villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models
Authors: Xiaoyu Chen
Hangxing Wei
Pushi Zhang
Chuheng Zhang
Kaixin Wang
Yanjiang Guo
Rushuai Yang
Yucen Wang
Xinquan Xiao
Li Zhao
Jianyu Chen
Jiang Bian
Venue: ICLR 2026 Poster (2026)
Keywords: embodied AI
Primary area: applications to robotics, autonomy, planning

Abstract: Vision-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent works have begun to explore the incorporation of latent actions, abstract representations of motion between two frames, into VLA pre-training. In this paper, we introduce villa-X a novel Vision-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies.
Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. We demonstrate that villa-X can generate latent action plans in a zero-shot fashion, even for unseen embodiments and open-vocabulary symbolic understanding. This capability enables villa-X to achieve superior performance across diverse simulation tasks in SIMPLER and on two real-world robotic setups involving both gripper and dexterous hand manipulation. These results establish villa-X as a principled and scalable paradigm for learning generalizable robot manipulation policies. We believe it provides a strong foundation for future research.

Published as a conference paper at ICLR 2026
VILLA-X, -X:: ENHANCING LATENT ACTION MODELING IN VISION-LANGUAGE-ACTOON MODELS
Xiaoyu Chen Hangxing Wei Pushi Zhang',, Chuheng Zhang', Kaixin Wang Yanjiang Guo", Rushuai Yang"t, Yucen Wang Xinquan Xiao Li Zhao * 8, Jianyu Chen', Jiang Bian'
Microsoff Research TTinnguu University Wuhan University
'Hong Kong University of Science and Technology
NNanjing University
ABSTRACT
Vision-Language-Actionn (VLA) models have emerged as popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent works have begun to explore the incorpo- ration of latent actions, abstract representations of motion between two frames, into VLA pre-training. In this paper, we introduce villa-X, novel Vision- Language-Laten Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. We demonstrate that villa-X can generate latent action plans in zero-shot fash- ion, even for unseen embodiments and open-vocabulary symbolic understanding. This capability enables villa-X to achieve superior performance across diverse simulation tasks in SIMPLER and on two real-world robotic setups involving both gripper and dexterous hand manipulation. These results establish villa-X as a principled and scalable paradigm for learning generalizable robot manipulation policies. We believe provides a strong foundation for future research.
INTRODUCTION
Latent action learning has emerged as a promising approach for the pretraining of vision-language- action (VLA) models (Collaboration et al. 2023 Black et al. 2024 Li et al. 2024a Chen et al.. 2024b Kim et al. 2024: Zhao et al. [2025] NVIDIA et al. 2025a), enabling learning from both robot data and human video data Ye et al.) [224|] Chen al. 2024b NVIDIA et al.) 2025a Bu et al.) 2025a). At the core of these methods a Latent Action Model (LAM), which is designed to capture the motion semantics between successive frames into compact latent tokens. These tokens, referred to as latent actions, serve as pseudo-action labels, enriching robot policy training by enabling imitation learning on abundant, action-free data.
While promising, the central challenge still lies in improving how latent actions can enhance robot policy learning- This motivates our investigation into two pivotal questions: how belier learn latent actions, and how to more effectively integrate them into VLA pre-training? In this paper, we introduce villa-X, a novel Vision Language-Latent-aeeee
(ViLLA) framework that advances both key aspects of latent action modeling. For the latent action learning component, existing latent action models (Ye al. 2024, Chen et al.) 2024b) NVIDIA et al.) 2025a Bu et al.) 2025a) typically compress latent actions based on visual signals, as shown in Figure D (a). However, while visual changes generally align with robot physical dynamics, certain motions, such as end-effector rotations or gripper movements, are subtle in pixel changes but critical for control. Vision-based models often pay less attention to these motions, a limitation also noted in recent work (Chen et al. (024a). As result, the learned latent actions remain physically ungrounded, hindering effective knowledge transfer. To address this, we move beyond purely visual signals by leveraging structural cues for physical grounding. Specifically, we integrate proprioceptive Forward Dynamics Model (proprio FDM) as an auxiliary decoder within our Latent Action Model (LAM), as shown in Figure (b). This module predicts future robot proprioceptive states and actions by including embodiment context as
"quual contribution
Interns at Microsoft Research
8 Project lead

---

Published as a conference paper at ICLR 2026
FON
FDW
IDWI
proprio
(8) Existing Latent Action Model
Action (b) Our Latent Action Model
Figure 1: (a) A standard Latent Action Model (LAM) learns a latent action &t primarily through visual reconstruction, predicting future frame 0t+ K from the current frame o, and latent action 2₁. (b) Our proposed model enhances this by adding proprio-FDM. FDM. This auxiliary module predicts future robot states +t+1:4+K and actions 0 conditioned on embodiment context Ce, enabling the latent actions to be better grounded in physical dynamics.
inputs to help distinguish heterogeneous data. As a result, the latent actions become better grounded by focusing on visual changes that align with the agent's physical dynamics. This makes the latent action a more effective bridge between vision and control, ultimately improving knowledge transfer. This framework is general and can be extended to other cues like end-effector keypoint detection or human hand posc estimation, which we leave for future work. To better leverage learned latent actions, we introduce a novel integration strategy for VLA pre-training. villa-X models latent and robot actions within joint diffusion framework composed of two components: latent action expert (ACT-lateet) and robot action expert (ACT-robot) as shown in Figure 2 Within this framework, an attention mechanism conditions robot action generation latent actions generation. Compared to existing methods, this framework facilitates a more effective and structured transfer of information.
We conduct comprehensive evaluations of villa-X across diverse environments. Our experiments yield two key findings. First, extensive ablation studies confirm that our proposed improvements to the latent action model and policy architecture outperform existing methods. Second, we show that by scaled pre-training, the latent action expert effectively plans for the future, and generalizes zero-shot to unseen embodiments and open- open-vocabulary symbolic icons. Collectively, villa-X achieves state-of-the-art performance in various tasks including numerous simulation tasks in SIMPLER and two real-world setups featuring various robotic platforms with both gripper and dexterous hand manipulators. This establishes robust foundation for future research in the field. In summary, our main contributions are as follows:
We improve latent action learning by introducing extra proprio FDM, which grounds latent actions in physical dynamics by aligning with underlying robot states and actions.
We propose jointly learn latent action expert and robot action expert through joint diffusion in the policy model, conditioning robot action prediction on latent actions to fully exploit their potential.
Through scaled pretraining, our latent action expert develops strong zero-shot generalization capabilities. This enables the effective transfer of knowledge across diverse simulated environments and real-world robotic tasks, leading superior performance.
2 RELATED WORK
Vision Maaion-Languag--cccton Models
Baaio-LLLanggg-AAcctin (VLA) models Black et al. 2024
Li et al. 2024a NVIDIA et al., 2025a Chen al. 2024b) Kim et al. 2024 Zhao 2025
et al. 2024c| Collaboration et al. 2023 leverage pre-trained vision-language models ( (VLMs) to generate robot actions using visual and language cues. They either directly repurpose VLMs for action prediction (Chen et al. 2024b; Ye et al. 2024; Li et al., [024cc Kim et al., 2024) or use action experts to map VL.M outputs to robot actions (NVIDIA et al.) 2025a Black et al. [024 Li
2024a). While training on large-scale datasets like Open X-Embodiment (Collaboration al. (023)
enhance the generalization ability of VLAs, cross-embodiment generalization remains challenging due to diverse robot setups and configurations. Utilizing unlabeled trajectory data with pseudo-labels such as latent actions Ye et al. 2024 Chen et al., 2024b), language sub-goals (Mu et al. 2023), or visual sub-go sub-goals (Zhao et al. 2025) supports overcoming these challenges. Our method advances the latent action framework by enhancing both the modeling of latent actions and their integration into VLA pre-training.
2

---

Published as a conference paper at ICLR 2026
attention
Re+
dz
dg+1
t
VLM
ACT-latent x
ACT-robot
Ot
O
Wrist
4t
Figure 2:
Architecture of ACT: hierarchical policy that predicts latent action plans and conditions robot action generation them, incorporating embodiment context and attention masking.
Modeling Latent Actions for VLA Prefraining Recent exploration into latent actions began with LAPO (Schmidt & Jiang 2023) and Genie (Bruce et al. 2024), which primarily focused on the video game domain. Dynamo (Cui et al. 2024) adopted a similar architecture, using inverse and forward dynamics models to shape state representations.
For robotic learning, methods have started to incorporate latent actions into VLA pretraining (Ye et al., 2024 Chen al., 2 b, NVIDIA et al. 2025a; Bu et al., 2025a b Yang al. 2025a 2025). LAPA (Ye et al. (024) proposes to learn from videos, and trains its latent actions and Vision- Language Model (VLM) using either human or robot video data. Concurrently, 2024a) learns latent actions from a mixture of human and robot videos, marking the first successful transfer of latent actions between humans and robots. Moto-GPT (Chen al.)
Liang et al.
IGOR (Chen et al.
2024b) co-fine-tunes both latent and robot action labels. GROOT (NVIDIA et al. 2025a) treats latent
actions distinct embodiment, while Go-1 (Bu al. 2025a) generates robot actions conditioned on discrete latent tokens. UniVLA (Bu et al., 2025b proposes a two-stage training pipeline to
learn task-centric latent actions. More recent works like Yang et al. (2025a): Liang al. (2025) explore the continuous latent actions. /Zhang et al. (2025) provides the analysis on the learned latent actions, while I.OOM (Nikulin et al., (2.55) uses supervision to learn latent actions that are robust distractors in MuJoCo environments. By contrast, approach jointly models latent and robot actions through a joint diffusion process, conditioning robot action generation on latent actions for more effective and structured information transfer. Our method improves upon prior work in several key aspects: it offers tighter integration than LAPA (Ye et al. 2024) and GROOT (NVIDIA) et al.) 2025a); incorporates immediate visual context, unlike Moto-GPP (Chen et al. (024b); and avoids teacher-forcing inconsistencies seen in Go-1 (Bu et al. 2025a). These advantages collectively contribute more robust reasoning test time.
METHOD
Our method, villa-X, learns a physically grounded latent action space and uses it to train a VLA policy. The framework has two components:
(i) Latent Action Model (LAM) infers latent actions from a pair of observations, aligning these latent actions with robot dynamics via additional proprioceptive supervision.
(ii) ACTor Module (ACT) builds on pre-trained vision-language backbone and jointly models sequences of latent and robot actions for improved planning and control.
Training proceeds in three stages: (i) LAM pretraining diverse datasets, (ii) ACT pretraining with joint latent-robot modeling, and (iii) embodiment-specifi finetuning.
3.1 LATENT ACTION MODEL (LAM)
Latent actions provide a compact intermediate representation, enabling the of abundant human videos and improving cross-embodiment generalization (Ye et al. 2024 Chen et al.) 20246). Prior works typically learn a quantized codebook of latent actions two modules: an Inverse Dynamics Model (IDM) and visual Forward Dynamics Model (FDM). The IDM predicts a latent token Zt from a frame pair (ot, ()-))))) k) while the FDM reconstructs the future observation ô+tK from (ot, 2)):
2t IDM(o 0++K), Ô FDM(o 2t).
(1)
This objective ensures consistency in visual change but ignores physical dynamics, producing latents that are insufficiently grounded when robot states available.
3

---

Published as a conference paper at ICLR 2026
Proprioceptive Grounding. To address this, we introduce additional proprioceptive Forward Dynamics Model (proprio-DDM that predicts both future robot states and actions steps ahead, given the current state 9, and latent 2z:
(... q+ K. d++1 k++k) = proprio-FDM(qq 21, Ce),
(2)
where C. denotes an embodiment context described below. Optimizing visual and proprioceptive forecasting jointly encourages latent tokens to emphasize physical dynamics alongside visual changes.
Disambiguating Heterogeneous Embodiments. Large-scale datasets mix embodiments with different morphologies and control frequencies. Naively conditioning the proprio-FDM. on (q., #)) risks encoding embodiment-speciin features into latents. We introduce context vector cg comprising:
Ce f (dataset ID, control frequency),
where dataset IDs are mapped to learnable embeddings, and frequencies are encoded using sinusoidal features passed through an MI... These embeddings, concatenated with qt, allow proprio-FDM to separate embodiment-specific dynamics while preserving latent action consistency across datasets.
The full LAM thus optimizes image reconstruction loss, proprioceptive prediction loss, and vector- quantization commitments jointly. For human video lacking proprio labels, the proprio term is omitted. Finally, we adopt the continuous vector from the VQ codebook center as our latent actions. We refer readers Appendix A for further implementation details.
In summary, our LAM extends prior latent action models beyond compressing only visual changes to jointly modeling physical state transitions. While this work leverages robot proprioception for grounding, the framework is generic: alternative structural cues like end-effector keypoint detection or human hand pose estimation could replace low-level states, which we leave for future exploration.
3.2 ACTOR MODULE (ACT)
Our ACT module extends traditional vision- sssion-language-ctton (VLA) approaches by explicitly mod- eling both latent actions (z ( (n- 1)K (2t, žt+K (x)((=)))))) and robot actions (a:=+m-11
(ac, O.-1:-.., 414m-1) with a joint diffusion process. We factorize the policy into two conditional distributions:
7 7(autm-11: (:::::ttnnnn-- (n-K) 1es, 9c. ce) = Trobot (0c:c+m-1
K Iit+ 1)K Oc, 4,, qe,
ACT-moool
Whstent
K (n- Ot:
ACT-latent
where O2 is the observation, the language instruction, qe the proprioceptive state, and ce the embodiment context. Additionally, the low-level policy can optionally incorporate wrist camera input if available. This explicit modeling and factorization improves upon prior methods, such as LAPA (Ye al. 2024), which rely latent actions only through pre-trained weight initialization. By contrast, our method treats latent actions as a distinct mid-level representation that bridges high-level vision and language prompts with low-level robot actions, and allow for allowing more effective and structured information transfer from latent actions to robot actions.
Architecture.
ACT (Figure comprises three experts with blockwise causal attention mask:
VLM: Encodes the visual- linguag inputs into high-level features.
ACT-latent:
Latent action expert that predicts latent action tokens for mid-level planning, conditioning on VLM features.
Robot action expert that produces the low-level action chunk, conditioning VLM features, predicted latents and additional control-specific inputs including propriocep- tive states and embodiment context.
ACT-robot:
4

---

Published as a conference paper at ICLR 2026
Attention Masking Strategies. A key aspect of ACT is how maintain a robust dependence on the latent tokens without letting the policy learn trivial shortcuts. Inspired by Moto (Chen et al.) 2024b) and RDT (Liu al., 2024), we adopted the stochastic Masking strategy. During training, we stochastically mask the attention from robot actions to latent actions. In 50% of the cases, all robot-to-latent attention masked; otherwise, 50% of the latent tokens randomly masked. This can prevent the robot-action branch from overly relying on latent actions and leading to improved robustness. We found this design erucial practice.
Joint Diffusion Modeling. For implementation, villa-X models the joint distribution of the future robot actions 0g:4m-1 and latent actions 2 (n= using conditional flow matching framework. For notational simplicity, we group these actions into a single variable 2; and denote the conditioning inputs (o,, qt, C.) O,. The objective is to train network vi that minimizes the flow matching loss:
(0) (..) | O.) ux 1x)]
(5)
where т E [o, 1] denotes flow matching timestep. In practice, we first sample random noise e ~ (0, 7) to create noisy target s/ T + 7)- The network (xr Ot) is then trained to predicted the denoising vector field u( xx) = Tt. During training, we sample t from beta distributions. Notably, the explicit factorization Eq. is achieved by block-wise causal attention.
EXPERIMENTS
In this section, we aim answer the following questions through experiments:
Does our improved LAM learn higher-quality latent actions?
Can the actor module effectively leverage the pre-trained latent actions?
By scaling pre- pre-training, can the latent actor module effectively plans for the future and generalize zero-shot unseen embodiments and open-vocabulary concepts in symbolic icons?
How does villa-X compare existing VLA baselines in both simulated benchmarks and real-world robot tasks?
4.1 DOES OUR IMPROVED LAM LEARN HIGHER- LATENT ACTIONS?
In this subsection, we evaluate whether our improved latent action modeling enhances the quality of the learned latent actions. The key component of our LAM is the incorporation of the proprio FDM module. To assess its impact, we compare our model (denoted w/ #/pp) to a variant without the proprio FDM module (denoted wo wo/pp).
Probing First, a core expectation for latent actions is that they should carry information useful for predicting low-level robot actions. To test this, we conduct a probing experiment. Specifically, after training the latent action models, we freeze them and train a simple 3-layer MLP predict the corresponding robot actions for each latent
ppp - ppppp
sample
200
action. Probing is conducted on the LIBERO dataset (Liu al. 2023a), which is not used for training latent action models. We train the MLP on the training split of LIBERO and evaluate it using the LI loss on the validation split.
sample
-20
01
07
03
We are interested in how closely the predicted actions match the ground truth. In LIBERO, the robot action space has eight dimensions: three for position, four for rotation, and one for the gripper. Rather than averaging the error across dimensions, we focus on the maximum LI error across all action dimensions, as we want to avoid large deviations in any single aspect of the action. For each model variant (w/ pp and wo/ pp), compute the number of validation samples whose maximum LI error falls below threshold. By sweeping this threshold, we count how many samples fall within each error bin. A better model should yield more samples with low errors.
Threshold
Figure 3: Probing experiment results.
5

---

method achieves significantly Published as a conference paper at ICLR 2026
Table 1: Evaluation results on SIMPLER for different variants of our villa-X(top group) and alternative approaches for incorporating latent actions (bottom group). "Ours" refers to the w w/pp described in the main text.
Google robot
WidowX robot
Method
Pick
Move
Drawer
Avg.
Carrot
Eggplant
Spoon
Cube
Avg.
Ours
81.7
55.4
38.4
58.5
24.2
71.7
48.3
19.2
40.8
wo wo/pp
77.0
52.7
42.6
57.4
22.5
57.5
43.3
5.9
32.3
wo LAM
42.1
24.6
38.4
35.0
25.8
60.8
36.7
9.2
33.1
LAPA-style
64.7
28.8
38.0
43,8
0.8
0.0
2.5
0.8
1.0
Go-l-style
44.3
46.3
41.2
43.9
7.5
91.3
30.4
16.7
36.5
For each error bin, compute the difference in the number of samples between the w/pp and wo wo/pp variants and present the results in Figure The w/pp variant produces more samples with smaller errors, while the wo/pp variant has more samples in the high-error bins. This demonstrates the effectiveness of the proprio FDM module in capturing information from the robot actions. We further visualize the learned latent actions, and perform more ablations on LAM. Please refer to Appendix for more details.
Policy Pre-training Next, we compare how the latent actions generated by different variants of LAM(w/PP) /pp and wo/pp) influence policy pre-training. Unlike the main experiments, we pre- train models in this section on mixture of 10% Fractal (Brohan al.] (022) data, 10% Bridge V2 (Ebert et al. 2021) data, and 100% Komething-Somettee computation cost while remaining a setting where limited robot data are available for training the VLA model. The resulting policies evaluated in the SIMPLER environment (Li et al.) 2024d), a simulation benchmark explicitly designed to mitigate the gap between simulated and real-world robotic environments. It comprises and the WidowX robot with four. We evaluate our method on the visual matching setting. The results summarized in Table [ effectiveness of incorporating the proprio FDM module. Additionally, we include a baseline that does use latent actions (denoted wo LAM) and is trained solely predict robot actions. The performance of wo LLM is significantly worse, indicating that pre- pre-training with latent actions is essential.
V2 (Goyal et al. 201 7a) data, to reduce
platforms: the Google robot with three manipulation tasks
We observe that w/pp pp clearly outperforms wo/pp, demonstrating the
4.2 CAN THE ACTOR MODULE EFFECTIVELY LEVERAGE THE PRE-TRAINED LATENT ACTIONS?
Given high-quality latent actions produced by the pre-trained LAM, investigate whether design can effectively leverage them to pre-train robot control policies. We compare our approach against two recent methods that also utilize latent actions, albeit in different ways: LAPA Ye et al. 2024
and GO-1 (Bu et al. 2025a).
To isolate the effect of how latent actions incorporated, implement LAPA-style and GO-1-style models based on our architecture for a fair comparison. For the LAPA-style model, we follow two-stage pre-training protocol: we first train the VLM to predict latent actions, then replace the latent action prediction head with robot action prediction head and continue training on data with robot action labels. For the --syle model, we implement separate latent planner that autoregressively predicts latent actions. The robot action prediction component remains largely unchanged in main design.
Following the experiment setup in the previous subsection, train all models on the same dataset mixture and then evaluate the resulting policies the SIMPLER environment (Li al. (024d). The results shown in Table Compared to other two approaches, higher performance, validating the effectiveness our design for incorporating latent actions into VLA pre-training. More ablation studies on policy designs can be found in Appendix

---

Published as a conference paper at ICLR 2026
Touch the Touch the bowl
Touch the leaf
Touch the hotdog
Touch the hotdog
Touch the hotdog
Visualization of zero-shot latent plans unseen embodiment. Each pair of images shows the starting frame (left) and the ending frame (right), with the instruction displayed above.
Figure 4:
4.3 ZERO-SHOT GENERALIZATION FOR LATENT ACTOR MODULE
To evaluate the zero-shot generalization capabilities of ACT-latent planning, we conducted a real-world visualization experiment focused on its ability handle new embodiments and understand novel open-vocabulary symbols. For embodiment generalization, we used Realman robot arm, a new embodiment never seen during training. To assess open-vocabulary generalization, designed a set of symbol cards, testing the model's ability to comprehend concepts typically absent from standard robotics datasets.
The evaluation process is as follows: given a starting image and a language instruction (e.g., "touch the corn"), ACT-latent first generates a sequence of latent actions. Then separately trained world model is used to render this sequence into video, allowing us to verify the effectiveness of the plan.
As shown in Figure 4 the rendered trajectories confirm that the model successfully generates latent plans that follow the commands. These results highlight two key capabilities of our approach:
Embodiment Generalization: ACT-latent successfully identifies and controls this unseen robot arm, indicating that its learned knowledge is embodiment-agnostic and readily trans- ferable to the new robotic platform.
Vocabulacy Understanding: The model's ability to interact correctly with symbol concepts reveals that villa-X retains the general- purpose vision- languaea capabilities of the original Visiua-Language Model (VLM) after pre-training.
More visualizations can be found in Appendix E
To evaluate how effectively villa-X uses this knowledge, we measure its success rates on a variety of control tasks in the following sections.
4.4 EVALUATING VILLA-X IN SIMULATION
Baselines and Experimental Setup We use the SIMPLER benchmark as described above. this section, we compare against several categories of prior work:
Vision-Languagg-Action (VLA) models: RT-1-X (Collaboration et al., 2023 Octo-
base (Octo Model Team et al. 2024), OpenVLA (Kim al. (024), Robo V MMs (Li et al.
2024b), To (Black et al. 2004). To- FAST (Pertsch 2025), OpenVLA-OFT (Kim et al.
2025), which learn policies solely from mixed robot datasets.
Joint policy learning and world modeling method: GROOT-NI.S NVVIDI et al. 2025b
which aligns the model with target future embeddings.
Visual trace methods: VLA (Zheng et al. 2024), Magma YYagg et al. learn plamning on the extract visual traces from videos.
2025b), which
Latent-Action based methods: MoTo (Chen et al., 2024b) and LAPA YY al
2024 which
additionally exploit unlabelled videos by inferring latent actions.
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparison on SIMPLER of villa-X and existing methods. Methods marked with are evaluated directly after pretraining, whereas other methods are evaluated after post-training on corresponding dataset.
Google Robot
WidowX X Robot
Method
Pick
Move
Drawer
Avg.
Carrot
Eggplant
Spoon
Cube
Avg..
RT-1-X
56.7
31.7
59.7
49.4
4.2
0.0
0.0
0.0
1.1
Octo-base *
17.0
4.2
22.7
14.6
8.3
43.1
12.5
0.0
16.0
VLA
16.3
46.2
35.6
32.7
0.0
4.1
0.0
0.0
1.0
Robo
72.7
66.3
26.8
55.3
25.0
0.0
20.8
8.3
13.5
RoboVLMs
77.3
61.7
43.5
60.8
20.8
79.2
45.8
4.2
37.5
To
72.7
65.3
38.3
58.7
0.0
62.5
29.1
16.6
27.1
To- FAST
75.3
67.5
42.9
61.9
21.9
66.6
29.1
10.8
32.1
Open VLA-OFF-
72.3
69.6
47.2
63.0
4.2
N/A
12.5
8.3
N/A
GROOT-NI.5
69.3
68.7
35.8
57.9
54.3
61.3
75.3
57.0
62.0
TraceVLA
45.0
63.8
63.1
57.3
16.6
65.0
12.5
16.6
27.7
Magma
75.0
53.0
58.9
62.3
29.2
91.7
37.5
20.8
44.8
MoTo
74.0
60.4
43.1
59.2
N/A 45.8
N/A 58.3
N/A 70.8
N/A 54.2
N/A
LAPA
N/A
N/P.
N/A
N/A
57.3
Ours w/o latent
56.3
25.8
27.3
36.5
31.3
74.6
61.7
28.3
49,0
Ours
98.7
75.0
59.3
77.7
46.3
64.6
77.9
61.3
62.5
Except where noted (+), all models follow two-stage pretraining-finetuinng protocol. including general pretraining phase on large-scale mixed data, followed by finetuning on a dataset of specific embodiment. We also include ablation (Ours w/o latent) that removes latent-action expert while keeping all other components unchanged. Baseline scores are cited from their original publications or other relevant literature, while missing entries are marked N/A.
Experimental Results Table summarizes the success rates on both platforms. Our full model achieves the highest score on average success rate on the Google robot (77.7% and the WidowX robot (62.5%). This improvement over VLA methods, which cannot exploit unlabelled video, demonstrates the benefit of incorporating human videos into policy learning. Moreover, our approach outperforms other video learning and latent-action methods, indicating that specific mechanism for leveraging video data more effective. Finally, the gap between our full model and the "villa-X w/o latent" ablation confirms that the latent-action expert is essential for achieving these gains.
4.5 EVALUATING VILLA-X ON REAL-WORLD ROBOTS
To assess real-world generalization, we deploy villa-X on two platforms: a Realman arm with gripper and XArm with 12-DoF XHand shown in Figure
Realman robot arm with gripper We use a 6-DoF Realman RM 75 with a 1-DoF Inspire gripper, fine-tuning and evaluating on five tasks: Pick-in (pick the block into bowl), Pick-out (pick the block out of bowl), Stack (stack the block onto another block), Unstack (unstack the block from another block), and Push (push the block to given location). The fine-tuning set contains 375 teleoperated trajectories (75 per task); the object layout and table are fixed, while object positions vary.
We conduct two sets of evaluation: In task evaluation, we remain the table setup the same as data change the color of the block and table cover. For each
collection; in generalization evaluation, task, we conduct 10 trials with distinct object positions; positions and lighting are identical across policies. As shown in Table 4 villa-X outperforms all baselines in both settings. Rollout videos are available the Appendix.
Xarm robot arm with Xhand dexterous hand On the dexterous-hand platform, the Xhand, a 12-DoF dexterous hand with five flexible fingers, mounted on a 7-DoF Xarm robot arm. Fine-tuning performed the Xhand Dataset (Hu et al. 2024), which comprises 4,000 trajectories spanning
8

---

Published as a conference paper at ICLR 2026
Arm Pletform
Pick in
Pick out
Push
Stack
Unstack
Dexterous Hand Platform
Figure 5: Real-world robot evaluation platforms: (top) Realman robot arm platform with gripper and (bottom) Xarm robot arm with Xhand dexterous hand. Platform setups are shown on the left, with corresponding evaluation tasks the right.
Table 3: Evaluation Xarm robot arm of villa-X and existing methods.
Pick & Place
Stack Cube
Place Cup Upright
Pour Water
Flick Ball
Method
seen
unseen
seen
unseen
seen
unseen
seen
unseen
seen
unseen
GR-1
56
40
15
0
0
0
40
10
GROOT
44
28
20
0
Ours w/o latent
72
60
70
40
40
30
40
10
50
30
Ours
84
68
75
50
60
30
60
30
50
40
Table Evaluation on Realman robot arm of villa-x and existing methods.
Method
Pick in
Pick out
Push
Stack
Unstack
Change block color
Change table
40
70
60
50
70
40
40
GROOT
30
70
10
10
60
50
30
Open VLA-OFT
50
70
10
50
80
40
50
Ours w/o latent
40
80
30
60
70
40
30
Ours
30
100
50
50
100
60
60
13 task categories. Since dexterous-hand data were used during pretraining, this evaluation
test embodiment transfer ability. We select five representative tasks-pick-and-pee cup upright placement, water pouring and ball flicking. The results summarized in Table for (i) seen tasks, where objects are randomly replaced or additional distractors are added, and (ii) unseen tasks, which use unseen objects backgrounds. The performances are evaluated under 50 runs for pick and place, 20 runs for stack cube and 10 runs for others. demonstrates that our method outperforms existing baselines.
cube stacking,
5 CONCLUSION, LIMITATIONS, AND FUTURE WORKS
In this paper, we presented villa-X, novel Visual-Lanne Languagg--Lateet--annne (ViLLA) framework that improves both the learning of latent actions and their incorporation into VLA pre-training. Our experiments demonstrate that our enhanced Latent Action Model learns higher-quality latent actions, and our improved policy model more effectively leverages these learned latents. The learned latent action expert can even generalize zero-shot to an unseen embodiment, showing strong generalization ability. Overall, our method exhibits superior performance in both simulated environments and real-world robotic tasks. One limitation that the proposed latent expert, although effective at future planning through both visual and proprioceptive state planning, is not fully explored in this work. For example, future research could learn a critic with prior knowledge from foundation vision-language models, allowing multiple samples from the latent expert and rejecting planned trajectories that do not follow the language instruction. We leave this aspect as future work to further improve the capability of the ViL.A framework.

---

Published as a conference paper at ICLR 2026
6 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our research, we provide comprehensive details of our methodology and implementation. thorough description our model architecture, experimental setup, and evaluation protocols can be found in Section - Additional implementation details required to reproduce our main results are documented in Appendix. The source code included in the supplementary material.
REFERENCES
Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Hydra: Hybrid robot actions for imitation learning. arxiv, 2023.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Fisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong. Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: A versatile 3b vlm for transfer, 2024. URL https://arxiv.or
abs 2 205/2407.077 07726
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri. Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. Toi A vision-language-action flow model for general robot control. arXiv preprint arXiv: 2410.24164, 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, K. Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, A. Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Levine, Yao Lu, U. Malla, D. Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, M. Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, S. Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Q. Vuong, F. Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale. Robotics: Science and Systems, 2022. doi: 22.48550/arXiv:22222666
Jake Bruce. Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng. Shenyuan Gao, Xindong He, Xu Huang. Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang. Yu Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen, Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song. Huijie Wang. Wenhao Wang, Dafeng Wei, Chengen Xie, Guo Xu, Junchi Yan, Cunbiao Yang, Lei Yang. Shukai Yang. Maoqing Yao, Jia Zeng, Chi Zhang, Qinglin Zhang, Bin Zhao, Chengyue Zhao, Jiaqi Zhao, and AgiBot- SgiBot-Worlde Sorld-Contributors Jianchao Zhu. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv: 2503.06666 2025a.
Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao. Ping Luo, and Hongyang Li. Univla: Learning to act anywhere with task-centric latent actions, 2025b. URL https: littps://arxive
htt///xivv.org/bs///////// 2505 06111
Lawrence Yunliang Chen, Simeon Adebola, and Ken Goldberg. Berkeley UR5 demonstration dataset. https://s goog http::///w...t.oooo..e..cc.////a
10

---

Published as a conference paper at ICLR 2026
Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Cathera Yang, Li Zhao, and Jiang Bian. Igor: Image-goat representations are the atomic control units for foundation models in embodied ai. arXiv preprint arXiv:2411.0000
2024a.
Yi Chen, Yuying Ge, Yizhuo Li, Yixiao Ge, Mingyu Ding, Ying Shan, and Xihui Liu. Moto: Latent motion token as the bridging language for robot manipulation. arXiv preprint arXiv: 2412.04445, 2024b.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241273666, 2023.
Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acom Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung. Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky. Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su, Hao-Shu Fang. Haochen Shi, Henghui Bao, Heni Ben Amor. Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang. Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang. Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go. Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart in-Mart in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu. Tianli Ding, Todor Davehev. Tony Z. Zhao, Travis Annstrong. Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang. Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open Embodiment: Robotic learning datasets and RT-X models. https: arxiv org abs 2310. 08864 2023.
11

---

Published as a conference paper at ICLR 2026
Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210.1004/ 2022.
Zichen Jeff Cui, Hengkai Pan, Aadhithya Iyer, Siddhant Haldar, and Lerrel Pinto. Dynamo: In- domain dynamics pretraining for visuo-motor control. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10- 15, 2024, 2024, http:tt::// 3b8db54b62
nips wtt.ippp..pcppppeeeeeeeeeeeeeee wwwpaperppppppppccppppptteee...e Lacreer___lleeee
/ html]
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. The epic-kitchens dataset: Collection, challenges and haselines. IEEE Transactions on Puttern Analysis and Machine
Intelligence, 43(11::42254411 2020.
Shivin Dass, Jullian Yapeter, Jesse Zhang, Jiahui Zhang, Karl Pertsch, Stefanos Nikolaidis, and Joseph J. Lim. CLVR jaco play dataset, 2023. URL http:t/p///ggittb...cm///aaaaaaa
jaco play dataset
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danillidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109. 13396, 2021.
Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Junbo Wang. Haoyi Zhu, and Cewu Lu. Rh20t: A robotic dataset for learning diverse skills in one-shot. In RSS 2023 Workshop on Learning for Task and Motion Planning, 2023.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Frrittgg Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017a.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyúska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Stueller-Freitag Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense, 2017b. URL http:///xxvv.rrgg///// abs/
1706. 04261
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao Siddhant Bansal, Dhruv Batra, Vincent Car- tillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li. Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullic Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo. Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the World in 3,000 Hours of Egocentric Video. In IEEE/CVF Computer Vision and Pattern Recognition (CVPR), 2022a.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022b.
12

---

Published as a conference paper at ICLR 2026
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL httts:://axxiy> wwtp:/iwwwrg/aa////a3a 03385
Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J. Lim. Furniturebench: Reproducible real-world benchmark for oong-horizon complex manipulation. In Roboties: Science and Systems, 2023.
Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv: 248, 2024.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 1002. PMLR, 2022.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. In CoRL, pp. 651 2018.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Gua- Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang. Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph Lim, Jitendra Malik, Roberto Martin, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: large-scale in-the-wild robot manipulation dataset. 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source lision-anguuage-action model. arXiv preprint arXiv:2446.09922 2024.
Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-languagg-action models: Optimizing speed and success. arXiv preprint arXiv:2502. 2ee225222999944 19645, 2025.
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19650 2024a.
Xinghang Li, Peiyan Li, Minghuan Liu, Dong Wang, Jirong Liu, Bingyi Kang, Xiao Ma. Tao Kong, Hanbo Zhang, and Huaping Liu. Towards generalist robot policies: What matters in building models. arXiv preprint arXiv:2412.1 14058, 2024b.
aasion-language-ace
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, and Tao Kong. Vision-language foundation models as effective robot imitators. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024c. URL 1GG
https:://pennre
ew net www.tw/fooummttt////// (efooommid=@Ff.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Oier Mees, Karl Pertsch, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sich, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real- world robot manipulation policies in simulation. In Pulkit Agrawal,
13

---

Published as a conference paper at ICLR 2026
Oliver Kroemer, and Wolfram Burgard (eds.), Conference on Robot Learning. 6-9 November 2024, Munich, Germany, volume 270 of Proceedings of Machine Learning Research, pp. -728-.788. PMLR, 2024d.
mmr wttpe///e////1/2.....c..
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies simulation. arXiv preprint arXiv:2405.059441 2024e.
Yiming Li, Ziang Cao, Andrew Liang, Benjamin Liang, Luoyao Chen, Hang Zhao, and Chen Feng. Egocentric prediction of action target in 3d. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European conference on computer vision (ECCV), pp. 619-635, 2018.
Anthony Liang. Pavel Czempin, Matthew Hong, Yutai Zhou, Erdem Biyik, and Stephen Tu. Clam: Continuous latent action models for robot learning from unlabeled demonstrations. arXiv preprint arXiv:2505.04999, 2025.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning, arXiv preprint arXiv:2306.03110, 2023a.
Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, and Yuke Zhu. Robot learning on the job: Human-in-the-logg autonomy and learning during deployment. In Robotics: Science and Systems (RSS), 2023b.
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-Ib: diffusion foundation model for bimanual manipulation. arXiv preprint arXiv: 2410.07864, 2024.
Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21013 June 2022.
Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Picter Abbeel, and Sergey Levine. Fmb: functional manipulation benchmark for generalizable robotic learning. arXiv preprint arXiv:2401.0e. 008553, 2024.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence, Interactive language: Talking robots real time. IEEE Roboties and Automation Letters, 2023.
Oier Mees, Jessica Borja-Diaz, and Wolfram Burgard. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023.
Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. CoRL, 2023.
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang. Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 46:25081-250999 2023.
Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for skill-based imitation learning. In Conference Robot Learning (CoRI.), 2022.
Alexander Nikulin, Ilya Zisman, Denis Tarasov, Nikita Lyubaykin, Andrei Polubarov, Igor Kiselev, and Vladislav Kurenkov. Latent action learning requires supervision in the presence of distractors, 2025. URL wtt.tttts::/ ptatps://arxie abb /20.lb//22222.00 00.37
14

---

Published as a conference paper at ICLR 2026
NVIDIA, :. Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da. Runyu Ding, Linxi "Jim" Fan. Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang. Zu Wang, Jing Wang. Qi Wang. Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. Gr00t nl: An open foundation model for generalist humanoid robots. arXiv preprint arXiv: 2503. 2025a.
NVIDIA, :. Johan Bjorck, Fernando Castañeda, Nikita Cherniade,, Xingye Da, Runyu Ding. Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang. Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany. Scott Reed, You Liang Tan. Guanzhi Wang. Zu Wang. Jing Wang, Qi Wang. Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhen- jia Xu. Sconghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. GrO0t n1: An open foundation model for generalist humanoid robots, 2025b. URL org/ abss//////////e abtp//03/////3///3 2503. 14734
https://arxs
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo. Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open- source generalist robot policy, In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
Baoqi Pei, Yifei Huang, Jilan Xu, Guo Chen, Yuping He, Lijin Yang. Yali Wang. Weidi Xie, Yu Qiao, Fei Wu, and Limin Wang. Modeling fine-grained hand-object dynamics for egocentric video representation learning. 2025. URL htttttps:/// /attps://arxiv.ortt http://awww....oo///////333 00986
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision- shiio-languagc-action models. arXiv preprini arXhv:2501.09747 2025.
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, Jia Yuan Gu, Bin Zhao, Dong Wang, and Xuelong Li. Spatialvla: Exploring spatial representations for visual-language-attoon model, 2025. URL
wwg. abs/ 2201 15830
Gabriel Quere, Annette Hagengruber, Maged Iskandar, Samuel Bustamante, Daniel Leidner, Freek Stulp, and Joem Vogel. Shared Control Templates for Assistive Robotics. In 2020 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp- 7, Paris, France, 2020.
Allen Z. Ren. open-pi-zero: Re-implemennaatoon of no vision lanbunn-anggag--acttinn model, 2025. URL httpp:///githuu.comm/aalleee
cecenn-pi-zenne zero
Erick Rosete-Beas. Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task agnostic offline reinforcement learning. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
Dominik Schmidt and Minqi Jiang. Learning to act without actions. arXiv preprint arXiv:2312.10812. 2023.
Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto. On bringing robots home, 2023.
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
Jikai Wang. Qifan Zhang, Wei Chao. Bowen Wen, Xiaohu Guo, and Yu Xiang. Ho-cap: A capture system and dataset for 3d reconstruction and pose tracking of hane-object interaction, 2024a. URL https: / http///wwxvv.or///.ooaao wttp///////a///// 2268/3666333333333333
15

---

Published as a conference paper at ICLR 2026
Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive-visual learn- ing with heterogeneous pre-trained transformers. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural In- formation Processing Systems, volume 37, pp. 124420-124450. Curran Associates, Inc., wttppppperrllleepppprrreeeeee 2024b. URL https::/ppooceeediiiiit Satpaaddaaatt--pppppreeee- Conferenceeeecccomeeeoeeeeoeee
mwurippee ps Confrr-eeeeeeeeeeeeeeeeeeeee pdf
Xin Wang. Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Mare Pollefeys. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20270-20281, Octoher 2023.
Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, and Hongkai Xiong. Spatial-temponts transformer networks for traffic flow forecasting. arXiv preprint arXiv: 2001.02908, 2020.
Jiange Yang. Yansong Shi, Haoyi Zhu, Mingyu Liu, Kaijing Ma, Yating Wang, Gangshan Wu, Tong He, and Limin Wang. Como: Learning continuous latent motion from internet videos for scalable robot learning, 2025a.
http::::/aaxxv..rgggg
/2055 17006
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijic Zheng. Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Joel Jang, et al. Magma: A foundation model for multimodal ai agents. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14203-14214, 2025b.
Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Man- dlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, and Minjoon Seo. Latent action pretraining from videos. arXiv preprint arXiv: 1410.1/758 2024.
Chuheng Zhang, Tim Pearce, Pushi Zhang, Kaixin Wang, Xiaoyu Chen, Wei Shen, Li Zhao, and Jiang Bian. What do latent action models actually learn?, 2025. https: org/
abs abs/2506.25 /2///.5066....
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma. Song Han, Chelsea Finn, Ankur Handa, Ming-Ya Yu Liu, Donglai Xiang, Gordon Wetzstein, and Tsung-Yi Lin. Cot-vla: Visual chain-of-thought reasoning for vision-language- action models. arXiv preprint arXiv: 2503.22020 2025.
Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. arXiv preprint arXiv:2412. 2024.
A ADDITIONAL IMPLEMENTATION DETAILS FOR LAM
In this appendix, we provide extended information on the architecture, training protocols, and inference behavior for our Latent Action Model (LAM).
A. ARCHITECTURE OVERVIEW
Our LAM comprises four main modules:
(i) Spatial-Temporall Transformer (ST-Transformer) Inverse Dynamics Model (IDM): Takes a video clip (by default, 224 x 224) as input. We employ patch embedding with a patch size of 14 and stack 12 ST-blocks (Xu et al. 2020), each with hidden dimension of 768 and 32 attention heads.
(ii) Vector Quantization (VQ) Module: Maps the continuous IDM outputs to discrete latent tokens, each associated with codebook entry. We set the codebook size to 32. While the model internally uses discrete token indices during training. the continuous codebook centers are used in downstream modules.
16

---

Published as a conference paper at ICLR 2026
(iii) Image Reconstruction Forward Dynamies Model (FDM): A 12-layer Vision Transformer (ViT)-base network that takes the current frame or and a latent action 2, to predict ō (iv) Proprioceptive Forward Dynamics Model (proprio-PDM): A 2-layer MLP with dual output heads to predict future robot states 4++5 and low-level robot actions uz+i- This module takes the current robot state gr. the latent 2,, and an embodiment context vector Ce.
Rather than predicting a single latent action per pair of frames, the ST-Transformer- bassed IDM processes sequence of TiAM frames, resulting in Tiam 1 latent tokens. We TIAM = 8. By reconstructing future frames with the FDM and future states/actions with the proprio-FDM, the model learns latent representation that is both visually and physically grounded.
A.2 TRAINING DETAILS
We train our LAM combination of human egocentric data (e.g., Ego4D (Grauman et al. 2022a and robot trajectories (e.g., OpenX (Collaboration et al. 2023)). Samples lacking low-level robot annotations (e.g., human videos) exclude the proprio-Fi objective.
-Dppro--DDMM-- FDM branch, using only the visual FDM
Hyperparameters. We use a batch size of 512 and learning rate of 1.5 x 10-4, with 2000-step linear warmup. Training lasts approximately days on 128 NVIDIA A100 GPUs. Both the visual FDM and proprio-FDM share the same weighting in the overall loss. Throughout training, each latent token is discretized via the VQ module but represented by its continuous codebook center in subsequent network components.
A.3 INFERENCE BEHAVIOR AND DIAGNOSTICS
During inference, only the IDM is required to extract latent tokens from consecutive frames. The FDM and proprio-FDM are typically retained for diagnostic and visualization purposes, allowing us to examine whether the learned latent tokens accurately capture future frame content, robot states, and actions. This reconstruction- based analysis aids in understanding and debugging the physical grounding of the latent representation.
B ADDITIONAL IMPLEMENTATION DETAILS FOR ACTOR MODULE
Our VLA model comprises three components. First, the vision- language encoder is based on PaliGemma (Beyer et al. 2024), 3B 3B-parameter VLM pretrained with 224 x 224 images and 128-token text inputs. Second and third, the latent-action expert and the robot-action expert are each implemented as 18-layer Transformer networks, mirroring PaliGemma's design, with a hidden dimension of 1,024 and 8 attention heads. For the latent action sequence, we select a sequence length of N = 6, and for the robot actions, select sequence length of M 4.
We extend our policy head with a variant of HPT Wang et al. 2024b). assigning each embodiment its own pair of state- and cection-projectinn layers while sharing all other parameters. Visual features from the wrist camera are extracted by a pretrained ResNet-18 (He et al. 2015) and fused into the main model via shared cross attention head that maps the ResNet features into 16 tokens. During training, wrist-view inputs are randomly masked 50% of the time. We also observed that the latent-action representation can be overly exploited by the robot-action expert, so we regularize this with two complementary dropout schemes. First, we add a 50% attention-weiiht dropout on the latent-action stream. For the remaining tokens, we randomly mask 50% latent action tokens. This combined masking strategy encourages the model to learn robust, generalizable policy that will balance the predicted latent actions as well as the input image and instruction. During training, we sample r from different beta distributions for latent actions and robot actions, which biases the timesteps for latent actions towards the noisier regime. Each expert contains approximately 300 M parameters and is trained from scratch. We train all components jointly using learning rate of 5c with 200-step linear warmup. We clip gradients to a maximum norm of 1.0 to ensure stable optimization. The pretraining takes 4 days on 64 NVIDIA 1100 GPUs.
17

---

Published as a conference paper at ICLR 2026
C DATASET
C.I DATA MIXTURE
We curated data mixture by combining both robot data and action-free human videos for our pretraining phase. For robot data, we draw primarily from OpenX (Collaboration et al. 2023) mixture and AgiBot (Bu et al. 2025a). For OpenX dataset, our base data mixture is created primarily based on (Kim et al. 2024, Octo Model Team et al.) (024). In total, we use 1.6M trajectories with 223.5M
frames of robot data. For human videos, we use a mixture of Ego4D (Grauman et al., 2022a),
EgoPAT3D (Li et al. 2022), EGTEA Gaze+ (Li et al.
2018), EPIC-KITCHENS (Damen et al. 2020
HO-Cap Wang et al. 2024a), HO14D (Liu ct al. [022 HoloAssist (Wang et al. 2023). RH20T (Fang
et al. 2023), Something Something V2 (Goyal al. 2017b). Altogether, this yields 3.6M clips of
human videos. During LAM pretraining, we exclusively utilize the primary third-person camera view.
For policy pretraining, optionally incorporate the wrist-mounted view (when available), applying
a 50% dropout. A full breakdown of our data mixture is listed in Table
C.2 DATA PREPROCESSING
For data cleaning, adopt EgoHOD (Pei et al. 2025), curated subset of Ego4D (Grauman ctt al. 2022a)). and further filter the videos based on visual quality to ensure high-quality inputs for training. For both robot data and human videos, apply random adjustments brightness, contrast, saturation, and hue as data augmentation. the case of robot data, we represent both proprioceptive states and actions using euler angles.
Dataset
Mix Ratio (%)
RT-1 Robot Action (Brohan et al. 2022
9.70
AgiBot World Beta (Bu et al. 2025a
20.0
Kuka (Kalashikkov al. 2018
1.97
Bridge (Walke et al.] 2023 Ebert et al.
2021
5.47
Taco Play (Rosete-Beas al. 2022 Mees et
2023)
0.76
Jaco Play (Dass et al. 2023)
0.12
Berkely Autolab URS (Chen et al.
0,31
Language Table (Lynch et al. 2023
0.11
Stanford Hydra Dataset (Belkhale et al.
2023
1.61
NYU Franka Play Dataset (Cu et 2022
0.22
Furniture Bench Dataset (Heo et al. 20233
0.63
Austin Sailor Dataset (Nasiriany et 2022
0.57
Austin Sirius Dataset (Liu et al. 2023b
0.45
BC-Z (Jang al. 2022
3.47
DLR EDAN Shared Control Quere et al.
2020
0.01
CMU Stretch (Medoncaa et al. 2023
0.04
FMB Dataset (Luo et al. 2024
0.73
DobbE Shafiullah et al. 2023
0.37
DROID (Kaazatsky et al. 2024
3,46
Ego4D
Grauman et al.
20226 Pei et
al.
2025
21,46
3goPAT3D Li al.
2022
0.94
EGTEA Gaze+ (Li el al. 201
0.89
EPIC-KITCHENS (Damen al.
2020
6.95
HO-Cap Wang et al 2024a)
0.63
HOI4D (Luu el 2022
1.99
HoloAssist (Wagg el al.
2023
4.77
RH20T Fang et 2023)
5.56
Something-SSmethinn V2 Goyal et al.
2017a)
6.82
Table 5: Our training data mixture used during the pretraining phase.
18

---

Published as a conference paper at ICLR 2026
Grasp Things
Pick Things Up
Move Left
Move Right
Figure 6: Visualization of image pairs with similar latent actions.
D LAM VISUALIZATION AND MORE ABLATIONS
D.1 IMAGE PAIRS WITH SIMILAR LATENT ACTIONS
Figure 6 visualizes image pairs sharing the same latent action, demonstrating that these pairs corre- spond to similar underlying robot behaviors.
The results demonstrate that similar latent actions represent the similar robot behaviors and low-level actions, in regardless of which embodiment (including human and different robots) is executing such action. This results support that villa-X learns cros--ebbodiment prior knowledge for manipulations with latent actions.
D.2 TRANSFER VIDEO DEMONSTRATIONS INTO ROBOT ACTIONS THROUGH LAM AND PROPRIO FDM
To further demonstrate the transfer ability of our LAM, extract latent actions from videos of task demonstrations, map them to robot actions using the proprio FDM, and execute the resulting robot actions in the SIMPLER simulator.
The results are presented in Figure and Figure demonstrations used by LAM extract latent actions, while the bottom displays the corresponding SIMPLER simulation results, where real actions decoded from the latent actions using proprioceptive FDM are executed. Specifically, Figure illustrates robot-to-robot transfer, and Figure 8 illustrates to-boot transfer. The simulated motions closely reproduce the original demonstrations. indicating that latent actions learned by villa-X are both aligned with and grounded in the robot's actions.
each figure, the top row shows the video
iillutrallss
D.3 MORE ABLATIONS ON LAM
To validate the contribution the embodiment context in our proprio-FDM, we further conducted an ablation study comparing our full method ("Ours"" against a version without the context ("Ours w/o context"). Both models were trained on 10% of the OXE dataset.
(1) Performance validation dataset: We measured the reconstruction loss of visual FDM and proprio FDM on the validation set:
(2) Zero-Shot Generalization to Novel Embodiment: We evaluated the model on our dataset collected on our Realman robot arm dataset (from Section 4.4), an embodiment completely unseen during training. We then conducted the action probing experiment described in Section 4.1 by inferring latent actions with IDM and training a new MLP to predict robot actions from the latent actions.
19

---

Published as a conference paper at ICLR 2026
M
Original Video
SIMPLER Rollout
Original Video
SIMPLER Rollout
Figure 7: Transfer robot video demonstrations into robot actions through LAM and proprio FDM in SIMPLER simulator. Upper: the SIMPLER rollout closely reproduce the motion of moving downwards, Bottom: the SIMPLER rollout closely reproduce the motion of moving right.
Method
Visual FDM loss (L)
Proprio FDM loss (3)
Ours w/o context
0.068
0.078
Ours
0.057
0.070
Relative improvement
16.2%
10.3%
Table 6: Performance comparison on the validation set.
Method
Probing loss ())
Probing loss (xyz) (1)
Probing loss (rot) (L)
Probing loss (gripper) (1)
Ours w/o context
0.165
0.0675
0.00861
0.928
Ours
0.152
0.0574
0.00619
0.873
Relative improvement
7.9%
15.0%
28. 1%
5.9%
Table 7: Zero-shot generalization to an unseen embodiment.
The results from both experiments demonstrate that the embodiment context improves performance and aids generalization to novel embodiments. We hypothesize that while the visual FDM provides general transfcrability by aligning latent actions with visual changes, the proprio-FDM grounds these latent actions in robot physical dynamics. However, due data heterogeneity (e.g., different action
20

---

Published as a conference paper at ICLR 2026
definitions / controllers, discussed previously), the model requires the embodiment context to disambiguate different embodiments and learn a more consistent, grounded latent action space.
E LATENT ACTION EXPERT VISUALIZATION
In this experiment, we demonstrate the performance of the latent expert by passing its prediction through the image reconstruction FDM that takes the latent action as inputs and predicts the future observations, which forms simulated environment for the iteratively executing the latent expert.
Starting from a single initial image, the latent expert and image reconstruction FDM jointly generate different behaviors in videos that follow diverse instructions using only latent actions. We experiment with initial images from RT-1 and Bridge dataset, and show the image clips of generated videos in Figure and Figure T0 with different language instructions. The results show that the latent expert properly follows the language instructions for task solving, where the latent expert properly recognizes the target objects and predict latent actions that move towards the target object.
F MORE ABLATIONS ON POLICY MODEL
We primarily conducted ablation studies on two main components: (1) the attention mask and (2) the embodiment context. Our experiments follow the same setting as Table in the main paper. The ablation results below show that both the attention mask and embodiment context are effective in improving performance on two robot platforms: Google Robot and WidowX Robot.
Table 8: Ablation study results for the policy model. The first columns (Pick, Move, Drawer) refer to the Google Robot, and the last columns (Carrot, Eggplant, Spoon, Cube) refer the WidowX Robot. All numbers are success rates (%).
Method
Pick
Move
Drawer
Avg.
Carrot
Eggplant
Spoon
Cube
Avg.
Ours
81.7
55.4
38.4
58.5
24.2
71.7
48.3
19.2
40.8
Ours w/o mask
80.3
30.6
48.8
53.2
18.3
52.5
38.3
26.7
34.0
Ours w/o context
86.6
21.3
39.3
49.1
28.3
67.5
25.8
32.5
38.5
G SIMULATION EVALUATION DETAILS
G.1 SIMPLER BENCHMARK
We evaluate on all eight SIMPLER (Li al. 2024e) tasks in the visual matching setting, which include two robot platforms: Google Robot and WidowX.
For Google Robot, the tasks are: (1) pick coke can (including horizontal, vertical and standing can configurations); (2) move an object near target object; (3) open / close top. middle or bottom drawer; and (4) place apple in a closed drawer, which includes two subtasks: first open top drawer, and then place the apple into the top drawer. On the widowX setup, the tasks consist of: (1) put a carrot on the plate; (2) put an eggplant on the basket; (3) put spoon on the towel; (4) stack a green cube on yellow one.
We follow the standard evaluation protocol to test by randomizing both configurations of the environ- ments. For the Google Robot tasks, we execute 300 trials of "Pick Coke Can", 240 of "Move Near", 216 of "Open/Close Drawer", and 108 of "Place Apple Closed Drawer". For each WidowX X task, use 24 unique configurations. To ensure statistical significance,
test each configuration 10 times, yielding 240 rollouts per task. Reported results (Table 2) are the average success rates across these trials. Please refer to SIMPLER (Li et al. 2024e for more details.
For a fair comparison, we adopt the published performance metrics for RT-1-X (Collaboration et al. 2023), Octo-base (Octo Model Team et al.] (0044), Open VLA (Kim al. 2024), Robo VLMs (Li et al. 2024b). MoTo (Chen et al. (024b), and LAPA Ye et al. 2024) directly from their respective papers. In the case of GROOT INVIDIA et al. 2025b),
the official pretrained checkpoint
21

---

Published as a conference paper at ICLR 2026
and performe fine-tune tine-tuning on the RT- RT-I/Bridge dataset following the authors' published guidelines accordingly.
H LIBERO BENCHMARK
The LIBERO benchmark (Liu et al. 2023a evaluates knowledge transfer in multitask and lifelong robot learning problems for robotic manipulation, consisting of four task suites: LIBERO-Spate evaluates the model's performance under novel layouts with the same task and object types, LIBERO- Goal evaluates the model's s performance under novel tasks with the same object types and layouts, LIBERO-Object evaluates the model's performance under novel object types with the same tasks and layouts, LIBERO-Long evaluates the model's performance under diverse set of objects, layouts and backgrounds. Each task suite contains 10 tasks with 50 human demonstrations per task for fine-tuning.
Baselines and Experimental Setup We compare with the following existing models: Diffusion
Policy (Chi et al.
2023) trained from scratch, Octo (Octo Model Team et al.) 2024). Open VLA (Kim Black et al. 2024), mg FAST (Pertsch et al.) 2025). Trace VLA (Zheng al. 2024)
et al. 2024). , no
and Spatial VLA
(Qu et al.] 2025). ). For wo, we use the open source version (Ren 2025 and the same
training set as our model. All models follow a two-stage pretraining- finttininninniingggggg protocol. We finetune villa-X and villa-X X w/o latent on the demonstration data of the each task suite separately, and
test on the LIBERO simulator for 10 tasks and 20 trials per task on each task suite.
Experimental Results Table summarizes the success rates each task suite of LIBERO. Our model achieves better performance than existing methods in all the four task suites. Also, our model with latent action achieves higher performance all the four task suites and average performance, confirming that the proposed latent action expert improves the manipulation performance.
Table 9: Evaluation on 4 LIBERO task suites of villa-a X and existing methods.
Method
Spatial
Object
Goal
Long
Average
Diffusion Policy (Chi ct al. 2023
78.3
92.5
68.3
50.5
72.4
Octo-base (Otto Model leam et
2024
78.9
85.7
84.6
51.1
75.1
Ope Kim el 2024
84.7
88.4
79.2
53.7
76.5
No (reimplement [Ken
2025
88.0
88.5
87.0
81.1
no- FASFAST [Pertsch et a..
2025
96.4
96.8
88.6
85.5
VLA 7Theng et
84.6
85.2
75.
54.1
74.8
SpatialVLA (Qu 2025
88.2
89.9
78.6
55.5
78.1
Ours w/o latent
86.0
86.5
85.0
70.0
81.9
Ours
97.5
97.0
91.5
74,5
90.1
REAL-WORLD ROBOT PLATFORMS EVALUATION DETAILS
1.1 REALMAN ROBOT ARM
The Realman robot arm setup is shown in Figure (upper). We mount the gripper for Inspire Robot to the Realman RM75 robot arm. We use two camera views, including a primary view camera with the same view point the images (used to demonstrate different tasks) shown in 5 and a wrist camera. For fine-tuning of our models, we reinitialize the linear state encoder, action encoder, and action decoder, and tune the full parameters (except for the vision encoder). We fine-tune all the models for 60k gradient steps.
We collect data the following five tasks with their task instructions:
Put-in: "Pick the green block from the table into the blue bowl"
Put-out: "Pick the green block from the blue bowl onto the table"
Push: "Push the green block to position X" where "X" indicates the nine positions written
the table.
22

---

Published as a conference paper at ICLR 2026
Stack: "Stack the wooden block onto the green block"
Unstack: "Unstack the wooden block from the green block"
We collect 375 trajectories (75 trajectories for each task) for fine-tuning, The trajectories are collected 10Hz. We post-process these trajectories remove statie frames with action, resulting 120 steps on average in one trajectory.
We evaluate the fine-tuned model seven groups with 10 trials for each group. The first five groups contain the tasks the same as data collection. The last two groups are designed to evaluate the generalization ability of the models. For the "change block color" group, we repeat the previous five tasks but change the green block into blue and red ones. For the "change table cover" group, we change the table cover from red brown and blue ones.
The visualization example of each task for our model can be found in Figure
I.2 XHAND DEXTEROUS HAND
The Xhand setup is shown in Figure (lower). The 12-dof Xhand is mounted on a 7-dof XArm robot arm. There two camera views, including a main 3-rd view camera, and wrist camera. During fine-tuning, we reinitialize linear encoder and decoder modules for both state and action to accommodate the hand's higher dimensionality.
We use the dataset collected (Hu et al. 2024) as our finetuning dataset, which comprises roughly 4,000 trajectories spanning 13 task categories and 50 unique objects. For evaluation, focus on five representative XHand tasks as depicted in Figure 5 namely pick-and-place, cube stacking, upright cup placement, water pouring, and ball flicking. Each task is assessed under "seen" and "unseen" conditions: in the seen setting, the same objects and backgrounds encountered during training are used, albeit with randomized tabletop positions and optional distractors; the unseen setting, either the target objects or the scene background (or both) were never encountered during finetuning, totaling more than 20 novel objects. During evaluation, we conducted 50 evaluation runs for the pick-and-plans task, 20 runs for cube stacking, and 10 runs for each of the remaining tasks. The visualization example each task can be found in Figure 12 and Figure
LLM USAGE
Throughout the preparation of this manuscript, we utilized Large Language Models (LLMs) as writing assistants. Their primary roles were to proofread for grammatical accuracy and to help refine the prose.
23

---

Published as a conference paper at ICLR 2026
K MORE RESULTS ON PROBING ANALYSIS
In this section, we provide additional results for the experiments described in Section 4.1 Figure 14 presents the probing results under the L1 loss using two complementary visualizations. In the left panel, we display overlapped bar plots where distinct colors represent different methods. In the right panel, we visualize the difference sample distribution across the same error thresholds.
L LATENT PLAN ROLLOUT USING LAM FDM
This section supplements Figure with visualizations using the LAM's image FDM. The procedure involves generating sequence of latent actions 21m via the ACT-latent policy, given an initial frame and instruction. We then recursively predict future frames via 6 DDM (ô,, 2e), where the input o is the output of the last iteration. We observe that the image FDM captures the correct motion intention, although the generated images suffer from severe blurring as errors accumulate during iterative rollouts. This visual degradation justifies our motivation for employing dedicated world model Figure 4
24

---

Published as a conference paper at ICLR 2026
Original Video
SIMPLER Rollout
Original Video
SIMPLER Rollout
Original Video
SIMPLER Rollout
Figure 8: Transfer human video demonstrations into robot actions through LAM and proprio FDM in SIMPLER simulator. Upper: the SIMPLER rollout closely reproduce the motion of moving right; Middle: the SIMPLER rollout closely reproduce the motion of moving forward and backward; Bottom: the SIMPLER rollout closely reproduce the motion of moving right.
25

---

Published as a conference paper at ICLR 2026
Move the the left side
Initial
Move the cone to the upper side
Pick the pot
Initial Frame
Pick up the red object
Move the blue into the bowl
Initial Frame
Pick up the spoon
Figure 9: Generated image sequence jointly by the latent expert and the world model via latent actions, following different instructions from the same initial image (Part I).
26

---

Published as a conference paper at ICLR 2026
Open bottom drawer
Initial
Open middle drawer
Close middle drawer
Initial Frame
Pick pesi can from the middle drawer
Close top drawer
Initial Frame
Pick brown chip bag from top drawer
Figure 10: Generated image sequence jointly by the latent expert and the world model via latent actions, following different instructions from the same initial image (Part II).
27

---

Published as a conference paper at ICLR 2026
Pick the green block from the table into the blue bowl
Pick the green block from the blue bowl onto the table
Push green block position
Stack the wooden block onto the green block
Unstack the wooden block from the green block
Figure 11: Realman evaluation trajectory examples.
28

---

Published as a conference paper at ICLR 2026
Pick the mango in the green plate
Pick the apple into the blue bowl
Pick the yellow toy into the basket
onion basket
Stack the blue cube the red cube
Figure 12: Xhand evaluation trajectory examples (part I).
29

---

Published as a conference paper at ICLR 2026
Straighten the cup
Flick the ball
Flick the
Pouring orange juice into cup
Pouring orange juice into the
Figure 13: Xhand evaluation trajectory examples (part II).
300
2500
sample
No vyvvopp pp sample
200
N popppp
2000
100
1500
N uppp sample
1000
-100
500
0.05
000
0.15
0.20
0.25
0.30
0.35
0.40
0.50
0.55
0.0
0.65
0.05
010
0.15
0.20
0.25
0.30
0.35
0.40
0.50
0.55
0.50
0.65
Figure 14: Probing results under Ln loss using different views.
30

---

Published as a conference paper at ICLR 2026
Touch the cookie
Touch the rice
Figure 15: The rollout of latent plan using LAM FDM.
31


Reviews:
[{"review_id": "Hy3gpjamvr", "reviewer": "Reviewer_kiPs", "summary": "The paper presents an approach to learn latent actions, and leverage latent actions when predicting robot actions through joint diffusion. In addition to learning forward and inverse dynamics models, as is done in existing work, the paper proposes to learn an embodiment-conditioned model which predicts future robot states and actions. During inference-time, the action head predicts both latent actions and robot actions given image observations, language instructions, proprioceptive states and embodiment embedding, where an attention mask is used to enforce the factorization of the conditional probability.", "questions": "- How is $K$, the number of future steps when training FDM and IDM, determined?\n\n- Why is the prediction only for $o_t$ and $o_{t+K}$ when training observation F/IDM in Equation (1), but for the entire sequence when prediction robot states $q_{t+1:t+K}$ and actions $a_{t+1:t+K}$ in Equation (2)?\n\n- In Equation (3), why is the context vector conditioned on dataset ID instead of robot ID. Wouldn’t it make sense for two datasets with the same robot to share an ID?\n\n- Why does the action head predict $(n-1)K$ latent actions and $m$ robot actions? How to choose $n$ and $m$?\n\n- Why is the robot action branch “overly relying on latent actions” harmful? Isn’t the latent action supposed to provide sufficient information for action prediction (Equation (2))?\n\n- What is the action space for the model? How does the gripper command translate to Xhand command in the realworld experiments?\n\n- In Appendix A.1, how is the codebook size of $32$ decided?\n\n- In Appendix B, can you explain how different $\\tau$ distribution can be used for latent actions and robot actions? This seems contradictory to (5) where all actions are bundled together during denoising. \n\n- In Appendix F, is the attention mask ablation referring to the block-wise causal attention mask or random masking of attention during training?\n\n- In Appendix H, why is OpenVLA-OFT not included in Table 9? Especially considering that it is included in Table 2.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach has strong empirical results. In particular, in Table 2, the proposed approach outperforms state-of-the-art VLA models, such as OpenVLA, OpenVLA-OFT, $\\pi_0$, and Gr00T. It also outperforms other algorithms such as MoTo and LAPA.\n\n- Using an embodiment-specific embedding is an interesting approach to leveraging diverse robot datasets, where different datasets have slightly different dynamics and action spaces. Ablation study shows that the embodiment embedding has a positive impact on the success rate.\n\n- The ablation study is in general quite thorough, showing the effectiveness of each component of the proposed approach, e.g., latent action model, proprioceptive state prediction, etc.", "weaknesses": "- Although the main design components are validated through ablation studies. Some finer-grained design choices lack discussion and study. See more details in the **Questions** section.\n\n- Figure 3, probing experiment results, is not very easy to understand. Perhaps a plot showing the distribution of error in different intervals, for both w/ pp and w/o pp,  would be more informative. It is also not very convincing why $L_\\infty$ (max across all dimensions) is preferred over $L_1$ (summing/averaging over all dimensions). \n\n- The results in 4.3, although helpful and intuitive, are not very informative scientifically. Since the images are obtained from a “separately trained world model”, it is hard to tell whether the latent action actually encodes the robot behavior or these outcome images just come from hallucination from the world model. It may be good to swap it with another ablation study in the appendix.\n\n- It is unclear why only Gr00t is used as the baseline for real-world experiments, where other state-of-the-art VLA models are left out.\n\n- The tasks evaluated in the experiment are mainly simple pick and place tasks, without much dexterity. \n\n- Minor: Incomplete sentence in Appendix D.3: \"Both models were trained on 10...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IarNAU1Dtb", "reviewer": "Reviewer_XiVL", "summary": "The paper introduces VILLA-X, a Vision-Language-Latent-Action framework that advances robot policy learning by improving both how latent actions are learned and integrated into Vision-Language-Action models. It enhances latent action learning through a proprioceptive forward dynamics module that grounds latent representations in physical robot dynamics, and it introduces a joint diffusion-based policy that conditions robot action generation on latent action planning.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured and clearly written. The problem motivation is sound, the technical approach is explained logically.\n\n2. The evaluation is thorough, encompassing systematic ablations, major simulation benchmarks (SIMPLER, LIBERO), and real-world deployment on two distinct platforms. \n\n3. The demonstrated capability for zero-shot generalization to novel embodiments addresses a core challenge in the field.", "weaknesses": "1. The technical contributions, while valuable, exhibit limited novelty relative to existing literature. The proposed proprioceptive Forward Dynamics Model (proprio-FDM), which grounds latent actions by predicting low-level states, is conceptually similar to the approach of Nikulin et al. [1], who employ a linear decoder on latent tokens to predict actions. The efficacy of this general principle for grounding has also been previously analyzed by Zhang et al. [2]. Furthermore, the architectural design of separate experts for latent and robot actions (ACT-latent and ACT-robot) bears a strong resemblance to the module separation employed in GO-1.\n\n2. The characterization of the GO-1 baseline may be inaccurate. Based on its open-source implementation, GO-1 does not appear to autoregressively predict latent actions using a next-token-prediction (NTP) loss, but rather uses an L1 loss for latent action learning. Consequently, the description in Section 4.2 and the subsequent analysis in Table 1 could be misleading regarding the true nature of this baseline.\n\n3. The experimental comparisons lack benchmarks against several highly relevant contemporary works, notably IGOR [3] and UniVLA [4]. These methods also focus on cross-embodiment generalization and leverage latent actions learned from web videos, making their inclusion critical for properly contextualizing the claimed advancements of this work.\n\n4. The zero-shot generalization analysis in Section 4.3 would be strengthened by clarifying the training procedure for the world model used for visualization. \n\n\n\n______\n\n[1] Nikulin, Alexander, et al. \"Latent action learning requires supervision in the presence of distractors.\" arXiv preprint arXiv:2502.00379 (2025).\n\n[2] Zhang, Chuheng, et al. \"What Do Latent Action Models Actually Learn?.\" arXiv preprint arXiv:2506.15691 (2025).\n\n[3] Chen, Xiaoyu, et al. \"IGOR: Image-goal representations are the atomic control units for foundation models in embodied ai.\" arXiv preprint arXiv:2411.00785 (2024).\n\n[4] Bu, Qingwen, et al. \"UniVLA: Learning to act anywhere with task-centric latent actions.\" arXiv preprint arXiv:2505.06111 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1xASDEm9S1", "reviewer": "Reviewer_oHXE", "summary": "This paper introduces a new framework called villa-X, whose core idea is to incorporate latent actions into the pretraining and policy learning of Vision-Language-Action (VLA) models.", "questions": "1. I recently came across latent action learning in a survey paper [1]. How does the latent action learning mentioned in your work differ from that?\n\n2. I also believe that latent learning is a promising approach for solving cross-embodiment transfer. What other potential solutions do you foresee in the future?\n\n3. Equation (3) makes me a bit confused — what exactly is the dataset ID, and why is the context vector ce composed of these two parts?\n\n4. How large is the VILLA-X model in terms of scale?\n\n[1] Towards a Unified Understanding of Robot Manipulation: A Comprehensive Survey. arXiv 2025.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The system design is simple, effective, and scalable.\n\n2. The experiments are sufficient and comprehensive.\n\n3. The writing is clear and provides a good reading experience.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iwxHv11EYd", "reviewer": "Reviewer_7xUJ", "summary": "This paper presents villa-X (vision-language-latent-action), a framework that integrates latent actions into vision-language-action (VLA) models. The core idea of villa-X consists of two components:\n- incorporating existing latent action models (LAMs) into a proprioceptive module to obtain more physically grounded latent actions\n- jointly modeling latent actions and real robotic actions.\n\nExperimental results demonstrate that (1) the proprioceptive module enables LAMs to extract higher-quality latent actions, and (2) the resulting VLA model significantly outperforms existing VLA baselines in both simulated and real-world environments.", "questions": "- [Q1] In Table 1, the performance of LAPA and Go-1 on the WidowX robot appears noticeably poor. Could the authors elaborate on the underlying cause of this phenomenon?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- [S1] villa-X achieves strong performance in both real-world and simulated settings. The authors conduct extensive experiments that convincingly demonstrate the effectiveness of the proposed framework.\n- [S2] The paper is clearly written and well-organized. Figures and equations are concise yet informative, effectively illustrating how villa-X operates.", "weaknesses": "- [W1] From a high-level perspective, this work largely follows the structure of the existing framework [1]. The process—first training latent action models (LAMs) with a VQ-VAE-style objective to generate pseudo labels (latent actions) for robot data, and then training the VLA model to predict those latent actions—remains similar. The proposed addition of a proprioceptive module and the joint prediction of latent and robotic actions, while useful, appear somewhat incremental.\n- [W2] The paper provides limited discussion on why villa-X performs well across diverse environments and tasks. The experiments primarily report performance improvements without deeper analysis of contributing factors or underlying mechanisms.\n- [W3] No statistical significance analysis is presented. The reported performance metrics lack measures of variance or confidence, making it difficult to assess the robustness of the claimed improvements.\n\n**References**\n\n[1] Ye et al., Latent Action Pretraining from Videos. In ICLR, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)