Latent Action Pretraining from Videos
Authors: Seonghyeon Ye
Joel Jang
Byeongguk Jeon
Se June Joo
Jianwei Yang
Baolin Peng
Ajay Mandlekar
Reuben Tan
Yu-Wei Chao
Bill Yuchen Lin
Lars Liden
Kimin Lee
Jianfeng Gao
Luke Zettlemoyer
Dieter Fox
Minjoon Seo
Venue: ICLR 2025 Poster (2025)
Keywords: vision-language-action models
robotics
Primary area: applications to robotics, autonomy, planning
TL;DR: We introduce Latent Action Pretraining for general Action models (LAPA), the first unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels.

Abstract: We introduce Latent Action Pretraining for general Action models (LAPA), the first unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels. Existing Vision-Language-Action models require action labels typically collected by human teleoperators during pretraining, which significantly limits possible data sources and scale. In this work, we propose a method to learn from internet-scale videos that do not have robot action labels. We first train an action quantization model leveraging VQ-VAE-based objective to learn discrete latent actions between image frames, then pretrain a latent VLA model to predict these latent actions from observations and task descriptions, and finally finetune the VLA on small-scale robot manipulation data to map from latent to robot actions. Experimental results demonstrate that our method significantly outperforms existing techniques that train robot manipulation policies from large-scale videos. Furthermore, it outperforms the state-of-the-art VLA model trained with robotic action labels on real-world manipulation tasks that require language conditioning, generalization to unseen objects, and semantic generalization to unseen instructions. Training only on human manipulation videos also shows positive transfer, opening up the potential for leveraging web-scale data for robotics foundation models.

Published as a conference paper at ICLR 2025
LATENT ACTION PRETRAINING FROM VIDEOS
Seonghyeon Yelv
Joel Jang 2+
Byeongguk Jeon'
Sejune Joo'
Jianwei Yang" Baolin Peng
Ajay Mandlekar
Reuben Tan
Yu-Wei Chao
Yuchen Lin
Lars Liden
Kimin Lee Jianfeng Gao 38
Luke Zettlemoyer Dieter 2.x
Minjoon Seo
KAIST University of Washington
Microsoft Research
NVIDIA
5 Allen Institute for Al
ABSTRACT
We introduce Latent Action Pretraining, the first unsupervised method for pre- training Vision-Language-Actioo (VLA) models without ground-truth robot ac- tion labels. Existing Lision-Language-Action models require action labels typ- ically collected by human teleoperators during pretraining, which significantly limits possible data sources and scale. In this work, we propose a method learn from internet-scale videos that do not have robot action labels. We first train an action quantization model leveraging VQ-VAE -aseeed objective to learn discrete latent actions between image frames, then pretrain latent VLA model to predict these latent actions from observations and task descriptions, and finally finetune the VLA on small-scale robot manipulation data map from latent to robot ac- tions. Experimental results demonstrate that our method significantly outperforms existing techniques that train robot manipulation policies from large-scale videos. Furthermore, it outperforms the state-of-the VLA model trained with robotic action labels real-world manipulation tasks that require language conditioning, generalization to unseen objects, and semantic generalization to unseen instruc- tions. Training only human manipulation videos also shows positive transfer, opening up the potential for leveraging web-scale data for robotics foundation models. We will open-source the model checkpoints and code latentactionpre- iotnion
INTRODUCTION
Vision LanoonLLannnggettttiin Models (VLA) for robotics (Brohan et al. 2023 Kim et al. 2024 are
trained by aligning large language models with vision encoders, and then finetuning on diverse robot datasets (Collaboration et al. (023); this enables generalization to novel instructions, unseen objects, and distribution shifts (Michat et al. 2024). However, diverse real-world robot datasets mostly require human teleoperation, which makes scaling difficult. Internet video data, on the other hand, offers abundant examples of human behavior and physical interactions scale, presenting promising approach to overcome the limitations of small, specialized robotic datasets (Yang et al.) 2024cp. However, it is challenging to learn from internet video data for two major challenges: first. much of the raw data on the web lacks explicit action labels; second, the data distribution from the web fundamentally different from the embodiments and environments of typical robotic systems (McCarthy et al. 2024). We propose Latent Action Pretraining, an unsupervised approach to pretraining robotic foundation model without the need for ground-truth robot action labels (Figure D..
Latent Action Pretraining consists of two models that are learned sequentially, followed by a fine- tuning stage to map the latent actions to real robot actions. In the first pretraining stage, we use
DDenotes equal contribution.
Work done during internship Microsoft Research. I Work done during internship at NVIDIA.
6 Denotes equal advising.

---

Published as a conference paper at ICLR 2025
Large-Scale Robot Datasets
Internet-scale Video Data
Expensive to collect
xxx
Widely available
Requires hardware Contains robot actions
Human robot x actions
LAPA
Robotic Foundation Model
Problem Formulation. We investigate building a generalist robotic foundation model from human motion videos without action labels.
Figure I:
VQ-VAE-I Aaaeda objective (Van Den Oord al., 2017) learn quantized latent actions between raw image frames. Analogous to Byte Pair Encoding (Sennrich et al., 2016) used for language modeling, this can be seen as learning to tokenize atomic actions without requiring predefined action priors (e.g., end-effector positions, joint positions). In the second stage, we perform behavior cloning by pretraining Vision-Language Model to predict latent actions derived from the first stage based on video observations and task descriptions. Finally, we fine-tune the model small-scale robot manipulation dataset with robot actions to learn the mapping from the latent actions to robot actions. In this work, refer to the resulting VLA models as LAPA.
We measure performance on diverse manipulation videos, including existing robot video datasets (without utilizing ground-truth actions) and human manipulation datasets. Our results show that the proposed method significantly outperforms baseline methods of training manipulation policies with- out ground-truth action labels, particularly in cross-enviromment and cross-embodiment scenarios. Furthermore, on real-world manipulation tasks, our method leads to a new monolithic VLA model, outperforming OPENVLA, the current state-of-the-ard model Vision Language Action (VLA) model trained on a diverse mixture of datasets with ground-truth actions. These results demonstrate the ef- fectiveness of learning unified quantized latent action representations across diverse robotic datasets featuring different embodiments (shown in Section 5.2) We further demonstrate that Latent Action Pretraining remains effective even when pretrained on only human manipulation video, outperform- ing models pretrained on Bridgev2 (Walke et al. 2023), one of the largest open-sourced robotic datasets. We observe that LAPA effectively captures environment-centrcs actions, including object and camera movements, which could be beneficial for downstream tasks like navigation dynamic, non-quasistatic tasks. We expect that our method opens up the potential for building foundation models for roboties by pretraining much larger web-scale video data.
We summarize our main contributions and findings below:
We propose Latent Action Pretraining, an unsupervised approach to pretraining robotic foundation model to encode robotic skills from web-scale video data.
Experiments on simulation and real-world robot tasks show that our method not only sig- nificantly outperforms baseline methods for training robotic manipulation policies that are pretrained without using ground truth action labels, but also leads to VLA model that outperforms the current state-of-the-art VLA model trained with ground-truth actions (by +6.22%), while achieving over 30x greater pretraining efficiency.
We qualitatively demonstrate that it is possible to use LAPA as the action prediction model and decoder of the latent action quantization model as the world model by predicting fu- ture frames conditioned the current observation and the latent action predicted by LAPA, effectively building a neural simulation capable of performing closed-loop evaluations en- tirely through neural inference.
RELATED WORK
Vision-Language-Action Models Vision-Language Models (VLMs), trained on large-scale inter- net datasets of text, images, and videos, have shown strong capabilities in understanding and gen- erating both text and multimodal data (Liu et al.) 2023 Team 2024 Liu et al. 2024: Abdin et al. 2024). Leveraging this, recent advancements have introduced monolithic Vision- Vision-Language sanguag--Action Models (VLAs), which extend VLMs by fine-tuning them with robotic action data for enhanced
physical grounding (Brohan et al. 2023 Kim al. 2024
Team et al. 2024 Collaboration al.
2

---

Published as a conference paper at ICLR 2025
X1
Enooder
Decoder
VLM
LAPA
4I
N2
Knock down the water bottle
Pick up the milk and put un the sink
Lawent
1. Latent Action Quantization
2. Latent Pretraining
Latent Action Pretraining
Action Finetuning
Figure 2: Overview of Latent Action Pretraining. (1) Latent Action Quantization: We first learn discrete
latent actions in a fully unsupervised manner using the VQ-VAE objective (Detail in Figure (). (2) Latent Pretraining: The VLM is trained predict latent actions, essentially performing behavior cloning. After pretraining, finetune LAPA on small set of action-labeled trajectories to map latent space the end effector delta action space.
Incorporating auxiliary objectives, such as visual traces (Niu et al., 2024), language rea- soning paths (Michal al. (2024), or creating conversational-style instruction datasets from robot trajectories (Li et al.) 2024a). have further improved VLA performance. However, these methods remain dependent on labeled action data, limiting scalability. In contrast, our approach reduces reliance on numan-tcleoperated data by requiring labeled actions only for fine-tuning.
2023).
Videos offer rich data for robot learning, but most lack action labels (McCarthy et al. 2024). Related work pretrains a vision encoder on egocentric human videos (Grauman al. 2022 improve visual representations (Nair al.
Training Robot Policies From Videos
2022 Dasari et al., 2023 or video generative models to generate future robot trajectories (Wu et al. 2024 Liang et al,,
2024
He al., 2024). Methods also extract diverse features from human videos such as interac-
tions Zeng et al. (0244). affordances (Bahl et al. 2023 Kannan et al., 2023 Srirama et al. 2024
Shaw et al. 2023), or visual traces (Wen et al., 2023 Bharadhwaj al.) (024b). Some perform
retargeting of human motions to robot actions to create robotic policies that involve hand pose es-
timators (Wang al. 2023 Zhu et al., 2024 Shaw et al. 2023 Bharadhwajj et al. 2023] Ye et al.) 2024a); these policies are usually task-specific or need aligned data human to robot data in the same environment. Finally, some train inverse dynamics models (IDMs), optical flow, reinforcement learning models that predict actions from future state rollouts generated by world models (Du et al.
2023 Qin et al. 2022) or motion capture systems (Yang et al.
2023; Ko et al.) 2024. Yang et al.
2024b
Bharadhwaj al. 2024a)
the IDM for active learning (Baker al. 2022
Latent Actions Previous works have employed latent actions across diverse scenarios. GE- NIE (Bruce et al.] 2024) maps user inputs (ground-truth actions) to latent space, allowing gen- crative models to create interactive environments. We adopt similar latent action model but apply it to label actionless data for training monolithic VLA to solve robotic tasks. Similarly, Edwards et al. (2018) and Schmidt & Jiang (2024) use latent actions to pretrain and fine-tune policies for focus learning latent actions from real-world
video games (Cobbe et al. (019)). In contrast, human motions for more complex, continuous robotic tasks. Unlike other work that leverages latent actions by converting ground-truth actions into latent capture better multimodality and task mantics Lynch et al. (2020); Jiang et al. (2023); Lee et al. (2024); Mete et al. (0024), our approach derives latent actions directly from observations, not ground- -ruth actions.
3 LAPA: LATENT ACTION PRETRAINING FOR GENERAL ACTION MODELS
Latent Action Pretraining consists of two models that learned sequentially: Latent Action Quan- tization and Latent Pretraining. The overall process illustrated Figure Note that we use the same pretraining dataset for Latent Action Quantization and Latent Pretraining.
3

---

Published as a conference paper at ICLR 2025
3.1 LATENT ACTION QUANTIZATION
To learn latent actions in fully unsupervised manner, we train a latent action quantization model following [Bruce et al.) (224) with a few modifications. Our latent action quantization model is an encoder-decoder architecture where the encoder takes the current frame 24 and the future frame Te-H of video with fixed window size H and outputs the latent action 2 The decoder is trained to take the latent action 2t and Tt and reconstruct H++ Unlike [Buce et al.] (2024), we use cross attention to attend 2 given 2t instead of additive embedding, which empirically leads to capturing more semantically meaningful latent actions. Our quantization model is a variant of C-ViVIT tokenizer (Villegas et al. 2023) where the encoder includes both spatial and temporal transformer while the decoder only contains spatial transformer since our model uses only two image frames input.
Our latent action quantization training model is based on the VQ-VAR objective (van den Oord et al,, 2017), where the nearest quantized representation from the continuous embedding is retrieved from an embedding space where each embedding corresponds to a codebook. The VQ-VAE objective enables the latent action 2 to be discrete tokens (codebooks), making easy for VLMs to predict 2,. The latent action is represented using sequences from |C] codebook vocabulary space. The sequence length is designated by the kernel size, stride and padding value of CNN network which is used right before the vector quantization process. To avoid gradient collapse often observed in VQ- we utilize NSVQ (Vali & Bäckström (022) which replaces the vector quantization error to a product of original error and a normalized noise vector. We also apply stop gradient to the patch embedding of the frame 2i during decoding to avoid representation collapse. Codebook replacement technique from NSVQ is applied during early training steps to maximize codebook utilization. Further model and training details are provided in Appendix
We utilize the encoder of our latent action quantization model as an inverse dynamics model in la- tent pretraining and the decoder for generating neural-based closed-loop rollouts. Unlike previous works (Bruce et al. 2024 Valevski al., 2024), our approach trains both world model that gen- erates rollouts from the latent actions and policy model that produces these latent actions through Latent Pretraining.
3.2 LATENT PRETRAINING
We use the encoder of the latent action quantization model as an inverse dynamics model to label all frames Ft. given frame 2(+1, with latent action 2,. Then, we do action pretraining by using a pretrained VLM to predict the & given the language instruction of video clip and the current image I₁. Instead of using the existing language model head of the VLM, we attach a separate latent action head (a single MI.P layer) of vocab size |C). By default, we freeze only the vision encoder and unfreeze the language model during training. Since latent pretraining does not rely on ground truth actions, it opens the possibility of using any type of raw video paired with language instructions. Also, in contrast to traditional action granularity used in robotics (e.g. end-effector positions, joint positions, joint torques, etc.), our approach does not require any priors about the action hierarchy/granularity and is learned in end-to-end manner simply by being optimized to best capture the 'delta' of consecutive observations in a given video dataset. We broadly refer to models having gone through latent pretraining as LAPA.
3.3 ACTION FINETUNING
VLAs that are pretrained predict latent actions are not directly executable real-world robots since latent actions are not actual delta end-effector actions or joint actions. To map latent actions to actual robot actions, we finetune LAPA on small set of labeled trajectories that contain ground truth actions (delta end-effector). For action prediction, we discretize the continuous action space for each dimension of the robot so that the number of data points allocated for each bin is equal following Kim et al. (2024) : Brohan et al. (023). We discard the latent action head (a single MLP layer) and
'Athhough Bruce et al. (202)) conditioned on multiple past observations, exclude previous frames due computational constraints. We leave prepending past observations as future work.
4

---

Published as a conference paper at ICLR 2025
replace it with new action head generate ground truth actions + freeze the vision encoder and unfreeze all of the parameters of the underlying language model
As with latent pretraining,
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of Latent Action Pretraining as a general-purpose pretaining method. Specifically, we focus on answering the following questions: QL. How does LAPA perform when there are cross-task, cross-environment and cross-embodiment gaps between pretaining and fine-tuning? Q2 Can LAPA learn superior priors compared to using ground-truth actions during pretraining in nuutt--embodiment setting? Q3. Can create performant LAPA solely from raw human manipulation videos?
4.1 BENCHMARKS AND ENVIRONMENTS
We evaluate the effectiveness of LAPA on 9 different task categories in 2 different simulation envi- ronments and different real-world robotic tasks. Table shows overview of the pretraining and fine-tuning dataset for each setup and Figure Sigureeinn Appendix visualizes the simulation benchmark and real-world setups. More details of each evaluation setup are provided in Appendix
Language Table (Lynch et al. 2023) is simulation where robot performs 2 DOF actions to push blocks (see Figure 9 (a)). includes subtask categories: BlocktoBlock, BlocktoAbsolute, BlocktoBlockRelatvv BlocktoRelative, and Separate. During evaluation, we evaluate models for both seen and unseen scenarios, where unseen includes new objects (color and shape) and unseen combinations of seen objects.
SIMPLER (Li et al., 2024b) set of simulated environments for evaluating generalist robot manipulation policies. We assess our models on 4 tasks (Figure 9 (b)) using the DOF WidowX robot arm. Since SIMPLER lacks fine-tuning trajectories, we collect 100 multi-task trajectories using successful rollouts from VLA model trained on V2 data (Walke et al. 2023).
Real-World Tabletop Manipulation experiments used a 7 DOF Franka Emika Panda robot arm in three environments (shown in Figure 9 (c)). We utilize three pretraining data sources: Bridgev2 (Walee et al.) 2023), Open-X (Collaboration et al. 2023). and Something Something finetune three multi-instruction tasks:
v2 GGoyal et al. 2017). Following Kim et al. (2024),
(1) Pick <object>> into Sink', (2) 'Cover <object>> with Towel', and (3) 'nookk <object>> Over'.
Each task involves 150 trajectories across 15 objects. We use task-specific partial success criterion
for evaluation, following Kim et al. (2024).
4.2 BASELINES
For the underlying VLM, we use the 7B Large World Model (LWM-Chat-IMP (Liu et al. 2024).
SCRATCH denotes the baseline model where we finetune our backbone VLM only the down- stream tasks, to quantify the gains we get from the pretraining stage.
UNIPT (Du et al., 2023) uses video diffusion model during pretraining generate video rollouts given a language instruction, which does not require any action labels during pretraining similar to our approach. For finetuning, an inverse dynamics model (IDM) trained to extract the ground truth actions given adjacent frames El We also finetune the diffusion model on the downstream task to match the target distribution.
VPT (Baker et al.) 2022) trains an IDM on action labeled data, and uses the IDM model to extract pseudo actions on raw videos. Then, we use the pseudo actions labeled by the IDM to pretrain our backbone VI.M on the pretraining data, identical to Latent Pretraining of LAPA.
We also tried leaving latent action head and adding additional head to decode the latent to ground-truth actions following Schmidt & Jiang (2024) However, we empirically found that re-initializing the action head
resulted in superior downstream task performance, likely due to the size of the underlying policy model (7B). 3 leave parameter efficient fine-tuning approaches as future work finetuning (Hu al. 2022
We do not compare with Yang et al. (2024h) since the model is not open sourced and Ko et al. it is behavior cloning baseline.
2024 since
5

---

Published as a conference paper at ICLR 2025
ACTIONVLA denotes the baseline that uses the actual ground-truth robot action labels during pre- training with the same backbone VLM. This may be seen as the upper bound, since it utilizes the actual ground-truth labels.
OPENVLA (Kim et al., 2024) is a state-of-the-ar VLA model that was pretrained on 970k real- world robot demonstrations from the Open X-Embodiment Dataset (Collaboration al. 2023), mostly collected through human teleoperation. This model has a comparable number of parameters LAPA (7B). We compare against OPENVLA for real-world robot experiments by fine-tuning the pretrained OPENVLA on our downstream tasks.
Further details of baseline models are provided in Appendix
4.3 LANGUAGE TABLE RESULTS
Language Table Results. Average Success Rate (%) + StdErr across the three different pretrain- finetune combinations from the Language Table benchmark described in Table 3 traeectories used for fine-tuning next each category.
Table I:
We also note the # of
In-domain (1k)
Cross-usk (7K)
Cross-eav (Ik)
Socn
Unseen
Seen
Unseen
Scen
Unsoon
SCRATCH
15.6
15.2_a.
27.2
22.4,110
15.6192
15.2_a.a
UNIPI
13.2 22.0
20.8
16.0
13.6
12.0 .7.
VPT
44.0g7.
32.8...
72.0a6s
60.8256
18.0
18.4_9.7
LAPA
62.0
49.6 u.5
73.2
54.81aa
33.61
29.6
ACTIONVLA
77.0ga.5
58.8_6.6
77.0ga.5
58.8266
64.8
54.0_7.0
In-Domain Performance First, we assess LAPA's ability learn from small subset of in- domain action label data by pretraining on 181k trajectories and finetuning on 1 action-laheled trajectories (0.5%). As shown in Table [ LAPA largely outperforms SCRATCH and narrows the gap with ACTIONVLE VLA despite not using action labels during pretraining, Additionally, LAPA sur- passes UNIP1 and VPT. Notably, while UNIPI handles simple tasks well, its diffusion model often generates incorrect plans for longer-horizon horizon tasks, aligning with Du et al. (2024) (see Figure [7] Appendix G.T VPT, with the same backbone VLM as LAPA, outperforms UNIPI, showing the su- periority of the VLA model, but still underperforms LAPA, highlighting the effectiveness of latent actions.
Cross-Task Performance We investigate whether LAPA's broad skills be retained after fine- tuning on a specific task. Pretraining LAPA on 181k trajectories and finetuning on only separate tasks (7k), evaluate all 5 task categories, similar the in-domain setup, assess latent pretrain- ing's 's benefits for unseen tasks. When comparing LAPA and SCRATCH in Table [ and Table 7 B in Appendix G..I, latent pretraining significantly benefits the separate task as well the other task categories, resulting in a significant boost in both seen and unseen setups. Like before, UNIPI is constrained by its diffusion model's planning limitations, while VPT performs strongly, even sur- passing ACTION VLA in the unseen setting. This is likely due to using more labeled data (7k vs. 1k), helping the IDM generate more accurate pseudo labels.
Cross-Environnen Performance We further investigate if Latent Action Pretraining benefits downstream performance when the pretraining and fine-tuning environments are different. We pre- train LAPA on 440k real-world trajectories, and then finetune 1k simulation trajectories, which can be seen as testing on a setup where a real2sim gap is present (Figure 9 (a)). From Table D observe that LAPA still significantly outperforms SCRATCHI, showing that latent pretraining leads to positive transfer even on cross-envirommett setting. Notably, both UNIPI and VPT sig- nificantly underperforms LAPA, showing that learning to predict latent actions is more robust to cross-enviromment transfer. VPT only results in minor positive transfer, indicating that the IDM is not robust to environment shifts.
4.4 REAL-WORLD RESULTS
We pretrain our models (1) Bridgev2 Walke al., 2023 to measure the cross-embodiment performance (WidowX embodiment for pretraining and Franka embodiment for finetuning) and (2) Open X-Embodiment Dataset (Collaboration al. 2023) to measure the effect of pretraining in

---

Published as a conference paper at ICLR 2025
Scrach
SperKLA (Br2gg)
Openiii 10
Bridge
LA 29 Mggul
LAN 10pazt)
4015 (44
22.3
Average
Kroc coljecto Over
Gover 520 Kc with Tikml
Pick conject > and put In sink
Figure 3: Real-world Tabletop Manipulation Results. We evaluate on total 54 rollouts each model encompassing unseen object combinations, unseen objects and unseen instructions. Average success rate (%) + StdEEr are shown (detailed results provided in Appendix G.3)
Table 2: Evaluation Results divided into eval types. We average the success rate across the tasks depending on what capability we trying quantify: seen objects but unseen combinations, (2) unseen objects, and new instructions requiring semantic reasoning. Best bolded and second best is underlined.
Seen Obj. Unseen Combo
Unseen Obj.
Seen Obj. Unseen Instr.
AVG
SCRATCH
18.0
20.3
25.4
21.2
ACTIONVLA (Bridge)
38.3
31.8
27.7
32.6
OPENVLA (Bridge)
34.6
22.1
LAPA (Bridge)
43.4
31.4
35.6
36.8
OPENVLA (Open-X)
46.2
42.1
43.4
43,9
LAPA (Open-X)
57.8
43.9
48.5
50.1
LAPA (Human Videos)
36.5
37.4
28.1
34.0
multi-embodiment setting. Figure 3 shows the average success rate across the tasks where each task encompasses unseen object combination, unseen object, and unseen instruction settings. We provide detailed results depending on the generalization type in Table 2
Bridgev2 Pretraining We compare models that were pretrained on the Bridgev2 dataset. Similar to previous results, all models pretrained on Bridgev2 result in significant performance enhancement compared SCRATCH. Furthermore, by comparing LAPA which does not leverage action-labeled trajectories during pretraining with models that use action-labeled trajectories during pretraining (ACTIONVLA and OPENVLA), we observe an interesting finding: LAPA outperform VLAs that use action labeled pretraining data on average success rate of the 3 tasks, unlike previous scenar- ios where VLAs pretrained on the ground-truth actions were upper bounds. LAPA significantly outperforms the other models in pick-and-place tasks; given that most tasks in Bridgev2 are pick- and-place, we hypothesize that VLA models pretrained on ground truth action labels have overfitted to the WidowX action space from the Bridgev2 dataset, hampering cross-embodiment adaptability to action distribution shifts during fine-tuning. In contrast, LAPA avoids this issue by not relying on ground truth action labels during pretraining.
Open-X Pretraining From Figure 3. we see that VLAs pretrained on the Open-X dataset out- performs VLAs pretrained on the Bridgev2 dataset, showing that data scaling during pretraining
demonstrates positive transfer for downstream tasks (Collaboration et al.) (023). This also suggests there could be significant further improvement when scaling the diversity and scale of the pretraining data, especially with large web-scale video data.
When comparing LAPA with OPENVLA, we see that LAPA significantly outperforms OPENVLA on 2 out of 3 tasks (Figure 3. Also, as shown in Table 2 VLA (Open-X) on all types of generalization settings. This highlights LAPA's effectiveness in a multi-embodiment setting by showcasing its ability to leverage a shared latent action space during pretraining, akin to how language and image representations are utilized. contrast, contemporary action pretraining methods may suffer from reduced positive transfer between datasets due to the variability in action representation spaces across different embodiments and datasets.
LAPA (Open-X) outperforms Open-
However, for pick and place task, LAPA underperforms OPENVLA. We observe that most failures of LAPA are due to early grasping. In fact, LAPA outperforms VLA in reaching performance (83.33% vs 66.67% (reaching performance for each task is provided separately in Appendix G.3
7

---

Published as a conference paper at ICLR 2025
This suggests that, although LAPA possesses stronger language conditioning and coarse- -aaaned planning abilities, there is room for improvement skills such as grasping. Since grasping occurs only once twice in each trajectory, the 150 labeled trajectories may not be sufficient for LAPA to accurately predict grasp actions based on the physical characteristics of diverse objects.
4.5
LEARNING FROM HUMAN MANIPULATION VIDEOS
47.8
41.0
13.3
10.6
2h.9
25.4
13.0
12.2
Sentcch Un
VA7
LARA
Average
Knock
Cover
Pick and place
(a) SIMPLER Results
(b) Real-world Tabletop Manipulation Robot Results
Figure 4: Pretraining from Human Video Results. Average success rate (%) StdErr LAPA and baselines
pretrained on human manipulation vidcos where the embodiment and environment gap is extreme. We evaluate on both simulation (left) and real-world robot setup (right).
In this section, we show results when we extend Latent Action Pretraining to human manipula- tion videos, which aligns with the main motivation of this work. Unlike robot trajectories, human videos have two challenges: human videos do not contain action labels, and the distribution of hu- man videos distinct from the robot embodiment (McCarthy et al.) 2024). We try to investigate whether our method well as baseline approaches could address these challenges by pretraining on Something-Somethieee V2 dataset (Goyal et al.] (2017) which consists of 220K videos that includes human performing actions with everyday objects.
We first evaluate the performance of LAPA pretrained on human videos on SIMPLER. In addition SCRATCH, we also compare with UNIPI and VPT pretrained with the same human video dataset. As shown in Figure 4a LAPA outperforms SCRATCH. showing that although the distribution of the pretraining data is distinct from the deployment setup, leveraging human videos for latent action pretraining results in positive transfer. Also, LAPA performs the best performance by outperforming UNIPI and VPT, implying that Latent Action Pretraining is robust to human to robot embodiment shifts. Note that it is impossible to train ACTIONVLA because the human videas do not have any robot action labels.
We report the real-world robot experiments in Figure 4b Surprisingly, we can see that LAPA trained with human videos outperforms OPENVLA (Bridge) on average. Despite the larger embodiment gap for LAPA (Human to robot vs. Robot to robot), it learns better prior for robot manipula- tion. Also, as shown in Table 2 LAPA (Human Videos) shows good generalization performance, especially for unseen objects. We conjecture that this is because Something Something V2 dataset interacts with much diverse objects compared to Bridgev2. This result highlights the potential of raw human manipulation videos from the web compared to expensive robot manipulation data, which quires time-intensive teleoperation to collect. We expect that applying our approach on large-scale internet videos (e.g., YouTube videos) could unlock the potential for large-scale pretraining of generalist action foundational model, similar to foundational models in NLP or Computer Vision.
4.6 PRETRAINING EFFICIENCY
The benefit of LAPA extends beyond downstream task performance include pretraining efficiency. For pretraining LAPA (Open-X), X), the best-performing model, use 8 H100 GPUs for 34 hours with a batch size of 128 (total of 272 H100-hours). In contrast, OPENYLA required a total of 21,500 A A100-hours with a batch size of 2048. Despite being approximately 30-40 times more efficient for pretraining, LAPA still outperforms OPENVLA
We calculate based on the fact that H100 GPUs lead to 2-3 times speedup compared to A A100 GPUs for training.
8

---

Published as a conference paper at ICLR 2025
We believe this efficiency stems from two factors: (1) the of the Large World Model (Liu al. 2024) as the backbone VLM model, and (2) the searse-grained actions LAPA compared to con- ventional action pretraining. First, the training objective during LWM pretraining includes generat- ing the next state, which corresponds to the next frame in video. We hypothesize that this objective enables the model implicitly understand high-level actions in video. Notably, ACTIONVLA (Bridge), which uses LWM as the backbone, and OPENVLA (Bridge), which uses Prismatic as the backbone, are trained on the same data and objective. However, ACTIONVLA reaches optimal per- formance (in terms of action token accuracy) in significantly fewer epochs (3 epochs) compared to OPENVLA's 30 epochs. Second, the action space for LAPA is much smaller than that for OPEN- VLA (8* vs. 256"), ) making learning the perception-and-lanee easier to learn. For all LAPA models (BridgeV2, Open-X, Human Videos), we observe that a single epoch of training is sufficient to achieve optimal performance.
to action generation problem
ABLATION AND ANALYSIS
5.1
SCALING MODEL, DATA, AND LATENT ACTION SIZE
257
56
54
54
56
52
50
50
46
55
48
54
75
150
300
10
50
100
46
30
(a) Model Scaling
(b) Data Scaling
(c) Latent Action Length
(d) Latent Action Vocab
Figure 5: Scaling Ablation Results LAPA. scale+ dimensions of LAPA: model parameters (in millions), data size (ratio among Bridgev2), and the latent action sequence and vocabulary size, and show the downstream average success rate (%) on the SIMPLER fine-tuning tasks.
Large Language Models (LLMs) have demonstrated scaling laws (Kaplan et al., 2020 where per- formance improves with increases in model size, dataset size, and computational resources used for training. Similarly, we attempt to analyze whether LAPA benefits from scaling across three dimen- sions: latent action quantization model size, data size, and latent action representation space. For a controlled setup, we apply our method to Bridgev2 and then fine-tune it on SIMPLER except for Language Table result of Figure 5c
As shown in Figure 5 scaling benefits LAPA across the three dimensions. Interestingly, we observe that the optimal scale of the latent action space depends the complexity of the action dimension contained in the pretraining dataset. For example, increasing the latent action sequence length is less effective compared increasing the vocabulary for Language Table (Figure 16). Except for Language Table, we maintain the generation space of LAPA at 84 throughout all of our main exper- iments. These results imply that when scaling pretraining Internet-scale videos that go beyond manipulation videos, scaling LAPA in terms of model, dataset, and latent action space could improve performance, especially capture higher action dimensions such whole-body control.
5.2 LATENT ACTION ANALYSIS
We qualitatively analyze the alignment of quantized latent actions with real continuous actions. For interpretation, we condition the current image observation xi and each latent action on the decoder of the latent action quantization model, and present the reconstructed images.
In Language Table, we observe that each latent action corresponds to distinct movement of the robot arm, with the distribution of latent actions being well-clustered in the actual 2D action space (shown in Figure 12] 13 of Appendix D Next, for human manipulation videos, observe that camera viewpoints also correspond to a latent action since the viewpoint changes within a video (shown in Figure 14 of Appendix E). We also analyze the latent actions learned from the Open- embodiment, which encompasses multiple embodiments, tasks, and environments. As shown in Figure 6 even though the embodiment and environment differ, conditioning on the same latent

---

Published as a conference paper at ICLR 2025
x
[1,1,3/2]
[3.2.0.1]
[4.2,4.1]
[5,1.2.7]
[5,3,5.2]
[6.7,0.2]
Actions
Down, Left
Up
Right, Rotate
Down
Left, Rotate
Up. Left
Figure 6: Latent Action Analysis. We condition the current observation Iι and quantized latent action to the decoder of the latent action quantization model. We observe that each latent action can be mapped into a semantic action. For example, latent action [1.1,3,2] corresponds to going down and left while [3,2,0,1] corresponds to going up little bit.
action results in a similar action in the reconstructed image. This supports our previous claim that latent actions are learned in shared representation space, regardless of the embodiment dataset, facilitating stronger positive transfer across diverse datasets.
X1
8
8
8
S
Figure 7: Closed loop rollout of LAPA. LAPA is conditioned on current image 21 and language instruction of 'take the broccoli out of the pot'. We generate rollout images by conditioning the decoder of Latent Action Quantization Model with latent actions generated by LAPA.
We also qualitatively analyze the coarse- -rarined planning capability of LAPA through a closed-loop rollout. We use a LAPA model that has only undergone pretraining, without any action finetuning. Since this model generates latent actions that are not directly executable in the real world, we con- dition the current observation æ1 and the predicted latent action from LAAAA with the decoder of the latent action quantization model. As shown in Figure 7 when conditioned on the current observa- tion and the instruction to 'take the broccoli out of the pot', LAPA generates robot trajectories that successfully reaches for the broccoli, moves down to grab it, and, the arm moves away from the pot, the broccoli disappears. This shows the potential for LAPA as a general-purpost robotic world model, not only predicting actions but also the outcomes of the actions. For example, this can lead to an extension of LAPA to act as a Task and Motion planning system, where it can first generate multiple plans given a natural language task instruction, choose the most optimal trajectory based on methods of quantifying the success among multiple trajectory candidates (Hwang et al.) (2024] Duan al. 2024).. and perform open-loop closed-loop inference. This can lead a paradigm where we aim to improve performance through scaling test-time compute, as with LLMs (Snell et al. (024).
6 LIMITATIONS AND CONCLUSION
In this paper, we introduce Latent Action Pretraining, a scalable pretraining method for building VLAs without using ground-truth action labels. Across three benchmarks spanning both simula- tion and real-world robot experiments, we show that our method significantly improves transfer to downstream tasks compared to existing approaches. We also present a state-of-the-art VLA model that surpasses current models trained on 970K action-labeled trajectories. Furthermore, we demon- strate that our method can be applied purely on human manipulation videos, where explicit action information is absent, and the embodiment gap is substantial.
We still face certain limitations. First, LAPA underperforms compared to action pretraining when comes to ine-grained motion generation tasks like grasping. We believe that increasing the latent action generation space could help address this issue. Second, similar to prior VLAs, LAPA also en- counters latency challenges during real-time inference. Adopting a hierarchical architecture, where a smaller head predicts actions at a higher frequency. Lastly, while we qualitatively demonstrate that latent action space captures camera movements (Figure [4), have not yet explored the application of LAPA beyond manipulation videos, such as those from self-driving cars, navigation, or landscape scenes. We leave these explorations for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Arhan Jain and Marius Memmel for helping out with the robot hardware and teleoperation setup. Also, we thank Minyoung Hwang, Jiafei Duan, Junsu Kim, and Changyeon Kim for helpful discussions and constructive feedback. This work was partly supported by Center for Advanced Urban Systems (CAUS) of Korea Advanced Institute of Science and Technology (KAIST) funded by GS E&C (40%) and the Institute of Information Communications Technology Planning & Evaluation(IITP) grant funded by the Korea governmen((MSIT) (RS- RRS-2024-00399966 Development of a Cybersecurity Specialized RAG-based sLLM Model for Suppressing Gen-Al Malfunctions and Construction of Publicly Demonstration Platform, 30%; No.RS-2022-112
Comprehensive Video Understanding and Generation with Knowledge-bar Deep Logic Neural Network, 20%; No.RS-2021-1212266. Artificial Intelligence Innovation Hub, 10%).
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from hu- videos versatile representation for robotics. In Proceedings ofthe the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Bowen Baker, Iige Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. In Advances in Neural Information Processing Systems, 2022.
Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hicrarchies using language. arXiv preprint arXiv:2403.0/822 2024.
Homanga Bharadhwaj. Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Zero-shot robot manipulation from passive human videos. arXiv preprint arXiv:2302.020//, 2023.
Homanga Bharadhwaj. Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arXiv:2409. /6283, 2024a.
Homanga Bharadhwaj. Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Pre- dicting point tracks from internet videos enables diverse zero-shot robot manipulation. arXiv preprint S00.::2-005005222 2024b.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Ri-2: laniu--annuag--actton models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 2023.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative inter- active environments. In Forty-first International Conference on Machine Learning, 2024.
Karl Cobbe. Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural genera- tion to benchmark reinforcement learning. arXiv preprint arXiv: 01.11..1888. 2019.
OX Emboodimeet Collaboration, A Padalkar, A Pooley, A Jain, A Bewley, A Herzog, A Irpan, A Khazatsky, A Rai, Singh, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.0886. 2023.
Sudeep Dasari, Mohan Kumar Srirama, Unnat Jain, and Abhinav Gupta. An unbiased look at datasets for visuo-motor pre-training. In Conference Robot Learning, 2023.
11

---

Published as a conference paper at ICLR 2025
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schu- urmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Thirty-eeentth Conference on Neural Information Processing Systems, 2023.
Yilun Du, Sherry Yang. Pete Florence, Fei Xia, Ayzaan Wahid, brian ichter, Pierre Sermanet, Tianhe Yu, Picter Abbeel, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tomp- son. Video language planning. In The Twelfth International Conference on Learning Representa- lions, 2024.
Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. Aha: A lasion-anguaagemoddll for detecting and reasoning over failures robotic manipulation, 2024.
Ashley D Edwards. Himanshu Sahni, Yannick Schroecker, and Charles L Isbell. Imitating latent policies from observation. arXiv preprint /805:00900 2018.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phat, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022.
Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, and Xuelong Li. Learning an action- able discrete diffusion policy via large-scale actionless video pre-training. In Advances in Neural Information Processing Systems, 2024.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Repreventations, 2022.
Minyoung Hwang, Joey Hejna, Dorsa Sadigh, and Yonatan Bisk. Motif: Motion instruction fine- tuning. arXiv preprint arXiv:2409. 10683, 2024.
Zhengyao Jiang, Yingchen Xu, Nolan Wagener, Yicheng Luo, Michael Janner, Edward Grefenstette, Tim Rocktäschel, and Yuandong Tian. H-gap: Humanoid control with a generalist planner. arXiv preprint arXiv:2312.02662. 2023.
Aditya Kannan, Kenneth Shaw, Shikhar Bahl, Pragna Mannam, and Deepak Pathak. Deft: Dexterous fine-tuning for real-world hand policies. arXiv preprini arXiv:2310. 19797. 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprini arXiv:2406.00 09246, 2024.
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences. In The Twelfth International Conference on Learning Representations, 2024.
Hadas Kress-Gazit, Kunimatsu Hashimoto, Naveen Kuppuswamy, Paarth Shah, Phoebe Horgan, Gordon Richardson, Siyuan Feng, and Benjamin Burchfiel. Robot learning as an empirical sci- ence: Best practices for policy evaluation. arXiv preprint arXiv:2409.094. 09491, 2024.
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. arXiv preprint a1X1v.2403.03111, 2024.
12

---

Published as a conference paper at ICLR 2025
Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burgert, Mu Cai, Yong Jae Lee, et al. Llara: Supercharging robot learning data for vision-language policy. arXiv preprint arXiv:2406.20092 2024a.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint M.Xxv::4005059441 2024b.
Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song, and Carl Vondrick. Dreamitate: Real-world visuomotor policy learning via video genera- tion. arXiv preprint wrXiv:2406.16800 2024.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08266. 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp. 1113- 1132. PMLR, 2020.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters. pp. 1-8. 2023. doi: 10. 111100/9RAA.22333992222
Robert McCarthy, Daniel CH Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas G Thuruthel, and Zhibin Li. Towards generalist robot learning from internet video: A survey. arXiv preprini arXiv:2404. 19664, 2024.
Atharva Mete, Haotian Xue, Albert Wilcox, Yongxin Chen, and Animesh Garg. Quest: Self- supervised skill abstractions for learning continuous control. arXiv preprint arXiv:2407.15840. 2024.
Zawalski Michat, Chen William, Pertsch Karl, Mees Oier, Finn Chelsea, and Levine Sergey. Robotic control via embodied
reasoning, arXiv preprint arXiv:2407.08693, 2024.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A univer- sal visual representation for robot manipulation. arXiv preprint (xXXv:2203.12600 2022.
Dantong NIu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roci Herzig. Llarva: Vision-action instruction tuning enhances robot learning. arXiv preprint arXiv:2406.1 1815, 2024.
Yuzhe Qin, Yueh-Hur Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, 2022.
Dominik Schmidt and Mingi Jiang. Learning to act without actions. In The Twelfth International Conference on Learning Representations, 2024.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016.
Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. videos. In Conference on Robot Learning, 2023.
Videodex: Learning dexterity from internet
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.
Mohan Kumar Srirama, Sudeep Dasari, Shikhar Bahl, and Abhinav Gupta. Hrp: Human affordances for robotic pre-training. arXiv preprint a40iv:2407. 189/ 2024.
13

---

Published as a conference paper at ICLR 2025
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv:2405.09818. 2024.
arXiv preprint
Octo Model Team, Dibya Ghosh. Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. /2211, 2024.
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408. 14837, 2024.
Mohammad Hassan Vali and Tom Bäckström. Nsvq: Noise substitution in vector quantization for machine learning. IEEE Access, 10:1398-13610, 2022. doi: 10.1 10.1109/ACCESS::2222.1111
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning- In Advances in Neural Information Processing Systems, 2017.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ruben Villegas. Mohammad Babacizadeh, Pieter-Jan Kindermans, Hernnn Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022.
Ruben Villegas. Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2023.
Homer Rich Walke, Kevin Black, Tony Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, 2023.
Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and An- ima Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422 2023.
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprini urXi::2401.00025, 2023.
Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In The Twelfth International Conference on Learning Representations, 2024.
Jingyun Yang, Zi-ang Cao, Congyue Deng, Rika Antonova, Shuran Song, and Jeannette Bohg. Equibot: Sim (3)-equivariant diffusion policy for generalizable and data efficient learning. arXiv preprint arXiv:2407.0l 01479, 2024a.
Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kael- bling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In The Twelfth International Conference on Learning Representations, 2024b.
Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Position: Video as the new language for real-world decision making. Proceedings of the 4/st International Conference on Machine Learning. 2024c.
Jianglng Ye, Jiashun Wang, Binghao Huang, Yuzhe Qin, and Xiaolong Wang. Learning contin- uous grasping function with a dexterous hand from human demonstrations. IEEE Robotics and Automation Letters, 5(5):2882-2889 2023.
Jia Zeng, Qingwen Bu, Bangjun Wang, Wenke Xia, Li Chen, Hao Dong, Haoming Song, Dong Wang, Di Hu, Ping Luo, et Learning manipulation by predicting interaction. arXiv preprint arXiv:2406. 20w.2000000 2024.
Yifeng Zhu, Arisrei Lim, Peter Stone, and Yuke Zhu. Vision-based manipulation from single human video with open-world object graphs. arXiv preprint arXiv:2405.2033 2024.
14

---

Published as a conference paper at ICLR 2025
A
LATENT ACTION QUANTIZATION DETAILS
X1
X
Latent Actions
21
Patch Emb
Pstch Emb
NSVO
Spatial Transformer
Spatial Transformer
Spatial Transformer
Patch Emb
Causal Transformer
Discretize
2
2
Encoder
Decoder
Figure 8: Model architecture of our Latent Action Quantization Model.
We show model architecture details of our latent action quantization model in Figure g We utilize the C- ViViT model architecture from Villegas al. (2022) to replicate the latent action model from GENIE (Bruce et al. 2024). During the encoding process, both and #2 are gone through patch embedding to obtain pl and P2 and gone through a spatial transformer. To convey temporal infor- mation, the representations for both outputs of spatial transformer (transformer with causal positional encodings) to get e₁ and e2 continuous embeddings. We then define d1 C2 ej and discretize d1 by finding the closest embedding from the codebook where the codebook size is determined as hyperparameter.
passed causal transformer
21 = arg min dd zk 2
(1)
When obtaining é, and 62, we go through a CNN network. The sequence length is designated by the kernel size, stride and padding value of CNN network. After quantization, we apply NSVQ technique before decoding.
|d] 21
di d1
(2)
where v ~ (O(,,)) 1). For decoding, we go through the following equation:
4 D(Attn(sglpp))
where stop gradient (sg) is applied to p1 to avoid representation collapse and cross attention used to attend dj given P1. Unlike the encoder, decoder D only include spatial transformer. The training objective is L2 reconstruction loss.
L |002 2f2113
15

---

Published as a conference paper at ICLR 2025
Table 3: Pretraining and fine-tuning dataset for each environment. Cross-Env denotes cross-enviromment, Cross-Emb denotes cross-embodimett, and Multi-Emb denotes multi-embodiment. For fine-tuning, notes multi-task training and MI denotes tasks with diverse multi-instructions. Category denotes the main capability we are trying to quantify.
Pretraining
Fine-tuning
Environment
Category
Dataset
Trajs
Dutaset
# Trajs
In-Domain
Sim (All tasks)
181k
Tasks (MT, MI)
1k
LangTable
Cross-Task
Sim (All tasks)
181k
Task (MI)
7k
Cross-Fnv
Real (All 5 tasks)
442k
5 tasks (MT, MI)
1k
In-Domain
Bridgev2
60k
4 Tasks (MT)
100
SIMPLER
Cross Cross-Emb
Something v2
200k
4 Tasks (MT)
100
Cross-Emb
Bridgev2
tasks (MI)
450
World
Multi-Emb
Open-X
970k
(MI)
450
Cross-Emb
Something
200k
tasks (MI)
450
After latent model training, we utilize the 21 as the latent action label for æi. The encoder can be seen as the inverse dynamics model and the decoder can be seen as the world model.
B DETAILS ON EXPERIMENTAL SETUP
BridgeData V2
Put Carrol Plate
Pul Spoon Towel
Stack Cubes
Put Engplant in Basket
(a) LANGUAGE TABLE
(b) SIMPLER
(c) REAL
Figure 9: Experimental Setups. (a) shows example from the 440k real-world trajectories (top) and the 181k simulation trajectories (boutom) from Language Table Benchmark. (b) shows the different evaluation tasks we use with the SIMPLER environment. (c) shows the three different tasks that we perform in the real-world.
Language Table Experimental Setup Figure 9 (a) shows examples of the Language Table setup. For Language Table experiments, we train VLA-based -aaed models to generate language directions (e.g. 'move up') before actual actions following Belkhale al. (2024). which significantly improved the performance [ For evaluation, evaluate 50 evaluation rollouts for each subtask category where the initial locations of the objects are randomized for each evaluation. Further details can be found in
SIMPLER Experimental Setup Figure 9 (B) shows examples of the SIMPLER setup. The SIM- PLER environment does not provide any fine-tuning data for their evaluation pipeline, Thus, we first train our underlying VLM on the Bridgev2 dataset and perform zero-shot rollout the 4 tasks in SIMPLER. Note that we use held-out trajectories differing in object orientation and position from the evaluation setup. We filter 25 successful trajectories for each task (total of 100) and use them as the fine-tuning dataset for all our experiments. For evaluation, we evaluate on 24 rollouts
For DOF robot experiments, we found the benefit of generating language directions to be marginal compared to the increased inference cost. Therefore, only generate delta end-effector actions other experiments.
16

---

Published as a conference paper at ICLR 2025
per task while randomizing the initial object locations. We consider Bridgev2 and SIMPLER to be in-domain since they show a high correlation between real-world and simulation results with their simulation benchmark. Further details can be found in https//ggttbbbcommmmmm/eeeees/sp
Real-world Tabletop Manipulation Experimental Setup Figure 9 (C) shows examples of the real-world tabletop manipulation experimental setup. For the teleoperation, we use the polymetis robotic stack] to collect 150 trajectories for each of the tasks. All of the tasks require multi- instruction following capabilities since there are 3 objects in the scene and the model has to condition on the task description infer which object to interact with. Figure 10 shows samples of each task. For each task, we aim to quantify 3 distinct capabilities:
(1) We test the ability to infer the correct object from the task description between unseen com- bination of seen objects during fine-tuninig, (2) We test the ability to infer the correct object from totally unseen objects during fine-tuning that may or may have not been observed during pretraining. Specifically, the knocking tasks was conducted with real-world objects that were highly unlikely to have been in any of the pertaining datasets. (3) We test the ability infer the correct object (among seen objects, unseen combinations) from a totally unseen instruction that requires semantic reason- ing (e.g. Pick up a spicy object), For each evaluation criteria, 6 rollouts are performed for each models, resulting in a total of 18 rollouts for each task category. Since there are three tasks, each model evaluated with 54 rollouts in the roaldworld. We provide the full list of all of the seen and unseen objects used for each rollout in Table 13
15
and the total average success rates in Table
16
Furthermore, for fair comparison, we match the image resolution during training of all of our models and use the exact same object initial positions for all of our evaluation, mostly on the same day to minimize variability. For evaluation metrics, we adapt a partial success criteria for fine- grained evaluation, following Kim et al. (2024), which we describe in detail below.
Knock down the <object>.
For knocking, we give 0.5 partial score if the robot reaches to the correct object and 1 if the robot knocks down the correct object.
Cover the <object>> with a towel.
For covering, we give 0.33 partial score if the robot picks up the towel correctly, 0.66 if the robot reaches to the correct object or if the towel partially covers the object, and if the correct object is completely covered by the towel.
Pick up the <object>> and put it in the sink.
For pick and place, we give 0.25 for reaching to the correct object. 0.5 for grasping the object, 0.75 for grasping and moving the object towards the sink, but failing to place the object in the sink, and 1 for placing the correct object in the sink.
C BASELINE DETAILS
For UNIPI, we use diffusion model from (Ko al., 2024) which can be trained on 4 A100 GPUs. For all experiments, we train with 128 batch. We use the same inverse dynamics model as VPT during inference. To mediate estimation errors between the predicted video plans and executed actions being accumulated, we periodically conduct replanning by regenerating new video plans after executing two actions. For VPT, we use ResNet18 followed by an MLP layer for the inverse dynamics mIdel(IDM). The IDM is trained to predict an action when given two frames on a single A6000 GPU using using Adam optimizer with a learning rate le-4. For OLE (Bridge), we pretrain on Bridgev2 for 30 epochs with batch size of 1024. For OpenVLA (Open-X), we the pretrained checkpoint from Kim et al. (2024). For finetuning, we use LoRA finetuning (Hu et al., 2022) with batch size of 32. We have observed that full-finetuning and lora finetuning leads to similar performance, so we use LoRA finetuning as default for efficient fine-tuning. We finetune the model until the training action accuracy reaches 95%. For ACTION and LAPA, we train with a batch size of 128 and with image augmentation for real-world finetuning.
17

---

Published as a conference paper at ICLR 2025
D EXPERIMENTAL RESULT ANALYSIS
Table 4: Pretraining trajectories statistics for downstream tasks. Number of trajectories that the task with evaluation task for each pretraining dataset: Bridgev2, Open-X, and Something Something (Sthv2) dataset.
Task
Bridgev2
Open-X
Sthv2
Knocking
7,969
6,655
Covering
898
5,026
6,824
Pick & Place
10,892
911,166
3,272
We further analyze the real-world robot results shown in Figures and 4b focusing on how the task distribution in pretraining data impacts downstream performance. Table g presents the num- ber of trajectories corresponding to each evaluation task (Knocking, Covering, and Pick & Place) across pretraining datasets (Bridgev2, Open-X, X. and Something Something V2 (Sthv2)), determined through lexical matching. We expect future work use other methods of analyzing the relationship between pertaining and fine-tuning task distributions that capture semantics of the task rather than simple lexical matching. We perform this analysis to get a sense of how the task distribution in the pretraining data affects downstream task performance.
Knocking There almost knocking-related trajectories in Bridgev2. This scarcity may plain why models trained on Bridgev2 performed worse compared to those trained on Sthv2, despite larger embodiment gap in the Sthv2 dataset (Figure 4b).
Covering similar trend observed for the covering task. Given that the number of covering tra- jectories in Bridgev2 is relatively small compared to the Sthv2 dataset, models trained on Bridgev2 occasionally underperform compared to LAPA trained on Sthv2.
Pick & Place For the pick and place task, the trend reverses. The number of pick and place tasks in Sthv2 is relatively small compared to Bridgev2 and Open-X, which might explain why LAPA trained on Sthv2 significantly underperforms models trained on Bridgev2 or Open-X. Based on these results, to a more robust generalist policy compared to training robot videos with narrower skill sets. We also expect future research to provide more in-depth analysis of the relationship between task distribution in pretraining data and performance on downstream tasks.
expect that pretraining videos encompassing wide range of skills will lead
illustrated in Fig- ure I1 LAPA outperforms VLE in 65.4% when disregarding the ties. When considering the ties, LAPA outperforms VLA in 31.5% of cases, while Open VLA prevails only 16.7%. In- terestingly, they tie in 51 31.9% of the trials, suggesting that in about half the instances, both models either fail or achieve similar partial success score. Note that these evaluations were performed while ensuring that the target and distractor objects were in identical initial locations during eval- uation, alternating the models during evaluation. These results provide insight into the statistical significance of the comparison, supporting the use of multiple metrics to ensure more comprehen- sive evaluation of physical robot performance in real-world scenarios (Kress-Gaas et al. 2024). not only the average success-rate across all of the evaluation rollouts.
We also present the win rate of LAPA (Open-X) against Open VLA (Open-X)
E DETAILED LATENT ACTION ANALYSIS
We provide further qualitative analysis of LAPA. First, we analyze latent actions learned from Lan- guage Table with vocabulary size of and sequence length of 1. In Figure [2 we show that each latent action corresponds to semantic action (0: Move left and forward, I: Move left and back,
2: Move right and back, 3: Move right slightly, 4: Move right, 5: Move back, 6: Do not move,
7: Move forward). We observe that increasing the latent action vocabulary size leads to capturing a more fine-grained information. We analyze the relationship between latent actions with ground- truth 2 DOF actions by mapping each instance into latent action space. As shown in Figure 13
18

---

Published as a conference paper at ICLR 2025
observe that latent actions well-clustered in the actual 2D action space, indicating that latent actions are meaningful representations that are highly related to actual continuous actions.
We further analyze the latent actions learned from human manipulation videos using the Something- Something V2 dataset. As illustrated in Figure [4 these latent actions capture not only hand move- ments but also camera movements. Since the camera viewpoint varies throughout the videos in the Something-SSmetiinne
V2 dataset due to the videos being egocentric, our latent action quantization model also learns represent camera movements. For instance, latent actions [3,5,2.7] and [5,6,7.6] correspond to slight downward camera movement, [4,0,0,4] and [2,3,6,6] indicate rightward move- ment, and [4,2,0,0] and [5,7,0.5] represent subtle upward camera shifts.
F ADDITIONAL ABLATION RESULTS
We first analyze the effect of window size H for latent action quantization process. For all robot manipulation videos, we have determined the window size depending on the fps of the video so that the next frame models 0.6 seconds ahead from the current frame. For human manipulation videos, we have set the next frame to be 2.4 seconds ahead since we qualitatively observed that many of the frames of the human videos contain much less dynamic actions compared robot videos. (However, we think that filtering these frames could make the window size the same as robot videos, which we leave as future work.). We have added an ablation experiment on the window size for robot videos (Bridgev2) by evaluating on SIMPLER in Figure [15a] Note that the default is H 3 because Bridgev2 collected with 5hz. The results show that LAPA quite robust to different window sizes. However, if the window size is extremely large, performance degradation is observed. This is expected since our quantization model is relatively small (300M parameters), faces difficulties modeling latent information when the visual deltas are significant.
We also analyze the data scaling in terms of fine-tuning data by comparing with SCRATCH. We evaluate SIMPLER. As shown in Figure [5b] LAPA (Bridge) consistently outperforms SCRATCH even when the fine-tuning data instances are small while the absolute performance increases with larger fine-tuning data.
For data scaling, we also analyze the data scaling of human videos. We compare LAPA trained from 10% of Sthv2 human video dataset with LAPA trained from the whole Sthv2 human video dataset. Results in Table [2 show that scaling the human video datasets boosts the performance for SIMPLER benchmark not only for the final success for all subtasks. We leave exploring scaling law for human videos more extensively or future work, since showing scaling law requires intensive computational resources do different ablations of model size, data size, and computational resources.
Finally, we vary the latent action length and vocabulary size in Language Table, extending the result of Figure which which was analyzed in Bridgev2 data. As shown in Figure 16 increasing the sequence and vocab size increases the performance. However, unlike SIMPLER, we observe that the increas- ing the latent action vocab size much more effective compared to increasing the latent action sequence length in terms of absolute performance. This implies that for environments that are visu- ally simple, increasing the latent action vocabulary might more effective compared to sequence length.
G DETAILED EXPERIMENTAL RESULTS
G.1 LANGUAGE TABLE
We provide the detailed results of the experiments performed on the Language Table benchmark
10 For all of the tables in the appendix, we bold the best result among the comparisons and underline the second best. Each value denotes the success rate (%). 50 evaluation rollouts are performed for each task category, resulting in 250 total evaluation rollouts per model for each table.
in Table 5 6
We also show the qualitative result of UNIPI where the diffusion model generates the correct plan for simple and short-horizon tasks (e.g. separate tasks). However, the diffusion model generates the wrong plan corresponding to the instruction when the task requires longer horizon planning (Figure 17
19

---

Published as a conference paper at ICLR 2025
Table 5: Language Table In-Domain Seen Results.
SCRATCH
UNIPI
VPT
LAPA
ACTIONVLA
Block2Block
4.0
14.0
36.0
58.0
76.0
Block2Absolute
6.0
4.0
38.0
56.0
72.0
Block2BlockRelaivee
10.0
12.0
48.0
52.0
76.0
Block2Relative
6.0
10.0
26.0
48.0
70.0
Separate
52.0
72.0
70.0
96.0
90.0
AVG
15.6
22.4
43.6
62.0
76.8
Table 6: Language Table In-Domain Unseen Results.
SCRATCH
UNIPI
VPT
LAPA
ACTIONVLA
Block2Block
4.0
26.0
62.0
Block2Absolute
10.0
42.0
58.0
BlocktveockRelativee
2.0
20.0
48.0
Block2Relative
8.0
32.0
44.0
Separate
48.0
44.0
44.0
82.0
AVG
15.2
13.2
32.8
49.6
58.8
Table 7: Language Table Cross- Task Seen Results.
SCRATCH
UNIPI
VPT
LAPA
ACTIONVLA
Block2Block
18.0
12.0
74.0
76.0
Block2Absolute
8.0
6.0
56.0
62.0
72.0
BlockcvveekRelativve
6.0
2.0
62.0
72.0
76.0
Beock2Relative
24.0
16.0
72.0
60.0
70.0
Sepurate
80.0
68.0
96.0
98.0
90.0
AVG
27.2
20.8
72.0
73.2
76.8
Table 8: Language Table Cross- Task Unseen Results.
SCRATCH
UNIPI
VPT
LAPA
ACTIONVLA
Block2Block
16.0
4.0
66.0
46.0
62.0
Block2Absolute
10.0
10.0
56.0
52.0
58,0
Block2BlockRelative
8.0
10.0
46.0
48.0
48.0
Block2Relative
12.0
4.0
52.0
38.0
44,0
Separate
66.0
52.0
84.0
90.0
82.0
AVG
22.4
16.0
60.8
54.8
58.8
Table Language Table Cross-Environment Seen Results.
SCrAtCh
UNIPI
VPT
LAPA
ACTIONVLA
Block2Block
4.0
4.0
16.0
26.0
66.0
Block2Absolute
6.0
4,0
8.0
16.0
58,0
Reockive
10.0
8.0
6.0
20.0
62,0
Block2Relative
6.0
4,0
12.0
22.0
54,0
Sepurate
52.0
48.0
48.0
84.0
84,0
AVG
15.6
13.6
18.0
33.6
64.8
G.2 SIMPLER
We provide results of various models evaluated on SIMPLER environment. Table II shows the setting where baseline models are pretrained Bridgev2 and then finetuned on SIMPLER roll- outs (100 videos). The results show detailed results for each task (stack green to yellow block, put carrot plate, put spoon otowel, put eggplant in basket) and subtasks (grasping and mov- ing). As shown in Ta] UNIPI significantly underperforms all other baselines on the SIMPLER Environment. We observe that, although the generated plans from the diffusion models quite
20

---

Published as a conference paper at ICLR 2025
Table 10: Language Table Cross-Eavironment Unseen Results.
SCRATCH
UNIPI
VPT
LAPA
ACTIONVLA
Block2Block
8.0
2.0
2.0
30.0
38.0
Block2Absolute
10.0
6.0
4.0
14.0
48.0
Block2BlockRelative
2.0
6.0
2.0
10.0
50.0
Block2Relative
8.0
4.0
40.0
54.0
Separate
48.0
42.0
44.0
76.0
80.0
AVG
15.2
12.0
18.4
29.6
54.0
accurate, the IDM lacks the capability to predict 7 DOF continuous actions accurately when given only 100 action-labeled trajectories. Specifically, we observe that UNIPI often fails to grasp the ob- ject within the maximum step limit. This implies the effectivness of using VLAs in scenarios with insufficient action-labeled data. Similar to the results of Section 4.3 LAPA outperforms baseline models that pretrain without using ground-truth action labels (UNIPI and VPT) and closes the per- formance gap with ACTIONVLA, which pretrained on all of the 60K action-labeled trajectories from the Bridgev2 dataset. This highlights the effectiveness of LAPA, OPENVLA fine-tuned on SIM- PLER trajecotries for reference. The performance of OPENVLA (36.4) is similar Scratch. The bad performance of OPENVLA on SIMPLER is a well known issue which is due to OPENVLA not being robust real-to-sin transfer for SIMPLER.
when the complexity
of the action space increases. We also evaluate the performance
We also provide detailed results of the setting where baseline models are pretrained on human ma- nipulation videos (Something Something V2 dataset) and then finetuned on SIMPLER rollouts (100 videos) in Table 12 We only compare to UNIPI, VPT, and LAPA since ACTIONVLA could not be trained without ground-truth action labels.
Table SIMPLER results Bridgev2 Pretraining. Success, Grasping, and Moving Rates (%) SIM- PLER environment. We pretrain UNIPI, VPT, I.APA on Bridgev2 dataset without using ground-truth action labels and ACTION on Bridgev2 using action lahels. We also add the result of OPEN VI.A fine- tuned on SIMPLER trajectories for reference. The main 4 tasks are: stack green to yellow block, carrot on plate, put spoon towel, and put eggplant in basket. Best is bolded and second best underlined.
Success Rate
SCRATCH
UNIPI
VPT
I.APA
ACTIONVLA
OPENVLA
Stack G2Y
29.2
2.7
45.8
54.2
75.0
41.6
Carro12Plate
29.2
37.5
45.8
50.0
Spwon2Towel
50.0
0.0
70.8
70.8
70.8
37.5
Figgllan/BBask
29.2
0.0
50.0
58.3
50.0
16.7
AVG
34.4
51.0
57.3
63.5
36.4
Grasping Rate
Grasp Green Block
66.6
20.8
62.5
62.5
87.5
50.0
Grasp Carrot
45.8
58.3
Grasp Spoon
70.8
22.2
79.2
83,3
83,3
45.8
Grasp Eggplant
62.5
16.0
70.8
83.3
75.0
37.5
AVG
61.4
23.1
66.7
71.9
80.2
50.0
Moving Rate
Move Green Block
58.3
29.1
58.3
66.6
91.6
70.8
Move Carrot
45.8
70.8
75.0
Move Spoon
70.8
34.6
79.2
83.3
79.2
75.0
Move Eggplant
87.5
70.8
87.5
AVG
65.6
42.6
68.7
27.1
88.5
63.7
G.3 REAL-WORLD
We also provide the full list of objects and the partial success recorded for each of the evaluation rollout: Knocking (Table [3). Covering (Table 14). and Pick & Place (Table 15). The total average success rate is provided in TTble|T6).
21

---

Published as a conference paper at ICLR 2025
SIMPLER results of Human Manipulation Video Pretraining, Success, Grasping. and Moving Rates (%) in SIMPLER environment. We pretrain UNIPI, VPT, and LAPA on Something-Sometinne V2 datasct without using ground-truth action labels. The main 4 tasks stack green yellow block, put carrot on plate, put spoon towel, and put eggplant basket. Best bolded and second best is underlined.
Table 12:
Success Rate
VPT
UNIPI
LAPA
LAPA (10%)
StackG2Y
50.0
0.0
50.0
45.8
Carrot2Plate
29.1
1.3
50.0
41.6
Spoon2Towel
37.5
1.3
50.0
66.6
Eggplant2Bask
66.6
0.0
58.3
45.8
AVG
45.8
0.7
52.1
50.0
Grasping Rate
Grasp Green Block
66.6
2.7
50.0
Grasp Carrot
45.8
31.7
62.5
41.6
Grasp Spoon
70.8
21.7
75.0
70.8
Grasp Eggplant
91.6
6.8
70.8
62.5
AVG
68.7
15.7
66.7
56.2
Moving Rate
Move Green Block
62.5
2.7
62.5
50.0
Move Carrot
58.3
37.5
70.8
58.3
Move Spoon
54.1
18.1
75.0
79.2
Move Eggplant
91.6
50.3
83.3
62.5
AVG
66.6
27.1
72.9
62.5
Table 13: Knocking Task Results
OpenVL.A (OpenX)
LAPA (OpenX)
OpenVLA (Bridge)
LAPA (Bridge)
ActionVI.A (Bridge)
LAPA (Sthv2)
Scratch
Seen Objects, Unseen Object Combinations
flamingo
0.5
0.5
0.5
pistachios soft scrub
0.5
1
0.5
0
white cup
0
0
0.5
mustard
1
0
water bottle
0.5
0.5
SUM
2.5
3.5
1.5
2
Unseen Objects
pringles
0.5
0.5
hersey's s chocolate syrup
0
0
popcorn
skittles
0
green board marker
0.5
0.5
0.5
0.5
0.5
paper towel
SUM
2
2
1.5
1.5
0.5
1.5
Seen Objects, Unseen Instructions
drink contains orange to eat with milk object used cleaning something to wash dishes the rectangle object
0.5
0
1
0
0
0.5
0.5
0.5
0.5
0.5
0.5
SUM
3.5
4
1
2.5
Success Rate (Strict)
27.78%
44.44 %
5.56%
11.11%
22.22%
0.00%
22.22%
Success Rate
38.89%
52.78%
25.00%
25.00%
33.33%
13.89%
30.56%
Reaching Success Rate
50.00%
61.11%
44.44%
38.89%
44.44%
27.78%
38.89%
22

---

Published as a conference paper at ICLR 2025
Cover <object> with towel
Pick up <object> and put it in the sink
Knock <object> down
Seen Objects
Unseen Objects
Figure 10: Real-world Tabletop Manipulation Examples.
23

---

Published as a conference paper at ICLR 2025
Win-Rate Percentage: OpenVLA vs LAPA
Percentage: OpenVIA
LANA
OperAs
Iescetoge
Pecoeengee
(a) Win rate (%) disregarding ties.
(b) Win rate (%) with ties.
Figure 11: Pairwise win rate (%). We compare pairwise win-rate of VLA and LAPA across the 54 evaluation rollouts the yeal-world (a) shows the win-rate while ignoring the ties and (b) shows the ties together with the individual wins.
1
3
4
6
Actions
Forward & Left
Back & Left
Back & Right
Right Slightly
Right
Back
Do not Move
Forward
Figure Latent Action Analysis in Language Table. We condition current observation an quantized latent action to the decoder of the latent action quantization model. We observe that each latent action can be mapped into semantic action. For example, latent action O corresponds to moving bit left and forward and corresponds to moving bit left and back.
Scatter Pot Actions and Coordinates
-20
Figure 13: Correlation of latent action with ground-truth actions When we map latent actions to ground- truth 2 DOF actions of Language Table, observe that latent actions are well-clustered in the actual 2D action space.
24

---

Published as a conference paper at ICLR 2025
1
[2,3,6,6]
[3,5,277]
[4,0,0,4]
[4,2,0,0]
[5,6,766]
[5,7,0,5]
Camera View
Right
Down
Right, Down
Up
Down
Left
Figure Latent Action Analysis Human Manipulation Videos. We condition the current observation 31 and quantized latent action to the decoder of the latent action quantization model. We observe that each latent action can be mapped into semantic action including camera movements. For example, latent action [3,5,277] corresponds to moving the camera hit down while [4,2,0.0] corresponds moving the camera slightly up.
AVG Success Rate 196)
57
AVG Success Rate 1%6)
LAPA
50
Scratch
55
40
53
30
51
220
49
5
10
10
30
50
100
(a) Window Size
(b) Fine-tuning Data Scale
Figure 15: Additional Ablation Results of LAPA. We further analyze the performance of LAPA by varying the window size latent action quantization and fine-tuning data scale. report average success rate SIMPLER.
64
59
58
62
57
60
56
58
55
56
54
54
2
256
1024
(a) Latent Action Seq
(b) Latent Action Vocab
Figure 16: Ablation Results of LAPA in Language Table. We try various latent action vocab and sequences of LAPA and show the downstream average success rate (%) on the Language Table fine-tuning tasks.
Figure 17: Success and Failure Cases of UNIP1. (Top) Given the instruction 'move the green block away from the red cube and red pentagon', the diffusion model of UNIPI successfully generates the plan. (Bottom) Given the instruction of "put the blue moon toward the yellow block', the diffusion model fails to generate the correct plan.
25

---

Published as a conference paper at ICLR 2025
Table 14: Covering Task Results
OpenVLA (OpenX)
LAPA (OpenX)
VLe (Bridge)
LAPA (Bridge)
Action (Bridge)
LAPA (Sthv2)
Scratch
Seen Objects, Unseen Object Combinations
icecream
0.33
0.33
0.33
0.33
0.33
0.33
strawberry
0.33
1
0.33
0.33
1
pepper
0.33
0
0.33
0.33
0.33
0.33
0.33
watermelon
0.33
0.33
0.33
0.33
0.33
0.33
lego block
0.66
0.33
0.33
pink duck
0.33
0.33
0.33
0.33
0
0.33
SUM
2.31
3.66
2.65
3.32
2.65
1.99
2.32
Unseen Objects
donut
0.33
0.66
0.66
0.66
0.33
orange
0.33
0.33
0.33
mushroom
0.33
0.33
0.33
0.33
0.33
0.33
0.33
yellow lego block
0.33
0.33
peas
0.33
ege
0.33
0.66
0
SUM
2.32
3.66
3.98
2.66
2.98
2.65
3,99
Seen Objects, Unseen Instructions
drink
0.33
0
0.66
0.33
0.33
0.66
yellow object
0.66
0.66
0.33
0
0.33
fruit
0.33
0.33
0.33
0.33
0.33
vegetable
0..3
0.33
0.33
0.33
0.33
0.33
edible object
0.33
0.33
0.33
0.33
condiment
0.33
0.33
0.33
0.33
0.33
0.33
SUM
2.31
1.98
1.98
1.66
1.98
2.32
2.31
Success Rate (Strict)
5.56%
33.33 %
16.67%
27.78%
11.11%
16.67%
22.22%
Success Rate
38.56%
51.67 %
47. 47.83%
42.44%
42.28%
38.67%
47.89%
Reaching Success Rate
16.66%
38.89 %
38.89 %
27.78%
22.22%
22.22%
27.78%
26

---

Published as a conference paper at ICLR 2025
Table 15: Pick & Place Sink Task Results
OpenVLA
LAPA (OpenX)
OpenVLA (Bridge)
LAPA (Bridge)
ALtion (Bridge)
LAPA
Seratch
(OpenX)
(Sthv2)
Seen Objects, Unseen Object Combinations
milk
1
1
0
orange lego block
0
0
0
ketchup
0.25
0.25
0.25
0.25
0
0
0.75
1
0.25
0,25
0.25
0.25
iceeream
0.25
0
0
1
0
salt
0
0.25
0
0
0
SUM
3.5
3.25
2.25
2.25
0.25
Unseen Objects
carrot
0.25
0
0.25
0.25
0.25
ycllow paprika
1
0
0.25
0.25
0
ycllow eube
0.5
0.25
0.5
0
0
salmon sushi
0.25
0
0.5
0
0
orange
0
0
0
0.25
blue cube
0,25
0.25
0
0
0
SUM
4.25
2.25
0.25
1.5
1.25
0.5
1.25
Seen Objects, Unseen Instructions
object that is yellow object that is round an object that is fruit object that you can drink
1
0
0.25
0
0
0.25
0
0
0.25
1
1
0
1
0.75
0
0,25
0
0.5
0
0
an object that is vegetable
0
0
0
0
0
0
an object that is an animal
0,25
0
0.25
0.25
0
SUM
2
2.75
1
2.75
0.5
1.25
0.75
Success Rate (Strict)
50.00
27.78%
16.67%
27.78%
16.67%
5.56%
16.67%
Success Rate
54.17
45.83%
19.44%
43.06%
22.22%
11.11%
23.61%
Reaching Success Rate
66.67%
83.33%
27.78%
72.22%
38.89%
27.78%
33.33%
Table 16: Summary of Total Success Rates (%)
OpenVLA (OpenX)
LAPA (OpenX)
OpenVLAA (Bridge)
LAPA (Bridge)
ActionVI.A (Bridge)
LAPA
Scratch
(Sthv2)
Total Success Rate
43.87% 27.78%
50.09% 35.19%
30.76% 12.96%
36.83% 22.22%
32.61% 16.67%
21.22%
34.02%
Total Success Rate (Strict)
7.41%
20.37%
27


Reviews:
[{"review_id": "h5BvA2NBQf", "reviewer": "Reviewer_CUQQ", "summary": "This paper proposes an unsupervised pre-training method, LAPA, for vision-language-action (VLA) models that eliminates the need for action labels. LAPA uses a VQ-VAE structure to learn quantized action latents from frame differences. Subsequently, the authors train a VLA model to predict these quantized latents based on frames and language instructions. This model can then be fine-tuned on small-scale robot manipulation datasets. Through experiments on both simulated and real-world datasets, LAPA demonstrates improved performance over selected baseline models to indicate its generalization ability.", "questions": "- Need clarification for pretraining stage: In table 1, authors provide details about the dataset for both pre-train and finetune phase, however, it remains unclear about the latent action quantization. My understanding is that latent action quantization is part of the pretraining so both latent quantization  (section 3.1) and latent pretraining (section 3.2) use the same dataset. Clarification of the dataset usage could provide a better understanding of the experiment set up. \n\n- In most of the experiments, LAPA is trained with a single large scale dataset for fair comparison with other models like OpenVLA. Since LAPA does not require any action label during pre-training, it would be interesting to see having all three datasets used together can provide benefit of downstream performance boost to further backup the scalable claim.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Originality:\n- The proposed method removes the need of action labels for pre-training VLA models which significantly increase the data availability.\n- Training VQ to predict delta between frames is a simple and scalable way of learning coarse latent action. \n- A significant performance improvement compared to SoTA (OpenVLA) model under various scenarios and relatively small performance gap between the upper bound case (ActionVLA) and LAPA. \n\nQuality:\n- The proposed method is technically sound.\n- Extensive experiments are conducted to evaluate LAPA performance under various scenarios.\n\nClarity:\n- The paper is overall well-written and easy to follow.\n\nSignificance:\n- LAPA provides a way of utilizing large amounts of videos without action labels with huge embodiment distribution shifts, which is a significant contribution to scalable robot learning.", "weaknesses": "- Lack of Experiments on Sequence Length in VQ Stage: There is a lack of experiments illustrating the effect of different sequence lengths during the VQ stage. It seems arbitrary that the latent code length is set to 4 (line 433-434), and for the language table dataset (line 933), the sequence length is set to 1. A discussion on the rationale behind these choices is missing. Incorporating experiments on various sequence lengths could help assess LAPA’s flexibility and robustness.\n\n- Limited Ability to Capture Complex Movements: Learning frame differences may only capture simple movement information. In visualizations of the learned latents, it appears that LAPA’s latent code primarily contains embodiment or camera movement information. Detailed results (Tables 13, 14, and 15) also indicate that LAPA performs better on coarse movement tasks like knocking and covering but struggles with finer actions like grasping or picking objects. This suggests that frame-difference learning might not be ideal for learning action latents. Including experiments with alternative approaches could provide a clearer evaluation. \n\n- Impact of Smaller Action Space: The smaller action space in LAPA compared to OpenVLA (256 bins for each action dimension in OpenVLA vs. a relatively smaller space in LAPA) may account for the observed performance improvement, rather than frame-difference learning alone. Reducing OpenVLA’s latent space for fairer comparison could better clarify the contributions of each part of LAPA.\n\n- Minor Wording Issue: The phrase “actionless video” on line 83 may be misleading, as it implies the absence of any action. Consider rephrasing for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Ba7e2me7y", "reviewer": "Reviewer_5VGj", "summary": "This paper proposes a novel pretraining method for VLAs. It first utilizes VQ-VAE to learn a encoder which encode the image into discrete token, and the decoder decode the next frame to learn the dyanmics of the adjacent frames. The discrete tokens could be treated as action tokens and replace the action labels in the human / robot rollouts.\nThe authors conduct  successive experiments to demonstrate the model's performance over in-domain scenes, cross-env scenes and cross embodiment scenairos to demonstrate the effectiveness of the proposed framework.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The innovative approach of using VQ-VAE to encode image dynamics into latent space and replacing labeled actions with these encoded tokens is particularly intriguing. This method holds significant importance for the research community, given the high costs associated with data collection for action labeling.\n\nThe experimental validation is comprehensive, with strong results obtained from both simulation environments and real-world settings, underscoring the reliability of the model. \n\nThe analysis is comprehensive, including the performance of in-domain, cross-environment and cross-embodiment setting.\n\nThe paper is well-structured and presented, making it easy for readers to follow the rationale and outcomes of the research.", "weaknesses": "The pretraining and finetuning setups in experiment section is a little confusing. For example, how is ActionVLA pretrained with action labels while there does not exist action labels in in something V2.\n\nThe utilized finetuning recipe of other baselines is not demonstrated in detail, which makes me concer the fairness of the comparison. I hope the authors could add detailed information in the appendix.\n\nAll the experiments in simulators are trained with only few trajectories, especially in Bridge V2, only 100 trajectories are utilized. I am wondering that if the model could learn multiple language conditioned tasks (like above 30 categories with diverse instruction) well. I suggests that tha authors add more experiments in a new simulator with more task catigories, and further train with RT-X data mixup and test the model performance in Simpler with Bridge and Google Robot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MTOMJ3DXji", "reviewer": "Reviewer_jNBu", "summary": "This paper proposes a method to pretrain vision-language-action (VLA) models for robotic manipulation using video data without robot action labels. Starting with actionless videos, they first train a VQVAE to infer latent actions between consecutive frames. They then train a VLM to predict these latent actions based on the current image and the language instruction. Finally, they finetune this model using robot trajectories with action labels for the target tasks. Their approach is evaluated in simulated and real-world multi-task experiment settings, including pretraining on actionless robot videos followed by finetuning on  robot trajectories of different tasks and pretraining on human videos from the Something-Something-v2 dataset.", "questions": "- Where do you think the performance gains by pretraining on videos come from (see above) ? \n- Have you tried pretraining on human video datasets other than SSv2, such as EpicKitchens, Ego4d, or even datasets less focused on manipulation ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Interesting approach: The proposed approach is both simple and practical, potentially easier to implement than the baselines considered in the experimental section. Pretraining VLAs on actionless data, especially human videos, is particularly relevant, and the use of inferred latent actions is a sensible solution.\n- Good experimental results: Through extensive comparative and ablation studies in both simulation and real-world robot settings, the authors clearly demonstrate the effectiveness of their proposed method. The experiments involving pretraining on human videos are impressive.\n- Clarity: The paper is well-written, clear, and easy to follow.", "weaknesses": "Since the latent actions are not directly used for downstream control and the model is finetuned on robot action labels, it’s unclear whether the performance gains come from leveraging temporal information/action priors in videos or simply from pretraining on data (robot trajectories/SSv2) that more closely aligns with the finetuning robot data compared to the base VLM’s original training data. Would a pretraining task without temporal information—such as image captioning— achieve similar results? For the same reason, it remains unclear if this approach can scale effectively to internet-scale video datasets, in particular beyond manipulation videos, as claimed by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sv7hr0LJld", "reviewer": "Reviewer_D2cm", "summary": "The paper presents LAPA, a method that aims to bridge the human-to-robot embodiment gap for VLA model training. \nLAPA consists of two stages. In the first stage, an encoder-decoder transformer learns a latent action representation from actionless videos with a VAE reconstruction loss. In the second state, given a current observation and task label, a VLM is trained to predict the learned latent actions from stage 1. The model is finetuned on a small in-domain dataset to transfer the model to actual robot actions.\n\nThe authors evaluate LAPA in two simulated environments, LanguageTable and Simpler, and a real-robot setup. They show their method outperforms similar action-less baselines while being competitive with VLAs trained on large-scale action-annotated datasets.", "questions": "See above. Also:\n\n- Why do you not evaluate OpenVLA on Simpler?\n- Why do you not evaluate pertaining on something-something in LangTable?\n- How does the framework perform with less finetuning data?\n- How is the fixed window size H determined? What granularity do the produced actions have?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The paper is well-motivated and tackles a very important problem in Robot Learning\n- The authors conduct several experiments in different environments while also including real-robot experiments\n- Combining learned latent actions with VLAs has not been explored before and is an interesting approach to bridging the human-robot embodiment gap.\n- The authors show that their pretraining strategy outperforms a recent state-of-the-art VLA trained explicitly with action labels.\n- The authors extend on [1] and show that latent action generation can be conditioned on language, enabling a more intuitive way to generate actions than codebook action generation.\n\n\n[1] Genie: Generative Interactive Environments, Bruce et al., 2022", "weaknesses": "The main weakness of the paper is its clarity.\n\n- It is not obvious throughout the paper that LAPA uses two distinct models.  The phrasing in the introduction, ‘LAPA has two pretraining stages…' gives the impression that you use one model for both latent action representation learning and action prediction. I would clarify early on that the method consists of two distinct models.\n- You refer to a learned world model in the introduction, but it is unclear where a world model is learned from the introduction.\n- The method section is oversimplified and hard to understand.\n    - In the text, you write that the Latent Action Quantization Model encoder produces a latent action $z_t$ for observation $x_t$, fed into the decoder. The referenced Figure 9 says that the decoder is conditioned on $x_t$ and $z_{t+H}$\n    - The VQ-VAE objective is not explained clearly. Although it is mainly based on prior work, it presents a large part of the method. Thus, I would recommend explaining the objective and the resulting latent action learned in more detail. For instance, you mention $z_t$ being a sequence $s$. How is the sequence length chosen? Does the model learn to predict $s$ latent actions at once? How is the vocabulary space $C$ chosen?\n    - You mention that you pretrain a VLM in Line 178. This indicates that it is trained from scratch, although you use an already pretrained model.\n    - In Line 194, you mention that the latent action head is a single mlp layer. This should be mentioned in the previous section.\n    - Using the abbreviation LAPA for the method and the model is confusing throughout the method section. Although you mention this at the beginning, this might be confusing for the reader. In Figure 2, LAPA is used to generate actions. In the text, you mention that LAPA trains both a world model and a policy. This isn't very clear.\n    - I think a more detailed Method (Architecture) figure than Figure 2 could help better understand the proposed method.\n- Although the extensive experiments and evaluations are good, the result section is too large. I would recommend cutting down here. Often, you just state the results from the table.\n- To improve the paper's clarity, I would reduce the results section and instead extend the method section.\n\nWhile the number of experiments is sufficient, I believe the focus could be improved. The experiment section emphasizes pretraining with large amounts of robot data where actions are available. However, in my opinion, the more interesting aspect is the pretraining step using real-world data. I would recommend shifting more attention to this area.\nI would also give more details on the real-world evaluation in the main part of the paper regarding the generalization of the method. For instance, do you use the VLA to perform the task on unseen objects? I know you show this in the appendix, but these results should be highlighted in the main part of the paper.\n\nRegarding the data scaling ablation, whether the experiment refers to finetuning or pretraining data is unclear. If it relates to pretraining data, an additional ablation regarding finetuning data would be interesting. After all, the method's goal is to enable transfer with a small amount of in-domain demonstrations. \n\n\nTypos and other formatting errors:\n\n- Line 25: Missing s in ‘model’\n- Line 71: Bridgev2 missing citation\n- Line 83: video missing s\n- Missing axis labels in Figure 6.\n- Appendix A Typo in Heading\n- Table 10: Wrong tasks. Unless you modified the simpler tasks, the tasks should be Carrot2Plate and Spoon2Towel.\n\n\nOverall, the paper's results are very promising, and the method applied to complex robot control is novel and interesting. Still, I think the clarity and presentation of the paper and results could be improved significantly, which would increase the contribution and relevance to the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fpV261R5Id", "reviewer": "Reviewer_2CRD", "summary": "This work aims to pretrain a Vision Language Action model on large scale internet video datasets.\n\nFour stage pipeline:\n1) Learn a codebook for latent actions from raw videos\n2) Pseudolabel a large dataset with these latent actions\n3) Pretrain a VLM on this pseudolabeled dataset to predict these latent actions\n4) Finetune the latent pretrained VLM on real labeled actions\n\nExperiments are performed in several simulated and real domains to demonstrate that LAPA can recover most of the performance of the same model architecture / finetuning setup trained on labeled versions of the pretraining dataset.", "questions": "1) What exactly are the error bars on all the plots? Standard deviation? Standard error? I didn't see it anywhere listed.\n2) In Figure 4 and 5, is there a reason ActionVLA (OpenX) is not included?\n\nNitpicks:\n\nSection 3 needs polish so that it's easier to read\n - Figure 2 is missing a label on step 3 for finetuning\n - e.g. on line 177/178, \"label all $x_t$, given $x_{t+1}$, with $z_t$\" ->  with \"label all frames $x_t$, given frame $x_{t+1}$, with latent action $z_t$\". If you're trying to skim, you have to go back to Section 3.1 to figure out the definitions of these math symbols.\n\nFigure 5 needs to have its colors made consistent between subfigure a and b; right now it looks like VPT, which is pink in figure a, is appearing in figure b", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Paper provides\n - A new recipe for training Vision Language Action models that can use data without action labels\n - A new foundation model that seems to produce good results when finetuned on real world tasks, including state of the art results on 2/3 real robot tasks involving gross motor skills\n\nPaper performs a number of experiments that provide hints at good performance when pretraining on unlabeled human data.", "weaknesses": "**TL;DR: There are data consistency issues throughout the paper, including in key result figures. With those fixes, method's proposed cross embodiment training recipe appears weak on actual cross embodiment data.**\n\n**Data consistency issues**\n\nTable 15 does not agree with the bar values in Figure 4 and 5 --- the superior performance of LAPA (Bridge) to ActionVLA (Bridge) is caused by swapping of values for ActionVLA and LAPA in the _Pick_ task, based on Table 15's results. Fixing this error makes LAPA (Bridge) worse than ActionVLA (Bridge), which is what one should reasonably expect given LAPA is unsupervised and ActionVLA is supervised and both use the same architecture. Quite honestly I think this should have been caught in a simple sanity check by the authors, as these results don't make much sense otherwise.\n\nTable highlighting of results seem to also consistently have errors\n - In Table 4, Block2BlockRelative, VPT has its score of 48 underlined (to indicate second place) when LAPA scored 52.0\n - In Table 7, row \"Separate\", ActionVLA has its score of 82.0 underlined (to indicate second place) when VPT scored 84.0\n\nThese are just the most obvious issues, I cannot be certain there are not deeper issues with the results. _Given the glaring nature of the errors in headline result plots, it makes me not trust the quantitative results of the authors._\n\nAs a personal note, I want to underscore that this is not the sole responsibility of the junior authors who likely prepared the manuscript and made the plots --- at least some of these errors are ones that could (and should) have been caught by senior authors reading the paper and skimming key results. \n\n**Most studies are robot to robot, providing limited signal on the value of actual cross embodiment training protocol**\n\nThese studies are robot to robot tasks, which don't address the question of usefulness of pretraining on domains where we aren't able to get action labels --- it's unreasonable to assume we don't have access to the actions associated with the observations for robot pretraining data. These ablations can be informative as \"Supplemental Experiments\", but we need to see the action codebook, pretrained on datasets where we do _not_ reasonably have access to actions, is providing real value. \n\nExperiments that fall into this category\n - The Language Table experiments are robot -> robot, and these are _very_ similar tasks\n - BridgeV2 -> SIMPLER is a larger gap, it's still robot -> robot\n\n**Studies on nonrobot to robot transfer provide weak proof of value**\n\nAssuming the label swapping issue only impacted LAPA (Bridge)'s Pick and Place performance and nothing else (I did not check the correctness other entries in the other tables), ActionVLA trained on the 54k trajectories of Bridgev2 outperforms LAPA pretrained on the 220K human videos. Notably, 54k trajectories is quite small; OpenX has over 1 million, but ActionVLA pretrained on OpenX is an important missing as a baseline from the results --- we need to tease out architecrture differences compared OpenVLA, as LAPA uses LWM-Chat-1M (taken directly from Liu et al 2024) which appears to be superior to OpenVLA's architecture --- but all (corrected) evidence seems to indicate that ActionVLA (OpenX) will not do worse than LAPA (OpenX).\n\nClearly pretraining on human data is not expected to match training on labeled robot data on a sample vs sample basis, but to make the argument that this pretraining is providing value, we need to see scaling laws showing that increased human data translates to increased performance; unfortunately, the authors made the puzzling decision to run their scaling laws experiments on Bridge (an already small dataset), instead of the salient dataset to this question, so we have no way of knowing how the method improves with data scale. \n\nGiven how the unsupervised action encoder works and the human hand latent analysis in Appendix E / Figure 15, my hypothesis is that if the task distribution of the human dataset (Table 3) remains the same, the gross motor base latent actions are still going to be discovered pretty early on and the marginal value of each additional sequence will quickly approach zero --- this of course needs to be verified or falsified, but there's no evidence in the paper to do that.\n\n**Actionable feedback**\n\n1) Please go review all of the numbers in detail from top to bottom. I happened to catch some of these issues, I have no idea how many more are hidden.\n2) I think the ideas presented here are interesting, but the story being told needs to be significantly refined. Pretraining on the human data needs to be front and center, and there needs to be a focus on performance as we scale up this dataset\n3) ActionVLM pretrained on OpenX is a useful artifact on its own. I don't know if someone's done this already or not, but such a release itself would be useful to the community.\n\n**UPDATE NOV 24th**\n\nIn light of the authors' comments on the correctness of Figures 3 and 4 I have raised my score from 1 to a 3. The remaining issues, related to the fundamental story of the paper, are detailed in my responses to the authors' comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "weTV3Wq3dO", "reviewer": "Reviewer_7bYx", "summary": "This paper proposes an unsupervised method for learning action knowledge from internet-scale video data. Without the need for action labels, the method implicitly models the action information contained in videos, making it possible to acquire generalizable knowledge from a vast amount of videos. The foundation model obtained from pre-training can be transferred to specific downstream robot manipulation tasks through supervised fine-tuning. The article validates the model's superiority over state-of-the-art VLA models by pre-training and fine-tuning with data from different domains.", "questions": "See above. I'm willing to discuss with other reviewers and authors to decide my final rating.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed unsupervised training method is highly significant. By employing VQ-VAE to learn action tokens, it is possible to model the action knowledge contained in videos without requiring action labels, which makes the pre-training with a larger scale of videos feasible for future applications.\n\nThe proposed method for reconstructing future frames during pre-training is reasonable and concise. The visualization results in the paper also confirm that the method can indeed learn structured action latent representations from unsupervised video data.\n\nThe experiments conducted in this paper are quite meticulous, including pre-training within the same domain, cross-task pre-training, as well as pre-training on large-scale real-world data like BridgeV2 and Open-X.", "weaknesses": "The method used in this paper is relatively simple. Pre-training through the prediction of future frames has been mentioned in some previous works. And the paper also points out that training directly with the VQ-VAE objective is quite similar to Genie. The ensuing question is, when the pre-trained video data is particularly abundant and spans a large number of domains, whether such a training method is sufficient to capture the patterns of action embeddings. Additionally, the selection of window size H for future frames is fixed during the training process, does such a choice bring difficulties to the modeling of actions? Could you explain the reason for choosing a fixed window size, or an ablation study on different window sizes?\n\nThe experimental validation part of the paper could be more robust. The current simulation environments use Language Table and SIMPLER, while real-world experiments only involve three tasks. How does OpenVLA perform in the simulation experiments used in this paper? And compared with OpenVLA, it does not seem to bring absolute advantages in all real-world tasks, such as \"Knock\" and \"Cover\" training with Bridge data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)