Latent Action Pretraining from Videos

## Abstract

Abstract We introduce Latent Action Pretraining for general Action models (LAPA), an unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels. Existing Vision-Language-Action models require action labels typically collected by human teleoperators during pretraining, which significantly limits possible data sources and scale. In this work, we propose a method to learn from internet-scale videos that do not have robot action labels. We first train an action quantization model leveraging VQ-VAE-based objective to learn discrete latent actions between image frames, then pretrain a latent VLA model to predict these latent actions from observations and task descriptions, and finally finetune the VLA on small-scale robot manipulation data to map from latent to robot actions. Experimental results demonstrate that our method significantly outperforms existing techniques that train robot manipulation policies from large-scale videos. Furthermore, it outperforms the state-of-the-art VLA model trained with robotic action labels on real-world manipulation tasks that require language conditioning, generalization to unseen objects, and semantic generalization to unseen instructions. Training only on human manipulation videos also shows positive transfer, opening up the potential for leveraging web-scale data for robotics foundation model. We open-source the model checkpoints and code at latentactionpretraining.github.io .

## 1 Introduction

Vision-Language-Action Models (VLA) for robotics (Brohan et al., ; Kim et al., ) are trained by aligning large language models with vision encoders, and then finetuning it on on diverse robot datasets (Collaboration et al., ); this enables generalization to novel instructions, unseen objects, and distribution shifts (Michał et al., ). However, diverse real-world robot datasets mostly require human teleoperation, which makes scaling difficult. Internet video data, on the other hand, offers abundant examples of human behavior and physical interactions at scale, presenting a promising approach to overcome the limitations of small, specialized robotic datasets (Yang et al., ). However, it is challenging to learn from internet video data for two major challenges: first, much of the raw data on the web lacks explicit action labels; second, the data distribution from the web is fundamentally different from the embodiments and environments of typical robotic systems (McCarthy et al., ). We propose Latent Action Pretraining for General Action Models (LAPA), an unsupervised approach to pretraining a robotic foundation model without the need for ground-truth robot action labels (Figure 1).

LAPA has two pretraining stages, followed by a fine-tuning stage to map the latent actions to real robot actions. In the first pretraining stage, we use a VQ-VAE-based objective (Van Den Oord et al., ) to learn quantized latent actions between raw image frames. Analogous to Byte Pair Encoding (Sennrich et al., ) used for language modeling, this can be seen as learning to tokenize atomic actions without requiring predefined action priors (e.g., end-effector positions, joint positions). In the second stage, we perform behavior cloning by pretraining a Vision-Language Model to predict latent actions derived from the first stage based on video observations and task descriptions.
Finally, we fine-tune the model on a small-scale robot manipulation dataset with robot actions to learn the mapping from the latent actions to robot actions. In this work, we refer to both the proposed method and the resulting VLA models as LAPA.

Figure: Figure 1: Problem Formulation. We investigate building a generalist robotic foundation model from human motion videos without action labels.
Refer to caption: x1.png

We measure performance on diverse manipulation videos, including existing robot video datasets (without utilizing ground-truth actions) and human manipulation datasets. Our results show that the proposed method significantly outperforms baseline methods of training manipulation policies from actionless videos, particularly in cross-environment and cross-embodiment scenarios. Furthermore, on real-world manipulation tasks, our method leads to a new monolithic VLA model, outperforming OpenVLA (Kim et al., ), the current state-of-the-art model Vision Language Action (VLA) model trained on a diverse mixture of datasets with ground-truth actions. These results demonstrate the effectiveness of learning unified quantized latent action representations across diverse robotic datasets featuring different embodiments (shown in Section 5.2). We further demonstrate that LAPA remains effective even when pretrained on only human manipulation video, outperforming models pretrained on Bridgev2 (Walke et al., ), one of the largest open-sourced robotic datasets. We observe that LAPA effectively captures environment-centric actions, including object and camera movements, which could be beneficial for downstream tasks like navigation or dynamic, non-quasistatic tasks. We expect that our method opens up the potential for building foundation models for robotics by pretraining on much larger web-scale video data.

We summarize our main contributions and findings below:

- •
We propose Latent Action Pretraining for general Action models (LAPA), an unsupervised approach to pretraining a robotic foundation model to encode robotic skills from web-scale video data.
- •
Experiments on simulation and real-world robot tasks show that our method not only significantly outperforms baseline methods for training robotic manipulation policies from actionless video, but also leads to a VLA model that outperforms the current state-of-the-art VLA model trained with ground-truth actions (by +6.22%), while achieving over 30x greater pretraining efficiency.
- •
With LAPA, we qualitatively demonstrate that it is possible to use the learned world and action prediction models to simulate full trajectories in diverse environments, effectively building a neural simulation capable of performing closed-loop evaluations entirely through neural inference.

## 2 Related Work

### 2.1 Vision-Language-Action Models

Vision-Language Models (VLMs), which are trained on extensive internet-scale datasets encompassing text, image, and video, have demonstrated the ability to understand and generate both text and multimodal data (Liu et al., ; Team, ; Liu et al., ; Abdin et al., ). Building on the generalization power of VLMs, recent work has introduced Vision-Language-Action Models (VLAs), which extend VLMs by fine-tuning them on robotic action data to improve physical grounding (Brohan et al., ; Kim et al., ; Team et al., ; Collaboration et al., ). Recently, incorporating auxiliary objectives, such as visual traces (Niu et al., ), language reasoning paths (Michał et al., ), or constructing a conversational-style instruction dataset using robot trajectory data (Li et al., ) during VLA training has been shown to improve performance. However, these methods still heavily rely on labeled action data, which limits the scalability of developing general VLAs, since they will be bounded by the amount of robotic data made available through human teleoperation. In contrast, our approach only requires labeled actions for finetuning, significantly reducing the burden of requiring a vast amount of human teleoperated robot data.

#### Training Robot Policies From Videos

Videos contain rich information about dynamics and behavior which can be potentially beneficial for robot learning. However, most raw videos do not contain any action labels (McCarthy et al., ). One line of work attempts to leverage egocentric human videos (Grauman et al., ) to improve visual representations for visuomotor robotic tasks (Nair et al., ; Dasari et al., ). Some recent works explore the benefits of video generative models pretrained on human videos for downstream robotic tasks (Wu et al., ; Liang et al., ). Another line of work aims to learn useful information from human videos by learning from interactions (Zeng et al., ), affordances (Bahl et al., ; Kannan et al., ; Srirama et al., ; Shaw et al., ), or visual traces (Wen et al., ; Bharadhwaj et al., ) extracted from human videos. However, all these works learn useful visual priors whereas our approach allows learning the mapping directly from perception to control during pretraining.

Another line of work aims to learn robot manipulation policies by retargeting human motions to robot motions. These works rely on off-the-shelf models such as hand pose estimators (Wang et al., ; Zhu et al., ; Shaw et al., ; Bharadhwaj et al., ; Ye et al., ; Qin et al., ) or motion capture systems (Yang et al., ) to retarget the human motions directly to robot motions. However, these works either learn only task-specific policies or require large in-domain perfectly aligned human-robot data and do not leverage the diverse, web-scale human motion data to train a single generalist policy.

The last line of work assumes one has access to a small-scaled robotic dataset with action labels to train an inverse dynamic model (IDM), optical flow, or reinforcement learning models that predicts actions from future state rollouts generated by world models (Du et al., ; Ko et al., ; Yang et al., ; Bharadhwaj et al., ) trained on diverse web-scale human videos. Another related work (Baker et al., ) trains an IDM with a small amount of action-labeled data and uses the IDM to label large-scaled actionless web data and pretrain a visuomotor robot policy. We consider these methods as direct baselines to our proposed methods since they are scalable methods with the same experimental assumptions and learn both visual and control priors to train generalist robot policies.

#### Latent Actions

Previous works have employed latent actions across diverse scenarios. GENIE (Bruce et al., ) maps user inputs (ground-truth actions) to a latent space, allowing generative models to create interactive environments. We adopt a similar latent action model but apply it to label actionless data for training a monolithic VLA to solve robotic tasks. Similarly, Edwards et al. () and Schmidt & Jiang () use latent actions to pretrain and fine-tune policies for video games (Cobbe et al., ). In contrast, we focus on learning latent actions from real-world human motions for more complex, continuous robotic tasks. Unlike other work that leverages latent actions by converting ground-truth actions into latent to capture better multimodality and task semantics Lynch et al. (); Jiang et al. (); Lee et al. (); Mete et al. (), our approach derives latent actions directly from observations, not ground-truth actions.

## 3 LAPA: Latent Action Pretraining for general Action models

Figure: Figure 2: Overview of LAPA. (1) Latent Action Quantization: We first learn discrete latent actions in a fully unsupervised manner using the VQ-VAE objective. (2) Latent Pretraining: The VLM is trained to predict latent actions, essentially performing behavior cloning. After pretraining, we finetune the LAPA model on a small set of action-labeled trajectories to map the latent space to the end effector delta action space.
Refer to caption: x2.png

LAPA is divided into two stages: Latent Action Quantization and Latent Pretraining.
First, we use a VQ-VAE based objective to capture the discretized latent delta information between consecutive frames in a video. Next, a pretrained VLM is trained to predict the latent action designated by the encoder of the Latent Action Quantization model, given the current image and the language instruction. After Latent Pretraining, we finetune the VLA model on a small number of ground-truth action-labeled trajectories to map the latent space to the actual action space.
The overall process is illustrated in Figure 2.

### 3.1 Latent Action Quantization

To learn latent actions in a fully unsupervised manner, we train a latent action quantization model following Bruce et al. () with a few modifications. Our latent action quantization model is an encoder-decoder architecture where the encoder takes the current frame $x_{t}$ and the future frame $x_{t+H}$ of a video with a fixed window size $H$ and outputs the latent action $z_{t}$(^1^11Although Bruce et al. () conditioned on multiple past observations, we exclude previous frames due to computational constraints. We leave prepending past observations as future work.). The decoder is trained to take the latent action $z_{t}$ and $x_{t}$ and reconstruct $x_{t+H}$. Unlike Bruce et al. (), we use cross attention to attend $z_{t}$ given $x_{t}$ instead of additive embedding, which empirically leads to capturing more semantically meaningful latent actions. Our quantization model is a variant of C-ViViT tokenizer (Villegas et al., ) where the encoder includes both spatial and temporal transformer while the decoder only contains spatial transformer since our model uses only two image frames as input. Further model details are provided in Appendix A.

Our latent action quantization training model is based on the VQ-VAE objective (van den Oord et al., ). The VQ-VAE objective enables the latent action $z_{t}$ to be discrete tokens (codebooks), making it easy for VLMs to predict $z_{t}$. The latent action is represented using $s$ sequences from $|C|$ codebook vocabulary space. To avoid gradient collapse often observed in VQ-VAE, we utilize NSVQ (Vali & Bäckström, ) which replaces the vector quantization error to a product of original error and a normalized noise vector. We also apply codebook replacement technique from NSVQ during early training steps to maximize codebook utilization.

We utilize the encoder of our latent action quantization model as an inverse dynamics model in the next stage of LAPA and the decoder for generating neural-based closed-loop rollouts. Unlike previous works (Bruce et al., ; Valevski et al., ), LAPA trains both a world model that generates rollouts from the latent actions and a policy model that produces these latent actions through Latent Pretraining.

### 3.2 Latent Pretraining

We use the encoder of the latent action quantization model as an inverse dynamics model to label all $x_{t}$, given $x_{t+1}$, with $z_{t}$. Then, we pretrain a VLM to predict the $z_{t}$ given the language instruction of a video clip and the current image $x_{t}$. Instead of using the existing language model head of the VLM, we attach a separate latent action head of vocab size $|C|$. By default, we freeze only the vision encoder and unfreeze the language model during training. Since latent pretraining does not rely on ground truth actions, it opens the possibility of using any type of raw video paired with language instructions. Also, in contrast to traditional action granularity used in robotics (e.g. end-effector positions, joint positions, joint torques, etc.), our approach does not require any priors about the action hierarchy/granularity and is learned in an end-to-end manner simply by being optimized to best capture the ‘delta’ of consecutive observations in a given video dataset. We broadly refer to models having gone through latent pretraining as LAPA.

### 3.3 Action Finetuning

VLAs that are pretrained to predict latent actions are not directly executable on real-world robots since latent actions are not actual delta end-effector actions or joint actions. To map latent actions to actual robot actions, we finetune LAPA on a small set of labeled trajectories that contain ground truth actions (delta end-effector). For action prediction, we discretize the continuous action space for each dimension of the robot so that the number of data points allocated for each bin is equal following Kim et al. (); Brohan et al. (). We discard the latent action head (a single MLP layer) and replace it with a new action head to generate ground truth actions.(^2^22We also tried leaving the latent action head and adding additional head to decode the latent to ground-truth actions following Schmidt & Jiang () However, we empirically found that re-initializing the action head resulted in superior downstream task performance, likely due to the size of the underlying policy model (7B).). As with latent pretraining, we freeze the vision encoder and unfreeze all of the parameters of the underlying language model.(^3^33We leave parameter efficient fine-tuning approaches as future work for finetuning (Hu et al., ).)

## 4 Experiments

In this section, we demonstrate the effectiveness of LAPA as a general-purpose pretaining method. Specifically, we focus on answering the following questions:

- •
Q1. How does LAPA perform when there are cross-task, cross-environment, and cross-embodiment gaps between pretaining and fine-tuning?
- •
Q2. Can LAPA learn superior priors compared to using ground-truth actions during pretraining in a multi-embodiment setting?
- •
Q3. Can we create a performant LAPA solely from raw human manipulation videos?

### 4.1 Benchmarks and Environments

**Table 1: Pretraining and fine-tuning dataset for each environment. Cross-Env denotes cross-environment, Cross-Emb denotes cross-embodiment, and Multi-Emb denotes multi-embodiment. For fine-tuning, MT denotes multi-task training and MI denotes tasks with diverse multi-instructions. Category denotes the main capability we are trying to quantify. Illustration of each environment is shown in Figure 3.**
| Environment | Category | Pretraining | Fine-tuning |  |  |
| --- | --- | --- | --- | --- | --- |
| Dataset | # Trajs | Dataset | # Trajs |  |  |
| LangTable | In-Domain | Sim (All 5 tasks) | 181k | 5 Tasks (MT, MI) | 1k |
| Cross-Task | Sim (All 5 tasks) | 181k | 1 Task (MI) | 7k |  |
| Cross-Env | Real (All 5 tasks) | 442k | 5 tasks (MT, MI) | 1k |  |
| SIMPLER | In-Domain | Bridgev2 | 60k | 4 Tasks (MT) | 100 |
| Cross-Emb | Something v2 | 220k | 4 Tasks (MT) | 100 |  |
| Real-World | Cross-Emb | Bridgev2 | 60k | 3 tasks (MI) | 450 |
| Multi-Emb | Open-X | 970k | 3 tasks (MI) | 450 |  |
| Cross-Emb | Open-X | 970k | 1 task (MI, Bi-manual) | 150 |  |
| Cross-Emb | Something v2 | 220k | 3 tasks (MI) | 450 |  |

Figure: (a) Language Table
Refer to caption: x3.png

We evaluate the effectiveness of LAPA on 9 different task categories in 2 different simulation environments and 4 different real-world robotic tasks. Table 1 shows an overview of the pretraining and fine-tuning dataset for each setup and Figure 3 visualizes the simulation benchmark and real-world setups. More details of each evaluation setup are provided in Appendix B.

Language Table (Lynch et al., ) is a simulation where a robot performs 2 DOF actions to push blocks (see Figure 3) (a)). It includes 5 subtask categories: BlocktoBlock, BlocktoAbsolute, BlocktoBlockRelative, BlocktoRelative, and Separate. Each trajectory from both sim and real is classified to one of 87k behaviors specified via natural language instruction (e.g. ‘slide the blue cube down a bit’).
During evaluation, we evaluate models for both seen and unseen scenarios, where unseen includes new objects (color and shape) and unseen combinations of seen objects.

SIMPLER (Li et al., ) is a set of simulated environments for evaluating generalist robot manipulation policies. We assess our models on 4 tasks (Figure 3 (b)) using the 7 DOF WidowX robot arm. Since SIMPLER lacks fine-tuning trajectories, we collect 100 multi-task trajectories using successful rollouts from a VLA model trained on BridgeV2 data (Walke et al., ), with held-out trajectories differing in object orientation and position from the evaluation setup.

Real-World Tabletop Manipulation experiments used a 7 DOF Franka Emika Panda robot arm in three environments and 14 DOF bi-manual robot in a single environment (shown in Figure 3 (c)).
We utilize three pretraining data sources: Bridgev2 (Walke et al., ), Open-X (Collaboration et al., ), and Something Something v2 (Goyal et al., ). Following Kim et al. (), for single arm experiments, we finetune on three multi-instruction tasks: (1) ‘Pick $<$object$>$ into Sink’, (2) ‘Cover $<$object$>$ with Towel’, and (3) ‘Knock $<$object$>$ Over’. For bi-manual robot experiments, the task is to ‘Put the $<$object1$>$ on the container and the $<$object2$>$ on the plate’. For all tasks, each task involves 150 trajectories across 15 objects. We use a task-specific partial success criterion for evaluation, following Kim et al. ().

### 4.2 Baselines

For the choice of our underlying VLM , we utilize the 7B Large World Model (LWM-Chat-1M) (Liu et al., ), which also leads to much efficiency gains during pretraining (further analysis provided in Section 4.7).
We also compare with existing behavior cloning baselines that do not require ground truth actions during pretraining.

Scratch denotes the baseline model where we finetune our backbone VLM only on the downstream tasks, to quantify the gains we get from the pretraining stage.

UniPi (Du et al., ) uses a video diffusion model during pretraining to generate video rollouts given a language instruction, which does not require any action labels during pretraining similar to our approach. For finetuning, an inverse dynamics model (IDM) is trained to extract the ground truth actions given adjacent frames.(^4^44We do not compare with Yang et al. () since the model is not open-sourced and Ko et al. () since it is not a behavior cloning baseline.) We also finetune the diffusion model on the downstream task to match the target distribution.

Vpt (Baker et al., ) trains an IDM on action labeled data, and uses the IDM model to extract pseudo actions on raw videos. Then, we use the pseudo actions labeled by the IDM to pretrain our backbone VLM on the pretraining data, identical to Latent Pretraining of LAPA.

ActionVLA denotes the baseline that uses the actual ground-truth robot action labels during pretraining with the same backbone VLM. This may be seen as the upper bound, since it utilizes the actual ground-truth labels. However, we find that LAPA is more performant in scenarios where there are embodiment shifts between pretraining and fine-tuning (Section 4.5).

OpenVLA (Kim et al., ) is a state-of-the-art VLA model that was pretrained on 970k real-world robot demonstrations from the Open X-Embodiment Dataset (Collaboration et al., ), mostly collected through human teleoperation. This model has a comparable number of parameters to LAPA (7B). We compare against OpenVLA for real-world robot experiments by fine-tuning the pretrained OpenVLA on our downstream tasks.

Further details of baseline models are provided in Appendix C.

### 4.3 Language Table Results

**Table 2: Language Table Results. Average Success Rate (%) across the three different pretrain-finetune combinations from the Language Table benchmark as described in Table 1. We also note the # of trajectories used for fine-tuning next to each category. We report the performance for individual tasks in Appendix E.1.**
|  | In-domain (1k) | Cross-task (7k) | Cross-env (1k) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Seen | Unseen | Seen | Unseen | Seen | Unseen |
| Scratch | 15.6±9.2 | 15.2±8.3 | 27.2±13.6 | 22.4±11.0 | 15.6±9.2 | 15.2±8.3 |
| UniPi | 22.0±12.5 | 13.2±7.7 | 20.8±12.0 | 16.0±9.1 | 13.6±8.6 | 12.0±7.5 |
| Vpt | 44.0±7.5 | 32.8±4.6 | 72.0±6.8 | 60.8±6.6 | 18.0±7.7 | 18.4±9.7 |
| LAPA | 62.0±8.7 | 49.6±9.5 | 73.2±6.8 | 54.8±9.1 | 33.6±12.7 | 29.6±12.0 |
| ActionVLA | 77.0±3.5 | 58.8±6.6 | 77.0±3.5 | 58.8±6.6 | 64.8±5.2 | 54.0±7.0 |

#### In-Domain Performance

First, we assess LAPA’s ability to learn from a small subset of in-domain action label data by pretraining on 181k trajectories and finetuning on 1k action-labeled trajectories (0.5%). As shown in Table 2, LAPA largely outperforms Scratch and narrows the gap with ActionVLA despite not using action labels during pretraining. Additionally, LAPA surpasses UniPi and Vpt. Notably, while UniPi handles simple tasks well, its diffusion model often generates incorrect plans for longer-horizon tasks, aligning with Du et al. () (see Figure 17 of Appendix E.1). Vpt, with the same backbone VLM as LAPA, outperforms UniPi, showing the superiority of the VLA model, but still underperforms LAPA, highlighting the effectiveness of latent actions.

#### Cross-Task Performance

We investigate whether LAPA’s broad skills can be retained after finetuning on a specific task. Pretraining LAPA on 181k trajectories and finetuning on only separate tasks (7k), we evaluate all 5 task categories, similar to the in-domain setup, to assess latent pretraining’s benefits for unseen tasks. When comparing LAPA and Scratch in Table 2 and Table 6, 7 in Appendix E.1, latent pretraining significantly benefits the separate task as well the other 4 task categories, resulting in a significant boost in both seen and unseen setups. Like before, UniPi is constrained by its diffusion model’s planning limitations, while Vpt performs strongly, even surpassing ActionVLA in the unseen setting. This is likely due to using more labeled data (7k vs. 1k), helping the IDM generate more accurate pseudo labels.

#### Cross-Environment Performance

We further investigate if LAPA benefits downstream performance when the pretraining and fine-tuning environments are different. We pretrain LAPA on 440k real-world trajectories, and then finetune on 1k simulation trajectories, which can be seen as testing on a setup where a real2sim gap is present.
(^5^55We only test on the real2sim scenario since simulation evaluation is easily replicable and easier to setup. We leave exploration in the sim2real scenario as future work.) Figure 3 (a) compares the two environments. From Table 2, we observe that LAPA still significantly outperforms Scratch, showing that latent pretraining leads to positive transfer even on cross-environment setting. Notably, both UniPi and Vpt significantly underperforms LAPA, showing that learning to predict latent actions is more robust to cross-environment transfer. Vpt only results in minor positive transfer, indicating that the IDM is not robust to environment shifts.

### 4.4 SIMPLER Results

We pretrain our models on the Bridgev2 (Walke et al., ) dataset and fine-tune on 100 trajectories collected from the SIMPLER environment (Li et al., ) (25 trajectories for each task).

Figure: Figure 4: SIMPLER Results. Avg. success rate (%) is shown across 4 tasks. Detailed results are in Appendix E.2.
Refer to caption: x6.png

#### Results

As shown in Figure 4, UniPi significantly underperforms all other baselines on the SIMPLER Environment. We observe that, although the generated plans from the diffusion models are quite accurate, the IDM lacks the capability to predict 7 DOF continuous actions accurately when given only 100 action-labeled trajectories. Specifically, we observe that UniPi often fails to grasp the object within the maximum step limit. This implies the effectivness of using VLAs in scenarios with insufficient action-labeled data. Similar to the results of Section 4.3, LAPA outperforms baseline models that pretrain on actionless videos (UniPi and Vpt) and closes the performance gap with ActionVLA, which is pretrained on all of the 60K action-labeled trajectories from the Bridgev2 dataset. This highlights the effectiveness of LAPA, even when the complexity of the action space increases. We provide the detailed results of individual tasks and subtasks (grasping and reaching) in Table 10 of Appendix E.2.

### 4.5 Real-world Results

Figure: Figure 5: Real-world Tabletop Manipulation Results. We evaluate on a total of 54 rollouts for each model encompassing unseen object combinations, unseen objects and unseen instructions. Average success rate (%) are shown. We provide detailed results depedning on the generalization type in Table 12 and individual results in Appendix E.3.
Refer to caption: x7.png

We pretrain our models on (1) Bridgev2 (Walke et al., ) to measure the cross-embodiment performance (WidowX embodiment for pretraining and Franka embodiment for finetuning) and (2) Open X-Embodiment Dataset  (Collaboration et al., ) to measure the effect of pretraining in a multi-embodiment setting.
Figure 5 shows the average success rate across the 3 tasks where each task encompasses unseen object combination, unseen object, and unseen instruction settings. We provide detailed results depending on the generalization type in Table 12 in Appendix E.3. We also measure the cross-embodiment performance of Open X-Embodiment Dataset pretraining by fine-tuning on bi-manual robots.

#### Bridgev2 Pretraining

We compare models that were pretrained on the Bridgev2 dataset. Similar to previous results, all models pretrained on Bridgev2 result in significant performance enhancement compared to Scratch. Furthermore, by comparing LAPA which does not leverage action-labeled trajectories during pretraining with models that use action-labeled trajectories during pretraining (ActionVLA and OpenVLA (Bridge)), we observe an interesting finding: LAPA outperform VLAs that use action labeled pretraining data on average success rate of the 3 tasks, unlike previous scenarios where VLAs pretrained on the ground-truth actions were upper bounds. LAPA significantly outperforms the other models in pick-and-place tasks; given that most tasks in Bridgev2 are pick-and-place, we hypothesize that VLA models pretrained on ground truth action labels have overfitted to the WidowX action space from the Bridgev2 dataset, hampering cross-embodiment adaptability to action distribution shifts during fine-tuning. In contrast, LAPA avoids this issue by not relying on ground truth action labels during pretraining.

#### Open-X Pretraining

From Figure 5, we see that VLAs pretrained on the Open-X dataset outperforms VLAs pretrained on the Bridgev2 dataset, showing that data scaling during pretraining demonstrates positive transfer for downstream tasks (Collaboration et al., ). This also suggests there could be significant further improvement when scaling the diversity and scale of the pretraining data, especially with large web-scale video data.

When comparing LAPA with OpenVLA, we see that LAPA significantly outperforms OpenVLA on 2 out of 3 tasks (Figure 5). This highlights LAPA’s effectiveness in a multi-embodiment setting by showcasing its ability to leverage a shared latent action space during pretraining, akin to how language and image representations are utilized. In contrast, contemporary action pretraining methods may suffer from reduced positive transfer between datasets due to the variability in action representation spaces across different embodiments and datasets.

However, for pick and place task, LAPA underperforms OpenVLA. We observe that most failures of LAPA are due to early grasping. In fact, LAPA outperforms OpenVLA in reaching performance (83.33% vs 66.67%) (reaching performance for each task is provided separately in Appendix E.3). This suggests that, although LAPA possesses stronger language conditioning and coarse-grained planning abilities, there is room for improvement in skills such as grasping. Since grasping occurs only once or twice in each trajectory, the 150 labeled trajectories may not be sufficient for LAPA to accurately predict grasp actions based on the physical characteristics of diverse objects.

To evaluate the cross-embodiment performance of LAPA pretrained with Open-X, we fine-tune both LAPA and OpenVLA on a bimanual robot with a 14-DoF action space, which presents a more challenging cross-embodiment scenario than the previous Bridgev2 pretraining experiments due to a cross-embodiment gap as well as the increase of the action space dimensions (there are no bimanual robot datasets in Open-X). Similar to the single-arm experiments, our evaluation covers seen object combinations, unseen object combinations, unseen objects, and unseen instructions. We observe that LAPA (30.21%), which does not use labeled action data during pretraining, slightly outperforms OpenVLA (26.04%) on average, confirming LAPA’s cross-embodiment capability even with the increase of the complexity of the action dimensions. However, the absolute performance of both models remains relatively low, indicating much room for improvement. We believe that incorporating bimanual human manipulation videos (Grauman et al., ; Damen et al., ) could enhance LAPA’s performance, which we plan to explore in future work. Individual results are provided in Table 17 in Appendix E.3.

### 4.6 Learning from Human Manipulation Videos

In this section, we show results when we extend LAPA to human manipulation videos, which aligns with the main motivation of this work. Unlike robot trajectories, human videos have two challenges: human videos do not contain action labels, and the distribution of human videos is distinct from the robot embodiment (McCarthy et al., ). We try to investigate whether our method as well as baseline approaches could address these challenges by pretraining on Something-Something V2 dataset (Goyal et al., ) which consists of 220K videos that includes human performing actions with everyday objects.

Figure: (a) SIMPLER Results
Refer to caption: x8.png

We first evaluate the performance of LAPA pretrained on human videos on SIMPLER. In addition to Scratch, we also compare with UniPi and Vpt pretrained with the same human video dataset. As shown in Figure 6(a), LAPA outperforms Scratch, showing that although the distribution of the pretraining data is distinct from the deployment setup, leveraging human videos for latent action pretraining results in positive transfer. Also, consistent with the result of Section 4.4, LAPA shows the best performance, implying that Latent Action Pretraining is robust to human to robot embodiment shifts. Note that it is impossible to train ActionVLA because the human videos do not have any robot action labels.

We report the real-world robot experiments in Figure 6(b). Surprisingly, we can see that LAPA trained with human videos outperforms OpenVLA (Bridge) on average. Despite the larger embodiment gap for LAPA (Human to robot vs. Robot to robot), it learns a better prior for robot manipulation. This result highlights the potential of raw human manipulation videos from the web compared to expensive robot manipulation data, which requires time-intensive teleoperation to collect. We expect that applying our approach on large-scale internet videos (e.g., YouTube videos) could unlock the potential for large-scale pretraining of a generalist action foundational model, similar to foundational models in Natural Language Processing or Computer Vision.

### 4.7 Pretraining Efficiency

The benefit of LAPA extends beyond downstream task performance to include pretraining efficiency. For pretraining LAPA (Open-X), the best-performing model, we use 8 H100 GPUs for 34 hours with a batch size of 128 (total of 272 H100-hours). In contrast, OpenVLA required a total of 21,500 A100-hours with a batch size of 2048. Despite being approximately 30-40 times more efficient for pretraining, LAPA still outperforms OpenVLA (^6^66We calculate based on the fact that H100 GPUs lead to 2-3 times speedup compared to A100 GPUs for training.).

We believe this efficiency stems from two factors: (1) the use of the Large World Model (Liu et al., ) as the backbone VLM model, and (2) the coarse-grained actions of LAPA compared to conventional action pretraining. First, the training objective during LWM pretraining includes generating the next state, which corresponds to the next frame in a video. We hypothesize that this objective enables the model to implicitly understand high-level actions in a video. Notably, ActionVLA (Bridge), which uses LWM as the backbone, and OpenVLA (Bridge), which uses Prismatic as the backbone, are trained on the same data and objective. However, ActionVLA reaches optimal performance (in terms of action token accuracy) in significantly fewer epochs (3 epochs) compared to OpenVLA’s 30 epochs. Second, the action space for LAPA is much smaller than that for OpenVLA ($8^{4}$ vs. $256^{7}$), making learning the perception-and-language to action generation problem easier to learn. For all LAPA models (BridgeV2, Open-X, Human Videos), we observe that a single epoch of training is sufficient to achieve optimal performance.

## 5 Ablation and Analysis

### 5.1 Scaling Model, Data, and Latent Action Size

Figure: (a) Model Scaling
Refer to caption: x10.png

Large Language Models (LLMs) have demonstrated scaling laws (Kaplan et al., ), where performance improves with increases in model size, dataset size, and computational resources used for training. Similarly, we attempt to analyze whether LAPA benefits from scaling across four dimensions: latent action quantization model size, data size, latent action sequence and vocabulary size. For a controlled setup, we apply our method to Bridgev2 and then fine-tune it on SIMPLER.

As shown in Figure 7, scaling benefits LAPA across the four dimensions. However, for the latent action space (sequence and vocabulary size), we believe there is a sweet spot where performance improves up to a certain point, after which it either plateaus or diminishes. Interestingly, we observe that the optimal scale of the latent action space depends on the complexity of the action dimension contained in the pretraining dataset. For example, increasing the latent action sequence length is less effective compared to increasing the vocabulary for Language Table (Figure 16). Except for Language Table, we maintain the generation space of LAPA at $8^{4}$ throughout all of our main experiments.

These results imply that when scaling pretraining to Internet-scale videos that go beyond manipulation videos, scaling LAPA in terms of model, dataset, and latent action space could improve performance, especially to capture higher action dimensions such as whole-body locomotion and manipulation.

### 5.2 Latent Action Analysis

We qualitatively analyze the alignment of quantized latent actions with real continuous actions. For interpretation, we condition the current image observation $x_{1}$ and each latent action on the decoder of the latent action quantization model, and present the reconstructed images.

Figure: Figure 8: Latent Action Analysis in Language Table. We condition the current observation $x_{1}$ and quantized latent action to the decoder of the latent action quantization model. We observe that each latent action can be mapped into a semantic action. For example, latent action 0 corresponds to moving a bit left and forward.
Refer to caption: x14.png

Figure: Figure 9: Correlation of latent actions with ground-truth actions.
Refer to caption: extracted/5929182/figures/langtable_analysis_map.png

First, we analyze latent actions learned from Language Table with vocabulary size of 8 and sequence length of 1. In Figure 8, we show that each latent action corresponds to a distinct movement of the robot arm, with the distribution of latent actions being well-clustered in the actual 2D action space. We observe that increasing the latent action vocabulary size leads to capturing a more fine-grained information. We analyze the relationship between latent actions with ground-truth 2 DOF actions by mapping each instance into latent action space. As shown in Figure 9, we observe that latent actions are well-clustered in the actual 2D action space, indicating that latent actions are meaningful representations that are highly related to actual continuous actions.

Figure: Figure 10: Latent Action Analysis in Human Manipulation Videos. We condition the current observation $x_{1}$ and quantized latent action to the decoder of the latent action quantization model. We observe that each latent action can be mapped into a semantic action including camera movements. For example, latent action [3,5,2,7] corresponds to moving the camera a bit down while [4,2,0,0] corresponds to moving the camera slightly up.
Refer to caption: x15.png

We further analyze the latent actions learned from human manipulation videos using the Something-Something V2 dataset. As illustrated in Figure 10, these latent actions capture not only hand movements but also camera movements. Since the camera viewpoint varies throughout the videos in the Something-Something V2 dataset due to the videos being egocentric, our latent action quantization model also learns to represent camera movements.

Figure: Figure 11: Latent Action Analysis in Multi-Embodiment Setting. We condition the current observation $x_{1}$ and quantized latent action to the decoder of the latent action quantization model. We observe that each latent action can be mapped into a similar semantic action even though the embodiments are different. For example, latent action [1,1,3,2] corresponds to going down and left while [3,2,0,1] corresponds to going up a little bit.
Refer to caption: x16.png

We also analyze the latent actions learned from the Open-X embodiment, which encompasses multiple embodiments, tasks, and environments. As shown in Figure 11, even though the embodiment and environment differ, conditioning on the same latent action results in a similar action in the reconstructed image. This supports our previous claim that latent actions are learned in a shared representation space, regardless of the embodiment or dataset, facilitating stronger positive transfer across diverse datasets.

Figure: Figure 12: Closed loop rollout of LAPA. LAPA is conditioned on current image $x_{1}$ and language instruction of ‘take the broccoli out of the pot’. We generate rollout images by conditioning the decoder of Latent Action Quantization Model with latent actions generated by LAPA.
Refer to caption: x17.png

We also qualitatively analyze the coarse-grained planning capability of LAPA through a closed-loop rollout. We use a LAPA model that has only undergone pretraining, without any action finetuning. Since this model generates latent actions that are not directly executable in the real world, we condition the current observation $x_{1}$ and the predicted latent action from LAPA with the decoder of the latent action quantization model. As shown in Figure 12, when conditioned on the current observation and the instruction to ‘take the broccoli out of the pot’, LAPA generates robot trajectories that successfully reaches for the broccoli, moves down to grab it, and, as the arm moves away from the pot, the broccoli disappears. This shows the potential for LAPA as a general-purpose robotic world model, not only predicting actions but also the outcomes of the actions. For example, this can lead to an extension of LAPA to act as a Task and Motion planning system, where it can first generate multiple plans given a natural language task instruction, choose the most optimal trajectory based on methods of quantifying the success among multiple trajectory candidates (Hwang et al., ; Duan et al., ), and perform open-loop / closed-loop inference. This can lead a paradigm where we aim to improve end-task performance through scaling test-time compute, as with LLMs (Snell et al., ).

## 6 Limitations and Conclusion

In this paper, we introduce LAPA, a scalable pretraining method for building VLAs using actionless videos. Across three benchmarks spanning both simulation and real-world robot experiments, we show that our method significantly improves transfer to downstream tasks compared to existing approaches. We also present a state-of-the-art VLA model that surpasses current models trained on 970K action-labeled trajectories. Furthermore, we demonstrate that LAPA can be applied purely on human manipulation videos, where explicit action information is absent, and the embodiment gap is substantial. We believe our work can be extended to build scalable robot foundation models.

We still face certain limitations. First, LAPA underperforms compared to action pretraining when it comes to fine-grained motion generation tasks like grasping. We believe that increasing the latent action generation space could help address this issue. Second, similar to prior VLAs, LAPA also encounters latency challenges during real-time inference. Adopting a hierarchical architecture, where a smaller head predicts actions at a higher frequency, could potentially reduce latency and improve fine-grained motion generation. Lastly, while we qualitatively demonstrate that our latent action space captures camera movements (Figure 10), we have not yet explored the application of LAPA beyond manipulation videos, such as those from self-driving cars, navigation, or landscape scenes. We leave these explorations for future work. We hope that our work can help overcome the data bottleneck in robotics and accelerate the development of generalist robot policies.

## Acknowledgements

We thank Arhan Jain and Marius Memmel for helping out with the robot hardware and teleoperation setup. Also, we thank Minyoung Hwang, Jiafei Duan, Junsu Kim, and Changyeon Kim for helpful discussions and constructive feedback.

## Appendix A Latent Action Quatization Model Details

Figure: Figure 13: Model architecture of our Latent Action Quantization Model.
Refer to caption: x18.png

We show model architecture details of our latent action quantization model in Figure 13. We utilize the C-ViViT model architecture from Villegas et al. () to replicate the latent action model from GENIE (Bruce et al., ). After latent model training, we utilize the $z_{2}$ as the latent action label for $x_{1}$. The encoder can be seen as the inverse dynamics model and the decoder can be seen as the world model.

## Appendix B Details on Experimental Setup

#### Language Table Experimental Setup

Figure 3 (a) shows examples of the Language Table setup. For Language Table experiments, we train VLA-based models to generate language directions (e.g. ‘move up’) before actual actions following Belkhale et al. (), which significantly improved the performance (^7^77For 7 DOF robot experiments, we found the benefit of generating language directions to be marginal compared to the increased inference cost. Therefore, we only generate delta end-effector actions on other experiments.). For evaluation, we evaluate on 50 evaluation rollouts for each subtask category where the initial locations of the objects are randomized for each evaluation. Further details can be found in [https://github.com/google-research/language-table](https://github.com/google-research/language-table).

#### SIMPLER Experimental Setup

Figure 3 (B) shows examples of the SIMPLER setup. The SIMPLER environment does not provide any fine-tuning data for their evaluation pipeline, Thus, we first train our underlying VLM on the Bridgev2 dataset and perform zero-shot rollout on the 4 tasks in SIMPLER. Note that we use held-out trajectories differing in object orientation and position from the evaluation setup. We filter 25 successful trajectories for each task (total of 100) and use them as the fine-tuning dataset for all of our experiments. For evaluation, we evaluate on 24 rollouts per task while randomizing the initial object locations. We consider Bridgev2 and SIMPLER to be in-domain since they show a high correlation between real-world and simulation results with their simulation benchmark. Further details can be found in [https://github.com/simpler-env/SimplerEnv](https://github.com/simpler-env/SimplerEnv).

Figure: Figure 14: Real-world Tabletop Manipulation Examples.
Refer to caption: x19.png

#### Real-world Tabletop Manipulation Experimental Setup

Figure 3 (C) shows examples of the real-world tabletop manipulation experimental setup.
For the teleoperation of single arm robot experiments, we use the polymetis robotic stack(^8^88[https://github.com/facebookresearch/polymetis](https://github.com/facebookresearch/polymetis)) to collect 150 trajectories for each of the tasks. All of the tasks require multi-instruction following capabilities since there are 3 objects in the scene and the model has to condition on the task description to infer which object to interact with. Figure 14 shows samples of each task. For each task, we aim to quantify 3 distinct capabilities:

(1) We test the ability to infer the correct object from the task description between an unseen combination of seen objects during fine-tuning, (2) We test the ability to infer the correct object from totally unseen objects during fine-tuning that may or may have not been observed during pretraining. Specifically, the knocking tasks was conducted with real-world objects that were highly unlikely to have been in any of the pertaining datasets. (3) We test the ability to infer the correct object (among seen objects, unseen combinations) from a totally unseen instruction that requires semantic reasoning (e.g. Pick up a spicy object). For each evaluation criteria, 6 rollouts are performed for each models, resulting in a total of 18 rollouts for each task category. Since there are three tasks, each model is evaluated with 54 rollouts in the real-world. We provide the full list of all of the seen and unseen objects used for each rollout in Table 13, 14, 15, and the total average success rates in Table 16.

Furthermore, for a fair comparison, we match the image resolution during training of all of our models and use the exact same object initial positions for all of our evaluation, mostly on the same day to minimize variability. For evaluation metrics, we adapt a partial success criteria for fine-grained evaluation, following Kim et al. (), which we describe in detail below.

Knock down the $<$object$>$.

For knocking, we give 0.5 partial score if the robot reaches to the correct object and 1 if the robot knocks down the correct object.

Cover the $<$object$>$ with a towel.

For covering, we give 0.33 partial score if the robot picks up the towel correctly, 0.66 if the robot reaches to the correct object or if the towel partially covers the object, and 1 if the correct object is completely covered by the towel.

Pick up the $<$object$>$ and put it in the sink.

For pick and place, we give 0.25 for reaching to the correct object, 0.5 for grasping the object, 0.75 for grasping and moving the object towards the sink, but failing to place the object in the sink, and 1 for placing the correct object in the sink.

Similar to single arm experiments, we collect 150 trajectories for bi-manual experiments. For the bi-manual task, one of the arms should choose one between two plates depending on the language instruction and put the chosen plate on the container. Next, the other arm should pick up a correct object from three object candidates and put the object on the plate. For evaluation, we quantify (1) seen combination of seen objects during fine-tuning, (2) unseen combinations of seen objects, (3) unseen objects, (4) unseen instructions of seen objects. 6 rollouts are performed for each sub-category, resulting in 24 rollouts for each model. We provide the full result in Table 17. For evaluation metrics, we adapt a partial success criteria for fine-grained evaluation.

Put the $<$object1$>$ on the container and $<$object2$>$ on the plate.

For bi-manual task, we give 0.25 partial score if the robot reaches to the correct plate, 0.5 if the robot puts the correct plate on the container successfully, 0.75 if the robot reaches to the correct object on the right, and 1 if the robot successfully pick the object and place the object on top of the plate.

## Appendix C Baseline Details

For UniPi, we use diffusion model from (Ko et al., ) which can be trained on 4 A100 GPUs. For all experiments, we train with 128 batch. We use the same inverse dynamics model as Vpt during inference. To mediate estimation errors between the predicted video plans and executed actions being accumulated, we periodically conduct replanning by regenerating new video plans after executing two actions. For Vpt, we use ResNet18 followed by an MLP layer for the inverse dynamics model(IDM). The IDM is trained to predict an action when given two frames on a single A6000 GPU using using Adam optimizer with a
learning rate 1e-4. For OpenVLA (Bridge), we pretrain on Bridgev2 for 30 epochs with a batch size of 1024. For OpenVLA (Open-X), we use the pretrained checkpoint from Kim et al. (). For finetuning, we use LoRA finetuning (Hu et al., ) with batch size of 32. We have observed that full-finetuning and lora finetuning leads to similar performance, so we use LoRA finetuning as default for efficient fine-tuning. We finetune the model until the training action accuracy reaches 95%. For ActionVLA and LAPA, we train with a batch size of 128 and with image augmentation for real-world finetuning.

## Appendix D Experimental Result Analysis

**Table 3: Pretraining trajectories statistics for downstream tasks. Number of trajectories that are the same task with evaluation task for each pretraining dataset: Bridgev2, Open-X, and Something Something V2 (Sthv2) dataset.**
| Task | Bridgev2 | Open-X | Sthv2 |
| --- | --- | --- | --- |
| Knocking | 2 | 7,969 | 6,655 |
| Covering | 898 | 5,026 | 6,824 |
| Pick & Place | 10,892 | 911,166 | 3,272 |

We further analyze the real-world robot results shown in Figures 5 and 6(b), focusing on how the task distribution in pretraining data impacts downstream performance.
Table 3 presents the number of trajectories corresponding to each evaluation task (Knocking, Covering, and Pick & Place) across pretraining datasets (Bridgev2, Open-X, and Something Something V2 (Sthv2)), determined through lexical matching. We expect future work to use other methods of analyzing the relationship between pertaining and fine-tuning task distributions that capture semantics of the task rather than simple lexical matching. We perform this analysis to get a sense of how the task distribution in the pretraining data affects downstream task performance.

#### Knocking

There are almost no knocking-related trajectories in Bridgev2. This scarcity may explain why models trained on Bridgev2 performed worse compared to those trained on Sthv2, despite a larger embodiment gap in the Sthv2 dataset (Figure 6(b)).

#### Covering

A similar trend is observed for the covering task. Given that the number of covering trajectories in Bridgev2 is relatively small compared to the Sthv2 dataset, models trained on Bridgev2 occasionally underperform compared to LAPA trained on Sthv2.

#### Pick & Place

For the pick and place task, the trend reverses. The number of pick and place tasks in Sthv2 is relatively small compared to Bridgev2 and Open-X, which might explain why LAPA trained on Sthv2 significantly underperforms models trained on Bridgev2 or Open-X. Based on these results, we expect that pretraining on videos encompassing a wide range of skills will lead to a more robust generalist policy compared to training on robot videos with narrower skill sets. We also expect future research to provide a more in-depth analysis of the relationship between task distribution in pretraining data and performance on downstream tasks.

Figure: (a) Win rate (%) disregarding ties.
Refer to caption: extracted/5929182/figures/pair_test2.png

We also present the win rate of LAPA (Open-X) against OpenVLA (Open-X). As illustrated in Figure 15, LAPA outperforms OpenVLA in 65.4% when disregarding the ties. When considering the ties, LAPA outperforms OpenVLA in 31.5% of cases, while OpenVLA prevails in only 16.7%. Interestingly, they tie in 51.9% of the trials, suggesting that in about half the instances, both models either fail or achieve a similar partial success score. Note that these evaluations were performed while ensuring that the target and distractor objects were in identical initial locations during evaluation, alternating the models during evaluation. These results provide insight into the statistical significance of the comparison, supporting the use of multiple metrics to ensure a more comprehensive evaluation of physical robot performance in real-world scenarios (Kress-Gazit et al., ), not only the average success-rate across all of the evaluation rollouts.

## Appendix E Detailed Experimental Results

### E.1 Language Table

We provide the detailed results of the experiments performed on the Language Table benchmark in Table 4, 5, 6, 7, 8, 9. For all of the tables in the appendix, we bold the best result among the comparisons and underline the second best. Each value denotes the success rate (%). 50 evaluation rollouts are performed for each task category, resulting in 250 total evaluation rollouts per model for each table.

We also show the qualitative result of UniPi where the diffusion model generates the correct plan for simple and short-horizon tasks (e.g. separate tasks). However, the diffusion model generates the wrong plan corresponding to the instruction when the task requires longer horizon planning (Figure 17).

For ablation of LAPA in Language Table, we experiment with various latent action vocab and sequence sizes. As shown in Figure 16, increasing the sequence and vocab size increases the performance. However, unlike SIMPLER, we observe that the increasing the latent action vocab size is much more effective compared to increasing the latent action sequence length in terms of absolute performance. This implies that for environments that are visually simple, increasing the latent action vocabulary might be more effective compared to sequence length.

Figure: (a) Latent Action Seq
Refer to caption: x20.png

Figure: Figure 17: Success and Failure Cases of UniPi. (Top) Given the instruction of ‘move the green block away from the red cube and red pentagon’, the diffusion model of UniPi successfully generates the plan. (Bottom) Given the instruction of ‘put the blue moon toward the yellow block’, the diffusion model fails to generate the correct plan.
Refer to caption: x22.png

**Table 4: Language Table In-Domain Seen Results.**
|  | Scratch | UniPi | VPT | LAPA | ActionVLA |
| --- | --- | --- | --- | --- | --- |
| Block2Block | 4.0 | 14.0 | 36.0 | 58.0 | 76.0 |
| Block2Absolute | 6.0 | 4.0 | 38.0 | 56.0 | 72.0 |
| Block2BlockRelative | 10.0 | 12.0 | 48.0 | 52.0 | 76.0 |
| Block2Relative | 6.0 | 10.0 | 26.0 | 48.0 | 70.0 |
| Separate | 52.0 | 72.0 | 70.0 | 96.0 | 90.0 |
| AVG | 15.6 | 22.4 | 43.6 | 62.0 | 76.8 |

**Table 5: Language Table In-Domain Unseen Results.**
|  | Scratch | UniPi | VPT | LAPA | ActionVLA |
| --- | --- | --- | --- | --- | --- |
| Block2Block | 8.0 | 4.0 | 26.0 | 50.0 | 62.0 |
| Block2Absolute | 10.0 | 6.0 | 42.0 | 48.0 | 58.0 |
| Block2BlockRelative | 2.0 | 6.0 | 20.0 | 28.0 | 48.0 |
| Block2Relative | 8.0 | 6.0 | 32.0 | 38.0 | 44.0 |
| Separate | 48.0 | 44.0 | 44.0 | 84.0 | 82.0 |
| AVG | 15.2 | 13.2 | 32.8 | 49.6 | 58.8 |

**Table 6: Language Table Cross-Task Seen Results.**
|  | Scratch | UniPi | VPT | LAPA | ActionVLA |
| --- | --- | --- | --- | --- | --- |
| Block2Block | 18.0 | 12.0 | 74.0 | 74.0 | 76.0 |
| Block2Absolute | 8.0 | 6.0 | 56.0 | 62.0 | 72.0 |
| Block2BlockRelative | 6.0 | 2.0 | 62.0 | 72.0 | 76.0 |
| Block2Relative | 24.0 | 16.0 | 72.0 | 60.0 | 70.0 |
| Separate | 80.0 | 68.0 | 96.0 | 98.0 | 90.0 |
| AVG | 27.2 | 20.8 | 72.0 | 73.2 | 76.8 |

**Table 7: Language Table Cross-Task Unseen Results.**
|  | Scratch | UniPi | VPT | LAPA | ActionVLA |
| --- | --- | --- | --- | --- | --- |
| Block2Block | 16.0 | 4.0 | 66.0 | 46.0 | 62.0 |
| Block2Absolute | 10.0 | 10.0 | 56.0 | 52.0 | 58.0 |
| Block2BlockRelative | 8.0 | 10.0 | 46.0 | 48.0 | 48.0 |
| Block2Relative | 12.0 | 4.0 | 52.0 | 38.0 | 44.0 |
| Separate | 66.0 | 52.0 | 84.0 | 90.0 | 82.0 |
| AVG | 22.4 | 16.0 | 60.8 | 54.8 | 58.8 |

**Table 8: Language Table Cross-Environment Seen Results.**
|  | Scratch | UniPi | VPT | LAPA | ActionVLA |
| --- | --- | --- | --- | --- | --- |
| Block2Block | 4.0 | 4.0 | 16.0 | 26.0 | 66.0 |
| Block2Absolute | 6.0 | 4.0 | 8.0 | 16.0 | 58.0 |
| Block2BlockRelative | 10.0 | 8.0 | 6.0 | 20.0 | 62.0 |
| Block2Relative | 6.0 | 4.0 | 12.0 | 22.0 | 54.0 |
| Separate | 52.0 | 48.0 | 48.0 | 84.0 | 84.0 |
| AVG | 15.6 | 13.6 | 18.0 | 33.6 | 64.8 |

**Table 9: Language Table Cross-Environment Unseen Results.**
|  | Scratch | UniPi | VPT | LAPA | ActionVLA |
| --- | --- | --- | --- | --- | --- |
| Block2Block | 8.0 | 2.0 | 2.0 | 30.0 | 38.0 |
| Block2Absolute | 10.0 | 6.0 | 4.0 | 14.0 | 48.0 |
| Block2BlockRelative | 2.0 | 6.0 | 2.0 | 10.0 | 50.0 |
| Block2Relative | 8.0 | 4.0 | 40.0 | 18.0 | 54.0 |
| Separate | 48.0 | 42.0 | 44.0 | 76.0 | 80.0 |
| AVG | 15.2 | 12.0 | 18.4 | 29.6 | 54.0 |

### E.2 SIMPLER

We provide detailed results of various models evaluated on SIMPLER environment. Table 10 shows the setting where baseline models are pretrained on Bridgev2 and then finetuned on SIMPLER rollouts (100 videos). The results show detailed results for each task (stack green to yellow block, put carrot on plate, put spoon on otowel, put eggplant in basket) and subtasks (grasping and moving).

We also provide detailed results of the setting where baseline models are pretrained on human manipulation videos (Something Something V2 dataset) and then finetuned on SIMPLER rollouts (100 videos) in Table 11. We only compare to UniPi, Vpt, and LAPA since ActionVLA could not be trained without ground-truth action labels.

**Table 10: SIMPLER results of Bridgev2 Pretraining. Success, Grasping, and Moving Rates (%) in SIMPLER environment. We pretrain UniPi, Vpt, and LAPA on Bridgev2 dataset without using ground-truth action labels and ActionVLA on Bridgev2 using action labels. The main 4 tasks are: stack green to yellow block, put carrot on plate, put spoon on towel, and put eggplant in basket. Best is bolded and second best is underlined.**
| Success Rate | Scratch | UniPi | Vpt | LAPA | ActionVLA |
| --- | --- | --- | --- | --- | --- |
| Stack G2Y | 29.2 | 2.7 | 45.8 | 54.2 | 75.0 |
| Carrot2Towel | 29.2 | 2.7 | 37.5 | 45.8 | 58.0 |
| Spoon2Plate | 50.0 | 0.0 | 70.8 | 70.8 | 70.8 |
| Eggplant2Bask | 29.2 | 0.0 | 50.0 | 58.3 | 50.0 |
| AVG | 34.4 | 1.3 | 51.0 | 57.3 | 63.5 |
| Grasping Rate |  |  |  |  |  |
| Grasp Green Block | 66.6 | 20.8 | 62.5 | 62.5 | 87.5 |
| Grasp Carrot | 45.8 | 33.2 | 54.1 | 58.3 | 75.0 |
| Grasp Spoon | 70.8 | 22.2 | 79.2 | 83.3 | 83.3 |
| Grasp Eggplant | 62.5 | 16.0 | 70.8 | 83.3 | 75.0 |
| AVG | 61.4 | 23.1 | 66.7 | 71.9 | 80.2 |
| Moving Rate |  |  |  |  |  |
| Move Green Block | 58.3 | 29.1 | 58.3 | 66.6 | 91.6 |
| Move Carrot | 45.8 | 48.6 | 66.6 | 70.8 | 91.6 |
| Move Spoon | 70.8 | 34.6 | 79.2 | 83.3 | 79.2 |
| Move Eggplant | 87.5 | 58.0 | 70.8 | 87.5 | 91.6 |
| AVG | 65.6 | 42.6 | 68.7 | 77.1 | 88.5 |

**Table 11: SIMPLER results of Human Manipulation Video Pretraining. Success, Grasping, and Moving Rates (%) in SIMPLER environment. We pretrain UniPi, Vpt, and LAPA on Something-Something V2 dataset without using ground-truth action labels. The main 4 tasks are: stack green to yellow block, put carrot on plate, put spoon on towel, and put eggplant in basket. Best is bolded and second best is underlined.**
| Success Rate | VPT | UniPi | LAPA |
| --- | --- | --- | --- |
| StackG2Y | 50.0 | 0.0 | 50.0 |
| Carrot2Towel | 29.1 | 1.3 | 50.0 |
| Spoon2Plate | 37.5 | 1.3 | 50.0 |
| Eggplant2Bask | 66.6 | 0.0 | 58.3 |
| AVG | 45.8 | 0.7 | 52.1 |
| Grasping Rate |  |  |  |
| Grasp Green Block | 66.6 | 2.7 | 58.3 |
| Grasp Carrot | 45.8 | 31.7 | 62.5 |
| Grasp Spoon | 70.8 | 21.7 | 75.0 |
| Grasp Eggplant | 91.6 | 6.8 | 70.8 |
| AVG | 68.7 | 15.7 | 66.7 |
| Moving Rate |  |  |  |
| Move Green Block | 62.5 | 2.7 | 62.5 |
| Move Carrot | 58.3 | 37.5 | 70.8 |
| Move Spoon | 54.1 | 18.1 | 75.0 |
| Move Eggplant | 91.6 | 50.3 | 83.3 |
| AVG | 66.6 | 27.1 | 72.9 |

### E.3 Real-world

We provide the detailed result of real world evaluation depending on the generalization type: (1) seen objects but unseen combinations, (2) unseen objects, and (3) seen objects but unseen instructions. The results are shown in Table 12. As shown in the table, LAPA (Open-X) outperforms OpenVLA (Open-X) on all types of generalization settings. Also, LAPA (Human Videos) shows good generalization performance, especially for unseen objects. We conjecture that this is because Something Something V2 dataset interacts with much diverse objects compared to Bridgev2.

We also provide the full list of objects and the partial success recorded for each of the evaluation rollout: Knocking (Table 13), Covering (Table 14), and Pick & Place (Table 15). The total average success rate is provided in Table 16).

**Table 12: Evaluation Results divided into eval types. We average the success rate across the 3 tasks depending on what capability we are trying to quantify: (1) seen objects but unseen combinations, (2) unseen objects, and new instructions requiring semantic reasoning. Best is bolded and second best is underlined.**
|  | Seen Objects | Unseen | Seen Objects | AVG |
| --- | --- | --- | --- | --- |
|  | Unseen Combinations | Objects | Unseen Instructions |  |
| Scratch | 18.0 | 20.3 | 25.4 | 21.2 |
| ActionVLA (Bridge) | 38.3 | 31.8 | 27.7 | 32.6 |
| OpenVLA (Bridge) | 35.6 | 34.6 | 22.1 | 30.8 |
| LAPA (Bridge) | 43.4 | 31.4 | 35.6 | 36.8 |
| OpenVLA (Open-X) | 46.2 | 42.1 | 43.4 | 43.9 |
| LAPA (Open-X) | 57.8 | 43.9 | 48.5 | 50.1 |
| LAPA (Human Videos) | 36.5 | 37.4 | 28.1 | 34.0 |

**Table 13: Knocking Task Results**
|  | OpenVLA | LAPA | OpenVLA | ActionVLA | LAPA | Scratch | LAPA |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | (OpenX) | (OpenX) | (Bridge) | (Bridge) | (Bridge) | (Sthv2) |  |
| Unseen Object Combinations |  |  |  |  |  |  |  |
| flamingo | 0 | 0.5 | 0.5 | 0.5 | 0 | 0 | 0.5 |
| pistachios | 0.5 | 1 | 0.5 | 0 | 1 | 0 | 1 |
| soft scrub | 0 | 0 | 0 | 0 | 0.5 | 0 | 0.5 |
| white cup | 1 | 0 | 0 | 0.5 | 0.5 | 0.5 | 0 |
| mustard | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| water bottle | 1 | 1 | 0.5 | 0 | 0 | 0.5 | 0 |
| SUM | 2.5 | 3.5 | 1.5 | 1 | 2 | 1 | 2 |
| Unseen Objects |  |  |  |  |  |  |  |
| pringles | 0.5 | 0.5 | 0.5 | 0 | 0 | 0 | 0 |
| hersey’s chocolate syrup | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| popcorn | 0 | 1 | 1 | 1 | 1 | 0 | 1 |
| skittles | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| green board marker | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 |
| paper towel | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| SUM | 1 | 2 | 2 | 1.5 | 1.5 | 0.5 | 1.5 |
| Unseen Instructions |  |  |  |  |  |  |  |
| a drink that contains orange | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| food to eat with milk | 0.5 | 0 | 0 | 0 | 0 | 0 | 0 |
| a object used for cleaning | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| something to wash dishes | 1 | 1 | 0 | 0.5 | 1 | 0.5 | 0 |
| the nuts | 1 | 1 | 0.5 | 1 | 1 | 0.5 | 1 |
| rectangle object | 1 | 1 | 0.5 | 0.5 | 0.5 | 0 | 1 |
| SUM | 3.5 | 4 | 1 | 2 | 2.5 | 1 | 2 |
| Success Rate (Strict) | 27.78% | 44.44% | 5.56% | 11.11% | 22.22% | 0.00% | 22.22% |
| Success Rate | 38.89% | 52.78% | 25.00% | 25.00% | 33.33% | 13.89% | 30.56% |
| Reaching Success Rate | 50.00% | 61.11% | 44.44% | 38.89% | 44.44% | 27.78% | 38.89% |

**Table 14: Covering Task Results**
|  | OpenVLA | LAPA | OpenVLA | ActionVLA | LAPA | Scratch | LAPA |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | (OpenX) | (OpenX) | (Bridge) | (Bridge) | (Bridge) | (Sthv2) |  |
| Unseen Object Combinations |  |  |  |  |  |  |  |
| icecream | 0.33 | 0.33 | 0.33 | 0.33 | 0.33 | 0.33 | 0 |
| strawberry | 0.33 | 1 | 0.33 | 1 | 0.33 | 1 | 1 |
| pepper | 0.33 | 0 | 0.33 | 0.33 | 0.33 | 0.33 | 0.33 |
| watermelon | 0.33 | 0.33 | 0.33 | 0.33 | 0.33 | 0 | 0.33 |
| blue lego block | 0.66 | 1 | 1 | 1 | 1 | 0.33 | 0.33 |
| pink duck | 0.33 | 1 | 0.33 | 0.33 | 0.33 | 0 | 0.33 |
| SUM | 2.31 | 3.66 | 2.65 | 3.32 | 2.65 | 1.99 | 2.32 |
| Unseen Objects |  |  |  |  |  |  |  |
| donut | 0.33 | 1 | 0.66 | 1 | 0.66 | 0.66 | 0.33 |
| orange | 0.33 | 0.33 | 1 | 0 | 0.33 | 1 | 1 |
| mushroom | 0.33 | 0.33 | 0.33 | 0.33 | 0.33 | 0.33 | 0.33 |
| yellow lego block | 0.33 | 1 | 1 | 0.33 | 0 | 0.33 | 0.33 |
| peas | 1 | 0 | 0.66 | 1 | 1 | 0.33 | 1 |
| egg | 0 | 1 | 0.33 | 0 | 0.66 | 0 | 1 |
| SUM | 2.32 | 3.66 | 3.98 | 2.66 | 2.98 | 2.65 | 3.99 |
| Unseen Instructions |  |  |  |  |  |  |  |
| drink | 0.33 | 0 | 0.66 | 1 | 0.33 | 0.33 | 0.66 |
| yellow object | 0.66 | 0.66 | 0 | 0 | 0.33 | 0 | 0.33 |
| fruit | 0.33 | 0.33 | 0.33 | 0.33 | 0.33 | 0.33 | 0.33 |
| vegetable | 0.33 | 0.33 | 0 | 0.33 | 0.33 | 0.33 | 0.33 |
| edible object | 0.33 | 0.33 | 0.66 | 0 | 0.33 | 1 | 0.33 |
| condiment | 0.33 | 0.33 | 0.33 | 0 | 0.33 | 0.33 | 0.33 |
| SUM | 2.31 | 1.98 | 1.98 | 1.66 | 1.98 | 2.32 | 2.31 |
| Success Rate (Strict) | 5.56% | 33.33% | 16.67% | 27.78% | 11.11% | 16.67% | 22.22% |
| Success Rate | 38.56% | 51.67% | 47.83% | 42.44% | 42.28% | 38.67% | 47.89% |
| Reaching Success Rate | 16.66% | 38.89% | 38.89% | 27.78% | 22.22% | 22.22% | 27.78% |

**Table 15: Pick & Place Sink Task Results**
|  | OpenVLA | LAPA | OpenVLA | ActionVLA | LAPA | Scratch | LAPA |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | (OpenX) | (OpenX) | (Bridge) | (Bridge) | (Bridge) | (Sthv2) |  |
| Unseen Object Combinations |  |  |  |  |  |  |  |
| milk | 1 | 1 | 1 | 1 | 1 | 0 | 1 |
| orange lego block | 1 | 1 | 0 | 1 | 0 | 0 | 0 |
| ketchup | 0.25 | 0.25 | 0.25 | 0.25 | 0 | 0 | 0 |
| corn | 1 | 0.75 | 1 | 0.25 | 0.25 | 0.25 | 0.25 |
| icecream | 0.25 | 0 | 0 | 0 | 1 | 0 | 1 |
| salt | 0 | 0.25 | 0 | 1 | 0 | 0 | 0 |
| SUM | 3.5 | 3.25 | 2.25 | 3.5 | 2.25 | 0.25 | 2.25 |
| Unseen Objects |  |  |  |  |  |  |  |
| carrot | 1 | 0.25 | 0 | 0.25 | 1 | 0.25 | 0.25 |
| yellow paprika | 1 | 1 | 0 | 0.25 | 0.25 | 0 | 1 |
| yellow cube | 1 | 0.5 | 0.25 | 0.5 | 0 | 0 | 0 |
| salmon sushi | 0 | 0.25 | 0 | 0.5 | 0 | 0 | 0 |
| orange | 1 | 0 | 0 | 0 | 0 | 0.25 | 0 |
| blue cube | 0.25 | 0.25 | 0 | 0 | 0 | 0 | 0 |
| SUM | 4.25 | 2.25 | 0.25 | 1.5 | 1.25 | 0.5 | 1.25 |
| Unseen Instructions |  |  |  |  |  |  |  |
| an object that is yellow | 1 | 1 | 0 | 1 | 0.25 | 0 | 0 |
| an object that is round | 0 | 0.25 | 0 | 0 | 0 | 0.25 | 0 |
| an object that is a fruit | 1 | 1 | 1 | 1 | 0 | 1 | 0.75 |
| an object that you can drink | 0 | 0.25 | 0 | 0.5 | 0 | 0 | 0 |
| an object that is a vegetable | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| an object that is an animal | 0 | 0.25 | 0 | 0.25 | 0.25 | 0 | 0 |
| SUM | 2 | 2.75 | 1 | 2.75 | 0.5 | 1.25 | 0.75 |
| Success Rate (Strict) | 50.00% | 27.78% | 16.67% | 27.78% | 16.67% | 5.56% | 16.67% |
| Success Rate | 54.17% | 45.83% | 19.44% | 43.06% | 22.22% | 11.11% | 23.61% |
| Reaching Success Rate | 66.67% | 83.33% | 27.78% | 72.22% | 38.89% | 27.78% | 33.33% |

**Table 16: Summary of Single Arm Total Success Rates (%)**
|  | OpenVLA | LAPA | OpenVLA | ActionVLA | LAPA | Scratch | LAPA |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | (OpenX) | (OpenX) | (Bridge) | (Bridge) | (Bridge) | (Sthv2) |  |
|  |  |  |  |  |  |  |  |
| Total Success Rate | 43.87% | 50.09% | 30.76% | 36.83% | 32.61% | 21.22% | 34.02% |
| Total Success Rate (Strict) | 27.78% | 35.19% | 12.96% | 22.22% | 16.67% | 7.41% | 20.37% |

**Table 17: Bi-manual Task Results (Seen and Unseen)**
|  | OpenVLA (Open-X) | LAPA (Open-X) |
| --- | --- | --- |
| Seen Object Combinations |  |  |
| purple, blue cup | 0.25 | 0.25 |
| gray, cupcake | 0.25 | 0.75 |
| gray, apple | 0 | 0 |
| purple, milk | 0.25 | 0.75 |
| gray, orange juice | 0.5 | 0.25 |
| purple, green cup | 0.25 | 0.25 |
| SUM | 1.5 | 2.25 |
| Unseen Object Combinations |  |  |
| purple, orange | 0.75 | 0 |
| purple, red block | 0.25 | 0 |
| purple, cupcake | 0.25 | 0 |
| gray, peach | 0.25 | 0.5 |
| gray, mustard | 0 | 0 |
| gray, spam | 0.25 | 0.5 |
| SUM | 1.75 | 1 |
| Unseen Objects |  |  |
| purple, pear | 0.25 | 0.5 |
| purple, mayonnaise | 0.75 | 0.25 |
| purple, yellow cup | 0.5 | 0.5 |
| plain white, tomato soup | 0.25 | 0.75 |
| plain white, wooden block | 0.25 | 0.25 |
| plain white, cheez it | 0 | 0 |
| SUM | 2 | 2.25 |
| Unseen Instructions |  |  |
| darker, fruit | 0.25 | 0.5 |
| darker, sweet dessert | 0.5 | 0 |
| darker, drink | 0.25 | 0.5 |
| lighter, rectangular object | 0 | 0.25 |
| lighter, roundest object | 0 | 0.5 |
| lighter, red object | 0 | 0 |
| SUM | 1 | 1.75 |
| Success Rate | 26.04% | 30.21% |

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: - Need clarification for pretraining stage: In table 1, authors provide details about the dataset for both pre-train and finetune phase, however, it remains unclear about the latent action quantization. My understanding is that latent action quantization is part of the pretraining so both latent quantization  (section 3.1) and latent pretraining (section 3.2) use the same dataset. Clarification of the dataset usage could provide a better understanding of the experiment set up. 

- In most of the experiments, LAPA is trained with a single large scale dataset for fair comparison with other models like OpenVLA. Since LAPA does not require any action label during pre-training, it would be interesting to see having all three datasets used together can provide benefit of downstream performance boost to further backup the scalable claim.
soundness: 4
strengths: Originality:
- The proposed method removes the need of action labels for pre-training VLA models which significantly increase the data availability.
- Training VQ to predict delta between frames is a simple and scalable way of learning coarse latent action. 
- A significant performance improvement compared to SoTA (OpenVLA) model under various scenarios and relatively small performance gap between the upper bound case (ActionVLA) and LAPA. 

Quality:
- The proposed method is technically sound.
- Extensive experiments are conducted to evaluate LAPA performance under various scenarios.

Clarity:
- The paper is overall well-written and easy to follow.

Significance:
- LAPA provides a way of utilizing large amounts of videos without action labels with huge embodiment distribution shifts, which is a significant contribution to scalable robot learning.
summary: This paper proposes an unsupervised pre-training method, LAPA, for vision-language-action (VLA) models that eliminates the need for action labels. LAPA uses a VQ-VAE structure to learn quantized action latents from frame differences. Subsequently, the authors train a VLA model to predict these quantized latents based on frames and language instructions. This model can then be fine-tuned on small-scale robot manipulation datasets. Through experiments on both simulated and real-world datasets, LAPA demonstrates improved performance over selected baseline models to indicate its generalization ability.
weaknesses: - Lack of Experiments on Sequence Length in VQ Stage: There is a lack of experiments illustrating the effect of different sequence lengths during the VQ stage. It seems arbitrary that the latent code length is set to 4 (line 433-434), and for the language table dataset (line 933), the sequence length is set to 1. A discussion on the rationale behind these choices is missing. Incorporating experiments on various sequence lengths could help assess LAPA’s flexibility and robustness.

- Limited Ability to Capture Complex Movements: Learning frame differences may only capture simple movement information. In visualizations of the learned latents, it appears that LAPA’s latent code primarily contains embodiment or camera movement information. Detailed results (Tables 13, 14, and 15) also indicate that LAPA performs better on coarse movement tasks like knocking and covering but struggles with finer actions like grasping or picking objects. This suggests that frame-difference learning might not be ideal for learning action latents. Including experiments with alternative approaches could provide a clearer evaluation. 

- Impact of Smaller Action Space: The smaller action space in LAPA compared to OpenVLA (256 bins for each action dimension in OpenVLA vs. a relatively smaller space in LAPA) may account for the observed performance improvement, rather than frame-difference learning alone. Reducing OpenVLA’s latent space for fairer comparison could better clarify the contributions of each part of LAPA.

- Minor Wording Issue: The phrase “actionless video” on line 83 may be misleading, as it implies the absence of any action. Consider rephrasing for clarity.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness.
soundness: 4
strengths: The innovative approach of using VQ-VAE to encode image dynamics into latent space and replacing labeled actions with these encoded tokens is particularly intriguing. This method holds significant importance for the research community, given the high costs associated with data collection for action labeling.

The experimental validation is comprehensive, with strong results obtained from both simulation environments and real-world settings, underscoring the reliability of the model. 

The analysis is comprehensive, including the performance of in-domain, cross-environment and cross-embodiment setting.

The paper is well-structured and presented, making it easy for readers to follow the rationale and outcomes of the research.
summary: This paper proposes a novel pretraining method for VLAs. It first utilizes VQ-VAE to learn a encoder which encode the image into discrete token, and the decoder decode the next frame to learn the dyanmics of the adjacent frames. The discrete tokens could be treated as action tokens and replace the action labels in the human / robot rollouts.
The authors conduct  successive experiments to demonstrate the model's performance over in-domain scenes, cross-env scenes and cross embodiment scenairos to demonstrate the effectiveness of the proposed framework.
weaknesses: The pretraining and finetuning setups in experiment section is a little confusing. For example, how is ActionVLA pretrained with action labels while there does not exist action labels in in something V2.

The utilized finetuning recipe of other baselines is not demonstrated in detail, which makes me concer the fairness of the comparison. I hope the authors could add detailed information in the appendix.

All the experiments in simulators are trained with only few trajectories, especially in Bridge V2, only 100 trajectories are utilized. I am wondering that if the model could learn multiple language conditioned tasks (like above 30 categories with diverse instruction) well. I suggests that tha authors add more experiments in a new simulator with more task catigories, and further train with RT-X data mixup and test the model performance in Simpler with Bridge and Google Robot.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - Where do you think the performance gains by pretraining on videos come from (see above) ? 
- Have you tried pretraining on human video datasets other than SSv2, such as EpicKitchens, Ego4d, or even datasets less focused on manipulation ?
soundness: 3
strengths: - Interesting approach: The proposed approach is both simple and practical, potentially easier to implement than the baselines considered in the experimental section. Pretraining VLAs on actionless data, especially human videos, is particularly relevant, and the use of inferred latent actions is a sensible solution.
- Good experimental results: Through extensive comparative and ablation studies in both simulation and real-world robot settings, the authors clearly demonstrate the effectiveness of their proposed method. The experiments involving pretraining on human videos are impressive.
- Clarity: The paper is well-written, clear, and easy to follow.
summary: This paper proposes a method to pretrain vision-language-action (VLA) models for robotic manipulation using video data without robot action labels. Starting with actionless videos, they first train a VQVAE to infer latent actions between consecutive frames. They then train a VLM to predict these latent actions based on the current image and the language instruction. Finally, they finetune this model using robot trajectories with action labels for the target tasks. Their approach is evaluated in simulated and real-world multi-task experiment settings, including pretraining on actionless robot videos followed by finetuning on  robot trajectories of different tasks and pretraining on human videos from the Something-Something-v2 dataset.
weaknesses: Since the latent actions are not directly used for downstream control and the model is finetuned on robot action labels, it’s unclear whether the performance gains come from leveraging temporal information/action priors in videos or simply from pretraining on data (robot trajectories/SSv2) that more closely aligns with the finetuning robot data compared to the base VLM’s original training data. Would a pretraining task without temporal information—such as image captioning— achieve similar results? For the same reason, it remains unclear if this approach can scale effectively to internet-scale video datasets, in particular beyond manipulation videos, as claimed by the authors.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: See above. Also:

- Why do you not evaluate OpenVLA on Simpler?
- Why do you not evaluate pertaining on something-something in LangTable?
- How does the framework perform with less finetuning data?
- How is the fixed window size H determined? What granularity do the produced actions have?
soundness: 4
strengths: - The paper is well-motivated and tackles a very important problem in Robot Learning
- The authors conduct several experiments in different environments while also including real-robot experiments
- Combining learned latent actions with VLAs has not been explored before and is an interesting approach to bridging the human-robot embodiment gap.
- The authors show that their pretraining strategy outperforms a recent state-of-the-art VLA trained explicitly with action labels.
- The authors extend on [1] and show that latent action generation can be conditioned on language, enabling a more intuitive way to generate actions than codebook action generation.


[1] Genie: Generative Interactive Environments, Bruce et al., 2022
summary: The paper presents LAPA, a method that aims to bridge the human-to-robot embodiment gap for VLA model training. 
LAPA consists of two stages. In the first stage, an encoder-decoder transformer learns a latent action representation from actionless videos with a VAE reconstruction loss. In the second state, given a current observation and task label, a VLM is trained to predict the learned latent actions from stage 1. The model is finetuned on a small in-domain dataset to transfer the model to actual robot actions.

The authors evaluate LAPA in two simulated environments, LanguageTable and Simpler, and a real-robot setup. They show their method outperforms similar action-less baselines while being competitive with VLAs trained on large-scale action-annotated datasets.
weaknesses: The main weakness of the paper is its clarity.

- It is not obvious throughout the paper that LAPA uses two distinct models.  The phrasing in the introduction, ‘LAPA has two pretraining stages…' gives the impression that you use one model for both latent action representation learning and action prediction. I would clarify early on that the method consists of two distinct models.
- You refer to a learned world model in the introduction, but it is unclear where a world model is learned from the introduction.
- The method section is oversimplified and hard to understand.
    - In the text, you write that the Latent Action Quantization Model encoder produces a latent action $z_t$ for observation $x_t$, fed into the decoder. The referenced Figure 9 says that the decoder is conditioned on $x_t$ and $z_{t+H}$
    - The VQ-VAE objective is not explained clearly. Although it is mainly based on prior work, it presents a large part of the method. Thus, I would recommend explaining the objective and the resulting latent action learned in more detail. For instance, you mention $z_t$ being a sequence $s$. How is the sequence length chosen? Does the model learn to predict $s$ latent actions at once? How is the vocabulary space $C$ chosen?
    - You mention that you pretrain a VLM in Line 178. This indicates that it is trained from scratch, although you use an already pretrained model.
    - In Line 194, you mention that the latent action head is a single mlp layer. This should be mentioned in the previous section.
    - Using the abbreviation LAPA for the method and the model is confusing throughout the method section. Although you mention this at the beginning, this might be confusing for the reader. In Figure 2, LAPA is used to generate actions. In the text, you mention that LAPA trains both a world model and a policy. This isn't very clear.
    - I think a more detailed Method (Architecture) figure than Figure 2 could help better understand the proposed method.
- Although the extensive experiments and evaluations are good, the result section is too large. I would recommend cutting down here. Often, you just state the results from the table.
- To improve the paper's clarity, I would reduce the results section and instead extend the method section.

While the number of experiments is sufficient, I believe the focus could be improved. The experiment section emphasizes pretraining with large amounts of robot data where actions are available. However, in my opinion, the more interesting aspect is the pretraining step using real-world data. I would recommend shifting more attention to this area.
I would also give more details on the real-world evaluation in the main part of the paper regarding the generalization of the method. For instance, do you use the VLA to perform the task on unseen objects? I know you show this in the appendix, but these results should be highlighted in the main part of the paper.

Regarding the data scaling ablation, whether the experiment refers to finetuning or pretraining data is unclear. If it relates to pretraining data, an additional ablation regarding finetuning data would be interesting. After all, the method's goal is to enable transfer with a small amount of in-domain demonstrations. 


Typos and other formatting errors:

- Line 25: Missing s in ‘model’
- Line 71: Bridgev2 missing citation
- Line 83: video missing s
- Missing axis labels in Figure 6.
- Appendix A Typo in Heading
- Table 10: Wrong tasks. Unless you modified the simpler tasks, the tasks should be Carrot2Plate and Spoon2Towel.


Overall, the paper's results are very promising, and the method applied to complex robot control is novel and interesting. Still, I think the clarity and presentation of the paper and results could be improved significantly, which would increase the contribution and relevance to the field.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 2
questions: 1) What exactly are the error bars on all the plots? Standard deviation? Standard error? I didn't see it anywhere listed.
2) In Figure 4 and 5, is there a reason ActionVLA (OpenX) is not included?

Nitpicks:

Section 3 needs polish so that it's easier to read
 - Figure 2 is missing a label on step 3 for finetuning
 - e.g. on line 177/178, "label all $x_t$, given $x_{t+1}$, with $z_t$" ->  with "label all frames $x_t$, given frame $x_{t+1}$, with latent action $z_t$". If you're trying to skim, you have to go back to Section 3.1 to figure out the definitions of these math symbols.

Figure 5 needs to have its colors made consistent between subfigure a and b; right now it looks like VPT, which is pink in figure a, is appearing in figure b
soundness: 2
strengths: Paper provides
 - A new recipe for training Vision Language Action models that can use data without action labels
 - A new foundation model that seems to produce good results when finetuned on real world tasks, including state of the art results on 2/3 real robot tasks involving gross motor skills

Paper performs a number of experiments that provide hints at good performance when pretraining on unlabeled human data.
summary: This work aims to pretrain a Vision Language Action model on large scale internet video datasets.

Four stage pipeline:
1) Learn a codebook for latent actions from raw videos
2) Pseudolabel a large dataset with these latent actions
3) Pretrain a VLM on this pseudolabeled dataset to predict these latent actions
4) Finetune the latent pretrained VLM on real labeled actions

Experiments are performed in several simulated and real domains to demonstrate that LAPA can recover most of the performance of the same model architecture / finetuning setup trained on labeled versions of the pretraining dataset.
weaknesses: **TL;DR: There are data consistency issues throughout the paper, including in key result figures. With those fixes, method's proposed cross embodiment training recipe appears weak on actual cross embodiment data.**

**Data consistency issues**

Table 15 does not agree with the bar values in Figure 4 and 5 --- the superior performance of LAPA (Bridge) to ActionVLA (Bridge) is caused by swapping of values for ActionVLA and LAPA in the _Pick_ task, based on Table 15's results. Fixing this error makes LAPA (Bridge) worse than ActionVLA (Bridge), which is what one should reasonably expect given LAPA is unsupervised and ActionVLA is supervised and both use the same architecture. Quite honestly I think this should have been caught in a simple sanity check by the authors, as these results don't make much sense otherwise.

Table highlighting of results seem to also consistently have errors
 - In Table 4, Block2BlockRelative, VPT has its score of 48 underlined (to indicate second place) when LAPA scored 52.0
 - In Table 7, row "Separate", ActionVLA has its score of 82.0 underlined (to indicate second place) when VPT scored 84.0

These are just the most obvious issues, I cannot be certain there are not deeper issues with the results. _Given the glaring nature of the errors in headline result plots, it makes me not trust the quantitative results of the authors._

As a personal note, I want to underscore that this is not the sole responsibility of the junior authors who likely prepared the manuscript and made the plots --- at least some of these errors are ones that could (and should) have been caught by senior authors reading the paper and skimming key results. 

**Most studies are robot to robot, providing limited signal on the value of actual cross embodiment training protocol**

These studies are robot to robot tasks, which don't address the question of usefulness of pretraining on domains where we aren't able to get action labels --- it's unreasonable to assume we don't have access to the actions associated with the observations for robot pretraining data. These ablations can be informative as "Supplemental Experiments", but we need to see the action codebook, pretrained on datasets where we do _not_ reasonably have access to actions, is providing real value. 

Experiments that fall into this category
 - The Language Table experiments are robot -> robot, and these are _very_ similar tasks
 - BridgeV2 -> SIMPLER is a larger gap, it's still robot -> robot

**Studies on nonrobot to robot transfer provide weak proof of value**

Assuming the label swapping issue only impacted LAPA (Bridge)'s Pick and Place performance and nothing else (I did not check the correctness other entries in the other tables), ActionVLA trained on the 54k trajectories of Bridgev2 outperforms LAPA pretrained on the 220K human videos. Notably, 54k trajectories is quite small; OpenX has over 1 million, but ActionVLA pretrained on OpenX is an important missing as a baseline from the results --- we need to tease out architecrture differences compared OpenVLA, as LAPA uses LWM-Chat-1M (taken directly from Liu et al 2024) which appears to be superior to OpenVLA's architecture --- but all (corrected) evidence seems to indicate that ActionVLA (OpenX) will not do worse than LAPA (OpenX).

Clearly pretraining on human data is not expected to match training on labeled robot data on a sample vs sample basis, but to make the argument that this pretraining is providing value, we need to see scaling laws showing that increased human data translates to increased performance; unfortunately, the authors made the puzzling decision to run their scaling laws experiments on Bridge (an already small dataset), instead of the salient dataset to this question, so we have no way of knowing how the method improves with data scale. 

Given how the unsupervised action encoder works and the human hand latent analysis in Appendix E / Figure 15, my hypothesis is that if the task distribution of the human dataset (Table 3) remains the same, the gross motor base latent actions are still going to be discovered pretty early on and the marginal value of each additional sequence will quickly approach zero --- this of course needs to be verified or falsified, but there's no evidence in the paper to do that.

**Actionable feedback**

1) Please go review all of the numbers in detail from top to bottom. I happened to catch some of these issues, I have no idea how many more are hidden.
2) I think the ideas presented here are interesting, but the story being told needs to be significantly refined. Pretraining on the human data needs to be front and center, and there needs to be a focus on performance as we scale up this dataset
3) ActionVLM pretrained on OpenX is a useful artifact on its own. I don't know if someone's done this already or not, but such a release itself would be useful to the community.

**UPDATE NOV 24th**

In light of the authors' comments on the correctness of Figures 3 and 4 I have raised my score from 1 to a 3. The remaining issues, related to the fundamental story of the paper, are detailed in my responses to the authors' comments.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See above. I'm willing to discuss with other reviewers and authors to decide my final rating.
soundness: 3
strengths: The proposed unsupervised training method is highly significant. By employing VQ-VAE to learn action tokens, it is possible to model the action knowledge contained in videos without requiring action labels, which makes the pre-training with a larger scale of videos feasible for future applications.

The proposed method for reconstructing future frames during pre-training is reasonable and concise. The visualization results in the paper also confirm that the method can indeed learn structured action latent representations from unsupervised video data.

The experiments conducted in this paper are quite meticulous, including pre-training within the same domain, cross-task pre-training, as well as pre-training on large-scale real-world data like BridgeV2 and Open-X.
summary: This paper proposes an unsupervised method for learning action knowledge from internet-scale video data. Without the need for action labels, the method implicitly models the action information contained in videos, making it possible to acquire generalizable knowledge from a vast amount of videos. The foundation model obtained from pre-training can be transferred to specific downstream robot manipulation tasks through supervised fine-tuning. The article validates the model's superiority over state-of-the-art VLA models by pre-training and fine-tuning with data from different domains.
weaknesses: The method used in this paper is relatively simple. Pre-training through the prediction of future frames has been mentioned in some previous works. And the paper also points out that training directly with the VQ-VAE objective is quite similar to Genie. The ensuing question is, when the pre-trained video data is particularly abundant and spans a large number of domains, whether such a training method is sufficient to capture the patterns of action embeddings. Additionally, the selection of window size H for future frames is fixed during the training process, does such a choice bring difficulties to the modeling of actions? Could you explain the reason for choosing a fixed window size, or an ablation study on different window sizes?

The experimental validation part of the paper could be more robust. The current simulation environments use Language Table and SIMPLER, while real-world experiments only involve three tasks. How does OpenVLA perform in the simulation experiments used in this paper? And compared with OpenVLA, it does not seem to bring absolute advantages in all real-world tasks, such as "Knock" and "Cover" training with Bridge data.

Meta-review: After careful consideration of the reviews, the subsequent author-reviewer discussions, and my own reading of the paper, I recommend accepting this submission. The paper presents an approach to pretraining Vision-Language-Action (VLA) models without requiring ground-truth robot action labels, and demonstrates potential for scaling robot learning through the use of widely available video data.

The paper's primary technical contribution, LAPA, addresses a challenge in robot learning by enabling the use of unlabeled video data for pretraining. The method combines VQ-VAE-based action quantization with a latent VLA model in a technically sound manner. The authors provide comprehensive experimental validation across multiple domains, including both simulated and real-world environments, which demonstrated the method's effectiveness and generalization capabilities.

The review process generated substantial discussion, particularly regarding the method's scalability and experimental validation. While reviewer 2CRD raised important concerns about the need for more extensive scaling experiments with the Something-Something V2 dataset, their expectation for GPT3-style scaling laws may be a bit overly demanding for an academic publication. The other reviewers (CUQQ, 5VGj, jNBu, D2cm, and 7bYx) provided balanced assessments that recognized both the method's current limitations and its contributions.

Several reviewers noted the paper's thorough experimental evaluation, including ablation studies and comparisons with state-of-the-art baselines. The authors' responses during the discussion period were constructive and thorough, addressing concerns about implementation details, clarifying methodological choices, and providing additional experimental results where feasible within computational constraints.

The decision to accept is based on the paper's technical contribution, thorough experimental validation within reasonable academic constraints, and potential impact on the field of robot learning. While more extensive scaling experiments would be valuable, their absence does not diminish the paper's immediate contributions to the community.

Decision: Accept (Poster)