AdaWorld: Learning Adaptable World Models with Latent Actions
Authors: Shenyuan Gao
Siyuan Zhou
Yilun Du
Jun Zhang
Chuang Gan
Venue: ICML 2025 poster (2025)
Keywords: World Model
Latent Action
Video Generation
Diffusion Model
Decision Making
Embodied AI
Primary area: applications
TL;DR: A highly adaptable world model that enables efficient adaptation and effective planning with minimal interactions and finetuning.

Abstract: World models aim to learn action-controlled future prediction and have proven essential for the development of intelligent agents. However, most existing world models rely heavily on substantial action-labeled data and costly training, making it challenging to adapt to novel environments with heterogeneous actions through limited interactions. This limitation can hinder their applicability across broader domains. To overcome this limitation, we propose AdaWorld, an innovative world model learning approach that enables efficient adaptation. The key idea is to incorporate action information during the pretraining of world models. This is achieved by extracting latent actions from videos in a self-supervised manner, capturing the most critical transitions between frames. We then develop an autoregressive world model that conditions on these latent actions. This learning paradigm enables highly adaptable world models, facilitating efficient transfer and learning of new actions even with limited interactions and finetuning. Our comprehensive experiments across multiple environments demonstrate that AdaWorld achieves superior performance in both simulation quality and visual planning.

Ada World: Learning Adaptable World Models with Latent Actions
Shenyuan Gao
Siyuan Zhou'
Yilun Du
Jun Zhang'
Chuang Gan
adaptable-world wwt.oddee.gggtthbeee github.
Abstract
Action-Agnostic Pretraining
World models aim to learn action-controlle fu- ture prediction and have proven essential for the development of intelligent agents. However, most existing world models rely heavily on substan- tial action-labeled data and costly training, mak- ing it challenging adapt novel environments with heterogeneous actions through limited inter- actions. This limitation can hinder their applica- bility across broader domains. To overcome this limitation, we propose Ada World, an innovative world model learning approach that enables effi- cient adaptation. The key idea is to incorporate action information during the pretraining of world models. This is achieved by extracting latent ac- tions from videos in self-supervised manner, capturing the most critical transitions between frames. We then develop an autoregressive world model that conditions on these latent actions. This learning paradigm enables highly adaptable world models, facilitating efficient transfer and learn- ing of new actions even with limited interactions and finetuning. Our comprehensive experiments across multiple environments demonstrate that AdaWorld achieves superior performance in both simulation quality and visual planning.
Video Models
World Models
unicieled viceos
masiive action costly traning
Action-Aware Pretraining
Efficient Action Transter
difforent contexts
kainn acliore
AdaWorld
videos with latent latent actions
Efficient World Model Adaptation
Interoctions
specialized aed world modelss
Planning Results
Act-agnostic
302
2037
AdoWorld
Robol Ern
Game Ery
1.
Figure Different world model learning paradigms. Prior meth- ods often require expensive labeling and training to achieve action controllability in new environments. To overcome this, we intro- duce latent actions unified condition for action-aware pretrain- ing from videos, enabling highly adaptable world modeling. Our world model, dubbed Ada World, readily transfer actions across contexts without training. By initializing the control interface with the corresponding latent actions, Ada World also be adapted into specialized world models efficiently and achieve significantly better planning results than the action- aagionic baseline.
1. Introduction
Intelligent agents should perform effectively across various tasks (Reed et al., 2022; Lee et al., 2022; Durante et al., 2024; Raad et al., 2024). A promising solution to this ob- jective is developing world models that can simulate dif- ferent environments (Wu et al., 2023; 2024: Yang et al., 2024c; Hansen al., 2024). Recent world models are typi- cally initialized from pretrained video models (Xiang al.,
proved generalization, these models still require substantial action labels and high training costs to acquire precise ac- tion controllability. While pseudo labels can be annotated for videos (Baker al., 2022: Zhang et al., 2022), defining unified action format for general environments is chal- lenging. As result, existing methods often require costly training when adapting to new environments with varying action specifications (Gao et al., 2024; Chi et al., 2024; Che et al., 2025). These limitations pose great challenges for transferring and learning new actions based on limited
2024; Agarwal al., 2025; He et al., 2025). Despite im-
'HKUST "Harvard UMMss Amherst "MIT-IBM Watson Lab. Primary contact to Shenyuan Gao c
Proceedings of the = International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Adaptable World Models with Latent Actions
interactions and finetuning.
ity of our latent actions, the resulting world model is readily adaptable various environments. In particular, since our world model has learned to simulate different actions repre- sented by any latent actions, adapting to a new environment is akin to finding the mapping of corresponding latent ac- tions for its action space. Given one demonstration of an action, our model can readily transfer the demonstrated ac- tion by extracting the latent action with latent action encoder and reusing it across different contexts. When action labels are provided, we can similarly obtain their latent actions and initialize the control interface efficiently. This enables us to adapt our model to specialized world models with minimal finetuning. When only a limited number of interactions are available (e.g.. 50 interactions), our approach is significantly more efficient than pretraining from aagtinnagnootic videos.
As humans, we can estimate the effects of different actions through limited experiences (Ha & Schmidhuber, 2018). This ability likely arises from our internal representations of actions learned from extensive observations (Rizzolatti et al., 1996; Romo et al., 2004; Dominici et al., 2011). These common knowledge can be reused across different contexts and associated with specific action spaces efficiently (Ry- bkin et al., 2019; Schmeckpeper al., 2020; Sun et al., 2024). Consequently, humans can easily transfer observed actions to various contexts and imagine the transitions of new environments through a few interactions (Poggio & Bizzi, 2004). These insights motivate us to ponder: can we achieve human- adaptability in world modeling by learn- ing transferrable action representations from observations?
Note that our approach is as scalable existing video pre- training methods (Seo et al., 2022; Mendonca et al., 2023; Wu et al., 2023; Agarwal et al., 2025; He et al., 2025; Yu et al., 2025). To enhance the generalization ability of Ada- World, we collect a large corpus of videos from thousands of environments through automated generation. The resulting dataset encompasses extensive interactive scenarios, span- ning from ego perspectives and third-person views virtual games and real- world activities. After action-aware pretrain- ing at scale, we show that the adaptability of Ada World can seamlessly generalize to a wide variety of domains.
In this paper, we propose AdaWorld, an innovative pretrain- ing approach for highly adaptable world models. Unlike pre- vious methods that only pretrain on action-agnostic videos, we argue that incorporating action information during pre- training will significantly enhance the adaptability of world models. As illustrated in Figure 1, the adaptability of Ada- World primarily manifests in two aspects: (1) Given one demonstration of an action, AdaWorld can readily transfer that action to various contexts without further training. (2) It can also be efficiently adapted into specialized world models with raw action inputs via minor interactions and finetuning, enabling more effective planning in various environments.
In summary, we make the following contributions:
Ada consists of two key components: latent action autoencoder that extracts actions from unlabeled videos, and autoregressive world model that takes the extracted actions conditions. Our main challenge is that in-the-wild videos often involve complicated contexts (e.g., colors and textures), which hinders effective action recognition. To overcome this challenge, we introduce an information bot- tleneck design to our latent action autoencoder. Specifically, the latent action encoder extracts a compact encoding from two consecutive frames. We refer to this encoding latent action hereinafter, as it is used to represent the transition between these two frames. Based on the latent action and the former frame, the latent action decoder makes its best effort to predict the subsequent frame. By minimizing the prediction loss using the minimal information encoded in the latent action, our autoencoder is encouraged to disentan- gle the most critical action from its context. Unlike previous methods (Bruce et al., 2024; Chen et al., 2024b; Ye et al., 2025) that focus on playability and behavior cloning, we compress the latent actions into a continuous latent space to maximize expressiveness and enable flexible composition. We find that our latent actions are context-invarian and can be effectively transferred across different contexts.
We present AdaWorld, autoregressive world model that is highly adaptable across various environments. It can readily transfer actions to different contexts and allows efficient adaptation with limited interactions.
We establish Ada World on a large-scale dataset sourced from extremely diverse environments. After extensive pretraining, Ada World demonstrates strong generaliza- tion capabilities across various domains.
We conduct comprehensive experiments across multi- ple environments to verify the efficacy of Ada AdaWorld. Our model achieves promising results in action transfer, world model adaptation, and visual planning.
2. Method
In this section, we first introduce the architectural design of our latent action autoencoder (Sec. 2).. By leveraging the latent actions as conditions, we then build an autoregressive world model through action-aware pretraining (Sec. 2.2). Finally, we demonstrate how our model facilitates highly adaptable world modeling (Sec. 2.3).
We then pretrain an autoregressive world model that condi- tions on the latent actions. Thanks to the strong transferabil-
2

---

Learning Adaptable World Models with Latent Actions
diliionn
Autoregressive World Mocial
feex
next frame
neat frame prediction
Latent Action Encoder
Latent Action Decoder
Latent Action Encoder
latent octio space
latent actions
Figure2 2. Latent action autoencoder. With an information bottle- neek design, our latent action autoencoder is able to extract the most critical action information from videos and compresses it into a continuous latent action.
short
noke cup
autorogressive rollout
Nistoriol
Figure 3. Action-aware pretraining. We extract latent actions from unlabeled videos using the latent action encoder. By lever- aging the extracted actions as unified condition, we pretrain a world model that can perform autoregressive rollouts at inference.
2.1. Latent Action Autoencoder
Instead of only taking actionless videos for world model pretraining, our key innovation is to incorporate action in- formation during the pretraining phase. Benefiting from the pretrained knowledge of action controllability, the result- ing world models can be efficiently adapted using limited ground truth actions. However, action labels are rarely avail- able for in-the-wild videos. While a common practice is to collect these labels through interactions, collecting them across a multitude of environments incurs significant labor. Furthermore, defining a unified action format across diverse environments often unfeasible, and current world models require costly training to accommodate new action formats.
The spatial attention can attend to all tokens within each frame, while the temporal attention has access the two tokens in the same spatial positions across the two frames. We also incorporate rotary embeddings (Su et al., 2024) in temporal attentions to indicate the causal relationship. After sufficient attention correlations, the learnable tokens d++++ can adaptively aggregate the temporal dynamics between the two input frames. We then discard all tokens and only project a++1 estimate the posterior of the latent action (Ma, oa) following the standard VAE (Kingma & Welling, 2014). Subsequently, we sample ā from the approximated posterior and attach it to ft. which is then sent to the latent action decoder. The latent action decoder is a spatial Trans- former that predicts the subsequent frame fe+1 in the pixel space. The whole latent action autoencoder is optimized with the VAE objective:
To address these challenges, instead of relying explicit ac- tion annotations, we propose extracting latent actions from videos as a unified condition for world model pretraining. Nevertheless, in general videos, the action information is of- ten entangled with the contexts, posing significant difficulty for effective action recognition. Inspired by the observation that agents' actions often drive the dominant variation in most interactive scenarios (Rybkin et al., 2019; Menapace et al., 2021; 2022; Bruce et al., 2024), we introduce an in- formation bottleneck to automatically differentiate actions from observations.
Cprod (s(+1) = (ggo(alSeeaa) (0 (1)
-Dxi ((f((((()
Compared to the original pixel space, the dimension of our latent action is extremely compact. Hence, it is challeng- ing to forward the entire subsequent frame to the decoder via latent action. To minimize the prediction error of the subsequent frame, the latent action à must encapsulate the most critical variations relative to the former frame. This re- sults in context-invarian action representations that closely correspond to the true actions taken by the agents.
To be specific, we instantiate a latent action autoencoder based on the Transformer architecture (Vaswani et al., 2017), where the encoder extracts the latent action à from two con- secutive frames fes+11 and the decoder predicts the sub- sequent frame f++ based on the latent action ă and the former frame ft. The latent action encoder divides two frames fes+1 into image patches of size 16 x 16. These patches are then projected to patch embeddings and flattened along the spatial dimension. Afterwards, they are concate- nated with two learnable tokens dt.++1. Sinusoidal position embeddings (Dosovitskiy et al., 2021) are also applied to each frame to indicate the spatial information. To efficiently encode the tokens from these two frames, we employ a spa- tiotemporal Transformer (Bruce et al., 2024) with L stacked blocks. Each block comprises interleaved spatial and tempo- ral attention modules, followed by a feed-forward network.
Nevertheless, we empirically find that our latent action au- toencoder, trained using the aforementioned formulation, struggles to express diverse transitions between frames. This problem arises because the standard VAE imposes a strong constraint on posterior distributions. Conversely, removing this constraint may compromise the disentanglement ability of VAE (Burgess et al., 2017). To remedy this, we adopt the /- B-VAE formulation (Higgins et al., 2017; Alemi et al.,
3

---

Learning Adaptable World Models with Latent Actions
source
target
source
target
target
source
target
Figure Action transfer from demonstrations. By extracting and reusing the latent actions in different contexts, AdaWorld can readily transfer the demonstrated actions from source videos to various target scenes without training. Please see Appendix C for more results.
2017), which introduces an adjustable hyperparameter f::
2.2. Action-Aware Pretraining
cered (fe+1) Egg(a 1) po fi)
(2)
After training the latent action autoencoder, we can use its encoder to automatically extract action information from videos. This allows us to incorporate action information for world model pretraining, which we refer to as action-aware pretraining. To realize this, we pretrain a world model that predicts the next frame conditioned on the current latent action. As shown in Figure 3, utilize the latent action encoder to extract the latent actions between frames and send them as inputs for our world model. Unlike previous methods that often predict video clips (Yang et al., 2024c:
B D
The additional hyperparameter enables flexibly control the information contained by the latent actions. In practice we empirically adjust this hyperparameter to achieve good trade-off' between expressiveness and context disentangling ability our latent actions. As shown in Figure 4, our latent action autoencoder can extract context-ine context-invariant actions that transferrable across different contexts.

---

Learning Adaptable World Models with Latent Actions
latent action A fright)
latent action B [Jump]
latent action A +
(jump right)
Figure Action composition. compose two actions by averaging their latent actions the continuous latent space, resulting in new action that merges the functions of both. This indicates that our latent action space is semantically continuous in the meanings of actions.
Xiang et al., 2024; Agarwal et al., 2025), our model supports frame-level control, offering finer granularity for interac- tion. To ensure smooth transitions, memory with K historical frames. During inference, our model can predict a sequence of future frames by autore- gressively repeating the next-frame prediction process and appending the predicted frames to the memory.
conditioning information which includes historical frames and the latent action á.
maintain short-term
2.3. Highly Adaptable World Models
After action-aware pretraining across various environments, the world model can be controlled by different latent ac- tions, making it highly adaptable for multiple applications, including efficient action transfer, world model adaptation, and even action creation.
Although may seem straightforward repurpose the la- tent action decoder the function of world model, it only makes coarse predictions via single forward pass, resulting in sigmificant quality degradation after several interactions. To achieve genuine predictions, we establish an indepen- dent world model based on diffusion models. Specifically, we initialize the world model with Stable Video Diffusion (SVD) (Blattmann et al., 2023), a latent diffusion model trained with the EDM framework (Karras al., 2022). Dif- ferent from the original SVD, we only denoise one noisy frame each time. To enable deep aggregation with the action information, the latent action is concatenated with both the timestep embedding and the CLIP image embedding from the original SVD. The last frame in memory is used as the condition image of SVD. To inherit the pretrained temporal modeling capability, we encode the historical frames us- ing the SVD image encoder and concatenate them with the noise latent map of the frame to predict. Since the number of available historical frames may vary in practice, we ran- domly sample historical frames with a maximum length of during training and send the memory length condition to the world model. Following previous practices (He al., 2022; Valevski al., 2025), noise augmentation also ap- plied to corrupt the historical frames during training. This augmentation effectively alleviate the long-term drift problem, even when no noise is applied during inference. We pretrain the world model on our large-scale dataset by minimizing the following diffusion loss:
Efficient action transfer. When presented with a demon- stration video, we use the latent action encoder extract a sequence of latent actions. This enables us disentangle the action from its context and replicate it across different contexts. Specifically, given the initial frame from new context, we can reuse the extracted latent action sequence as the conditions to generate a new video autoregressively, As demonstrated Figure 4, AdaWorld naturally transfers actions from source videos to various contexts.
Efficient world model adaptation. AdaWorld also allows efficient world model adaptation with limited action labels and training steps. Specifically, after collecting few action- video pairs through interactions, we use the latent action encoder to infer their latent actions. Thanks to the conti- nuity of our latent action space, latent actions for the same label can be averaged directly. We empirically find that the averaged embedding consistently represents the intended ac- tion. Thus, for new environment with N discrete actions, we initialize specialized world model using N averaged latent actions and finetune the whole model for few steps. For environments with continuous action spaces, since there are infinite options, we add a lightweight MLP to map raw action inputs to the latent action interface. The interface can also be efficiently initialized by finetuning the MLP with minimal action-latent action pairs. Figure shows that the models initialized in aforementioned ways can be efficiently adapted to take control inputs through minimal finetuning.
|1000 2o(#1, L {)
Cpretrain Emo....
(3)
where do the prediction of our world model and is the
Action composition and creation. It is also noteworthy
5

---

Learning Adaptable World Models with Latent Actions
that Ada World enables several unique applications com- pared to existing world models. For instance, it allows the composition of new actions by interpolating observed ac- tions within the latent space, as demonstrated in Figure 5. In addition, by collecting and clustering latent actions, we can easily create a flexible number of control options with distinct functions and strong controllability. This suggests that World could serve as an alternative to generative in- teractive environments (Bruce et al., 2024). See Appendix C for experimental details on action creation.
Table 1. Action transfer comparison. In both datascts, AdaWorld excels at transferring the demonstrated actions to different contexts.
LIBERO
S8v2
Method
FVD1
ECSt
Humant
FVDJ
ECS?
Human"
Act-agnostic
1545.2
0.702
0%
847.2
0.592
1%
Flow cond.
1409.5
0.724
2%
702.8
0.611
10.5%
Discrete cond.
1504.5
0.700
3.5%
726.8
0.596
21.5%
AdaWorld
767.0
0.804
70.5%
473,4
0.639
61.3%
3.1. Action Transfer
AdaWorld can readily transfer a demonstrated action to various contexts without further training. Below, we provide both qualitative and quantitative evaluations to showcase how effectively Ada World performs action transfer.
3. Experiments
In this section, we first demonstrate AdaWorld's strengths in action transfer in See. 3.1. We then study how efficient world model adaptation enables better simulation and plan- ning in Sec. 3.2. Lastly, analyze the effectiveness of our designs with ablation studies in Sec. 3.3.
Qualitative results. We transfer action sequences of length 20 through autoregressive generation in Figure 4. It shows that Ada World can effectively disentangle the demonstrated actions and emulate them across contexts. Qualitative com- parison with other baselines can be found in Appendix C.
To thoroughly understand the adaptability of our approach, we compare AdaWorld with three representative baselines:
Quantitative results. To quantitatively compare with other baselines, we construct evaluation set sourced from the un- seen LIBERO (Liu et al., 2023) and Something- Soomeiinnn v2 (SSv2) (Goyal et al., 2017) datasets. Specifically, we select and pair videos from the same tasks in LIBERO and the same labels among the top-10 most frequent labels in SSv2, resulting in 1300 pairs for evaluation (more details in Appendix D). While the selected video pairs contain similar actions, we find that the video pairs from LIBERO often differ in the arrangement of objects, and those from SSv2 have significant differences in contexts. For each video pair, we take the first video as the demonstration video and use the first frame from the second video as the initial frame. We then generate videos by extracting action conditions from the demonstration video and employing different models to autoregressively predict the next 20 frames from the initial frame. The evaluation is performed by measure the gener- ated videos against the original videos using Fréchet Video Distance (FVD) (Unterthiner et al., 2018). To complement the FVD evaluation, which reflects overall distribution simi- larity, we additionally employ Embedding Cosine Similarity (ECS) (Sun et al., 2024) that performs frame-level measure- ments with I3D (Carreira & Zisserman, 2017). We further conduct human evaluation on a set of 50 video pairs from LIBERO and SSv2, respectively. Four volunteers are invited to judge whether the action is successfully transferred or not. Both the automatic and human evaluations in Table I demonstrate that our continuous latent action achieves the best action transfer performance, underscoring its capability to express more nuanced actions without losing generality.
action-agnostic pretraining, In this setup, we train a world model that shares the same architecture as Ada- World but always takes zeros as action conditions dur- ing pretraining. This baseline is used to demonstrate the effect of the predominant pretraining paradigm that relies only on action- videos (Mendonca et al., 2023; Wu et al., 2023; Gao et al., 2024; Che et al., 2025; Agarwal et al., 2025; He et al., 2025).
Optical flow as an action-aware condition. We auto- matically predict optical flows from videos using Uni- Match (Xu et al., 2023a). The flow maps are downsam- pled to 16 x 16 and flattened as conditional encodings to replace the latent actions during pretraining. This baseline serves as alternative solution for extracting action information from unlabeled videos.
Discrete latent action as an action-aware condition. We also implement variant of the latent action autoen- coder based on the standard VQ-VAE (Van Den Oord et al., 2017). Instead of using a continuous latent ac- tion space, this variant adopts a VQ codebook with 8 discrete codes following Genie (Bruce et al., 2024).
Except for the above modifications, we align other training settings for the baselines and our method. The world models of all compared methods are trained for 50K iterations to en- sure a fair comparison. Our training dataset comprises four publicly accessible datasets (Goyal al., 2017; Grauman et al., 2022; O'Neill et al., 2024; Ju et al., 2024) and videos collected automatically from 1016 environments in Gym Retro (Nichol et al., 2018) and procgen Benchmark (Cobbe et al., 2020). This results in about 2000 million frames of in- teractive scenarios in total. More details about our datasets and implementation are provided in Appendices A and B.
3.2. World Model Adaptation
We also investigate how the proposed method benefits effi- cient world model adaptation in terms of simulation quality
6

---

Learning Adaptable World Models with Latent Actions
Table 2. Action-controlled simulation quality when adapting to four unseen environments. All results are tested after 800 finctuning steps, using 100 samples for cach discrete action (Habitat, Minecraft, DMLab) or 100 continuous trajectory samples (nuScencs).
Habitat (discrete action)
Minecraft (discrete action)
DMLab (discrete action)
nuScenes (continuous action)
Method
PSNR1
LPIPS|
PSNR*
LPIPS]
PSNR*
LPIPS!
PSNRT
LPIPS]
Act-agnostic
20.34
0.450
19.44
0.532
20.96
0.386
20.86
0.475
Flow cond.
22.49
0.373
20.71
0.492
22.22
0.357
20.94
0,462
Discrete cond.
23,31
0.342
21.33
0.465
22.36
0.349
21.28
0.450
AdaWorld 23.58
0.327
21.59
0.457
22.92
0.335
21.60
0.436
50 Winerroft Samples for Each Action
1:00 Minecroft Samples for Each Action
150 Minecraft Samples for Each Action
200 Minerroft Somples for Each Action
Cuas ved Atiesgrmix
Ouas cedricd
vodricd Atissgpnni
sedi vd Ainsupenic
1124
300
20
200
430 Traning Steps
430
403 Thaning Stepe
Thaiinn Steps
Thaniing Steps
100
Sumplees
Outs ruscemmes Semples
Cas nusicmmes Action agrottic
Duns ALL oe:
Attice agrictic
Jct Jc sost
PAR
ralling steps
Traiiing Stepp
training 50035
Figure 6. PSNR curves for world model adaptation. With limited samples and training steps, AdaWorld adapts to the action controls of new environments more rapidly than conventional pretraining methods.
and visual planning performance.
action-latent action pairs for 3K steps, which takes less than 30 seconds on single GPU.
3.2.1. SIMULATION QUALITY
Results. As reported in Table 2, Ada achieves the best fidelity after finetuning with limited interactions and compute. The comparison results suggest that the proposed method allows the world models to efficiently simulate new action controls in unseen environments. Note that all action- aware variants significantly outperform the action-agnoss baseline, underlining the importance of our key innovation, Le., incorporating action information during pretraining.
Setup. To evaluate the simulation quality after adaptation, we choose three environments with discrete action space (Habitat (Savva et al., 2019), Minecraft, DMLab (Beattie et al., 2016)) and one environment with continuous action space (nuScenes (Caesar et al., 2020)) that not included in our training dataset. Each environment has validation set consisting of 300 samples, which is used to evaluate the adaption quality in terms of PSNR (Hore & Ziou, 2010) and LPIPS (Zhang et al., 2018). To demonstrate the adaptability with restricted labels, we collect only 100 samples for each action in every discrete environment and 100 trajectories for nuScenes. Using the limited interaction data, then finetune all compared world models for 800 steps with batch size of 32 and learning rate of 5 10-5, The learning rate for the pretrained weights is discounted by factor of 0.1. For the action-agnostic baseline, we initialize action embeddings with random parameters. For the other three models, we use the averaged action conditions extracted from the 100 samples initialize the action embeddings as described in Sec. 2.3. Note that for nuScenes, add two-layer MLP to map continuous displacements to the latent action interface. The MLP is fine-tuned with limited
To further demonstrate our sample efficiency and finetuning efficiency. we conduct more comparative experiments using different sample numbers and finetuning steps Minecraft and nuScenes. We show the evolving curves of PSNR in Figure 6. In all cases, Ada Aorld performs much better at the beginning and improves significantly faster after a few finetuning steps. This suggests that our approach provides a superior initialization for efficient world model adaptation compared to conventional pretraining methods.
3.2.2. VISUAL PLANNING IN GAMES
Setup. After learning action controls, world models can be utilized for planning. To demonstrate the superiority of Ada- World in planning performance, we first compare it with the
7

---

Learning Adaptable World Models with Latent Actions
Table 3. Visual planning results in games. For all selected scenes from four Proegen environments, report the success rate and standard error with random seeds. Ada AdaWorld achieves higher average success rates than other baselines as well as Q-learning even without finetuning. Oracle:
We use the ground truth simulator for MPC, which indicate the upper bound of this planning strategy.
Success Rate!
Method
Heist
Jumper
Maze
CaveFilyer
Average
Random
19.33++4.41%
22.00+2.50%
5..43++5.44%
22.00+2..50%
26. 2.555%
Act-agnostic
33.55%
20.47.12.45%
39.33:12.87%%
23.33 11911...44
26.00.10.98%
AdaWorld
w/o finctunc
38.67+2.011% %
28.000+2.25%%
A1.33:+2.725%
31.33++2.505%
44.83=+1133%
w/ finetune
66.67:14.09%
58.67
1..000:::.995
33,3331..805
56.67-12.16%
Q-learning
22.67++3.87%
47.33++6.77 1%
4.67=0.81%
34.00
|
27.17++1.27%%
Oracle (GT env.)
86.67.1.16%
77.33.12.67%
84.67.12.91%
74.0011..99%
80.67.12.111
Table 4. Visual planning results in robot tasks. The success rates and standard errors are obtained over 4 runs for each task from VPa. We also report the aggregated success rates normalized by the scores of the ground truth simulator the right.
Success Ratet
Method
Robosuite push
Open slide
Blue button
Green button
Red button
Upright block
Aggregate
Act-agnostic
07.50=0.50%
1.67 5.83+2.85%
1.00%
3.33-+0.0055 10.83=2.5005
0.00-10000%
1.67=
5.03
AdaWorld
63.50=1.71%
29.17+2.50%%
10.00=2.366%
5.00=0.96%
21.54
action-agnostic baseline in video game environments using sampling-based model predictive control (MPC) optimized by Cross-Entropy Method (De Boer et al., 2005; Chua et al., 2018). The MPC planning and optimization procedure are deferred to Appendix B.4. We define a goal-reaching task based on the Procgen benchmark (Cobbe et al., 2020) and select 30 scenes from each of four environments (Heist, Jumper, Maze, CaveFlyer). This ensures that the specified goals can be reached within acceptable number of steps (more details in Appendix E). For each scene, randomly collect 100 samples for each action in the default action space (LEFT, DOWN, UP, RIGHT), Based on the collected samples, the pretrained world models are finetuned for 500 steps with batch size of 32 and learning rate of 5 x 10-5 We then use the finetuned world models to perform MPC planning in the selected scenes. The reward is defined as the cosine similarity between the predicted observations and the image of the final state. The planning is deemed successful if the agent reaches the final state within 20 steps.
To further demonstrate the effectiveness of our approach, we also compare the planning results with Q-learning (Sutton & Barto, 2018), classical model-free reinforcement learning method. For each scene, we construct a Q-table using the same samples collected for the MPC planning. The states of Q-table are represented by quantized images, and the rewards are obtained by computing the cosine similarity with the goal image. As shown Table 3, Ada signifi- cantly dominates the Q-learingg method, suggesting that approach makes more effective use of limited interactions.
3.2.3. VISUAL. PLANNING IN ROBOT TASKS
Setup. To verify our efficacy in robot control tasks, we pre- train a low-resolution variant of AdaWorld and evaluate the planning performance on the Vp benchmark (Tian et al., 2023) after adaptation. The planning is also sampling- based and is performed using the model-predictive path integral (MPPI) (Williams et al., 2016; Nagabandi et al., 2020). We focus on a similar compute- -fffcciie setting and finetune the pretrained variant and an cction-agnostie baseline for 1K steps. The evaluation is conducted on 100 tabletop Robo- suite tasks (Zhu et al., 2020) and 7 RoboDesk tasks (Kannan et al., 2021). More details are in Appendix B.5.
Results. Table 3 presents the success rates averaged over 5 random seeds. While the action-agnostic baseline performs similarly to random planning, Ada World substantially in- crease the success rates across all environments. This indi- cates that our approach not only adapts more efficiently but also enables more effective planning. Visit our project page to see planning demonstrations of agents in games.
Results. Table 4 reports the success rates on Vp2, We omit the Flat block and Open drawer tasks from RoboDesk, as they do not yield meaningful scores under our constrained adaptation setting. The results show that AdaWorld adapts more efficiently with limited finetuning steps and improves the planning performance by clear margin.
Additionally, we evaluate the visual planning performance without finetuning our model using the collected samples. In particular, we only utilize the averaged latent actions derived from these samples as action embeddings for the corresponding scenes. The results in Table 3 indicate that even without updating model weights, our variant still out- performs the finetuned action-agnostic pretraining baseline.
3.3. Ablation and Analysis
Interface initialization. We ablate the latent action initial- ization approaches in Sec. 2.3 with random initialization
8

---

Learning Adaptable World Models with Latent Actions
Table 5. Impacts of training data diversity. Increasing data diver- sity enhances the generalization of latent actions to new domains.
- 2x10-+ [AdaWorld]
- 2x10-1
Proegen
Training Data
PSNR1
LPIPS|
OpenX
25.51
0
Retro
26.43
0,250
Retro+OpenxX 26.62
0.234
Table 6 Generality of AdaWorld. Applying action-aware pre- training to iVideoGPT also significantly improves its adaptability,
serrigg
Model
BAIR
Figure 7. UMAP of latent actions. Reducing the value of B in- creases expressiveness but sacrifices disentanglement from context.
PSNR1
LPPPS..
iVideoGPT
16,59
0.220
WiioooGPT+AdWWold
17.40
0.204
Hyperparameter choice. In Eq. (2), the hyperparameter / is adjusted to achieve a good trade-off between expressive- ness and context disentangling ability of latent actions. To provide more intuitive illustration, we randomly collect 1000 samples for each action from Habitat, Minecraft, and DMLab, and use UMAP (McInnes et al., 2018) for visual- ization. Figure 7 shows that the same actions, even from different environments, are clustered together, which val- idates the Note that noise exists because the action inputs cannot be executed in certain states (e.g., cannot go ahead when an obstacle is in front). We also compare samples inferred by model trained with a lower B. Although this results in more differentiable latent actions, it also reduces action overlap across environments thus sacrificing disentanglemen ability. We therefore set B as 2 x 10 by default.
and adapt the resultant model to the unseen Minecraft and nuScenes. Figure 6 shows that randomly initializing the control interface of AdaWorld results in a quality drop at the beginning. Nevertheless, it is noteworthy that although our variant is slightly worse than the action-agnoste pretraining baseline when the finetuning begins, it rapidly surpasses the action-agnostic baseline after just 200 steps. This is because Ada World has learned a highly adaptable control interface through action- -ware pretraining, allowing it to efficiently adapt to an unseen environment by simply fitting the action embeddings for the new action space.
property of our latent actions.
Data diversity. We also study the impact of data mixture to the generalization ability of latent actions. To this end, we implement three latent action autoencoder with different combinations of Open X-Embodiment (OpenX) (O Neill et al., 2024) and Gym Retro (Nichol et al., 2018) datasets for 40K steps. We then assess the latent action decoder predic- tions of these three variants on Procgen benchmark (Cobbe et al., 2020) in Table 5. Surprisingly, we discover that even though OpenX mainly consists of real-world robot videos, incorporating OpenX helps the latent action autoencoder generalize to the unseen 2D virtual games in Procgen. This suggests that further increasing data diversity may positively affect the generalization of our latent actions.
4. Conclusion
In this paper, we introduce AdaWorld, new world model learning approach that facilitates efficient adaptation across various environments. It is highly adaptable in transferring and learning new actions with limited interactions and fine- tuning. Extensive experiments and analyses demonstrate the superior adaptability of AdaWorld, highlighting its potential as a new paradigm for world model pretraining.
Method generality. To demonstrate the generality of our method, we use iVideoGPT (Wu et al., 2024) as state-of- world
Limitations. While AdaWorld promotes adaptable world modeling, several challenges remain. First, it does not oper- ate at real-time frequeney. Future work could incorporate distillation and sampling techniques (Feng et al., 2024; Yin et al., 2025) to accelerate inference speed. Similar to prior works (Yang et al., 2024c), AdaWorld struggles to create novel content when the rollout exceeds the initial scene. This issue is likely to be solved by scaling model and train- ing data (Bruce et al., 2024; Bar et al., 2025). Additionally, our model falls short in achieving extremely long-term roll- outs, and we will explore potential solutions (Chen et al., 2024a; Feng et al., 2024; Ruhe et al., 2024) in future work. We also append some primary failure cases in Appendix C.
the-art baseline. iVideoGPT is an model with an autoregressive Transformer architecture. It is pretrained by action-agnostic video prediction and adds a linear projection to learn action control during finetuning. For fair comparison, we implement a variant by condition- ing iVideoGPT with our latent actions during pretraining. The training details can be found in Appendix B.6. After finetuning, we compare action-controlled simulation quality on BAIR robot pushing dataset (Ebert et al., 2017) in Ta- ble 6. The proposed action-aware pretraining significantly enhances the adaptability of iVidcoGPT, suggesting that our method is generally applicable to different world models.

---

Learning Adaptable World Models with Latent Actions
Acknowledgements
Colosseo: A Large-Scale Manipulation Platform for Scal- able and Intelligent Embodied Systems. arXiv preprint arXiv:2503.06669, 2025.
Shenyuan Gao and Jun Zhang were supported by the Hong Kong Research Grants Council under the NSFC/RGC Col- laborative Research Scheme grant CRS HKSS6603/332
Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. Understanding Disen- tangling in 3-VAE. In NeurIPS Workshops, 2017.
Impact Statement
This paper presents work whose goal is to advance the field of machine intelligence. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q.. Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuScenes: A Multimodal Dataset for Autonomous Driving. In CVPR, 2020.
Carreira, J. and Zisserman, A. Quo Vadis, Action Recogni- tion? A New Model and the Kinetics Dataset. In CVPR, 2017.
References
Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E.,, Cai, T., Chattopadhyay, P., Chen, Y., Cui, Y., Ding, Y., al. Cosmos World Foundation Model Platform for Physical Al. arXiv preprint arXiv:2501.. 103555, 2025.
Che, H., He. X., Liu, Q., Jin, C., and Chen, H. GameGen- Interactive Open-
Open-World World Game Video Generation. In ICLR, 2025.
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep Variational Information Bottleneck. In ICLR, 2017.
Chen, B., Monso, D. M., Du, Y., Simchowitz, M., Tedrake, R., and Sitzmann, V. Diffussin Forcing: Next-Token Prediction Meets Full-Sequence Diffusion. In NeurlPS, 2024a.
Alonso, E., Jelley, A., Micheli, V., Kanervisto, A., Storkey, A., Pearce, T., and Fleuret, F. Diffusion for World Model- ing: Visual Details Matter in Atari. In NeurIPS, 2024.
Chen, X., Guo, J., He, T., Zhang, C., Zhang, P., Yang. D. C., Zhao, I.., and Bian, IGOR: Image-GOal Representa- tions are the Atomic Control Units for Foundation Mod- els in Embodied AI. arXiv preprint arXiv:2411.00770 2024b.
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos. In NeurIPS, 2022.
Bar, A., Zhou, G., Tran, D., Darrell, T., and LeCun, Y. Navigation World Models. CVPR, 2025.
Chen, Y., Ge, Y., Li, Y., Ge, Y., Ding, M., Shan, Y., and Liu, X. Moto: Latent Motion Token as the Bridg- ing Language for Robot Manipulation. arXiv preprint arXiv:2412.044455, 2024c.
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wain- wright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., et al. DeepMind Lab. arXiv preprint 1eXiv:661.003800 2016.
Chi, X., Zhang, H., Fan, C.-K., Qi, Zhang, R., Chen, A., Chan, C.-m., Xue, W., Luo, W., Zhang, S., et al. EVA: An Embodied World Model for Future Video Anticipation. arXiv preprint arXiv:24100 241.:11611.. 2024.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets. arXiv preprint arXiv:2311.15122 2023.
Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep Reinforcement Learning in Handful of Trials using Probabilistic Dynamics Models. In NeurIPS, 2018.
Bruce, J., Dennis, M. D., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steiger- wald, R., Apps, C., et al. Genie: Generative Interactive Environments. In /CML, 2024.
Cobbe, K., Hesse, C., Hilton, J., and Schulman, Leverag- ing Procedural Generation to Benchmark Reinforcement Learning. In ICML, 2020.
Bu, Q., Zeng, J., Chen, L., Yang. Y., Zhou, G., Yan, J., Luo, P., Cui, H., Ma, Y., and Li, Closed-Loop Visuo- motor Control with Generative Expectation for Robotic Manipulation. In NeurIPS, 2024.
Cui, Z. J., Pan, H., Iyer, A., Haldar, S., and Pinto, L. Dy- naMo: In- In-Domain Dynamics Pretraining for Visuo-Motor Control. In NeurlPS, 2024.
De Boer, P.-T,, Kroese, D. P., Mannor, S., and Rubinstein, R. Y. A Tutorial on the Cross- cooos-Entropy Method. Annals of Operations Research, 2005.
Bu, Q.. Cai, J., Chen, L., Cui, X., Ding, Y., Feng, S., Gao, S., He, X., Huang, X., Jiang, S., et al. AgiBot World
10

---

Learning Adaptable World Models with Latent Actions
Dominici, N., Ivanenko, Y. P., Cappellini, G., d' Avella, A., Mondi, V., Cicchese, M., Fabiano, A., Silei, T., Di Paolo, A., Giannini, C., al. Locomotor Primitives in Newborn Babies and Their Development. Science, 2011.
Hansen, N., Su, H., and Wang, X. TD-MPC2: Scalable, Robust World Models for Continuous Control. In ICLR, 2024.
Hassan, M., Stapf, S., Rahimi, A., Rezende, P., Haghighi, Y., Brüggemann, D., Katircioglu, I,, Zhang, L.,, Chen, X., Saha, S., et al. GEM: A Generalizable Ego-Vision Multimodal World Model for GrnnnGr Ego-Motion, Object Dynamics, and Scene Composition Control. In CVPR, 2025.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly. S., Uszkoreit, J., and Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2021.
Du, Y., Yang, S., Dai, B., Dai, H., Nachum, O., Tenenbaum, J., Schuurmans, D., and Abbeel, P. Learning Universal Policies via Text-Guided Video Generation. In NeurIPS, 2023.
He, H., Zhang, Y., Lin, L., Xu, Z., and Pan, L. Pre-Trained Video Generative Models as World Simulators. arXiv preprint arXiv:2502.07825, 2025.
He, Y., Yang, T., Zhang, Y., Shan, Y., and Chen, Q. Latent Video Diffusion Models for High-Fidelity Long Video Generation. arXiv preprimt arXiv:2211.1322) 2022.
Du, Y.. Yang, M., Florence, P., Xia, F,, Wahid, A., Ichter, B., Sermanet, P., Yu, T., Abbeel, P., Tenenbaum, J. B., et al. Video Language Planning. In ICLR, 2024.
Higgins, I,, Matthey, I.., Pal, A., Burgess, C. P., Glorot, X., Botvinick, M. M., Mohamed, S., and Lerchner, A. beta- VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In ICLR, 2017.
Durante, Z., Sarkar, B., Gong, R., Taori, R., Noda, Y., Tang, P. Adeli, E., Lakshmikanth, K., Schulman, K., Mil- stein, A., al. An Interactive Agent Foundation Model. arXiv preprint wrxv:2202.055229 2024.
Hong, Y., Liu, B., Wu, M., Zhai, Y., Chang, K.-W., Li, L.,
Ebert, F., Finn, C., Lee, A. X., and Levine, Self- Supervised Visual Planning with Temporal Skip Con- nections. In CoRL, 2017.
Lin, K., Lin, C.-C., Wang. J., Yang, Z., et al. SlowFast- VGen: Slow-Fast Learning for Action-Driven Long Video Generation. In ICLR, 2025.
Edwards, A., Sahni, H., Schroecker, Y., and Isbell, C. Imi- tating Latent Policies from Observation. In ICML, 2019.
Hore, A. and Ziou, D. Image Quality Metries: PSNR vs. SSIM. In ICPR, 2010.
Feng, R., Zhang, H., Yang, Z., Xiao, J., Shu, Z., Liu, Z., Zheng, A., Huang, Y., Liu, Y., and Zhang, H. The Ma- trix: Infinite-Horizon World Generation with Real-Time Moving Control. arXiv preprint arXiv:2412.03366 2024.
Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., and Corrado, G. GAIA-1: A Generative World Model for Autonomous Driving. arXiv preprint arXiv:2309. 17080, 2023.
Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., and Li, H. Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controlla- bility. In NeurIPS, 2024.
Ju, X., Gao, Y., Zhang, Z., Yuan, Z., Wang, X., Zeng, A.. Xiong, Y., Xu, Q., and Shan, Y. MiraData: A Large- Scale Video Dataset with Long Durations and Structured Captions. In NeurlPS Datasets and Benchmarks, 2024.
Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzyn- ska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. The "Something Something" Video Database for Learning and Evaluating Visual Common Sense. In ICCV, 2017.
Kaiser, L., Babaeizadeh, M., Milos. P., Osinski, B., Camp- bell, R. H., Czechowski, K., Erhan, D., Finn, C., Koza- kowski, P., Levine, S., et al. Model-Based Reinforcement Learning for Atari. In ICLR, 2020.
Kannan, H. Hafner, D., Finn, C., and Erhan, D. RoboDesk: A Multi-Task Reinforcement Learning Benchmark. https: /ttp:s////ttuuu..////// google- rresearch/robodede 2021.
Grauman, K., Westbury, A., Bymee E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. Ego4D: Around the World in 3,000 Hours of Egocentric Video. In CVPR, 2022.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the Design Space of Diffusion-Based Generative Models. NewrlPS, 2022.
Ha, D. and Schmidhuber, J. Recurrent World Models Facili- tate Policy Evolution. In NewrIPS, 2018
Hafner, D.,, Pasukonis, J., Ba, J., and Lillicrap, T. Mastering Diverse Domains through World Models. arXiv preprint MrXiv:2301.04100 2023.
Kazemi, N., Savov, N., Paudel, D., and Van Gool, L. Learning Generative Interactive Environments by Trained Agent Exploration. In NeurlPS Workshops, 2024.

---

Learning Adaptable World Models with Latent Actions
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. OpenVLA: An Open-Source Vision- Language-Action Model. In CoRL, 2024.
Menapace, W., Lathuiliere, S., Tulyakov, S., Siarohin, A., and Ricei, E. Playable Video Generation. In CVPR, 2021.
Menapace, W., Lathuilière, S., Siarohin, A., Theobalt, C.,
Tulyakov, S., Golyanik, V., and Ricci, E. Playable En- vironments: Video Manipulation in Space and Time. In CVPR, 2022.
Kim, S. W., Zhou, Y., Philion, J., Torralba, A., and Fidler, S. Learning to Simulate Dynamic Environments with GameGAN. In CVPR, 2020.
Mendonca, R., Bahl, S., and Pathak, D. Structured World Models from Human Videos. In RSS, 2023.
Kim, S. W., Philion, J.. Torralba, A., and Fidler, Drive- GAN: Towards Controllable High-Quality Neural Sim- ulation. In CVPR, 2021.
Micheli, V., Alonso, E., and Fleuret, F. Transformers are Sample-Effice Efficient World Models. In ICLR, 2023.
Kingma, D. P. K. and Welling, M. Auto-Encoding Varia- tional Bayes, In ICLR, 2014.
Nagabandi, A., Konolige, K., Levine, S., and Kumar, V. Deep Dynamies Models for Learning Dexterous Manipu- lation. In CoRL, 2020.
Ko, P.-C., Mao, J., Du, Y., Sun, S.-H., and Tenenbaum, J. B. Learning to Act from Actionless Videos through Dense Correspondences. In ICLR, 2024.
Nichol, A., Pfau, V., Hesse, C., Klimov, O., and Schulman, J. Gotta Learn Fast: A New Benchmark for Generalization in RL. arXiv preprint arXiv: 037220, 2018
Kong, W. Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong. J., Li, X., Wu, B., Zhang. J., et al. Hunyuan Video: A Systematic Framework For Large Video Generative Models. arXiv preprint arXiv:2412.00 2.36603 2024.
Nikulin, A., Zisman, I., Tarasov, D., Lyubaykin, N., Pol- ubarov, A., Kiselev, I., and Kurenkov, V. Latent Action Learning Requires Supervision in the Presence of Dis- tractors. arXiv preprint arXiv:2502.00377 2025.
Lee, K.-H., Nachum, O., Yang, M. S., Lee, L., Free- man, D., Guadarrama, S., Fischer, I., Xu, W., Jang, E., Michalewski, H., al. Multi-Game Decision Transform- ers. In NeurlPS, 2022.
O'Neill, A., Rehman, A., Gupta, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Man- dlekar, A., et al. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. In ICRA, 2024.
Ling, P., Bu, J., Zhang, P., Dong, X., Zang, Y., Wu, T., Chen, H., Wang, J., and Jin, Y. MotionClone: Training-Fri Motion Cloning for Controllable Video Generation. In ICLR, 2025.
Pearce, T., Rashid, T., Bignell, D., Georgescu, R., Devlin, S., and Hofmann, K. Scaling Laws for Pre- Pre-Tnain Agents and World Models. arXiv preprint ar.44:2411.04434, 2024.
Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., and Stone, P. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. In NeurIPS Datasets and Benchmarks, 2023.
Peebles, W. and Xie, S. Scalable Diffusion Models with Transformers. In ICCV, 2023.
Loshchilov, I. and Hutter, F. Decoupled Weight Decay Regularization. In ICLR, 2019.
Poggio, T. and Bizzi, E. Generalization in Vision and Motor Control. Nature, 2004.
Lu, T., Shu, T., Xiao, J., Ye, L., Wang, J., Peng, C., Wei, C., Khashabi, D., Chellappa, R., Yuille, A., et al. GenEx: Generating an Explorable World. arXiv preprint arXiv:2412.09624, 2024.
Qi, H., Yin, H., Du, Y., and Yang, H. Strengthening Genera- tive Robot Policies through Predictive World Modeling. arXiv preprint arXiv:250..0062. 2025.
Lu, T., Shu, T., Yuille, A., Khashabi, D., and Chen, J. Gen- erative World Explorer. In ICLR, 2025.
Raad, M. A., Ahuja, A., Barros, C., Besse, F., Bolt, A., Bolton, A., Brownfield. B., Buttimore. G., Cant, M., Chakera, S., et al. Scaling Instructable Agents Across Many Simulated Worlds. arXiv preprint M.Xiv:2404.1019, 2024.
Mazzaglia, P., Verbelen, T., Dhoedt, B., Courville, A., and World Models for Generalization in Embodied Agents. NeurIPS, 2024.
Rajeswar, S. GenRL:
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., et al. A Generalist Agent. TMLRR 2022.
McInnes, L., Healy, J., and Melville, J. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv preprint arXiv: / metp://022.0000 2018
12

---

Learning Adaptable World Models with Latent Actions
Ren, Z., Wei, Y., Guo, X., Zhao, Y., Kang, B., Feng, J., and Jin, X. VideoWorld: Exploring Knowledge Learning from Unlabeled Videos. In CVPR, 2025.
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards Accurate Gen- erative Models of Video: A New Metric & Challenges. arXiv preprint arXiv: arXiv:1812.01717, 2018.
Rigter, M., Gupta, T., Hilmkil, A., and Ma, C. AVID: Adapt- ing Video Diffusion Models to World Models. arXiv preprint arXiv:2410. /2822, 2024.
Valevski, D., Leviathan, Y., Arar, M., and Fruchter, S. Dif- fusion Models are Real-Time Game Engines. In ICLR, 2025.
Rizzolatti, G., Fadiga, L., Gallese, V., and Fogassi, L. Pre- motor Cortex and the Recognition of Motor Actions. Cog- nitive Brain Research, 1996.
Van Den Oord, A., Vinyals, O., et al. Neural Discrete Representation Learning. In NeurIPS, 2017.
Romo, R., Hernández, A., and Zainos, A. Neuronal Corre- lates of a Perceptual Decision in Ventral Premotor Cortex. Neuron, 2004.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, t.,, and Polosukhin, I. Attention is All You Need. In NeurlPS, 2017.
Ruhe, D., Heek, J.. Salimans, T., and Hoogeboom, E. Rolling Diffusion Models. In ICML, 2024.
Villar-Corrales A. and Behnke, S. PlaySlot: Learn- ing Inverse Latent Dynamics for Controllable Object- Centric Video Prediction and Planning. arXiv preprint arXiv:2502.07600 2025.
Rybkin, O., Pertsch, K., Derpanis, K. G., Daniilidis, K., and Jaegle, A. Learning What You Can Do before Doing Anything. In ICLR, 2019.
Wang, I.,, Zhao, K., Liu, C., and Chen, X. Learn- ing Real- Woal-World Action-Vider Dynamics with Het- erogeneous Masked Autoregression. arXiv preprint arXiv:2502.04296, 2025.
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu. J., Koltun, V., Malik, J.. et al. Habitat: A Platform for Embodied Al Research. In ICCV, 2019.
Wang, X., Zhu, 7., Huang, G., Chen, X., Zhu, J., and Lu, J. DriveDreamer: Towards Real-World-Drine World Models for Autonomous Driving. In ECCV, 2024.
Schmeckpeper, K., Xie, A., Rybkin, O., Tian, S., Daniilidis, K., Levine, S., and Finn, C. Learning Predictive Models From Observation and Interaction. In ECCV, 2020.
Watter, M., Springenberg, J., Boedecker, J., and Riedmiller, M. Embed Control: A Locally Linear Latent Dynamics Model for Control from Raw Images. In NeurIPS, 2015.
Schmidt, D. and Jiang, M. Learning to Act without Actions. In ICLR, 2024.
Willi, T., Jackson, M. T., and Foerster, J. N. Jafar: An Open-Source Genie Reimplemention in JAX. In ICML Workshops, 2024.
Seo, Y., Lee, K., James, S. L., and Abbeel, P. Reinforcement Learning with Action-Free Pre- Teo-rrinnng from Videos. In ICML., 2022.
Williams, G., Drews, P., Goldfain, B., Rehg, M., and Theodorou, F. A. Aggressive Driving with Model Predic- tive Path Integral Control. In ICRA, 2016.
Seo, Y., Hafner, D., Liu, H., Liu, F., James, S., Lee, K., and Abbeel, P. Masked World Models for Visual Control. In CoRL, 2023.
Wu, J., Ma, H., Deng, C., and Long, M. Pre-Training Contextualized World Models with in-the-Wild Videos for Reinforcement Learning. NeurIPS, 2023.
Su, J., Ahmed, M., Lu, Y., Pan, S. Bo, W., and Liu, Y. RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing, 2024.
Wu, J.. Yin, S., Feng, N., He. X., Li, D., Hao, J., and Long, M. iVideoGPT: Interactive VideoGPT's are Scal- able World Models. In NewrlPS, 2024.
Sun, Y., Zhou, H., Yuan, L., Sun, J. J., Li, Y., Jia, X., Adam, H. Hariharan, B., Zhao, L., and Liu, T. Video Creation by Demonstration. arXiv preprint 1xX5552412.09551, 2024.
Wu, P., Escontrela, A., Hafner, D., Abbeel, P., and Gold- berg, K. DayDreamer: World Models for Physical Robot Learning. In CoRL, 2022.
Sutton, R. S. Dyna, an Integrated Architecture for Learning, Planning, and Reacting, ACM Sigart Bulletin, 1991.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. MIT Press, 2018
Wu, Y., Tian, R., Swamy, G., and Bajcsy, A. From Fore- sight to Forethought: VLM-In-the-Logg Policy Steering via Latent Alignment, arXiv preprint arXiv:2502.01825 2025.
Tian, S., Finn, C., and Wu, J. A Control-Centric Benchmark for Video Prediction. In /CLR, 2023.
13

---

Learning Adaptable World Models with Latent Actions
Xiang, J., Liu, G., Gu, Y., Gao, Q.. Ning, Y., Zha, Y., Feng, Z., Tao, T., Hao, S., Shi, Y., al. Pandora: Towards General World Model with Natural Language Actions and Video States. arXiv preprint 2xXi::2400.09455.. 2024.
Zhang, J., Zhu, R., and Ohn-Bar, E. SelfD: Self-Learning Large-Scale Driving Policies From the Web. In CVPR, 2022.
Zhang, L.,, Kan, M., Shan, S., and Chen, X. PreLAR: World Model Pre- Pre-Training with Learnable Action Representa- tion. In ECCV, 2024.
Xu, H., Zhang, J., Cai, J., Rezatofighi, H., Yu, F,, Tao, D., and Geiger, A. Unifying Flow, Stereo and Depth Estimation. JEEE TPAMI, 2023a.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, The Unreasonable Effectiveness of Deep Features as Perceptual Metric. In CVPR, 2018.
Xu, M., Xu, Z., Chi, C., Veloso, M., and Song. S. XSkill: Cross Embodiment Skill Discovery. In CoRL, 2023b.
Yang. J., Gao, S., Qiu, Y., Chen, L., Li, T., Dai, B., Chitta, K., Wu, P., Zeng, J., Luo, P., al. Generalized Predictive Model for Autonomous Driving. In CVPR, 2024a.
Zhen, H., Qiu, X., Chen, P., Yang. J., Yan, X., Du, Y., Hong, Y., and Gan, C. 3D-VLA: 3D AssionLanguagge-Acte Generative World Model. ICML, 2024.
Yang. M., Du, Y., Dai, B., Schuurmans, D., Tenenbaum, B., and Abbeel, P. Probabilistic Adaptation of Text-to- Video Models. In ICLR, 2024b.
Zhou, G., Pan, H., LeCun, Y., and Pinto, L. DINO-WMF World Models on Pre-Trained Visual Features enable Zero-Shot Planning. arXiv preprint arXiv:241 014411.0498.. 2024a.
Yang, M., Du, Y., Ghasemipour, K., Tompson, J., Schuur- mans, D., and Abbeel, P. Learning Interactive Real-World Simulators. In ICLR, 2024c.
Zhou, S., Du, Y., Chen, J., Li, Y., Yeung, D.-Y,, and Gan, C. RoboDreamer: Learning Compositional World Models for Rohot Imagination. In ICMI., 2024b.
Yang. M., Li, J., Fang. Z., Chen, S., Yu, Y., Fu, Q. Yang, W.. and Ye, D. Playable Game Generation. arXiv preprint arXiv:2412.00887, 2024d.
Zhu, F., Wu, H., Guo, S., Liu, Y., Cheang, C., and Kong, T. IRASim: Learning Interactive Real-Robot Action Simulators. arXiv preprint arXiv:2406. 145.00 2024.
Yang. S., Walker, J., Parker- Holder, J., Du Y., Bruce, J., Barreto, A., Abbeel, P., and Schuurmans, D. Video as the New Language for World Decision Making. In ICMI., 2024e.
Zhu, Y., Wong, J., Mandlekar, A., Martin-Martin, R., Joshi, A., Nasiriany, S., and Zhu, Y. robosuite: A Modular Sim- ulation Framework and Benchmark for Robot Learning. arXiv preprint arXiv: 12293, 2020.
Yang. Z., Teng. J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang. Y., Hong, W., Zhang, X., Feng, G., et al. CogVideoX: Text-to-Vide Video Diffusion Models with An Expert Transformer. In ICLR, 2025.
Yatim, D., Fridman, R., Bar-Tal, O., Kasten, Y.. and Dekel, T. Space-Time Diffusion Features for Zero-Shot Text- Driven Motion Transfer. In CVPR, 2024.
Ye, S., Jang, J., Jeon, B., Joo, S., Yang, J., Peng, B., Man- dlekar, A., Tan, R., Chao, Y.- Lin, B. Y., et al. Latent Action Pretraining from Videos. ICLR, 2025.
Ye. W. Zhang, Y., Abbecl, P., and Gao, Y. Become Proficient Player with Limited Data through Watching Pure Videos. In ICLR, 2023.
Yin, T., Zhang, Q.. Zhang, R., Freeman, W. T., Durand, F,, Shechtman, and Huang, X. From Slow Bidirectional to Fast Causal Video Generators. In CVPR, 2025.
Yu, J., Qin, Y., Wang, X., Wan, P., Zhang, D., and Liu, X. GameFactory: Creating New Games with Generative Interactive Videos. arXiv preprint arXiv:2501.0832. 2025.
14

---

Learning Adaptable World Models with Latent Actions
A. Datasets
A.1. Data Collection and Generation
Our training data is primarily sourced from four publicly accessible datasets (Goyal et al., 2017; Grauman et al., 2022; O'Neill et al., 2024; Ju al., 2024) that represent various actions in typical interactive scenarios. For the videos from Open X-Bmbodiment (O'Neill al., 2024), keep both egocentric and exocentric demonstrations encompass wide spectrum of action patterns. Note that manually remove the unfavorable subsets that dominated by static, nonconsecutive, low- cowffrequency and low-resolution frames.
To further enrich the diversity of our data, we also automatically generate a massive number of transitions from two gaming platforms developed by OpenAl (Nichol et al., 2018; Cobbe et al., 2020). We manually search on the Internet and import the ROMs we found into Gym Retro (Nichol et al., 2018), resulting a total of 1000 interactive environments (see Table for the full list). For the 16 environments in Proegen (Cobbe al., 2020), we hold out 1000 start levels for evaluation and use the remaining 9000 levels for training. All samples are generated using the "hard" mode. We take a random action at each time step to collect 1M transitions for each Gym Retro environment and 10M for each Proegen environment. Unlike Genie (Bruce et al., 2024) that samples all actions uniformly in their case study, we employ biased action sampling strategy to encourage broader exploration. Specifically, we increase the probability of selecting particular action for short period and then alternate these probabilities in the subsequent period. As shown in Figure 8, this simple strategy leads to much more diverse scenes in our data. Generating data with reinforcement learning agents may further boost the scene diversity (Kazemi et al., 2024; Yang et al., 2024d; Valevski et al., 2025), which we leave for future work. In Figure 9, we visualize some representative environments in our final dataset.
A.2. Data Mixture
Since the datasets vary size and diversity, it challenging to balance them perfectly. Therefore, we simply weight all subsets according the number of videos during training. We report detailed statistics of our training data in Table 7,
Table 7. Data organization. Data sources, generation procedures, approximated frame counts, and mixture ratios for our training.
Category
Data Source
Automated
Frames
Ratios
Gym Retro (Nichol et al., 2018)
1000M
49%
2D Video Game
Proegen Benchmark (Cobbe et al.. 2020)
144M
Robot Data
Open X-Embodimnnnt (O Neileil al., 2024)
X
170M
30%
Ego4D (Grauman al., 2022)
330M
1%
Human Activity
Something SomeehhnggSomethinn (Goyal el al., 2017)
7M
3%
3D Rendering
MiraData (Ju ct al., 2024)
X
200M
14%
City Walking
MiraData (Ju al., 2024)
X
120M
1%
uniform action sampling strategy (1000 steps)
biased action sampling strategy (1000 steps)
Figure 8. Effect of our biased action sampling strategy. Compared to the uniform action sampling strategy, our biased scheme enables agents explore longer horizons.
15

---

Learning Adaptable World Models with Latent Actions
Table 8. Full list of collected Gym Retro environments. We import 1000 ROMs from the web for automated data generation scale.
Boder
B. Implementation Details
B.1. Architecture
The latent action autoencoder adopts Transformer architecture with 500M parameters. It consists of 16 encoder blocks and 16 decoder blocks using 1024 channels and 16 attention heads. The dimension of the latent actions is 32. The autoregressive world model adopts 3D UNet architecture following SVD (Blattmann et al., 2023), with 1.5B trainable parameters and memory length of 6. The default input resolution for both models is 256 x 256.
B.2. Training
The latent action autoencoder is trained for 200K steps from scratch with a batch size of 960. We employ the AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate of 2.5 x 10-5 and weight decay of 0.01. The hyperparameter 3 is set to 2 10 to achieve good balance between representation capacity and context disentangling ability.
The autoregressive world model is trained for 80K steps with batch size of 64 and a learning rate of x 10-5 on 16 NVIDIA A 100 GPUs. We adopt cosine learning rate scheduler with 10K warmup steps. To enhance prediction quality,
16

---

Learning Adaptable World Models with Latent Actions
ZSTART
Figure 9. Diversity of our training dataset. Our curated dataset aggregates an extremely wide range of scenarios.
exponential moving average is also applied during pretraining. The noise augmentation level is randomly selected from range of 0.0 to 0.7, with an interval of 0.1.
For both latent action autoencoder training and world model pretraining, we randomly jitter the brightness of input frames to augment generalization ability. For all frames with varying aspect ratios, we apply center cropping to avoid padding irrelevant content. We check the meta information of all datasets and downsample the videos to approximately a frame rate of 10 Hz for training.
B.3. Sampling
By default, we use sampling steps with a classifier-free guidance scale of 1.05 to generate new frames. We do not add noise the historical frames but condition the world model on an augmentation level of 0.1 we find this improves the results, A timestep shifting strategy (Kong et al., 2024) is also applied enhance generation quality.
B.4. Visual Planning on the Procgen Benchmark
We summarize our model predictive control process for visual planning below:
1. Given the current observation and the image of the final state, where the environment is about to restart with a positive reward, we calculate the current reward as the cosine similarity between them in RGB space. The planning objective is defined as the maximum reward obtained along the planned trajectory.
2. At each iteration, we sample a population of N action sequences, each with a length of T., from a distribution. The initial distribution is set to a uniform distribution over four actions (LEFT, DOWN, UP, RIGHT).
3. For each sampled action sequence, the world model is used to predict the resulting trajectory, and the reward is calculated for each trajectory.
4. The top K action sequences with the highest rewards are selected, and we update the distribution by increasing the sampling probabilities of these selected actions.
5. A new set of N action sequences is sampled from the updated distribution, and the process repeats for i Cross-Entropy Method iterations.
6. After i optimization iterations, the first T action in the action sequence with the highest probability is executed in the environment. The planning terminates either when the final state is achieved or when the search limit is exceeded.
In practice, we use = 2 Cross-Entropy Method iterations. For each iteration. = 100 action sequences with a length of L. are sampled, and the best K samples are selected to update the action sampling distribution. After the
17

---

Learning Adaptable World Models with Latent Actions
optimization procedure is done, the first T actions are executed in the environment. We the search limit to 20 steps. For efficiency, only denoising steps and disable classifier-free guidance during planning.
B.5. Visual Planning on the Vp Benchmark
We train low-resolution variants a resolution of 64 64 for control-centric evaluation on the Vp² benchmark. We follow the official protocol of vP² to evaluate all models. During adaptation, use 5K given trajectories for Robosuite and 35K scripted trajectories with perturbations for RoboDesk for finetuning. Each world model is adapted for IK steps with a batch size of 32 and a learning rate of 5 10-4. A cost below 0.05 is considered as success in Robosuite tabletop pushing tasks.
B.6. IVideoGPT Training Details
We resumee from the official OpenX checkpoint of iViieoGPP and do not finetune its tokenizer. The model is designed to predict 15 future frames from an initial frame. After pretraining iVideoGPT and our action-aware variant on OpenX for 27K extra steps, we finetune each model with robot actions for IK steps on BAIR robot pushing dataset. The resulting models are tested on 256 test videos.
C. Additional Results
C.1. Action Transfer
Qualitative comparison. We qualitatively compare AdaWorld trained for 50K steps with other baselines in Figure 10. The results show that our approach is able to represent nuanced actions and exhibit strong transferability across contexts.
Visualizations. We attach more action transfer results in number of environments in Figures 12 to 16. Each sample is generated by transferring a latent action sequence with a length of 20.
Failure case study. We present typical failure cases of action transfer Figure 21. AdaWorld does not always perfectly understand physics and dynamics. It also struggles produce high-quality content during long-term rollouts or dramatic view shifts.
C.2. World Model Adaptation
To demonstrate the advantages of our approach, in Figure 11, we visualize some simulation results on the held-in test set using the finetuned models in Sec. 3.2.1. While the model pretrained with action aagiosic videos struggles to faithfully execute the action inputs, model has rapidly acquired precise action controllability using the same number of steps. This underscores that our approach could serve as a better initialization method for world model adaptation compared to conventional methods.
C.3. Action Creation through Clustering
As mentioned in Sec. 2.3, AdaWorld can also easily create flexible number of control options through latent action clustering. Specifically, we process our Procgen and Gym Retro training set using the latent action encoder to obtain the corresponding latent actions. To generate different control options, we apply K-means clustering to all latent actions, setting the number of clustering centers the desired number of control options. To examine the controllability of varying actions derived with World, we adopt the APSNR metric following Genie (Bruce et al., 2024). Table 9 shows the APSNR of the latent action decoder predictions. The larger the APSNR, the more the predictions are affected by the action conditions and therefore the world model is more controllable. The results in Table demonstrate that the control options derived with Ada World represent distinct meanings and exhibit comparable controllability to the discrete counterpart, while the latter does not support a customizable number of actions, as it is fixed once trained.
Table Action controllability evaluation. AorWorld supports customizing different numbers of actions with strong controllability.
APSNR
Method
7
10
Discrete cond.
N/A
N/A
N/A
N/A
6.47
N/A
N/A
N/A
N/A
AdaWorld
5.67
5.15
7.28
8.23
6.26
7.32
6.07
6.68
6.53
18

---

Learning Adaptable World Models with Latent Actions
D. Something-Something v2 Categories for Action Transfer
In Sec. 3.1, we utilize the top-10 most frequently appearing categories from SSv2 for action transfer evaluation, including "Putting [something] on surface", "Moving [something] up". "Pushing down", "Covering [something] with [something]". "Pushing [something] from right left", "Uncovering [something]". "Taking [one of many similar things on the table]", "Throwing [something]", and "Putting [something] into [something]".
"Pusunug [something] from left right", "Moving [something]
E. Selected Scenes for Visual Planning
To guarantee an effective evaluation, we conduct an exhaustive search using the ground truth environments to identify viable Proegen test scenes that can be completed in an acceptable number of steps by goal image matching. This results in a total of 120 scenes for our visual planning experiments in Sec. 3.2.2. In Figures 17 20, we illustrate the initial states of all selected scenes. The scenes from survival-oriented environments (e.g.. Bighish and BossFight) are not considered in our evaluation.
F. Related Work
F. World Models
Driven by advances in deep generative models (Peebles & Xie, 2023; Yang et al., 2025), world models have made encouraging
strides recently. They aim to replicate the transitions of the world through generation (Sutton, 1991; Ha & Schmidhuber, 2018; Yang et al., 2024c; Lu et al., 2024; 2025; Agarwal et al., 2025), enabling agents to perform decision making entirely in imagination even when confronted with unseen situations (Kaiser et al., 2020; Micheli et al., 2023; Hafner et al., 2023;
Du et al., 2023; Mazzaglia al., 2024). This capability has proven beneficial for gaming agents (Kim et al., 2020; Alonso et al., 2024; Pearce et al., 2024; He et al., 2025), autonomous vehicles (Kim et al., 2021; Hu et al., 2023; Yang al., 2024a;
Gao et al., 2024; Wang et al., 2024; Bar et al., 2025; Hassan et al., 2025), and real robots (Seo et al., 2023; Wu et al., 2022;
Ko et al., 2024; Du et al., 2024; Zhen et al., 2024; Zhou et al., 2024b; Wu et al., 2024; Zhu et al., 2024; Zhou et al., 2024a;
Bu et al., 2024; Wang et al., 2025; Qi et al., 2025; Wu et al., 2025).
Despite the substantial benefits, existing works often suffer from costly training that requires extensive annotations to learn new actions in different environments. To mitigate this issue, some efforts have explored more efficient world modeling through pretraining from passive videos (Watter al., 2015; Seo et al., 2022; Mendonca et al., 2023; Wu al., 2023). However, these methods primarily focus obtaining compact world representations rather than learning control interfaces that can be readily adapted with limited interactions and computations. Another family of research investigates efficient online adaptation (Yang et al., 2024b; Rigter et al., 2024; Hong et al., 2025), but they primarily focus on parameter efficiency by assuming that the parameters of the pretrained video models are frozen or inaccessible.
While a recent study (Bruce et al., 2024) has explored learning interactive environments from 2D gameplay videos. is limited to 8 fixed actions, which constrains its ability express and adapt to more fine-grained actions. In contrast, we advocate for using continuous latent action space, maximizing flexibility for efficient action transfer and enabling several unique applications for adaptable world modeling.
E.2 Latent Action from Videos
Humans can comprehend the notion of various actions by observing others to behave (Rizzolatti et al., 1996; Rybkin et al., 2019; Schmeckpeper et al., 2020; Yatim et al., 2024; Ling et al., 2025). To develop intelligent agents, it is intriguing to automatically extract similar action primitives from videos. Some approaches align human and robot manipulation videos in a shared space to derive meaningful action prototypes (Xu et al., 2023b). However, they require collecting semantically paired videos, which limits their scalability for most real-world tasks.
To avoid reliance on video pairs, some works extract implicit latent actions through fully unsupervised learning (Edwards et al., 2019; Menapace et al., 2021: 2022; Ye et al., 2023; Schmidt & Jiang, 2024; Zhang et al., 2024; Sun et al., 2024; Villar- Corrales& & Behnke, 2025). However, they mainly focus on a single environment with limited complexity. While more recent works extend similar approaches to multiple datasets (Cui et al., 2024; Chen et al., 2024b;c; Ye et al., 2025; Bu et al., 2025; Ren et al., 2025), they mainly leverage latent actions as the objectives for behavior cloning. Consequently, they always adopt discrete actions to fit the policy networks (Kim et al., 2024), leading to significant ambiguity due to discretization (Nikulin et al., 2025). Thus, the potential of latent actions for adaptable world modeling remains underexplored.
19

---

Learning Adaptable World Models with Latent Actions
source video
Act-agnostic
Rlow cond.
Discrete cond.
AdaWorld
Figure Qualitative comparison of action transferability. World can accurately identify the demonstrated action and precisely transfer it another context, while the other baselines fall short in doing so.
ground truth
Act-agnostic
AdaWorid
Figure Minecraft adaptation results after finetuning 800 steps with 100 samples for each action. Our approach efficiently achieves precise action controllability with minimal action-labeled data and finetuning, while the action-agnostic pretraining baseline fails to perform the correct actions.
20

---

Learning Adaptable World Models with Latent Actions
source
target
source
HALLS
target
source
target
C
source
target
source
target
Figure 12, Additional action transfer results by AdaWorld. Best viewed with zoom-in.
21

---

Learning Adaptable World Models with Latent Actions
source
target
source
target
source
target
source
target
source
target
Figure 13, Additional action transfer results by AdaWorld, Best viewed with zoom-in.
22

---

Learning Adaptable World Models with Latent Actions
source
target
Cuki
Cuki
Coca
Cuki
source
target
source
target
source
target
source
target
Figure 14, Additional action transfer results by AdaWorld. Best viewed with zoom-in. in.
23

---

Learning Adaptable World Models with Latent Actions
source
target
source
target
source
target
source
target
source
target
Figure 15. Additional action transfer results by AdaWorld. Best viewed with zoom-in.
24

---

Learning Adaptable World Models with Latent Actions
source
target
source
target
source
target
source
target
source
target
Figure 16. Additional action transfer results by AdaWorld. Best viewed with zoom-in.
25

---

Learning Adaptable World Models with Latent Actions
PE
TE
E.
iE
51
C
FF
H
LE
24
ry
Figure 17. Heist scenes for planning evaluation. The figure illustrates the initial states of all selected scenes.
0000
0000
20000 000000000000
0000
0000
Figure 18. Jumper scenes for planning evaluation. The figure illustrates the initial states of all selected scenes.
26

---

Learning Adaptable World Models with Latent Actions
2
Figure 19. Maze scenes for planning evaluation. The figure illustrates the initial states of all selected scenes.
Figure 20. CaveFlyer scenes for planning evaluation. The figure illustrates the initial states of all selected scenes.
27

---

Learning Adaptable World Models with Latent Actions
target
source
target
source
target
source
target
Figure 21. Examples of failure cases. AdaWorld is by no means perfect in simulating real-world physics. dynamic agents, long-term rollouts, and significant view changes.
28


Reviews:
[{"review_id": "Bfr9DfLVls", "reviewer": "Reviewer_kWaF", "summary": "This paper introduces AdaWorld, a world model learning approach that leverages self-supervised latent action extraction from videos to capture key transitions. It also introduces an autoregressive world model (single-frame SVD) conditioned on these latent actions and historical frames, enabling transfer and learning of new actions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful feedback. We answer each question below and will include all results in the revision.\n\n> The model requires a reference video to guide actions. Clarify how the model can achieve similar controllability to Genie or OASIS at inference time.\n\n**R1-1**: Our world model does not necessitate a reference video for prediction. As shown in Sec. 3.2, our model can be efficiently adapted to various action inputs. After adaptation, our model can be directly controlled by raw actions (e.g., Minecraft actions like OASIS) without using reference videos. Moreover, as mentioned in Sec. 2.3, one can easily obtain a customizable number of control options by clustering latent actions from videos.\n\n> A user study may be necessary to evaluate the effectiveness of action/motion transfer.\n\n**R1-2**: We conduct a user study with the baselines as requested. We follow the same setup in Sec 3.1 and generate 50 video pairs on LIBERO and SSv2, respectively. We then invite four volunteers to judge whether the action is successfully transferred. The success rates are reported below:\n\n| | LIBERO | SSv2 |\n| :--- | :---: | :---: |\n| Act-agnostic | 0% | 1% |\n| Flow | 2% | 10.5% |\n| Discrete | 3.5% | 21.5% |\n| AdaWorld | **70.5%** | **61.5%** |\n\n\nThe results indicate that our interface can transfer actions more effectively, especially on LIBERO, where the robot actions are more nuanced.\n\n> Include motion transfer approaches for comparison.\n\n**R1-3**: Thanks for the suggestion. First, we want to emphasize that our paper aims to develop a highly adaptable interface for world models. In contrast, motion transfer methods mainly rely on transferring video-level feature maps, which are not applicable for interactive control and are not suitable for action adaptation. It is also notable that our action transfer ability does not require strictly aligned spatial structures, which is much more flexible than common motion transfer settings (see **R1-7** below).\n\nDue to time limit, we compare the suggested MotionClone using 32 official demos released on its GitHub page. To avoid potential bias, we crop them to square and ensure that the text at the top is excluded. We use AdaWorld to autoregressively predict 16 frames and resize them to MotionClone’s resolution. Since ground truth target videos are not available, we invite four volunteers for a user study. As a result, AdaWorld is preferred 21.09% of time in action transfer accuracy, showing our ability to perform motion transfer tasks (though optimizing video-level motion transfer is not our main purpose).\n\n> Whether the latent action autoencoder effectively captures important actions. During training, smooth transitions between frames dominate the sequence, whereas sudden transitions constitute only a small fraction of the total frames. Whether smooth transitions should be considered \"actions\" or merely \"motion\".\n\n**R1-4**: We add more action transfer results to show the capability of our latent action autoencoder [[LINK](https://icml2025-1014.github.io)]. Without any special handling of training data, our model can effectively capture and transfer sudden transitions. Rebalancing the training data may further enhance the learning of these transitions. Note that the latent action autoencoder is encouraged to encode the most critical actions due to the information bottleneck, while minor motions and background changes that can be predicted by the decoder are likely not to be encoded.\n\n> Missing motion transfer references.\n\n**R1-5**: We have included the two references in the related work and will update them accordingly in the revision.\n\n> The input frame in Figure 3.\n\n**R1-6**: Sorry for the confusion. Figure 3 illustrates our training process, where $f_{t+1}$ is the frame to be denoised and is used to generate latent actions. During inference, $f_{t+1}$ is not used and the predictions are made based on past frames. The control interface can take latent actions transferred from other videos, selected from a known set, or raw actions after efficient adaptation. We will clarify this in the revision.\n\n> Can the method still function effectively when the source and target videos are misaligned due to differences in camera poses or character locations?\n\n**R1-7**: Unlike typical motion transfer settings, our method does not require strong spatial alignment to capture actions. As shown in this [[LINK](https://icml2025-1014.github.io)], our latent actions can effectively recognize and transfer actions from various poses and embodiments. Moreover, we want to clarify that the main objective of this work is to develop a generally adaptable world model rather than maximize action representation precision. Although our latent action may not faithfully represent all kinds of actions, it is general enough to serve as a unified interface for pretraining, which significantly improves the adaptability of world models compared to existing training methods. We will add more results in the revision.", "claims_and_evidence": "The paper largely supports its claim regarding action/motion representation learning. However, since it is presented as a world model, it differs from traditional world models where direct user input (e.g., via keyboard or mouse) influences predictions. Instead, this approach requires a reference video to guide actions in the generated video. It would be helpful for the authors to clarify how the model can achieve similar controllability to Genie or OASIS at inference time.", "methods_and_evaluation_criteria": "The proposed method makes sense, as learning motion representation through pretraining a latent action autoencoder and applying it to a one-frame denoising diffusion model is a reasonable approach. However, I have some concerns regarding the training data and process (see the section below).\n\nRegarding the evaluation criteria, I am skeptical about the robustness of the ECS metric based on I3D features for assessing action/motion transfer in open-domain scenarios. A more comprehensive user study may be necessary to better evaluate the effectiveness of action/motion transfer.", "theoretical_claims": "The theoretical claims appear to be valid. However, I have some concerns about whether the latent action autoencoder effectively captures important actions. Smooth transitions between frames, such as a character simply moving forward, dominate the sequence, whereas sudden transitions, like jumping or performing distinct actions, constitute only a small fraction of the total frames. Yet, during training, it seems that f_t and f_{t+1} are sampled uniformly. This raises the question of whether smooth transitions should be considered \"actions\" or merely \"motion.\"", "experimental_designs_or_analyses": "For the comparison methods, I suggest that the authors include motion transfer approaches, as referenced earlier. Also consider add user study for action transferring performance.", "relation_to_broader_scientific_literature": "This paper is related to to diffusion models, world models, and action-driven video generative models.", "essential_references_not_discussed": "A group of motion transfer methods are missing for inference:\n\nSpace-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer, CVPR 2024\nMotionClone: Training-Free Motion Cloning for Controllable Video Generation, ICLR 2025", "other_strengths_and_weaknesses": "Strengths: Modeling action or motion using latent representations is a crucial challenge in world models, and this work takes a promising direction by expanding the action vocabulary beyond traditional action-labeled approaches. The proposed method—first learning to extract latent actions and then incorporating them into diffusion models—seems well-motivated and conceptually sound.\n\nWeakness: Please refer to the sections above, as well as the \"Questions for Authors\" section, for my specific concerns and areas where further clarification is needed.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. In Figure 3, the input appears to be the next frame  f_{t+1}, but this seems to result in a reconstruction rather than a next-frame prediction. Based on the text in Line 248, the last frame in the memory is used as the condition image, which suggests it might be f_t instead?\n\n2. The current approach seems to entangle motion with structural elements, leading to a strong similarity in overall flow between the source and target videos. It would be helpful for the authors to discuss potential limitations of this, particularly in scenarios where the source and target videos are misaligned due to differences in camera poses or character locations. Can the method still function effectively under such conditions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sb4EbptsZv", "reviewer": "Reviewer_UZ2f", "summary": "This paper proposes a pretraining framework for learning world models that can generalize to various contexts.  AdaWorld first learns a latent action representation using an unsupervised forward prediction objective. Subsequently, AdaWorld learns an autoregressive world model that conditions on latent actions to produce future frames. Interestingly, AdaWorld finds that the learned latent actions are context-invariant, allowing for transfer between contexts. AdaWorld is evaluated on a variety of downstream tasks in action transfer, model adaptation, and visual planning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful feedback. We answer each question below and will include all results in the revision.\n\n> Compare to state-of-the-art baselines.\n\n**R2-1**: To demonstrate the generality of our method, we use iVideoGPT as a state-of-the-art baseline. iVideoGPT is an action-controlled world model with an autoregressive Transformer architecture. It is pretrained by action-agnostic video prediction and adds a linear projection to learn action control during finetuning. For fair comparison, we implement a variant by conditioning iVideoGPT with our latent actions during pretraining. We resume from the official OpenX checkpoint and do not finetune its tokenizer. After pretraining iVideoGPT and our action-aware variant on OpenX for 10k extra steps, we finetune each model with robot actions for 1k steps on BAIR robot pushing dataset. The result is shown below:\n\n| | PSNR &uarr; | LPIPS &darr; |\n| :--- | :---: | :---: |\n| iVideoGPT | 16.69 | 0.221 |\n| iVideoGPT + AdaWorld | **17.33** | **0.207** |\n\nIn our paper, we also compare Genie's latent action setting. The results validate our superior adaptability compared to recent arts.\n\n> Discuss LAPA and PVG. Method is technically not very novel.\n\n**R2-2**: Our key innovation is to incorporate action information into world model pretraining, which significantly enhances its adaptability. To achieve this, we extract continuous latent actions as a scalable condition for our world model. While prior works have studied latent actions, they mainly focus on imitation learning (LAPA, LAPO) and playability (PVG, Genie). As a result, they use discrete latent actions, which struggle to express various actions and fall short in adaptation. As mentioned in Sec. 2.3., our continuous design also enables several unique applications.\n\n> How does the latent action model avoid collapse?\n\n**R2-3**: Our model is less likely to collapse for three main reasons. (1) A large portion of our data is collected randomly. Thus, our model must learn from latent actions. Otherwise, it has no way to predict the next step. (2) Our data contains thousands of environments, making learning a shared action space easier than remembering all behaviors as a decoder policy. (3) The parameter $\\beta$ is adjusted to allow sufficient information to pass the bottleneck, ensuring that the latent actions capture meaningful information.\n\n> In Figure 4, what is meant by context-invariant?\n\n**R2-4**: The latent action sequence from the source video is directly applied to the target scene for autoregressive prediction, where the same actions are replicated even when context is totally different.\n\n> Why optical flow as a condition is not as powerful as image reconstruction?\n\n**R2-5**: The dense and uniform optical flow is highly sensitive to spatial and structural misalignment. In contrast, our latent action can adaptively allocate its capacity to represent the most critical actions, making it more robust in recognizing misaligned actions [[LINK](https://icml2025-1014.github.io)].\n\n> Does the average embedding also work in continuous control environments?\n\n**R2-6**: While the average embedding is not directly applicable, AdaWorld still exhibits strong adaptability for continuous action spaces. To verify this, we use nuScenes, an autonomous driving dataset where the vehicle takes continuous displacements at each timestep, as a typical example. During adaptation, we add a two-layer MLP to map actions to the latent action interface. The interface can also be efficiently initialized by finetuning the MLP with minimal action-latent action pairs (3k steps take less than 30 seconds on a single GPU). The result is shown below:\n\n| | PSNR &uarr; | LPIPS &darr; |\n| :--- | :---: | :---: |\n| Act-agnostic | 20.86 | 0.475 |\n| Flow | 20.94 | 0.462 |\n| Discrete | 21.28 | 0.450 |\n| AdaWorld | **21.60** | **0.436** |\n\nWe also plot the PSNR curves [[LINK](https://icml2025-1014.github.io)], where AdaWorld adapts more rapidly in all cases. **R2-1** also shows that our method works for another world model with continuous control, while **R4-4** shows that AdaWorld enables better planning for robot arms.\n\n> Pretraining data for main experiments.\n\n**R2-7**: Except for Table 4, all models are pretrained on the data mixture specified in Appendix A.2, including the full OpenX and Retro datasets.\n\n> What part ensures that the latent actions are clustered semantically? Is it the beta-VAE term that controls the amount of disentanglement? Is it only an application in the discrete domain?\n\n**R2-8**: The semantic clustering and disentangling ability result from the low dimensionality of latent actions and the regularization on posterior distributions. The parameter $\\beta$ controls the disentanglement of latents [[LINK](https://icml2025-1014.github.io)] (see **R4-1**). This ability also holds for continuous actions. As [[LINK](https://icml2025-1014.github.io)] shows, continuous actions can be effectively disentangled and transferred across contexts.", "claims_and_evidence": "Yes, the paper claims that the learned world model can generalize to new, unseen contexts. The experiments show that by applying latent actions from one demonstrations, AdaWorld can generalize the same behavior to generate behavior for a unseen scene conditioned on just a single frame. There are also experiments demonstrating AdaWorld's capabilities on visual planning and adaption. However, these experiments are not as comprehensive and could use additional comparisons to state-of-the-art baselines.", "methods_and_evaluation_criteria": "Yes, the proposed method is sound and the evaluation is based on standard metrics used in the literature.", "theoretical_claims": "There are no theoretical claims and proofs.", "experimental_designs_or_analyses": "To evaluate their approach, they measured AdaWorld's ability to generalize to new contexts on a single demonstration. The paired data between LIBERO and Something Something v2 is reasonable. Also the evaluation environments used are quite common in the literature.", "relation_to_broader_scientific_literature": "The proposed work is relevant to literature in latent action models and learning from actionless videos including LAPO and Genie among others. The proposed method of learning a latent action space to guide world model generation is sound and a good idea.", "essential_references_not_discussed": "[1] Ye, Seonghyeon, et al. \"Latent action pretraining from videos.\" arXiv preprint arXiv:2410.11758 (2024).\n[2] Menapace, Willi, et al. \"Playable video generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "other_strengths_and_weaknesses": "Strengths:\n- Paper is well written and easy to follow\n- Implementation details and framework is described thoroughly\n- AdaWorld has diverse set of applications in task transfer and visual planning\n\nWeakness:\n- Method is technically not very novel. Prior work has already proposed latent action models in LAPO and Genie and similarly there is a plethora of work on generative video models\n- No state-of-the-art baselines. Baseline methods are mostly ablations or modifications to the approach (e.g. predicting optical flow as pretraining objective) so the comparison is quite weak\n- It would be nice to incorporate some baselines on controllable video generation to highlight that AdaWorld is more adaptable than prior works as a result of the learned latent action space.\n- The data diversity experiments seem a bit orthogonal to the purpose of the paper and is not really an ablation. The action-agnostic video pretraining is more of an ablation than an actual baseline.", "other_comments_or_suggestions": "See weaknesses and questions section.", "questions_for_authors": "Question:\n- How does the latent action model avoid collapse, e.g. how do you ensure that the decoder or the forward dynamics model in this case does not just pass f_t through and learn a policy on the demonstration data?\n- In Figure 4, what is meant by context-invariant? Does that mean the same latent actions are applied to autoregressively predict the future frames across each of these environments?\n- Why do you think optical flow as a condition is not as powerful as image reconstruction?\n- Does the average embedding also work in continuous control environments? The highlight example seems to be only in the Procgen discrete action space setting.\n- What is the pretraining data for the main set of experiments? Do those use the full OpenX and Retro datasets for pretraining the latent actions?\n- What part of the model ensures that the learned latent actions are clustered semantically? Is it the beta-VAE term that controls the amount of disentanglement in thel latent space? Again, is this only application in the discrete domain? Is there any evidence that this holds in continuous environments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ziPJANRrEA", "reviewer": "Reviewer_vV4B", "summary": "The authors propose a method to incorporate latent actions into the pre-training stage of World-models allowing for more efficient adaptation to downstream tasks. The authors curate a dataset spanning from ego perspectives and third-person views to virtual games and real-world environments. Finally, they evaluate their model on action transfer, model adaptation and visual planning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the helpful feedback. We answer each question below and will include all results in the revision.\n\n> The authors mention using two action tokens $a_{t:t+1}$ during the latent action autoencoding but they mention that they use $a_{t+1}$ to approximate the posterior. What do the authors do with $a_{t}$? Also the authors mention that in the action-aware pretraining, the next frame is predicted based on a sequence of the latent actions. Which latent action $a_{t+1}$ or $a_{t}$? These details need to be carefully described and mathematically formulated instead of just via text.\n\n**R3-1**: Sorry for the confusion. The $a_{t}$ ensures that the total number of tokens is aligned between $t$ and $t+1$, simplifying the implementation of spatiotemporal attention. Let $f_{t}$ and $f_{t+1}$ represent image tokens and [;] denote concatenation operation. The spatial attention can be formulated as:\n\n[$f_{t}$’;$a_{t}$’] = SpatialAttn([$f_{t}$;$a_{t}$]),\n\n[$f_{t+1}$’;$a_{t+1}$’] = SpatialAttn([$f_{t+1}$;$a_{t+1}$]),\n\nand the temporal attention can be formulated as:\n\n[$f_{t}$’;$f_{t+1}$’] = TemporalAttn([$f_{t}$;$f_{t+1}$]),\n\n[$a_{t}$’;$a_{t+1}$’] = TemporalAttn([$a_{t}$;$a_{t+1}$]).\n\nAfter encoding, all tokens except for the $a_{t+1}$ will be discarded, and we only project $a_{t+1}$ to estimate the posterior. Formally,\n\n[$\\mu$;$\\sigma$] = FC($a_{t+1}$).\n\nThus, for two consecutive frames $f_{t:t+1}$ in the video sequence, the corresponding latent action is predicted from $a_{t+1}$. We will revise this part accordingly and open-source all code for reproducibility.\n\n> In line 317-319 authors say that \"We randomly initialize action embeddings for the action-agnostic video pretraining baseline.\" However in lines 282-285 the authors say \"pretrain a world model that shares the same architecture as AdaWorld but does not take latent actions as conditions.\" These two statements seem inconsistent.\n\n**R3-2**: Our latent action is concatenated with the timestep embedding and CLIP image embedding in the original SVD, which is equivalent to adding a linear projection layer to these two layers. For the action-agnostic baseline, we use the same architecture but input zeros into the additional linear projection during pretraining. When adapting to the discrete action space, the inputs are chosen from an action codebook. We utilize the averaged latent actions to initialize the embeddings of this codebook. Since the latent actions are not learned in the action-agnostic baseline, we initialize its action codebook with random parameters. We will clarify this in the revision.", "claims_and_evidence": "Yes the authors' claims are evidence backed. The action aware pre-training does show promising results in action transfer, model adaptation and visual planning.", "methods_and_evaluation_criteria": "The authors mention using two action tokens  $a_{t:t+1}$  during the latent action autoencoding but they mention that they use $a_{t+1}$ to approx the posterior. What do the authors do with $a_{t}$ ? Also the authors mention that in the action aware pretraining, the next frame is predicted based on a sequence of the latent actions. Which latent action $a_{t+1}$ or $a_{t}$? These details need to be carefully described and mathematically formulated instead of just via text.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "In line 317-319 authors say that \"We randomly initialize action embeddings for the action-agnostic\nvideo pretraining baseline.\" However in lines 282-285 the authors say \"pretrain a world model that shares the same architecture as AdaWorld but does not take latent actions as conditions.\" These two statements seem inconsistent.", "relation_to_broader_scientific_literature": "The proposed action aware pre-training is quite novel although encoding latent actions from videos is not.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fn7IAnJ10i", "reviewer": "Reviewer_jpmM", "summary": "This paper focuses on learning world models from general videos. Unlike previous approaches that solely rely on video-based learning, this work extracts latent actions in a self-supervised manner and leverages action information for large-scale world model pretraining. With the aid of latent actions, the model can efficiently transfer actions across different contexts and adapt to new environments with limited interactions. Comprehensive experiments are conducted across multiple environments to validate the effectiveness of AdaWorld.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback. We answer each question below and will include all results and discussions in the revision.\n\n> A visualization or feature similarity analysis of the action latents.\n\n**R4-1**: As suggested, we randomly collect 1000 samples for each action from three environments (Habitat, Minecraft, DMLab) and use UMAP projection [1] to visualize them [[LINK](https://icml2025-1014.github.io)]. The visualization shows that the same actions, even from different environments, are clustered together, which validates the context-invariant property of our latent actions. Note that noise exists because the action inputs cannot be executed in certain states (e.g., cannot go ahead when an obstacle is in front). We also compare the latent action autoencoder trained with a different hyperparameter choice in the right figure. Although a lower $\\beta$ results in more differentiable latent actions, it also reduces action overlap across environments thus sacrificing disentanglement ability.\n\n> This paper follows a similar idea to Genie, aiming to learn action latents in a self-supervised manner to improve world model learning. Additionally, the approach of learning difference information between two given frames is commonly used in the robotics field.\n\n**R4-2**: While the concept of latent action is not new, we are the first to find its usefulness in world model pretraining. Unlike existing works that mainly adopt a discrete action set for imitation learning or playability, we propose a continuous latent action space that enables more effective adaptation. As mentioned in Sec. 2.3, our design also enables several unique applications compared to prior works like Genie, such as action composition and clustering.\n\n> The learned latent actions seem quite similar to those in Genie and Genie2, primarily representing simple movements such as up, down, left, and right. However, they struggle to effectively capture complex and long-range motion representations.\n\n**R4-3**: Unlike Genie, which is limited to 8 fixed actions, we develop a continuous latent action space that can express a wide range of diverse actions. Compared to Genie’s discrete design, our model can capture and transfer more nuanced actions (Table 1). Since our model is pretrained with various kinds of actions, adapting it to a new environment is akin to matching the corresponding latent actions for the action space, which leads to better simulation quality (Table 2). We also add some action transfer results showing that our model captures complex actions [[LINK](https://icml2025-1014.github.io)].\n\nRegarding the range of actions, our current model opts for frame-level control to achieve finer granularity. In future work, we will explore extending our general training recipe to long-range settings, e.g., predicting multiple frames with one latent action.\n\n> Visual planning evaluation in robotics simulators, where a goal image is provided for generation, followed by evaluating the corresponding planning performance using a video-action model.\n\n**R4-4**: Thanks for the suggestion. We use the 100 robosuite control tasks from the VP2 benchmark [2] to evaluate the effectiveness of our method in robotic tasks. We focus on a compute-efficient setting and finetune our pretrained world model and the action-agnostic baseline for only 1k steps. The finetuned models are then used to perform goal-conditioned model predictive control following the official protocol of the VP2 benchmark. The success rates are reported below:\n\n| | robosuite |\n| :--- | :---: |\n| Act-agnostic | 14% |\n| AdaWorld | **61%** |\n\nThe results demonstrate that our action-aware pretraining significantly enhances robot planning performance with limited finetuning steps. We will experiment with more robotic environments in the revision.\n\n> How does the model perform when generating longer temporal sequences?\n\n**R4-5**: The current model can be stably controlled for about 20 frames. Generating longer sequences is likely to result in quality degradation. While this paper mainly focuses on enabling adaptable world models, we believe our study complements other progresses in this field. We will explore potential solutions for long-horizon rollouts in future work.\n\n[1] UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction\n\n[2] A Control-Centric Benchmark for Video Prediction", "claims_and_evidence": "This work claims that by learning latent actions:\n\n1. AdaWorld can directly transfer actions to different contexts.\n2. It can efficiently adapt into specialized world models with limited interactions and fine-tuning.\n\nThe authors validate these claims through experiments and visualizations. Figure 2 demonstrates action transfer to support Claim 1, while Table 1&2 showcases the model's rapid adaptation capabilities. Compared to other conditioning methods, AdaWorld achieves superior performance.", "methods_and_evaluation_criteria": "The method employs a β-VAE to learn latent actions and a conditional diffusion model for video generation. The evaluation primarily focuses on the consistency and quality of generated videos, with FVD and ECS serving as reasonable metrics.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "The paper evaluates the approach through both quantitative and qualitative experiments. Tables 1 & 2 validate the advantages of prediction under the action latent space. Overall, the experimental setup is reasonable. However, it lacks visualization or feature similarity analysis of the learned action latent representations.", "relation_to_broader_scientific_literature": "This paper follows a similar idea to Genie, aiming to learn action latents in a self-supervised manner to improve world model learning. Additionally, the approach of learning difference information between two given frames is commonly used in the robotics field.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "1. The learned latent actions seem quite similar to those in Genie and Genie2, primarily representing simple movements such as up, down, left, and right. However, they struggle to effectively capture complex and long-range motion representations.", "other_comments_or_suggestions": "1. The paper validates the benefits of the learned world model through visual planning. A more convincing approach could be testing in robotics simulators, where a goal image is provided for generation, followed by evaluating the corresponding planning performance using a video-action model.\n2. Providing an analysis of the action latents would make the findings more convincing.", "questions_for_authors": "1. From the visualizations, a decline in generation quality over time can be observed. How does the model perform when generating longer temporal sequences?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces AdaWorld, a two-stage training framework designed to learn world models. AdaWorld initially learns a latent action representation using an unsupervised forward prediction objective, and then learns an autoregressive world model that generates future frames by conditioning on the learned latent actions. AdaWorld's effectiveness is demonstrated through evaluation on various downstream tasks, including action transfer between different video contexts and visual planning.

All reviewers agreed on the quality of the paper, and that the paper's claims were well supported by the experimentation. I agree with the reviewers' consensus and therefore support acceptance.
Decision: Accept (poster)