GenRL: Multimodal-foundation world models for generalization in embodied agents
Authors: Pietro Mazzaglia
Tim Verbelen
Bart Dhoedt
Aaron Courville
Sai Rajeswar
Venue: NeurIPS 2024 poster (2024)
Keywords: world models
foundations models
reinforcement learning
multitask generalization
Primary area: reinforcement_learning
TL;DR: Connecting multimodal foundation models' representations with world models' representations for RL enables specifying task by vision or language prompts and learning the corresponding embodied behaviors in imagination.

Abstract: Learning generalist embodied agents, able to solve multitudes of tasks in different domains is a long-standing problem. Reinforcement learning (RL) is hard to scale up as it requires a complex reward design for each task. In contrast, language can specify tasks in a more natural way. Current foundation vision-language models (VLMs) generally require fine-tuning or other adaptations to be adopted in embodied contexts, due to the significant domain gap. However, the lack of multimodal data in such domains represents an obstacle to developing foundation models for embodied applications. In this work, we overcome these problems by presenting multimodal-foundation world models, able to connect and align the representation of foundation VLMs with the latent space of generative world models for RL, without any language annotations. The resulting agent learning framework, GenRL, allows one to specify tasks through vision and/or language prompts, ground them in the embodied domain’s dynamics, and learn the corresponding behaviors in imagination.
As assessed through large-scale multi-task benchmarking in locomotion and manipulation domains, GenRL enables multi-task generalization from language and visual prompts. Furthermore, by introducing a data-free policy learning strategy, our approach lays the groundwork for foundational policy learning using generative world models. 
Website, code and data: https://mazpie.github.io/genrl/

GenRL: Multimodal-foundation world models for generalization in embodied agents
Pietro Mazzaglia' IDLab, Ghent University
Tim Verbelen
Bart Dhoedt
VERSES Al Research Lab
IDLab, Ghent University
Aaron Courville Mila, University of Montreal
Sai Rajeswar ServiceNow Research
Vision-language user prompts
Embodied task behavlors
GenRL
Running
Headstand
Doing spits
Multimodal- world model
Figure 1: colltinodal-fuundattt world models connect and align the video-language space of foundation model with the latent space of generative world model for reinforcement learning, requiring vision-only data. Our GenRL framework turns visual and/or language prompts into latent targets and learns to realize the corresponding behaviors by training in the world model's imagination.
Abstract
Learning generalist embodied agents, able to solve multitudes of tasks in different domains is a long-standing problem. Reinforcement learning (RL) is hard to scale up it requires a complex reward design for each task. In contrast, language specify tasks in a more natural way. Current foundation vision-language models (VLMs) generally require fine-tuning other adaptations be adopted in embod- ied contexts, due to the significant domain gap. However, the lack of multimodal data in such domains represents obstacle developing foundation models for embodied applications. In this work, we overcome these problems by presenting multimodal-founattin world models, able connect and align the representation of foundation VLMs with the latent space of generative world models for RL, without any language annotations. The resulting agent learning framework, GenRL, allows one to specify tasks through vision and/or language prompts, ground them the embodied domain's 's dynamics, and learn the corresponding behaviors in imagination. As assessed through large-scale multi-task benchmarking in locomo- tion and manipulation domains, GenRL enables multi-task generalization from language and visual prompts. Furthermore, by introducing data free policy learn- ing strategy, our approach lays the groundwork for foundational policy learning using generative world models.
Website, code and data: mazpie.github.rr wit.eggi.htbbbb..n...... io/genrl
Work done while interning at Mila/SorviceNow Research. Email: pietro. www.ooaaaaaaaaaaaaaoaooo be
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Foundation models are large pre-trained models endowed with extensive knowledge of the world, which can be readily adapted for given task 44 These models have demonstrated extraordinary
generalization capabilities in a wide range of vision [28 45 67 and language tasks [43 19 [3 [T. As we aim extend this paradigm with objects and other agents in their environment, we require generalist agents that are capable of reasoning about these interactions and executing action sequences within these settings [6].
embodied applications, where agents physically interact
Reinforcement learning (RL) allows agents to learn complex behaviors from visual and/or propriocep- tive inputs [1] 26 (27 by maximizing specified reward function. Scaling up RL to multiple tasks and embodied environments remains challenging as designing reward functions is a complicated process, requiring expert knowledge and prone errors which can lead undesired behaviors [I] Recent work has proposed the adoption of visual-language models (VLMs) to specify rewards for visual environments using language [4] 48 39, e.g. using the similarity score computed by CLIP 45 between an agent's input images and text prompts. However, these approaches mostly require fine-tuning of the VLM 38 otherwise, they tend work reliably only in few visual settings (48)
In most RI settings, we lack multimodal data to train or fine-tune domain-specifie foundation models, due to the costs of labelling agents' interactions and/or due to the intrinsic unsuitability of some embodied contexts to converted into language. For instance, in robotics, it's non-trivial to convert a language description of task to the agent's s actions which are hardware-level controls, such as motor currents or joint torques. These difficulties make hard to scale current techniques to large-scale generalization settings, leaving open the question:
How does one effectively leverage foundation models for generalization in embodied domains?
In this work, we present GenRL, novel approach requiring no language annotations that allows training agents solve multiple tasks from visual or language prompts. GenRL learns multimodal- foundation world models (MFWMS) Ms), where the joint embedding space of a foundation video-language model (57) is connected and aligned with the representation of a generative world model for RL (23), using only vision data. The MFWM allows the specification of tasks by grounding language or visual prompts into the embodied domain's 's dynamics. Then, we introduce an RL objective that enables learning to accomplish the specified tasks in imagination 24 by matching the prompts in latent space.
Compared to previous work in world models and VLMs for RL, one emergent property of GenRL is the possibility to generalize to new tasks in a completely data-free manner. After training the MFWM, it possesses both strong priors over the dynamics of the environment, and large-scale multimodal knowledge. This combination enables the agent interpret a large variety of task specifications and learn the corresponding behaviors. Thus, analogously to foundation models for vision and language, GenRL allows generalization to new tasks without additional data and lays the groundwork for foundation models in embodied RL domains [4).
2 Preliminaries and background
Additional related works can be found in Appendix A
Problem setting. The agent receives from the environment observations x E X' and interacts with through actions a E A. In this work, we focus on visual reinforcement learning, so observations are images the environment. The objective of the agent is to accomplish a certain task т, which can be specified either in the observation space 2,, e.g. through images videos, or in language space yo. where Y' represents the space of all possible sentences. Crucially, compared to a standard RL setting, we do not assume that a reward signal is available to solve the task. When reward function exists, is instead used to evaluate the agent's performance.
Generative world models for RL. In model-based RL, the optimization of the agent's actions done efficiently, by rolling out and scoring imaginary trajectories using (learned) model of the environment's dynamics. In recent years, this paradigm has grown successful thanks to the adoption of generative world models, which learn latent dynamics by self pelf-prrdic the agent's inputs [23]. World models have shown impressive performance in vision- based environments 124..

---

Luck
1+1
114k
t-k
conn
conn
conn
comn
co'n1
t:tik tik
Pr-1
naise
emb
elign
(a) Connecting and aligning (Section 3.2)
(b) Learning task behavior (Section) 3.3
Figure 2: Overview of GenRL. The agent learns caattimoddl-founndaa world model that connects and aligns (a) the representation of a foundation VLM with the latent states of a generative world model. Given a certain task prompt, (b) the model allows embedding the task and translating into targets in the latent dynamics space, which the agent can learn to achieve by using RL in imagination.
improving our ability solve complex and open-ended tasks (26). Generative world models have been successfully extended to many applications, such as exploration [5T) skill learning 42), solving
long-term memory tasks [50 and robotics [58 16
Foundations models for RL. Large language models (LLMs) have been used for specifying behaviors using language [4 29 56 59] but this generally assumes the availability of textual interface with the environment or that observations and/or actions can be translated to the language domain. The adoption of vision- language models (VLMs) reduces these assumptions, it allows the evaluation of behaviors the visual space. However. this approach has yet to show robust performance, it generally requires fine-tuning of the VLM 4 15], prompt hacking techniques 9 modifications to the environment
or visual
Vision-language generative modelling. Given the large success of image-language generative models [49], recent efforts in the community have focused on replicating and extending such success to the video domain, where the temporal dimension introduces new challenges, such temporal consistency and increased computational costs [30 3]. Video generative models are similar to world models for RL, with the difference that generation models outputs typically not conditioned on actions, but rather conditioned on language (30 or on nothing at all (i.e. an unconditional model).
GenRL
3.1 World models for RL
GenRL Iearns a task-agnostic world model representation by modelling the sequential dynamics of the environment in compact discrete latent space S [4 26 Latent states E S are sampled from independent categorical distributions. The gradients for training the model are propagated through
the sampling process with straight-through estimation 5
The world model made following components:
Encoder: (x)))))))
Sequence model:
h4 = (1 hi-1)
Decoder: (x)(st))
Dynamics predictor:
trained with the loss:
Ls =
Σ DKL (st (s) st-1, a1-1)]
E Pog
(1)
dyn loss
rooon loss
(= where Ps (s (81|8)-11 a shorthand for Po (f 01-1 l4-))) The sequence model is
implemented linear GRU cell (8). Differently from recurrent state space models (RSSM; (25).

---

for our framework, encoder and decoder models are not conditioned on the information present in the sequence model. This ensures that the latent states only contain information about a single observation, while temporal information is stored in the hidden state of the sequence model. Given the simpler encoder-decoder strategy of our model, the encoder can be seen as a probabilistic visual tokenizer, which is grounded in the target embodied environment (64]
3.2 Multimodal-foundation world models
Multimodal VLMs are large pre-trained models that have the following components:
Vision embedder: e(o)
SPT (IE++*),
<(2)
Language embedder: e f (v).
where T is sequence of visual observations and y is text prompt. For video-language models, k is generally a constant number of frames (e.g. k E {4, 8, 16] frames). Image- language models 1 as the vision embedder takes single frame as an input. For our implementation, we adopt the Intern Video2 video-languaal model [57] (with k=8).
are a special case where
To connect the representation of the multimodal foundation VLM with the world model latent space, we instantiate two modules: a latent connector and a representation aligner:
Connector: Pe (*(ssst))e), Aligner: - fo(e!))
Leonn =
Σ DKL (s 10)0((
Lalign = fo
where sg(-) indicates stop gradients propagating.
The connector learns to predict the latent states of the world model from embeddings in the VLM's representation space. The connector's 's objective consists of minimizing the KL divergence between its predictions and the world model's encoder distribution. While more expressive architectures, such transformers [55] state-space models [21] could be adopted, we opt for a simpler GRU-based architecture for video modelling. This way, we keep the method simple and the architecture of the connector is symmetric with respect to the world model's components.
Aligning multimodal representations. The connector learns to map visual embeddings from the pretrained VLM to latent states of the world model. When learning the connector from visual embeddings e!*), we assume it can generalize to the (theoretical) corresponding language embedding e!!)) if the angle 0 between the two embeddings is small enough, as shown in Fig. 3a. This can be
expressed as cos 0 > or 0 arccos c, with e small positive constant [68]
Multimodal VLMs trained with contrastive learning exhibit a multimodality gap spherical embeddings of different modalities not aligned. Given a dataset of vision-langue data, this projective function can be learned. However, in embodied domains vision-language data is typically unavailable. Thus, we have to find way to align the representations using no language annotations.
34 where the
Previous methods inject the noise into the vision embeddings during training [66 68 This leads to the situation shown in Figure Bb. where c grows larger with the noise. This allows language embeddings to be close enough to their visual counterparts.
In our work, we instead learn an aligner network, which maps points surrounding el closer to e(v)) As represented in Fig. 3. this way, c is unaltered but the aligner will map e(!)) close enough to e(r). Since we use noise to sample points around e(*) the aligner model can be trained using vision-only data and thus, no language annotations.
Figure 3: When training the connec- tor on (a) the VLM's representation we can address the multimodality gap in multiple ways: (b) prior works adopt noise during the training of the connec- tor, (c) we adopt aligner network that learns to map points in proxim- ity of the visual embedding close the corresponding embedding.
(a) VLM representation
(b) Effect of noise e/ connector training
(e) Effect of aligner map to
efl
elt
eM
Hypersphere surface where points have high cosine similarity with eM
Hypersphere surface where the aligner network mape points to el

---

The aligner allows us to train noise-free connector, which has two main advantages: (i) it yields higher prediction accuracy for visual embedding inputs while maintaining a similar alignment for language embedding inputs; and (ii) it is more flexible; it's easier to re-train/adapt for different noise levels, as it only requires re-training the aligner module, and its use can avoided if unnecessary.
3.3 Specifying and learning tasks in imagination
World models can be used to imagine trajectories in latent space, using the sequential and dynamics models. This allows to train behavior policies in model-based RI fashion 224. Given task specified through a visual or language prompt, our MFWM can generate the corresponding latent states by turning the embedder's output, taskk into sequences of latent states Stc+& (decoded examples are shown in Figure D. The objective of the policy model mTe is then to match the goals specified by the user by performing trajectory matching.
The trajectory matching problem can be solved as divergence minimization problem 13 between the distribution of the states visited by the policy ma and the trajectory generated using the aligner- connector networks from the eperiseed prompt:
0 = arg min E molll))
Σ Zy (distance(po(sst1lee 1864, ae llpolllpssee 1
lemat))
with Ctack spr(.).
0
(2)
The KL divergence is natural candidate for the distance function 13 However, in practice, we found that using the cosine distance between linear projections of the latent states notably speeds up learning and enhances stability. We can then turn the objective in Eq. 2 into reward for RL:
'GenRL = coso(( (g Io (144))) with
$41 Po(S az), sank ~ PP (eusk),
where 9s represents the first linear layer of the world model's decoder. We train an actor-critic model to maximize this reward and achieve the tasks specified by the user [26]. Additional implementation details are provided in Appendix B
Temporal alignment. One issue with trajectory matching is that it assumes that the distribution of states visited by the agent starts from the same state as the target distribution. However, the initial state generated by the connector may differ from the initial state where the policy is currently in. For example, consider the Stickman agent on the right side of Figure [ If the agent is lying on the ground and tasked to run, the number of steps to get up and reach running states may surpass the temporal span recognized by the VLM (e.g. in our case 8 frames), causing disalignment in the reward.
To address this initial condition alignment issue, we propose a best matching trajectory technique, inspired by best path decoding in speech recognition 20 Our technique involves two steps:
1. We compare the first b states of the target trajectory with b states obtained from the trajectories imagined by the agent by sliding along the time axis. This allows one to find at which timestep fa the trajectories are best aligned (the comparison provides the highest reward).
2. We align the temporal sequences in the two possible contexts: (a) if state from the agent sequence comes before la, the reward uses the target sequence's 's initial state; and (b) if the state comes steps after la, it's compared to the st+* state from the target sequence.
In all experiments, we fix = 8 (number of frames of the VLM we use (57), which we found to strike good compromise between comparing only the initial state (b = 1) and performing no alignment (b- imagination horizon). An ablation study can be found in Appendix
4 Experiments
Overall, we employ set of locomotion environments (Walker, Cheetah, Quadruped, and a newly introduced Stickman environment) (54] and one manipulation environment (Kitchen) [22]. for total of tasks where the agent trained without rewards, using only visual or language prompts. Details
about the datasets, tasks, and prompts used can be found in the Appendix C
5

---

Table 1: Language-to-act in-distribution. Offline RL from language prompts on tasks that are included in the agent's 's training dataset. Scores are episodic rewards averaged over 10 seeds (+ standard error) rescaled using min-max scaling with (min random policy, max expert policy).
Image-language VLM
Vidco-langunge VLM
IQL
TD3+BC
TD3
WM-CLIP
IQL
TD3+BC
TD3
WM-CLIP
GenRL
walker stand
0.67 0.00
0,92
0,93 193:000
1.01 0.0
0.66 0:05
0.64
1.01 :00
0.94 0.0
1.02 0.0
walker run
0.24
0.27 = 0.0
0.09 4 0.02
0.05 0.02
0
0.24
0.35 4 000
0.7 0:01
0.77 = 0.02
walker walk
0,34 0.05
0.04
0.21 a 0.01
0.4 0.03
0.44:
0.88 + 0.00
0,91 + 0.00
1.01 0.0
cheetah run
0
0.0
-0.01
-0.0
0.15
-0.01 0.0
0.37 001
0.56 0.00
0.74 0.01
quadruped stand
0.56 + onz
0.64 + 0.04
0.65 + 0.04
0.97 + 0.0
0.52 0:06
0.43 ons
0.61 # 0.00
0.97 0.0
0.97 #
quadruped run
0.3 0.03
0,28 + 0.02
0.24
0.27 1 0.0
0.38 + 0.03
0,25 0.02
0,26 0.01
0.61 + 0.02
0.86 0.02
quadruped walk
0.26 1
0.31 0.02
0.28
0.47 0.02
0.32
0:01
0.28 0.02
0.92
0.93 0.01
stickman stand
0.45 + on6
0.58 + 0:04
0.06 + 0.04
0.71 + 0.02
0.43 0:004
0.45 + o.os
0.08 + 0.02
0.32 + 0.01
0.7 = 0.02
stickman walk
0.4 = 0.04
0,48 + 0:01
0.18
0.23 0:01
0.51 0.02
0,46 0.03
0.41 + 0.02
0.65 0.05
0.83 0.01
stickman
0.2
0.22 2: 0.02
0.19 0.0
0.23
0.21 0.19
0.35
0.35
kitchea microwave
0.06 0.04
0,22 + 0.11
0.0 1.0::00
0.0 0.0
0.01 0.01
0.0
0.11 + 0.08
0.9 + 0.09
0.97 = 0.02
kitchen light
0.14
0,11 + 0.11
0.59 + 0.16
0.1 # 0.09
0.02 0.01
0.0
0.18 2 0.11
0,26 + 0.13
0.46 2 0.09
kitehen burner
0.21 0:05
0.18 0.05
0.09 0.05
0.03 0.03
0.02
0.02
0.31
0.78
kitchen slide
0.02+
0.0 0.0
0.0
0.0
0.04 +0002
0.02 0.02
0.7 +:014
0.88 + 0.04
1,0 + 0.0
overall
0
0,33 + 0:01
0,23 + 0.04
0.30 0.02
0.29 0.02
0.24 0.02
0.41 + 0.05
0,70 + 0.01
0.80 = 0.02
4.1
Offline RL
In offline RL, the objective of the agent is to learn to extract a certain task behavior from a given fixed dataset [33]. The performance of the agent generally depends on its ability to 'retrieve' the correct behaviors in the dataset and interpolate among them. Popular techniques for offline RL include behavior cloning, such as IQL [31. and behavior-regularizee approaches, such as CQL (32 TD3+BC (17).
off-policy RL methods, such as TD3 [8] advantage-weighee
We aim to assess the multi-task capabilities of different approaches for designing rewards using VLMs. We collected large datasets for each of the domains evaluated, containing a mix of structured data (i.e. the replay buffer of an agent (26) learning to perform some tasks) and unstructured data (i.e. exploration data collected using (5I).. The datasets contain no reward information and no text annotations of the trajectories. The rewards for training for given task must be inferred by the agent, i.e. using the cosine similarity between observations and the given prompt or, in the case of GenRL, using our reward formulation (Eq. 3).
We compare GenRL to two main categories of approaches:
Image-language rewards: following [8 the cosine similarity between the embedding for the language prompt and the embedding for the agent's s visual observation is used as a reward. For the VLM, we adopt the SigL
[65] model as it's reported to have superior
performance than the original CLIP [45).
Video- Video-language rewards: similar to the lange-angguage rewards, with the difference that the vision embedding is computed from a video of the history of the last & frames, as done in [1]. For the VLM, we use the Intern Video2 model [57], the same used for GenRL.
The evaluation compares GenRL to various offline RL methods from the literature, including IQL, TD3+BC, and TD3. We also introduce a model-based baseline, WM-CIIP. This baseline is the antithesis of GenRL as, rather than learning a connector and an aligner, it learns a "reversed connector". This module learns to predict VLM embeddings from the world model states (GenRL does the opposite). This makes it possible to compute rewards in imagination in a similar way to the model- free baselines, by computing the cosine similarity between the visual embeddings predicted from imagined states and the task's 's language embeddings.
All methods are trained for 500k gradient steps, and evaluated on 20 episodes. For each task, model- free agents require training the agent from scratch, including the visual encoder, actor, and critic networks on the entire dataset. Model-based agents require training the model once for each domain and then training an actor-critie for each task. Other training and baseline details are reported in Appendix D

---

walker (6 taskst
quadruped tasks)
stickman (10 tasks)
cheetah taskst
overall tasks)
0.6
0.4
0.2
1QL-1
TD3+9C+ TD3+
WW-CUPP
IQL-V
TD3+BC'V
TD3-V
WK-CLIP-
GenRL
Figure 4: sanguagg--t-accionn generalization. Offline RL from language prompts tasks that not deliberately included in the training dataset. Performance averaged over 10 seeds and standard error reported with black lines. Detailed results per task in Appendix K
Language-to-action in-distribution. We want to verify whether the methods can retrieve the task behaviors that are certainly present in the training data, when specifying the task only through language. We present results Table ] with episodic rewards rescaled so that represents the performance of random agent, while represents the performance of expert agent.
GenRL excels in overall performance across all domains and tasks, outperforming other methods particularly in dynamic tasks like walking and running in the quadruped and cheetah domains. However, some static tasks of the kitchen domain, other methods occasionally outperform GenRL. outpere
This can be explained by the fact that the target sequences that GenRL infers from the prompt are often slightly in motion, even static cases. To address this, we could set the target sequence length to for static prompts, but we opted to maintain the method's simplicity and generality, acknowledging this as minor limitation.
As expected, video-language rewards tend to perform better than image-language rewards for dynamic tasks. The less conservative approach, TD3, performs better than the other model-free baselines in most tasks, similarly to what is shown in [62] The model-based baseline's performance, WM-CLIP-V, the closest to GenRL's.
generalization. To assess multi task generalization, we defined set of tasks not included in the training data. Although don't 't anticipate agents matching the performance of expert models, higher scores in this benchmark help gauge the generalization abilities of different methods. We averaged the performance across various tasks for each domain and summarized the findings in Figure with detailed task results in Appendix K
Language-to-acte
Overall, we observe similar trend for the in-distribution results. GenRL significantly outperforms all model-free approaches, especially in the quadruped and cheetah domains, where the performance is close to the specialized agents' performance. Both for image-language (-I in the Figure) and lideo-language (-V the Figure) more conservative approaches, such as IQL and TD3+BC tend to perform worse. This could be associated with the fact that imitating segments of trajectories is less likely to lead to high-rewarding trajectories, as the tasks are not present in the training data.
quadi upod
0.6
0.4
Gon
Figure 5: Video-to-action GenRL. allows grounding video prompts into the target environment's dynamics. It allows visualization of the model's interpretation of the prompts, using the decoder (top row), and it allows turning prompts into behaviors, leading generally higher performance than other approaches. 10 seeds. Additional visualizations on the project website.
7

---

(a) GenRL (offline policy learning)
(b) GenRL (data-free policy learning)
ovenal (35 twees)
1.0
ttak prompt
teak prompt
actionn
actions
0.4
states Intemallyy
0.2
Offine
observations
Iatent
neveercle
Agent
Agent
0.0
datasot
Pro trained
150t Freet
Figure 6: By removing data dependencies on actions (on-policy learning in the model's imagination), rewards (computed using only the prompt and latent states, using Eq. 3}. and observations (by sampling latent states within the model), GenRL agents can be adapted for new tasks data-free
fashion. Performance is averaged over 10 seeds and standard error is reported with black lines. Detailed results per task Appendix K
Video-to-action. While language strongly simplifies the specification of task, in some cases providing visual examples of the task might be easier. Similarly as for language prompts, GenRL allows grounding visual prompts (short videos) into the embodied domain's dynamics and then learning of the corresponding behaviors.
In Figure [] we provide behavior learning results from video prompts. The tasks included are of static and dynamic nature and span across 4 different domains. Visualizations of the videos used as prompts are available on the project website, where we also present a set of "grounded videos" generated by the model using the prompts (see snapshots at the top of Fig. 5). These can be obtained by inferring the latent targets corresponding to the vision prompts (left images, in the Figure) and then using the decoder model to decode reconstructed images (right images, in the Figure).
The results show similar trend to the language prompts experiments and the performance when using video prompts is aligned to the uinuage-to-action performance, for the same tasks. In general, we found it interesting that the VLM allows us to generalize to very different visual styles (drawings, realistic, Al-generated), very different camera viewpoints (quadruped, microwave), and different morphologies (cheetah tasks).
Summary. The experiments presented allow us to establish more clearly the main ingredients that contribute to the stronger performance of GenRL: (i) the languuan model helps in dynamic tasks, (ii) model-based algorithms lead to higher performance, (iii) the connection comnetion-alignment system presented generally outperforms the "reversed" way of connecting the two representations.
4.2 Data-free policy learning
In the previous section, we evaluated several approaches for designing reward using foundation VLMs. Clearly, model-free RL approaches require continuous access to a dataset, to train the actor-critic and generalize across new tasks. Model-based RL can learn the actor-critic in imagination. However, in previous work (26, 24). imagining sequences for learning behaviors first requires processing actual data sequences. The data is used initialize the dynamics model, and obtain latent states that represent the starting states to rollout the policy in imagination. Furthermore, in order to learn new tasks, seward-labelled data is necessary to learn reward model, which provides rewards to the agent during the task learning process.
Foundation models 44 are generally trained on enormous datasets in order to generalize to new tasks. The datasets used for the model pretraining are not necessary for the downstream applications, and sometimes these datasets are not publicly available [4] 19]. In this section, a new paradigm for foundation models in RL, which follows the same principle of foundation models for vision and language. We call this paradigm data-free policy learning and we define it as the ability to generalize to new tasks, after pre-training, by learning a policy completely in imagination, with no access to data (not even to the pre- pre-training dataset).
aim to establish
GenRL enables frte-free policy learning thanks to two main reasons: the agent learns a task-agnostic MFWM on a large varied dataset during pre-training, and the MFWM enables the possibility of specifying tasks directly in latent space, without requiring any data. Thus, in order to learn behaviors in imagination, the agent can: (i) sample random latent states in the world model's representation,

---

walker stand
walker run
walker walk
walker (9 tasks)
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
all expl data
data
walk stand
0.0
Figure 7: Training data distribution. Analysing the impact of the training data distribution on the generalization performance of GenRL. Performance is obtained by training behaviors data-free mode, after training the MFWM on different subsets of the training dataset. Performance averaged over 10 seeds (black lines indicate standard error). Full results in Appendix
(ii) rollout sequences in imagination, following the policy's 's actions, and (iii) compute rewards, using the targets obtained by processing the given prompts with the connector-aligner networks.
In Figure 6 we provide a diagram that further clarifies the differences between training GenRL in an offline and in a data-free policy learning fashion. Then, we present results that compare data- free policy learning with offline RL baselines, as discussed in Section |.1 While data-free policy learning shows slight decrease in overall performance, its performance remains close to the original GenRL's performance and still outperforms other approaches. In Appendix K the difference in performance is minimal across most domains, and data-free policy learning performs better in the kitchen domain.
we further show that
By employing data-free learning, after pre- vre-training agents can master new tasks without data. By requiring no CPU-GPU memory transfers of the data, data-free policy learning also reduces the training time of the policy, often allowing convergence within only 30 minutes of training. As we scale up foundation models for behavior learning, the ability to learn data-free will become crucial. Although very large datasets will be employed train future foundation models, GenRL adapts well without direct access to original data, offering flexibility where data may be proprietary, licensed or unavailable.
4.3 Analysis of the training data distribution
As demonstrated in Sections 4.1 and 4.2 after training on large dataset, a GenRL agent can adapt to multiple new tasks without additional data. The nature of the training data, detailed in Appendix [ combines exploration and task specifc data. Thus, we ask ourselves what subsets of the data are the most important ones for GenRL's training.
To identify critical data types for GenRL, trained different MFWMs on various dataset subsets. Then, we employ data-free behavior learning to train task behaviors for all tasks. We present an analysis over subsets of the walker dataset in Figure 7
The results confirm that a diverse data distribution is crucial for task success, with the best performance achieved by using the complete dataset, followed by the varied exploration data. Task-specific data effectiveness depends on task complexity, for instance, 'run data' proves more useful and generalizable than 'walk data' or 'stand data' across tasks. Crucially, 'stand data', which shows minimal variation, limits learning g for a general agent but can still manage simpler tasks like 'lying down' and 'sitting on knees' as detailed in Appendix K
Moving forward with training foundation models RL, will essential to develop methods that extract multiple behaviors from unstructured data and accurately handle complex behaviors from large datasets. Thus, the ability of GenRI. to primarily leverage unstructured data a significant advantage for scalability.

---

5 Discussion
We introduced GenRL, a world-model based approach for grounding vision-language prompts into embodied domains and learning the corresponding behaviors in imagination. The multimodal- foundation world models of GenRL can be trained using unimodal data, overcoming the lack of multimodal data in embodied RL domains. The data-free behavior learning capacity of GenRL lays the groundwork for foundation models in RL that can generalize to new tasks without any data.
A framework for behavior generation. A common challenge with using LLMs and VLMs involves the need for prompt tuning to achieve specific tasks. As GenRL. relies on a foundation VI.M, similar to previous approaches 4
48 it is not immune from this issue. However, GenRI. uniquely allows for the visualization of targets obtained from specific prompts. By decoding the latent targets, using the MFWM decoder, we can visualize the interpreted prompt before training the corresponding behavior. This enables a much more explainable framework, which allows fast iteration for prompt tuning, compared to previous (model-free) approaches which often require training the agent to identify which behaviors are rewarded given a certain prompt.
Limitations. Despite its strengths, GenRL presents some limitations, largely due to inherent weak- nesses in its components. From the VLMs, GenRL inherits the issue related to the multimodality gap [4 66) and the reliance prompt tuning. We proposed connection-ale -laigmmmnnnlligmgentt mechanism to mitigate the former. For the latter, we presented an explainable framework, which facilitates prompt tuning by allowing decoding of the latent targets corresponding to the prompts. From the world model, GenRL inherits a dependency on reconstructions, which offers advantages such as explainability but also drawbacks, such as failure modes with complex observations. We further investigate this limitation in Appendix and present other potential limitations in Appendix |
Future work. As we strive to develop foundation models for generalist embodied agents, our framework opens up numerous research opportunities. One such possibility is learn multiple behaviors and have another module, e.g. an LLM, compose them to solve long- mor-oooizon tasks. Another promising area of research investigating the temporal flexibility of the GenRL framework. We witnessed that for static tasks, greater temporal awareness could enhance performance. This concept could also apply to actions that extend beyond the time comprehension of the VLM. Developing general solutions to these challenges could lead to significant advancements in the framework.
Acknowledgments and Disclosure of Funding
Pietro Mazzaglia is funded by Ph.D. grant of the Flanders Research Foundation (FWO). This research was supported by a Mitacs Accelerate Grant.
References
[1] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety, 2016.
[2] B. Baker, I. Akkaya, P. Zhokhov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Vidco pretraining (vpt): Learning to act by watching unlabeled online videos, 2022.
[3] O. Bar-Tal, H. Chefer, O. Tov, C. Herrmann, R. Paiss, S. Zada, A. Ephrat, J. Hur, G. Liu, A. Raj, Y. Li, M. Rubinstein, T. Michaeli, O. Wang, D. Sun, T. Dekel, and I. Mosseri. Lumiere: A space-time diffusion model for video generation, 2024.
[4] K. Baumli, S. Baveja, F. Behbahani, Chan, G. Comanici, Flennerhag, M. Gazeau, K. Hol- sheimer, D. Horgan, M. Laskin, et al. Vision-language models as a source of rewards. arXiv preprint axXiv2311..09187. 2023.
[5] Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprini 138.::33...333,, 2013.
[6] Bruce, M. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, Hughes, M. Lai, A. Mavalankar, R. Steiger- wald, C. Apps. Y. Aytar, S. Bechtle, F. Behbahani, S. Chan, N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rocktäschel. Genie: Generative interactive environments, 2024.
10

---

[7] E. Cetin, A. Tirinzoni, M. Pirotta, A. Lazaric, Y. Ollivier, and A. Touati. Simple ingredients for offline reinforcement learning, 2024.
[8] Chung, Gulcehre, K. Cho, and Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling, 2014.
[9] Y. Cui, S. Niekum, A. Gupta, V. Kumar, and A. Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation?, 2022.
[10] F. Deng, I. Jang, and S. Ahn. Dreamerpro: Reconstruction-fee free model-based reinforcement learning with prototypical representations. International Conference on Machine Learning, pages 4956 4975. PMLR, 2022.
[11] J. Devlin, M.W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2019.
[12] Embodiment Collaboration et al. Open x-embodiment: Robotic learning datasets and rt-x models, 2024.
[13] P. Englert, A. Paraschos, J. Peters, and M. P. Deisenroth. Model-based imitation learning by probabilistic trajectory matching. In 2013 IEEE international conference on robotics and automation, pages 1922 1927. IEEE, 2013.
[14] A. Escontrela, A. Adeniji, W. Yan, A. Jain, X. B. Peng, K. Goldberg, Y. Lec, D. Hafner, and P. Abbeel. Video prediction models as rewards for reinforcement learning. Advancey in Neural Information Processing Systems, 36, 2024.
[15] L. Fan, G. Wang. Jiang, Mandlekar, Yang. H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anand- kumar. Minedojo: Building open- open-ended embodied agents with internet-seat knowledge. Advances in Neural Information Processing Systems, 35:18 118881333362 2022.
[16] S. Ferraro, P. Mazzaglia, T. Verbelen, and B. Dhoedt. Focus: Object-centric world models for robotics manipulation, 2023.
[17] S. Fujimoto and Gu. minimalist approach to offline reinforcement learning, 2021.
[18] S. Fujimoto, H. van Hoof, and Meger. Addressing function approximation emorr actor-critic methods, 2018.
[19] Gemini Team et al. Gemini: A family of highly capable multimodal models, 2024.
[20] A. Graves. S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages (69-376, 2006.
[21] A. Gu and T. Dao. Mamha: Linear-time sequence modeling with selective state spaces, 2023.
[22] A. Gupta, V. Kumar, C. Lynch. S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning, 2019.
[23] D. Ha and J. Schmidhuber. World models. 2018.
[24] D. Hafner, Lillicrap, Ba, and M. Norouzi. Dream control: Learning hehaviors by latent imagination, 2020.
[25] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels, 2019.
[26] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.41100 2023.
[27] N. Hansen, H. Su, and X. Wang. Td-mpe2: Scalable, robust world models for continuous control, 2024.
[28] A. Kirillov, Mintun, N. Ravi, H. Mao, Rolland, L. Gustafson, Xiao, S. Whitehead, C. Berg. W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything, 2023.
[29] M. Klissarov, P. D'Oro. S. Sodhani, R. Raileanu, P.-. Bacon, P. Vincent, A. Zhang, and M. Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. 2023.
11

---

[30] D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, R. Hornung, H. Adam, H. Akbari, Y. Alon, V. Birodkar, et al. Videopuet: A large language model for zero-shot video generation. arXiv preprini 2312222....4125, 2023.
[31]
Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. 2021.
[32] A. Kumar, A. Zhou, G. Tucker, and Levine. Conservative q-learning for offline reinforcement learning. 2020.
[33] S. Levine, A. Kumar, G. Tucker, and J. Fu, Offline reinforcement Iearning: Tutorial, review, and perspec- tives on open problems, 2020.
[34] Liang, Zhang. Y. Kwon, Yeung, and Zou. Mind the gap: Understanding the modality gap in multi modal contrastive representation learning, 2022.
[35] S. Lifshitz, K. Paster, H. Chan, J. Ba, and S. Mcllraith, Steve-l: A generative model for text-to-behavior in minecraft, 2024.
[36] J. Lin, Y. Du, O. Watkins, D. Hafner, P. Abbeel, D. Klein, and A. Dragan. Learning model the world with language, 2023.
[37] H. Liu, W. Yan, M. Zaharia, and P. Abbeel. World model on million-length video and language with blockwise ringattention, 2024.
[38] E. S. Lubana, J. Brehmer, P. de Haan, and T. Cohen, Fomo rewards: Can we cast foundation models as reward functions?, 2023.
[39] C. Luo, M. He, Zeng, and C. Sun. Text-aware diffusion for policy learning, 2024.
[40] Y. J. Ma, Liang. Som, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image representations and rewards for robotic control, 2023.
[41] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models, 2024.
[42] P. Mazzaglia, T. Verbelen, B. Dhoedt, A. Lacoste, and S. Rajeswar. Choreographer: Learning and adapting skills in imagination. International Conference on Learning Representations, 2023.
[43] OpenAl et al. Gpt-4 technical report, 2024.
[44] R. Bommasani et al. On the opportunities and risks of foundation models, 2022.
[45] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[46] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and Chen. Hierarchical text-conditional image generation with clip latents, 2022.
[47] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky,
J. Kay, J. T. Springenberg, T. Fecles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordhar, and N. de Freitas. A generalist agent, 2022.
[48] Rocamonde, V. Montesinos, E. Nava, E. Perez, and Lindner. Vision-language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310. 12921, 2023.
[49] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models, 2022.
[50] M. R. Samsami, A. Zholus. J. Rajendran, and S. Chandar. Mastering memory tasks with world models. In The Twelfth International Conference on Learning Representations, 2024.
[51] R. Sekar, O. Rybkin, K. Daniilidis, Abbeel, D. Hafner, and D. Pathak. Planning to explore via self- supervised world models, 2020.
[52] D. Tarasov, V. Kurenkov. A. Nikulin, and S. Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning, 2023.
12

---

[53] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023.
[54] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez. J. Merel. T. Erez, T. Lillierap. N. Heess, and Y. Tassa, dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
[55] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention you need, 2023.
[56] G. Wang, Y. Xie. Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023.
[57] Y. Wang, K. Li, X. Li, J. Yu, Y. He, G. Chen, B. Pei, R. Zheng, J. Xu, Z. Wang. Y. Shi, T. Jiang, S. Li, H. Zhang, Huang. Y. Qiao, Y. Wang, and Wang. Intemvideo2: Scaling video foundation models for multimodal video understanding, 2024.
[58] P. Wu, A. Escontrela, D. Hafner, K. Goldberg, and Abbeel. Daydreamer: World models for physical robot learning, 2022.
[59] T. Xie, Zhao, H. Wu, Y. Liu. Q. Luo, Zhong. Y. Yang, and Yu. Text2reward: Reward shaping with language models for reinforcement learning, 2024.
[60] Yang, Y. Du, Ghasemipour, J. Tompson, L. Kaelbling, D. Schuurmans, and Abbeel, Learning interactive real-world simulators, 2024.
[61] S. Yang, Nachum, Du, J. Wei, Abbeel, and D. Schuurmans. Foundation models for decision making: Problems, methods, and opportunities, 2023.
[62] D. Yarats, D. Brandfonnreneer H. Liu, Laskin, Abbeel, A. Lazaric, and L., Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning, 2022.
[63] Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data- augmented reinforcement learning, 2021.
[64] L. Yu, Lezama, B. Gundavarapu, L.. Versari, Sohn, Minnen, Cheng, Birodkar, A. Gupta, X. Gu, A. G. Hauptmann, B. Gong, M.-H. Yang, I. Essa, D. A. Ross, and L. Jiang. Language model beats diffusson tokenizer is key to visual generation, 2024.
[65] X. Zhai, B. Mustafa, A. Kolesnikov, and I.. Beyer. Sigmoid loss for language image pre-training, 2023.
[66] Y. Zhang, E. Sui, and Yeung-Levy. Connect, collapse, corrupt: Learning cross-modal tasks with uni moda data, 2024.
[67] I.. Zhao, B. Gundavarapu, L.. Yuan, H. Zhou, S. Yan, Sun, L., Friedman, R. Qian, Weyand, Y. Zhao, R. Hornung, F. Schroff, M.-H. Yang, D. A. Ross, H. Wang, H. Adam, M. Sirotenko, T. Liu, and
B. Gong. Videoprism: A foundational visual encoder for video understanding, 2024.
[68] Y. Zhou, R. Zhang, C. Chen, C. Li, C. Tensmeyer, T. Yu, J. Gu, J. Xu, and T. Sun. Lafite: Towards language-free training for text-to-image generation, 2022.
13

---

A Extended Related Work
[inked to Section
World models. Recent research has focused on the question of how to learn world models from large-scale video datasets (37 60). In [6. they leverage latent action representation, but their work is mostly focussed on 2D platform vidcogames or simple robotic actions. [14] they use hraeeby-frame vidco prediction
way to provide rewards for RL. DynaLang [66] studies the incorporation language prediction as part the world model, to train multimodal world models also from datasets without actions or rewards. The representation in DynaLang is shared in the world model between vision and language, while for GenRL, the world model representation is trained on vision-only data and connected-aligned to the multimodal foundation representation.
Foundation models for actions, Few cases of foundation models for embodied domains have been developed. Notable mentions are GATO large-scale model trained on Minecraft data, using human expeet labeled trajectories. The model learns strong behavioral priors by behavior cloning which can be fine-tuned using RL. STEVE-1 [35] connects VPT's behavioral prior with the MineCI.IP model representation [15]. using the unCLIP approach [46]. RT-X [12 are large-scale transformer models trained on expert robotics dataset, sharing common action space (end-effector pose) across different embodiments.
large-scale behavior cloning agent, trained on 604 tasks. VPT (2)
B Implementation details
Linked Section 3]
Actor-critic. Rewards can be maximized over time in imagination a RL fashion, using actor-critic models of the form:
Actor: [e(a)))))
Critic:
U6 0o (s)), where R = + 7(11 Am++1) : + XR
For the actor-critic, we follow the implementation advances proposed in Dreamer (26] (version of the paper, dated January 2023), such using two-hot distribution for lcarning the critic network and scaling returns in the actor loss.
When computing the reward TGuRL,, we the mode of the distribution for the target s141 ~V pe (8t+1/eusk) improve stability.
paper. dated January 2023). Differences from the default hyperparameters or model size choices are illustrated in Table[]] For instance, a main difference is that we use difference batch sizes/lengths for training the MFWM and the actor-critic as these stages
Hyperparameters. For the hyperparameters, we follow DreamerV3 20
(version
independent from each other.
The connector network the same hyperparameters and architecture the sequential dynamics of the world model. The aligner network employs small U-Net, with bottleneck that is half the size of the embedding representation. The embedding representation 768-dimensional. Further details can found in our accompanying code implementation.
Name
Value
Multimodal Foundation World Model
Batch size
48
Sequence length
48
GRU recurrent units
1024
CNN multiplier
48
Dense hidden units
1024
MLP layers
Actor-Critic
Batch size
32
Sequence length
32
Table 2: World model and actor-critic hyperparameters.
14

---

C
Tasks
[Linked to Section 4]
Stickman environment. The Stickman environment is based on the Walker environment from the dm control suite. We designed the Stickman environment to explore tasks that require upper body limbs (c.g. boxing, doing a handstand) without the complexity of training humanoid (which requires a significantly larger amount of data be solved ). The number of joints is increased by 4: joints per arm, is for the shoulder, the other for the elbow. The total number joints is 10. action space is normalized to [-1,1] as dm control tasks. The robot also presents a head, to resemble a humanoid.
Prompts and scores. We present the list of tasks employed, along with the language prompts used for specifying the task, in Table 3 For the newly introduced tasks, the goal can be easily inferred by reading the task's 's name its prompt. For the 'flipping' tasks, consider flips both in the forward dircction and backward direction, VLM struggles distinguish directions. The reward functions used evaluate agent's score can be found in our open-source code.
The prompts we use have been fine-tuned for the Intern Video2 model [57] However, we found that they mostly improved performance for the SigLIP model too [65] One common observation is that these models are generally biased towards human actions. Thus, specifying the embodiment in the prompt is sometimes helpful, e.g. 'spider running fast' 'running like quadruped'. Another observation is that for some behaviors the agent can produce very different styles, e.g. agent can be walking in slow or way, or a more or less composed manner. Specifying words like 'fast' or 'clean' helps clarifying what kind of behavior is expected.
Table 3: Task and prompt used for each task Prompt
Specialized agent
agent
score
score
quadruped
spider rumning fast
930
quadruped
spider walking
quadruped stand
spider standing
990
15
quadruped jump
spider jumping
875
15
quadruped logs
logs
875
quadruped lie down
lying down
965
750
chectsh
running like quadruped
890
9
standing
standing human
cheetah lying down
lying down
920
430
stickran walk
robot walk fast clean
960
35
stickman
stickman stand
standing
970
70
stickman flipping
doing flips
790
45
stickman foot
stand foot stand up and kick
865
stickman high kick stickman lying down stickman sit knees
920
55
lying down horizontally
965
380
praying
966
40
stickman lunge pose
lunge pose
stickman headstand
headstand
935
180
stickman boxing
punch
920
80
stickman up
standing hands up
walker walk
walk fast clean
960
45
walker
fast clean
770
30
stand
standing up straight
walker flipping
doing backflips stand on one foot up and lying down borizonially
720
20
walker one foot
955
20
walker high kick
walker lying down
975
170
walker sit knees
praying
945
100
walker lunge pose
lunge pose
945
kitchen microwave
opening the microwave fully open
kitchen light
activate the light
1
0
kitchen burner
the buroer becomes red slide cabinet above the knobs
kitchen slide
15

---

D Experiments settings
Linked Section 41
Baselines. In order implement performant model-free offiine RI. baselines adopt the findings of [2 and 7. adopting larger deeper networks and layer normalization. Inputs 64x64x3 RGB images. We frame stack of 3. The encoder architecture is adapted from the DrQ-v2 encoder images, e.g. random shifts, to hurt performance.
6 We find augmentations on the
The WM-CLIP baseline learns a "reversed connector" from the world model representation to the VLM representation (GenRL does the opposite). The "reversed connector", given the latent state corresponding to certain observation, predicts the corresponding embedding. Formally:
Reversed connector:
Crev comn [|e(0) 2
After training the reversed connector, visual embeddings can be inferred from latent states. For policy learning, rewards are computed using the cosine similarity between embeddings inferred from imagined latent states and the prompts' embedding.
The reversed connector is implemented 4-layer MLP, with hidden size 1024, For fair comparison, adopt the same world model for WM-CLID GenRL. For WM-CLIP we pre-train the additional reversed connector, while for GenRL the connector and aligner.
Offline RL. For each task, training model-free agents (IQL, TD3, TD3+BC) requires re- re-training the full agent (visual encoder, actor, critic) on the entire dataset, from scratch, while training model-based agents (GenRL, WM-CLIP) requires training the model once for each domain and then training an actor-critic for each task. Moreover, for training the actor-critic in GenRI,, only 50k gradient steps, the policy converges significantly faster than for the other methods.
Datasets composition. We present datasets' composition in Table
Table 4: Datasets composition.
Domain
~u num of observations
Subset
Subcount
walker
2.5M
walker
500k
walker walk
500k
walker stand
500k
walker expl
IM
chectah 1.8M
chectah run
1M
cheetah expl
820k
quadrupod
2.5M
quadruped expl
1M
quadruped run
500k
quadruped stand
500k
quadruped walk
500k
kitchen
3.6M
kitchen slide
700k
kitchen light
700k
kitchen bottom humer
700k
kitchen microwave
700k
kitchen expl
800k
stickman
2.5M
stickman stand
500k
stickman walk
500k
stickman expl
1M
stickman run
500K
minecraft
Compute resources. We cluster of V100 with 16GB of VRAM for all experiments. To enable efficient training, image and video-CLIn embeddings are computed advance and stored with the datasets. Training the MFWM for 500k gradient steps takes days. After pre-training the MFWM, training the actor-critic for a prompt for 50k gradient steps takes less than 5 hours. In data-free mode, it takes less than 3 hours. In both cases, convergence normally arrives after 10k gradient steps, but we keep training. Model-free baselines take around 7 hours train for 500k gradient steps.
On single GPU, model-free RL faster train for small number of GenRL starts becoming advantageous when using the world model training for more than runs (which often case, considering the number of runs = seeds x M tasks per domain). When adopting the data-free policy learning strategy, GenRL doesn't rely on the dataset at all. This halves the time required for training. as there are no data transfers between the CPU (where the dataset is normally loaded) and the GPU for training.
16

---

E
Temporal alignment ablation
[Linked to Section
In Figure S we investigate the impact of our best-matching trajectory strategy for temporally aligning imaginary latent states to target states for GenRL's reward computation.
vakernn
soberwal
sistrupei-mat
unddupedion
conineed wss
0.4
0.2
0.0
Mirees
Whikee Iiee
Wivves bumer
Whrves 641
0.4
0.2
0.0
mucling translary
no algggnmment
Ne 13 tapis)
chestall (1 tasks)
ouadruped 13 tacks)
stickmann (3 taskst
Mrchen tackst
oiverall 114 (adk3)
1.0
0.2
0.0
best matching Uajedory
best matching in tal sate
algnmenn
Figure 8: Temporal alignment ablation. We analyze the impact of temporal alignment in our proposed
RL objective for matching sequential targets. Results averaged over 10 seeds.
F
Aligner model ablation study
Linked to Section 3]
To establish the importance of the aligner network, in Table we report the results of additional ablations:
GenRL aligner: this is ablation of GenRL where the the language prompt's embedding is directly fed into the connector, rather than processing first with the aligner;
TD3-V and WM-CLIP-V + aligner: embedding using GenRL's pre- pre-trained aligner. Then, we use it to compute the cosine similarity for the reward function, as for the original baselines.
these ablations, we first process the language prompt's
Table 5: Aligner ablation study.
GenRL no aligner
GenRL
WM-CLIP-V WM-CLIP-V aligner
TD3-V
TD3-V + aligner
cheetah
0.32 0.02
0.93 0.0
0.82 0.17
0.84 0,02
0.31 0.09
0.77 0.04
stckmann
walker
0.19 + 0.01
0.75 + aot
0.70 + 0:2
0.74 = 0:01
0.56 0.94
0.48 + 0.04
kitchen
0.25 0.00
0.76 0.08
0.71
0.32
0.27
quadrupod
0.17 = 0.00
0.91 + 0.02
0.81 0.04
0.76 0.05
0.32 2 0.04
0.33 4 0.04
overall
0.17 0.00
0.76
0.67
0.68 0.02
0.40
0.42 0.02
We can observe that: the aligner mechanism crucial GenRL's functioning. ii) processing the language embedding in the reward function of the WM-CLIP-V and TD3-V basclines changes performance on some tasks (performance domain varies). However, using the aligner provides no advantage overall.
We believe the aligner is very important in GenRL because its output, the processed language embedding, is fed to another network, the connector. If the language embeddings were not processed by the aligner, they would have been different from the embeddings used train the connector, which the visual embeddings.
Instead, for the baselines, we process the language embedding with the aligner and then a similarity score with the visual embeddings. This overall renders very similar performance to no aligner processing. hinting that the aligner network doesn't improve the cosine similarity signal. At the same time, this also suggests that the aligner network doesn't it hurt the generality of the VLM's embeddings, as the cosine similarity after processing the embedding provides similarly useful signal before processing.
compute
17

---

G
Comparison with LIV
[inked to Section 4]
We also tested all baselines with the LIV's representation [0] in the Kitchen tasks. We used LIV's open-source code download and instantiate the model. Note that the available model the general pre-trained model, the one line-tuned for the Kitchen environment.
LIV's results confirm original paper's claims (see Appendix G4) that the representation does not work well for vision-language rewards without fin-tunnin on somain-specific lisoonlanguaae pairs (which are unavailable in our settings, we use no language annotations).
Table 6: Comparison with LIV 40 on kitchen tasks.
IQL LIV
TD3-BC LIV
TD3 LIV
WM CLIP LIV
GenRL
kitchen microwave
0.03 ox3
0.0 a 0.0
0.2 = 0.16
0.0
0.97 0.32
kitchen light
0.53 a 0.24
0.05 a 0.04
0.0 0.0
0.0
0.46 a 0.09
kitchen burner
0.2
0 0.0
0,0
0.0
0.62 0.07
kitchen slide
0.03 0.00
# 0.03
0.67 0.27
1.0 0.0
overall
0,20 0.11
0.02 a 0.02
0.05 0.07
0.17 0.12
0.76 0.08
H
Extended Discussion on Data-free Policy Learning
Linked Section
(a) Offline RL
(b) Offline RL VLM
Requires reward-enginenn engineered task!
task prompt
observations, actions, rewards
observations, actions
rewards
Offline
Agent
Offline dataset
Pre-trained VLM
Agent
(c) GenRL (offline policy learning)
(d) GenRL (data-free policy learning)
task prompt
task prompt
actions
No data
actions
sample states internllyy
observations
latent states,
latent states, rewards
rewards
Agent
Agent
Offline dataset
Pre-trained
Pre-trained MFWM
Figure 9: Representing how the different agents process the data for policy learning. By removing data dependencies actions (on- (on-policy learning in the model's imagination), rewards (computed using only the prompt and latent states, using Eq. (3). and observations (by sampling latent states within the model), GenRL agents can be adapted for new tasks in a data-free fashion.
Offline RL methods (Fig. 9 learn from offline datasets of observations, actions and rewards.
Offine RL methods (Fig- 9b), combined with VLMs, can learn to perform tasks zero-shot from new prompts, but they need sample observations and actions from the dataset for computing rewards and for policy learning.
GenRL (Fig. (l)) needs sample (sequences of) observations from the dataset, infer the initial latent states for learning in imagination. Afterwards, rewards can be computed on the imagined latent sequences, enabling policy learning.
Data-free free GenRL (Fig. Id),. samples the initial latent states internally by combining: (i) random samples of the latent space. (ii) randomly sampled embeddings, which are mapped to "actual embeddings" using the aligner, and turned into latent states, by the connector. Thus, policy learning requires data sampling all.
18

---

model in providing accurate Initial states distribution. Uniform sampling from the latent space of the world model often results in meaningless latent states. Additionally, the sequential dynamics model of the MFWM, using GRU, requires some 'warmup' steps to discern dynamic environmental attributes, such as velocities.
To address these issues we perform two operations. First, we combine uniformly sampled states from the discrete latent spaces with states generated by randomly sampling the connector model, as sequences generated by the connector tend to have more coherent structure than random uniform samples. Sccond, we perform rollout of five steps using mix of actions from the trained policy and random actions. This leads to varied distribution of states, containing dynamic information, which use the initial states for the learning imagination process.
I Scaling to complex observations
Linked to Section 5]
Stone cave
Beach
Taiga
Lake
Snow
Badlands
1x model
8x model
Figure 10: Decoded language prompts in Minecraft
Generalist embodied agents should able to scale to open-ended learning settings. Using GenRI, we explored this by training an agent in the Minecraft environment using a small dataset collected by a DreamerV3 agent [26]. Note that GenRL is also trained using similar experimental settings as DreamerV3, e.g. setting the environment in the same way. The primary challenge found was the model's difficulty reconstructing complex observations this open-ended environment.
Reconstructing complex observations common issue with world models [T]. overcome this limitation. while keeping the method unaltered, we attempted to scale up the number of parameters the MFWM. Qualitative reconstruction results are presented in Figure TO We observe that the agent is able to identify different biomes from language, significantly blurrier compared to the other environments reconstructions gain some details the results still highlight the difficulty targets from prompts.
with the smaller size of the model. However, the reconstructions analyzed. When using larger model, the
While this might not be an issue for simple high-level tasks, c.g. 'navigate to a beach', inferring unclear targets might make it difficult to perform more precise actions, c.g. 'attack a zombic'. Future research should aim to address this issue, for instance, by improving simple GRU-based architecture, leveraging transformers diffusion models improve the quality of the representation []
J Additional limitations
Linked Section
Pre-training data requirements. As developed framework, observed that, in order solve complex tasks, the agent requires some expert data/demonstration the complex behavior: We observed and analyzed this aspect the experiments of Section 4.3. We believe this limitation is, to some extent, inevitable, as data-driven Al agents need to observe complex behaviors during training in order to be able to replicate them.
In this work, we used some exploration data, obtained using an exploration agent (Plan2Explore), and some task-specific data, collected using expert RL agent (DreamerV3) Alternatively, one could investigate the adoption of small set of demonstrations.
Precise manipulation skills. Adding additional tasks the Kitchen environment, for lesting multi-task generalization, showed to harder than for other domains. This to difficulties of exploring meaningful behaviors in manipulation environments.
We are able to use GenRL to retrieve the four tasks present in the dataset and we are generally able to achieve similar tasks, such as "reach the microwave". From decoding the language prompts, we also see that GenRL sensibly decode prompts such as "moving the left" or "staying still". more 'unusual" prompts such as
19

---

"swan pose" (where the robot arm would imitate the pose of swan). While these prompts show that the system works well in this environment, of these tasks would be particularly useful.
When we prompt the system with struggles. For instance, if ask open double door cabinet on the top left, the agent may reach for but will not open The reason behind this is that probably never opened that cabinet the training dataset and thus it's impossible to reproduce such behavior in an offline manner. The same issues are most likely present with all the baselines well, as they all use the same dataset and foundation models for training.
interesting tasks that out of the distribution, the agent generally
For the locomotion domains, instead, the exploration data is varied enough that new tasks, outside of the tasks present in the training dataset, are enabled through the exploration data/
K Detailed results
[Lnned to Scction 4
1.0
0.8-
0.6
quadfuoed live
sesdruped dean
Juckman layinng
Vicomen one
aicemen ying down
Wilmun tops
0.0
0.6-
0.4
snewe
crAraa kuaae poce
chdkran headenee
ctnoraa beako
diekmon bandcap
diverh shadin
dheenh Viln
0.6
0.4
02
Figure 11: Multi-task generalization detailed results. Results averaged over 10 seeds.
20

---

Asllerttnd
valorwak
stand
1.0
1.0
0.8
0.6
0.4
0.2
vuller hab MOC
Nor Nary dtan
Velte at Ene0
watter LEDOO 2014
weller Ropag
quat-aao hare
0.4
0.2
0.0
wushupol las
Juskmnn
Hegn
Wichmun hpp
02-
0.0
1.0
0.8-
06-
Figure 12: Data-free RL detailed results. Results averaged over 10 seeds.
Figure 13: Training data distribution detailed results. Results averaged over 10 seeds.
21

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: All the claims presented in the abstract and introduction are further discussed while presenting the method and the results. The experiments support the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the
results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the approach are discussed in the experiments and analysis sections of the paper. All potential limitations are also restated and summarized in the final discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" scction in their paper.
The paper should point out any strong assumptions and how robust the results are violations of these assumptions (c.g., independence assumptions, noiseless settings, model well- spel--ccffiiction asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't a acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve integrity the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not provide theoretical results. The new objectives and models introduced are evaluated empirically.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
22

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly refereneed.
4. Experimental Result Reproducibility
Question: Daes the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The dctails to reproduce the results are provided in the experiments section and in the supplementary material (appendix). To further facilitate reproduction of our work, the code and data will be made publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the paper includes experiments,
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to deseribe the particular way they provide for reproducibility. In the of closed-sourg source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
See the project website for access data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines htttpp::/niips.ccppbblle
guides/Codee
for more details.
While we encourage the release of code and data, we understand that this might not be possible,
so "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g.. for a new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/ //tp///....../////// cc/public/ qut.dess/oooeSuubllessssooP..... for more details.
23

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why. At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details to reproduce the results provided in the experiments section and in the supplementary material (appendix). To further facilitate reproduction of our work, the code and data will made publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
TTe full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Dnes the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We provide standard errors in tables and error bars in the plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error hars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example. train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a
library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
should be clear whether the error mean.
the standard deviation or the standard error of the
is OK to report I-sigma error hars, but one should state it. The authors should preferably report a 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
For asymmetric distributions,
8. Experiments Compute Resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
Information about the compute exploited is provide in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
24

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethicss http:///wuurppp.ccccppppd/////aa
Answer: [Yes]
Justification:
The work does not deviate from the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [NA]
Justification: The work has no societal impact at the current stage.
Guidelines:
The answer NA means that there is no societal impact of the work performed. If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e-g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed point out that generic algorithm for optimizing neural networks
could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misusc or dual-use should be released with necessary safeguards allow for controlled
of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should
describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
25

---

Question: Are the creators or original owners of assets (e.g.. code. data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
corresponding papers the models and the environments used our experiments, e.g. SigLIP, InternVideo2, DmControl.
Justification:
Guidelines:
The answer means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL..
The name of the license (e.g., CC-BY 4.0) should he included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswithccseeo thcode. com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are alongside the assets?
introduced in the paper well documented and is the documentation provided
Answer: [Yes]
Justification: New assets will be released and documented at later date.
Guidelines:
The answer means that the paper does not release new assets. Researchers should communicate the details of the dataset/coue/moodel as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsoureing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: the paper does not involve crowdsoureing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval. you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as institution conducting the review,
27


Reviews:
[{"review_id": "MUgfCxgHWG", "reviewer": "Reviewer_X1kr", "summary": "The paper wants to leverage the large-scale pre-training of foundation models trained on internet data to train a world model for embodied agents that generalizes across tasks and domains. This is done by training a world model in the standard way, but in addition training aligner and connector networks that (1) map language embeddings to video embeddings and (2) map video embeddings to world model latent states. At inference time, this allows conditioning the world model on a task language prompt and then training in imagination to learn policies.", "questions": "- What is the value of $k$ (number of frames predicted by the connector)? What duration does this correspond to? What happens if the task is longer than this duration?\n-  Just to confirm, in the offline RL evaluation, first the world model is trained on the offline data (only for that particular domain) and then the policy is trained in imagination? In that case, why is there a difference in the time taken for the actor-critic to converge in the data-free setting (see line 524)\n-  In the single task, offline RL case, is the aligner trained with only as many language prompts as the tasks in that domain? If that is the case, it would be trained to reconstruct $e^{(v)}$ corresponding to many different videos in the offline dataset, some of which might contain suboptimal trajectories which have nothing to do with the language prompt. How can we expect the aligner to learn anything useful in this case?\n- If the aligner is trained only on a few language prompts, how is it able to generalize to new tasks?\n- What exactly is the multi-task generalization setting? In this evaluation, does the method get access to offline data from the OOD task? If yes, how is it used to train the policy? If no, how are the model-free baselines trained in this setting?", "limitations": "Yes, the authors adequately assessed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- On the website, the reconstruction results from language and video are nice and quite unexpected (I'm unsure why the aligner and connector networks are able to generalize to new prompts) \n- The problem the paper is trying to solve is relevant, especially given the mismatch in data availability between embodied and vision / language settings", "weaknesses": "- The main claim of the paper is strong generalization performance, leveraging the internet scale pre-training of video-language models. The bottleneck is the generalization ability of the networks which map embeddings from the video-language model to the world model latent states, and on the quality of the world model itself. I don't see why the aligner and connector should generalize.\n- Given the main claim, I would like stronger baselines / ablations in the generalization and data-free settings. Currently, there are no baselines in the data-free case which makes it impossible to assess how well the method generalizes.  \n- Many of the experimental details are unclear in the paper (please see my questions). I encourage the authors to explain these better in the rebuttal and camera-ready, and also provide some intuition for why their method is better than the baselines.  \n- In the single task, offline RL case, all the baselines are model-free, whereas the proposed method utilizes a model. I would have liked to see at least one model-based baseline to confirm that the improvement is because of the better reward signal and not because of the model-based optimization.\n- In the single task, offline RL case, reward is computed by looking at the similarity between the representations of the task prompt and the image / video. In the case of the base lines, these representations are fixed (eg. CLIP / Intern2Video representations), whereas for the proposed method these are taken from the last layer of the model learnt on the data itself. This is also reflected in the compute budget - the model takes 5 days to train (in addition to the 5 hours of training in imagination).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments.\n\n**Connector-aligner generalization**\n\nFirst, we would like to make clear that, as stated in multiple parts of the paper, the connector and aligner networks are trained using **vision-only data** and **no language annotations**. We have provided additional clarification for this in our main rebuttal message.\n\nThe connector-aligner synergy in the MFWM allows to rely on the multimodal foundation model's knowledge for grounding visual and language prompts into the embodied domains dynamics. The extent to which this system generalizes depends on the knowledge possessed by the two main components: (i) the foundation VLM and (ii) the world model. Given a certain concept, expressed through a prompt, GenRL is able to understand the concept and learn the corresponding behavior given two conditions: \n\n* the VLM has been pre-trained on vision-language pairs that allow understanding that concept. We assume this is the case as foundation models are trained on massive datasets; \n* the world model has been trained on a dataset that contains visual observations that match the given concept. Given a mixed dataset, containing many tasks and exploration data, we know this is true for the tasks in the dataset, but we cannot know for other tasks beforehand.\n\nIn practice, we show that the tasks we deliberately put in the dataset can be retrieved (Behavior retrieval experiments) and the system also generalizes to many other prompts/tasks. This means the behaviors are likely to be found in the exploratory data or as part of the tasks data. We assess this generalization capability quantitatively, in the Multitask generalization experiments, and qualitatively, with the visualizations on the website.\n\n**Stronger baselines**\n\nAs we describe in the main rebuttal message, we added new model-based baselines to our main experiments. These baselines follow the recommendation of Reviewer Yb6H of learning an \"inverse connector\" from the world model representation to the VLM representation (GenRL does the opposite). This baseline is stronger than the model-free baselines and it allows as to more clearly establish the main ingredients that contribute to the stronger performance of GenRL: (i) using a video-language model helps in dynamic tasks, (ii) using a model-based algorithm is beneficial, (iii) the connection-alignment system presented outperforms the other straightforward way of connecting the two representations (world model and VLM).\n\nAs for the data-free settings, we do not have knowledge of any other data-free behavior learning strategies. As, to the best of our knowledge, this is a new paradigm for behavior learning, we aim to establish its performance compared to more established paradigms, e.g. offline RL. We have provided additional information about our unique data-free pipeline in the main rebuttal message.\n\n**Missing implementation details**\n\nThe following information has been added to the paper:\n\n**Value of k**: we adopt a number of frames $k=8$ (as stated at Line 158. We also added this information at Line 104 to make it clearer)\n\n**Multi-task generalization settings**: all methods are trained on the same dataset, made of structured data (the behavior retrieval tasks) and unstructured data (exploration data). Assuming that the data distribution is varied enough, many behaviors that are not part of the behavior retrieval tasks are likely observed. The goal of all methods is to understand the received prompt and learn the \"best-matching behavior\". This can be done from the given dataset, for the model-free RL baselines, or in imagination after pre-training the world model, for the model-based baselines and GenRL. We observed that GenRL is able to succeed in many tasks that we didn't deliberately add to the dataset.\n\n**Training time**\n\nGenRL pretrains a single world model per dynamic, for 5 days, and uses it for learning behaviors in imagination, for 5 hours per task. Model-free RL methods require 7 hours but have no pre-training stage. On a single GPU, model-free RL is faster to train for a small number of runs. GenRL starts becoming advantageous when using the world model for training for more than 60 runs (which is often the case, considering the number of runs = N seeds x M tasks per domain).\n\nWhen adopting the data-free learning strategy, GenRL doesn't rely on the dataset at all (see main rebuttal message). This halves the time required for training, as there are no data transfers between the CPU (where the dataset is loaded) and the GPU for training.\n\nWe hope to have satisfied all the reviewer's concerns and we look forward to receiving updated feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xaOe9RDTac", "reviewer": "Reviewer_jfoT", "summary": "This paper proposes to combine a DreamerV3-style world model with a pretrained vision language model (VLM). By training two small adaptors to align the latent space of the VLM with that of the world model, the aligned representations from the VLM can be used as a reward signal to train agents in the world model.\n\nThe training process consists of two main parts. \n\n1) There is a large offline dataset needed in the environments of interest (prompt and trajectories of states and actions), generated by expert RL agents and a random policy. This trains the world model and the adaptors. Each environment (domain) uses a separate world model.\n\n2) Actor-critic agents are trained purely within this world model’s imagination, a separate policy for each task. The paper shows these agents outperform standard model-free offline RL methods trained on only the large offline dataset. It also shows some effectiveness at generalizing to new tasks within an environment, specified with a new text prompt.", "questions": "See weaknesses.", "limitations": "Fine.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "- The core idea of the paper is very nice. There is a lot of interest from the community in working out how to get value from the broad general knowledge locked away in LLMs and VLMs, into RL agents. This paper offers a novel way to attack this – to my knowledge world models have not been used in this context before. \n\n- The results are not dazzling, but they indicate the approach works and it outperforms standard (though perhaps weak) offline-RL baselines. Section 4.2 shows promise in generalizing to knew text prompts in an existing environment.", "weaknesses": "My main criticism of the paper is that the narrative oversells what the core work actually supports. I detail examples below. Overall I’d suggest either presenting the work that has been done comprehensively in a more reserved manner, or adding the required work to support the broader claims and experiments. Either way, I think changes would be large enough to require a resubmission.  I’m disappointed to not be able to give the paper a higher score as I liked the main idea. \n- The capability of the model to condition on visual goals is presented as a main functionality of the model – featuring in the first figure, the abstract, and throughout the paper. But the only evidence to support this is a very brief and qualitative experiment (Figure 6a). Everything else is conditioned on text. I am of the opinion that conditioning on visuals would likely work, but the paper must present good evidence to support this.\n- Several aspects of the title ‘Multimodal foundation world models for generalist embodied agents’ are misleading. 1) Only one modality is really tested (as in prior point). 2) ‘Foundation world models’ suggested I'd see a single very general world model. But in Appendix D is an important detail -- each environment learns a separate world model, so they are only general or foundational within a specific Mujoco embodiment. This kind of detail is important and should be honestly discussed in the main paper. 3) A ‘generalist agent’ is referred to, but every agent in the paper only performs a single specialist task, there is nothing general about the agent’s themselves.\n- The method is reported as needing ‘no language annotations’ (line 42). This is not true. The large offline dataset requires text prompts accompanying each trajectory.\n- The paper claims to be ‘the first large-scale study of multitask generalization from language in RL’ (line 165), but I can think of others. Language table is the first that comes to mind. \n- One of the motivations for the work is that reward functions can be hard to specify, while language is a more natural form. However, the large offline dataset is generated by using multiple expert agents which need reward functions.\n- ‘Data free RL’ is suggested as a new paradigm for foundation models in RL. I’d argue that this is simply know as zero-shot generalization to most in the community.\n- Main experiments are presented in Table 1. Whilst the offline-RL methods are one comparison point, I’m not sure how comparable they are, since they are all model-free while GenRL is model based. Are there any model-based variants that would be easily considered as baselines? The differences are reflected in the different compute times required – GenRL takes 5 days for world model training +5 hours per policy, while the baselines take 7 hours per policy. This seems like an unfair comparison, especially to withhold the detail to the appendix.\n- Results in Minecraft are briefly mentioned in Section 5. But so few details are given that I am lost as to what it is showing. This should either be removed or full details added. \n- The paper presents a new stickman environment. But details are sparse. The authors have failed to correctly identify this in Checklist Section 13.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments.\n\n**Learning from visual prompts**\n\nIn order to support our claims with empirical evaluation, we have provided results of behavior learning from video prompts. The results can be found in our main rebuttal message and the videos on the website.\n\nWe have three main observations about these experiments: (i) GenRL performs overall better than the baselines, (ii) the performance from video prompts are generally close to the performance from language prompts, (iii) leveraging the foundation VLM knowledge allows to generalize in very different visual settings.\n\n**Title**\n\nWe hope to have resolved the reviewers' concern about the multimodality nature of our work with the additional experiments. \n\nThe term we coined as a shortand for our MFWM models is not used to indicate a class of foundation (world) models. Instead, as we state several times in the paper (Lines 9, 42, 105, Figures 1 and 2), it is used to indicate world models whose representation is connected to the knowledge of multimodal foundation models. In order to make this more clear, we added an hyphen to our name (\"Multimodal-foundation world models\") so that it is clear that the first two words are used as an \"adjective\" for the world model, rather than for indicating a multimodal foundation model.\n\nWe understand the definition of a \"generalist agent\" may be ambiguous, as we have a \"generalist\" model that allows multi-task and multimodal prompts generalization by training multiple specialist policies, as correctly stated by the reviewer. \n\nGiven the above observations, we will update our title to: \n**\"Multimodal-foundation world models for generalization in embodied domains\"**\n\nMoreover, following the reviewer suggestion, we moved the statement about training one world model per dynamic, currently in the Appendix, to the main paper (Experiments section).\n\n**No language annotations**\n\nThere must have been a misunderstanding, probably due to an ambiguous statement at Lines 176-177. However, we use **no language annotations** in our datasets. Please, see our main rebuttal comment about this.\n\n**Large-scale study**\n\nTo the best of our knowledge, our work is the first large-scale study where agents can generalize to as many embodied tasks and domains (35 tasks, in 5 embodied domains). Nonetheless, we will remove the adjective \"first\", if the reviewer finds it inaccurate.\n\n**Data collection**\n\nAs we developed our framework, we observed that, in order to solve more complex tasks, the agent requires some expert data/demonstration of the complex behavior. We analyse this in our \"Training data distribution\" experiments. We believe this limitation is, to some extent, inevitable, as data-driven deep learning agents need to observe complex behaviors during training in order to be able to replicate them. \n\nIn this work, we used an expert RL agent (DreamerV3) to collect the data for us. Using a small set of demonstrations might be an alternative. We added further discussion about this and other limitations in the Appendix of the paper.\n\n**Data-free**\n\nWe discuss the differences between Offline RL, GenRL with data and GenRL data-free in our main rebuttal comment. \n\nGenerally, we would argue that a single offline RL policy cannot generalize to new tasks zero-shot, without (re)training on the data. Skill-based approaches hold the potential to do so. However, training skill-based policies that cover many different complex behaviors in a zero-shot fashion is an open research question. Our data-free strategy offers an alternative view on this problem.\n\n**Comparison with baselines**\n\nAs we describe in the main rebuttal message, we added model-based baselines to our main experiments. These baselines follow the recommendation of Reviewer Yb6H of learning an \"inverse connector\" from the world model representation to the VLM representation (GenRL does the opposite). Please see the main rebuttal message for further details.\n\nGenRL pretrains a single world model per dynamic, for 5 days, and uses it for learning tasks in imagination, for 5 hours per task. Model-free RL requires only 7 hours, as it takes longer to converge, but has no pre-training stage.  On a single GPU, model-free RL is faster to train for a small number of runs. GenRL becomes advantageous when using the world model for training policies for more than 60 runs (runs = tasks x seeds). If using data-free learning (3 hours per task), the advantages become significant after 30 runs.\n\nGiven the nature of our work, focusing on behavior generalization performance, rather than on computing budget, and the limited space for the main text, we have kept this information (now including the above discussion) in Appendix.\n\n**Experimental settings**\n\nTo make room for the new visual prompted experiments, we are moving the Minecraft results in Appendix. We have also added more details about the training setup we adopted. As we stated in Line 295, we used DreamerV3 to collect the data and thus used the same settings as their work. Nonetheless, we also summarized the main information in our work.\n\nThe Stickman environment is based on the Walker environment from the _dm_control_ suite. We designed the Stickman environment to explore tasks that require upper body limbs (e.g. boxing, doing a handstand) without the complexity of training a humanoid (which requires a significantly larger amount of data to be solved [1]). The number of joints is increased by 4: 2 joints per arm, one is for the shoulder, the other for the elbow. The total number joints is 10. The action space is normalized to be in [-1,1] as all _dm_control_ tasks. The robot also presents an head, to resemble a humanoid.\n\nFurther details can be found in our accompanying code implementation, which we shared with the Area Chair.\n\n[1] Mastering Atari with Discrete World Models, Hafner et al, 2020\n\nWe hope to have satisfied all the reviewer's concerns and we look forward to receiving updated feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ga9LEYTBXF", "reviewer": "Reviewer_bBQY", "summary": "The paper looks at a method for leveraging foundation multimodal models for learning world models in RL. They do so by aligning the latent space of a video language model with that of a generative model that can be used for learning in imagination. This is done by training connector-and-aligner networks . The rewards for a task can then be derived by measuring the cosine sim between representations of the states visited by a policy and the states generated by the connector aligner network when it is conditioned on a language-based task prompt. A policy can be optimised to maximise this alignment based reward.", "questions": "1. Why is setting b=imagination horizon the same as doing no alignment? (Line 160)\n2. I’m not completely sure how you train the aligner-connector network: is it done by 1) using images collected from the downstream domain (in this case from the mujoco sim), 2) getting their VLM visual embedding and their world model encoder embedding and aligning those? As for the text part, is this done by corrupting the VLM visual embedding (to approximate the language embedding) and aligning it again with the world model encoder?  What is the policy used to collect the data and resulting data distribution? I understand that Fig 5 is somehow related to this question but this could be made clearer. For eg. which task’s policy is chosen to collect the data to train the MWFM for the results in the main table (how is this policy related to the task being evaluated)? \n3. The discussion around Figure 5 is not very clear to me - how do we infer that “’run’ data proves more generalizable than  ’walk’ or ’stand’ data across tasks“ - the figures suggest that training on ‘stand’ led to the highest rewards for downstream tasks\n4. “This can be explained by  the fact that the target sequences that GenRL infers from the prompt are often slightly in motion“ - could you explain why that would be the case (it inferring the closest matching state as one that is in motion)?", "limitations": "The paper includes a brief discussion on limitations of their method.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Transferring foundation model knowledge to improve policy learning is an open problem of interest to the community. \n\nThe paper provides a successful recipe for aligning a foundation model with the world model for a specific domain that we want to do policy learning in. \n\nThe paper is written well.\n\nI'm currently being conservative in giving a borderline accept score, since some aspects of the method are not clear to me (I have addressed this in my questions below) - but I will be happy to raise my score after engaging with the authors once they have addressed these questions.", "weaknesses": "1. I would have expected that simple tasks with clearly distinguishable static end states (such as standing) should have worked equally well with CLIP rewards, however the table shows a big difference between the proposed method and the image-language reward baselines even on those tasks, which leads me to think that the baselines may be missing out on some component that the proposed method has. What could be missing, or is this intuition wrong? \n2. The generations in Fig 6a are actually not accurate at all - many of the poses don’t correspond to the humanoid pose if you look closely and would actually optimize learning to strike the wrong pose if a policy is trained with it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments.\n\n**Image-language CLIP results**\n\n> simple tasks with clearly distinguishable static end states (such as standing) should have worked equally well with CLIP rewards\n\nWe agree with the reviewer's intuition and we believe the results confirm their statement. If we look only at the static tasks (stand tasks and kitchen tasks) we see that the performance of Image-based and Video-based baselines are very comparable. We observe this also for the new model-based baselines (please, see main rebuttal message). GenRL, which uses a video-language VLM, tends to outperform the other approaches. \n\n**Video prompts' poses**\n\nThe visualizations in Figure 6a are the interpretations of what the visual prompt provided would look like in the embodied domain, according to the model. We agree these do not match the poses provided exactly, but only in their \"semantic\" meaning (e.g. doing crunch abs, despite the legs position being different). This issue is due to two aspects of our framework: (i) VLMs shine with semantic interpretation but struggle to represent precise geometrical information, and thus they struggle with precise pose estimation, (ii) the world model may have not seen certain poses in its training dataset, and thus it will provide the closest behavior to the prompt. Despite this limitation, we found that our video prompting allows to learn behavior from a variety of visual conditions, e.g. draw sketches, very different camera views. Please, see our main rebuttal message for the additional experiments.   \n\n**Sequence alignment**\n\nIf we set $b$=imagination horizon we only have one possible comparison between the two sequences.\n\nExample with a sequence of length 3. Target sequence states (T), agent sequence states (A).\n\nb = 1 (initial state alignment), 3 possible alignments\n\n| T | |   |\n|---|---|---|\n| A | A | A |\n\n|  | T |   |\n|---|---|---|\n| A | A | A |\n\n|  | |  T |\n|---|---|---|\n| A | A | A |\n\nb = 2, 2 possible alignments\n\n\n| T |T |   |\n|---|---|---|\n| A | A | A |\n\n|  | T | T  |\n|---|---|---|\n| A | A | A |\n\n\nb = 3 (no alignment), 1 possible alignment\n\n| T |T |  T |\n|---|---|---|\n| A | A | A |\n\n**Training procedure and data collection**\n\nWe think the reviewer correctly understood our training procedure, and which we further clarify in our main rebuttal comment for clarity. As currently stated in Lines 174-176, we collect data using two kind of agents: (i) a Dreamer V3 agent learning to perform the task (we added the details about the agent being DreamerV3 in the main text) and (ii) a Plan2explore agent, collecting exploration data in the domain.\n\nThe datasets are detailed in Appendix. However, we provided detailed results per task on the behavior extraction tasks so that a reader can quickly see which tasks are contained in the dataset (indeed, the behavior extraction tasks). We have made this clearer in the main text, by stating it at Line 190 as follows:\n\n\"Behavior extraction. We want to verify whether the methods can retrieve the tasks behaviors that are\ncertainly present in the dataset. *For these tasks, the replay buffer of an expert agent learning to solve the tasks (DreamerV3) was already included in the mixed dataset used for training*\"\n\n**Training data distribution**\n\nWe believe there is a misunderstanding in the way Figure 5 visualizes the data. The labels (all, expl, run walk, stand) indicate the portion of the data used for training the agent. Each plot represents performance of the agent trained with different portions of data on a task / group of tasks. Thus, the agent trained only on 'stand data' is the one with the lowest performance. We changed the labels to \"all/expl/run/walk/stand **data**\" to improve clarity.\n\n**Static poses matching**\n\nExamples of predicted sequences being in motion for static tasks are present in our website. For example, if we look at the decoding of the \"downward facing dog\" prompt, we observe that the pose of the agent is in motion, rather than being completely static. We observed similar issues with some of the task prompts.\n\nWe hope to have satisfied all the reviewer's concerns and we look forward to receiving updated feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9JrOmW91C7", "reviewer": "Reviewer_Yb6H", "summary": "In this work, the authors propose learning a pixel-based reconstructive world model, and then separately learn networks to convert the representations of a pretrained VLM into the learned world model latent space.  By using a VLM trained via contrastive alignment, this essentially enables the projection of both image as well as text inputs into the latent space of the world model, and therefore simple similarity can be used to provide rewards for downstream policy learning.", "questions": "Why Stickman instead of Humanoid?  Humanoid has been able to be solved with pixel-based world models in the past (Appendix A of Dreamerv2).  What are the specifications of Stickman and with what criteria was it designed?\n\nWhy did the authors use a simple GRU?  Why was a more advanced world model not used, like RSSMs?  Was this tested or ablated over?\n\nWhy were there no multi-task generalization experiments performed for Meta-World kitchen?\n\nWould the authors consider Text2Reward and other approaches that learn a reward function as data-free RL, as there do not need additional data to generalize to learning new policies?  Alternatively, the data-free RL paradigm sounds like zero-shot generalization for policy learning, which is already offered by VLM-RM and other similar works.\n\nWhy did the authors choose to generate the video demonstrations synthetically, rather than use actual natural video clips?  Would the performance not be better when using natural videos which are more in line with what the base VLM was trained on?\n\nHow are the rewards computed using temporal alignment?  Essentially, are only the rewards for the most-aligned segments across the target trajectory and the agent used as rewards, and for all other timesteps a 0 reward is provided?  This computation seems rather expensive for long-horizon trajectories.", "limitations": "The limitations section seems acceptable.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This reviewer is a supporter of the idea of unifying the representation spaces of a large-scale pretrained VLM and that of a world model.  This author appreciates the benefits: matching behavior of a world model with natural language can enable text-conditioned generalization.  The preliminary experiments show promise.\n\n**Originality**: This work appears decently original.\n\n**Quality**: This works quality is acceptable.\n\n**Clarity**: The clarity of the work is acceptable, the core ideas are communicated clearly.  However, there are lots of open questions surrounding this work that could be elaborated upon further.\n\n**Significance**: This work appears to be decently significant, as a preliminary investigation in this space.", "weaknesses": "Chief amongst the weaknesses of this work is the limited environments applied to, and also the limited baselines (essentially, the only existing work the authors compare against is VLM-RM).  The authors can consider comparing against other forms of text-conditioned policy learning, such as LIV for the kitchen setting, or Text2Reward and similar approaches for the general case.  It also seems a strange setup to normalize in-between expert and random, and report results in this way.  This reviewer is unaware of prior work that performs evaluations in this way.  What is the rationale behind this evaluation strategy compared to what is used in prior work?\n\nDetails about certain components of the model and how they are implemented are sparse.  For example, is the aligner a video generative model (text-to-video model)?  How is it implemented?\n\nIt is a bit dissatisfying to rely on a corrupted version of vision as a language embedding.  It seems strange that the aligner should on one hand be learning to bring language embeddings meaningfully across modalities to the image/video space, which the authors motivate is necessary because of the multimodality gap.  However, the authors then treat language embeddings as a noisy corruption of a video embedding - so essentially the training objective for the aligner is essentially a denoising?  And rather than bridging a modality gap, the aligner is essentially a denoiser?\n\nWhy do we not learn the reverse direction, where we optimize a world model's latent space that projects into existing VLM space?  This design decision is not elaborated upon, but seems more intuitive to this reviewer.\n\nFrom the video demonstrations, on the associated project website, it is rather unclear what is happening.  Are Behavior Retrieval videos from expert policies in an offline dataset that are matched with a particular text prompt/input video?  What are those text prompts/input videos?  It's not clear what the retrieval setup is.  For Multitask Generalization, it is also not obvious what the corresponding prompts are.  Furthermore, the results for multitask generalization do not seem smooth and natural, despite being simplistic DM_Control environments (especially the case for their proposed simplified Stickman environment) and they are missing Kitchen environments.  In the end, it appears that their method is still good as a retrieval technique (retrieving already-achieved expert behaviors in \"Behavior retrieval\") due to the underlying VLM, and is decent at reconstructing video prompts, but still suffers in terms of learning coherent policies (e.g. what is visualized in \"Multitask generalization\"), which is ultimately what is of interest.\n\nFor the video prompts that are decoded, it appears as if almost all of them are rather stationary (with the exception of the cheetah/dog example and the human dancing example) - they collapse to a stationary goal pose.  Perhaps this is because the clips are so short (8 frames) that essentially it boils down to pose-matching.  It is not obvious that this is that beneficial in supervising motion; so why does this improve upon static image supervision?  Indeed, many of the results that are shown learned by the policy are rather stationary and do not have much movement (most are just jitters around a stationary pose).  It then begs the question how this approach improves upon just a static goal supervision.  However, the authors simultaneously find that in static tasks other methods outperform the authors' approach.\n\nThis reviewer pushes back on the term \"data-free RL\", as there still needs data (and interaction data) to learn their method.  This is a very confusing terminology, and honestly the generalization comes from the large-scale pretrained VLM - it would be more appropriate to reuse the terminology of zero-shot reward models or zero-shot policy learning used in prior works across alignment methods (vision-language models are zero-shot reward models for reinforcement learning, [Rocamonde, '23]) and diffusion (text-aware diffusion for policy learning, [Luo, '24]).\n\nThis reviewer really enjoys the work but believes there are many open questions that warrant further explanation.  Furthermore, the evaluation suite (environments) and comparison suite (benchmarks) is rather weak.  The idea is indeed neat, but the execution leaves much to be desired, and therefore this reviewer believes the work is of borderline quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments.\n\n**Experiments**\n\nWe added new experiments, including stronger model-based baselines. Details are in the main rebuttal message.\n\nWe also tested all baselines with LIV's representation in the Kitchen tasks. We used LIV's open-source code to download and instantiate the model. Note: the available model is the general pre-trained model, not the one fine-tuned for the Kitchen environment.\n\n|                   | IQL-LIV     | TD3+BC-LIV  | TD3-LIV     | WM-CLIP-LIV | GenRL       |\n|-------------------|-------------|-------------|-------------|-------------|-------------|\n| kitchen microwave | 0.03 | 0.0    | 0.2   | 0.0    | 0.97  |\n| kitchen light     | 0.53 | 0.05  | 0.0    | 0.0    | 0.46  |\n| kitchen burner    | 0.2  | 0.0 | 0.0    | 0.0    | 0.62  |\n| kitchen slide     | 0.03  | 0.03  | 0.0    | 0.67  | 1.0    |\n\nLIV's results confirm the original paper's claims (see Appendix G4) that the representation does not work well for vision-language rewards without fine-tuning on domain-specific vision-language pairs (which are unavailable in our settings, as we use **no language annotations**).\n\nText2Reward requires a different experimental setup than ours. We focus on visual RL. The agent observes the environment only through images and doesn't have any privileged information about which entities are present in the scene e.g. joints, limbs, objects.\n\nNormalizing results using expert performance and random performance as max-min values is one of the de-facto standards for the Atari suite. This is referred to as Human Normalized Score (HNS), see [1].\n\n**Aligner**\n\nThe aligner is not a generative model and, as the reviewer suggests, it's closer to a denoiser network. In the main rebuttal comment, we have provided an additional explanation about how and why we reduce the multimodality gap by adding noise to the embeddings. Additional insights are available in related work [2,3]. \n\nAs described in the Appendix (Line 491-492), \"the aligner network employs a small U-Net, with a bottleneck that is half the size the embedding representation.\".  The embedding representation is 768-dimensional. Further details can be found in our accompanying code implementation, provided to the Area Chair.\n\n**Inverse connector idea**\n\nWe have implemented this idea and added it as a baseline (WM-CLIP) to our work. The reviewer's intuition is correct as the idea works. However, without the connecting-aligning process, the system does not perform as well for some prompts/tasks. Please, see the main rebuttal comment for further details.\n\n**Website visualizations**\n\nWe added the corresponding task labels in the website and we provided the language prompts in the Appendix.\n\nIn the videos, we observe that less natural behavior is correlated with a lower normalized score on the task. Thus, we expect those behaviors to be less smooth, e.g. see 'stickman flipping' or 'walker high kick'.\n\n**Dynamic tasks**\n\nWe provided additional results and visualizations, including more dynamic tasks. Please, see the main rebuttal comment and the updated website.\n\nNonetheless, we agree with the reviewer that the model struggles more to sync with input videos for dynamic actions. One of the reasons for this is that the control frequency of the agent is different from the motion frequency from the provided video prompt. This makes it hard to match the prompt accurately.\n\n**Data-free**\n\nWe provided an additional explanation in the main rebuttal comment. We believe that both Rocamonde et al, Luo et al, and Text2Reward fall in the category that we represent as \"Offline RL + VLM\", as they require data for training the behavior policy.\n\nGiven that the term \"data-free RL\" may be confusing, we have changed our naming convention to \"Data-free Policy Learning\", similar to Luo et al, which we now cite for reference.\n\n**Humanoid**\n\nWe designed the Stickman environment to explore tasks that require upper body limbs (e.g. boxing, doing a handstand) without the complexity of training a humanoid, which requires a significantly larger amount of data to be solved, as reported in the DreamerV2 paper.\n\nBuilding on the Walker model, the number of joints is increased by 4: 2 joints per arm, one is for the shoulder, the other for the elbow. The total number joints is 10. The action space is normalized to be in [-1,1] as all dm_control tasks.\n\n**Connector architecture**\n\nThe architecture of the connector network is the same as the RSSM architecture used for the world model, which employs a GRU for computing the hidden state. We corrected this detail in the paper. \n\n**Additional tasks**\n\nMeta-World environments adopt different environments for different tasks, i.e. with different objects, workspaces and thus visual dynamics. Instead, we focussed on evaluating on domains where we can learn a single MFWM for the environment's dynamics and use it to generalize to new tasks. \n\nThe Franka Kitchen environment allows for more tasks in a single environment (we evaluate on 4). However, it's hard to design new tasks as the number of objects to interact with is limited. \n\n**Video prompts**\n\nNatural videos actually work well (and often better) than AI-generated videos. We found intriguing the idea of generating the prompts using another GenAI model, rather than having to look for a video online. In the additional results, we chose to also show some natural videos.\n\n**Temporal alignment**\n\nBefore the aligned segments, the reward is computed using the initial state of the target sequence. This is currently stated at Line 155.\n\nWe hope to have satisfied all the reviewer's concerns and we look forward to receiving updated feedback.\n\n[1] A Review for Deep Reinforcement Learning in Atari: Benchmarks, Challenges, and Solutions, Fan el al\n\n[2] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning, Liang et al\n\n[3] Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data, Zhang et al", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Training with no language annotations

We stated several times that the system is trained with vision-only data (Fig. 1, Line 46, Line 469) and no language annotations (Line 11, Line 42). Nonetheless, some reviewers expressed doubts on this matter. We believe the source of confusion is the statement at Line 176-177 ("We have removed the explicit reward information about the task and replaced it with a short task description, in language form.").

To improve clarity, we replaced it with:
"The datasets contain no reward information and no text annotations of the trajectories. The rewards for training for a given task must be inferred by the agent, i.e. using the cosine similarity between observations and the given prompt or, in the case of GenRL, using our reward formulation (Eq. 3)."

**How can the system work with language prompts, if it's not trained on language data?**

The connector learns to map visual embeddings from the pre-trained VLM to latent states of the world model. When learning the connector from visual embeddings $e^{(v)}$, we assume it can generalize to the corresponding language embedding $e^{(l)}$ if the angle $\theta$ between the two embeddings is small enough, see Fig. 11a in attached PDF. This can be expressed as $\cos{\theta} > c$ or $\theta < \arccos{c}$, with $c$ a small positive value [1]. 

Previous work [1,2] leverages noise during training (of the connector), leading to the situation in Fig. 11b, where $c$ grows larger with the noise. This allows language embeddings to be close enough to their visual counterparts.

In our work, we instead learn an aligner network, which maps points surrounding $e^{(v)}$ closer to $e^{(v)}$ (Fig. 11c). This way $c$ is unaltered but the aligner will map $e^{(l)}$ close enough to $e^{(v)}$. Since we use noise to sample points around $e^{(v)}$ the model can be trained using vision-only data (no language annotations). 

We hope this provides a cleaner explanation, which should replace the previous one (Line 118-127).

[1] LAFITE: Towards Language-Free Training for Text-to-Image Generation, Zhou et al

[2] Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data, Zhang et al

## Data-free settings

After pre-training the MFWM, we claim that is possible to train policies for new tasks in a data-free fashion. **How does this differ from the standard GenRL setting and offline zero-shot RL methods?** We answer this question in Fig. 12 (attached PDF).

Offline RL methods (Fig. 12a), combined with VLMs, can learn to perform tasks zero-shot from new prompts, but they need to sample observations and actions from the dataset for computing rewards and for policy learning.

GenRL (Fig. 12b), and potentially other model-based RL methods combined with VLMs, need to sample (sequences of) observations from the dataset, to infer the initial latent states for learning in imagination. Afterwards, rewards can be computed on the imagined latent sequences, enabling policy learning.

In addition, for data-free GenRL (Fig. 12c), we sample the initial latent states internally by combining: (i) random samples of the latent space, (ii) randomly sampled embeddings, which are mapped to "actual embeddings" using the aligner, and turned into latent states, by the connector. Thus, policy learning requires no data sampling at all. 

Finally, following the suggestion of Rev. Yb6H, we renamed this paradigm "Data-free policy learning". To the best of our knowledge, there are no previous works that can learn multiple policies in a data-free fashion (after pre-training of a model) which is the reason why we are unable to provide additional baselines.

## Baselines

We added model-based baselines to our main experiments. These baselines follow the recommendation of Rev. Yb6H of learning an "inverse connector" from the world model representation to the VLM representation (GenRL does the opposite). The "inverse connector", given the latent state corresponding to a certain observation, predicts the corresponding embedding. Formally:

Inverse connector: $\hat{e}^{(v)}_t = f(s_t, h_t) $

$\mathcal{L_{inv-conn}} =  || e^{(v)} - \hat{e}^{(v)}_t ||^2_2$

After training the inverse connector, visual embeddings can be inferred from latent states. For policy learning, rewards are computed using the cosine similarity between embeddings inferred from imagined latent states and the prompts' embedding. We call this method **WM-CLIP**.

The inverse connector is implemented as a 4-layer MLP, with a hidden size of 1024. For a fair comparison, we adopt the same world model for WM-CLIP and GenRL. For WM-CLIP we pre-train the additional inverse connector, while for GenRL the connector and aligner. We use one world model for each domain and then train N policies (for N seeds). We have also re-run the main experiments, increasing the number of seeds to 10 for all methods.

In Table 5 and Fig. 13 (attached PDF), we observe that WM-CLIP is stronger than the model-free baselines. This allows us to clearly establish the main ingredients that contribute to the stronger performance of GenRL: (i) the video-language model helps in dynamic tasks, (ii) model-based algorithms lead to higher performance, (iii) the connection-alignment system presented outperforms the "inverse" way of connecting the two representations.

## Behaviors from video prompts

In Fig. 14 of the attached PDF, we provide behavior learning results from video prompts. The videos are also on the project website. The tasks included are static and dynamic, across 4 different domains.

The results show a similar trend to the language prompts experiments and the performance of using video prompts is aligned to language prompts, for the same tasks. In general, we found it interesting that the VLM allows us to generalize to very different visual styles (drawings, realistic, AI-generated), very different camera viewpoints (quadruped, microwave), and different morphologies (cheetah tasks).

Meta-review: This work explores utilizing large-scale pre-trained models for interactive & embodied tasks via multimodal world models that generalize across domains. This is achieved by training connector and aligner networks, using vision-only data, to align the latent space of large-scale pre-trained models with that of the generative world models.

In initial reviews, while the submission received mixed overall ratings, all reviewers praised the novelty of the proposed approach and the valuable contribution of leveraging large-scale pre-trained models for RL and embodied AI. During the discussion phase, two of the reviewers (X1kr, Yb6H) moved from borderline rejects to borderline accepts, based on clarifications provided by the authors regarding language annotations, the role of connector & aligner networks, introduction of new baselines and ablations, and authors’ commitment to improving the clarity of the draft. Another reviewer (bBQY) implicitly championed the submission by raising their rating further to a clear acceptance. Part of the committee expressed valid concerns regarding overstatements of the contributions and some confusion about key components of the proposed methodology. The authors engaged in constructive discussions with the reviewers to ground their claims more concretely, clarify technical details, and improve the overall explanation. Reviewer jfot found the new edits warranted a fresh round of reviews. However, they recognized the core idea of the submission as innovative, the results as suitable and indicative of generalization, and the positioning and scope of the work as relevant to the community. 

The AC recommends acceptance and agrees with the largely shared opinion of the committee. This work introduces a novel, pertinent, and timely contribution -- that has the potential to significantly benefit the community. The authors are strongly encouraged to improve the draft with the discussed writing edits, new experimental analysis, and provide a clearer description of the key components of their methodology in the final draft.
Decision: Accept (poster)