FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
Authors: Yucen Wang
Rui Yu
Shenghua Wan
Le Gan
De-Chuan Zhan
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
World Models
Foundation Models
Open-ended Tasks
Goal-Conditioned Reinforcement Learning
Multi-task Reinforcement Learning
Learning Reward Functions
Primary area: reinforcement_learning->deep_rl
TL;DR: We propose FOUNDER that grounds Foundation Model task representations into World Model goal states, enabling open-ended task specification and completion in embodied environments by capturing deep-level task semantics.

Abstract: Foundation Models (FMs) and World Models (WMs) offer complementary strengths in task generalization at different levels. In this work, we propose FOUNDER, a framework that integrates the generalizable knowledge embedded in FMs with the dynamic modeling capabilities of WMs to enable open-ended task solving in embodied environments in a reward-free manner. We learn a mapping function that grounds FM representations in the WM state space, effectively inferring the agent's physical states in the world simulator from external observations. This mapping enables the learning of a goal-conditioned policy through imagination during behavior learning, with the mapped task serving as the goal state. Our method leverages the predicted temporal distance to the goal state as an informative reward signal. FOUNDER demonstrates superior performance on various multi-task offline visual control benchmarks, excelling in capturing the deep-level semantics of tasks specified by text or videos, particularly in scenarios involving complex observations or domain gaps where prior methods struggle. The consistency of our learned reward function with the ground-truth reward is also empirically validated. Our project website is https://sites.google.com/view/founder-rl.

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
Yucen Wang" 12
Rui Yu 12 Shenghua Wan 22 Le Gan 12 De-Chuan Zhan 12
Abstract
bodied domains poses significant challenges. Reinforce- ment Learning (RL) offers pathway to bridge this gap by enabling FMs to learn and adapt through real-world in- teractions and feedback. In turn, FMs provide agents with valuable reasoning and generalization abilities. This synergy between RL and FMs has driven extensive research aimed advancing embodied decision-making (Brohan al., 2023b; Du et al., 2023; Brohan et al., 2023a; Liu et al., 2024).
Foundation Models (FMs) and World Models (WMs) offer complementary strengths in task gen- eralization at different levels. In this work, we propose FOUNDER, a framework that integrates the generalizable knowledge embedded in FMs with the dynamic modeling capabilities of WMs to enable open-ended task solving in embodied environments in reward-free manner. We learn mapping function that grounds FM representa- tions in the WM state space, effectively inferring the agent's physical states in the world simula- tor from external observations. This mapping enables the learning of goal-conditioned pol- icy through imagination during behavior learn- ing, with the mapped task serving as the goal state. Our method leverages the predicted tem- poral distance to the goal state an informative reward signal. FOUNDER demonstrates superior performance on various multi-task offline visual control benchmarks, excelling capturing the deep-level semantics of tasks specified by text or videos, particularly in scenarios involving com- plex observations or domain gaps where prior methods struggle. The consistency of our learned reward function with the ground- -trtt reward is also empirically validated. Our project website is https://sites. google.com. wwweeeccoo/ciiiieeee cll//iee//ouuuderr.l
Simultaneously, World Models (WMs) have emerged as crit- ical components for embodied RI. agents (Wu et al., 2022; Zhang et al., 2024), which capture the underlying dynam- ies of physical environments and provide actionable state representations essential for control tasks. WMs enable agents generate synthetic trajectories through imagina- tion, significantly improving sample efficiency (Moerland et al., 2023; Luo et al., 2022). Furthermore, the ability of WMs to model environment dynamics highlights their po- tential for task generality, as the same learned dynamics model can be applied across multiple tasks within the en- vironment. However, leveraging WMs for specific tasks typically requires the design of tailored reward functions, and challenging. This con-
which can be straint limits the broader applicability of WMs, especially for tasks specified in human-intuitive formats, such as text descriptions or videos, where rewards cannot be directly inferred.
Since WMs are primarily trained on embodied interaction data, they inherently lack the prior knowledge and high- semantic understanding required to interpret such open- ended tasks. However, this gap can be filled by FMs, which exhibit powerful capabilities in understanding and represent- ing complex patterns across multiple modalities. This raises a fundamental yet unexplored question: Can the comple- mentary strengths in task generalization offered by WMs (low-level dynamics modeling of the target environment) and FMs (high-level knowledge of the world) be integrated to enable embodied decision-making agents capable of solv- ing diverse. open- open-ended tasks?
1. Introduction
Foundation Models (FMs), pre-trained on diverse datasets, encapsulate rich and generalizable knowledge, demonstrat- ing exceptional capabilities in vision and language tasks (Moroncelli et al., 2024). However, adapting FMs to em-
'School of Artificial Intelligence, Nanjing University, China Natiooaa Key Laboratory for Novel Software Technology, Nan- jing University, China. Correspondence to: De-Chuan Zhan <zdel nj..euu..uu..........
In this work, we propose a framework, Foundation mOdels groUNDed in woRld modEls for open-ended decision mak- ing (FOUNDER), that bridges the gap between FMs and WMs by grounding FM-generated task representations
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodicd Decision Making
into actionable WM states, enabling model-based Goal- Conditioned Reinforcement Learning (GCRL) to solve open-ended tasks indicated by texts or videos. While FMs task prompts, the output general-
and low-level dynamics, offering general framework for open-ended task interpretation and accomplishment.
can handle purpose representations may not align well with the target RL environment, making them unsuitable for direct use in WMs or policy learning. To address this issue, we ground FM representations within the target environment by learn- ing mapping function from FM representations to states in the WM. This can be seen as identifying the underlying physical states in the environment simulator that correspond to external observations, represented as FM-generated vi- sual or language embeddings, with the learned WM serving as an environment simulator. The mapping allows multi- modal task prompts to be translated into the WM state space, where they can be treated as goal states. Solving the task then reduces to efficient GCRL in the WM. Specifically, we learn the mapping function using an autoencoder, as well as enforcing alignment between the mapped states and their corresponding WM states using KL constraints. Addition- ally, we learn a model to predict the temporal distance be- tween WM states, with the predicted distance to the mapped goal state serving as the reward function for policy learning.
2. Related Work
Task/Reward Specification using Foundation Models Foundation models have proven to be powerful tools for task and reward specification in RL. Large language mod- els have been used to design reward functions and apply regularization for improved RL performance (Kwon et al., 2023; Hu & Sadigh, 2023; Klissarov et al., 2023). Vision- language models, in particular, excel in reward and task specification in visually rich environments. Some works (Fan et al., 2022; Sontakke et al., 2024; Baumli et al., 2023) compute semantic similarity between agent states and task descriptions, providing dense reward signals for RL from visual observations. Other studies (Lee et al., 2023) lever- age Vision-language models to analyze visual inputs and offer preference- based feedback for reward model training. While some approaches (Baumli et al., 2023; Rocamonde et al., 2023; Cui et al., 2022) use off-the-shelf FMs for zero- shot task specification, others (Fan et al., 2022; Sontakke et al., 2024) fine-tune the FMs on domain-specific datasets for better reward design. Our method differs from these previous works by specifyyng tasks and rewards through world models.
We evaluate FOUNDER on DeepMind Control Suite tasks, Franka Kitchen, and Minecraft. Our method consistently performs well in multi-task decision-making based on of- fline reward-free RL from visual observations with tasks specified in multi-modal prompts. Furthermore, empirical studies in cross-domain settings demonstrate the superior ability of FOUNDER to capture deep-level task semantics, especially in scenarios where previous methods struggle.
Model -Baee Goal-Conditioned RL (MBGCRL) Previ- ous methods have explored GCRL in WMs. LEXA (Men- donca et al., 2021) is an unsupervised RL method that en- courages agents to discover and achieve complex goals in imagination, enabling zero-shot adaptation to visual goals. Choreographer (Mazzaglia et al., 2022) exploit the learned WM to learn skills from an offline dataset and uses a meta- controller to deploy them for efficient task adaptation. Di- rector (Hafner et al., 2022) decomposes long-horizan tasks into goals and achieves them in imagination. These methods use MBGCRL for exploration or subtask learning, whereas we apply it directly for behavior learning to reach goal states specified by multi-modal prompts.
The most similar prior method, GenRL (Mazzaglia et al., 2024), relies on step-by-step visual alignment in connecting FM representations with WMs as well as in policy learning. While effective in some cases, this approach may cause mis- leading task interpretations and reward signals. particularly in tasks with complex visual observations, inherent static nature, or significant visual discrepancies with the prompt. This is because GenRL uses only visual features to infer and match specific agent behaviors, lacking temporal awareness and a deeper understanding of the task semantics.
3. Problem Setting
In contrast, FOUNDER enables deeper task interpretation by mapping the prompt representation the correspond- ing goal state in the WM. This mapping, combined with our cemporal-disttace-basee approach, yields a highly infor- mative reward function that effectively guides policy learn- ing. Unlike GenRL's rigid step-by-step trajectory matching, FOUNDER operates within a more flexible GCRL paradigm, allowing for more adaptive behavior learning. The con- sistency of our learned reward function with the ground- truth reward is also empirically validated. From those sides, FOUNDER creates bridge between high-level knowledge
We focus on training agents capable of solving open-ended tasks in the context of offline RL. from visual observations. The agent has access to pre-collected dataset of interac- tion trajectories in embodied environment, denoted as B = (ob.r- ) where represents the observations (images) and a represents the corresponding actions. No- tably, the dataset B consists solely of observation-action pairs, and does not contain reward information or task- specific label annotations.
In this setting, the offline RL. agent is tasked with addressing open-ended tasks specified by multi- multi-modal prompts, such
2

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodicd Decision Making
as text descriptions or videos. We leverage Foundation Model, i.e., Video-Language Model (VLM), aid the agent in interpreting the task semantics from the prompt. The agent cannot interact with the real environment or obtain the ground-truth reward of the task during training. Thus, the agent's objective is to use the VLM to understand the task, derive the necessary supervisory signals for accom- plishing it, and utilize the reward-free free offline dataset B to learn policy conditioned on the task prompt.
learning of more effective mapping function that captures deeper environmental attributes from the VLM to the WM.
4.2. Mapping Function Learning
The learned world model can be viewed an environment simulator. Intuitively, when a task is specified by video, the video embedding generated by Foundation Model can be considered an external observation of a specific agent state = in the environment: e ~ U(( =). where U/ is the emission function in the context of POMDP (Kaclbling et al., 1998). This is because task-related semantics, which specify particular agent behavior, are embedded within the video; however, the underlying physical state information is not inherently included in the original representation. Thus, learning the mapping function g from the FM representation space E' to the world model state space Z, is essentially the process of inferring the corresponding agent state in the simulator: 2 ~ Q(- e)..
4. Method
illustrates the key steps of FOUNDER. In the pre- training stage, we first train a Dreamer-style world model using the offline data. Next, we employ this world model to learn temporal distance prediction model between world model states. We then learn a mapping function that con- nects VL.M representations to model states, optimizing the reconstruction likelihood of an autoencoder subject to KL divergence constraints. Finally, for downstream tasks spec- ified by text or video prompts, we utilize the components developed in the previous phases to perform behavior learn- ing. We then describe our method in detail.
Figure I
These insights are reminiscent of prior work on inferring latent states in pre-defined domain from external visual ob- servations (Chen et al., 2021), which employs variational in- ference when paired data from the two domains are unavail- able. However, given the availability of a Video-Language Model, a pre-trained world model, and an offline dataset, we can construct paired data for the VI.M and WM space, using the dataset trajectories. This is done by processing consec- utive observation sequences (short videos) in the dataset trajectories using the VLM and encoding the image-action pairs to latent states using the WM, where k is the number of video frames that the VLM can process:
4.1. World Model Learning
We train the world model in the paradigm of DreamerV3 (Hafner et al., 2023) and employ the RSSM backbone (Hafner et al., 2019) from the offline dataset, consisting of the following components:
Posterior (Encoder):
90(21 | ogi, a<t)
Prior (WM transition):
po(2 22-1, ar1))
e. VLM(OL-ke))
t= k=k,.". .
Observation Decoder: po(ar] | 2x)
(3)
21 ~ WM( lgge aet); 1 1= ...
k......? T
The learning objective is optimizing the evidence lower- bound on the likelikel of the offline data (or, T=)
Here, is sampled from the encoded distribution of the WM states, capturing the agent's 's hehavior by incorporating historical trajectory information. ee typically represents the semanties of the recent image sequence up to step & from the perspective of the VLM. The paired data (e,, 24) establish the correspondence between the VLM and WM representation spaces. We then learn mapping function Qs to obtain the mapped state & by aligning the distribution of the mapped states with the ground-truth states:
T Bziaas In p& (oe 2x)
(2)
DKL [go (. Oses azx) lIpo (- |1-1111,, a1]]
There is no reward model since the dataset contains no re- ward information. The WM state z is composed of both deterministic and stochastic states: 2;- = (he, st). In GenRL, the encoder and decoder do not condition on hy which con- tains rich historical information, and the latent states only contain information about a single visual observation (Maz- zaglia et al., 2024). This limits GenRL to visual-level multi- step alignment, where FM representations are aligned solely with the visual states 8; in the WM. In contrast, we maintain the original structure of V3, preserving all the infor- mation in the WM states. This ensures that the learned WM retains richer environmental information, allowing for the
min Qu
DEL 1Q (.) [[WM o act)l (4)
The distribution of 2 is parameterized in the world model, and we similarly parameterize the distribution of the mapped result &. This makes the objective of minimizing the KL divergence tractable, and Q2 can be updated using the repa- rameterization trick. Additionally, since e, captures the semantics of a short video, including timesteps preceding
3

---

FOUNDER: Grounding Foundation Models in World Models for Open- Enen-EEndd Embodicd Decision Making
Offine Dataset
Brakadingg
Tenjo Proskder Pia
Terpll Predicter
Predlcttr
Mapping Function Qp
Encoding
WM
Learning
Mapping Funeeton Qg
VIX Emberding
WM imagination
D
Pelt.))) 1-1
Video Prompe
Preducter
Figure 1. FOUNDER encompasses pretraining phase and hehavior learning phase. During pretraining, we first train Dreamer-style WM, then use it to learn mapping function and a temporal distance predictor. During behavior learning, the VI.M representation of task prompt is mapped to goal state within the WM and conl-conditione policy is learned through imagination, where the predicted temporal distance to the goal state serves as reward. The three pretrained components collectively enable successful behavior learning.
Ot, aligning only the last step's states may overlook prior information. To avoid seq2seq-style modeling, which re- quires learning complex generative mapping function in GenRL, we instead introduce simple reconstruction loss to recover the complete VLM representation from the mapped state, using an autoencoder:
Temporal Distance Predictor: We introduce a model Dg to predict the temporal distance between two WM states. The temporal distance represents the number of timesteps required to transition from one state to another in the WM, effectively capturing the agent's 's progression towards the goal state when used in the reward function. We sample states pairs 2,, 2tte from the same trajectory sequence in the offline dataset, encoded by the pre-trained WM, and feed these pairs into D9 to predict the distance c, normalized by the sequence length T. Also, we incorporate negative sample pairs 2 o from different trajectories, where the target for these pairs is the maximum possible distance in terms of timesteps:
min Qu.Py
B In Pe (ex 2.]
(5)
The two losses together result in an objective akin to varia- tional inference, with the key difference being that we use the alignment of WM states at the corresponding timesteps as KL constraint, rather than relying on random state dis- tributions. Although the mapping is learned solely inter- action data from the target environment, the generalization power of the VLM enables the mapping to extend to data from other domains, as demonstrated in our experiments and in (Mazzaglia et al., 2024). This enables multi-task behavior learning by mapping diverse external VLM embeddings cg into corresponding WM goal states zg:
MSE (Do(ze, zete): 7)
min
Do
min Do
MSE 22)(2),=))..)
The insights behind these loss designs are widely adopted in prior work (Mendonca et al., 2021; Hartikainen al., 2019). Temporal distance provides a more robust reward signal by extracting environmental dynamic information, promoting goal-oriented behavior and capturing deeper task semanties beyond visual details. The temporal distance predictor is also pre-trained before policy learning.
2g ~ Qpl lg))
(6)
4.3. Behavior Learning
When a task prompt is provided, the agent performs standard GCRL within the world model to reach the goal state 2g mapped from the prompt. The key challenge then becomes determining the reward function used to guide learning to- wards the goal. Previous methods typically employ cosine similarity with the goal state the reward function (Hafner et al., 2022; Mazzaglia et al., 2024). However, we observe that this approach often fails due to reward hacking, particu- larly in locomotion tasks (Section 5.1). To address this, we propose using temporal distance (Mendonca et al., 2021) as an alternative reward signal.
Goal-conditioned Policy Learning. The behavior learning process the WM becomes straightforward. After imagin- ing trajectory rollouts in the WM, we employ the learned D to predict the temporal distance between each state and the mapped goal state, assigning rewards to each timestep in the imagined trajectories:
TD kg) = --Do DD((E, zy)
(8)
Using this reward signal, we apply Dreamer-style Actor- Critie learning to train a goal-conditioned policy and value
4

---

FOUNDER:
Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
Table 1. Normalized test performance of FOUNDER and baselines on DMC and Kitchen benchmarks, Mcan scores (higher is better) with
standard deviation are recorded across seeds for each task.
Task
GenRL
WM-CLIP
GenRL-TTmppD
FOUNDER w/o TempD
FOUNDER
Cheetah Stand
0.93 0.03
0,93 = 0.03
0.42 + 0.04
0.91 + 0.01
1.02 0.01
Cheetah Run
0.68 0.06
0,51 = 0.04
0.37 = 0.06
0.21 + 0.01
0.81 = 0.02
Cheetah Flip
-0.04 + 0.01
-0.11 + 0.11
0.06 + 0.05
-0.26 + 0.01
0.97 = 0.02
Walker Stand
0.81 0.16
1.01 + 0.02
0,92 + 0.06
1.02 + 0.01
1.01 = 0.02
Walker Walk
0.95 + 0.02
0.95 + 0.03
0.42 + 0.10
0.19 + 0.02
0.94 + 0.04
Walker Run
0.81 + 0.02
0.69 + 0.05
0.68 + 0.03
0.21 + 0.01
0.78 + 0.04
Walker Flip
0.48 0.04
0.59 + 0.04
0.50
0.28 + 0.02
0.47 + 0.03
Stickman Stand
0.60 + 0.11
0.41 + 0.06
0.49 0.05
0.53 + 0.04
0.91 + 0.04
Stickman Walk
0.83 0.03
0.69 + 0.13
0.84
0.26 0.03
0.91 0.03
Stickman Run
0.38 0.03
0.37 + 0.04
0.38 + 0.03
0.17 0.00
0.48 + 0.02
Stickman Flip
0.29 0.05
0.62 + 0.03
0.38 1 0.04
0.25 0.03
0,41 + 0.03
Quadruped Stand
0.95 0.06
0.84 + 0.20
0.97 + 0.04
0.99 + 0.02
0.98 - 0.01
Quadruped Walk
0.73 + 0.19
0.64 + 0.21
0.60 + 0.17
0.51 + 0.02
0.90 = 0.05
Quadruped Run
0.72 4: 0.21
0.58 + 0.13
0.51 1 0.09
0.52 + 0.01
0.94 1 0.03
Kitchen Light
0.00 + 0.00
0.35 + 0,48
0.92 + 0.28
1.00 + 0.00
0,97 = 0.18
Kitchen Slide
0.62 + 0.49
1.00 + 0.00
1.00 + 0.00
0.97 + 0.18
1.00 = 0.00
Kitchen Microwave
1.00 0.00
0.63 + 0.48
1.00 + 0.00
0.98 0.13
1.00 0.00
Kitchen Burner
0.35 0.48
0.10 + 0.30
0.63 + 0.48
1.00 + 0.00
0.60 + 0.49
Kitchen Kettle
0.35 + 0.48
0.07 + 0.25
0.05 + 0.22
0.05 + 0.22
0.33 + 0.47
Overall
0.60
0.57
0.59
0.52
0.81
function:
tions. Details of environments and tasks are in Appendix A.
Action model: a₂ ~ T (at) 24, kg)
Experimental Setup. The agent is required to solve di- verse downstream multi multi-modal tasks using a reward-free offline dataset and a Video-Language Model. We collect offline data consisting of image-action pairs for each do- main, which generated through exploration strategies like Plan2Explore (Sekar et al., 2020), and from the training buffers of RL. agents designed to complete particular tasks. The agent must generate necessary learning signals for the underlying task and learn a corresponding policy. Details of our pre-collected data can be found in Appendix B. To en- sure fair comparison, FOUNDER and all baseline methods employ the same VLM, Video2 (Wang al., 2025), as adopted by (Mazzaglia et al., 2024).
11
Value model: v (zt, ao Ba,wM
Σ
T (Ze, mq)
(9) The behavior learning phase of FOUNDER integrates the high-level generalizable knowledge encoded in the VLM, the environment modeling capability of the world model, the task grounding ability of the mapping function, and the reward signal generation of the temporal distance predictor. The task-agnostic nature of these components allows for their potential application to wide range of downstream tasks, facilitating open-ended multi-modal task specification and solving in embodied environments.
Baselines. Model-free offline RL methods like IQL (Kostrikov et al., 2021b) and TD3+BC (Fujimoto & Gu, 2021) involve the cumbersome process of retraining from scratch for every task, and zero-shot FM-generated rewards using task prompts and observations has been shown to perform poor in (Mazzaglia et al., 2024). Therefore, we compare FOUNDER mainly with model-base methods. We select GenRL, which performs step-by by-step state sequence matching in learning mapping and policy, WM-CLIP (Maz- zaglia et al., 2024), which learns a mapping from WM states to FM representations and uses the similarity FM space as the reward. We also evaluate FOUNDER w/o TempD, which uses cosine similarity in reward function dur- ing behavior learning. The final baseline is GenRL-Templl which replaces the distance metric in GenRL with tempo-
5. Experiments
We empirically evaluate the performance of FOUNDER in solving open-ended tasks specified by text or videos, using four locomotion environments (Cheetah, Walker, Quadruped and Stickman proposed in (Mazzaglia et al., 2024)) from the DeepMind Control Suite (DMC) (Tassa al., 2018), and one manipulation environment (Franka Kitchen (Gupta et al., 2019)). For video tasks, place special focus on cross-domain videos that cause significant visual gap. Furthermore, we validate the reliability of the reward function learned by FOUNDER. Finally, we extend FOUNDER to Minecraft, a more complex environments with challenging task instructions and more intricate observa-
5

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
Walker
0.-
Run
0.2
52
0.0
64
0.0
walker
stickman
cheet
stickman
cheetah
09
54
0.0
02
-02 walker
stidmmnn
cheetah
stickman
cheetah
Welker
Genki
Figure 2. Normalized evaluation performance on cross gross-embodimen tasks built upon DMC. Each row corresponds to one of the Run or Flip tasks, while cach column represents the domain in which the agent evaluated. Each subplot presents the results respectively using videos from the remaining two domains as task prompts. This yields 6 domain combinations: (Cheetah, Walker), (Cheetah, Stickman), (Walker, Cheetah), (Walker, Stickman), (Stickman, Cheetah), (Stickman. Walker), each evaluated on Flip. totaling 12 evaluation tasks. Mean scores with standard deviation recorded across 4 seeds.
ral distance, designed to assess whether temporal distance alone contributes the performance of FOUNDER. Each method pre-trains WM and other components for 500K gra- dient steps, and then is constrained to maximum of 50K gradient updates in behavior learning. Implementation de- tails of these model-based baselines are in Appendix C. The comparison of our approach with model-free methods, in- cluding an analysis efficiency and computation costs, is detailed in Appendix D.I and Appendix D.2.
klichen
kitcher
mahh
TIZn
right
10p
Spht top
5.1. Language Task Solving on DMC and Kitchen
Figure 3. Performance of FOUNDER and baselines over seeds two tasks in Kitchen. The agent's 's observations is captured from the Main viewpoint, while task video prompts provided from the Right or Top viewpoints, yielding 4 tross-view tasks.
We evaluate the ability of interpreting and solving open- ended text-based tasks of FOUNDER DMC and Franka Kitchen. For DMC, select four locomotion tasks that are generic across different agent embodiments: Stand. Walk, Run and Flip, and evaluate all possible environment-tas task combinations. For Kitchen, we evaluate on five commonly used tasks. We incorporate the aligner from GenRL to denoise the text embeddings of task prompts into visual embeddings. The results are presented in Table 1, where performance normalized between the expert policy and random policy.
as demonstrated by the low performance of FOUNDER w/o TempD.
Moreover, when studying the failure cases of GenRL and FOUNDER w/o TempD, we find that their failures can be attributed to reward hacking which leads to unintended outcomes. In these scenarios, GenRL or FOUNDER w/o TempD agents only mimie the visual features of the task without truly accomplishing it, revealing their inability to interpret and exploit deep-level task semantics. A GenRL agent might simply wave its arms near the light switch instead of actually turning it on. A FOUNDER agent tasked with running may remain stationary if not use the temporal distance predictor. We provide case study for these failures on our website https://siitss https://ssies.ggggle..see www.gggl.ccomvwe//foonerr. and in Appendix E.3.
FOUNDER achieves the best overall performance, ranking among the highest 14 out of 19 tasks. Ge on kitchen tasks with inherently static properties, under- scoring its lack of temporal awareness and the timestep disalignment introduced by step-by-step trajectory match- ing. The poor performance of WM-CLIP emphasizes the importance of aligning representations in WM state space rather than in VLM space, as WM states encapsulate action- able information beyond surface-level visual or linguistic semantics. GenRL-TempD does not outperform GenRL, indicating that temporal distance alone does not contribute significantly to performance; its effectivvness emerges only when integrated within the GCRL paradigm of FOUNDER,
5.2. Cross-domain Video Task Solving
We then evaluate FOUNDER'S capability in solving video- based tasks, with a partieular focus on cross-domain scenar-
6

---

FOUNDER: Grounding Foundation Models in World Models for Open- Bnen-Enned Embodied Decision Making
cheatah
walker walk
walker flipping
stickman walk
stickman
0.30
0.4
0.5-
0.3
0.4
0:2
0.5
0.2
0.3
045
0.3
0.2-
010
0.
0.1
0.1
0.0
GenRL
GarRL D
FOUNDER TempO
FOUNDER
Figure 4. Performance ovaluation of FOUNDER and baselines over 4 seeds across 5 cross-viewpoint video tasks in DMC. Visualizations of video prompts indicating the task semantics and the agent's observation the corresponding environment is presented first row,
ios where the visual appearance task videos differs sig- nificantly from the agent's environment observations. This includes videos captured from alternative camera viewpoints or showcasing behaviors performed by morphologically dif- ferent agent embodiments. Notably, each task is defined by single 8-frame video, as Intern InterVVideo2 can process maximum of 8 frames. This constraint further increases the difficulty of the tasks, resulting in problem setting of (Third-Person) One-Shot Visual Imitation Learning (Stadie al., 2017; Wan et al., 2023; Huang et al., 2024), which is challenging for prior methods.
Table 2. Evaluation of the consistency between learned pseudo rewards and ground-truth rewards, averaged on 7 tasks in DMC. The results for each task are shown Appendix D.4.
Methods
Corrt Regret. Precision Recall" FIf
GenRL
0.12
0.37
0.47
0.44
0.44
WM-CLIP GenRL-TempD FOUNDER w/o TempD
0.40
0.26
0.61
0.69
0.63
0.05
0.75
0.46
0.40
-0.02
0.90
0.16
0.15
0.15
FOUNDER
0.54
0.07
1.0
0.47
0.59
curately aligns them with the corresponding physical states in the world model. While (Mazzaglia et al., 2024) shows that GenRL can effectively translate real-world videos into videos in the simulator and learn policy, experiments reveal its failure interpret the semantics of cross-domain, simulation-generann videos. This contrast highlights that GenRL primarily operates style-transfer- like approach, aligning visual appearances rather than capturing deeper physical states. In contrast, FOUNDER excels extracting and grounding underlying physical information from task videos WM space, making it more effective in scenarios requiring true semantic understanding.
For cross-embodiment evaluation, assess behavior leamn ing for the Run and Flip tasks within one of the three do- mains-Cheetah, Walker, or Stickman--wiie using task videos sourced from the remaining two domains. This setup results in six domain combinations and total of 12 tasks. The evaluation results are presented in Figure 2, where FOUNDER achieves the best performance on out of 12 tasks. Notably, FOUNDER is the only method that succeeds in Cheetah-bases domains, with other baselines completely failing to connect tasks specified in Walker and Stickman videos into Cheetah embody. FOUNDER excels in challeng- ing domain transfers, such as (Cheetah, X) or (X, Cheetah), where the morphological differences between Cheetah and the other two domains are significantly larger than those between Walker and Stickman.
Furthermore, to provide more comprehensive evaluation of F FOUNDER's performance on out-of-diitribution video task prompts, we also conduct experiments on real-world video tasks, following the protocol (Mazzaglia et al., 2024), as detailed Appendix D.3. The results again indicate that FOUNDER demonstrates solid performance in comparison to GenRL.
In the cross-view evaluation, we select several tasks from DMC and Kitchen where the task video prompts are cap- tured from alternative camera viewpoints. The evaluation results are presented Figure 3 and Figure 4. FOUNDER outperforms or matches the performance of baselines across most tasks, and the only method to learn successful policy in Cheetah Run, Walker Walk and Kitchen Burner (Right). Notably, GenRL exhibits the poorest performance among all baselines in Kitchen tasks, where the domain gap across different viewpoints is huge.
5.3. Evaluation of the Learned Reward Function
We assess the learned reward functions of FOUNDER and baseline methods in comparison with ground-trute -trut rewards. The learned reward functions depend on the pre-trained WM state space, the mapping function, and the similarity metric, comprehensively reflecting the properties of each method. We use the metrics RankCorr and Regret to evaluate whether the learned pseudo-reward function ac-
The above results demonstrate that FOUNDER captures deep-level task semantics beyond visual features and ac-

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodicd Decision Making
Hunt Cow Sunflower Plains
Shear Sheep Plains
Chop Trees Forest
Milk Cow Plains
Hunt Sheep Forest
0.23
2.33
5,0025 Vale
Sarras Sate
0.33
0.25
Success Sate
0.10-
0.131
9.02
0:33
0.3
3.0
5.00
0.87
1.00
26 Timestopt UP1
46
86
-2.00
20
66 Timestast DP1
40
66
Timcsaph-ie"
Timesoplxie1
Figure 5. Performance of POUNDER and haselines over 3 seeds across 5 tasks in Minecraft during behavior learning. Fach task is specified in text prompt. The solid curves and the shaded region indicate the average episodic success rates and the 95% confidence intervals across different runs. We apply a moving average to smooth the curves.
curately estimates the ground-truie -ruth reward. Additionally, we adopt the evaluation approach from (Fan et al., 2022), which converts the reward model into a binary success clas- sifier, and measure performance using Precision, Recall, and F1-score. We evaluate all methods on the testing buffers stored during language task behavior learning (Section 5.1). The metrics are computed at the trajectory level, where the ground- truth and pscudo rewards for each timestep are aver- aged over each trajectory. More detailed information about the evaluation and metrics can be found in Appendix C.4.
biomes, which are specified in text prompts, listed in Table 3. The agent relies solely on visual observations for decision- making, without access to manually designed dense rewards, success signals, or underlying state information.
We evaluate the performance of FOUNDER on Minecraft tasks with GenRL and MineCLIP-IQL. Following the same setup as previous experiments, the agents are provided only with a VLM and an offline dataset for task understanding and adaptation. For both FOUNDER and GenRL, we first perform pre-training on the offiine dataset for 500K gra- dient steps and then conduct behavior learning for down- stream tasks for 100K steps. Additionally, we incorporate MineCLIP in (Fan et al., 2022), a cext-condiiioned reward model pre-trained on an internet-scale Minecraft knowledge base, to establish a model-free offline RL baseline (IQL), where MineCLIP is used to assign rewards to the offline dataset for each text task. The resulting MineCLIP-IQL, is oracle method for comparison. Implementation de- tails of MineCLIP-IQL can be found in Appendix C.3, The evaluation results are presented in Figure 5.
The results are presented in Table 2. We observe that FOUNDER achieves the best consistency with the ground- truth reward as measured by Correlation and Regret. For binary classification metrics, FOUNDER demonstrates per- feet Precision, though it is outperformed by WM-CLIP on Recall and the final score. However, during policy learn- ing based on our assigned pseudo-rewards, avoiding the misclassification of low-reward behavior as high-reward is far more critical than identifying all high-reward behaviors. This mirrors cost-sensitive learning (Zhou & Liu, 2005), where the cost of False Positive samples is much larger than that of False Negative samples. In this context, precision outweighs recall, as mistakenly favoring low-reward behav- iors can lead to reward hacking and undesirable outcomes, as seen with FOUNDER w/o TempD. Overall, FOUNDER exhibits the highest consistency in reward estimation, ef- fectively translating task prompts into WM goal states and generating reliable reward functions that guide policy learn ing.
We observe that FOUNDER method achieves similar perfor- mance to GenRL on two of the five tasks while consistently surpassing GenRL, on the other three. Notably, FOUNDER'S average success rate on these three tasks consistently ex- ceeds GenRL's upper confidence interval, demonstrating statistically significant gains.
Moreover, FOUNDER matches or surpasses the oracle MineCLIP- approach in 3 out of tasks, underscoring the effectiveness of our paradigm for leveraging FMs in RL. Instead of finetuning FMs with vast amounts of related Inter- net data, our approach learns an extra module that connects FMs to WMs using limited environment data, providing more efficient and adaptable solution.
5.4. Extending FOUNDER to Minecraft Environment
We then try extending FOUNDER to more challenging open- ended embodied environments such as Minecraft, where the visual observations are significantly more complex and the task instructions are harder to interpret and complete. We adopt the Minedojo benchmark (Fan et al., 2022), which introduces thousands of diverse tasks specified in free-form language. Specifically, we evaluate FOUNDER and base- lines on five tasks: "Hunt Cow", "Shear Sheep", "Chop Trees", "Milk Cow", and "Hunt Sheep" within different
6. Discussion
We introduce FOUNDER, a framework that integrates FMs and WMs enable open-ended task specification and com- pletion in embodied environments. By grounding FM-
8

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodicd Decision Making
generated task representations into corresponding goal states within the WM, FOUNDER effectively bridges high-level knowledge with low-level dynamics, facilitating efficient GCRL for diverse downstream tasks. The framework's s physical-statt--liineed mapping function, paired with a reward signal, allows the agent to go beyond visual features, obtain a deeper interpreta- tion of the task, and rapidly adapt to multi-modal prompts. FOUNDER demonstrates superior performance in complex or cross-domain environments where prior methods strug- gle, underscoring its potential for robust and generalizable embodied task solving.
Finn, C., al. Rt-2: Viston-language-action models transfer web knowledge to robotic control, arXiv preprint wrXiv:2307.15811 2023a.
Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on robot learning, pp. 287- 318. PMLR, 2023b.
lemporal-distance-basse
Chen, X.-H.. Jiang, S., Xu, F., Zhang, Z., and Yu, Y. Cross- modal domain adaptation for cost-efficient visual rein- forcement learning. Advances in Neural Information Prucessing Systems, 34: 34: 15220-22-2,,,, 2021.
Our method relies on offline environment data to learn the WM and mapping function, meaning that prompt elements not represented in the dataset may not be properly grounded. As the performance of FOUNDER is upper- roper-bounded by the offline data, future work should explore more data collection strategies or incorporate real-world videos into WM learn- ing. Additionally, while our current experiments mainly focus on short-horizon tasks, integrating the reasoning ca- pabilities of FMs into our framework could enhance its ability handle long-horizon tasks that require task decom- position. Incorporating more powerful sequence modeling methods could also improve long-term memory within the agent to handle complex, long-horizon tasks. We leave these explorations to future work.
Cui, Y., Niekum, S., Gupta, A., Kumar, V., and Rajeswaran, A. Can foundation models perform zero-shot task specifi- cation for robot manipulation? In Learning for dynamics and control conference, pp. 893-905. PMLR, 2022.
Du, Y., Watkins, O., Wang, Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, pp. 8657- 8677. PMLR, 2023.
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang. Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo:
Building open-ended embodied agents with internet-scale knowledge. Advances Neural Informa- tion Processing Systems, 35:18343-18862 2022.
Acknowledgements
This work is partially supported by National Science and Technology Major Project (2022ZD01 14805), NSFC (No. 62476123), Postgraduate Research & Practice Innovation Program of Jiangsu Province IKYCX25_03260 and the Young Scientists Fund of the National Natural Science Foun- dation of China (PhD Candidate) (624B200197).
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 54:20132-2011 2021.
Gao, C.-X. and Kong, R. OffineRL-Libb: Bench- marked Implementations of Offline RL Algorithmss February 2023. URL sxpoverflow/Offllnett Lib.
Impact Statement
This paper aims to advance the field of machine learning by proposing framework that integrates the power of FMs and WMs to facilitate open-ended embodied task solving across domains such as gaming, motion control, and autonomous manipulation. This has the potential to generate significant positive impacts on industry and society.
Gupta, A., Kumar, V., Lynch, C., Levine, S., and Hausman, K. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning, arXiv preprint arXiv: 11966, 2019.
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp. 2555 2565. PMLR, 2019.
References
Baumli, K., Baveja, S., Behbahani, F., Chan, H., Comanici, G., Flennerhag, S., Gazeau, M., Holsheimer, K., Horgan, D., Laskin, M., et al. languaee models source of rewards. arXiv preprint arXiv:23 2023.
Hafner, D.,, Lee, K.-H,, Fischer, I., and Abbeel, P. Deep hierarchical planning from pixels. In Koyejo, S., Mo- hamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (, Advances in Neural Information Processing Sys- tems, volume 35, pp. 26091-26104 Curran Associates, Inc., 2022.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A.,

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
Hafner, D., Pasukonis, J., Ba, and Lillicrap. T. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.
Ma, H., Wu, J., Feng, N., Xiao, C., Li, D., Hao, J., Wang,
J., and Long, M. Harmonydream: Tass harmonization inside world models. arXiv preprint arXiv:2310.00344 2023.
Hartikainen, K., Geng, X., Haarnoja, T., and Levine, S. Dynamical distance leamnng for semi-supervised and unsupervised skill discovery. arXiv preprint arXiv:/907.0822 2019.
Mazzaglia, P., Verbelen, T., Dhoedt, B., Lacoste, A., and
Rajeswar, Choreographer: Learning and adapting skills in imagination. arXiv preprint arXiv:2211.13350 2022.
Hu, H. and Sadigh, D. Language instructed reinforcement learning for human-ai coordination. In International Con- Serence on Machine Learning, pp. 13584-13598. PMLR, 2023.
Mazzaglia, P., Verbelen, T., Dhoedt, B.. Courville, A., and Rajeswar, S. Genrl: Multimodal-foundatie world mod- els for generalization in embodied agents. arXiv preprint arXiv:2406. 280.: 2024.
Huang, K., Shao, M., Wan. S., Sun, H.-H., Feng, S., Gan, L., and Zhan, D.C. Sensor: Imitate third-person ex- pert's behaviors via active sensoring. arXiv preprint arXiv:2404.000 033366 2024.
Mendonca, R., Rybkin, O., Daniilidis, K., Hafner, D., and Pathak. D. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems, 34:24379-24391, 2021.
Kaelbling, I. P., Littman, M. L., and Cassandra, A. R. Plan- ning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-13, 1998.
Moerland, T. M., Broekens, J., Plaat, A., Jonker, C. M., al. Model-based reinforcement learning: survey, Foundations and Trends@) in Machine Learning, 16(1):
Klissarov, M., D'Oro, P. Sodhani, S., Raileanu, R., Bacon, P.-L,, Vincent, P., Zhang, A., and Henaff, M. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint
1-118, 2023.
Moroncelli, A., Soni, V., Shahid, A. A., Maccarini, M., For- gione, M., Piga, D., Spahiu, B., and Roveda, L. Integrat- ing reinforcement learning with foundation models for autonomous robotics: Methods and perspectives. arXiv preprini arXiv:2410. 16411, 2024.
2023.
Kostrikov, I., Nair, A., and Levine, S. Offline rein- forcement learning with implicit q-learning. CoRR, abs/2110.06169, 2021a. 0bs/2110.06161
Park, S., Kreiman, T., and Levine, S. Foundation poli- cies with hilbert representations. In Proceedings of the 4 International Conference on Machine Learning, pp. 39737-39761, 2024.
Kostrikov, I,, Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning arXiv preprint arXiv:21 10.06169, 2021b.
Qin, Y., Zhou, E., Liu, Q., Yin, Z.,, Sheng, L., Zhang, R., Qiao, Y., and Shao, J. Mp5: A multi-modal open-ended embodied system in minecraft via active perception. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16307-16316. IEEE, 2024.
Kwon, M. Xie, S. M.. Bullard, K., and Sadigh, D. Re- ward design with language models. arXiv preprint arXiv:2303.00001, 2023.
Lee, H., Phatale, S., Mansoor, H., Lu, K. R., Mesnard, T., Ferret, J., Bishop, C., Hall, E., Carbune, V., and Rastogi, A. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023.
Rocamonde, J., Montesinos, V., Nava, E., Perez, E., and Lindner, D. Vision-language models are zero-shot re- ward models for reinforcement learning. arXiv preprint 2023.
Liu, S., Yuan, H., Hu, M., Li, Chen, Y., Liu, S., Lu, Z., and Jia, J. RI-gpt: Integrating reinforcement learning and code-as-policy. arXiv preprini urXiv:2402. 19299, 2024.
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. Plamning to explore via self-supervised world models. In International conference on machine learning, pp. 8583 8592. PMLR, 2020.
Lu, C., Ball, P. J., Rudner, T. G., Parker-Holder, J., Osborne, M. A., and Teh, Y. W. Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint ar XiX:222660047799 2022.
Sontakke, S., Zhang, J., Arnold, S., Pertsch, K., Biyik, E., Sadigh, D., Finn, C., and Itti, L. Roboclip: One demonstration is enough to learn robot policies. Advances Neural Information Processing Systems, 36, 2024.
Luo, F.M.., Xu, T., Lai, H., Chen, X.-H., Zhang, W., and Yu, Y. A survey on model-based reinforcement learning. arXiv preprint 22066:2206. 20..2220..038, 2022.
10

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
Stadie, B. C., Abbeel, P., and Sutskever, I. Third-person im- itation learning. arXiv preprint urXiv:/703.01770 2017.
Tassa, Y. Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L.., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., al. Deepmind control suite. arXiv preprint arXiv: 1801.00690, 2018.
Team, O. L., Stooke, A., Mahajan, A., Barros, C.. Deck, C., Bauer, J., Sygnowski, J., Trebacz, M., Jaderberg, M., Mathieu, M., al. Open-ended learning leads gener- ally capable agents. arXivv preprint wxXi::217..12808 2021.
Wan, S., Wang, Y., Shao, M., Chen, R., and Zhan, D.- C. Semail: eliminating distractors in visual imitation via separated models. In International Conference on Machine Learning, pp.
PMLR, 2023.
Wan, S., Chen, Z., Gan, L., Feng, S., and Zhan, D.- C. Semopo: Learning high-quality model and policy from low-quality offline visual datasets. arXiv preprint arXiv:2406. 2eXiv:2406.09486, 09486, 2024.
Wang, Y., Li, K., Li, X., Yu, J., He, Y., Chen, G., Pei, B., Zheng, R., Wang. Z., Shi, Y., al. Internvideo2: Scaling foundation models for multimodal video understanding. In European Conference Computer Vision, pp. 396- 416. Springer, 2025.
Wu, P., Escontrela, A., Hafner, D., Goldberg, K., and Abbeel, P. Daydreamer: World models for physical robot learning, 2022. URL https: wttxiv.xvvvoroooo/ooo org abs/
2206. 14176.
Yu, T., Thomas, G., Yu, L.,, Ermon, S., Zou, J. Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142. 2020.
Zhang, 7.,, Chen, R., Ye, J., Sun, Y., Wang, P., Pang, J.- C., Li, K., Liu, T.-S., Lin, H., and Yu, Y. Whale-x: Learning scalable embodied world models with enhanced generalizability. 2024.
Zhou, B., Li, K., Jiang, J., and Lu, Learning from visual observation via offline pretrained state-to-go transformer. Advances in Neural Information Processing Systems, 36, 2024.
Zhou, Z. and Liu, X.-Y. Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Transactions on knowledge and data engineering, 18(1):63-77) 2005.

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
Figure 6. Demonstration of the Minecraft environment, where the agent performs tasks such chopping trees, hunting cows, milking cows, hunting sheep, and shearing sheep.
A. Environments and Tasks
Minecraft environment. The Minecraft environment is based on popular sandbox game and is widely recognized benchmark for Al research due to its complex and dynamic nature. The raw observation space consists of images with a resolution of 160x256. Minedojo has an 18-dimensional multi-discrete action space, allowing for diverse interactions such as movement, tool usage, and resource collection. However, since the original action space is too large and we only evaluate on task categories of "Harvest" and "Combat", we modify the action space in the same way as (Ma al., 2023; Zhou et al., 2024), resulting in 5-dimensional multi distrid action space. We select programmatic tasks (Fan et al., 2022) including "Hunt Cow", "Shear Sheep", "Chop Trees", "Milk Cow", and "Hunt Sheep". The prompts we use for each task are the same as Minedojo, which are listed in Table 3. A demonstration of the environment and tasks is shown in Figure 6. In our experiment, the resolution of image observations is 128x 28 and the action repeat is 1. The evaluation metric is the success rate that can better reflect the agent's performance compare to manually-designed rewards.
DMC and Franka Kitchen We use the implementation in (Mazzaglia et al., 2024) of tasks not originally existing in the DMC benchmark, such as several "stand" and "flip" tasks, for evaluation. The text prompts of language-based tasks Section 5.1 evaluated in DMC and Franka Kitchen are listed in Table 3. The text prompts we use in FOUNDER for DMC and Kitchen are mostly similar to GenRL (Mazzaglia et al., 2024). When evaluating baseline methods based on GenRL, we remain using the original prompts in the GenRL paper. In our experiment, the resolution of image observations is 64x64 and the action repeat 2.
B. Offline Dataset
We reuse the offline dataset collected by (Mazzaglia et al., 2024) for DMC and Kitchen, which consists of trajectories generated through performing Plan2Explore in the environment (Sekar et al., 2020) and from training buffers of RL agents designed to complete particular tasks within the corresponding domain. Additionally, (Mazzaglia et al., 2024) does not provide the Quadruped dataset, so we collect it on our own. For Minecraft, we use the same strategy to collect the environment data in Minedojo. Details the offline dataset information are listed in Table 4.
C. Implementation Details
C.1. Implementation Details of FOUNDER
Pretraining. At first, train DreamerV3 world model (WM) where WM states include deterministic and stochastic ones. The stochastic states are sampled from the categorical distribution as in DreamerV3. We then fix the WM to learn the remaining components (mapping function and temporal distance predictor), by encoding offline trajectory data to WM state space. During mapping function learning, the states mapped by Qu are in the same space of WM states, also include deterministic and stochastic parts. Therefore, We parameterize the two parts in the output layer of Qu for computing the KI. divergence in distribution alignment, and respectively using Gaussian and categorical distribution to parameterize the
12

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
Table 3. Prompts for each task in language-based evaluation, as well as expert and random agent score
Task
Prompt
Specialized agent score
Random agent score
cheetah stand
standing on the floor like a human
930
5
cheetah run
running like quadruped
890
cheetah flip
quadruped rotating flips
990
250
walker stand
standing up straight like a human walk fast clean like human run fast clean like human doing backflips
970
150
walker walk
960
45
walker run
770
30
walker flip
720
20
stickman stand
standing like a human robot walk fast clean like a human robot run fast clean like a human doing flips like human
970
70
stickman walk
960
35
stickman run
830
25
stickman flip
790
45
quadruped stand
spider standing
990
15
quadruped run
running fast clean like quadruped spider walking fast clean like a quadruped spider
930
10
quadruped walk
960
10
kitchen light
activate the light slide cabinet above the knobs opening the microwave fully open the burner becomes red pushing up the kettle
0
kitchen slide
0
kitchen microwave
0
kitchen burner
0
kitchen kettle
O
minecraft hunt cow
hunt cow shear sheep chop trees to obtain log obtain milk from a cow hunt sheep
minecraft shear sheep
minecraft chop trees minecraft milk cow minecraft hunt sheep
distribution of the two parts. Additionally, we learn an ensemble of mapping function, where we learn multiple output layers of stochastic state. The learning procedure of this ensemble module similar to plan2explore (Sekar et al., 2020), and the ensemble number is 5. During our learning of the temporal distance predictor Dg. we use the two-hot trick in Dy output layer, which is also used in the output layer the reward model in D3camerV3.
Behavior learning. We use the mean value of the goal distribution mapped from the VLM embedding as the goal state in DMC and Kitchen, while sampling goal states from the goal distribution in Minecraft for exploring more behaviors. FOUNDER and all the baseline methods do not use MOPO-style reward punishment (Yu et al., 2020; Lu et al., 2022: Wan et al., 2024) for unreliable transition prediction during policy learning, same as the GenRL code (Mazzaglia et al., 2024), ensuring a fair comparison. We apply normalization on the temporal cemporal-distance- distance- rewards for efficient policy learning.
Policy rollout. When acting with the environment, FOUNDER'S goal-conditioned policy needs goal state for every timestep's decision making. We use the same goal sampling approach in behavior learning, while only selecting the goal every 8 steps. This ensures short-perion
maniing consistency and robustness (Hafner al., 2022).
C.2. Implementation Details of baselines in DMC and Kitchen
For GenRL, follow the code in (Mazzaglia al., 2024) to train the MFWM learn policies for different tasks. For WM-CLIP, we follow the instruction in (Mazzaglia et al., 2024) to train reversed mapping from WM state space VLM space, minimizing the MSE loss of the predicted VL.M embedding with the ground-truth one. For GenRI-TTempD, we use the trained WM in GenRL to learn a temporal distance predictor, using only stochastic states as WM states input to the predictor since GenRL's WM encoder only a visual tokenizier (Mazzaglia et al., 2024) and does not include deterministic states during trajectory encoding. Then we use each step's 's predicted temporal distance between imagined trajectories and connected task sequence as the reward in behavior learning, replacing the original cosine similarity.
13

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
Table 4. Datasets composition.
Domain cheetah
num of observations
Subset
Subcount
1.8M
cheetah run
820K
cheetah plan2explore
1M
walker
2.5M
walker stand
500k
walker walk
500k
walker run
500k
walker plan2explore
1M
stickman
2.5M
stickman stand
500k
stickman walk
500k
stickman
500k
stickman plan2explore
IM
quadruped
2.5M
quadruped stand
500k
quadruped walk
500k
quadruped run
500k
quadruped plan2explore
1M
kitchen
3.6M
kitchen light
700k
kitchen slide
700k
kitchen microwave
700k
kitchen bottom burner
700k
kitchen plan2explore
800k
minecraft
3.4M
minecraft hunt
IM
minecraft shear sheep
610K
minecraft chop trees
8 810K
minecraft plan2explore
1M
C.3. Implementation of MineCLIP-IQL
We employ MineCLIP-IQL as a strong baseline for our experiments in the Minecraft environment. For policy learning, we utilize Implicit Q-Learning (IQL) (Kostrikov et al., 202 implemented via OfflineRL-Lib (Gao & Kong, 2023).
For the reward function, we derive the MineCI.I score by measuring the similarity between the video embedding and the prompt embedding, using the direct reward approach from (Fan al., 2022). Specifically, the reward is computed as:
PG (PG-NA Ny. .00
r max
(10)
where Nr denotes the number of prompts passed to MineCLIP, and Nr represents the baseline probability of randomly guessing the correct text-video match. To prevent highly uncertain probability estimates from falling below the random baseline, r is thresholded at zero. Following the policy learning approach in (Fan et al., 2022), we perform IQL on 512-dimensional vectors, which are low-dimensional embeddings of the original image observations in the offline dataset, encoded by MineCLIP's pre-trained encoder.
C.4. Evaluation Metrics
The metries and methods used for evaluating the pseudo rewards generated by FOUNDER and baselines during behavior learning are defined and outlined as follows. Since calculate all the metrics in Section 5.3 at the trajectory level, they measure the relationship between the pseudo-returns and the real returns of the trajectories.
Rank correlation quantifies the correlation between the ordinal rankings of the generated pseudo returns Vi.N and the corresponding ground-truth ones V1.N. It is formulated as:
RankCorr Cov(VI.N Vi.N)
(11)
(VVNN)o(VI.N)
14

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodicd Decision Making
where 1 N represents the indices of the trajectories.
Regret @ measures the gap between the return of the best trajectory and the return of the best trajectory selected within the top-k set, where the top-k set is determined based on pseudo returns. It is expressed as:
Regret@
max Vi" :EI:N
max VJ",
(12)
JEtopk(L:N)
where topk(1: : .V denotes the indices of the top K trajectories, ranked according to their pseudo returns V.
Binary bassificationn-asee evaluation Following (Fan et al., 2022), we convert our trained reward model into binary classifier and evaluate its performance. (Fan et al., 2022) incorporates human expert to label 100 good and 100 bad trajectories as ground-truth labels, and then use K-means clustering (K=2), to determine decision boundary 8 from the centroids' mean, classifying trajectories above 5 as successful. Following their practice, we use all methods' testing buffers stored during language task behavior learning (Section 4.3) and use the ground-truth reward to label the ground-truth "good" or "bad" trajectories. We then compute step-wise pseudo rewards using FOUNDER and baseline methods, averaging them per trajectory. We then use the same clustering-based approach to find the decision boundary and obtain the classification results with ground-truth labels. The classifier's performance is measured using Precision, Recall, and Fi-score, showing high agreement with ground-truth labels, making it reliable automatic evaluation method.
D. Additional Results
D.1. Model-Free Baselines
Prior work (Mazzaglia et al., 2024) has proven that traditional model-free offline RI. methods (IQI., TD3+BC, TD3) perform poorly in our multi-task reward-free setting compared to model-based counterparts, and we mainly compare FOUNDER to model-based baselines in Table 1. Traditional single-task model-free methods require training from scratch for each task, making them inefficient and impractical for multi-task solving.
To ensure a more comprehensive evaluation, we include HILP (Park et al., 2024), multi-task model-free method that achieves strong performance both in zero-shot offline RL and goal-conditioned RL, and TD3, the best-performing single-task model-free bascline according to GenRL's experiment (Mazzaglia et al., 2024). We assess their performance across eight tasks in the Cheetah and Kitchen domains, and the results are presented in Table 5. Following (Mazzaglia et al., 2024) for implementing single-task model-free methods like TD3, we use learnable DrQ-v2 encoder process the visual observations and a frame stack of 3. The reward each timestep is calculated as the cosine similarity between the VLM embedding of the text task prompt and the video embedding of the last k frames (k=8, same F FOUNDER and GenRL). For the implementation of HILP, we leverage their official code for zero-shot offline RL, and adopt the VLM-produced reward same as in TD3, for HILP''s downstream zero-shot policy learning.
Table 5. Normalized test performance of FOUNDER and model -ree baselines on Cheetah and Kitchen benchmark. We also include the performance of GenRL for a clearer comparison.
Task
TD3
HILP
GenRL
FOUNDER
Cheetah Stand
0.71 + 0.11
0.56 + 0.24
0.93 0.03
1.02 + 0.01
Cheetah Run
0.28 + 0.03
0.22 0.14
0.68 0.06
0.81 0.02
Cheetah Flip
-0.03 + 0.03
0.20 + 0.03
-0.04 + 0.01
0.97 + 0.02
Kitchen Light
0.10
0.00
0.00 + 0.00
0.97 = 0.18
Kitchen Microwave
0.10
0.00
1.00 + 0.00
1.00 + 0.00
Kitchen Slide
0.09
0.62 - 0.49
1.00 - 0.00
Kitchen Burner
0.37 + 0.12
0.00 +
0.35 - 0.48
0.60 + 0.49
Kitchen Kettle
0.03 + 0.07
0.00 + 0.00
0.35 0.48
0.33 + 0.47
We observe that FOUNDER consistently outperforms both single-task and multi-task model-free baselines. Model-free methods rely on zero-shot pseudo-rewards generated by FMs through cosine similarity between FM representations of task prompts and observations. However, the lack of embodied grounding in these FM rewards restricts their effectiveness. Notably, although HII.P demonstrates strong performance in multi-task reward- free offline RL. settings in a zero-shot manner,
15

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
when ground-truth rewards are given during downstream policy learning (Park et al., 2024), its performance deteriorates significantly when relying on FM-generated rewards, underscoring the importance of the grounding process. In contrast, model-based methods learn world model (WM) that captures environment dynamics, facilitating knowledge reuse across diverse downstream tasks and enabling explicit grounding of FMs. This fundamental advantage motivates us to integrate WMs and FMs to facilitate open-ended task solving.
D.2. Efficiency and Computational Costs
All experiments were conducted on an RTX 3090 GPU. For pretraining, training the MFWM in GenRL for 500k gradient steps requires approximately five days, whereas FOUNDER reduces this overhead to about three days by avoiding the seq2seq-style generative sequence modeling used GenRL. Moreover, pretraining the Hilbert representation and foundation policy HILP for 500k gradient steps takes around 30 hours.
For downstream tasks, training the actor-critic in the learned world model given task prompt for 50k gradient steps takes under five hours for both GenRL and FOUNDER. In the case of model-free methods, HILP enables zero-shot adaptation to downstream tasks, whereas single-task model-free methods like TD3 require approximately seven hours to train from scratch for 500k gradient steps.
We present a comprehensive comparison of the aforementioned methods the single-task model-free TD3, the multi-task model-free HILP, GenRL, and FOUNDER consideriig both performance and computational efficiency, Overall, we find that FOUNDER strikes strong balance between performance and learning efficiency, as shown Table
Table Overall comparison between TD3, HILP, GenRL and FOUNDER, considering both performance and computational efficiency
TD3
HILP
GenRL
FOUNDER
Performance
Low
Low Zero-shot
Medium
High
Multi- Task Adaptation
Learning from scratch (~ 7h)
Finetuning (~ 5h)
Finetuning (~ 5h)
Pretraining Overhead
N/A
30h
120h
72h
Overall Computational Cost
Low
Medium-Low
High
Medium
D.3. Real-world Video Tasks
We present experimental results on real-world video tasks using videos provided GenRL's code repository. The used file names of real-world videos are from the side', "guy_walking" walking', open microwave' respectively in GenRL's code repository, for specifying
withh_annds_up Sands_up_seen_ffee wwwpseeeeeommmmeeeeeee foooffrrohheee frrom_he_siee _hh__sid', spiderrdraw'
'dog running each task of "Cheetah Stand', 'Quadruped Walk", 'Cheetah Run', 'Stickman Walk' and 'Kitchen Microwave', respectively. The results are presented in Table 7.
Table 7. Normalized test performance of F FOUNDER and GenRL on real-works video tasks. We also list the performance of GenRL and FOUNDER corresponding language tasks upper bounds for comparison.
Task
GenRL_video
FOUNDER video
GenRL
FOUNDER Janguuge
Cheetah Stand
1.05 = 0.00
0.77 0.01
0.93 0.03
1,02 = 0.01
Quadruped Walk
0.66 + 0.01
0.99 0.01
0.73 0.19
0.90 + 0.05
Cheetah Run
0.63 + 0.01
0.60 0.01
0.68 + 0.06
0.81 + 0.02
Stickman Walk
0.52 = 0.02
0.85 0.02
0.83 + 0.03
0.91 = 0.03
Kitchen Microwave
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
Overall
0.77 + 0.01
0.84 + 0.01
0.83 + 0.06
0.93 + 0.02
FOUNDER again demonstrates solid performance compared to GenRL when generalizing to real-world video task under- standing and grounding, even matching GenRL's language- task solving performance. These results, coupled with FOUNDER's performance on cross-ents cross-embbddmeen and cross- viewpoint video tasks detailed in Section 5.2, confirm FOUNDER'S strong generalization capabilities to disti-iiittiittionn tasks.
16

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
D.4. Reward Evaluation
We offer the complete reward evaluation results in Section 5.3 on each of the 7 tasks, as shown in Table
E. Additional Discussions
E.1. Clarification on open-ended task solving
In our context, open-endedness specifically refers to agent's ability to ground and solve tasks specified through free-form user-defined task prompts via multimodal interfaces without predefined reward functions. FOUNDER develops this capability through integrating the high-level knowledge embedded in Foundation Models with the environment-speciffe modeling capabilities of World Models. The terminology "open-ended" is widely used in a similar way in (Fan et al., 2022; Team al., 2021; Qin et al., 2024), which aim to build generally capable agent, e.g. capable of solving free-form language tasks in (Fan et al., 2022).
E.2. Is FOUNDER'S performance gain solely comes from temporall-distance--ased reward?
FOUNDER grounds VLM representations into corresponding WM states without GenRL's computationally intensive sequence matching and enables deeper task understanding. It is FOUNDER'S overall architecture- -not isolated reward engineering techniques like temporal distance- that drives the performance gains.
From Table 1, even without temporal distance, FOUNDER w/o TempD already outperforms GenRL on static tasks (Stand tasks and Kitchen tasks). This demonstrates the inherent efficacy of our grounding framework, independent of reward shaping. FOUNDER w/o TempD struggles in dynamic tasks (Walk, Run) since single-frame goal grounding loses temporal awareness compared to GenRL (which uses multi-step seq2seq alignment) when using cosine similarity as rewards. At this time, the role of TempD-based rewards is to provide temporal information to the agent, with much lower training cost than GenRL's complex sequence matching. This is detailed in Appendix E.3.
Moreover, the performance degradation observed when simply adding temporal distance to GenRL (GenRL+TempDD, Table 1) further suggests that, it is FOUNDER'S 's overall architectural design, rather than isolated reward engineering, that drives the gains.
E.3. Case study: the influence of temporal distance
As mentioned in Appendix E.2, tempooal-ddistancc-baasd rewards mainly serve to enhance temporal awareness and provide complconn information for the FOUNDER agent. In this section, we present a study of the failure cases of FOUNDER w/o TempD on dynamic tasks. Analysis of GenRL's failure cases can be found on our website https///stesggoogl..omvviieee/
As shown in Table 1, while exhibiting strong performance static tasks (Stand and Kitchen), FOUNDER w/o TempD struggles in dynamic tasks (Walk, Run). When we plot the return curves during policy learning of dynamic tasks, we observe that, the agent can perform well at the very beginning of the behavior learning stage, but its performance deteriorates as training progresses, eventually reaching very low performance by the end (Figure 7). We also discover that the real return and pscudo return curves exhibit completely opposite trends during behavior lcarning, as shown in Figure 7. The real performance worsens as the agent maximizes the pseudo reward, causing reward hacking problem.
We then visualize the resulting trajectories of the policy the early stage and the final policy. Despite performing well the beginning, we find that at the end, the agent's behavior becomes more static. The agent may stay its initial position, appearing as if it is running walking, but in reality, it is only 'running' or "walking" at the original place without progressing forward, or moving forward at an extremely slow pace. This explains the poor final performance of FOUNDER w/o TempD in Table 1. Visualization of these trajectories can be found on our website https://sites. https://siee.goggle..ooe wtwpe////o//we///ooeetttttt
We conclude that, reward functions based on cosine similarity or other direct distance metrics may lead the policy to mimic the visual appearance of the goal, while overlooking the underlying task semantics and multi-step movement, particularly in dynamic tasks like running walking. Since FOUNDER-bard method maps the target sequence to single goal state in the world model, we hypothesize that using direct distance functions may result in a lack of temporal awareness.
At this time, the role of TempD-bases rewards is to provide temporal information and crucial task-completion information
17

---

FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
Eval Return (Real Rewarrd
400
Walker Run
475
Return (Pscudo Roward)
Eval Return (Real Reward)
700
Cheetah Run
Evoll Return [Pseuso Rewaard)
Eval Return (Real Reward
800
Stickman Walk
Eval Return PPSodOO Rowarr)
425
450
600
47%
700
575
700
100
100
500
450
600
650
500
250
300
100
400
325
200
375
300
20
20
200
55550
(xmesteppx<100)
Timestop(=10")
Timestep(x1o7"
Figure 7. Testing returns of the FOUNDER w/o TempD agent during behavior learning stage for Walker Run, Cheetah Run and Stickman Walk in terms of real performance and pscudo return. As the agent maximizes pecudo-eeturss derived from cosine similarity (blue curves), the agent's 's performance (red curves) degrades over the course of training, starting strong (Point A) but ending poorly (Point B).
Walker Run
Cheetah
Stickman Walk
Eval Return (Pseudo Reward)
Eval Return (Pseudo Reward)
Eval Return (Real Reward)
600
Eval Retum (Real Rewarg rr))
700
Eval Return (Real Reward
12
10
300
40
Eval Return (Peeddo
600
400
500
| M
600
30
300
400
5
200
20
40
200
20
200
100:
10
20
30
40
50
20
30
40
50
10
20
30
40
50
Timestep(xx10)
Timestep(x110))
Timestep(x10')
Figure& 8. Testing returns of the POUNDER agent during behavior learning stage for Walker Run, Chectah Run and Stickman Walk in terms of real performance and pseudo returm, when using TempD-hased rewards. The trends of the two return curves now generally consistent.
to the agent, with a much lower training cost than GenRL's complex sequence matching. FOUNDER then avoids the reward hacking problem, as the trends of the two return curves are now generally consistent, as shown in Table The incorporation of TempD-based rewards enables FOUNDER to achieve superior performance on dynamic tasks such Walk and Run, compared to FOUNDER w/o TempD and GenRL.
Moreover, Table 1 shows that incorporating TempD will not only benefit temporal dynamic tasks, but also match or surpass the performance of FOUNDER w/o TempD on 8 of the 9 static tasks (Stand and Kitchen tasks). This demonstrates that the positive influence of Tempp-based rewards is consistent.
18

---

FOUNDER: Grounding Foundation Models in World Models for Open-Endee Enend Embodied Decision Making
Table Evaluation of the consistency between learned pseudo rewards and ground-truth rewards for different tasks in DMC.
Methods
Corrt Regret, Precisionf Recallf F11
cheetah run
GenRL
0.49
0.19
0.50
0.77
0.60
WM-CLIP
0.80
0,42
0.57
1.00
0.73
GenRI-TTempll
0.23
0.22
0.40
0.52
0.44
FOUNDER w/o TempD
-0.05
1.11
0.00
0.00
0.00
FOUNDER
0.61
0.01
0.52
0.69
walker stand
GenRL
-0.11
0.12
0.77
0.53
0.63
WM-CLIP
0.78
0.02
0.81
1.00
0.89
BemRL-TempD
-0,34
0.12
0.70
0.15
0.24
FOUNDER w/o TempD
0.78
0.00
1.00
0.88
0.94
FOUNDER
-0.11
0.09
1.00
0.02
0.04
walker walk
GenRI.
0.11
0.22
0.52
0.29
0.37
WM-CLIP
0.65
0.00
0.97
0.67
0.79
GenRL-TempD
0.09
1,21
0.43
0.33
0.37
FOUNDER w/o TempD
-0.20
1.56
0.00
0.00
0.00
FOUNDER
0.55
0.12
1.00
0.33
0.50
walker run
GenRL
0.77
0.09
0.95
0.85
0.90
WM-CLIP
0.71
0.07
1.00
0.90
GenRL-TempD
0.12
0.96
0.67
0.49
0.56
FOUNDER w/o TempD
-0.19
0.97
0.00
0,00
0,00
FOUNDER
0.58
0.04
1.00
0.33
0.50
stickman stand
GenRI.
0.23
0.24
0.08
0.17
0.11
WM-CLIP
-0.36
0.86
0.07
0.14
0.09
GenRI-TTemD
0.06
0.95
0.07
0.11
0.09
FOUNDER w/o TempD
0.27
0.79
0.10
0.17
0.12
FOUNDER
0.54
0.08
1.00
0.72
0.84
stickman walk
GenRL
-0.46
1.19
0.43
0.53
0.47
WM-CLIP
0.02
0.28
0.77
0.50
0.61
GenRL-TempD
-0.14
1.24
0.58
0.42
0.49
FOUNDER w/o TempD
-0.64
1.33
0.00
0.00
0.00
FOUNDER
0.82
0.10
1.00
0.41
0.59
stickman run
GenRL
-0.21
0.55
0.00
0.00
0.00
WM-CLIP
0.23
0.13
0.29
0.62
0.39
G
0.36
0.56
0.39
0.98
0.56
FOUNDER w/o TempD
-0.15
0.57
0.00
0.00
0.00
FOUNDER
0.82
0.02
1.00
0.95
0.97
19


Reviews:
[{"review_id": "YCdvPVhrWk", "reviewer": "Reviewer_e54P", "summary": "The paper proposed FOUNDER, a novel framework that integrates Foundation Models (FMs) with World Models (WMs) to enable reward-free, open-ended decision-making in embodied environments. The central idea is to ground FM representations into the WM state space, allowing GCRL through imagination. Instead of relying on manually crafted reward functions, FOUNDER estimates the temporal distance to goal states as an intrinsic reward signal, leading to superior task generalization. The proposed method is evaluated on multi-task offline visual control benchmarks, including the DeepMind Control Suite, Franka Kitchen, and Minecraft, demonstrating strong performance in learning task semantics from text or video prompts, especially in challenging cross-domain settings. Empirical results show that FOUNDER significantly outperforms prior methods like GenRL by leveraging deeper semantic understanding rather than relying on step-by-step visual alignment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful comments and valuable feedback on our paper. We are delighted to receive recognition of our method's solid performance and strong task generalization demonstrated by extensive and detailed experiments. Please find below our detailed responses to the reviewer’s comments.\n\n**Other model-free baselines**:\nPrior work (e.g., GenRL) has proven that traditional model-free offline RL methods (IQL, TD3+BC, TD3) perform poorly in this setting compared to model-based approaches. These methods rely on zero-shot pseudo-rewards generated by Foundation Models (FM) through cosine similarity between FM representations of task prompts and observations. However, FM rewards are not explicitly grounded in the embodied domain, limiting their effectiveness. Additionally, traditional model-free methods require training from scratch for each task, making them inefficient and impractical for multi-task solving.  \n\nHowever, to ensure a comprehensive evaluation and address your concerns, we now include HILP [1], a multi-task model-free method that achieves strong zero-shot RL performance in goal-conditioned RL, and TD3, the best-performing single-task model-free baseline according to GenRL’s experiment. We assess their performance across eight tasks in the Cheetah and Kitchen domains, alongside GenRL for a clearer comparison. The results are on [our website](https://sites.google.com/view/founder-rl). We find that FOUNDER consistently outperforms both single-task and multi-task model-free baselines, and our prior claim is reinforced. We sincerely appreciate the reviewer’s suggestion regarding model-free baseline comparisons.\n\n**Efficiency and computational costs**:\nWe appreciate the reviewer's emphasis on computational efficiency and provide a comparison of the four typical approaches mentioned. All experiments were conducted on an RTX 3090 GPU. For pretraining, training the MFWM in GenRL for 500k gradient steps requires approximately five days, whereas FOUNDER reduces this overhead to about three days by avoiding the seq2seq-style generative sequence modeling used in GenRL. Pretraining the Hilbert representation and foundation policy in HILP for 500k gradient steps takes around 30 hours.\n\nFor downstream tasks, training the actor-critic on a given task prompt for 50k gradient steps takes under five hours for both GenRL and FOUNDER. In the case of model-free methods, HILP enables zero-shot adaptation to downstream tasks, whereas single-task model-free methods like TD3 require approximately seven hours to train from scratch for 500k gradient steps. Overall, we find that FOUNDER strikes a strong balance between performance and learning efficiency.\n|   | TD3 | HILP | GenRL| FOUNDER |\n| :--- | :--- | :--- |:--- | :--- |\n| **Performance** | Low | Low | Medium | High |\n| **Multi-Task Adaptation**| From scratch (~7h) | Zero-shot | Finetune (~5h) | Finetune (~5h) |\n| **Pretraining Overhead**| N/A | ~30h | ~120h | ~72h |\n| **Overall Computational Cost**| Low | Medium-Low | High | Medium |\n\n\n**Scalability of FOUNDER**\nWe have included a brief discussion on the scalability of FOUNDER to long-horizon tasks in Section 6. While our current experiments focus on short-horizon tasks, FOUNDER’s architecture is inherently designed to support long-horizon reasoning through two key mechanisms: (1) the world model’s temporal abstraction via latent state transitions and (2) the foundation model’s semantic grounding, which enables hierarchical task decomposition. To fully unlock this potential, there may exist three enhancements: (i) integrating more powerful sequence modeling architectures (e.g., [2]) or continual learning methods (e.g., [3]) a for persistent context tracking, (ii) implementing FM-guided curriculum learning [3] to phase in complex tasks, and (iii) developing a meta-controller for dynamic subgoal generation. We thank the reviewer for raising this concern, and we will explore it in our future work.\n\n[1] Park S, Kreiman T, Levine S. Foundation Policies with Hilbert Representations[C]//International Conference on Machine Learning. PMLR, 2024: 39737-39761.\\\n[2] Samsami M R, Zholus A, Rajendran J, et al. Mastering memory tasks with world models[J]. arXiv preprint arXiv:2403.04253, 2024.\\\n[3] Feng T, Wang X, Zhou Z, et al. EvoAgent: Agent Autonomous Evolution with Continual World Model for Long-Horizon Tasks[J]. arXiv preprint arXiv:2502.05907, 2025.", "claims_and_evidence": "The paper claims that FOUNDER enhances task generalization by bridging FM knowledge with WM-based decision-making, which is supported by empirical results showing higher success rates across diverse domains. Another claim is that the temporal distance-based reward function provides a more reliable training signal than traditional visual similarity metrics, validated through improved reward consistency analysis. The authors also claim that FOUNDER performs well in cross-domain tasks, where large domain gaps exist between task prompts and the embodied environment. This is demonstrated by its superior performance in generalizing across different camera viewpoints and agent embodiments in tasks like cheetah and Minecraft.", "methods_and_evaluation_criteria": "The evaluation protocol covered both locomotion and manipulation tasks with multiple baselines, including GenRL, WM-CLIP, and ablations of FOUNDER without temporal distance prediction. Except there are no real-world applications, the experiments are extensive and detailed.\n\nThe author talked about efficiency or inefficiency at a few places, so maybe a quantitative result and analysis for efficiency would be more convincing.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experiments are extensive, with evaluations on both text-based and video-based task prompts across multiple simulated environments.", "relation_to_broader_scientific_literature": "The paper is well-positioned within the literature on foundation models for reinforcement learning, goal-conditioned RL, and world models. It builds upon prior work such as GenRL, DreamerV3, and Choreographer, differentiating itself by focusing on task grounding through state-space alignment.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper is well-written and straightforward to read, and the experiments are extensive and solid. One major concern is I am not sure how the proposed method compared with other methods, especially the model-free method, in both the performance, efficiency and even real-world applications. I noticed the author stated that \"Model-free offline RL methods like IQL (Kostrikov et al., 2021b) and TD3+BC (Fujimoto & Gu, 2021) involve the cumbersome process of retraining from scratch for every task, and zero-shot FM-generated rewards using task prompts and observations has been shown to perform poor in (Mazzaglia et al., 2024). Therefore, we compare FOUNDER only with model-based methods.\" I may hold a more conservative opinion on this: for the \"cumbersome process of retraining from scratch for every task\" I believe there are lots of multi-task on-policy work that may also be a good and fair comparison, and seemed the published date of 2021 is far more behind 2024. Nevertheless, requiring intensive extra experiments for model-free methods may be hard in the rebuttal phase, I may expect more analysis on the efficiency part.", "other_comments_or_suggestions": "No", "questions_for_authors": "All experiments primarily focus on tabletop and 2D short-horizon tasks. Given this, do the authors anticipate that FOUNDER’s integration of  WM could be even more beneficial for long-horizon tasks that require memory and temporal abstraction? Specifically, could the WM’s ability to model latent state transitions and predict future states provide an advantage in tasks where agents must recall and utilize past information over extended time horizons? If so, what modifications or enhancements might be necessary for FOUNDER to effectively scale to such settings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xkpecVGaiw", "reviewer": "Reviewer_2yAX", "summary": "This work proposes FOUNDER, a method that leverages Visual-Language Models (VLMs) to get representations from visual observations and train RL agents on World Models imagination, starting from an offline dataset of trajectories. It proposes a method of aligning the embeddings from the VLM with the latent states learned by the World Model via a learned mapping function. This mapping function is optimized to reconstruct the VLM embeddings while also covering the WM latent distribution (via KL divergence minimization). For behavior learning, FOUNDER first learns a temporal distance predictor model, which is then used as reward model for the imagined rollouts. The agent then is learned via actor-critic RL in these imagined trajectories. The work presents experiments in DMC/Kitchen/Minecraft environments, claiming improvements over closer baselines (GenRL and WM-CLIP [1]).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful comments and valuable feedback. Below are our responses to each of their concerns:\n\n**Universally applicable claim**: Our claim of task-agnosticism refers to FOUNDER’s architecture—world model, mapping function, and reward generator—enabling deployment across downstream tasks specified via text or video inputs without architectural changes. This highlights its broad applicability, not guaranteed universal performance. We appreciate your feedback and will clarify this in the final version.\n\n**Open-endedness**: In our context, “open-endedness” refers to the agent’s ability to ground and solve user-defined tasks from free-form prompts via multimodal interfaces, without predefined reward functions. This differs from definitions emphasizing self-evolving exploration or unbounded learnability. FOUNDER achieves this capability by jointly learning the WM, mapping function, reward generator, and downstream policy from offline data. This definition aligns with similar works [1-3]. We will clarify this in the final version.\n\n**Performance of GenRL**: Our reproduction of GenRL follows its official code and datasets. Our reproduced results (GenRL_ours) align with GenRL’s arXiv v1 metrics on in-distribution DMC tasks but underperform on Kitchen tasks, likely due to domain properties. Despite GenRL v2’s improved performance, its implementation was not updated. FOUNDER outperforms GenRL in both cases.\n\n|  | GenRL_v1 (reported) | GenRL_v2 (reported) | GenRL_ours (reproduced) | FOUNDER |\n| --- | --- | --- | --- | --- |\n| **DMC** | 0.74 $\\pm$ 0.02 | 0.82 $\\pm$ 0.01 | 0.75 $\\pm$ 0.08 | 0.87 $\\pm$ 0.03 |\n| **Kitchen** | 0.69 $\\pm$ 0.15 | 0.76 $\\pm$ 0.05 | 0.50 $\\pm$ 0.32 | 0.89 $\\pm$ 0.17 |\n\n**Effectiveness of FOUNDER's components:** Our key innovation is grounding VLM representations into WM states without GenRL’s costly sequence matching, enhancing task understanding beyond reward correction. As shown in Table 1, FOUNDER w/o TempD outperforms GenRL on static tasks (e.g., stand, Kitchen), while it requires TempD for dynamic tasks (e.g., Walk, Run) to improve temporal awareness. TempD efficiently provides temporal structure at a much lower computational cost than GenRL’s sequence matching. TempD’s addition to GenRL does not improve its performance, confirming FOUNDER’s architecture drives success.\n\n**The “+1 reward” heuristic**:  The +1 operation smooths rewards, improving learning. Experiments show agents can achieve similar results without this shaping given more training steps. Training curves comparing shaped and unshaped rewards are available on [our website](https://sites.google.com/view/founder-rl).\n\n**Minecraft Experiment**: High variance in Minecraft experiments is mainly due to the environment’s stochasticity. To better reflect statistical significance, we now report 95% confidence intervals and provide clearer learning curves on our website. FOUNDER matches GenRL on 2/5 tasks and outperforms it on the remaining three.\n\n**Comparison with model-free baselines**: We directly compare **HILP**, a multi-task model-free approach, and **TD3**, the strongest single-task model-free baseline per GenRL. As detailed in our response to reviewer e54P, these comparisons justify the need for a model-based approach, with FOUNDER achieving a strong balance between performance and efficiency.\n\n**Limitations related to the temporal distance predictor**:\n\n1. The predictor learns with a predefined sequence length during training, but during testing, it outputs the predicted temporal distance independent of sequence length.\n2. Temporal distance excels in cross-domain tasks and metric evaluation, where GenRL and cosine similarity methods may fail due to reward hacking. While extreme OOD scenarios may still pose challenges, FOUNDER’s success shows its robustness in capturing deep task semantics rather than relying on brittle visual correlations.\n3. We appreciate the reviewer’s insight on mining negative samples. We follow LEXA [4], but acknowledge the risk of mislabeling. This can be mitigated by filtering out high-similarity negative pairs or actively selecting challenging cross-sequence pairs via feature-space searching. We will include an ablation on this in the final version.\n\n**Limitations from offline data quality**: We agree with the reviewer that dataset properties limit reward-free offline RL methods and have discussed this in Section 6. We welcome further discussion on dataset dependency concerns, as the reviewer did not fully elaborate on the complete questions.\n\n[1] Fan L, et al. Minedojo: Building open-ended embodied agents with internet-scale knowledge. NeurIPS 2022. \n\n[2] Team O E L, et al. Open-ended learning leads to generally capable agents. arXiv preprint.\n\n[3] Qin Y, et al. Mp5: A multi-modal open-ended embodied system in minecraft via active perception. CVPR 2024.\n\n[4] Mendonca R, et al. Discovering and achieving goals via world models. NeurIPS 2021.", "claims_and_evidence": "I found two core claims from the paper:\n\n1) The proposed method improves the alignment between the VLMs representations and the WM representations, which allows it to capture the semantic of the tasks better. The evidence would be the downstream results in the DMC/Kitchen/Minecraft environments. While I agree this is a proper form of evidence, I have some concerns on the experimental design (to be described later)\n\n2) The temporal distance predictor provides a more consistent and informative reward signal, while prior methods are prone to reward hacking. The presented evidence is on qualitative analysis of failure cases and correlation analysis between the proxy reward and ground truth reward.\n\nThere is one additional claim, which is not central as the previous ones, but still debatable: The proposed components of the method are task-agnostic and therefore “universally applicable to any downstream task and effectively facilitating open-ended task solving in embodied environments”. While it is clear that the modeling assumptions are task-agnostic, I think it is too strong to claim that they are universally applicable to any downstream task as there is no evidence these components would actually generalize/learn any task and also it is unclear how much this actually relies on the properties of the offline dataset.", "methods_and_evaluation_criteria": "I believe that the proposed method, considered baselines, and evaluation criteria make sense. Nonetheless, I am not sure if the described problem setting and parts of the narrative really align. More concretely, the problem setting describes as a goal “solving open-ended tasks in the context of offline reinforcement learning from visual observations”, and it is unclear the open-ended terminology here, which is often related to interestingness (information gain) and learnability [2]. From what was presented in the paper, it is not clear how a method in a pure offline setting, with a clear definition of goals (as explicitly model as goal conditioned policy) would be learning open-ended behavior. \n\n\nApart from that, there are some design choices which are questionable/unjustified (See weaknesses below)", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "I have some major concerns in terms of the experimental design.\n\n- The presented numbers (Table 1) does not reproduce what was claimed in GenRL paper [1]. In fact, there are substantial changes in some environments. It is unclear why this happens. Is the offline dataset used in this work different? If so, why? Given these discrepancies, it is hard to tell if GenRL was fully reproduced or if these methods are too sensitive for the employed offline dataset.\n\n- The paper does not bring any offline RL method as baseline, arguing that it involves a “cumbersome process for retraining from scratch for every task”. While I understand this, I believe they are necessary here to justify the need for a more complex, model-based approach. Furthermore, this is the standard in prior work as well - for instance, GenRL does bring these methods, which clarifies this point. \n\n- The minecraft experiments look very inconclusive. The standard errors are too spread for all methods in a way that is hard to claim that FOUNDER-based methods consistently outperform GenRL with statistical significance.  This is concerning since it is supposed to be the most challenging benchmark evaluated in the work, and the gains are very unclear.", "relation_to_broader_scientific_literature": "In my perspective, the main contribution of this work is to identify and highlight that the reward modeling aspect of GenRL leads to reward misspecification and hacking. The proposed method for alignment embeddings also looks relevant, although it is also unclear if this really works better than GenRL method (from Table 1, GenRL works better than Founder w/o TempD).", "essential_references_not_discussed": "I couldn’t find any essential references not discussed.", "other_strengths_and_weaknesses": "I believe the paper should discuss more some potential limitations:\n\n1) Limitations related to the temporal distance predictor:\n\n    - The temporal distance predictor relies on a predefined sequence length T, but on test-time this is not available (unless the problem setting is limited to fixed-horizon episodic tasks, but this is not specified). \n\n    - Also, why isn’t the temporal-distance predictor reward model also not prone to reward hacking? In a harder environment I would expect it to misgeneralize and present the same effect.\n\n    - There is also a potential limitation on mining negative examples for the temporal distance predictor training, which the paper does not explore and only performs randomly. \n\n2) The “+1 reward” is not justified/discussed in proper depth. As a simple reward shaping, I would not expect to have such a big impact in the final performance as presented in Table 5. It would be very important to discuss the effect of this heuristic.\n\n3) One last thing, while not specific to FOUNDER, but more broadly for these reward-free, offline settings: there is a strong reliance on the properties of the employed offline dataset. I believe this should be discussed in a limitations sections, cont", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see my questions in the previous points.\n\n**References** \n\n[1] Mazzaglia et. al. GenRL: Multimodal-foundation world models for generalization in embodied agents. NeurIPS, 2024.\n\n[2] Hughes et. al. Position: Open-Endedness is Essential for Artificial Superhuman Intelligence. ICML, 2024.\n\n\n**Please refer to my rebuttal comment for the updated score**", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "56QZS52To3", "reviewer": "Reviewer_GtTJ", "summary": "This paper introduces FOUNDER, a framework for grounding foundation models (FMs) in world models (WMs) to enable generalizable representation learning, multi-modal prompting, and dense reward prediction. FOUNDER shares conceptual similarities with GenRL but incorporates temporal information to enhance reward prediction and goal-conditioned reinforcement learning (RL) for more flexible behavior cloning. Experimental results on DMC and Kitchen benchmarks demonstrate that FOUNDER outperforms GenRL baselines.\n\n## update after rebuttal\n\nThe author's response addressed my concerns. I would like to remain my original ratings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their valuable feedback on our paper. We are delighted to receive recognition of our method's sound experimental design and solid experimental results. Please find below our detailed responses to the reviewer’s comments.\n\n**The influence of temporal distance**: We appreciate the reviewer for raising these important questions regarding temporal distance (TempD). Generally, the role of TempD-based rewards is to enhance temporal awareness and provide task-completion information to the FOUNDER agent.\n\nFrom Table 1, using cosine similarity between VLM representations of task and observations as rewards, FOUNDER w/o TempD already performs well on static tasks (Stand tasks and Kitchen tasks). However, FOUNDER w/o TempD struggles in dynamic tasks (Walk, Run). We observe that in dynamic tasks, the agent can perform well at the very beginning of the behavior learning stage, but its performance deteriorates as training progresses, eventually reaching very low performance by the end. We also discovered that the real return and pseudo return curves exhibit completely opposite trends during behavior learning. The real performance worsens as the agent maximizes the pseudo reward, causing a reward hacking problem. We then visualize the resulting trajectories of the policy at the early stage and the final policy. Despite performing well at the beginning, we find that at the end, the agent’s behavior becomes more static. The agent may stay at its initial position, appearing as if it is running or walking, but in reality, it is only 'running' or 'walking' at the original place without progressing forward, or moving forward at an extremely slow pace. This explains the poor final performance of FOUNDER w/o TempD in Table 1. These training curves, trajectory visualization, and detailed analysis can be found on [our website](https://sites.google.com/view/founder-rl).\n\nWe conclude that reward functions based on cosine similarity or other direct distance metrics may lead the policy to mimic the visual appearance of the goal, while overlooking the underlying task semantics and multi-step movement, particularly in dynamic tasks like running or walking. Since the FOUNDER-based method maps the target sequence to a single goal state in the world model, we hypothesize that using direct distance functions may result in a lack of temporal awareness.\n\nAt this time, the role of TempD-based rewards is to provide temporal information and crucial task-completion information to the agent, with a much lower training cost than GenRL's complex sequence matching. In Table 1, FOUNDER then avoids the reward hacking problem and achieves superior performance on dynamic tasks like Walk and Run, compared to GenRL and FOUNDER w/o TempD.\n\nMoreover, Table 1 shows that incorporating TempD will not only benefit temporal dynamic tasks but also match or surpass the performance of FOUNDER w/o TempD on static tasks. Kitchen Burner is the only 1 of the 9 static tasks in Table 1 where FOUNDER is outperformed by FOUNDER w/o TempD, which does not constitute statistical significance.\n\nFurthermore, for Minecraft tasks, FOUNDER w/o TempD performs surprisingly good, and incorporating TempD could weaken the performance on several tasks. We hypothesize that the data quality and the stochasticity nature of the Minecraft environment may have a negative impact on the learning process of the TempD predictor. When temporal predictions become noisy, falling back to cosine similarity serves as a safer baseline, and using visual similarities as pseudo-rewards in short-horizon Minecraft tasks is beneficial and will not cause problems on Walk or Run tasks in DMC. However, when TempD learns accurate temporal dynamics, its temporal credit assignment mechanism provides provable advantages in reward-free multi-task RL, as is proved in prior works [1-4]. We will incorporate approaches discussed in our rebuttal with Reviewer 2yAX, as well as techniques in [1], for better TempD learning under stochastic and complex environments.\n\n[1] Myers V, Zheng C, Dragan A, et al. Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making[C]//International Conference on Machine Learning. PMLR, 2024: 37076-37096.\n\n[2] Park S, Kreiman T, Levine S. Foundation Policies with Hilbert Representations[C]//International Conference on Machine Learning. PMLR, 2024: 39737-39761.\n\n[3] Park S, Rybkin O, Levine S. Metra: Scalable unsupervised rl with metric-aware abstraction[J]. arXiv preprint arXiv:2310.08887, 2023.\n\n[4] Mendonca R, Rybkin O, Daniilidis K, et al. Discovering and achieving goals via world models[J]. Advances in Neural Information Processing Systems, 2021, 34: 24379-24391.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "NA.", "experimental_designs_or_analyses": "The experimental design appears sound.", "relation_to_broader_scientific_literature": "FOUNDER shares similarity with GenRL but introduces temporal distance prediction and a goal-conditioned RL framework, resulting in more robust performance. It also relates to WM-CLIP, which learns a mapping from WM states to FM representations. However, FOUNDER instead grounds FM representations in the WM state space, leading to improved performance.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strength\n- The paper is clearly written and well-organized.\n- The experimental results effectively demonstrate the advantages of the proposed FOUNDER framework.\n\nWeakness\n- The proposed temporal distance prediction does not consistently improve performance.\n  - For example, in Kitchen Burner (Table 1), the mean score drops significantly from 1.0 to 0.6.\n  - Similar performance degradation is observed in the Minecraft environment, where Figure 5 shows that FOUNDER w/o TempD consistently outperforms FOUNDER.\n  - It remains unclear whether temporal distance prediction is a broadly applicable method or merely a task-specific trick. The paper does not provide sufficient analysis regarding: 1) Which types of tasks benefit from temporal distance prediction? 2) Under what conditions does it fail? 3) What are the underlying reasons for performance degradation in certain tasks?", "other_comments_or_suggestions": "None.", "questions_for_authors": "Please refer to the Weaknesses section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "18iGr6GV7I", "reviewer": "Reviewer_bGqN", "summary": "FOUNDER proposes a method that leverages the generalization capability of world model dynamics alongside the prior knowledge embedded in foundation models to improve embodied decision making, and demonstrate its effectiveness through extensive experiments across diverse domains.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful comments and valuable feedback on our paper. We are delighted to receive recognition of our method's effectiveness, robustness, and scalability through extensive experiments, and for acknowledging that our claims are supported by convincing evidence with well-justified evaluation criteria (baselines/metrics). Please find below our detailed responses to the reviewer’s comments.\n\n**1. Novelty concerns:** We appreciate the reviewer's comparison between FOUNDER and LEXA. While both methods utilize world models and temporal distance prediction, their technical contributions differ fundamentally. LEXA focuses on discovering diverse goals and learning to achieve them during online exploration, enabling adaptation to image goals for downstream tasks. In contrast, the core idea of FOUNDER is to ground foundation model (FM) representations (from multimodal tasks) into the world model (WM)'s latent space through explicit mapping functions, operating entirely offline. This enables physics-consistent translation of VLM embeddings into actionable WM states. It should be noted that world model and temporal distance predictor in FOUNDER only serve as tools to implement this paradigm, whereas the core contribution resides in bridging the gap between FM's general knowledge with WM's physical dynamics, establishing a new methodology for open-ended task solving and setting FOUNDER apart from prior methods using VLMs in RL.\n\n**2. Hyperparameter ablation**: We appreciate the reviewer's feedback regarding the hyperparameter settings. For frame window size (k=8), this choice strictly aligns with two critical considerations: (1) InternVideo2's pretrained temporal encoder requires fixed 8-frame inputs, as validated in its implementation. (2) Retained GenRL's 8-frame configuration for fair comparison. For the kl weight β, we use β=1.0 by default. To address your concerns, we conducted an ablation study on the Cheetah and Kitchen domains to validate their sensitivity, and the results are on [our website](https://sites.google.com/view/founder-rl). We find that FOUNDER is not sensitive to the kl weight parameter, while 1.0 is the best choice.\n\n**3. Goal sampling in behavior learning**: In fact, the goal state zg of a given video/text task during behavior learning is sampled from the goal distribution inferred from the video/text VLM embedding, e.g., using the learned mapping function: zg∼Qψ(⋅∣eg). This is discussed in Equation (6) and Appendix C.1.\n\n**4. Question about temporal distance learning**: We thank the reviewer for raising this valuable point. Learning a temporal distance predictor from quasi-static data may pose risks. However, the world model provides proper regularization to the input states, and such datasets are rare in real applications. To address your concerns, we provide experiments on learning a temporal distance predictor on the Stand datasets, where trajectories are generated by the expert policy in Walker Run, and validate the predictor's accuracy in predicting the same WM states as near-zero temporal distance. The results on [our website](https://sites.google.com/view/founder-rl) show robustness of temporal prediction in such settings.\n\n**5. Out-of-distribution tasks**: In section 5.2, we evaluate the performance of FOUNDER on out-of-distribution video tasks, where the visual appearance of task video prompts differs significantly from the agent's environment observations and training data, including cross-embodiment and cross-viewpoint settings on DMC and Kitchen, and FOUNDER demonstrates superior performance than other baselines. Also, in Sections 5.1 and 5.4, we test FOUNDER on out-of-distribution language tasks like Cheetah Flipping and Hunt Sheep, where the training dataset does not contain buffers of these tasks, and the performance of FOUNDER exhibits its generalization abilities. To further address your concerns, we present experimental results on real-world video tasks on [our website](https://sites.google.com/view/founder-rl) using videos provided in GenRL's code repository, and FOUNDER also demonstrates solid performance compared to GenRL.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence; further details are discussed in the following section.", "methods_and_evaluation_criteria": "The proposed methods are appropriately evaluated using diverse criteria in environments such as Minecraft and DMC.", "theoretical_claims": "The mathematical formulations are well-structured and clearly presented, making them easy to understand.", "experimental_designs_or_analyses": "I checked the novelty of this submission and discussed the details in the section addressing its strengths and weaknesses.", "relation_to_broader_scientific_literature": "The paper provides a thorough analysis of the strengths and weaknesses of both existing world models and foundation models, proposing a method that leverages their complementary benefits to enhance embodied decision-making performance.", "essential_references_not_discussed": "This paper appropriately cites and discusses the prior works necessary to understand and explain the proposed method.", "other_strengths_and_weaknesses": "- strength:\n    - The authors conduct extensive experiments across multiple domains, demonstrating the approach’s robustness and scalability.\n    - The selection of baselines appears well-reasoned, and the chosen metrics for comparison seem appropriate for evaluating the method’s improvements.\n- weakness:\n    - It seems that FOUNDER can be viewed as essentially LEXA [1] plus Foundation Model : LEXA already utilizes a similar world model framework along with temporal distance prediction. While FOUNDER’s approach and results appear compelling, it is less clear how novel the core idea is compared to existing solutions.\n    - The key hyperparameters (e.g., the frame window size k, the KL weight, etc.) are not clearly summarized, and the paper would benefit from an ablation study detailing how these values affect performance.", "other_comments_or_suggestions": "No", "questions_for_authors": "- In the behavior learning phase, it’s unclear in the paper how goal sampling is done during policy training. Do you randomly pick from the offline dataset, take one as the goal?\n- If you’re calculating temporal distance with MSE as in Equation (7), wouldn't there be an issue if, for instance, the robot is standing still in the dataset or exhibits periodic behavior, resulting in incorrect learning?\n- Since you’re using a foundation model, you really need to demonstrate generalization in the experiments. Does it still work if you query out-of-distribution goal images or prompts that weren’t used in training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors introduce FOUNDER, a novel framework that integrates Foundation Models with World Models for reward-free decision-making in embodied environments by grounding visual-language model representations into the World Model state space through a learned mapping function. This approach enables goal-conditioned reinforcement learning with stronger generalization capabilities compared to traditional manually crafted reward functions. Experiments across DeepMind Control Suite, Franka Kitchen, and Minecraft demonstrate FOUNDER's superior performance over baselines like GenRL and WM-CLIP. 

After the rebuttal process, the paper received positive scores [3,3,3,4], with all reviewers supporting acceptance. While using foundation models to provide self-supervised reward signals for reinforcement learning is not a new concept, FOUNDER's impressive experimental results clearly demonstrate the framework's significance and broad applicability to open-ended embodied AI.  Given these strengths and the consensus among reviewers, I recommend accepting this paper for publication.
Decision: Accept (poster)