RLZero: Direct Policy Inference from Language Without In-Domain Supervision
Authors: Harshit Sikchi
Siddhant Agarwal
Pranaya Jajoo
Samyak Parajuli
Caleb Chuck
Max Rudolph
Peter Stone
Amy Zhang
Scott Niekum
Venue: NeurIPS 2025 poster ()
Keywords: Zero-shot RL
Unsupervised RL
Behavior Foundation Models
Primary area: reinforcement_learning
TL;DR: A method to produce policies directly from language instructions without in-domain supervision

Abstract: The reward hypothesis states that all goals and purposes can be understood as the maximization of a received scalar reward signal.  However,  in practice, defining such a reward signal is notoriously difficult, as humans are often unable to predict the optimal behavior corresponding to a reward function.  Natural language offers an intuitive alternative for instructing reinforcement learning (RL) agents, yet previous language-conditioned approaches either require costly supervision or test-time training given a language instruction. In this work, we present a new approach that uses a pretrained RL agent trained using only unlabeled, offline interactions—without task-specific supervision or labeled trajectories—to get zero-shot test-time policy inference from arbitrary natural language instructions. We introduce a framework comprising three steps: *imagine*, *project*, and *imitate*. First, the agent imagines a sequence of observations corresponding to the provided language description using video generative models. Next, these imagined observations are projected into the target environment domain. Finally, an agent pretrained in the target environment with unsupervised RL instantly imitates the projected observation sequence through a closed-form solution. To the best of our knowledge, our method, RLZero, is the first approach to show direct language-to-behavior generation abilities on a variety of tasks and environments without any in-domain supervision. We further show that components of RLZero can be used to generate policies zero-shot from cross-embodied videos, such as those available on YouTube, even for complex embodiments like humanoids.

RLZero: Direct Policy Inference from Language Without In-Domain Supervision
Harshit Sikchi . Siddhant Agarwal*:! Pranaya Jajoo" :2, Samyak Parajuli* Caleb Chuck"., Max Rudolph".! Peter Stone Amy Zhang Scott Niekum
1 The University of Texas at Austin, University of Alberta Sony AI, UMass Amherst
Abstract
The reward hypothesis states that all goals and purposes can be understood as the maximization of received scalar reward signal. However, in practice, defining such reward signal is notoriously difficult, as humans are often unable to predict the optimal behavior corresponding to a reward function. Natural language offers an intuitive alternative for instructing reinforcement learning (RL) agents, yet approaches either require costly supervision or
previous Iinguage-conditions test-time training given language instruction. In this work, we present new approach that uses pretrained RI. agent trained using only unlabeled, offline task-specific supervision zero-shot test-time policy inference from arbitrary natural language instructions. We introduce a framework comprising three steps: imagine, project, and imitate. First, the agent imagines a sequence of observations corresponding to the provided language deseription using video generative models. Next, these imagined observations are projected into the target environment domain. Finally, an agent pretrained in the target environment with unsupervised RL instantly imitates the projected observation sequence through a closed-form solution. To the best of knowledge, Language-to-beaavee generation abilities on a variety of tasks and environments without any in-domain supervision. We further show that components of RLZero can be used to generate policies zero-shot from cross-embodied videos, such as those available YouTube, even for complex embodiments like humanoids.
interactions
labeled trajectories-to get
method, RLZero, is the first approach show direct
1 Introduction
Underlying the many successes of RL lies the engineering challenge of reward design. often takes a skilled expert to engineer reward function which still might not be fully aligned to the true task specification [9]. Not only does this challenge restrict the scaling of RL agents, but also makes those agents inaccessible any inexperienced in reward design. Even for experts, confidently specifying reward functions is generally infeasible because learning agents have often been found hack them [38, 4, 18, 31]; i.e. the learned policies for the reward function produce behaviors that do not align with what the human intended. Language is an expressive communication channel for human intent and allows bypassing traditional reward design, but learning mapping from language to behaviors has historically required collecting and annotating behaviors that correspond language [26, 33, 54] in the domain where the agent is be deployed. Scaling this strategy is impractical as it requires labeling the agent's 's large space of behaviors with corresponding language descriptions, and repeating the process every new domain.
This paper is motivated by the vision that a true generalist agent should be able to solve new tasks that presented to it in a way that is natural for people to specify. Language offers intuitive channel
Equal contribution, Equal Advising. Correspondence to hsikchi @ uduxxas.edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Imagined trajectory
Behavior Foundation Model
"Do a lunge"
Projection
lungle Masiing
Imitation/
Distribution
Language Instruction
Matching
RLZero framework of imagine, project, imitate: A video trajectory is imagined using text prompt and each frame is projected to agent's observation space. A closed form solution to imitation learning for BFMs trained with unsupervised RL is used to obtain policy that mimics the projected video behavior.
Figure I:
for task specification, but prior language-based policy learning methods have required significant agent training for specific language commands. In contrast, the objective of this work to enable generalist agents to translate language commands into behaviors, with no nost-command training. The solution we identify can be divided into two parts: (1) being able communicate the task specification through language commands, and, (2) quickly producing policies for the inferred task. Large-scale multimodal foundation models [79] provide us with the first part of the solution. Trained on large amounts of internet data, they can generate video segments that communicate what performing task entails. Unfortunately, these video segments can be -fist--diittibution the agent's s domain with different dynamics and embodiments. Thus, this work addresses this challenge by projecting individual frames generated video to agent observations using semantic similarity [61, 89], thus grounding them in the agent's ' observation space.
This presents us with the next part of the problem: How do we quickly infer a policy that solves the communicated task i.c mimics the grounded imagined trajectories? We utilize the recent success of successor- wewtrrfettttuttteeeeeeeeeeeeeeee unsupervised RL methods, sometimes called Behavior Foundation Models (BFMs) [76, 55, 1, 67] for direct policy inference. These models are pretrained in the agent's domain using tank-sgnostic prior interactions of arbitrary quality from the environment to encode wide variety of behaviors. During inference, near optimal policies corresponding to any reward function can be obtained zero-shot manner, i.e. in closed form without any gradient updates.
Putting these two parts together, we frame the problem of language-to-policc as a translation from one space, of language, another space, of behaviors, as specified by successor features. To address this problem, we leverage the capabilities both of video language models to act as a pretrained medium for translating language to states, and of unsupervised RL methods to provide zero-shot solution for distribution matching. The resulting approach is inspired by the imagination capabilities of humans to picture in their mind possibilities in the real world [63, 64, 59] and then ro rely on past experiences, memories, and abilities to inform their actions. Our framework RLZero (illustrated figure 1) comprises three main steps: a) Imagine: Imagine trajectories given language command; b) Project: The frames of imagined trajectories are projected to real observations of the agent; c) Imitate:
Leverage a pretrained BFM to directly output policy that matches the state visitation distribution of the imagined trajectories. Our experiments indicate that RLZero is a promising approach to designing an interpretable link connecting humans to RL agents. We find that RLZero a variety of tasks where reward function design would require expert
an effective method reward engineer. We further find that RLZero exhibits successful direct cross-embodiment transfer, the first approach to be able to do to the best of our knowledge. This paper's 's main contribution is the framework of imagine, project, and zero-shot imitate, which diverges from the prior approach of using VLMs reward functions- wwich can be hacked and instead focuses zero-shot imitation with unsupervised RL, which admits a unique solution that matches the imagined behavior.
2 Related Work
Language and Vision Based Control: There has been a rich history of using vision [13, 32, 68, 45] and language [73, 69, 27, 48] as task primitives in RL. While several of these infer a reward function from language instructions and/or VLMS [62, 6, 68, 85, 88, 43] and require training an RI. agent using this reward, others have used innttrctoon pairs to train the agents with the hope

---

of generalizing to related instructions. A number of recent works have utilized large pretrained
foundation models to generate video plans [16, 17. 44] or plans containing textual actions [29, 3, 80). Some similar methods generate these video plans using diffusion models or energy models [43].
But these methods often require in-domain labeled trajectories to finetune the large models or train these diffusion models [43, 44]. These often rely on APIs [39], inverse dynamics models [16, 44], or goal-conditioned policies trained using text-expert trajectory pairs [17] to convert these plans to actions in the agent's space. This means that these methods obtain open-loop sequence of actions that is a translation from the video or textual plan to the agent's 's action space. RLZero differs from all of these as: (1) It does not have any test time training or planning and directly obtains executable closed loop policy, (2) it does not require any in-domain task desk-dependent supervision like expert labeled trajectories to either train or finetune the video trajectory generation or for policy training. Recent work [49] proposed an approach to generate the video plans without requiring in-domain supervision. But this work still requires training the RI. agent for each given task prompt which can be tedious and time consuming.
Zero-shot RL: Zero-shot RL [77] promises the ability to quickly produce optimal policies for any given task defined by a reward function. A wide variety of methods have been developed to achieve zero-shot RL, which are in some ways generalizations of multi-task RL [12]. Most of these works assume a class of tasks where they can produce policies zero-shot. These tasks can be conl-connii [34, 19, 2, 65, 47], linear span of certain features [15, 5] or some combination of some skills [20, 21, 56]. Recent works [84, 76, 77, 55, 1] employ a successor measure- mneauree-based representation learning objective to be able to provide near-optimal policies for arbitrary reward function subject to model capacity constraints. Our work leverages these methods and finds the best reward supported by the representations that will produce the language-condttioned imagined trajectory.
3 Preliminaries
RLZero uses generative models to imagine trajectories from language prompts and directly produces a policy that results in observations imitating the imagined trajectory. In this section, we introduce the notion of trajectory generation, imitation learning, and unsupervised RL.
Multimodal Video-Foundation Models (ViFMs) and In-Domain Video Generation: Multimodal
ViFMs [78, 79, 75] enable the understanding of video data in shared representation space of other modalities such as text or audio. These shared representations can be used to condition video generation on different input modalities [36, 7]. Notably, these models guide content, style, and motion, or employ an image as the initial frame for a subsequent video sequence. For this work, we use video generation models VM that generate a sequence of video frames {ú1, i.n... given a task specified in natural language I by first converting the language prompt to common embedding space across modalities; formally, VM : + (4-))))--
utilize text prompts to
Imitation Learning through Distribution Matching: We consider learning agent in a Markov Decision Process (MDP) [58, 71] which is defined as tuple: M (8. p,,,,, d,o) where S and A denote the state and action spaces respectively, p denotes the transition function with MMs'|s, a) indicating the probability of transitioning from s s' taking action a; denotes the reward function, 7 E (0, specifies the discount factor and dy denotes the initial state distribution. The reinforcement learning objective is to obtain a policy π S + (AA) that maximizes expected return: E, 0 E ('r():),, where we use Ex to denote the expectation under the distribution induced by a (+)8)), 8841 ~ p(-(st;a)) and A(A) denotes a probability simplex supported over A.
An imitation learning agent does not have access to the reward function, R, but has access to an "expert" trajectory (or set of "cxpert" trajectories) from policy that maximizes the reward function. Distribution matching objectives [25, 52] for imitation learning have been commonly used in some recent work [24, 66]. The objective in these methods is min, D(p", p²) where 5² is the visitation distribution of the policy æ (defined by the probability of being in state s starting from the initial state distribution su and following the policy T). p is the visitation distribution exhibited by the "expert" trajectory and D is a function to compare the closeness of the distributions. f- f-Divergences commonly used as measure of distance between distributions.
Successor Measure based Unsupervised RL:: Successor Measure [8] based learning has been recently studied [76, 1] as unsupervised RL objective for its ability to describe long-term behavior
3

---

of the policy in the environment. Mathematicallyy, successor measures define the measure over future states visited as M
as, X) ET Σ ( E X(s,,)
XA C.
(I)
( (s p(s*) and trained dataset dº of offline interactions of the form {s, a, s'}. these methods facilitate extraction a state-representation (ss) that is suitable for RL. Then, policies (where the policies are represented using latents 2) are learned offline that are near-optimal for a reward function defined in the span of learned state-features (s) <p(s) At test time, the policy for any given reward function (r(s)) can then be obtained with such BFM (zero-shot, with no additional experiential data or gradient updates) using closed-form solution to the following linear regression:
Representing the successor measure for any policy
ž = min E [(r(s) <p()) =)]² = zR = Eg (x)(((()) ((s)")(=))))))) (+(()) r(*)
For this work, we use (f(s), TT)) to represent a BFM trained with successor- measur based unsupervised RI..
4 RLZero: Zero-Shot Prompt to Policy
RLZero uses components trained without any explicit in-domain supervision to map language to behaviors. For each domain, we consider dataset do of reward-free interactions (s, a, s'} and BFM (pp(s,, π2) pre-trained on d In the following sections, we describe the steps involved in detail. First, we present how an imagined trajectory is generated from a prompt. Then, we discuss how this imagined trajectory projected to real observations of an agent. Finally, we describe the zero-shot procedure for inferring policy that matches the behavior in the imagined trajectory. The latter two steps can also be used to directly infer policy when instead language prompt, a cross-embodied video demonstration is provided.
4.1 Imagine: Unsupervised Generative Text-Conditioned Video Modeling
Grounding language to tasks has historically [26, 33, 54] required costly annotation labels that map language to task examples specified through image or state trajectories. Large video-language foundation models (ViFMs) help lift that requirement by training on vast amounts of internet videos, thus giving us rich prior of grounding language commands to videos. We leverage the unsupervised video modeling approach from GenRL [49]. In the absence of any in-domain text labels, the approach uses an off-the- thel- rideo-language encoder with video encoder fo and language encoder fi trained on internet scale data to encode sequence of observation frames (01:T) a embedding space shared with its language description. A generative video model conditions on this latent embedding to generate sequence of image observations (11:)). The video model (VM) trained to reconstruct the original observation frames (oIrT) and at test time can simply be queried with a language instruction. Similar to GenRL, we use Intern Video2 as the video-language encoder and use a simplified GRU architecture for video generation. We refer the readers to GenRL for more details training the video generation model and accounting for multimodality gap between video-text encodings. Training the video generation model does not require text labels mapping language to tasks and is fully unsupervised. Thus during inference, given a language instruction e', we obtain a sequence of frames (M(()()') (fz, fg...r)) that represents the environment domain. Figure shows an example of what these imaginings look like using the trained video generation model. Given the generated video of text description, GenRL learns cetion-conditional world model to train a new policy for each text-instruction with model-based RI. This makes policy inference for variety of tasks costly, unsafe, and memory-ines -eefor-inefficient. In contrast, with RLZero we present a way to lift this limitation with unsupervised RL. to directly infer a policy given generated video in the sections below.
imagination of what the task looks like in
4.2 Project: Grounding to Agent's Observation Space
The imaginings produced by VM can be noisy, unrealizable, and not exactly representative of the domain. We propose to use semantic-similarity based retrieval to obtain nearest

---

frames in the dataset of the agent's prior environmental interactions do to project individual frames of imagined trajectories to real observations as shown Semantic matching also allows us the flexibility to replace imaginings with video demonstration of a task by a different
Fig 2.
embodiment agent in a different domain (e.g.
as we will demonstrate in Section 5.2). In this work, we use an off-the-shelf vision-language embedding approach for retrieval, SigLIP [89], to map both the imagined frame and agent observation to the same latent embedding space, which has been pre-trained for similarity matching
A
Figure 2:
Grounding Imagination Observations: We use nearest image retrieval defined by cosine similarity in the embedding space to output real observation from the dataset that matches the imagined frame.
Real
internet-scale dataset. As single image observation is not informative of key state variables such velocity or acceleration, we use the frame stacking technique common in RL [50] when finding the nearest observation from the dataset corresponding to any imagined frame. We use an encoding function & : I + Z to individually map sequence of images shared image-text embedding space.:
{(o k(i-ka) [E(0,- (i+-:a))
0t arg max
Vt [7].
Above we use k previous frames (usually set to 3) to identify state variables in order to improve matching accuracy. Using the offline interaction dataset, we find corresponding proprioceptive states in addition to the real observation that we will subsequently use as target sequence to imitate. We rely on proprioceptive states for the subsequent unsupervised RL phase, but in general BFMs may be trained with image observations if the state information is unavailable. While this step allows us to ground imagination to a sequence of real observations that the video expects the agent to follow, action inference still remains a challenge as the sequence might not be dynamically feasible or be available as a trajectory in the dataset to behavior clone.
4.3 Imitate: Zero-Shot Distribution Matching with Unsupervised RL
The resulting projected sequence of states might be dynamically infeasible and out of distribution from the trajectories in the offline dataset. Our key insight is to leverage unsupervised RL techniques to discover new skills from an offline dataset and use state-marginal matching to find policy that induces the closest dynamically feasible sequence imitating the imagination. This section demonstrates that after pretraining a successor-meant approximate closed-for fore-formm solution to state marginal matching exists without requiring any further gradient steps or interaction with the environment.
based unsupervised RL agent with reward-free transitions, an
We take the distribution matching perspective of imitation learning [28, 25] and find a policy that matches the state marginal distributions of the grounded imagined trajectories (expert). Given a dataset
of oflline environmental interactions (d with distribution p). we pretrain a successor-measures based unsupervised RL agent [1, 77] (Gp(s), π²) that provides us with mapping from reward function R to the corresponding ²n that induces a near-optimal policy Tzn as shown in Eq. 2. For the case of state-marging matching, finding the optimal policy reduces to finding the optimal latent 2 minimizing the following distribution matching objective:
Zimit arg min (s) (s)))
where _ and pT* are state visitation distribution of the "expert" imagined trajectories and of the policy T respectively and 'D can be chosen to be wean-ssuared error, f-divergence, Integral Probability Metrics (IPM), etc. In general, minimizing the distance via gradient descent can provide solution zimit to distribution matching. For the special case of KL divergence, Theorem shows that Zimit can be obtained using learned distribution ratio between expert and offline interaction dataset p² /p.
Theorem 1. Define J(Hr r) to be the expected return of policy n under reward r. For an offline (=(s) ) DKI(p",,
dataset d with density p. a learned log distribution ratio: v(s) =log( p(.)
pB)
5

---

Imigelnngnng rwward (VLM-IND)
Vidro-hnginge miard
Canki.
(Bas:
TDH
I0L.
Loing Down
231 04-112611
-316697.69770 71)
215 (120448.50.45 47)
S/S (419 56)
415 (199 de
(0.54)
515 (524.34-31.70) -3,778
Walk like a luman
1/5 (35.051.78.000
(30.36-34 34,58
3/5 (40 499491455775
45 (145
(146861.44.0 41l)
5/S (632 09
16. 313
Si5 (704.68- (6)
Wulker
Run human
(50 (5800155.00
161
US (100 24:89 (6)
2/5 (285 14463911 91)
5IS (316
S03
(875.49
Cartohral
4S (3200NN-22880
-1677600-440 04- 12)
SS (1SL 16-67
4SS 454 50.55)33
51S (252.44 (3.96
465 (254.80 12.74)
Strut like a horse
5/5 (310.05:500.22
(16461=100.45
1/5 (102) (5:03-5::69-00 45)
315 (21) 24125015
5:5 (361445-100:077
S:5 (45.6661111.30
Crast like a wores
4/5
(16.48:272.99)
(44.5:166.18)
1:5 (84.45:33697))
2/5 (127.144110011
0.5 015(127 (127.48::34.81
315 (141.86-223.0))
Quatruppl
Dunce
89:26
(8.19:4.755
2.2
3/5
2442
115 (14.0745
4/5 (15, 115.1127771
001229-333 5.29)
Walk using three legs
215 (15 9425.60) (..5
(6.003.61)
2/5 (7 9hd (97)
315 (Tab6el 75)
455 (3470-66.56)
5/S (27 20.22--722 7.29)
Bullnainng wae two lags
2/5 01..0..2 52)
-6.11-995
2/5 (5.60 69 19)
(5 3.05)
515 0.20 2.75)
SI5 (22.50 2.15)
(82.93
7.54)
3/5 88.8
35
2/5 (100)
Handsaand
255 (15.
0842.
(5065-2.9)
Lie downs
3/5 (219 3510.55-588.771
(203 99.216.39) 39)
2/5 (255 (2:4002:444 25)
3/5 (360.7645)) 764433
3/5 (469.52 17.45)
2/5 (202 00:2657::34473 75)
Cheetal
Bunny hup
215 (218 44:134.50 56)
(11 241322235553
1/5 (192 78455 55)
315 (129 26 (37)
415 (220 62 591
SIS (234.70::446
Jump high
(17: 104-99 (2)
41.00.01)
IUS (18) 96-4 40
SSS 824 983
51%
17)
(212.23
Jump
leps backdlip
Quadraped walk
(379, 34:21447)1
1993664:20.300 30)
315 (187 11877244--66677
3/5 28.20.9
2/5 (318
4/5 (388 02-377277
Stand place like dog
4/5
(474855:4344)
515 (258.58:88 26)
US (169.9814.00)
2/5 (238.25:55))
25:2222::7700 (3)
315 (4..05
down stable
(2200777
(1322119.27)
(50.86661444)
1S5 (28.8013.0))
(686.56 386.6)
4/5 (841.48 226.34)
Sichon
138,
(191
Praying
150..66:22.22 27)
13432.511
015 760.669.177
3/S (192.75:42.00 30)
4/5 1147744:54.00
Headhtand
215 (2t) 14::23
(14.5417:133 133
215 (48, 111-2.511
1/5 (13.8416602)
455 (71.7
33,77)
45 (7187-66.20 6.28)
Punch
2/5 (73.422:119 19.00)
(50.23 23 58,08)
3:5 (74.7847.01)
415 (65.761:22.6) 26)
S/S (181.08 69.58)
415 (216.44 37,81)
Plank
(394 204:161 27)
(507 46--299 41)
(16.19155..11 19:45. /70
0V5 664-5343345 433
US (19104:41.00 (5)
(653.0
Averago
$1.25 (148.97)
Rose Model (114.50)
40% (67.55)
(4.85((18.79)
76.8% (246.48)
83.2 % (295.11)
Table I: Winrates computed by GPT-44o of policies output by different methods when compared to base policies trained by IDa+-aaalla reward. Bolded distribution-matching returns denote statistically significant improvement over the second best method under a Mann-Whitne Whitney U test with a significance level of 0.05.
+ DKI (p" (8, a), p(s, a)) where rimmit (s) = u(8) Vs. The corresponding Zimit m(s) (ss)] where minimizing the upper bound is given by Zimitt = (s))(() = 1 denotes state features learned by the BFM.
-T, pimstt )
Thus, with the reward function pimit (s) r(s), we can use the solution of Zimit = Ep (s) (s) to retrieve the policy that mimics the grounded imagined behavior. However this reward function still requires learning a discriminator to obtain the distribution ratio for each task, which can lead to instabilities. A heuristic yet performant alternative is to use shaped reward function r(s) = ex(s), similar to Pirotta et al. [57]. which allows closed-form inference (Zimsit E, ( ((s)] =
E, [+)] = e (s)(s)) without learning a discriminator, any gradient steps or interaction with the environment. We compare both approaches in Appendix C.5. The performance for both these methods are almost identical and we defer to the latter one (gradient- frree inference approach) in all our experiments. The complete algorithm for RLZero can be found in Algorithm 1.
5 Experiments
Our experiments seek to understand the quality of behaviors that RLZero is able to produce in two settings (a) Language to Behavior, and (b) Cross-embodied Video to Behavior Generation. We note that none of the experiments below assume in-domain supervision such as annotations of trajectories in the environment with their task label, or expert demonstrations corresponding the specified tasks either for training the video generation model or training the behavioral foundation model.
We consider continuous control tasks from the DeepMind control suite (Cheetah, Walker, Quadruped, Stickman) and HumEny (3D Humanoid). The selected environments are diverse and complex, with their proprioceptive state space ranging from 17-358 dimensions and action space ranging from 6-69 dimensions. All of the environments use an environment horizon of 1000 steps, except HumEnv, which has horizon of 300 steps. We use Cheetah, Walker, Quadruped, and Stickman environments to demonstrate language to behavior generation capabilities of RI 1.rero and use Stickman and HumEnv environments to demonstrate cross-embodied abilities of RLZero to directly produce behaviors from -utto--distribution videos, such as those available on Yobe or AI Al-generated. Our choice of Stickman and HumEnv environments for cross- cross-embodied behavior generation is motivated by the ease of availability of third-person individual human videos on internet as these environments reflect human morphology. We use an off-the-shelf pretrained BFM [74] trained for HumEnv with large MoCap datasets of real human behavior without assuming access to dataset used for training the BFM. This allows us to demonstrate the generality of the RL.Zero framework to plug policy model trained with a different unsupervised RI. algorithm to produce behaviors from emoss-dmbodied videos. Our evaluation spans 25 tasks for language-to-behall and 17 tasks for aidco-to-bchavion
Setup:

---

Implementation: For the environments Cheetah, Walker, Quadruped, and Stickman, we collect dataset d0 of environmental interactions of the form (s, a, s' purely by a random exploration algorithm RND [10] (10 million transitions for Stickman and 5 million transitions for the rest the Stickman environment, we additionally augment this dataset with replay buffers of the agent trained for walking and running to increase the diversity of behaviors, increasing the dataset size by another 2 million transitions. The detailed composition the datasets can be found in Appendix B.3. We use the method from Section 4.1 to learn a text-conditioned video generation model using dataset dº without requiring any in-domain task-trajectory labeling. We
of the environments).
successor-featuus based unsupervised RL algorithm, Forward-Backward Backward [76], to pretrain general purpose policy BFM that can output a near-optimal policy given any reward function without test-time training or learning. For the projection step involved in language to behavior generation, we use SigLIP [89], an off-the-shelf image-text embedding model.
Baselines: For our evaluations, we consider the setting where the agent is not allowed environment interactions for learning during test time for fair comparison and all baselines have access to same offline dataset d This setting reflects the ability of the agents to use prior interaction data to learn meaningful behaviors. We compare against approaches that use foundation models trained with internet scale data to label rewards for offline transitions conditioned on text. These rewards can be computed using cosine similarity with text prompt per image observation of the agent in VLM-RL [62] or per sequence of image observations using video-embedding model. We use SigLIP as our image-text embedding model and Intern Video2 as video-text embedding model for fair comparison. Video-language embeddings take into account context and can potentially lead to more accurate reward estimation. Once the rewards are obtained, we use TD3 [23] and IQL [37] as the representative offline RL algorithms to obtain policies. We also compare to recent approach GenRL. that performs model-based RL in the environment to learn policy test time by maximizing similarity to text-conditioned generated video. All baselines train a seperate policy for each task whereas RLZero trains a single policy for all tasks per environment unsupervised RL algorithm that uses TD3 as the underlying RL method.
Figure 3: Examples for cross embodied imitation:
RLZero can mimic motions demonstrated in YouTube Al generated videos zero-shot. Top 2 rows: Stickman (2D Humanoid), Bottom 2 rows: SMPL. 3D Humanoid.
using
Evaluation Protocol: We provide both qualitative and quantitative comparisons with baselines. The evaluation of behaviors generated given text prompt a video can be challenging unlike the traditional RL setting, we do not have access to a ground truth reward function. Instead, we have prompts that can be inherently ambiguous but reflect the reality of human-robot interaction. An obvious evaluation metric is to ask humans how much the generated behavior resembles their expectation of the behavior given the prompt. We multimodal LLMs to qualitatively evaluate such preferences as a proxy to human preferences, as recent studies [14] have shown them to correlated (up to 79.3%). To quantitatively compare the generated behaviors, we compute returns under discriminator as a reward function that is trained to distinguish proprioceptive states corresponding to the projected frames of video (imagined real) offline dataset do similar to [46].
5.1 Benchmarking Language to Continuous Control
The ability specify prompts and generate agent behavior allows us to explore complex behaviors that might have required complicated reward function design. We curate set of 25 tasks across DM-control environments. Each of the agents has unique capabilities as a result of its embodiment,
7

---

and the prompts are specified to be reasonable tasks to expect for the specific domain. For each prompt, we generate behaviors for 5 seeds. The qualitative performance of any given method is evaluated as the win rate over the base method. We chose the base model for our comparisons as the policies trained via TD3 on image-language rewards. For each seed, we present the observation frames that the output policy generates and pass it to Multimodal L.M capable of video understanding, which is used as a judge. Since the number of tokens can get quite large with the long default horizon of the agent (1000 frames), we subsample the videos by choosing every 8 frames and selecting the first 64 frames of size 256 x 256. We observed this subsampling to retain temporal consistency and the effective horizon (8 x 32 256) to be long enough to demonstrate the task requested by the prompt. We consider a more fine- ime-grained quantitative metric for comparison - average return under the distribution eestrbbutioo-matccingg matching reward function. We learn a discriminator between the projected states for a given imagination (p and the offline interaction dataset p. Under the shaped reward function (Section 4.3), r(s)= e"(s) pB/ we compute returns for all the methods; higher return indicates that a method is better able to match the projected imaginations.
Table 1 demonstrates the win rates by different methods when evaluated by -- as well as distribution matching
Video Descriptions
SMODICE
ML2ero
Hurtaan backflip position
(16.05 = 3.44)
21 + 0.02)
Doowwardd-B fancind-facinng dog yoga pose
(15.63 + 2.74)
+ (/4.
0.04)
Stickinon (2D (eDanneddd))
Cow yoga pose
16.72 7.81)
5/4.99 1 0.02)
Downward dog one log raised
-7.04
(/5.35 = 0.08)
Lying on back, leg mised
19.07 1.207)
(12.55 1 0.01)
returns. We find that RLZero achieves win rate of 83.2% when compared
Lying on back, both legs raised
3.77)
(9,55 : 0.08)
High plank yoga pase
(12.83 1: 7.76)
5/5 (15.08 1 0.03)
Sitting down, legs out front
(12.01 + 1.06)
S/S (15.24 + 0.02)
Warrior III pose
(17.27 = 5.09)
(15.22 + 0.01)
to the best baseline (GenRL), which achieves win rate of 76.8%.
Front splits
(13.44 +: 2.39
4/5 (15.21 + 0.01)
kurate kick position
(50.02 1. 0.02)
5/5 (19999 0.01)
HunnErr (3D Hemamoid)
A cat doing handstand
(0.19 0.24)
S15 (199.86 0.02)
An urabesque ballet position
(10.02 1: 0.01)
S/5 (199.92 0.04)
RLZero shows statistically significant improvement over baselines under distribution matching Table 2: Cross- Embodied Eval.: Distribution Matching Return & Winrates
Animated wikiHow cartwheel Running
(0.19 0.24)
(199.91 0.04)
(58.08 = 0.46)
S15 (199.87 0.01)
Lying crunches
(0.10 0.20)
S15 89 0.04)
Plank position
(0.018 = 0.03)
S15 (199.91 0.04)
returns. This indicates that RLZero is able to produce qualitatively better behaviors as well as do quantitatively better imitation than baselines. An important point to note that all methods except RLZero requires test-time learning. whereas RLZero provides a gradient- freee approach to instantly generating behaviors from text prompts. As an example, the closest performing baseline, GenRL requires test-time training with every task averaging > 3-5 hours of training on NVIDIA-A40 GPU compared to our method, which requires Rv 25 seconds to output a policy. The main bottleneck in RLZero is the projection step, which requires finding the closest real observation to each frame under CLIP-like similarity metric and can be made significantly faster by using rector-databases specialized for retrieval. Figure 7 shows examples of behaviors output by RLZero some of the prompts from our evaluation set. Additionally, all basclines require training a new policy model at test time for cach prompt, opposed to RLZero, which uses a single policy, making our approach parameter efficient.
5.2 Video to Continuous Control: Can RLZero succeed at cross-embodied imitation?
The intermediate stage in RLZero of matching the closest observations in the offline dataset to a frame from video is based on semantic similarity. This means that RLZero can generalize to out-of-domain demonstrations building on the zero-shot generalization of CLIP-like representations. Subsequently, we can skip the imagine step completely if we are given an expert video demonstration. To investigate this, we consider a collection of videos scraped from Youtube as well as videos generated by open-source vidco generation tools like MetaAl and empirically test if RLZero is able to replicate the behaviors. We focus Humanoid environments (Stickman, i.c 2D Humanoid, and HumEnv, i. 3D Humanoid) here. For HumEnv [42], we use an open-source BFM [74] for policy generation given a reward function and do not assume access to the dataset used for training the BFM.
Table 2 shows the results for cross-embodee cross-embodied imitation across 17 video-clips. We use similar metrics to Section 5.1, but modified the GPT-4o prompt to take in the frames from the original video instead of specified task description. We compare against SMODICE [46] which allows for using state-only observational data in conjunction with suboptimal offline data for imitation learning. This allows us to ablate the quality of imitation produced by a successor measure-based method that uses one policy model for all tasks as opposed to SMODICE which trains new policy for each task. GenRL
8

---

requires a world model of the environment which is not available for HumEnv as we only have access the pretrained policy. RLZero achieves win rate of 80% against SMODICE for Stickman and 100% for HumEnv. This matches the observation from [57] that DICF-based methods lag behind in performance on observation-only imitation tasks. Figure 3 shows a qualitative comparison of the video and the obtained behavior on a few videos. Details about videos used can be found in Appendix C.3.
5.3 Ablation and Limitations
Imagination-free behavior generation: While the imagine, project, and imitate framework allows for the interpretability of the agent's behavior, we investigate if we can amortize the imagination and embedding search cost by directly mapping the language embedding to the skill embedding the BFM's latent space. For this, we consider sampling 2 uniformly in the latent space the BFM and embedding the generated image observation sequence through video- lacoo-laggagee encoder, which we denote by e. Given the observation sequence, generate the Zimst using the zero-shot inference process and learn mapping from + Zimit using small 3-layer MLP. On the same tasks considered in Fig 3, we observe imagination-free RLZero to have a win rate of 65.71 % over TD3 base model on Walker environment when compared to RLZero that had a win rate of 91.4% (detailed results in Table 7). A thorough explanation of imagination-free RLZero can be found in Appendix B.6.
(a) Imagined behaviors: Top: stickman: 'raise hand while standing in place', Bottom: walker: 'kick'
(b) Failed projection for cross-embodied video.
Figure 4: Failure Cases in RLZero
Our proposed method RLZero is not without failures. The stages of imagination and projection fail individually, but the failures remain interpretable, i.e., by investigating the videos and the closest state match, we can comment on the agent's 'ents ability to faithfully complete that task to certain extent.
Failures:
What cannot imagine, I cannot imitate: The video generation model used our work is fairly small and limited in capability. We encountered limitations when generating complex behaviors with this model and found to be sensitive to prompt engineering. Fortunately, as models get bigger and are trained on larger set of data, this limitation can be overcome. Figure 4a shows some examples of these failures with the corresponding prompts.
2. Limitation of semantic search- based image retrieval: In this work, we used SigLIP, which has shown commendable performance for image retrieval tasks. We observed failure cases in the following scenarios (e.g. Figure 4b): a) Background distractors: We observe the image-similarity to latch on to features from the background and produce incorrect retrieval; b) Rough symmetries: In tasks where the agent is roughly symmetric (e.g. Walker when the head and legs are almost identical with slight difference in width) the image retrieval fails by giving an incorrect permutation w.r.t the rough symmetries.
6 Conclusion
Specifying desired behavior to a learned agent is a longstanding problem in sequential decision-making. In this work, we take step towards leveraging intuitive mediums such as language and cross-embodiment to specify desired policies. The RLZero framework bypasses costly annotations, reward hacking, and deployment time training by leveraging the rich capabilities of behavioral foundation models. Our introduced framework demonstrates a technique for directly translating from intuitive mediums for task specification to the latent space of policies without any test-time training. We demonstrate that RLZero not only recovers policies with zero gradient steps evaluation time, but these policies also achieve qualitative and quantitative improvement over prior baselines that leverage VLMs as source reward or perform test-time training.

---

Acknowledgments
We thank Matteo Pirotta, Ahmed Touati, Andrea Tirinzoni, Alessandro Lazaric and Yann Ollivier for enlightening discussion on unsupervised RL. This work has in part taken place in the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) at The University of Massachusetts Amherst and Machine Intelligence through Decision-making and Interaction (MIDI) Lab at The University of Texas at Austin. SCALAR research supported in part by the NSF (US-2333384), the Center for Al Safety (CAIS), and the Long-Term Term Future Fund. HS, SA, SP, MR, and AZ are supported by NSF This work has
2340651, NSF 2402650, DARPA HR001 12490431, and ARO W911NF-24-I-011 taken place in the Learning Agents Research Group (LARG) the Artificial Intelligence Laboratory, The University of Texas at Austin. LARG research is supported in part by the National Science Foundation (FAIN-201983)
NRT-2125858) the Office of Naval Research (N00014-24-1-25500 Army Research Office (W91 17--1-22-0181, W911NF-23-2-000 (Cooperative Agreement HR001 12520004 Ad Hoc Teamwork), Lockheed Martin, and Good Systems, a research grand challenge at the University of Texas at Austin. The views and conclusions contained in this document are those of the authors alone. Peter Stone serves as the Chief Scientist of Sony AI and receives financial compensation for that role. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.
191 DARPA
References
[1] S. Agarwal, H. Sikchi, P. Stone, and A. Zhang. Proto successor measure: behavior space of an agent. In Forty- secty- International Conference on Machine Learning.
Representing the
[2] S. Agarwal, I. Durugkar, P. Stone, and A. Zhang. f-policy gradients: A general framework for goal-conditioned RL using f-divergences. In Teirtt-seventh Conference on Neural Information Processing Systems, 2023. URL https::////// //tttt:ss//ppeeeeeee //ppr//viieeeeeeee/emmeee www./neefouunnoee httt//foummiddhhhPeee riddhhhhttssAss
[3] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. Do as can, not as i say: Grounding language in robotic affordances. arXiv preprint MrXiv:2204.01691, 2022.
[4] D. Amodei, C. Olah, J. Steinhardt, Christiano. J. Schulman, and D. Mané. Conerete problems in ai safety. arXiv preprint 1rXiv:160.006500 2016.
[5] A. Barreto, W. Dabney, R. Munos, J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
[6] K. Baumli, S. Baveja, F. Behbahani, H. Chan, G. Comanici, S. Flennerhag, M. Gazeau, K. Holsheimer, D. Horgan, M. Laskin, et al. Vision-language models source of rewards. arXiv preprint arXiv: 2023.
[7] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, V. Jampani, and R. Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. https:tts:///attt wttp://.rrr/aba///aeeaaae/ee///e 1511111777..1.....
[8] L. Blier, C. Tallec, and Y. Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. arXiv preprini arXiv:2101.07/233 2021.
[9] S. Booth, W. B. Knox, J. Shah, S. Niekum, P. Stone, and A. Allievi. The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications. In Proceedings of the -eventh AAAI Conference Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence,
AAAI Press, 2023. ISBN URL https://ddot org. 1609
978-1-57735-880-00 doi: 10.1609/aaai.v33755.225 aaai. 1..755.57773
[10] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint 11Xiv:1810.1284. 2018.
10

---

[11] Calisthenicmovement.. The most underrated exercise you're re not doing!. 2021. URL https: YouTube video.
[12] R. Caruana. Multitask learning. Machine Learning, 28:41-75, 1997. URL https://api.
semanticscholar. www.eeechh.aaa.orrrrrrrpueeeeeee :59998814 45998148.
[13] E. Chane-Sane. C. Schmid, and I. Laptev. Learning video-conditioned policies for unseen manipulation tasks, 2023. URL https: pettps//arxaae //aaxii.vv.//////////e////aaaala wwwpsii..or//bbbe. org/abs/2305 06289.
[14] D. Chen, R. Chen, S. Zhang, Y. Liu, Y. Wang, H. Zhou, Q. Zhang, Y. Wan, P. Zhou, and L. Sun. Milm-as-a-judge: Assessing multimodal Ilm-as-a-judge with vision-language benchmark. arXiv preprint 2442::22002047888, 2024.
[15] P. Dayan. Improving generalization for temporal difference learning: The successor representation. Neural computation,5(426113-622222 1993.
[16] Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation, 2023. URL https://arxia org/abs/2302 00111.
[17] Y. Du, M. Yang, P. Florence, F. Xia, A. Wahid, B. Ichter, P. Sermanet, Yu, P. Abbeel, J. Tenenbaum, et al. Video language planning. arXiv preprint 0rXiv22310.10625 2023.
[18] G. Dulac-Amold, N. Levine, D. J. Mankowitz, Li, C. Paduraru, Gowal, and Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 1009))2419-2468, 2021.
[19] I. Durugkar, M. Tec, S. Niekum, and P. Stone. Adversarial intrinsic motivation for reinforcement learning. Advances in Neural Information Processing Systems, 0188662286366 2021.
[20] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. CoRR, abs/1802.06077, 2018. URL http: /ttp:///axxii org/ agg/abs/
1802. 1802.06070.
[21] B. Eysenbach, R. Salakhutdinov, and S. Levine The information geometry of unsupervised reinforcement learning. In International Conference on Learning Representations, 2022, URL https://openreviee wwwnnee/foruue d.0UUmXXOEE.
[22] S. Fujimoto and S. Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-200 -211222 2021.
[23] Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[24] D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-leam: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028-4033 2021.
[25] K. S. Ghasemipour, R. Zemel, and S. Gu. divergence minimization perspective on imitation learning methods. In Conference Robot Learning, pages 1259-1277. PML. 2020.
(26) P. Goyal, R. J. Mooney, and S. Niekum. Zero-shot task adaptation using natural language. arXiv preprint arXiv:2 02972, 2021.
[27] P. Goyal, S. Niekum, and R. Mooney. PixI2r: Guiding reinforcement learning using natural language by mapping pixels to rewards. In Conference on Robot Learning, pages 485-497. PMLR, 2021.
[28] Ho and Ermon. Generative adversarial imitation learning. Advances neural information processing systems, 29:4565-4577 2016.
[29] W. Huang, Abbeel, Pathak, and Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07200 2022.

---

[30] G. Ilharco. M. Wortsman, R. Wightman. C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong. J. Miller, H. Hajishirzi, A. Farhadi, and Schmidt. Openclip, July 2021. URL petps://doo.ore org/10.....00 /t////11..221111 5001//2211//// 5143773. If you use this software, please cite it as below.
[31] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. (0055): siioo-language-action model with open-world generalization. arXiv preprint arXiv:2504.16055 2025.
[32] V. Jain, M. Attarian, N. Joshi, A. Wahid, D. Driess, Q. Vuong, P. Sanketi, Sermanet, S. Welker, C. Chan, I. Gilitschenski, Y. Bisk, and D. Dwibedi. Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers, 2024.
[33] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2022.
[34] L. P. Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence, 1993. URL https: //app.app..p// semantccscoolar org/ httpg/CCrpuSD:::::0 :553p68
[35] G. Kim, S. Seo, J. Lee, W. Jeon, H. Hwang. H. Yang. and K. Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In ICLR. OpenReview.net 2022.
[36] D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, G. Schindler, R. Hornung, V. Birodkar, J. Yan, M.-. Chiu, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312. 14125, 2023.
[37] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv: /0.06666,, 2021.
[38] V. Krakovna. Specification gaming examples in ai. Available at vkrakovna. wordpress. com, 2018.
[39] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In arXiv preprint arXiv:2209.07753, 2022.
[40] W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL
[41] LivestrongWoman. Lying leg raises, 2014. URL http:::////out be/NN44BXXTALT1TTT751 Jdmjjuu05kjmöKK1L YouTube video.
[42] M. Loper, N. Mahmood, J. Romero, G. Pols-Moll, and M. J. Black. Smpl: A skinned multi-persoo linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume pages 851-866 2023.
[43] C. Luo, M. He, Z. Zeng, and C. Sun. Text-aware diffusion for policy learning. Advances in Neural Information Processing Systems, 37:46226-46231 2024.
[44] C. Luo, Z. Zeng. Y. Du, and C. Sun. Solving new tasks by adapting internet video knowledge. In The Thirleenth International Conference on Learning Representations, 2025. URL https: //openreview. nww/teetuunaneeee ft///orrmmmi///
[45] H. Luo and Z. Lu. Learning video-conditioned policy on unlabelled data with joint embedding predictive transformer. In The Thirteenth International Conference on Learning Representations, 2025. URL httpp::://////evveeeeeeee net/ forum?id=TTM0hi
[46] Y. J. Ma, A. Shen, D. Jayaraman, and O. Bastani. Smodice: Versatile offline imitation learning via state occupancy matching, arXiv preprint arXiv: 2022.
12

---

[47] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030, 2022.
[48] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 12931, 2023.
[49] P. Mazzaglia, T. Verbelen, B. Dhoedt, A. Courville, and S. Rajeswar. Multimodal foundation world models for generalist embodied agents. arXiv preprint arXiv:2406. 18043, 2024.
[50] V. Mnih, K. Kavukcuoglu, Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint /3/..5622 2013.
[51] A. Moves. How to do downward dog I adho mukha svanasana tutorial with dylan werner, February 2019, video.
YouTube
[52] T. Ni, H. Sikchi, Y. Wang, T. Gupta, L. Lee, and B. Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning, pages 529-551. PMLR, 2021.
[53] M. W. Nicole. 35 min full body workout l| intermediate pilates flow, 2021. URL https: //youtu. 1A0
YouTube video.
[54] A. O'Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint 2rXiv:23/0008800 2023.
[55] S. Park, T. Kreiman, and S. Levine. Foundation policies with hilbert representations. arXiv preprint arXiv:2402.15560 15567, 2024.
[56] S. Park, O. Rybkin, and S. Levine. METRA: Scalable unsupervised RL with metric-aware abstraction. In The Twelfth International Conference on Learning Representations, 2024. URL
[57] M. Pirotta, A. Tirinzoni, A. Touati, A. Lazaric, and Y. Ollivier. Fast imitation via behavior foundation models. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
[58] M. L.. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[59] Z. W. Pylyshyn. Mental imagery: In search of theory. Behuvioral and brain sciences, 25(2): 157-182, 2002.
[60] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[61] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. PMLR, 2021.
In International conference on machine learning, pages
[62] Rocamonde, Montesinos, E. Nava, E. Perez, and Lindner. Vision-language models are zero-shot reward models for reinforcement learning. arXiv preprint 2xX.v:2310.12000 2023.
[63] T. R. Sarbin. The role of imagination. Narrative analysis: Studying the development of individuals in society, page 5, 2004.
[64] T. R. Sarbin and B. Juhasz. Toward theory of imagination. Journal of personality, 38(1): 52-76, 1970.
[65] H. Sikchi, R. Chitnis, A. Touati, A. Geramifard, A. Zhang, and S. Niekum. Score models for offline goal-conditioned reinforcement learning. In The Twelfth International Conference on Learning Representations, 2023.
13

---

[66] H. Sikchi, Q. Zheng, A. Zhang, and S. Nickum. Dual RL: Unification and new methods for reinforcement and imitation learning. In The Twelfth International Conference on Learning Representations, 2024. URI. https: httpps:////peeeeeee /ptp//eeeeeeeeeeeeeeeeeeeeeeeeee net/foormmmddxxttthBBgg 1d=xt9Bu66rqgg.
[67] H. Sikchi, A. Tirinzoni, A. Touati, Y. Xu, A. Kanervisto, S. Niekum, A. Zhang, A. Lazaric, and M. Pirotta. Fast adaptation with behavioral foundation models. arXiv preprint arXiv:2504.078%. 2025.
[68] Sontakke, J. Zhang, Arnold, K. Pertsch, E. Bıyık, D. Sadigh, C. Finn, and L. Itti. Roboclip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems, 36, 2024.
[69] S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor. Language-condition imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33:13139-1311 13150, 2020.
[70] D. Suarez. Get your splits hip flexibility 28 day splits challenge 17 min danicla suarez, YouTube video.
2022. URL https: //yuuuy.uuuu// be hatpaa/ppeaaaeeeis-oooEooooooooo
[71] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[72] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, Abdolmaleki,
J. Merel, A. Lefrancq. T. P. Lillicrap, and M. A. Riedmiller. Deepmind control suite. CoRR, abs/1800 180.00669, 2018.
[73] Thomason, Zhang, R. Mooney, and P. Stone. Learning to interpret natural language commands through human-robot dialog. In Twenty-Fourth International Joint Conference Artificial Intelligence, 2015.
[74] A. Tirinzoni, A. Touati, J. Farebrother, M. Guzek, A. Kanervisto, Y. Xu, A. Lazaric, and M. Pirotta. Zero-shot whole-bode humanoid control via behavioral foundation models. In NeurlPS 2024 Workshop on Open-World World Agents.
[75] Z. Tong, Y. Song, J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training, 2022. URL UtL. 12602.
[76] A. Touati and Y. Ollivier. Learning one representation to optimize all rewards. In NeurlPS, pages 13 13-23, 2021.
[77] A. Touati, Rapin, and Y. Ollivier. Does zero-shot reinforcement learning exist? In ICLR. OpenReview.net, 2023.
[78] Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, Xu, Y. Liu, Z. Wang, al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03399) 2022.
[79] Y. Wang, K. Li, X. Li, Yu, He, C. Wang, G. Chen, B. Pei, Yan, R. Zheng, Xu, Z. Wang, Y. Shi, T. Jiang, S. Li, H. Zhang, Y. Huang, Y. Qiao, Y. Wang, and L. Wang. Internvideo2: Scaling foundation models for multimodal video understanding, 2024. URL https://arxiv. www.oogg./........oo............ a/aabb///233 15377.
[80] Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma, and Y. Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents, 2024. URL https: //arxiv../i//////aa org/ abs/ / 2202 01560.
[81] Well+Good. How to do push-up the right way well+good, 2019. URL https::///you be//ttbbbbs.000///ai--GGEEEEEEEd 3VV6.v. YouTube video.
[82] Well+Good. How to do crunches the right way well+good, July 2019. URL https: Catixsa__tDDpppC...m YouTube video.
[83] wikiHow. How to do a cartwheel, September 2019. URL, htRlsttts::// httttps://youu..u be http://yttttu.beellgc//33/11 si=boT_th BaL_tttM_EEE_________ YouTube video.
14

---

[84] Y. Wu, G. Tucker, and O. Nachum. The laplacian in rl: Learning representations with efficient approximations. arxiv preprint arXiv:1810.045588 04086, 2018.
[85] Xie, S. Zhao, C. Wu, Y. Liu, Q. Luo, Zhong, Yang. and Yu. Tex(2reward: Reward shaping with language models for reinforcement learning, 2024. URL abs/2309. 11489.
[86] D. Yarats, D. Brandffonbrener, H. Liu, M. Laskin, P. Abbeel, A. Lazaric, and L. Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprini arXiv:2201.13425 2022.
[87] J. Yoga. Full body standing yoga improve flexibility and balance, 2022. URL https: //youtu.be/vAlle
YouTube video.
[88] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. Chiang, Erez,
I.. Hasenclever, J. Humplik, et al. Language rewards for robotic skill synthesis. arXiv preprint arXiiv:2306. 08647, 2023.
[89] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 11975-11986, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in abstract and introduction are limited to the papers contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Section 5.3.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
The proofs and assumptions are available in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-refernced..
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question:
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, give extensive experimental details in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: We make the code available as supplementary material and will release publically upon acceptance of the manuscript.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/guides/ httpe//oooSSuuuiiiooooooroooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CoodeSummas
//n //iip..cccc.. https/ccpppbbc///cceeee.cc / ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We give all training and test details in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We perform appropriate statistical tests to demonstrate the improvements obtained from our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Our experiments were performed NVIDIA-A40 and AMD EPYC 7763 64-Core Processor machine.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: stttps:/neuurppe wwtpp//cc//tcciiccoooooooocccccc www.ieeeeeeeeeeeeeeeeeeeeeeeeeee ines?
Answer: [Yes]
Justification: Yes this work complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due to laws regulations in their jurisdictot)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work involves small models and simulated environments. It does no impact to the society at large, beyond improving our understanding of certain aspects of deep reinforcement learning.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We use open source models in our work and the RL models trained in the work are in simulated environments that do not pose a risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our work uses open source simulators and models with proper license attribution.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwoodeethcode..... com conddataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We do not release any assets other than our code for training simulated agents.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/modell as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not use human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use video generation models as component of our method. Further, we use multimodal LLM for evaluation using the L.LM as judge framework and describe it in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy pectps:/nnuurgge http:///ecc.coooeeeeee////////// for what should or should not be described.
22

---

Appendix
A Proof For Theorem 1
Theorem 1.
Define J(T,T) to be the expected return of a policy π under reward r. For an offline
dataset do with density p. a learned log distribution ratio: v(s) log( e' ), DKi(p", , ,F) S
pls)
-J( + (x (s, a.,, p(s, a.) where (s) v(s) vss. The corresponding Zimit minimizing the upper bound is given by Zemsst = =sprrim(ss)p((ss) = Ep ((s) ( where ip denotes state features learned by the BFM.
Proof. Let p be the density of the offline dataset, p" be the visitation distribution w.r.t. policy π and p be the expert density. The distribution matching objective mentioned Equation using KI. divergence is given as:
min D D DxL(p"Ile")
(5)
With simple algebraic manipulation, the divergence can be simplified to,
D 11==) =Ep- [log R
+
Ep
[log _]]
(6)
((s)
=Ep- [log
+ Dxu(p" (s))l((s)
(7)
p
J(T, log
+ DKI(Ø" (s)lpp(s)
(8)
p
S JT, log
+ DxL(Ø (s.a)l(sss))
(9)
The last line follows from the fact that D (p" (o))(((s)l)(()) -
p² p=(s.a)
DKL (s) a)) = E (s.))
(10)
als,a)
Ep (s 1" (=)s)(aass)
(11)
P (als)
p- (s)
(als)
Eprceayloog
E Ep log
(12)
(())
x (a(s)
(o) (0(
<alss)
+E (epe flog
(13)
p(s)
π (a]s)
DKL(p" (s)(())) + Egpp' (Kz(p"(s)]lp(())
(14)
(15)
Rewriting the minimization of the upper bound of KL as a maximization problem by reversing signs, we get:
max J(T log - -x,
(16)
The first term an RL objective with reward function given by log( log({"), and the second term an offline regularization to constrain the behaviors of offline datasets. Following prior works [35, 46]. since our BFM is trained on an offline dataset and limited to output skills in support of dataset actions, and we can ignore the regularization to infer the latent z parameterizing the skill. A heuristic yet performant alternative is to use a shaped reward function of - discriminator completely and was shown to lead to performant imitation in [57].
- which allows us to avoid training the
23

---

B Experimental Details
B.1 Environments
B.1.1 DM-control environments
We continuous control environments from the DeepMind Control Suite [72].
Walker:
The agent has 24 dimensional state space consisting of joint positions and velocities and 6 dimensional action space where each dimension of action lies in [-1, The system represents planar walker.
The agent has a 17 dimensional state space consisting of joint positions and velocities and 6 dimensional action space where each dimension of action lies in [-1, 1]. The system represents planar biped "cheetah".
Cheetah:
The agent has 78 dimensional state space consisting of joint positions and velocities and 12 dimensional action space where each dimension action lies in [-1, 1|. The system represents 3-dimensional ant with 4 legs.
Quadruped:
Stickman: Stickman was recently introduced as a task that bears resemblance to humanoid in [49]. It has a 44 dimensional observation space and a 10 dimensional action space where each dimension of action lies in [-1,
SMPL 3D Humanoid: The agent has a 358 dimensional state space consisting of joint positions and velocities and 69 dimensional action space where each dimension of action lies [-1, 1]. The system represents a 3-dimensional humanoid.
For all the environments consider image observations of size 64 64. All DM Control tasks have an episode length of 1000.
B.2 Evaluation Protocol
To evaluate models for behavior generation through language prompts. we considered set of prompts environment. One key consideration in designing these prompts was the generative video model's capability of generating reasonable imagined trajectories. Due to computing limitations, we were restricted to using fairly small video embedding (33 million parameters). The interpretability of our framework allows us declare failures before they happen by looking at the generations for imagined trajectories.
billion parameters) and generation model
For the of task prompts specified by language, there is ground truth reward function and there does not exist a reliable quantitative metric to verify which of the methods perform better. Instead, since humans communicate their intents via language, humans agent has demonstrated the behavior they intended to convey. In this work we use Multimodal LLM as a judge, following studies by prior works demonstrating the correlation of LLMs judgment to humans [14]. We use GPT-4o model as the judge, where the GPT-4o model is provided with two videos, one generated by a base method, and another generated by one of the methods we consider,
the best judge of whether the
and asked for preference between which video is better explained by the text prompt for the task. When inputting the videos the judge, we randomize the order of the baseline and proposed methods to reduce the effect of anchoring bias. The prompt we use to compare the two methods is given here:
For prompt to policies:
response = client. chat completion create(
model-MODEL,
messages-[
("role": "system", "content": "For the given summarization:
"[task prompt)', which video is more aligned with the summarization?"), ("role": "user", "content"; "Video A",
mmambaa {"type": "image_uri" url", 'image_url", ("url": : (x)>> }b, video1),
10
I1
"Video B" *map (lambda ("type": "image_url" url",

---

"image_ur"" _ ("url": f' i'data:image/jp: base64, ( (x) }}. video2),
15
"FIRST provide one-sentence comparison the two videos) and explain which you feel the given summarization explains better. SECOND, on a new line, state only 'A' or 'B' to indicatel
16
17
which video is better explained by the given summarization. Your reeponse should use the format:\
20
21
22
Comparison: Cone-sentence comparison and explanation>) Better explained by summarization: <'A' "B>
24
25
26
},
For cross-embodiment video policies:
crose_ embodied_ video, deeerpppeciiiiion [+map (lambda ("type": "image_url", "image_ur"" url": {"url": d'data: :datta:maage/ jpp base64, (x)
reaponse client. chat comppeetion.crratte
model=MODEL,
messages-
("role": "system", "content"; f"For the original video: umbodddd_iid___dcccpiipool which of the following given videos describe behavior similar to the original video?"},
11
12
{"role": "user", "content":
13
"Video A", (1amdaa ("type": "image_ure url", "image_url" ("URL"; d'data: imaes//mggep::: base64, (x}', }), video1),
15
16
17
"Video B"
*map (lambda {"type": "image_um url",
20
"image_url" {"URL": f iddata:iaage/ppe base64, {x}
21
22
}), video2),
"FIRST provide one-sentence comparison of the two videos and explain which you feel matches the behavior shown in original video better SECOND,
24
25
26
new line, state only 'A' or \ "B' indicate which video is better aligned the task demonstrated the original video. Your response should use the format:
29
30
Comparison: <one-sentence comparison and explanation>
33
Better matches the original video: <'A' or "B'>"
34
35
],
38
B.3 Dataset Collection for Zero-Shot RI.
For Cheetah, Walker, Quadruped, and Stickman environments, our data collected following pure exploration algorithm with no extrinsic rewards. In this work, we use intrinsic rewards obtained from Random Network Distillation [10] to collect our dataset based on the protocol by ExoRL [86] and using the implementation from repository ExoRL repository. For Cheetah, Walker,
25

---

and Quadruped, our dataset comprises 5000 episodes and equivalently 5 million transitions, and for Stickman, our dataset comprises 10000 episodes or equivalently 10 million transitions. Due to the high dimensionality of action space in Stickman, RND does not discover a lot of meaningful behaviors; hence we additionally augment the dataset with 1000 episodes from the replay buffer of training for a 'running' reward function and 1000 episodes of replay buffer trained on a 'standing' reward function.
B.4 Baselines
Zero-shot text to policy behavior has not been widely explored in RL literature. However, Offline RL using language-based rewards utilizes an offline dataset to learn policies and is thus zero-shot in terms of rolling out the learned policy. This makes it a meaningful baseline to compare against. Offline RL uses the same MDP formulation as described in Section to learn a policy
S - (AA), given a reward function : S + R and offline dataset D. The offline dataset consists of state, action, next-state, reward transitions (s, a, s', r())). One of the core challenges of Offline RL is to learn a Q-function that does not overestimate the reward of unseen actions, which then at evaluation causes the agent to drift from the support of the offline dataset D.
We implement two offline RL baselines to compare with RLZero- Implicit Q-learning (IQL. [37]) and Offline TD3 (TD3, [22]) Both of these methods share the same offline dataset as used to learn the successor measure in RLZero, which is described in Section 5, and gathered using RND. Since these datasets are reward-free, we must still construct reward function that provides meaningful rewards for an agent achieving the behavior that aligns with the text prompt. Formally, given language + 2,
instruction et E frame stack (ot-k, 0 0z) E I, and embedding VLM é which can also embed frame stacks é I + Z (and where observations o, E I, and we use for this section), the reward for a corresponding language instruction and frame stack k is the cosine similarity between the stacked language embedding and the frame embedding:
r(or- ket, el) H(e') ((or-ke) (ll())=))(((o-==))))
(17)
For any individual task, et is fixed and this is a reward function dependent on observations (as represented by a frame stack oz-k:)). Notice that this representation closely matches that in Equation 3, but instead of finding the optimal sequence of observations, we simply compute reward as the cosine similarity between language and frames. Since the strength of the embedding space is vital to the quality of the reward function for offline RL, we evaluate two different vision-lange laiiunnl models:
Image-language reward (SigLIP [89]): take stack of 3 frames encode them using SigLIP, then the reward is computed as the cosine distance of the embeddings and the SigLIP embedding of language.
Video-language reward (Intern Video2 [79]): this method takes in previous frames 00a- as context and uses generate an embedding of the current frame observation o,. The video encoder then takes the cosine similarity of ((Oua) ) and (ee'). This allows the reward function to provide rewards based not only on reaching certain states, but the agent exhibiting temporally extended behaviors that match the behavior. In practice, providing rewards using an image- based encoder for frame stacks can be challenging for tasks such as walking because they require context, and video-based rewards offer a way to better encode the temporal context.
B.4.1 Offline RL
Implicit Q-learning [37] Implicit Q-learning builds on the classic TD error (revised in our context of sunguage-instruction rewards):
L(0) = -Essas + Qo(,,)))
to learn Q function Qo. IQL builds on this loss to handle the challenge of ensuring that the Q-values do not speculate on out-of-distribution actions while also ensuring that the policy is able to exceed the performance of the behavior policy. Exceeding the behavior policy is important because the dataset is collected using RND, meaning that any particular trajectory from the dataset unlikely to perform well on language reward. The balance of performance is achieved by optimizing the objective with expectile regression:
(²) = |T 1(u < 0)/u²
26

---

Where 5 > 0.5 is the selected expectile. Expectile regression gives greater weight to the upper expectiles of distribution, which means that the function will focus more on the upper values of the Q function.
Rather than optimize the objective with , directly, IQL uses value function to reduce variance to give the following objectives:
Lv(
(0) = Eis,,,'', (r(s, et) + Vo(5') - Qo(s,a))) Using the Q-function, policy be extracted using advantage weighted regression:
L( (
a) (=(())) log (a]ss]]
Where 8 is the inverse temperature for the advantage term.
TD3 [23]:
TD3 was demonstrated to be the best performing algorithm when learning from exploratory RND datasets in [86]. While TD3 does not explicitly address the challenges discussed in implicit Q-lernning and learns using Bellman Optimality backups, the approach is simple and works well in practice. The algorithm uses a deterministic policy extraction πT S + to give the following objective:
arg max E B(s.))=(((ss.T( (()))((ss))
B.5 RLZero
Algorithm 1 RLZero
1: Init: Pretrained Video Generation Model VM, Pretrained BFM TT, Offine Exploration Datasct d
2: Given: text prompt L
Generate imagination video given the text prompt: (in, i2) VM
4: Project the imagined frames to real observations using embedding similarity as in Eq 3.
5: Use Theorem for zero-shot inference to obtain BFM $1, 82, 82, s1}) Zontt and return Wzmmit
B.5.1 Text to Imagined Behavior with Video models
To generate a proposed video frame sequence, we utilize the GenRI. architecture and provide the workflow using equations from the original paper [49]. First, the desired text prompt is embedded
with the underlying video foundation model internVideo'' [79] e = f()) fp)()) These embeddings are then repeated n. fromes times (we 12 frames = 32) to match the temporal structure expected by the world model. The repeated text embeddings The aligner implemented as a UNet and is used to address the multimodality gap [40] when embeddings from different modalities occupy distinct regions in the latent space. Next, the aligned video embeddings are concatenated with temporal embeddings. The temporal embeddings are one-hot encodings of the time step modulo n frames providing frame-level positional information. The first embedding is passed to the world model connector (syle) to initialize the latent state. For each subsequent time step. the sequence model he = fo 1 h4-1) (implemented updates the deterministic state h4s The deterministic state h; is mapped to a stochastic latent state (st) using the dynamics predictor Po (s. The dynamics predictor, implemented as an ensemble of MLPs, predicts the sufficient statistics (mean and standard deviation) for a Normal distribution over 8t During inference, the mean of this distribution used as the latent state. Finally, the latent state is passed to convolutional decoder Pr(2x) s,) to reconstruct the video frame Z,. This process is repeated for all time steps (t = 1,.; frames)
passed through an aligner module e(v) f (e()).
a GRU)
B.5.2 Grounding imagined observations to observations in offline dataset
As described in Section 4, we ground imagined sequences by retrieving real offline states based on similarity in an embedding space. This enables us to create a suitable z-vector for distribution matching which is the expected value of the state features under the distribution of imagined states (Pimeginez). During our dataset collection phase, we save both the agent's proprioceptive state as well as the corresponding rendered images and search over the images then find the corresponding
27

---

state. Our code supports both fracked-frame embeddings and single- frame embeddings. We find that stacked-fraae embeddings were helpful in modeling temporal dependencies through velocity and acceleration, which are crucial for recreating the intended behavior. SigI.P [89], which replaces CLIP's [60] softmax-base contrastive loss with pairwise sigmoid loss, resulted in qualitatively better matches to exact positions within sequences, imitating behavior more accurately than CLIP. For both models, we use the OpenCLIP [30] framework. Our matching process first involves precomputing embeddings offline, which are stored in chunks of up 100,000 frames to optimize memory usage and retrieval speed. During inference, we load this file and embed the query frame sequence from GenRL [49] into the same latent space. We process these query embeddings by dividing them into chunks k-frame sequences (A is generally 3 or 5), where each sequence consists of the current frame and the k 1 preceding frames. If there
not enough preceding frames, repeat the first frame to fill the gap. For each chunk of saved embeddings, we compute dot products between the query chunk and all subsequences of size in the saved embeddings. We track the highest similarity score for each query chunk and return the frames corresponding to the closest embedding sequences.
B.5.3 Training a zero-shot RL agent
In this work, we chose -ackwwdd (FB) [76] our zero-shot RL algorithm and trained it on proprioceptive inputs. Our implementation follows closely from the author's 's codebase. Specifically, used to map
FB trains Forward, Backward, and Actor networks. The backward networks demonstration or a reward function to a skill, which is then used to learn a latent-condittonal Actor. Our experiments were performed on NVIDIA-A40 and AMD EPYC 7763 64-Core Processor machine. The hyperparameters for our FB implementation are listed below:
Implementation: We build upon the codebase for FB https://githe ggubttttuuuu.. com/ wtt.krressarrccccooooalabaaa agent and implement all the algorithms under a uniform setup for network architectures and the same hyperparameters for shared modules across the algorithms. We keep the same -ngosid-ggoostcc hyperparameters and use the author-suggested -ethod-specific hyperparameters. The hyperparameters for all methods can be found in Table 3:
Table 3: Hyperparameters for zero-shot RL with FB.
Hyperparameter
Value
Replay buffer size Representation dimension
10 10 x 10 (for stickman)
128
Batch size
1024
Discount factor 7
0.98
Optimizer
Adam
Learning rate
3 10-4 4
Momentum coefficient for target networks
0.99
Stddey a for policy smoothing
0.2
Truncation level for policy smoothing Number of gradient steps Regularization weight for orthonormality loss (ensures diversity)
0.3
2 10
1
FB specific hyperparameters
Hidden units (F))
1024
Number of layers (F)
3
Hidden units (b)
256
Number of layers (b)
2
B.6 Imagination-Free RLZero
In this section, we propose an alternate method (Figure 5) for mapping a task description into usable policy. Instead of first embedding a text prompt e², generating a video, then mapping the video to policy parametrization, we propose to map the text prompt directly to policy parametrization. To this, we learn latent mapper m e + Zimitation that relates the latent space of ViLM to the latent space of our policy parametrization. The mapper is a layer ML. with hidden size of 512.
Pretraining: We first generate dataset of episodes containing diverse behaviors by rolling out the
28

---

MLM Embedding Space
mmitation Embedding Space
arals, z)
"Do a lunge"
Learned Latent Mapper
Figure 5: Illustrative diagram of imagination- free RLZero inference
behavior foundation model conditioned on a uniformly random sampled z. The resulting image observation sequences then down-sample (by 8) and sliced to break up each episode into smaller chunks of length 8; this preprocessing step helps increase the behavioral diversity and improves the ability of the ViFM to capture semantic meaning. The resulting clips are then embedded using ViFM (InternVide22 [79]) where each embedding is denoted by e (as in Section 5.3). Now, we have of sequences of length consisting of image observation along with their proprioceptive states, and the embedding for the image sequence.
Now, an obvious option is to map the embedding of image sequence to the that generated the trajectory. Unfortunately, the way BFMs are trained, they not account for optimal policy invariance to reward functions. That is multiple reward functions that induce the same optimal policy are mapped to different encodings in the Z-space. This presents a problem for the latent mapper, as it becomes a one-to-many mapping for any language encoding. We present alternative solution which ensures that only one target z is used for a given distribution of states induced by a language encoding. To achieve this turn back to the imitation learning objective where the sequence of proprioceptive states is used to obtain a policy representation using Lemma which gives the latent & corresponding to the policy that minimizes the distribution divergence to the sequence of given states. We refer the policy representation embedding space from the Forward- Barkadd---a representation as Zimitation Space.
When optimizing the latent mapper m, we minimize the following loss:
m(e) Zimitation
L(D,m) E (ammater e)
[m)]] Pimitation
The latent space of the Backward representation is aligned with the latent space of the policy parametrization, learning a mapping from the ViFM space the Backward space is equivalent to learning mapping from the ViFM space to the policy parametrization space.
Inference:
During inference, the language prompt embedded with multimodal embedding models is the embedding gap [40], which makes the video embeddings unaligned with text embeddings. To account for this gap, we use an aligner trained in an unsupervised fashion from previous work [49] to align the language embedding (catigned). Then the aligned embedding is passed through the latent mapper to get the policy conditioning Zimitaation which gives us the policy that achieves the desired behavior specified through language.
latent vector e A known issue
C Additional Results
C.1 Visualizations
A
Figure Example Imagined Trajectories: The video model imagines frames conditioned task specified as a text prompt 'do lunges'.
29

---

'Bunny Hop
Crawl like worm'
"Cartwneel"
Figure 7: RLZero in action:
Qualitative examples of RL converting the given language prompis into behaviors across different domains. Top to boltom: Cheetah, Walker, Quadruped, Stickman.
C.2 Zero-shot imitation: Discriminator vs Discriminator-free
We experiment whether optimizing a tighter bound to KL divergence the expense of training an additional discriminator in Lemma leads to performance improvements. Table shows that using discriminator does not lead performance improvement and training-free inference time solution achieves slightly higher win rate.
C.3 Cross Embodiment experiments
Tables 5 and describe the videos used for embodiment along with the win rate of the behaviors generated by RLZero when compared to base model which trains SMODICE [46] on the nearest states found with the same grounding methods as RLZero.
C.4 Imagination-Free RLZero Complete Results
We consider an ablation of our method by understanding the need for imagination by replacing the step with an end-to-end learning alternative. This is a novel baseline described in Appendix B.6. Table 7 shows the results of this end-to-end alternative which maps the shared latent space of video language models behavior policy.
C.5 Ablation on BFM
We perform an ablation on the choice of BFM. For RLZero, we used Forward-Backward Representations [76] to train the BFM but we experiment with alternate method: Proto Successor Measures [1] for training BFMs. Table 8 shows that Forward Backward Representations performed better than Proto Successor Measures in our experiments.
C.6 More Failure Cases
We include more failure cases in Figure 8 and Figure as they can help in understanding the limitations of RLZero better and may inform future work.
C.7 RLZero evaluation with Video-Embedding similarity
In this section, we experiment with another metric for comparison - embedding similarity between a video of the generated behavior and the text. We use InternVideo2 embed the videos and take the cosine similarity with the prompt used to generate the behavior. Table 9 shows the results for this metric of comparison. Unfortunately, for behaviors that differ significantly from the prompt. This points to a limitation of using this metric for evaluation. Some reasons for this failure could be the limited context length of 8 for the video embedding model or a misalignment between video and text embedding vectors [40].
observed that the similarity score is frequently higher
30

---

RI.Zero with discriminator
RLZero
Walker
Lying Down
5/5
5/5
Walk like human
5/5
5/5
Run like human
5/5
5/5
Do lunges
5/5
5/5
Cartwheel
5/5
4/5
Strut like horse
5/5
5/5
Crawl like a worm
3/5
Quadruped
Cartwheel
4/5
4/5
Dance
5/5
5/5
Walk using three legs
4/5
5/5
Balancing on two legs
4/5
5/5
Lie still
2/5
2/5
Handstand
4/5
3/5
Cheetah
Lie down
2/5
Bunny hop
5/5
5/5
Jump high
5/5
5/5
Jump back legs and backflip
5/5
5/5
Quadruped walk
2/5
4/5
Stand in place like a dog
4/5
3/5
Stickman
Lie down stable
5/5
4/5
Lunges
5/5
5/5
Praying
4/5
4/5
Headstand
5/5
4/5
Punch
4/5
4/5
Plank
4/5
3/5
Average
82.4%
83.2%
Table 4: Win rates computed by GPT-4o of policies trained by different methods when compared to base policies trained by TD3+Image-lange reward.
Prompt Descriptions
Video Link/Meta Al Prompt
Win rate SMODICE
Stickman (2D Humarrid
Human in backflip position
animated human uying backtlip
275
Downward facing dog yoga pose
right profile yoga pose downward facing dog
Cow yoga pose
5/5
Downward dog with one leg raised the air in the
[1]
5/5
Ling on back
one leg
Lying back with both legs raised in the air
[41]
5/5
High plank yogu pose
[8]]
515
Sitting down with logs laid the front
S/S
Warrior III pose
[7]
3/5
Front splits
[70]
4/5
Table Comparison Win rates vs SMODICE for Stickman
Prompt Deseriptions
Video Link/Meta Al Prompt
Win SMODICE
SMPL (3D Humanokd)
A karate kick position A cat doing a handstand arabesque ballet position
karate kick
side profile of cat doing headstand
ballet movement
5/5
Animated wikiHow demo of cartwheel Running
[83]
running [82]
Lying crunches
Plank position
[81]
5/5
Table Comparison of Win rates vs SMODICE for SMPL Humanoid
31

---

Environment/Task
RLZero
RLZero (Imagination- Free)
Walker
Lying Down
5/5
5/5
Walk like a human
5/5
4/5
Run like a human
5/5
1/5
Do lunges
5/5
5/5
Cartwheel
4/5
5/5
Strut like horse
5/5
3/5
Crawl like a worm
3/5
0/5
Table 7: Win rates computed by GPT-4o of policies trained different methods when compared to base policies
reward. RLZero shows marked improvement using embedding cosine similarity as reward functions.
trained by TD3+Image-langue
RLZero (PSM)
RLZero (FB)
Walker
Lying Down
5/5
5/5
Walk like a human
5/5
5/5
Run like human
2/5
5/5
Do lunges
4/5
5/5
Cartwheel
4/5
4/5
Strut like a horse
4/5
5/5
Crawl like worm
1/5
3/5
Quadruped
Cartwheel
3/5
4/5
Dance
5/5
5/5
Walk using three legs
5/5
5/5
Balancing on two legs
4/5
5/5
Lie still
1/5
2/5
Handstand
4/5
3/5
Cheetah
Lie down
0/5
2/5
Bunny hop
5/5
5/5
Jump high
5/5
5/5
Jump on back legs and backflip
5/5
5/5
Quadruped walk
3/5
4/5
Stand in place like dog
3/5
3/5
Average
71.6%
84.2%
Table 8: Win rates computed by GPT-4o of policies trained by different methods when compared to hase policies trained by TD3+Image languugg reward.
32

---

Figure 8: More examples of failed imagination the video generation model used in RLZero. From top bottom: Walker 'kick', Quadruped 'bunny hop", Cheetah 'frontroll', Stickman 'raise hands while standing place'
Innge-knguuge reward
Video-lanpugge rewcrd
RL.2m
101.
TD3 (Rasc Madel
TD1
101.
Walker
Laing Down
205 (0.05.0.000
-0.89200000)
215 (0.35000)))
SIS (0.04:0.001
5.5 (..33:n000)
Walkk hide human
00.91.0.000 (0.00)
-8081100003
315 (0925000))
48 (0:90:10000)
5/5 (000
Run like humnn
58 00:900000000 02)
-8085-00013
1/5 (0111000)
2/ 00:44:0000)
3/5 (0:9500000 0.00
Do langes
45.00-4-00.01
-0.0--0.00
2/5 (0:92-00000
US (0:01-0.000
305 (0:94-0001)
Canoheel
10.901110.001
(0.01:0.01)
2/5 (0.51:0000)
4/5 (0.90:10.01)
(0:95-00001
Stree like borse
(0.99:10..00)
(09110000)
1/5 (0.11:0000)
35 (0:90110003)
(0:00-1000000
(0.03::0.000)
-0.92:60.011
(0.92::0.00)
(0.05:0.01)
Quadruped
Canwheel
(0.051:000))
(0.05:10.01)
00.92:000033
Dance
(0.94:00000
(0.941:000)
(0:94110.0)
logs
(0.93:0000)
2/
Lic
15.00.7:40.000
-8.0020.00)
(0.94100.00)
2/5 (0.05:10.20)
Handanad
300,01
(0922000))
25 (0:00100000
00:91-0000
Chestahh
Lin down
385 (0.90:00.02)
-0887-000000
2/5 (0945000))
38 0004:000)
2:5 (0-90-000))
Ranny
10.0000-0..000
-0.05--0.000
(0.950000))
US 00:97-00222
5:5 (0 (0:56-0000 (.00)
Pumg high
325 (0.94110.01)
(0.4100.01)
UU5 (0.11200))
Si (0:9210001)
5/5 (0.93:0001)
Jump on back legs and bookftip
325 (0.99140..01
(0 (0.22-0000)
US5 (0911000)
25 (0:22:0.011
5/5 0..11
Quadraped walk
1/5 (0.96-10.000
10.00000001
3 (1.961000))
58 (0:9930011)
4/S (0:77-0001
Stand place like dag
45 (0.93:1..01)
10 (0.80:00.00)
3/5 (0.98:0000)
05 (0.98:30.00)
3/5 (0.97:00000)
Stickman
Lie down stable
25.00.220..000
(0.1110001)
(0.931:0000)
15 (0:93:10000)
4/5 (0.911100000
Lunges
05 05:0092:00000
(UM11:0000)
2500922000
05 (0:92-10.00)
$/50.66::0000
Prasinn
-0089:500003
(0.375:000)
(0.67:0.01)
Heashtinnd
2/500.00:0.01)
-8.90060.00)
(0.90::0.0))
(0.57:0..01
Punch
(0.0099::0002)
(0.88::0002)
Averae
(1M27)
(0996)
Table 9: Win rates computed by GPT-4o of policies trained by different methods when compared to base policies trained ID3+Image-langus reward. RLZero shows marked improvement over using embedding cosine similarity reward functions.
33

---

7
Figure 9: More examples of failed grounding by the image retrieval model used in RLZero. The top image shows the imagined frame frame from the embodied video, and the bottom the nearest frame obtained from the agent's prior interaction dataset.
34


Reviews:
[{"review_id": "TDNH8DkI6D", "reviewer": "Reviewer_uCNH", "summary": "This paper proposes to leverage foundation models for policy learning by first imagining a trajectory conditioned on language instructions, then projecting this imagined trajectory to real observations from the dataset, and finally imitating those observations using distribution matching. They show results on language-conditioned continuous control tasks from DM Control Suite and on video-conditioned continuous control tasks on Humanoid environments (for cross-embodiment transfer).", "questions": "See Strengths and Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Leveraging off-the-shelf foundation models for continuous control is a promising approach to address the scarcity of data for control, and it is especially interesting that compared to other methods this does not require in-domain trajectories for learning inverse dynamics models\n- The paper is clearly written\n- The results on the humanoid tasks, which are more realistic than DM Control, show that the method can be quite general\n\nWeaknesses\n- The approach relies heavily on the “project” step which can be extremely brittle. The authors themselves acknowledge this: “We observe the image-similarity to latch on to features from the background and produce incorrect retrieval.” It would be informative to see results on more “in-the-wild” videos with noisier backgrounds. \n- A strong alternative to semantic similarity would be using off-the-shelf pose detection models and matching the joints. Could the authors compare to those methods?", "was_revised": true, "final_justification": "The authors addressed my concerns and therefore I increased my score.", "rebuttal": "We thank the reviewer for their detailed feedback. We appreciate the reviewer for acknowledging that our method is promising in sparse data regimes without requiring in-domain demonstrations, and finding our paper to be clearly written and our method to be quite general. We discuss some of the highlighted weaknesses below:\n\n> Project step can be brittle\n\nAs the reviewer notes correctly, the projection step can be brittle as we are aiming to match semantic information about poses that are relevant for the embodiment. While we explicitly highlight this as a limitation in Section 5.3, our work opens up new avenues for better semantic matching relevant to robot control. An example improvement would be to use a multimodal language model to obtain conditional embeddings that can be prompted to match the relevant parts. Our use of SigLIP/CLIP style approach goes to show that even unconditional embeddings can lead to performant imitation when learning from relatively clean third-person videos. \n\n\n\n> A strong alternative to semantic similarity would be using off-the-shelf pose detection models and matching the joints\n\nWhile we agree that this could be an interesting experiment, our work and baselines share the common foundation of not assuming access to embodiment-specific models. Our unsupervised learning approach only assumes access to reward-free environment interactions of the specific agent and proposes an approach to enable zero-shot imitation with general foundation models. A number of hurdles make use of the proposed pose detection approach limiting: 1. Video and robot may have similar yet differing embodiments/skeletons, like different numbers of joints, 2. Pose detection models require domain-specific models and a domain-specific projection step involving complex retargeting [1,2] 3. Matching 3D skeleton in 2D images can create false positives exploiting symmetries.\n\n[1]: Liu, Junjia, et al. \"Human-Humanoid Robots Cross-Embodiment Behavior-Skill Transfer Using Decomposed Adversarial Learning from Demonstration.\" arXiv preprint arXiv:2412.15166 (2024).\n\n[2]: Cao, Zhe, et al. \"Openpose: Realtime multi-person 2d pose estimation using part affinity fields.\" IEEE transactions on pattern analysis and machine intelligence 43.1 (2019): 172-186."}, {"review_id": "MmRjvgH1cL", "reviewer": "Reviewer_BfVk", "summary": "The paper presents RLZero, which trains a video foundation model to generate videos given a reward-unlabeled offline dataset. This video model is used at test time to generate videos of an agent performing a task, given a language description, and then project the videos to the dataset to find nearest matching states. A state-distribution matching objective is applied to learn a policy to imitate these behaviors.", "questions": "When projecting generated observaiton sequences to closest matches in the offline dataset, won’t some of the closest-match observation sequences come from different trajectories? If so, there could be some discontinuities in the states retrieved, making the state distribution matching objective valid but potentially suboptimal?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n**Analysis:** THere’s some nice analysis and discussion on inference time of the pipeline. \n\n**Results:** Results across many tasks in a few embodiments demonstarte RLZero works pretty well. Analysis is comprehensive too, and the reported metrics make sense given the open-ended evaluation. \n\n**Method:** I like that the method intelligently combines the best aspects of large foundation models (to use for training the video gen model and perform projection) while also combining with more traditional RL techniques such as successor feature representations and state distribution matching. \n\n## Weaknesses\n\n**Writing Clarity:** \n\n- The intro doesn’t really flow well from one point to the next. It’s fine, but I think the logic takes a couple of turns by presenting a multi-step solution that spans paragraphs 2 and 3. Perhaps hinting at (1) imagining and (2) projecting far earlier; e.g., paragraph 1, would help with understanding the intro.\n- Similarly the methods section is written at an extremely low level without having a high-level to refer to. It feels that each part of the method is written after the previous, but there’s no high-level intuition about how the method works at the top of each subsection (4.2 and 4.3). This could just be me, but the intro and methods took me multiple times to read to fully understand the contributions of the paper and how exactly it works.\n\n**Lots of prior data:**\n\n- Experiments are performed with 5-12million initial transitions. This is a LOT of data. Some experiments with less data would be beneficial to demosntrating the scalability o the method.\n\n## Minor Things\n\nSome related work in learning generalist agents that can immediately produce actions for a given language description:\n\n- BOSS: https://clvrai.github.io/boss/\n- https://arxiv.org/abs/2306.01711\n- DECKARD: https://deckardagent.github.io/", "was_revised": true, "final_justification": "The authors have addressed my concerns during the rebuttal. Results are good and motivation makes sense. I am recommending an accept.", "rebuttal": "We thank the reviewer for their detailed feedback on our work. We address the reviewers' questions and concerns below:\n\n> The intro doesn’t really flow well from one point to the next. It’s fine, but I think the logic takes a couple of turns by presenting a multi-step solution that spans paragraphs 2 and 3. Perhaps hinting at (1) imagining and (2) projecting far earlier; e.g., paragraph 1, would help with understanding the intro.\n\nWe will reword the introduction to give an overview of the imagine and project steps before diving into some of the details. Note that NeurIPS does not allow updating the paper during the rebuttal period, but we will ensure to make these changes in the final version of the paper.\n\n\n> Similarly the methods section is written at an extremely low level without having a high-level to refer to. It feels that each part of the method is written after the previous, but there’s no high-level intuition about how the method works at the top of each subsection \n\nWe will also provide a high level overview of our method before diving into details in the method sections. We thank the reviewer for pointing this out.\n\n\n> Experiments are performed with 5-12million initial transitions. This is a LOT of data\n\n\nSome of the most sample-efficient model-based learning algorithms require 500k learning steps to learn a *single* task (Figure 3 in [1]). On the other hand, we are leveraging a model-free algorithm to learn a continuous space of tasks (effectively an infinite set of tasks) parameterized by a latent vector z in a single policy model. By learning a continuous space of tasks, in fact the model is more efficient in learning by leveraging a relatively moderate number of transitions to learn a large continuous space of behaviors.\n\n\n\n> When projecting generated observaiton sequences to closest matches in the offline dataset, won’t some of the closest-match observation sequences come from different trajectories? If so, there could be some discontinuities in the states retrieved, making the state distribution matching objective valid but potentially suboptimal?\n\nThe reviewer brings up an interesting point about closest matches coming from different trajectories. We highlight two key properties of the method: 1. The objective of state marginal matching does not require realizability. Our analytical solution is an outcome of ‘best-effort’ distribution matching – by finding the observation trajectory that is dynamically feasible and minimizes the KL divergence to the projected trajectory 2. Unsupervised RL allows one to stitch together behaviors and learn behaviors going beyond the trajectories in the dataset. In particular, it optimizes behaviors that maximize a parameterized class of reward functions given an offline dataset, solutions of which may require stitching multiple sub-trajectories.\n\n> Related Works\n\nWe thank the reviewer for bringing the works to our attention and we will add a discussion on them in our related work section.\n\nPlease let us know if our clarifications resolve your questions and concerns.\n\n[1]: Hansen, Nicklas, Xiaolong Wang, and Hao Su. \"Temporal difference learning for model predictive control. Proceedings of the 39 th International Conference on Machine Learning"}, {"review_id": "EC0VWgdGza", "reviewer": "Reviewer_f371", "summary": "This work proposes an imagine, project, and imitate framework to enable zero-shot language-to-behavior grounding. The transfer depends on a pretrained general video generation model, a projection function (semantic similarity), and an embodiment-specific unsupervised RL training stage. Crucially, none of the components need language-annotated in-domain demonstrations: the foundation models used for video generation and image similarity search are pretrained on general, scalable datasets, and the successor measure based unsupervised RL can be trained on mixed-quality offline interactions in the target domain. The offline interaction dataset is also used by the semantic retrieval step in order to ground the plan into the target observation space. Either a language command or a cross-embodiment video can be used to specify a task, with experimental evaluation in DeepMind control and HumEnv simulation environments.", "questions": "1. How exactly is the video generation model adapted to the offline dataset? Do you adopt the Aligner and Connector ideas from GenRL, even if you do not learn a world model? Moreover, why not condition the video generation to start from the current image observation from the environment?\n2. Is the assumption of the reward being linear with respect to the pretrained $\\varphi(s)$ a limitation?\n3. Can your reward definition also be hacked? Such as by visiting the expert’s state distribution but doing so out-of-order?\n4. Why do you refer to GenRL training as less safe than RLZero?\n5. Would a vision-only embedding model work for the projection step or does the model need to be trained for multimodal matching?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- A neat way to use video generative models as a prior for task specification, and especially powerful when combined with unsupervised RL.\n- Presents several desirable qualities over prior vision-language reward models, such as supporting immediate policy extraction, requiring less in-domain supervision, and reducing the risk of reward hacking.\n- Only (or mostly) unsupervised in-domain data is needed.\n- A policy can be derived in closed form without gradient updates or additional interaction, leading to impressive policy generation speed.\n- The proposed framework is interpretable: failures can be inspected for trajectory imagination and projection separately.\n\n**Weaknesses**:\n- Random exploration with RND seems to not quite be sufficient to produce relevant state space coverage in Stickman, which is why the offline data needed to be augmented with walking and running demonstrations (not annotated with natural language, however). The problem of generating an informative exploration dataset for complex domains remains a key issue.\n- Excluding GenRL as a baseline in HumEnv doesn’t seem entirely justified simply because the authors choose to start from a pretrained BFM rather than the offline dataset that was used to create the BFM.\n- The chosen discriminator-based quantitative evaluation metric may be biased in favour of the proposed method. Specifically, the discriminator is trained to consider projected frames as high-scoring behavior and the offline dataset as low-scoring. In this light, it might be expected that RLZero would perform well on this metric as it directly uses projected frames obtained in a similar way as an imitation target, whereas the baselines do not.\n- The paper could be improved by making the explanations more self-contained. In particular, a corresponding appendix section or a specific reference would help clarify how the functions $\\psi$ and $\\varphi$ are trained (and how a policy is derived from them). Same with how the video generation model is adapted with the offline dataset.\n\n**Minor**:\n\nSmall issues with the notation:\n- d_0 and s_0 are both used to denote the initial state distribution\n- it seems to be redundant to use both expectation and probability in Eq. 1\n- s+ is not defined\n- in Eq. 2, I believe the square should be inside the expectation", "was_revised": true, "final_justification": "While minor issues around clarity remain, they don’t affect my overall positive assessment. I maintain my acceptance rating.", "rebuttal": "We thank the reviewer for their detailed feedback. We respond to the reviewer’s concerns and questions below:\n\n> Random exploration with RND seems to not quite be sufficient to produce relevant state space coverage in Stickman, which is why the offline data needed to be augmented with walking and running demonstrations (not annotated with natural language, however). The problem of generating an informative exploration dataset for complex domains remains a key issue\n\nWe appreciate the reviewer's observation and acknowledge in the paper that state-coverage remains a key challenge for unsupervised learning. For any unsupervised method, the skills learned are limited to data coverage, and investigating how to enable scalable data collection is an interesting research topic, but beyond the scope of the work.\n\n\n> Excluding GenRL as a baseline in HumEnv doesn’t seem entirely justified simply because the authors choose to start from a pretrained BFM rather than the offline dataset that was used to create the BFM.\n\nFollowing the reviewers’ suggestion, we aim to include the comparison to GenRL for the HumEnv environment by using the dataset provided by MetaMotivo. The HumEnv environment requires learning a world model for a complex 3D humanoid, and our initial experiment resulted in a collapsed world model. Training the world model is a computationally intensive task (5 days on single GPU [1]) and might require longer time than the rebuttal to ascertain the hyperparameters that work for this setting. Since our language-to-policy experiments utilize a common video model to generate imaginations, our comparisons already show that RLZero is competitive to GenRL while bypassing test-time training and being faster at inference.\n\n> The chosen discriminator-based quantitative evaluation metric may be biased in favour of the proposed method. \n\nThe return under discriminator reward has been shown to be a valid metric to measure the distribution matching ability as it is an upper bound to KL divergence between the imitator and expert’s state visitation distribution[2,3] rather than a hand-crafted one. While methods based on image-language rewards or video-language rewards use a proxy objective for imitation, our method directly optimizes the objective we actually care about. As a result of this, our method and the GenRL baseline are expected to optimize the discriminator better. We also compare win-rates, which we believe is a truly unbiased metric to gain understanding of the effect of the entire pipeline from language-to-behaviors.\n\n> The paper could be improved by making the explanations more self-contained.\n\nWe appreciate the reviewers’ suggestion and will update the appendix with more details on unsupervised RL training and video generation model training. We will also fix the notation issues that the reviewer pointed out. Note that NeurIPS does not allow updating the paper during the rebuttal period, but we will ensure to make these changes in the final version of the paper. \n\n> How exactly is the video generation model adapted to the offline dataset? Do you adopt the Aligner and Connector ideas from GenRL, even if you do not learn a world model? Moreover, why not condition the video generation to start from the current image observation from the environment?\n\nYes, we follow the unsupervised video model training method from GenRL. Conditioning the video on the current image observation is an interesting idea that can enable closed-loop control with fewer compounding errors and is an interesting avenue for future research.\n\n> Is the assumption of the reward being linear with respect to the pretrained \\phi(s) a limitation?\n\nNo we do not believe reward being linear with respect to pretrained  \\phi(s)  to be a limitation. Note that we are learning rewards that are linear in terms of arbitrarily non-linear features of states - this can be viewed as a neural network that is learned to predict rewards from state, and we take the last layer of the NN to be features of the state under which the reward function remains linear.\n\n> Can your reward definition also be hacked? Such as by visiting the expert’s state distribution but doing so out-of-order?\n\nYes, we agree the way we define the reward function can potentially be hacked in the sense of permutation of states if allowed by the dynamics of environment. While we did not observe this to be a limitation in the environments we considered, this can be fixed in a principled way by matching visitations on {s,s’} and requires minimal changes to training the BFM.\n\n> Why do you refer to GenRL training as less safe than RLZero?\n\nRLZero allows us to partly interpret the output of the policy before deploying the policy by looking at the closest projected images. Tasks which do not seem realistic/unsafe can be discarded this way; in GenRL we know policy outcome only after deploying the learned policy.\n\n> Would a vision-only embedding model work for the projection step or does the model need to be trained for multimodal matching?\n\nSince we are interested only in image-to-image semantic similarity, it is possible that vision-only embedding would work, but language provides task information that might assist in better embedding similarity computation.\n\n\n\n[1]: Mazzaglia, Pietro, et al. \"GenRL: Multimodal-foundation world models for generalization in embodied agents.\" Advances in neural information processing systems 37 (2024): 27529-27555.\n\n[2]: Ma, Yecheng, et al. \"Versatile offline imitation from observations and examples via regularized state-occupancy matching.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3]: Zhu, Zhuangdi, et al. \"Off-policy imitation learning from observations.\" Advances in neural information processing systems 33 (2020): 12402-12413."}, {"review_id": "3z9J4lMU24", "reviewer": "Reviewer_u2oR", "summary": "The paper transfers a pre-trained reinforcement learning (RL) model to new tasks by aligning video of the target task with existing demonstration data.  \nIt then uses a shaped reward function with a successor features model to update the imitation policy without gradient-based optimization.\nThe input video can come from a video-generation system or be collected from publicly available sources.  \nDuring alignment, the method selects the most similar short frame sequences between the target video and the demonstration dataset.\nExperiments on continuous-control domains use videos generated from a text to video model or demonstrations from YouTube.\nThe approach outperforms GenRL (which relies on gradient updates) and avoids the need to train a separate policy for each new task, unlike the baseline RL methods evaluated.", "questions": "- For language-to-continuous control: why use win rate instead of score differences?\n- How fast is the projection step in the various experiments?\n\t- How does it scale with more demonstrations or longer task videos?\n\t- How long were task videos?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# quality\n- Good to show reasonable baselines and report statistical differences.\n\t- Also to show why reward models are inefficient.\n- Demonstrates reasonably good distribution matching\n- Lacks metrics on task outcomes: the objectives are distribution matching rather than achieving an outcome. While this makes some sense for language commands, it would help if the commands aligned with task-based objectives in at least some cases.\n\n\n# clarity\n- Experiments would benefit from a clearer narrative and explicit summary of results / takeaways.\n\n\n# significance\n- Domain adaptation with new data is a useful technique.\n- May be of interest in adapting models to new tasks through videos of demonstrations. Thus of potential interest to the robotics community.\n\n\n# originality\n- Mapping observations and mapping successor features are separately known, but the combination is novel. The novelty is in the steps of the process.", "was_revised": true, "final_justification": "The authors provided additional details on their evaluation methodologies and results. It is hard to tell whether the policies are succeeding in a ground truth sense, but the new results are modestly persuasive. I've revised my score somewhat accordingly.", "rebuttal": "We thank the reviewer for their thoughtful review. We answer the reviewers' questions below:\n\n> For language-to-continuous control: why use win rate instead of score differences?\n\nOur use of win-rate is motivated by the fact that there is no ground truth reward function associated with the prompted tasks. For instance, “walk like a human” may involve walking at 5 m/s or 10 m/s velocity. As a result, manually assigning a reward function can incorrectly favor one method over another. We use a denser metric in tables 1 and 2: discriminator as a reward function. This allows us to compare how well the agent is able to behave in a manner corresponding to the imagined trajectory for the language prompt and assign a dense score.\n\n\n> How fast is the projection step in the various experiments? a. How does it scale with more demonstrations or longer task videos? b. How long were task videos?\n\nThe projection step takes a wall clock time of 10-20 seconds on AMD EPYC 7763 64-Core Processor CPU and 1 NVIDIA-A40 GPU. The costs involved here are forward inference to obtain the embedding of each frame in the video and then searching over the dataset, which requires O(n_video_frames x n_dataset_transitions) operations, thus scaling linearly with video length size. Note that the dataset embeddings are pre-computed and cached. The task videos were of variable lengths, from 36 frames to 120 frames.\n\n\n   > Lacks metrics on task outcomes: the objectives are distribution matching rather than achieving an outcome. While this makes some sense for language commands, it would help if the commands aligned with task-based objectives in at least some cases\n\n\n\nWe appreciate the reviewers’ suggestion, but to reiterate from the first point above, that language to reward is inherently ambiguous, in particular for the domains we study in this work. As an example, language instructions of walking, strutting, or a yoga pose create ambiguity on what is the desired velocity, direction, height, etc. Looking at the output behaviors of one of the trained agents (RLZero, GenRL, VLM as reward) to get an outcome pose/goal pose would result in an unfair comparison.  We show one such comparison with hand-designed rewards below on some tasks - we compute outcome rewards for some tasks with hand-designed reward functions:\n\n| Task | TD3 (VLM-RM) | GenRL | RLZero |\n| --- | --- | --- | --- |\n| walker – lying down | 986.61 | 989.6 | 1000.0 |\n| walker – Walk like a human | 151.91 | 570.13 | 704.63 |\n| quadruped – handstand | 7.99 | 139.5 | 210.53 |\n\n\nWe would also like to clarify that we do not use a demonstration dataset, as mentioned in the reviewer’s summary.  Our dataset is collected mostly by random exploration in language-to-policy experiments. This makes learning imitation policy challenging from offline data.\n\n\nPlease let us know if our clarifications resolve your questions and concerns."}]

Author rebuttal:

Meta-review: This paper introduces RLZero, a framework for zero-shot language-to-policy inference that does not require in-domain supervision or labeled trajectories. The method follows an imagine–project–imitate pipeline: a video model imagines a trajectory from language input, a projection step aligns it with the target environment’s observation space, and an unsupervised RL agent instantly imitates the projected trajectory through closed-form policy inference. The work is evaluated on DM Control Suite and humanoid domains, including cross-embodiment transfer from third-person videos.

The reviewers agree that the main strength lies in the novelty of combining foundation video models with unsupervised RL to bypass costly language-annotated or in-domain demonstrations. The paper demonstrates generality across tasks and embodiments, immediate policy extraction without gradient updates, and interpretability of failures through the modular pipeline. Results are competitive with strong baselines such as GenRL, and the approach shows promise for scaling to real-world settings where supervision is scarce.

The main weaknesses highlighted include (i) brittleness of the projection step, which sometimes latches onto irrelevant features, (ii) reliance on large offline datasets (millions of transitions), which raises questions about scalability, (iii) limited comparisons on outcome-based metrics, and (iv) clarity issues in the introduction and method presentation. Some reviewers also noted missing baselines (GenRL on HumEnv) and potential bias in the evaluation metric. The authors responded constructively, adding high-level overviews for clarity, clarifying evaluation methodology, and providing additional outcome-based results to address reviewer concerns. While some limitations remain, particularly around projection robustness and dataset efficiency, the consensus is that these are natural challenges for this early-stage approach rather than fatal flaws.

Overall, this is a technically solid paper with interesting and original contributions at the intersection of RL, generative video modeling, and language grounding. It does not yet rise to the level of a spotlight due to evaluation and clarity limitations, but it represents a meaningful step forward and will be of broad interest to the NeurIPS community.
Decision: Accept (poster)