Title: RLZero: Direct Policy Inference from Language Without In-Domain Supervision
Venue: NeurIPS 2025
Accepted: yes
Keywords: Zero-shot RL

Unsupervised RL

Behavior Foundation Models
Abstract: The reward hypothesis states that all goals and purposes can be understood as the maximization of a received scalar reward signal.  However,  in practice, defining such a reward signal is notoriously difficult, as humans are often unable to predict the optimal behavior corresponding to a reward function.  Natural language offers an intuitive alternative for instructing reinforcement learning (RL) agents, yet previous language-conditioned approaches either require costly supervision or test-time training given a language instruction. In this work, we present a new approach that uses a pretrained RL agent trained using only unlabeled, offline interactions—without task-specific supervision or labeled trajectories—to get zero-shot test-time policy inference from arbitrary natural language instructions. We introduce a framework comprising three steps: *imagine*, *project*, and *imitate*. First, the agent imagines a sequence of observations corresponding to the provided language description using video generative models. Next, these imagined observations are projected into the target environment domain. Finally, an agent pretrained in the target environment with unsupervised RL instantly imitates the projected observation sequence through a closed-form solution. To the best of our knowledge, our method, RLZero, is the first approach to show direct language-to-behavior generation abilities on a variety of tasks and environments without any in-domain supervision. We further show that components of RLZero can be used to generate policies zero-shot from cross-embodied videos, such as those available on YouTube, even for complex embodiments like humanoids.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper transfers a pre-trained reinforcement learning (RL) model to new tasks by aligning video of the target task with existing demonstration data.  \nIt then uses a shaped reward function with a successor features model to update the imitation policy without gradient-based optimization.\nThe input video can come from a video-generation system or be collected from publicly available sources.  \nDuring alignment, the method selects the most similar short frame sequences between the target video and the demonstration dataset.\nExperiments on continuous-control domains use videos generated from a text to video model or demonstrations from YouTube.\nThe approach outperforms GenRL (which relies on gradient updates) and avoids the need to train a separate policy for each new task, unlike the baseline RL methods evaluated.\n\nSTRENGTHS AND WEAKNESSES:\n# quality\n- Good to show reasonable baselines and report statistical differences.\n\t- Also to show why reward models are inefficient.\n- Demonstrates reasonably good distribution matching\n- Lacks metrics on task outcomes: the objectives are distribution matching rather than achieving an outcome. While this makes some sense for language commands, it would help if the commands aligned with task-based objectives in at least some cases.\n\n\n# clarity\n- Experiments would benefit from a clearer narrative and explicit summary of results / takeaways.\n\n\n# significance\n- Domain adaptation with new data is a useful technique.\n- May be of interest in adapting models to new tasks through videos of demonstrations. Thus of potential interest to the robotics community.\n\n\n# originality\n- Mapping observations and mapping successor features are separately known, but the combination is novel. The novelty is in the steps of the process.\n\nQUESTIONS:\n- For language-to-continuous control: why use win rate instead of score differences?\n- How fast is the projection step in the various experiments?\n\t- How does it scale with more demonstrations or longer task videos?\n\t- How long were task videos?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an imagine, project, and imitate framework to enable zero-shot language-to-behavior grounding. The transfer depends on a pretrained general video generation model, a projection function (semantic similarity), and an embodiment-specific unsupervised RL training stage. Crucially, none of the components need language-annotated in-domain demonstrations: the foundation models used for video generation and image similarity search are pretrained on general, scalable datasets, and the successor measure based unsupervised RL can be trained on mixed-quality offline interactions in the target domain. The offline interaction dataset is also used by the semantic retrieval step in order to ground the plan into the target observation space. Either a language command or a cross-embodiment video can be used to specify a task, with experimental evaluation in DeepMind control and HumEnv simulation environments.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- A neat way to use video generative models as a prior for task specification, and especially powerful when combined with unsupervised RL.\n- Presents several desirable qualities over prior vision-language reward models, such as supporting immediate policy extraction, requiring less in-domain supervision, and reducing the risk of reward hacking.\n- Only (or mostly) unsupervised in-domain data is needed.\n- A policy can be derived in closed form without gradient updates or additional interaction, leading to impressive policy generation speed.\n- The proposed framework is interpretable: failures can be inspected for trajectory imagination and projection separately.\n\n**Weaknesses**:\n- Random exploration with RND seems to not quite be sufficient to produce relevant state space coverage in Stickman, which is why the offline data needed to be augmented with walking and running demonstrations (not annotated with natural language, however). The problem of generating an informative exploration dataset for complex domains remains a key issue.\n- Excluding GenRL as a baseline in HumEnv doesn’t seem entirely justified simply because the authors choose to start from a pretrained BFM rather than the offline dataset that was used to create the BFM.\n- The chosen discriminator-based quantitative evaluation metric may be biased in favour of the proposed method. Specifically, the discriminator is trained to consider projected frames as high-scoring behavior and the offline dataset as low-scoring. In this light, it might be expected that RLZero would perform well on this metric as it directly uses projected frames obtained in a similar way as an imitation target, whereas the baselines do not.\n- The paper could be improved by making the explanations more self-contained. In particular, a corresponding appendix section or a specific reference would help clarify how the functions $\\psi$ and $\\varphi$ are trained (and how a policy is derived from them). Same with how the video generation model is adapted with the offline dataset.\n\n**Minor**:\n\nSmall issues with the notation:\n- d_0 and s_0 are both used to denote the initial state distribution\n- it seems to be redundant to use both expectation and probability in Eq. 1\n- s+ is not defined\n- in Eq. 2, I believe the square should be inside the expectation\n\nQUESTIONS:\n1. How exactly is the video generation model adapted to the offline dataset? Do you adopt the Aligner and Connector ideas from GenRL, even if you do not learn a world model? Moreover, why not condition the video generation to start from the current image observation from the environment?\n2. Is the assumption of the reward being linear with respect to the pretrained $\\varphi(s)$ a limitation?\n3. Can your reward definition also be hacked? Such as by visiting the expert’s state distribution but doing so out-of-order?\n4. Why do you refer to GenRL training as less safe than RLZero?\n5. Would a vision-only embedding model work for the projection step or does the model need to be trained for multimodal matching?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents RLZero, which trains a video foundation model to generate videos given a reward-unlabeled offline dataset. This video model is used at test time to generate videos of an agent performing a task, given a language description, and then project the videos to the dataset to find nearest matching states. A state-distribution matching objective is applied to learn a policy to imitate these behaviors.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n**Analysis:** THere’s some nice analysis and discussion on inference time of the pipeline. \n\n**Results:** Results across many tasks in a few embodiments demonstarte RLZero works pretty well. Analysis is comprehensive too, and the reported metrics make sense given the open-ended evaluation. \n\n**Method:** I like that the method intelligently combines the best aspects of large foundation models (to use for training the video gen model and perform projection) while also combining with more traditional RL techniques such as successor feature representations and state distribution matching. \n\n## Weaknesses\n\n**Writing Clarity:** \n\n- The intro doesn’t really flow well from one point to the next. It’s fine, but I think the logic takes a couple of turns by presenting a multi-step solution that spans paragraphs 2 and 3. Perhaps hinting at (1) imagining and (2) projecting far earlier; e.g., paragraph 1, would help with understanding the intro.\n- Similarly the methods section is written at an extremely low level without having a high-level to refer to. It feels that each part of the method is written after the previous, but there’s no high-level intuition about how the method works at the top of each subsection (4.2 and 4.3). This could just be me, but the intro and methods took me multiple times to read to fully understand the contributions of the paper and how exactly it works.\n\n**Lots of prior data:**\n\n- Experiments are performed with 5-12million initial transitions. This is a LOT of data. Some experiments with less data would be beneficial to demosntrating the scalability o the method.\n\n## Minor Things\n\nSome related work in learning generalist agents that can immediately produce actions for a given language description:\n\n- BOSS: https://clvrai.github.io/boss/\n- https://arxiv.org/abs/2306.01711\n- DECKARD: https://deckardagent.github.io/\n\nQUESTIONS:\nWhen projecting generated observaiton sequences to closest matches in the offline dataset, won’t some of the closest-match observation sequences come from different trajectories? If so, there could be some discontinuities in the states retrieved, making the state distribution matching objective valid but potentially suboptimal?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes to leverage foundation models for policy learning by first imagining a trajectory conditioned on language instructions, then projecting this imagined trajectory to real observations from the dataset, and finally imitating those observations using distribution matching. They show results on language-conditioned continuous control tasks from DM Control Suite and on video-conditioned continuous control tasks on Humanoid environments (for cross-embodiment transfer).

STRENGTHS AND WEAKNESSES:
Strengths
- Leveraging off-the-shelf foundation models for continuous control is a promising approach to address the scarcity of data for control, and it is especially interesting that compared to other methods this does not require in-domain trajectories for learning inverse dynamics models
- The paper is clearly written
- The results on the humanoid tasks, which are more realistic than DM Control, show that the method can be quite general

Weaknesses
- The approach relies heavily on the “project” step which can be extremely brittle. The authors themselves acknowledge this: “We observe the image-similarity to latch on to features from the background and produce incorrect retrieval.” It would be informative to see results on more “in-the-wild” videos with noisier backgrounds. 
- A strong alternative to semantic similarity would be using off-the-shelf pose detection models and matching the joints. Could the authors compare to those methods?

QUESTIONS:
See Strengths and Weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01