Regularized Latent Dynamics Prediction is a Strong Baseline For Behavioral Foundation Models
Authors: Pranaya Jajoo
Harshit Sikchi
Siddhant Agarwal
Amy Zhang
Scott Niekum
Martha White
Venue: ICLR 2026 Poster (2026)
Keywords: Behavioral Foundation Models (BFMs)
Zero-shot Reinforcement Learning
Zero-shot RL
Representation Learning
Unsupervised RL
Primary area: reinforcement learning

Abstract: Behavioral Foundation Models (BFMs) have been recently successful in producing agents with the capabilities to adapt to any unknown reward or task. In reality, these methods are only able to produce near-optimal policies for the reward functions that are in the span of some pre-existing _state features_. Naturally, their efficiency relies heavily on the choice of state features that they use. As a result, these BFMs have used a wide variety of complex objectives, often sensitive to environment coverage, to train task spanning features with different inductive properties. With this work, our aim is to examine the question: are these complex representation learning objectives necessary for zero-shot RL? Specifically, we revisit the objective of self-supervised next-state prediction in latent space for state feature learning, but observe that such an objective alone is prone to increasing state-feature similarity, and subsequently reducing span of reward functions that we can represent optimal policies for. We propose an approach, RLDP, that adds a simple regularization to maintain feature diversity and can match or surpass state-of-the-art complex representation learning methods for zero-shot RL. Furthermore, we demonstrate the prior approaches diverge in low-coverage scenarios where RLDP still succeeds.

Published as a conference paper at ICLR 2026
REGULARIZED LATENT DYNAMICS PREDICTION IS A STRONG BASELINE FOR BEHAVIORAL FOUNDATION MODELS
Pranaya Jajoo
Harshit Sikchi*
Siddhant Agarwal
Amy Zhang*
Scott Niekum
Martha White
Department of Computing Science, University of Alberta, Canada
2 Alberta Machine Intelligence Institute (Amii)
3 Canada CIFAR AI Chair
4 The University of Texas at Austin
5 University of Massachusetts Amherst
f Equal contribution
ABSTRACT
Behavioral Foundation Models (BFMs) produce agents with the capability to adapt to any unknown reward or task. These methods, however, are only able to produce near-optimal policies for the reward functions that are in the span of some pre-existing state features, making the choice of state features crucial to the expressivity of the BFM. As a result, BFMs are trained using a variety of complex objectives and require sufficient dataset coverage, to train task useful spanning features. this work, we examine the question: are these complex representation learning objectives necessary for zero-shot RL? Specifically, we revisit the objective of self-supervised next-state prediction in latent space for state feature learning, but observe that such an objective alone prone increasing state-feature similarity, and subsequently reducing span. We propose an approach, Regularized Latent Dynamics Prediction (RLDP), that adds a simple orthogonality regularization to maintain feature diversity and can match or surpass wate-of-the- art complex representation learning methods for zero-shot RI. Furthermore, we empirically show that prior approaches perform poorly in low-coverage scenarios where RLDP still succeeds.
INTRODUCTION
Zero-shot reinforcement learning (RL) (Touati et al. 2022) is a problem setting where we learn an agent that can solve any task in the environment without any additional training or planning, after an initial pretraining phase. shro RL has significant practical potential in developing generalist agents with wide applicability. For instance, robotics applications, like robotic manipulation or drone navigation, often require agents to solve wide variety of unknown tasks. A general-purpose household robot needs to possess the capability to flexibly adapt to various household chores without explicit training for each new task.
Behavioral Foundation Models (BFMs) have been shown to be promising for zero-shot RL (Touai al. 2022 Agarwal et al. 2024). BFMs are trained on dataset of reward-free interactions, with the aim to provide near-optimal policy for a wide class reward functions without additional learning training during test-time. BFMs are trained by learning state representation y b) learning policy π conditioned on a latent vector 2 E Rd, where the can seen as a task embedding for reward r(s) = pps) T In this way, the BFM consists of space of policies, where different policies can be extracted by querying the learned policy using different 2. At test time, given any reward function test (s), the near-optimal policy Nz,teu is obtained zero-shot by solving for zyte such that yeat (s) RJ (ps) Zptest. This assumption on the reward also used for successor
Rdd, and
"Correspondence pranaya jooqua barta.ca

---

Published as a conference paper at ICLR 2026
features (Barreto al. (2016)), which consist of the discounted cumulative sum of feature vectors under a policy. Successor features zero-shot produce the action-values for a new reward, given by vector and BFM approaches often successor features to learn the policies.
The performance of the BFM relies heavily on the state representation. which is both used to extract for the reward and for the policy. State-of-the-an methods (Tuuai & Ollivier (002)); Agarwal et al. (2024); Park et al.] (2024) usually learn state representations that retain information represent successor measures under wide class of policies. A successor measure captures the (discounted) state visitation of a policy, given any starting state. They are the generalization of successor representations Dayan (1993) continuous states, and have simple linear relationship to successor features Touati & Ollivier (022). They can therefore be used to both encourage learning a generalizable state representation as well as simultaneously learning the successor features for the BFM. Successor measures are usually learned for an explicitly defined class of policies (Agarwal
2024) or implicitly by first defining class of reward functions (Touati al.) 2022 Park jt al.
2024 and considering optimal policies for those reward functions as the set of policies. The main intuition behind predicting successor measures as a target for state representation learning is that representations sufficient to explain future state- visitaiistttion for wide range of policies capture features that are relevant for sequential decision making under various reward functions.
al.
Unfortunately, learning state representations by estimating successor measures requires iteratively applying Bellman evaluation backups or Bellman optimality backups, both of which are known to result in a variety of learning difficulties. They suffer from various forms of bias Thrun & Schwwrtz (2014); Fujimoto et al. (2018): Lu et al.) (2018); Fu et al. (2019) and can suffer from feature collapse due the instability inherent in bootstrapping in the function approximation regime Kummar et al. 202T). Using Bellman backups to learn a representation requires choosing a class of policies or class of reward functions a priori. Further, because the state representation is trained from batch of offline data, unless chosen carefully, the policies may select out-of- distribution actions, leading to incorrect generalization and degenerate representations.
A simple alternative that sidesteps these issues is to use latent dynamics learning: predicting future latent states given the current state and the sequence of actions. Learning the state representation by predicting the latent dynamics has the benefit of being independent of the policy and thus avoids using Bellman backups and these out-of-distribution issues. This work investigates the following question:
Does latent next-state prediction produce state features that enable performant zero-shot RL?
Our investigation is inspired by the work of Fujimoto al. (025)) which showed that using dynamics prediction losses as auxiliary losses boosted performance of a single-task RL agent. Unlike the single task RL setting examined by Fujimoto al.] (2025), find that in its naive form, this objective leads to a mild form of feature collapse where the representation of different states increase in similarity over training. This collapse results in poor zero-shot RL performance when evaluated on number of downstream tasks. With simple orthogonality regularization to prevent collapse, we show that the representations learned are competitive and present scalable alternative to representations learned via complex successor measure estimation methods for zero-shot RL.
In summary, the contributions of this paper are as follows. 1. dynamics prediction (RLDP) as a simple alternative to learn state features for zero-shot RL. We identify as well as mitigate feature collapse plaguing latent dynamics prediction. 2. We show that our method remains competitive through an extensive empirical evaluation of representations for task generalization across variety of domains, online and offline RL settings, including in humanoid with large state-action space. 3. We show that the RLDP objective can learn performant policies in low-coverage settings where other methods fail.
We propose regularized latent-
RELATED WORK
Unsupervised RL encompasses the class of algorithms that enable learning general- purpose skills and representations without relying reward signal in the data. Works that have focused on intent or skill discovery have used diversity-drived objectives (Eysenbach et al. 20188 Achiam et al. 2018). maximizing mutual information (Warde-Farley al. (2018). Eysenbach et al. (011))
2

---

Published as a conference paper at ICLR 2026
Achiam et al. (2018) Eysenbach al. 22221) minimizing the Wasserstein distance (Park et al. (2023)) between latents and the induced state-iisitatin distribution. These discovered skills can be used to compose optimal policies for several rewards. Our work, the other hand, focuses on learning representations capable of producing optimal value functions for any arbitrary function reward specification.
There are also a variety of pre-training approaches for representations that can be fine-tuned for downstream control. Recent pre-training approaches (e.g., Ma al. (2022); Nair et al. (2022)
borrow self-supervised techniques such as temporal contrastive objectives to extract embeddings
from large-scale datasets (Grauman al. (202T). HILP (Park al. (2024) goes beyond standard masked autoencoding approaches by using Hilbert-space representations to preserve temporal dynamics. Auxiliary objectives involve complementary predictive tasks to get richer semantic or temporal structures (Agarwal et al.) Schwaarrrr et al.) 2020). Although representations from auxiliary objectives can accelerate policy learning, new policy still needs be learned from scratch for each new reward function.
Behavioral Foundation Models are obtained by training an RL agent in an unsupervised manner using task-agnostic reward-free offline transitions. Forward-Backward representations Touati & (202I) and PSM (Agarwal et al. (2024)) provide one such framework for training BFMs by learning representations that capture set of successor measures, which several successive works are hased. Fast Imitation with BFMs (Pirotta et al. (2023) demonstrates the ability of successor- measure-based BFMs imitate new behaviors from just few demonstrations, while [Sikchi] iet al. (2025) builds upon this by fine-tuning the BFM's latent embedding space, yielding 10-40% improvement over their zero-shot performance. Recent progress in imitation learning has led to the development of BFMs tailored for humanoid control tasks (Peng et al. (2022), Won et al.) 20022) Luo et al.] (2023), Tirinzoni et al. (2025) which can produce diverse behaviors trained using human demonstration data. Our work differs from these in that it provides a new, simpler state-representation learning objective for training BFMs.
Ollivier (02I))
PRELIMINARIES
We consider a reward-free Markov Decision Process (MDP) (Puterman) 2014) which is defined as a tuple M = (S, A. P. T), where S and A respectively denote the state and action spaces, P denotes the transition dynamics with P(s'|s. [sa[ss a) indicating the probability of transitioning from s to s' by taking action a, do denotes the initial state distribution and y E (0, specifies the discount factor. A policy π is a function π S ((A) mapping state s to probabilities of action in A. We denote by Pr(- 188 18,, TT) and E[- 8, a, m] the probability and expectation operators under state- action sequences (s,, 42)220 starting at (s, a) and following policy π with s ~ P(. 81-1 4111) and a T(- st). Given any reward function S + R. the Q-function of π for r is (7 a) E120 (1+1))))+1) T].
Behavioral Foundation Model (BFM) using Successor Features is a tuple (ep, th, n.) for state features, & : S + Z, successor features defined as ()ss, a, T) E,, 1. 7 (s:)), and the task- conditioned learned policy næ that inputs any task embedding = that corresponds to a reward function r; = x z. To define this policy иg, we use action-valus produced by the successor features vb. The action-value function for reward Tz and fixed policy π can be written as
(T F.z Σ (
2/ = E, Σ ( (s, =::=))=><]]==( ((,,,,))
The policy æ= is the optimal policy for reward rz, obtained by iteratively greedifying over Q (s, a). Using the successor feature notation, we can iteratively update n, for all states until satisfies the following fixed-point equation
T (-1ss) arg
for all
We overload notation and write (s, a, 2) to mean (sss, a, T2), because later we will directly input into network learn these successor features for near-optimal policy for r2-
The BFM can be used for any new reward function as long as we can obtain the z corresponding to that reward. This is straightforward to do if we are given dataset p, the corresponding
3

---

Published as a conference paper at ICLR 2026
can be extracted by solving the linear regression problem, min; -p[(p==1111 - (p-
Naturally, BFMs depend heavily on the choice the state representation yp; how to learn an effective Jp is the subject of this work.
Learning the successor features can be done using successor measures, instead of directly estimating the discounted sum of features yo. The successor measure (Dayan 1993 Blier et al.,
2021) of state-action (s, a) under a policy π is the (discounted) distribution over future states obtained by taking action in state and following policy ı thereafter:
M (x)
Σ /PP(Sst+1 E (,,, VXcs.
(M st)rts))) This simple linear The action-value can be represented as, -( = relationship between action-value functions and successor measures is similar to that of successor features and has been exploited by recent works (Touati & Ollivier 202 Agarwal et al. [024] Park ct al. 2024) to train BFMs. It has been shown by Touati & Ollivier (2021) that parameterizing the
successor measures as M (s, a, s)) at (s, 2) Ø(s+) yields 인 s, 2) successor features for the state feature (s() = (øe (=) 12 of Touati & Ollivier (2021)). Since, the
closed form solution for for any reward function was (ypP) ) pr,, using the parameterization of M implies = TTr.
To train the BFM, we alternate between successor measure learning phase to get and policy improvement phase. The successor measure learning phase learns to model densities M (s, 8* using the contrastive objective (Blier et al. (02T):
Successor-measure estimation:
(M sp (MF=
1 + 2 -. (M (s, ( Tz (s'),
s))3). (3)
This objective can be used assuming a fixed state representation o, only training the successor features v (as we will do for our approach), or allows for both é and 1b to be jointly optimized, as was done in the Forward-Backwarr (FB) algorithm (Touati & Ollivier 2021), PSM (Agarwal
jt al. 2024) and HILP (Park et al. 2024).
The policy improvement step greedily optimizes the action-value function given by this successor measure
Tg (s) = arg max Q (s, arg max Σ
T (s, -r(ss))r((""" r(st)
arg max ((b(s,a,z) (0(s*) ((((s)) r(s+))]
arg max (s a, 2)
Σ o(s r(s*)*+++))=) = arg max
(4)
In practice, we cannot directly set the policy to this argmax. Instead, we optimize the following loss.
Policy Improvement:
CP(mz)= -ag- (1) a,)(s,,,)))3
(5)
Appendix A.2 provides further overview of approaches train BFMs. In this work, we leverage this machinery for BFMs and focus on a new approach to estimate the state representations ø
4 METHOD
This method can be broadly divided into two parts: representation learning and zero-shot RL using successor features. The state representation encoder is trained using latent dynamics prediction with diversity regularization. We will show that these representations lead to reduction in the prediction error for successor measures for any policy. Leveraging these robust state embeddings, we then pretrain a Behavioral Foundation Model (BFM) to predict successor measures, enabling zero-shot inference of optr-optimal policies for unseen reward functions. We refer to this method as RLDP (Regularized Latent Dynamics Prediction based Behavioral Foundation Policies).
4

---

Published as a conference paper at ICLR 2026
4.1 LEARNING REPRESENTATION WITH REGULARIZED LATENT DYNAMICS PREDICTION
Zero-shot RL based on successor features relies on learning a state representation denoted by Φ(s). This state representation will define the span of reward functions that the zero-shot RL method is guaranteed to output optimal policies for.
The primary representation learning objective unrolled latent dynamies prediction. We learn a state representation encoder ø : S + IR (Z = Rd) and a latent state-action representation encoder : R4 x A - R2 such that latent dynamics can be expressed as d(s') = g(ó(s), a) w with some learned weights w, informing our loss function for representation learning. A sub-sequence of horizon { ag. si ..... ..., 1 - sH A sequence of future latent states h are obtained by encoding the initial state hg ((s)) and unrolling using the defined dynamics model h/+1 g(hi, ae) w. Then the objective is to predict the encoded future latent states:
sampled from the offline interaction dataset P given by 7²
H Σ t=1
Ca(o, 9, w) = Erindo
h
o(s))
(6)
where === {shr ab, s1, af. ... 1 a, - m hō target, which is periodically set to ø.
and ø is the slowly moving encoder (st)),
Latent dynamics models have been shown to significantly improve sample efficiency for
Average Cosine Similariy
8.6 0.5 0.
single task RL when models are used for
planning (Hansen et al. 2022), learning (Hafner jet al. 2019). or as representations (Fujimoto et al. (025) for model-free RL, but their suitability as general-purpose representations
6.3
84
for multi-task and zero-shot RL remains understudied. Most successful methods (Touati & Ollivier, 2021] Agarwal et al.] 2024) for zero-shot RL train representations to predict successor measures. However, directly estimating successor measures requires learning future otate-occupancies under a predefined set of policies. This poses a problem in the low-coverage setting as Bellman backups with policies that choose out of distribution action will result in incorrect predictions and negatively affect representation learning. In contrast, latent dynamics prediction is policy-independent representation learning objective.
Representation Learning Gradient Steps
Environment Chootah
Pointmass
-Chetahh [A 1.0)
Pointmass (A 1.01
Figure 1: Average Cosine similarity between state- representations sampled uniformly from the training datasct. Feature similarity increases over the course of training; once adding our orthogonality regularizer (with = 1), we obtain more diverse representations. Shaded region shows standard deviation over 4 seeds.
However, solely learning with the latent dynamics objective can lead to convergence to a collapsed solution. This is unsurprising trivial solutions of predicting constant zero vector achieves perfect loss in equation 6 To combat this, prior works (Grill et al., 2020) have proposed the use of a semi-gradee gradient update where stop-gradien is used for target h in equation along with a slowly updating target. However, we find these techniques insufficient to maintain representation diversity. We investigate this by computing the cosine similarity of state representations of gradient steps trained via minimizing equation [o an offline dataset collected by an exploration algorithm RND (Burda al. 2018). Figure shows that while the solutions do not collapse, there is an increase in feature similarity over the course of learning, which we refer to as a mild form of collapse. As the space of reward functions is spanned by state features, such an increase in feature similarity directly reduce the class of reward functions for which we can learn optimal policies and negatively impact task generalization.
function
Mitigating collapse in latent dynamics prediction: In order to prevent the mild form of feature collapse discussed earlier, we propose to add an auxiliary regularization objective that encourages diversity. Orthogonal regularization has been also studied in self-supervised learning (He et al.) 2024| Bansal et al.) 2018) as a way to mitigate collapse. We project all state representations as well as predicted latent next-state g(o(s), a) w in hypersphere: gd-1 {x E Rd Vd) and regularize by minimizing cosine similarity between any states. We
||0|||
5

---

- be defined as Published as a conference paper at ICLR 2026
ablate the choice of hyperspherical normalization 9 in Appendix A.4.2 and observe it give consistent improvements. We note that a similar regularization was applied to state features in the implementation for identifiability and uniqueness. In the case of latent dynamics prediction this step becomes crucial to mitigate the increase in representation similarity.
representations (Touati al. 2022) to encourage solution
The orthogonal regularization loss takes the following form:
Reward-free
trannations
(x) (x. ((8')
(7)
$o
ENCODER
where Φ E S Our final loss is a weighted
DYNAMICS
combination of dynamics prediction combined
with orthogonal diversity regularization
a,
S1
CRLDP(O. w)= Ca(ó, g, w) + (o) (8)
ENCODER
Similarity
where X controls the regularization strength. We visualize this loss in figure [ where the encoder given by o, the dynamics by g and and the diversity across encoded states is encouraged with L. We find that adding this regularization prevents collapse, shown in figure I even with a relatively small regularization coefficient À 0.01. We evaluate the impact of orthogonality regularization further with other coefficients in Appendix section A.4.2.
Q2
$2
Z22
Qg
$3
ENCODER
Z3
RLDP leads to representations capable of predicting successor measures: The representation learning objective simply latent dynamics prediction with an orthogonal regularization. Through this objective, we are enforcing the representations to good for predicting successor measures, which forms the basis of the BFMs that will be constructed using these. While prior work (Agarwal et al., 2025) already indicate that these representations are suitable for predicting successor measures, latent space abstract MDP M defined using the state representation Φ
Figure 2: RLDP combines latent next state prediction
regularization for diversity (an orthogonality regularizer) leam representations for BFMs.
further formalize this intuition. Let's begin by looking at the
Definition 4.1. Let MDP M corresponding to the state abstraction < (,,,, P(-lo(s), ,,
Apart from facilitating the construction of BFMs and zero-shot RL, one of the utilities of the staterreprssentationns is to compress the state space to smaller space. MDP M represents this compression. We will assume that M is Lipschitz. Formally, Assumption 4.2. M is (KR, KP) Lipschitz. (Gelada et al. 2019
We now have all the components to show in the Lemma 4.3 that minimizing LRLDP will lead to a reduction in the prediction error of successor measure for any Ky--Lipshiitz valued policy.
Lemma 4.3. Given MDP M, let be any Ky-Lipschitz valued policy, M be the successor measure for π and M be the corresponding successor measure on M, LRLDP(0, 9, w) upper bounds the prediction error in successor measure,
Eg a a (M - CRLDP(Ó.g,w) 1
4.2 ZERO-SHOT RL WITH RLDP REPRESENTATIONS
The first step RLDP is to train the representations from batch of reward-free offline environment transitions. The RLDP representation loss given in equation g does not rely on reward, because it only the latent dynamics prediction loss and the orthogonality regularizer.

---

Published as a conference paper at ICLR 2026
It is straightforward to optimize this loss the batch of offline data, to obtain learned state representation c. Key choices to be made include the regularization coefficient A, the length of the rollout H for the latent dynamics prediction loss and the update frequency for encoder target ő.
The next step is to train the BFM, by alternating successor measure estimation and policy improvement. The RLDP representations are kept frozen in the successor measure parameterization (* a, s*) (" 2) o(s*) and 1b(s, a, 2) and Ws are trained using losses 10 and respectively.
(1srl(1)) xa as, 2)((s')
R. aa. - (s,,,,,, -5 ( ())) (10)
Following prior work, in our experiments we consider variations of the policy improvement step (5 where we use an expert regularization in the policy update (Trrinzoni et al.) 2025) to guide exploration during online RL. for high-dimensional state-action spaces or use behavior cloning regularization (Fujimoto & Gu (202]) when learning offline for low-coverage datasets. These modifications are discussed in detail in the next section. We provide the full representation and policy learning pipeline for RLDP in Appendix section A.6
5
EXPERIMENTS
The goal of our experiments is to perform an extensive empirical study of the suitability of state representations learned by a regularized latent next-state prediction objective when compared to other methods that employ more complex strategies. In particular, we aim to answer the following questions: (a) Keeping all other learning factors similar, how does our method compare baselines in enabling generalization to unseen reward functions? We compare the representations learned by training multi-task policies with zero-shot RL both in the offline setting and the online setting. (b) By avoiding querying actions out of distribution does RLDP provide robust choice for learning representations in low coverage datasets? (c) What design decisions are crucial to the success of our method? We perform extensive ablation studies to understand our design choices.
For all datasets, we pretrain a BFM using the successor feature approach outlined in our method section 4 Each algorithm is given the same budget of gradient steps during pretraining, controlling the state representation dimension, and the final performance is obtained by taking the pre-trained model the end and querying it for different task-rewards for 50 episodes.
5.1 BENCHMARKING ZERO-SHOT RL FOR CONTINUOUS CONTROL.
Baselines: We broadly
compare RLDP against commonly used state-
Task
Random Features
FB
PSM
RLDF
Stand
392.40:1:8..3
918.29--28883
899.54:1:0.73
890.40--27.33
Walker
of-the-art baselines for zero-shot RL such as: FB,
Run
75.39::220997
381.31+:7.322
450.57::28.95
334.26-1-49.9
Walk
193.84=+11290
779.29=66.60
875.61-133.44
779.77=137.16
Flip
132.02-166.85
977.08:12.76
621.36:175.62
492.94
PSM, and HILP. These
baselines represent of diverse and strong approaches in the area of zero-shot RI..
set
Cheetah
Run
31.82:136.88
129, 729.39::37.63
181885154.17
157.12::29.92
Backward
60.08.111.82
142.41.136.77
158.64118.56
175.52.L15.30
Walk
155.52
604.54:180.51
$76.98:209.44 .45
592.92-1 104.66
Walk Backward
272.77:1:42.40
1.0.40.1:144.23
817.92.1 98.86
821.51 500.62
Pointinass Quadruped
Stand
240.01-1:66.6
732.59:1101.33
708.03-1.34999
794.94:1-43.25
Run
114.19:1:0.222
425.15:::2.02
404.32:1-23.26
457.41:1:44.70
5.1.1 OFFLINE ZERO-SHOT RL
Walk
137.65.147.57
492.91.1
$23.94.1.2..1
465.40 185.29
Jump
190.62-1-46.63
567.27:148.90
733.32-1:55330
Top Left
258.59:1183.56
143.85117.33
924.20 10.64
990.4111.00.99
Setup: We consider
Top Right
189.30=:18900
$50.84:1:82441
133.00
795.47--21.10
Bottom Left
193.32--90.37
672.28-1115306
10.6933215.62
17::204440
continuous control tasks from DeepMind control suite Tassa et al. 2018) Pointmass, Cheetah, Walker, Quadruped under similar setup considered
Bottom Right
64.08:472.21
272.97:+274.99
123.44=138.22
193,38=116766
seeds) of zero-shot RL performance between using an untrained initialized encoder, FB, PSM, and RLD? with representation size d 512. Bold indicates the best mean and any method whose mean plus one standard deviation overlaps with the best mean.
Table 1:
Comparison (over

---

Published as a conference paper at ICLR 2026
by prior works in zero-shot RL. We datasets from the ExoRL suite (Yarats al. (2022) that are obtained by an exploratory algorithm RND (Burda et al. (2018). Random features use representations from randomly initialized NN encoder.
Evaluation: To evaluate the different zero-shot RL methods we take the pretrained policies and variety of tasks. For each environment, consider 4 tasks similar prior
query them works (Touati et al., 2022 Park et al.; [2024] Agarwal et al. 2024). We conduct our experiments across two axes: a) Table pretrains all the BFMs same number of representation dimensions (512) and gradient steps. For RLDP, we use an encoding horizon of 5. We train representations for 2 million steps and train policy for additional million steps. b) Table the Appendix compares against representation dimension for & found to be best for prior methods and RLDP with the same number of gradient updates for pretraining each BFM.
Results: Overall, using learned representations (FB, PSM, RLDP) outperforms random features, confirming that representation learning is crucial for zero-shot RL. Among learned methods, PSM and RLDP generally achieve the strongest performance. Furthermore, training FB and PSM baselines sensitive to hyperparameters and we rely on author's implementation to tune hyperparameters.
5.1.2 ONLINE EERO RL
Previous section validated that RLDP representations lead to competitive zero-shot RL when the learning policies use offline interaction data. We explore if the learned representation enable competitive multi-task learning when agent is allowed interaction with the environment.
Sctup: We consider the SMPL @ooper itt al. (2023)) Humanoid environment that aims to mimic real human embodiment complex learning
and provides
challenge with a 358 dimensional observation space and 69 dimensional
action space. Due to the exploratory challenge of the environment, Tirinzoni ctt al. (2025) presented a new approach, Conditional Policy Regularization (CPR), to guide RL learning regularized with expert real-human trajectories. CPR trains successor measures in a similar way 3 but adds a regularization objective to policy encouraging it to jointly maximize Q-function while staying close to expert. This allows for better exploration and more realistic motions. Further implementation details can be found in appendix sections A.3.2 A.5.2.
Baseline
Figure 3: Pair-wise comparison of RLDP against prior offline representation learning methods using per-task oracle normalized performance differences (A = RLDP Baseline) in SMPL Humanoid environment. The gray diamond represents the IQM (Interquartile Mean).
as equation
Evaluation: Our representation learning phase offline and we use the metamotivo 5M transition datase(] collected from replay buffer of an online RI agent to learn state-representatiios and then use the CPR approach to train zero-shot policies. We train representations for million gradient steps and policy for 20 million environment steps. The offline phase of representation learning helps us remove the exploration confounder and test the quality of representations obtained by different approaches. The evaluation is performed on the full suite of 45 tasks provided by Tirinzoni et al. (2025). ). For each task, we present the normalized scores with respect to fully-online trained representations and policy in table 8 and present the aggregates results across tasks in figure over 4 seeds.
collectee
Results: Figure summarizes the results of RLDP representations with respect to baseline methods across all 45 tasks. Positive values indicate tasks where RLDP achieves higher normalized returns. These results suggest that overall RLDP fares competitively this evaluation are provided in table Further analysis shows that the performance is task dependent on some tasks (such as raisearms and lieonground), RLDP outperforms the baselines, even beating
the baselines. Complete results for
8

---

Published as a conference paper at ICLR 2026
the oracle performance for some tasks (shown in table 8). In others (like crawl rotate tasks), all methods perform subpar to oracle.
5.2 LEARNING REPRESENTATION WITH Low COVERAGE DATASETS
RLDP learns a policy-independent representation through latent dynamics prediction. Prior approaches assume class of policies learn representations predictive of successor measures, and this strategy can lead to poor out-of-distribution generalization when actions proposed by the policy are not covered by the dataset.
Setup. To evaluate this hypothesis concretely, we consider the D4RL. benchmark of OpenAl Gym MuloCo tasks (Fu al. (2020), [Todoroy al. (2012), [Brockman al. (2016)). This dataset has been widely used to examine the effects of value estimation error from out-of-distribution actions due to low coverage, which many offline RL algorithms struggle with (Kostrikov al. (202T); Fujimoto & Gu (2021); Kumar et al. (2020); [Wu et al. (2019); Sikehi et al. (2023)). We consider halfcheetah, hopper, and walker2d domains, and medium and medium-exper datasets.
Evaluation: evaluate the different zero-shot RI. methods, we first pretrain the representation learning methods on these datasets for I million gradient steps. We use a modified zero-shot policy learning approach that alternates between equation 3 and equation 5 that is additionally augmented policy improvement loss with a behavioral regularization inspired by Fujimoto & Gu (222I). This regularization allow the RL approach learn without overestimation bias and enabling us to establish fair comparison among representations learned by different approaches. We use the corresponding reward function provided by each dataset to do reward inference and evaluate the zero-shot policy. Further details are provided in Appendix sections A.3.3 and A.5.3
Busciine
80
RLDP
40
-20
Baseline
Figure Pair-wise comparison RLDP against baseline representation learning methods low-coverage D4RL dataset. Each point represents A RAIOP Rasseline for single {task, seed} pair. The gray diamond represents the IQM (Interquartile Mean).
Figure shows one-to-one comparison of normalized returns of RLDP against baseline methods using paired per-seed performance differences across low-coverage D4RL tasks over 10 seeds. RLDP outperforms all baseline methods in 5 of tasks. The overall mass of the violin lies above zero and IQM is positive, indicating that RLDP achieves higher normalized returns compared the baseline. Overall, the results suggest that RLDP reliable choice for feature learning in low coverage datasets while providing a simpler alternative to otherwise complex representation learning approaches. Per task normalized scores and statistical significance testing is reported in the Appendix section A.9 and table 9 We further bisect the individual and combined impact of using Bellman backups (similar to FB which may query out-of-distribution actions), latent next- state prediction, and orthogonality regularization for representation learning in section A.11 and find explicit Bellman backups to hurt performance learned policy.
Results:
5.3 WHAT MATTERS FOR SUPERVISING REPRESENTATION SUITABLE FOR CONTROL?
In section 4 we introduced RLDP method of representation learning with the loss used (equation g and the encoder training process. In this section, aim ablate components of this loss and the architecture of the encoder.
Orthogonality regularization: Keeping the encoding horizon constant (H =5), change the
orthogonality regularization coefficient. The results, presented in Figure 5 show that for zero
regularization (A 0), the average return decreases compared to X 0. This shows that

---

Published as a conference paper at ICLR 2026
diversity regularization is critical to the representation loss. For fixed encoding horizon, orthogonality regularizer = performs best. To further understand the role of the orthogonality regularizer in representation learning and how it helps prevent feature collapse, we refer to section and section A.4.2 where we show that the learned representations increase in cosine similarity
that
without regularization.
In section 4.1 introduce encoder training, where we project the latent next state representation ( a..
Encoder architecture:
Encoding Horizon
700
600
to hypersphere. Here, we ablate the importance of this projection. The results are presented in table 2 We observe that RLDP consistently outperforms its variant without spherical normalization on most tasks. The standard deviation is also higher for most results on the variant without hypersphere projection. This indicates that spherical normalization important design choice for stabilization and improving performance.
Average Return
500
Average
Cheetah
400
Pointmass
Quadruped
300
Walker
200
100
0.01
Orthogonality Regularization Coefficient
Figure 5: Evaluating the impact of Orthogonality Regularization: We one-sided Mann-Whitney U tests the per-seed returns over 4 seeds to compare different values the orthogonality regularization, and we observe that adding small orthogonality regularization coefficient À = 0.01 gives statistically significant improvement coefficient 0.0.
Results for all environments are reported in table 5 We provide additional discussion of the complete encoder architecture in section A.4 where we further ablate
encoder architecture.
CONCLUSION
This paper introduces RLDP, a representation learning objective for effective task generalization enabling performant behavioral foundation models. Our objective takes the simple form of regularized latent- dynamics prediction, objective that does not require any reconstruction, making it able to handle high-dimensional observation space and does not require explicit Bellman backups. making it more amenable to optimization. We identify that simply using latent-dynamies prediction leads a mild form of feature collapse where the state-representation similarity increases over time. To combat this issue, we propose using orthogonal regularization as way maintain feature diversity and prevent collapse. Using our method enables learning generalizable, stable, and robust representations that can achieve competitive performance compared to prior zero-shot signals. Importantly, we show that prior
Task
RLDP
RTDP wlo SN
Stand
794.94:1:3.25
661.73-95.75
Run
457.41474.00
378.97 148.47
Quadruped
Walk
185.40
219.39-151..1
Jump
733.32:4:5.30
495.9811133.8
Average(*)
612.77:48.17
514.02=86.94
Top Left
890.41.1:0.79
41..13
Top Right
795.47-4-11.10
122.92+122.99
Pointmass
Bottom Left
805,17::20.444
683. 12-76.22
Right
193.38:1167.66
22.5441.9.04
Average(*)
671.111292.58
181.6311341.00
Table 2: Study of encoder architecture (subset). Table
shows mean + std; RLDP significantly outperforms RLDP w/o SN on Pointmass. Quadruped, and pooled. SN: Spherical Normalization
RL techniques without relying on reinforcement-driee approaches struggle in low coverage setting and RLDP works robustly across different dataset types, making it practical unsupervised learning approach. This work, thus, paves the way for simpler yet effective approaches learn zero-shot policies in behavioral foundation models.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGEMENTS
We thank Siddarth Chandrasekar, Dikshant Shehmar, and Diego Gomez for enlightening discussions on unsupervised RL. This work has been conducted at the Reinforcement Learning and Artificial Intelligence (RLAI) lab at the University of Alberta, the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) the University of Massachusetts Amherst, and Machine Intelligence through drcision-making and Interaction (MIDI) Lab at The University of Texas at Austin. Support for this work was provided by the Canada CIFAR AI Chair Program, the Alberta Machine Intelligence Institute, and the Natural Sciences and Engineering Research Council of Canada (NSERC). HS, SA, and AZ. are supported by NSF 2340651, NSF 2402650, DARPA HR001 12490431, and ARO W911NF-24-100193. We are also grateful for the computational resources provided by the Digital Research Alliance of Canada.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery 2018.
algorithms. arXiv preprint
Siddhant Agarwal, Aaron Courville, and Rishabh Agarwal. Behavior predictive representations for generalization in reinforcement learning. In Deep RL Workshop Neural Information Processing Systems 2021. URL
forum? wttp://eee/oormmm..oooso.... d=w.odexs5JJ5..
Siddhant Agarwal, Harshit Sikchi, Peter Stone, and Amy Zhang. Proto successor measure: Representing the space of all possible solutions of reinforcement learning. arXiv preprint arXiv:2411.19416 2024.
Siddhant Agarwal, Caleb Chuck, Harshit Sikchi, Jiaheng Hu, Max Rudolph, Scott Niekum, Peter Stone, and Amy Zhang. A unified framework for unsupervised reinforcement learning algorithms. In Workshop on Reinforcement Learning Beyond Rewards@ Reinforcement Learning Conference 2025, 2025.
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations in training deep cans? arXiv preprint 100000181,.09102, 2018.
André Barreto, Rémi Munos, Tom Schaul, and David Silver. Successor features for transfer in reinforcement learning. arXiv preprint arXiv:1606.053/1 2016.
Léonard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent
values: A mathematical viewpoint. arXiv preprint arXiv:2101.07122. 2021.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 16Xi::6666.01... 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprini 1810. 12894, 2018.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural computation, 5(4):613-624, 1993.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprint arXiv: /802.06070 2018.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of unsupervised reinforcement learning. arXiv preprint 2rX.v:2110.02719, 2021.
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q- learning algorithms. arXiv preprint arXiv:1902.
2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning.
arXiv preprint arXiv: 106600660000 2021.
11

---

Published as a conference paper at ICLR 2026
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. corr abs/18 abs/1812.02900 (2018). arXiv preprint 11X.iv:8110029000 2018.
Scott Fujimoto, Pierluca D' D'Oro, Amy Zhang, Yuandong Tian, and Michael Rabbat. Towards general-purpose model-free reinforcement learning. arXiv preprint arXiv:2501.16142, 2025.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International conference on machine learning, pp. 2170-2179. PMLR, 2019.
Grauman, Westhury, Byrne, Z. Chavis, Furnari, Girdhar, J Hamburger, H Jiang, M Liu, X Liu, et al. Ego4d: around the world in 3,000 hours of egocentric video. arxiv 2021. arXiv
preprint 1rX.v:2110.07058 2021.
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning.
eprint arXiv: 2006.07733. 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint UrXiv:1801.000 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 2019.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04900 2022.
Junlin He, Jinxiao Du, and Wei Ma. Preventing dimensional collapse in self-supervised learning via orthogonality regularization. arXiv preprint arXiv:2411.00392, 2024.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:21 10.0661,,, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779 2020.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine.
Dr3: Value-based deep reinforcement learning requires explicit regularization. arXiv preprint arXiv:21 2021.
Yitang Li, Zhengyi Luo, Tonghe Zhang, Cunxi Dai, Anssi Kanervisto, Andrea Tirinzoni, Haoyang Weng, Kris Kitani, Mateusz Guzek, Ahmed Touati, Bfi--zero A promptable behavioral foundation model for humanoid control using unsupervised reinforeement learning. arXiv preprint arXiv:2511.044 1.04131, 2025.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi- perton linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851-866. 2023.
Tyler Lu, Dale Schuurmans, and Craig Boutilier. Non-delusional q-learning and value-iteration. Advances in neural information processing systems, 31, 2018.
Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng Xu. Universal humanoid motion representations for physics-based control. arXiv preprint arXiv:2310.04582. 2023.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implici pre-training. arXiv preprint 2tt::2210000000,0 2022.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 12601, 2022.
12

---

Published as a conference paper at ICLR 2026
Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric-aware abstraction. arXiv preprint arXiv:2310.0888 08887, 2023.
Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. arXiv preprini arXiv:2402.15507, 2024.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), 4144):1-17 2022.
Matteo Pirotta, Andrea Tirinzoni, Ahmed Touati, Alessandro Lazaric, and Yann Ollivier. Fast imitation via behavior foundation models. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
Martin I. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self self-predictive representations. arXiv preprint 1XXx:22007005929 2020.
Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. arXiv preprint arXiv:2302.08560) 2023.
Harshit Sikchi, Siddhant Agarwal, Pranaya Jajoo, Samyak Parajuli, Caleb Chuck, Max Rudolph, Peter Stone, Amy Zhang, and Scott Niekum. Rizero: Direct policy inference from language without in-domain supervision. arXiv preprint arXiv:2412.05718. 2024.
Harshit Sikchi, Andrea Tirinzoni, Ahmed Touati, Yingchen Xu, Anssi Kanervisto, Scott Niekum, Amy Zhang, Alessandro Lazaric, and Matteo Pirotta. Fast adaptation with behavioral foundation models. arXiv preprint WrXY:22504.0786 2025.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 000900,0000,,, 2018.
Chen Tessler, Yifeng Jiang, Erwin Coumans, Zhengyi Luo, Gal Chechik, and Xue Bin Peng. Maskedmanipulator: Versatile whole-body control for Iuco-mamipulation. arXiv preprint arXiv:2505. 2025.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 connectionist models summer school, pp. 255-263. Psychology Press, 2014.
Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, and Matteo Pirotta. Zero-shot whole-body humanoid control behavioral foundation models. arXiv preprint arXiv:2504. 1054, 2025.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. arXiv preprint arXiv:2103.07945 2021.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? arXiv preprint arXiv:2209. /4935, 2022.
David Warde- Farle- Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv:1811.11333 2018
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Physics- character controllers using conditional vaes. ACM Transactions on Graphies (TOG), 41(4):1-12, 2022.
13

---

Published as a conference paper at ICLR 2026
Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in rl: Learning representations with efficient approximations. arXiv preprint arXtv:8110005886, 2018.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprini arXiv:1911.13300 2019.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint mXXv:2201.13422) 2022.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 PROOF OF LEMMA 4.3
Lemma A.I. Given MDP M, let be any Ky-Lipschitz valued policy, M be the successor measure for π and M= be the corresponding successor measure on M, LRLDP(0, 9, w) upper bounds the prediction error in successor measure,
M ( CRLDP(0.g,w0 w)
Eg a at Ms,
1-7
Proof. Let's begin with Bg,and [IM (s, M (s, st]]] For fixed &
Ex [|M (8),, - M (8,88,,88)) S Ra 1p(s' = st) p(((s') d(s ((s+))|+
and (P V (7' - E ("
I LR + BB B B (am) (s') - (=((()))l++
a. ( o( ((s))) - (d
< LR + BBa B [V (5') 1'' (0(s')]1
#KyEEgande Di (-18, a), P(-1o(s),
(D is the distance metric used by - policy)
=LR+ + 7BB,and [Bx-P()),,a) [V (5') (o(''))
"Ky Ld
EE P( S CR + - V (8') - (d(s'))
KKr fud
S Ln + BBs [V (8) (d())))
TKvLa
=Ln + -Be,aaa (s, (d(s))),a, a(s ))++
JKvLa
This implies, (1 - Br (sM a. MM S Ln + yKvLa where Ca =
Be ip(s' -- - p(p(s') = 6(s ))/-
Taking expectation under s ~ p(st),
(1 (1-7) (a
E a h = =s) p(((s)) d(st))) + La La
= LR + -Kv Lad
This implies, Es (IM (s,a, - M a, st]] S Lotskyha CRLDP
1-y
A.2 PRIOR APPROACHES FOR REPRESENTATION LEARNING IN BFM'S
Prior work has often relied on complex objectives enable learning of and for BFMs. Forward- Backward (FB) (Touati et al. 2022) combine learning the state representation, ò with successor features, s and the policy. @ and vo are jointly learned represent successor measures for a class of reward-optimal policies. FB alternates minimizing the successor measure loss below jointly for th, á alongside policy improvement by optimizing Eq FB uses the following loss minimizing Bellman residuals to learn representations:
L( v)
+ E o l (() (b(+))))t+t)))))) n0(6', 2()))))) (II)
15

---

Published as a conference paper at ICLR 2026
HILP (Park et al. 2024) learns state representation o that are suitable to predict value function for goal-reaching which is subsequently used for zero-shot RL in the same way as RLDP. HILP parameterizes the value function to be V(s,g) = |1Ō(s) *(g))l and then minimizes:
()) Eg,,' grrp ( 1(s + g) (V VVs,
(12)
where & is an expectile loss (Kostrikov et al.
2021
The Laplacian approach (Wu et al. 2018) is action-independent and learns state representation using eigenvectors of graph-Laplacian induced by random-war walkk operator. The representation objective for Laplacian approach takes the following form:
(
Ex-p, A' (11) [llo(s) - -6s' + (
(13)
2
A.3
EXPERIMENTAL DETAILS
A.3.1 EXORL.
ExoRL (Exploratory Offline Reinforcement Learning) a benchmark suite that provides large, diverse offline datasets generated by exploratory policies across multiple domains (c.g., locomotion, manipulation, navigation). We consider three locomotion and one goal- goal-based navigation environments Walker, Quadruped, Cheetah, Pointmass from the Deepmind Control Suite TTasa iet al. (2018)). For offline training, we use data provided from the EXORL benchmark trained using RND agent. These domains explained further in table 3
All DM control tasks have an episode
length of 1000.
Observation/Action
Domain
Description
Type
Tasks
Reward
Dimension
stand walk run flip jump
Walker
two-legged robot
Locomotion
24/6
Dense
walk
Quadruped
four-legged robot
Locomotion
78/12
Dense
stand
walk
Cheetah
planar, 2D robot
Locomotion
17/6
run
Sparse
walk backward
backward
reach top left reach top right reach bottom right reach bottom left
Pointmass
navigation in 2D plane
Goal-reaching
4/2
Sparse
Table 3: ExoRL dataset summary. Domain is the environment name in the ExoRL benchmark. Description is natural language description of the agent embodiment and environment. Type refers the broad task category. Observation/Action Dimension refers the size of observation and action vectors from environment. Tasks refers to suite evaluation tasks provided in the ExoRL benchmark. Reward refers the density of non- zero reward signals from the environment.
A.3.2 SMPL 3D HUMANOID
SMPI. (Skinned Multi-Person Linear Model) is 3D parametric model of the human body that is widely used for character animation. has a 358 dimensional proprioceptive observation space that includes body pose, rotation, and velocities. The action space is 69 dimensional where each action dimension lies in [-1,1]. All episodes are of length 300.
A.3.3 D4RL
16

---

Published as a conference paper at ICLR 2026
D4RL (Datasets for Deep Data- Driven Reinforcement Learning)
Domain
Task Name
# Samples
(uu al. (2020) is an offline RL
hopper-medium
10
benchmark suite built on the v2 Open
hopper-medium-expee
2 x 106 108
AI Gym (Brockman al.] (20T6)
halfcheetah-meclee
Gym-MuJoCo
that provides standardized datasets and evaluation protocols across simulated and real-world tasks. We consider three simulated locomotion tasks Hopper, HalfCheetah,
halfcheetah settiraa-mmiiim-epeee
2 x 106
walker2d- walker2d-mediee expert
106
2 x 10"
Table 4: Gym-MuloCo tasks from D4RL..
Walker2D and two datasets medium and medium-expert. As described in Fu et al. (2020), the medium dataset is generated by online training Soft-Actor Critic (Haarnoja et al. (2018) agent, carly-stopping the training, and collecting million samples from this partially-traie trained policy. The "medium-experr" dataset is generated by mixing equal amounts of expert demonstrations and suboptimal data, generated via a partially trained policy or by unrolling a aafoorm-aa-randoo policy. Further details about these tasks have been provided in table 4 Episodes have inconsistent length depending on permination/rruncatiin with maximum of 1000.
A.4 REGULARIZED LATENT DYNAMICS PREDICTION
RLDP aims to learn state representation encoder such that latent state dynamics can be expressed as ((s') = g(((())))) a) where y is latent-state action encoder and w are some constant weights.
A.4.1 ARCHITECTURE
The architecture of the RLDE latent next-state prediction network is as pictured in figure 6
The state representation network Φ is a feedforward MLP with two hidden layers of 256 d-dimensional
A
units that maps state embedding. In our default RLDP architecture, the action is mapped to 256-dimensional space using linear network A. this section, make this distinction clear and use denote raw action input to the network and A to denote projection of action as input network. The outputs of these two networks are concatenated and passed through feedforward neural network that has two hidden layers of 512 units and d' d-dimensional output. The output of the g network is passed through a linear layer w. The final d-dimensional representations are spherically normalized.
((s)
g
W
(
During encoder training, the encoder map is unrolled perform next latent state prediction from current latent state and action as ((s') = g()(s),A) TTw. After encoder training, the encoder network frozen. To obtain latent state embeddings, the states are passed through the state representation network get (ss). The encoder architecture for RLDP is kept consistent across all methods and datasets.
Figure 6: Architecture of latent next-state prediction network in RLDP.
A.4.2 ABLATIONS
In this section, aim to examine the components of the RLDP state encoder to understand which parts of the method are crucial to learn representations that can maximize the span of reward functions represent optimal policies for.
17

---

Published as a conference paper at ICLR 2026
We pretrain state representation network & and policy using the ExoRL dataset generated with RND exploration policy and evaluate the performance in DMC environments cheetah, pointmass, quadruped, and walker.
Does orthogonality regularization matter?
Figure 7 shows the impact of changing orthogonality regularization while keeping constant encoding horizon (H = 5). The figure shows how the cosine similarity between latent states changes during encoder training for different regularization coefficients.
For regularization coefficient = O, the cosine similarity increases, indicating that all states are getting mapped to similar representations. For any regularization coefficient > 0, observe that the cosine similarity follows a steep descent, indicating that the states are being mapped to diverse representations. These results indicate that adding even small orthogonality regularization can reduce representation collapse significantly.
Encodinng Horizon Cheetah
Horizon Pointmass
Average Coo ne Sintirtty
64-
swet: 0.0
req coet:
sweft
reg. coef: 0:01
63-
6.20
1ug
mg coeft
mg zoett 10.0
mg zoeft 130.0
Average
mg coeft 100.0
02-
0.13
0.5
0.0
0.25
0.50
1.00
3.25
3:50
0.50
0.25
3.50 Represantation Leai Graclennt Stopo
0.50
1.5
2.00
Represontation Leamiin Gradicnt Stops
Encoding Hortzon Quadruped
Encoding
0.6
Rg 0.01
Avcragc Costre Sinitsrty
rg
ng
100.5
64
reg
01
Gradient
Learning Gradient Steps
Figure 7: Evaluating the impact of Orthogonality Regularization representations learned four environments: Cheetah (top left), Pointmass (lop right), Quadruped (bottom left), Walker (bottom right).
How does
encoding As discussed in section RLDP
horizon
impact
performance?
trained with the objective do
Orthogonality Regularization =
latent next state prediction from latent current state and action. This prediction can be done multiple steps into the future latent states (equation (6), depending the choice of encoding horizon H.
Average Retur
600
Average
Cheetah
500
Pointmass
Quadruped
Walker
In this section, we examine if the choice of encoding horizon impacts performance. To this end, we set the orthogonality regularization coefficient À 1.0 and sweep over encoding horizon (1, 5, 10, 20).
10
20
Encoding Horizon
Figure 8: Evaluating the impact of Encoding Horizon
The results are presented in figure
The average performance across en environments is relatively stable with a small dip at H = 10, indicating that encoding horizon does not significantly impact performance. For our experiments, we use encoding horizon H = 1 or H = 5 depending on the setting. Specific encoding horizon
18

---

Published as a conference paper at ICLR 2026
We do not choose higher encoding horizon H 20 despite comparable performance in figure because higher encoding horizon can result in slower encoder training. This is because each additional future state prediction involves forward pass through the encoder network.
setting for each experiment is discussed in section A.5
What is important for the encoder architecture?
In this section, we aim to ablate components of the encoder map to understand which factors contribute to RLDP's performance. For this setting, fix encoding horizon H =5 and orthogonality regularization coefficient = 1.0.
We focus two components of the encoder architecture linear layer A and spherical normalization SN on
We compare the complete RLDP encoder network with its variations a) RLDP w/o SN where spherical normalization is replaced with an identity mapping; b) RLDP w/o A where A is replaced with identity mapping: c) RLDP w/o SN & where A is also replaced with an identity mapping.
The results are shown in table 5 Although per-task results are variable, the full RLDP encoder delivered the strongest average performance on all four domains. Removing spherical normalization lowers returns and increases variance on most tasks and removing A also degrades performance. There are isolated wins for all variants, but these do not impact the domain-level results that favor the full RLDP encoder network. Thus, both SN and A contribute meaningfully to representation learning.
A.4.3 WHAT DO THESE REPRESENTATIONS LOOK LIKE?
To qualitatively assess the learned state
Task
SLD2
KLDP w/o SN
RLDE wlo A
w/o SN &
representations, we
29..40.1.7.33
860.74.162.47
810.79. 100.90
S81 66..69.16.92
use the Pointmass environment, where
Walker
Run
334.26-1:49.69
324.04116.33
290.78:126.52
276. 47::021177.21
Walk
779.77::137.16
728.29:::4300
715.83-192..8 43
583.60 26
Flip
492.94:::22799
501.59:::5.04
477.95 477.955+37..m 88
447.73:4:3.59
we uniformly sampled 10,000 equidistant states from the underlying state space (figure (a)).
Average(*)
624.34
603.66
573.84
547.33
Run
157.12:129.92
84.99 84.99:167.31
115.25=114.1
118667432.67
Cheetah
Run Backward
170.52::15.30
193.69:1:0.100
192.20:+42.07
156.566245.00 98.98
Walk
$92.92-t:104.66
387.50--2444.7
526.02-1:52.89
559.82=177.29
Backward
821.5111:50.22
838.12::145.37
133.29:173310
668.461186.17
We initialize representation encoder and pass these states through the encoder get latent embedding before training (figure (b)).
state
Average(*)
435.52
376.08
417.44
375,88
ruadruped
794.94:1:3..5
661.73:195775
518.61
687. 187.43-155.33
Run
457.41=:4.70
378.9771148.4
358.55+453.61
475.07=45.666
Walk
465.40=t:85.29
$19.39 119.39::251.. 2.111
384.92::19..
575.32=120.82
Jump
733.32:1:55.30
193..811133.81
319.18:1:55.18
$10.55=151.18
Average(*)
612.77
514.02
395.34
562.09
Top Left
890.41:200.9
492.133+41.44
186.19
890.89:
We then train two encoders with different losses: a. 0.0 in equation and train using only latent state prediction loss (figure{(c)). b. we set
Pointimass
Top Right
795.47:121.10
728.72:+122.999
809.64::1.23
799.599 19.44
Bottom Left
205.17:1:0.44
683.12:176.22
730.74:163.72
42-16681
we set X
Bottom Right
193.38:::167.68
22.54::39.04
206.59
178.77=130.17
Average(*)
671.11
$15.02
$47.62
583.78
Table 5: Study of encoder architecture. Cells show mean + std over 4 seeds; boldface indicates the highest mean per task.
X = 1.0 in equation & and train using latent state prediction loss and orthogonality regularization.
We project all these embeddings into two dimensions t-distributed Stochastic Neighbor Embedding (t-SNE). This visualization highlights the geometric structure captured by the representation and provides intuition about how the encoder organizes states in latent space.
The results in SeureeIshon that training an encoder using only latent state prediction loss (figure (c)) is ineffective at capturing the layout of the environment and maps different states to similar latent representations. Using both latent state prediction loss and orthogonality regularization enables the encoder to better capture the environment layout (figure(d)).
19

---

Published as a conference paper at ICLR 2026
(a) Pointmass observations
(b) Latent states before encoder training
(c) Latent states after training (prediction error)
(d) Latent states after training (prediction error orthogonality)
Figure i-SNE visualizations of state features in Pointmass. Each panel shows the 2D projection of 10,000
uniformly sampled states.
A.4.4 FUTURE DIRECTIONS: EXTENSION REAL-WORLD EMBODIMENTS
RLDP presents simple, stable and performant approach to train behavior foundation models in applications like robotics. An agent can promptable obtain low-level actions with such BFM. Recent works (Tessler et al. 2025 Li et al. 2025) have made promising attempts to extend BFM algorithms real-world domains and prior works have made possible to prompt BFMs with language and videos (Sikchi et al., 2024) which can be more intuitive interface for humans than reward functions. We believe the simplicity of this method and stability across hyperparameter choices, as demonstrated in table 5, makes it promising candidate for real-world embodiments.
A.5 IMPLEMENTATION DETAILS
In this section, we discuss the implementation details of all the methods and experiments described in the paper.
A.5.1 OFFLINE ZERO-SHOT RL
We use the same architecture for forward and policy networks presented in (Touati et al. 2022) for all representation learning methods.
The forward network F(s, a, a) has two parallel embedding layers that take in (s, a) and (s, 2) independently using feedforward networks with a single hidden layer of 1024 units, projecting to 512 dimensions. Their outputs (each with one hidden layer of 1024 units), which output a d-dimensional vector.
concatenated and passed into two separate feedforward heads
The policy network (s) has two parallel embedding layers that take inputs and (s, and embeds them similar to the forward network (one hidden layer of 1024 units mapping to 512 dimensions). The outputs of the embedding layers concatenated, and then passed into another
20

---

Published as a conference paper at ICLR 2026
Single-hideen feedforward network (1024 units) to produce action vector of dimension dA- final Tanh activation ensures that actions lie in the space [-1, 1-1
For results in table
For all methods, the backward representation network (ss) is implemented as feedforward neural network with two hidden layers of 512 units each, mapping state to a 512-dimensional embedding.
For results in t
For RLDP, wC sweep over representation dimensions (64, 128, 256, 512, 1024) and report the results for the dimension that achieves the highest average performance across all tasks within each environment.
For FB, PSM, HILP, and Laplacian, the representation dimensions previously identified as optimal for each respective method.
A.5.2 ONLINE ZERO-SHOT RL
Results for oracle baseline, FB-CPR, taken from Tirinzoni et al. (2025) where the model was trained for 30M environment steps and averaged across five seeds.
For the offline representation learning methods (HILP, PSM, FB, RLDP), the backward representation network Bss) follows the architecture of the backward network FB-CPR. It is 2-layer MLP with 256 hidden dimension that maps state a 256-dimensional embedding. We train this for 2 million timesteps on a dataset provided by Tirinzoni et al. (2025), which is generated by online training an FB-CPR agent for 30 million environment steps and saving the final million steps. The RLDP representations are trained with encoding horizon
We integrate the learned representation network into an FB-CPR agent train the forward and policy networks. This training is performed online for 20 million environment steps where no updates are performed on the representation network.
A.5.3 Low COVERAGE DATASETS
For offline representation learning methods (HILP, PSM, FB, RLDP), the backward representation network Bss) a feedforward neural network with two hidden layers of 256 dimension that maps state to 512-dimensional embedding. The RLDP representations horizon
trained with encoding
The forward network Fss, 2) and policy network r(s, =) follow the same architecture FB, We introduce an additional loss term for training the policy network that resembles TD3+BC (Fujimoto & Gu 2021 The policy improvement loss is defined
LP(mz) = >1b(8, + (T2(8) - a)²
(14)
where
A 1 Σ (m)aa)) Q(s;,
a
N
Following Fujimoto & Gu
2021
we sel a = 2.5
A.6 ALGORITHM
In algorithm I present the full algorithm for pre-training and inference of BFMs. In pretraining, we present RLDP representation learning as well as successor measur estimation and policy learning.
21

---

Published as a conference paper at ICLR 2026
Algorithm
Require: Offline dataset of trajectories D.
Require: Randomly initialized encoder di successor-measume model yb, actor m.
Require: Representation-learinng steps Nepr. total steps N.
Part I: Pretraining (offline)
2: for learning step 1, N do
3: if n < Niepr then
4:
Sample segment batch T = (56.11 o 1B
5:
16 = (())). Mi+1 - g(i at) w
La(d,g, w) Brindo > 1ij (0(())))) |
6:
7:
(o) = By (0s) ((55)]
8:
Lio, g, w) + Lo(d, g. w) + +d)
9:
Update 0, 9,
10:
else
11:
Sample transitions {(s, a, s', done)] ~ D
12:
Sample ~ Uniform Mix (random prior + goal-encoded)
13:
Policy Evaluation:
14:
Lean (v) from Equation 10
16:
Policy update:
a (Ts,
18:
Q = as,
19:
π T as Q (s)
20: end if
21: end for
22:
23: Part II: Inference (reward-basee task embedding)
Require: Task specification for the test task (e.g., name, parameters).
24: Set up the task-specific reward function Tisk (s) using the environment's reward routine
25: Sample transitions {(St, ai, s D
26: = = N Σ (8 (x1)
A.7 ADDITIONAL RESULTS
This section details additional experiments conducted evaluate RLDP against baseline methods, visualize the successor measures learned by RLDP, and study the effect of different loss components on representation learning.
A.7.1 OFFLINE ZERO-SHOT RL IN DMC
In table 6 we compare the returns for the representation dimension found to the best for the baseline methods and the RLDE. Across all methods, RLDP fares competitively to baselines that employ complex strategies such as FB, PSM to learn representation optimizing for successor measures across the environments despite its simplicity.
22

---

Published as a conference paper at ICLR 2026
Task
Laplace
FB
HILP
PSM
Stand
243.70::11..4
902.633::38.44
607.07++165.2
872.61:138.81
877.69-1:55.3
Walker
Run
63.6543LL02
39.276=311.9
107.84+34.24
19,46
324.85+:54..7
Walk
190.53+168.4
877.10=81.05
399.67
891444-46.81
790.94+67..55
Flip
48.73:: 17.66
206.22::16..2
277.95::59.61
640.75:1:1188
491.644::37.00
Cheetah
Run
96.32-4:5560
587.59
68.22:4:7.08
244.38:180.00
236.31+420.75
Run Backward
106.38=:29.40
14.99
37.99+
296.44:::0.14
322.08:+:39.28
Walk
109.1511-6606
799.831167.51
118.30-1168.444
984.21-1:.49
895.31
Walk Backward
654.29:219.81
980.76:1:2.32
249.611:236.29
979.011=7.73
984.76=10.85
Pointmass Quadruped
Stand
854.50:141.44 .47
740.05: 107.15
409.5441.7.55
842.86.182,
794.9411:3.25
Run
412.98.154.03
205,44.1:47..
431.77.1.44.9
457.411174.70
Walk
494.56:162.49
563.57:::53.22
86..544:86667
603.97::3.67
465.40=:185.00
Jump
642.84:::14.15
1511288::177.33
325.511:33.06
596.37:1:9.23
733.32-1:55.00
Top Left
713.46:1:58.99 90
897. 897.83:+35579
144946=t12.94
83143::66951
890.41-:00:79
Top Right
581.1442214.9
274.95=197.99
96.04++166.34
730.27+:5.10
795.47:+21.10
Hottom Left
689.05
$17.23:::00.66
192.34-177.68
451.38:13.466
805.17::20.44
Bottom Right
21.29:::2.54
19.37=#33.44
0.17=00.29
38.49:1:8.40
193.38 167.63
Table 6:
Comparison of zero-shot offline RL performance between different methods. Entries in bold
within one standard deviation of the per-task best mean (i.c., p4 2 p o²) aggregated over seeds.
A.8 TRAINING USFAS ON TOP OF RLDP REPRESENTATIONS
In table we examine the impact of directly learning Universal Successor Features top of RLDP representations. Typically for offline zero-shot RL on RLDP representations, we use loss equation I0 update the critic network. To train USFAs, we use the following loss:
(USFA( Bs,a,'' palls//ssstllllllo pl(viss, a, 2) - [o(s) + rb(s' n (s)),
(15)
We find that across a wide range of control tasks, training a USFA module on top of RLDP's state representations does not consistently outperform directly using successor measure loss TO for policy evaluation.
Learning USFAs on RLDP representations
RLDP
Stand
890-40::-27.33
854.84=641.50
Walker
Run
334.26:::49.66 69
304.89-:33115
Walk
137.77
665.98 6.64
Flip
492.94:1-2..79
497.77=170.80
Critic
Run
157.124429.92
139.00=2672
learned using successor measure loss achieves overlapping-bess performance on most Walker, Quadruped, and Pointmass tasks, while USFAs occasionally match slightly exceed Cheetah behaviors. Overall, these results indicate that RLDP'S learned representations capture most of the structure required for effective zero- shot generalization using either loss. The critic trained with successor measure loss typically achieves the strongest overall performance.
Cheetals
Run Bockward
170.52+t:5.30
172.52+:5.77
Walk
592.92:1104.00
647. 647.35
Walk
821.51:1.00.2
$00.021
Pointmass Quadruped
Stand
794.94-1-33.25
294.09-1:65.00
457.41:174.70
138.42:t:21.54
Walk
465.40:1185.29
166.661111..55
certain
Jump
733.32-4:55500
220.66:266.59
Left
890.4111:60.79
723.80
Top Right
795.47--21.10 10
723.79::57.89
Bottom Left
805.17:4-20.44
753.66:t:16. 16.59
Bottom Right
193.38:217.66
94.39
Table 7: Comparison (over 4 seeds) of zero-shot RL performance when equation I0 is used train the critic and when Universal Successor Features are trained on top state features. For fair comparison, we set RLDP representation dimension d' 512 for both methods.
A.8.1 ONLINE ZERO-SHOT RL
FB-CPR is an off-policy online unsupervised RL algorithm that introduces latent conditional- discriminator in the form of Conditional-Polie Regularization to output policies close unlabeled demonstration dataset M. The results for FB-CPR are as reported in Tirinzoni et al.
2025
In table 8 and figure 10 we provide the full suite of results on 45 SMPL Humanoid task for all baseline methods, RL.DP, and the oracle method FB-CPR.
23

---

Published as a conference paper at ICLR 2026
metric
FR-CPR (Oracle)
FB
PSM
HIPF
R:DP
craw)-0---0d
191.75 = 43.00
26.06 = 39.43
38.38
14.73
52.31 = 23.15
86,48
45.87
craw1-0.400-u
15.90
saw-l0044---
19.00 4.00
3,05 1 3.65
6.52 1 1.27
8.97 1 5.62
11.21 1: 4.72
cmwl--0.-2.u
0.64
299
cmw-1-0.500-d
131.13 64.97
43.27 + 34,06
46.17 13.56
52 4 27.82
55.82 + 18.40
crw.1-0.5-0-U
101.92 = 16.39
4.04 + 5.87
4.18
21.14 24.26
20.22
30.91
craw1-0.5-2-d
22.93
4.10
5.44 1.79
8.64 1.44:440 4.21
5.69
2.18
craw1-0.5-2-U
15.81 + 6.10
0,94 = 0.99
0.77 - 0.70
2.67 + 1.03
2,95 3.28
crouch-0
76.25
headstand
41.27 1: 10.20
0.00 1: 0.00
0.00 1 0.01
0.11 0.16
2,63 1.99
jump-2
21.21
27.89 1.66
liconground-don
193,50 18.89
35.41 = 26.08
63,87 26.68
69.79 + 27.02
74.69 30,03
uponggroudd-up
193.66 33.18
20.83 12.70
13.92 13.92 5.54
30.81 + 1.37
54.38 31.06
move-ego-90-2
210.99 6.55
207.47 9.92
179.67 49.64
196.81 40.36
178.82 45.10
move-cgo-9004
202.99 9.33
16184 12.65
102.35 35.15
102.98 + 40.47
99.96 32.58
egw--ego-D-0
move-ego-0.2
260.93 5.21
87.46 +: 21.99
252.75 1: 15.03
260.35 2.58
250,92 1 6.75
move-ego-O-4
235.44
133.47
233.02 14.75
201.90 38.55
move-cgo-180-
227.34 = 27.01
232.14 20.35
141.56 + 32.41
139.12 83.74
222.83 = 28.29
move-ego-180-
205.54 14.40
109.04 27.89
71.22 +: 19.98
53.37 25.65
81.92 29.90
move-ego-90-2
210.99 6.55
217.16 26.35
214.64 +..64+37..00
178.96 45.28
33.90
move-ego-90-4 90-4
202.99 9.33
154.20 + 41.82
104.73 20.95
102.51 + 63.37
160.31 = 37.02
move ego
90.2
nawe-ego-low-O..
215.61 + 27.63
168.33 5.95
150.34 1: 62.07
188.29 + 49.41
133.68 52,10
move-cgo-low-0-2
207.27 58.01
82.66 - 20.55
73.60 + 49.86
104.77 + 23,00
66.84 44,92
move-ego-low- 1oo-cgooloo--88-22
65,20 32.64
$2.38 + 27.67
46.28 + 22.28
43,90 39.86
28.71 12.52
move-egoo low 90-2
222.81 21.94
100.75 1: 39.27
53.20 1: 21.26
85.54 82.04
63.19 42.99
muscarms-I-h
71.90
raiseamms-h b-1
167.98 1 82.03
226.33 1: 35.55
90.57 1 68.37
82.42 + 43.38
201.33 + 87.51
misearms h-m
1:
miscarms-I-h
243.16 19.18
255.41 + 1.55
128.56 + 63.06
136.49 + 85.25
233.82 27.26
raiscarms-i-l
270.43 0.37
251.82 9.70
260.48 3.52
258.50 6.07
39.87 34.16
massearms---n
97.66 81.17
135.05 1: 80.31
254.91 3.78
91.49 46.58
217.42
39.30
misearms m-h
75.05 1: 69.32
79.25 31.99
41.58 1 13.58
126.62 1. 80.07
107.70 1: 79.18
misearms-m-I
134.83 70.28
218.22 46.82
173.28 72.83
155.21 71.93
220.67 50.89
miseaamss m.m
87.25 1: 98.42
179.60 1: 74.63
109.47 91.62
82.36 1. 38.59
211.30 48.89
notare-x-5-0.8
0.29
rotate-x-5-0.8
7,42 5.69
2,55 = 1.29
0.53 + 0.43
0.32 + 0.28
6,43 = 3.15
rotate-y-5-0.8
199.08 51.78
5.87 3.63
2,13
1.04 104+0.111 0.11
8.18 4.71
217.70 1: 43.67
4.86 1.44
1,58 0.44
0.89 1 0.13
14,03 12.12
mit0te77-5008
124.95 + 17.61
0.72 4: 0.79
0,2 1.200.300 0.30
0.31 0.23
17.09 9.10
95.23 15.75
1,71 1.67
0.39
0.38 138-:0.2
0.66 0.76
sitonground
199.44 1: 22.15
5,88 1. 4.75
27.39 1: 22.19
26.12 21.69
97.88 34.91
split-0.5
232.
12.64
34.31 32.98
87.22
split-1
117.67 61.27
6.80 = 9.14
6.12 7.17
6.13 5.72
13,02
16.90
Table 8: Comparing (over 4 seeds) FB, PSM. HIIP,, RLDP performance SMPL Humanoid. FB-CPR (online
oracle baseline) results from Tirinzoni al. (2025)
Bold indicates the best across methods.
Methods' performance as % of online oracle agent across tasks
PSM
RLDP
HILP
250
Performance
200
150
100 Oradle 100%
cant-0.400
caul-04-2u
caul-o
S:24
Jnm
moor cgp 80-3
moverese 180-2
move opo 180-4
moverss 90-2
moxe ego 90
move ago aegeoosee move 1ow.-ooo0o0
0.0
move 1ow-00
90-2
move maleernnn
ratcaam
riscamo
lm
move
Figure 10: Evaluating offline representation learning methods using online oracle policy in high dimensional
3D humanoid. Solid lines shows mean performance across tasks for each of the method.
A.9 FULL RESULTS ON D4RL
Table shows the normalized average returns of RLDP and baseline methods six low-coverage D4RL environments over 10 seeds. RLDP achieves the best mean performance on 5/6 tasks. Using Welch's t-tests with Holm correction, RLDP significantly outperforms FB on all tasks and
24

---

Published as a conference paper at ICLR 2026
Task
FB
PSM
HILP
RLD2
halfcheetah- seafeheetah-mediummmm
www.mmm-eeeeet-eye.....ee expert-v2
12.17:111.7
49,92.22.89
68.4748.20
86.03:18.36
hulfcheetah v2
39.27:18.71
42.64:10..4
43.85:11499
19,98:11.93
hopper- medium expert-y2
19.64-19447
18.19
hopper-medium--2
43.75:26.65
33,49=26.71
42.19:4.022
44.93=13.00
60.17:228.4
79.32=41.16
93.72-221.12
103.87-:3.31
maulker2d-mediumm
43.72:424.95
55.70=22.01
16.34115.08
83.83:12.66
Table 9: Normalized returns comparing FB, PSM, HIIP,, and RLDP in low-coverage setting. RLDP shows
significant gains over approaches that rely explicit Bellman backups for representation learning. Table shows mean std 10 seeds; Boldface indicates the highest use per-seed returns Welch's t-test and Holm correction; cases where the bolded method is not significantly better than the runner-up up are discussed in the text..
return per environment. Statistical comparisons
significantly outperforms PSM 5/6 tasks. RLDP's gains over HILP are statistically significant are not
on 3/6 tasks, while differences on pepper-medium-expeee and Balkkerddmmdiumm-caa statistically decisive with 10 seeds due high variance. On Hopper-medium, HILP has the highest mean but the HIIP-RIDDP difference is not significant under Welch's 's test, indicating comparable performance under seed variability.
A.10 VISUALIZATIONS OF LEARNED SUCCESSOR MEASURES
We used four room gridworld (as used in Touati & Ollivier 2021 Agarwal et al. (2024)
to plot the successor measures learned by RLDP. We collect a dataset all transitions and run RLDP with horizon to learn representations é. successor features v and policy Note that any policy parameterized by latent produces a successor measure parameterized by MM* at,,a,s)) = (sss, a, "0,
We have plotted the observed successor measures: M (su, ag. )) where fix Su and ag for a few different in figure [T] We have fixed 8p to the corner state and ag to action:
right. We have plotted the policy for visualizing the policy represent by 2.
A.11 EVALUATING THE EFFECT DIFFERENT LOSSES ON REPRESENTATION LEARNING
In table 10 we ablate the objectives used to learn representations and compare (a) Bellman-styy loss used FB, (b) latent prediction loss used in RLDP, and c) orthogonality regularization, that both FB and RLDP use. We evaluate these loss objectives individually and combined with unit scaling.
CBellmaa 1)) -Ee,,, (
+ Eg P ((*()) - -)))
(16)
2
Lonto = E ( d(s')l
(17)
#
Pprediction () = Eydno
Σ t=
|
क( x)
hg = (( = =0(hi, at) )
(18)
The results in table 10 highlighh that the interaction between different objectives matters much more than any individual Toss in isolation. Across all three expert datasets, we find that a combined objective of prediction loss and orthogonality regularization yicld the largest returns. In addition, adding Bellman backup to this objective with unit scaling results in inconsistent final returns.
Overall, these results support the design choice of using latent prediction with orthogonal regularization as the primary representation learning objective in low-coverage settings due to its peghhperformnnc and robustness across multiple settings.
We leave more exhaustive study of this setting to future work, including evaluating these objectives wider set of environments and exploring principled scaling for the loss components.
25

---

Published as a conference paper at ICLR 2026
(c) Figure Visualization successor measures A (So, Cg, s*) randomly sampled (a) and (b); and goal-conditioned (c) and (d).
(d)
26

---

Lonbo Published as a conference paper at ICLR 2026
Environment
Returns (mean + std)
halfcheetah medi expert
Lomho
51.36 + 6.93
Cellmmm
64.42 7.59
Predditionn
19.00 1.34
Bellman + Prrediction + Lonho
89.45 3.81
LBtlian + Comm (FB)
55.46 7.75
Priediction Licto (RLDF)
88.55 + 6.31
hopper med' i um- expert
Lombo
56.13 : 3.40
L Celliaan
59.02 = 12.19
Prodiction
54.63 + 3.51
Dellmaan + Preediction Lonbo
49.25 + 11.90
Bellman + LOnbo (FB)
49.93 + 28.78
Prrediction Lonho (RIDPP
75.53 + 12.70
wa 26err2-meede um-expert
95.10 12.28
Bettinan
89.12 + 19.36
L Prediction
97.52 = 14.27
Dellman Levediction + LOnbo
27.10 + 22.51
Dellmman + COnbo (FB)
36.85 + 14.89
Priediation Coeho (RLDP)
101.30 3.92
Table 10: Results D4RL. expert environments across different combinations aggregated over seeds.
27


Reviews:
[{"review_id": "xSAF3pz358", "reviewer": "Reviewer_2m5w", "summary": "This paper introduces a novel representation learning objective for unsupervised RL, where rewards are not given in the data, focused on enabling downstream Behavioral Foundation Models to perform effective zero-shot RL. Their objective broadly encompasses two components: a next latent state prediction loss motivated by theory, and an orthogonal regularization loss motivated through empirical observation. Through a series of experiments in robotics simulators, they validate their method, and show that their simple framework is a strong baseline for learning BFMs in a variety of settings including offline zero-shot RL, online zero-shot RL, and low coverage datsets.", "questions": "- Aside from ease of implementation, what other benefits arise from the simplicity of the proposed methods? Can we also any significant improvements in compute costs or sample efficiency?\n- As remarked in [1], next latent state prediction is generally a strong objective for representation learning in RL. However, in some tasks, other objectives such as observation reconstruction can also be beneficial. Due to the similar nature of all the environments in the paper (robotics physics-based simulators), it is possible that the proposed objective is only a strong baseline in environments of that class. For example, [1] remarks that while observation reconstruction is not beneficial in MuJoCo tasks, it can be good in low-dimensional clean environments such as grid worlds. Have you considered the benefit of other simple and popular representation learning objectives, and possibly experimenting with other environments such as pixel-based environments?\n\n\n### Minor Comments\n- Typo in line 287: \"Lemma 4.33 implies that minimizes\".\n- Typo in line 370: \"Due to **the** exploratory challenge [...]\"\n- Typo in line 412: \"a class of **policies** to learn [...]\"\n- The text is not consistent with its paragraph titles. For example, paragraph titles are sometimes bolded with a period (e.g. lines 320, 347, 353, 368 ...), bolded with a colon (e.g. lines 385, 423, 431), or italicized (e.g. lines 441 and 448).\n- The text is not consistent with line breaks between paragraphs. There looks to be missing line breaks in line 368, 375, 385, 423, 430, 440. \n- Figure 3 does not show any variance statistics for the \"average return\" datapoints. It can be difficult to parse whether the stated improvement from the orthogonality regularization is statistically significant.\n- It could be benficial to further contextualize the results by adding one or two naive baselines that are not state-of-the-art. For example, what is the performance of zero-shot RL on a random representations?\n- The results in Figure 2 are difficult to parse both in detail and at a glance. It could be beneficial to aggregate these results by normalizing the scores and providing an IQM [2]\n\n\n### References\n[1] Ni et al., 2023 -  https://arxiv.org/pdf/2401.08898\n[2] Agarwal et al., 2022 - https://arxiv.org/pdf/2108.13264", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Their proposed method is well-motivated both theoretically and empirically, and the authors present strong experimental evidence for their framework relative to other works in its area.\n- The experiments in their paper suggests that their proposed method can produce competitive results in the zero-shot RL setting, all the while being both simple to understand and implement compared to existing methods. Thus, this work is an important entry point for new researchers in the field of BFMs for zero-shot RL, and an important reference for existing researchers as well.\n- Their method is further studied beyond final performances, as the authors provide ablation studies for almost all components of their framework in either the main text or the supplementary material.", "weaknesses": "- It would be nice to further contextualize RLDP with the existing representation learning methods, by either providing the exact objectives of the baselines used in the paper(Laplace, FB, HILP, PSM) or being more explicit about the different assumptions used. For instance, the authors also suggest that their method is simpler in implementation and lacks some assumptions made by prior methods such as a prior class of policies. Showing the objectives used in the other works would clarify these differences", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "34O576qqhx", "reviewer": "Reviewer_Eidy", "summary": "This paper introduces **RLDP (Regularized Latent Dynamics Prediction)** as a pretraining objective for zero-shot reinforcement learning.  \nThe main contributions include:\n\n1. Proposing RLDP, a method to pretrain state representations for zero-shot RL via latent next-state prediction with regularization.  \n2. Theoretically proving that training in the latent space can approximate training in the original state space, preserving the predictive power of successor measures.  \n3. Conducting extensive experiments under both online and offline settings, showing that RLDP achieves competitive performance.  \n4. Demonstrating the advantage of RLDP under low-coverage conditions, where other methods tend to fail.", "questions": "1. In Figure 1, the paper mentions a “mild form of collapse.”  \n   However, the cosine similarity range of 0.4–0.6 does not obviously indicate collapse. Why is this value considered high?  Providing reference baselines (e.g., expected cosine similarity for random embeddings) would make this claim more convincing.\n\n2. The paper evaluates RLDP on D4RL’s medium and medium-expert datasets.  \n   Why not also include medium-replay? Those datasets have broader coverage, which could provide further insight into RLDP’s robustness under high-variance data.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Comprehensive experimental settings, including both online and offline environments.  \n2. Theoretical guarantees that connect the latent training objective to successor-measure consistency.  \n3. Rich intuition and ablation studies that help interpret how RLDP improves representation quality.", "weaknesses": "There are no major flaws, but several aspects could be improved for clarity and consistency.  \n\n1. Structure and exposition could be better organized:\n   - The paper does not clearly specify which parameters are optimized for each loss.  \n     The training process is only briefly introduced around line 295, and it’s unclear which components receive gradients.  \n     A concise summary of the full optimization pipeline (e.g., which modules update under each loss) would make the method section more readable.  \n   - In lines 180–190 describing policy improvement, it is unclear whether a separate policy network is maintained.  \n     In Equation (4), π(s) is written as if it equals a Q-value—perhaps “max” should be “argmax”?  \n     Equation (5) is understandable but would be more precise if an expectation operator **E[…]** were added.  \n   - The variable $\\bar{M}$ is first defined in *Lemma 4.3* but appears earlier in Equation (3).  \n     This may indicate an input mismatch: it should likely take $\\Phi(s′)$ instead of $s′$ as argument.\n\n2. The claim in line 198—“these representations minimize the prediction error for successor measures for any policy”—is not directly demonstrated.  \n   Subsequent quantitative analyses focus on returns rather than direct prediction errors.  \n   Since this is primarily a representation learning method, showing explicit results for prediction errors (e.g., L2 distance between predicted and ground-truth successor measures) would greatly strengthen the argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rnogeu67pv", "reviewer": "Reviewer_Th7h", "summary": "This work presents Regularized Latent Dynamics Prediction (RLDP), a representation learning approach for zero-shot reinforcement learning that replaces complex, reward-dependent objectives with a simple, self-supervised latent dynamics prediction task. The method augments next-state prediction with an orthogonality regularization term that mitigates feature similarity and prevents collapse in the learned state representations. Empirical evaluations across multiple benchmarks—including DeepMind Control Suite, D4RL, and high-dimensional humanoid control—show that RLDP achieves performance comparable to or exceeding state-of-the-art Behavioral Foundation Model baselines such as FB, PSM, and HILP. In particular, RLDP maintains stable performance in low-coverage datasets.", "questions": "What challenges do the authors suspect when adapting RLDP to real-world domains?\n\nHow would the approach perform in settings where the dynamics themselves evolve or where the environment exhibits non-stationarity—common in human-in-the-loop or adaptive control scenarios?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper makes a simple yet effective case for introducing latent dynamics into zero-shot offline RL. \n\nRLDP achieves performance on par with or exceeding that of more complex methods, demonstrating that simple latent dynamics prediction can yield strong generalization.\n\nPolicy-independent formulation avoids instability from Bellman backups, leading to more reliable learning in challenging or low-coverage environments.", "weaknesses": "Evaluation is performed on simulated continuous control benchmarks. Real-world data or transfer is not evaluated.\n\nEmpirical results show comparable but not universally superior performance, suggesting that the method’s advantages depend on task characteristics and data diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HBen9SYw12", "reviewer": "Reviewer_GfJN", "summary": "This paper proposes to use regularized latent dynamics prediction as an objective to learn representations for behavior foundation models. The main idea is to learn a state representation using a latent dynamics prediction objective, where the encoded latent states $\\phi(s_t)$ are trained to regress the unrolled predicted latent states $h_t := g(h_{t-1}), a_t, h_0 = \\phi(s_0)$. This objective alone suffers from collapse (shown in Figure 1), so the paper proposes to use an orthogonal regularization to mitigate it. The state representation is then used as the backward feature in the Forward-Backward (FB) representation framework. The paper shows theoretically that optimizing the latent dynamics objective also reduces prediction error in successor measures (i.e. the FB objective). Empirically, they evaluate the method on the offline zero-shot RL benchmark ExoRL, where the method matches or outperforms other behavior foundation model baselines. Their method also achieves online zero-shot RL in a Humanoid environment and is capable of learning representations from low-coverage datasets. The key design choice of orthogonal regularization is justified by ablation experiments. Overall, the paper proposes a simple yet effective method for zero-shot RL.", "questions": "1. Can you explain the mismatch between theory and practice as outlined in Weakness 1?\n2. Can you try an experiment where you learn a Universal Successor Feature on top of the state features, and do zero-shot RL with linear regression (assuming the features linearly span rewards)? This can potentially substantiate the broad applicability of the learned representations.\n\nReferences:\n\n[1] Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado van Hasselt, David Silver, Tom Schaul. Universal Successor Features Approximators. ICLR 2019.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is simple yet effective, performing favorably to baselines across multiple settings while suffering less from feature collapse.\n2. The theoretical justification is sound.\n3. The paper conducts thorough analysis on the feature collapse phenomenon and the effect of orthonogal regularization.", "weaknesses": "1. The theory and practice are somewhat mismatched. Theory says that learning successor measure on top of the latent representations improves as the representation learning objective improves. However, in practice, the forward feature is learned from scratch, and only the backward feature is carried over. \n2. The method does not improve much compared to baselines in the ExoRL benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)