Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
Authors: Vlad Sobal
Wancong Zhang
Kyunghyun Cho
Randall Balestriero
Tim G. J. Rudner
Yann LeCun
Venue: NeurIPS 2025 poster ()
Keywords: self-supervised learning
representation learning
dynamic learning
offline rl
reward-free RL
goal-conditoined RL
zero-shot RL
Primary area: reinforcement_learning
TL;DR: In this paper, we systematically evaluate RL and control-based methods on a suite of navigation tasks, using offline datasets of varying quality.

Abstract: A long-standing goal in AI is to develop agents capable of solving diverse tasks across a range of environments, including those never seen during training. Two dominant paradigms address this challenge: (i) reinforcement learning (RL), which learns policies via trial and error, and (ii) optimal control, which plans actions using a known or learned dynamics model. However, their comparative strengths in the offline setting—where agents must learn from reward-free trajectories—remain underexplored. In this work, we systematically evaluate RL and control-based methods on a suite of navigation tasks, using offline datasets of varying quality. On the RL side, we consider goal-conditioned and zero-shot methods. On the control side, we train a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning. We investigate how factors such as data diversity, trajectory quality, and environment variability influence the performance of these approaches. Our results show that model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts and is more data-efficient, while achieving trajectory stitching performance comparable to leading model-free methods. Notably, planning with a latent dynamics model proves to be a strong approach for handling suboptimal offline data and adapting to diverse environments.

Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
Vlad Sobal*!
Wancong Zhang*
Kynghyun Cho
Randall Balestriero
Tim G. J. Rudner
Yann LeCun
New York University Genentech
Brown University
Universitty of Toronto *Meta FAIR
Website
n Code
Abstract
A long-standing goal in Al is to develop agents capable of solving diverse tasks across range of environments, including those never seen during training. Two dominant paradigms address this challenge: (i) reinforcement learning (RL), which learns policies via trial and error, and (ii) optimal control, which plans actions using a known or learned dynamics model. However, their comparative strengths in the agents must learn from reward-free trajectories-rcet remaii underexplored. In this work, we systematically evaluate RL and control-basee methods on a suite of navigation tasks, using offline datasets of varying quality. On the RI. side, we consider goal-conditioned and zero-shot methods. On the control side, we train latent dynamics model using the Joint Embedding Predictive Archi- tecture (JEPA) and employ it for planning. We investigate how factors such data diversity, trajectory quality, and environment variability influence the performance of these approaches. Our results show that model-free RL benefits most from large amounts high-quality data, whereas model-based planning generalizes better unseen layouts and is more data-efficient while achieving trajectory stitching performance comparable to leading model-free methods. Notably, planning with latent dynamics model proves to be strong approach for handling suboptimal offline data and adapting to diverse environments.
offline setting-
Introduction
How can we build system that performs well on unseen combinations of tasks and environments? One promising approach avoid relying on online interactions or expert demonstrations, and instead leverage large collections of existing suboptimal trajectories without reward annotations [16, 32, 53]. Broadly, two dominant fields offer promising solutions for learning from such data: reinforcement learning and optimal control.
While online reinforcement learning has enabled agents to master complex lasks-from Atari games [44], Go [60], to controlling real robots [49]-it demands massive quantities of environment interactions. For instance, OpenAl et [49] used the equivalent of 100 years of real-time hand manipulation experience to train robot to reliably handle Rubik's 's cube. To address this inefficiency, offline RL. methods [18, 33, 41] have been developed learn behaviors from state-action trajectories with corresponding reward annotations. However, these methods typically train agents for a single task, limiting their reuse in other downstream tasks. To overcome this, recent work has explored learning behaviors from offline reward-free trajectories [32, 52, 53, 67]. This reward-free paradigm is particularly appealing as allows agents to learn from suboptimal data and use the learned policy to solve variety of downstream tasks. For example, system trained on low-quality robotic interactions with cloth can later generalize to tasks like folding laundry [9].
* Equal contribution. Author ordering determined by coin flip.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

23 different datasets varying quality Limited layout coverage
various methous offline reward- free data
Evaluations desirable properties
Dynamtcc
HILP
GCIOL
PLDM
Generalizing to new layouts
X
X
feature divensity
Kandom policy tejjectories
Generllizing tasks
X
trajectories
Trajectory strtchingg
aatent dyromicc model leaming
Figure 1: Overview of our analysis. We test six methods for learning from offline swardd-free trajectories 23 different datasets across several navigation environments. We evaluate for six generalization properties required to scale to large offline datasets of suboptimal trajectories. We find that planning with a latent dynamics model (PLDM) demonstrates the highest level of gencralization. For a full comparison, see Table 1. Right: diagram of PLDM. Circles represent variables, rectangles loss components, half-ovals - trained models.
Optimal control tackles challenge differently: instead of learning policy function via trial and error, it plans actions using known dynamics model [7, 63, 64] to plan out actions. Since real-world dynamics are often hard to specify exactly, many approaches instead learn the model from data [20, 71, 77J. This model-based approach has shown generalization in manipulation tasks involving unseen objects [17]. Importantly, dynamics models can be trained directly from reward-free offline trajectories, making this a compelling route [16, 56].
Despite significant advances in RL and optimal control, the role of pre-training data quality on reward- free offline learning remains largely unexplored. Prior work has primarily focused on RL methods trained on data from expert or exploratory policies [22, 76], without isolating the specific aspects of data quality that influence performance. In this work, we address this gap by systematically evaluating the strengths and limitations of various approaches for learning from reward-free trajectories. We assess how different learning paradigms perform under offline datasets that vary in both quality and quantity. To ground our study, we focus on navigation tasks an essential aspect many real- world robotic systems where spatial reasoning, generalization. and trajectory stitching play critical role. While this choice excludes domains such as manipulation, it offers controlled yet challenging testbed for our comparative analysis.
Our contributions can be summarized as follows:
1. We propose two new navigation environments with granular control over the data generation process, and generate total of 23 datasets of varying quality;
2. We evaluate methods for learning from offline, reward-fire -fee trajectories, drawing from both re- inforcement learning and optimal control paradigms. Our analysis systematically assesses their ability to learn from random policy trajectories, stitch together short sequences, train effectively limited data, and generalize to unseen environment layouts and tasks beyond goal-reaching:
3. We demonstrate that learning a latent dynamics model and using for planning is robust to suboptimal data quality and achieves the highest level of generalization environment variations;
4. We present list of guidelines to help practitioners choose between methods depending on available data and generalization requirements.
To facilitate further research into methods for learning from offline trajectories without rewards, we release code, data, environment visualizations, and more at latent- prventtplanning caamingggithub.... github. io.
2 Related Work
Reward-free offline RL refers learning from offline data that does not contain rewards in task-agnostic way. The goal is to extract general behaviors from offline data to solve a variety of downstream tasks. One approach uses goal-conditioned RL, with goals sampled in similar manner as in Hindsight Experience Replay [1]. Park et al. [52] show that this can be applied to learn goal-conditioned policy using IQL, as well as learn hierarchical value function. Hatch et [29] proposes using small set of observations corresponding to the solved task to define the task and learn from reward-free data. Hu et al. [30], Yu et al. [79] propose to use labeled data to train a reward function, than label the reward-fire free trajectories. Zero-shot methods go beyond goal-reaching from

---

on Diverse Maps
Test on Unseen Maps
H
80
80-
Sucoess rate
=0
40
train nng mass
distribution
disribution
GCBC GCIQL
HIQL
PLDM
Figure 2: Left: We train offline goal-conditioned agents on trajectories collected in a subset of maze layouts (left), and evaluate held out layouts, observing trajectories shown the right. Only PLDM solves the task (see Figure for more). Right: Success rates of tested methods on held-out layouts, of training layouts. Rightmost plot shows success rates of models trained on data from five layouts, evaluated on held-out layouts ranging from those similar to training layouts to out-of out-of- distiriiibttinnn ones. We use map layout edit distance from the training layouts as a measure of distribution shift. PLDM demonstrates the best generalization performance. Results are averaged over 3 seeds, shaded area denotes standard error. See Figure I for more details PLDMM.
function of the number
offline data and aim to solve arbitrary tasks specified at test time. HILP [53] proposes learning a representation space such that the distance in that space is proportional to the number of steps between two states, similar to Laplacian representations [69, 70, 73]. Forward- Backward representations [67, 68] tackle this with approach akin to muccessor-features [5].
Optimal Control, similar RI., tackles the problem of selecting actions an environment to optimize given objective (reward for RI, cost for control). Classical optimal control methods typically assumes that the transition dynamics of the environment are known [7]. This paradigm
has been used to control aircraft, rockets, missiles [1 I] and humanoid robots [35, 57J. When the transition dynamics cannot defined precisely, they can often be learned [65, 71J. Many RL methods approximate dynamic programming in the context of unknown dynamics [6. 62]. In this work, we use the term RL to refer to methods that either implicitly or explicitly use rewards information to train a policy function, and optimal control for methods that dynamics model and explicitly search for actions that optimize the objective.
The importance of offline data has been highlighted in works such ExORL, [76] which demon- strates that exploratory RI data enables off off-policy algorithms to perform well in offline RI.; however, it only compares exploratory vs. task-specific data, without analyzing which data aspects affect per- formance. Buckman et [12] investigates the data importance for offline RL with rewards. Recently proposed OGBench [51] introduces multiple offline datasets for variety of goal-conditioned tasks; in contrast, we conduct more fine-grained analysis of how methods perform in top-down navigation under suboptimal data conditions and generalize to new tasks and layouts. Yang et al. [74] also study generalization of offline GCRL, but focus on reaching -it-----ditiitutionn goals. Ghugare et al. [24] study stitching generalization.
The Landscape of Available Methods
In this section, we formally introduce the setting of learning from state-action sequences without reward annotations and overview available approaches. We also introduce method we call Planning with Latent Dynamics Model (PLDM).
3.1 Problem Setting
We consider Markov decision process (MDP) M (S,A,, where S is the state space,
A is the action space, E P(S) denotes the initial state distribution, p E S x A + S denotes the transition dynamics (we only consider the deterministic case), and r S reward function. We work in the offline setting, where we have access to a dataset of state-action sequences D which consists of transitions (8o, 4n 81,... 11, 1 We emphasize again that offline dataset in our setting does not contain any reward information. The goal is, given D, to find a policy æ E x 2 > A, to maximize cumulative reward r., where the space of possible task definitions. Our goal is to make the best use of the offline dataset D to enable the agent to
+ R denotes the
3

---

Table 1: Road-map of our generalization stress-lesting experiments. We test 4 offline goal-conditioned meth- ods HIQI,, GCIQL,, CRI,, GCBC; zero-shot RI. method HII.P, and leared latent dynamics planning method PLDM. *** denotes good performance in the specified experiment, * ft denotes average performance, and *e denotes poor performance. We see that HILP and PLDM are the best-performing methods, with PLDM standing out as the only method that reaches competitive performance in all settings.
Property (Expesimen seciio)
HILP
HIQL
GCIQL
CRL
GCBC
PLDM
Transfer to new environment layouts (4.8)
*
*
***
Transfer to new task (4.6)
*
*
*
*
*
***
Data efficiency (4.3)
*
*
***
*
**
***
Best-case performance (4.2)
***
***
***
**
***
Can learn from random policy trajectories (4.9)
***
* Sr
***
*
*
Can stitch suboptimal trajectories (4.4)
*
solve a variety of tasks in given environment with potentially different layouts. During evaluation, unless otherwise specified, the agent is tasked to reach goal state Sq. so the reward is defined as rg (s) = I||s $g]. and Z is equivalent to S.
3.2 Reward-free Offline Reinforcement Learning
In this work, we study methods that solve tasks purely from offline trajectories without reward annotations. Reward-free offline RL methods fall into two categories: goal-conditioned RL and zero-shot methods that treat the task as a latent variable. We evaluate state-of-the-ar methods from both categories on goal-rea. and test zero-shot methods on their ability to transfer new tasks. The methods we investigate are:
GCIQL [52] goal condiconnd version of Implicit Q-Learning [33], a strong and widely- used method for offline RL;
HIQL [52] - a hierarchical GCRL method which trains two policies: one to generate subgoals, and another one reach the subgoals. Notably, both policies the same value function; HII.P [53]- -a method that learns state representations from the offline data such that the distance in the learned representation space is proportional the number of steps between two states. A direction-conditionn, policy is then learned to be able to move along any specified direction in the latent space;
CRL [19]- uses contrastive learning to learn compatibility between states and possible reachable goals. The learned representation, which has been shown to be directly linked to goal-conditioned Q-function, is then used to train goal-conditionss policy;
GCBC [23, 43] Roal-Conditioned Behavior Cloning the simplest baseline for goal-reaching.
3.3 Planning with Latent Dynamics Model
The methods in Section 3.2 are model-free, none explicitly model the environment dynamics. Since we do not assume known dynamics in classical control, we can instead learn dynamics model from offline data, similar to [46, 55], which propose a model-based method for goal-reaching using an image reconstruction objective.
We propose a model- model-based method named Planning with a Latent Dynamics Model (PLDM), which learns latent dynamics using reconstruction frreetiiionoeeeeeeee SSL objective and the JEPA architccture [39]. At test time, we plan in the learned latent space to reach goals. We opt for an SSL approach that predicts the latents opposed to reconstructing the input observations [3, 21, 26, 81] motivated by findings that reconstruction yields suboptimal features [2, 42), while reconstruction-fres representation leam- ing works well for control [27, 59]. Appendix G provides empirical support: features trained with reconstruction- based methods such as DreamerV3 [26] underperform in test-time planning.
Given agent trajectory sequence (8o, 00, .1...., 11, 8T), we specify the PLDM world model as:
Encoder: : E0 = 20 = hs()s)
(I)
Predictors: : a = fb ( Vk E (I....,K)
(2)
where at is the latent state predicted by predictor l: and is the encoder output at step L. When K 1, we train an ensemble of predictors for uncertainty regularization at test-time. The training
objective involves minimizing the distance between predicted and encoded latents summed over

---

Table 2: Performance of tested methods good- quality data and data with no trajectories pass- ing through the door. Values are average success rates (taaandard error) across 3 seeds.
goal
start
Good-quality data
No door-passing trajectories
Method
CRL
89.3 = 0.7
14.7 = 4.1
Figure 3: Left: The Two-Roome environment. The agent starts random location and is tasked with reaching the goal at another randomly sampled location in other room using 200 steps or less. Observations are 64 x 64 pixels images. Right: Examples of trajectories in the offline data. Red: cach step's 's direction is sampled from Von Mises dis- tribution. Blue: each step's 's direction is sampled uniformly.
GCBC
86.0 = 2.0
8.4 = 1.2
GCIQL
98.0 + 0.9
99.6 = 0,4
HILP
100.0 + 0.0
100.0 = 0,0
HIQL
96.4 1 1.3
26.3
5.6
PLDM
97.8 + 0.7
34.4 = 2.7
all timesteps. Given target and predicted latents Z, 2t E "INNN where S T is the model prediction horizon, N is the batch dimension, and D the feature dimension, the similarity objective between predictions and encodings is:
Siim
Zr,b112
To prevent representation collapse, we use VICReg- VICReg-inspired [4] objective, and inverse dynamics modeling [40]. We show a diagram of PLDM in Figure 1. See Appendix D.1.1 for details.
Goal-conditioned planning with PLDM. In this work, mainly focus the task of reaching specified goal states. While methods outlined in Section 3.2 rely on trained policies to reach the goal, PLDM relies on planning. At test time, given the current observation 8μ, goal observation sgr pretrained encoder he predictor fe, and planning horizon H, our planning objective is:
Vk {1. , K} at
z
ho(s,),
at =f
(2f a.-1))
K
H
1
(a (a, 80, 8g)
l1b((8g)
(
(5)
K
k=1
1=
H
d
Cuncertainty (a, So: 8z)
=
Var( (st k
(6)
1=
arg min minn(Cooui (a,80: 80; 8g) 3Cuncertainy (a, 80, (m.8g)))
Cgea is the goal-reaching objective and Cuncertainty penalizes the model from choosing state-action transitions that deviate from the training distribution, with > E [0. 1] as the temporal discount. This regularization resembles how GCIQL, HIQL, and HILP use expectile regression to learn policies that remain in-distribution with respect to the dataset [34]. See Appendix E for ablations on Cuncentannty
Following the Model Predictive Control framework [45], PLDM re-plans every interactions with the environment. By default, we use i = 1 for all experiments, making PLDM ~ 4x slower than the model-free baselines. The replanning interval can be increased accelerate MPC with only a minor loss in performance (see Appendix F). We use MPPI [72] in all our experiments with planning. note that PLDM is not using rewards, neither explicitly nor implicitly, and should be considered as optimal control method. We also note that to apply PLDM to a new task, we do not need to retrain the encoder he and dynamics fe. we only need to change the cost in Equation (5). We test this flexibility in Section 4.6, where invert the sign of the cost to make the agent avoid given state.
4 Every Method Can Excel but Few Generalize
In this section, we conduct thorough experiments testing methods spanning RL and optimal control outlined in Section 3.2 and Section 3.3. We evaluate navigation tasks where the agent is either point mass (Section 4.1, Section 4.8) or quadruped (Section 4.7). We generate datasets of varying size and quality and test how specific data type affects given method. See Table I for overview.
5

---

100
100
100
75
75
75
Success
50
50
50
25
25
25
0.05
0.10
20
10a
10"
10a
10
Fraction of non-random data
Sequence length
Dataset size
GCBC
GCIQL
HILP
HIQL
PLDM
Figure 4: Testing the selected methods' performance under different dataset constraints. Values and shaded regions means and standard over 3 seeds, respectively. Left: To test the importance of the dataset quality, we mix random policy trajectories with good quality trajectories (see Figure 3). the amount of good quality data goes to O, methods begin to fail, with PLDM, GCIQL, and HILP being the most robust ones. Center: We measure methods' performance when trained with different sequence lengths. We find that many goal- conditioned methods fail when train trajectories -utoo--distrbuttion
short, which causes fär-awa away goals become resulting policy, Right: We measure methods' performance with datasets of varying sizes. see that PL.DM and GCTQL are most sample efficient, and manage to get almost 80% success rate even with a few thousand transitions. See Appendix H for the analysis of statistical significance.
4.1 Two-Rooms Environment
We begin with navigation task called Two-Rooms, featuring a point-mass agent. Each observation Ty E [22x64x61 is a top-down view: the first channel encodes the agent, the second the walls (Figure 3). Actions a R2² denote the displacement vector of the agent position from one time step to the next, with norm limit of 2.45. The goal is to reach randomly sampled state within 200 steps. See Appendix C.2 for more details. This environment allows for controlled data generation ideal for efficient and thorough experimentation, while still not being too trivial.
Offline data. To generate offline data, we place the agent in random location within the environment, and execute a sequence of actions for steps, where denotes the episode length. The actions are generated by first picking a random direction, then using Von Mises distribution with concentration to sample action directions. The step size is uniform from 0 to 2.45. Unless otherwise specified, the episodes' length is T 91, and the total number of transitions in the data is million.
4.2 What Methods Excel In-Distribution with Large High-Quality Dataset?
To get the topline performance of the methods under optimal dataset conditions, we test them in setting with abundant data, good state coverage, and good quality trajectories long enough to traverse the two rooms. With 3 million transitions, corresponding to around 30,000 trajectories, all methods reach good performance in the goal-reaching task Two-Rooms (Table 2), with HIQL, GCIQL, HILP, and PLDM nearing 100% success rate.
Takeaway: All methods can perform well when data is plentiful and high-quality.
4.3 What Method is the Most Sample-Efficient?
We investigate how different methods perform when the dataset size varies. While our ulti- mate goal is to have method that can make use of a large amount of suboptimal offline data, this experiment serves distinguish which methods can glean the most information from avail- able data. We tried ranges of dataset sizes all the way down to few thousand transitions. In Figure 4, we see that the model-hased PL.DM and the model-free GCIQL outperform other model-free methods when data is scarce. In particular, HILP is more data-hungry than other model-free methods but achieves perfect performance with enough data.
Takeaway: PLDM and GCIQL are more sample-efficient than other methods.

---

4.4 What Methods Can Stitch Suboptimal Trajectories?
Can we learn from short trajectories? We vary the episode length T during data generation to test whether methods can stitch together short training trajectories to reach oong-horizon goals. In real-world scenarios, collecting long episodes is often difficult- especially in open-ended environ- ments- so the ability to learn generalizable policies from short trajectories is crucial. The hardest scenario of Two-Rooms requires the agent to navigate from the bottom left corner to the bottom right comer, and involves ~ 90 steps, meaning that with short episode lengths such as 16, the goal is never observed. To succeed, methods must stitch together multiple offline trajectories.
We create datasets with episode length of 91, 64, 32, 16, keeping total transitions number at million. Results in Figure (center) show that with the exception of GCIQL, goal- condiiiond model-free methods struggle when trained with shorter episodes. We hypothesize that these methods are limited by dynamic programming on the short transitions, which can be sample-ineficient for stitching many short trajectories to reach far-away goals. In contrast, HILP performs well by learning to follow directions in latent space - even from short episodes. Similarly, PLDMM can learn an accurate model from short trajectories and stitch together a plan during test time.
Can we learn from data with imperfect coverage? We artificially constrain trajectories always stay within one room within the episode, and never pass through the door. Without the constraint, around 35% trajectories pass through the door. During evaluation, the agent still needs go through the door to reach the goal state. This also reflects possible constraints in real-life scenarios, as the ability to stitch offline trajectories together is essential to efficiently learn from offline data. The results are shown in Table 2. We see that HILP and GCIQL achieve perfect performance, while PLDM performance drops but is still higher than the rest of offline GCRL methods. We hypothesize that HII.P's P's latent space structure enables effective stitching, while PLDM retains some performance due to the learned dynamics. With the exception of GCIQL, other model-free GCRL methods fail to learn to compose trajectories across rooms.
Takeaway:
When solving the task requires 'stitching'. HILP and GCIQL work great. The performance of PLDM drops, but is better than that of most offline model-free GCRL methods.
4.5 What Methods Can Learn From Trajectories of a Random Policy?
In this experiment, we evaluate how trajectory quality affects agent performance. In practice, random policy data is easy to collect, while expert demonstrations are often unavailable. Thus, algorithms that can generalize from noisy data are crucial. We create a dataset where actions are sampled uniformly at random, causing agents to oscillate near the starting point. this setting, the average maximum distance between any two points in trajectory is 10 (when the whole environment is 64 by 64), while when using Von Mises to sample actions as in the case of good quality data, it is ~ 28. Example trajectories from both types of action sampling are shown in Figure 3.
We see that HILP, GCIQL and the model- PLDM outperform the rest of goal-conditioned RL methods in this setting (Figure 4). We hypothesize that because random trajectories on average do not go far, the sampled state and goal pairs during training are close to each other. As a result, faraway goals become out of distribution, and GCRL methods struggle with long-horizon tasks when TD learning fails to bridge distant trajectories. In contrast, PLDM uses the data only to learn the dynamics model, and random policy trajectories are still suitable for the purpose. HILP learns the latent space and how to traverse it in various dimensions even from random data.
Takeaway:
When the dataset quality is low, HILP, GCIQL and PLDM perform better than other offline GCRL methods.
4.6 What Methods Can Generalize a New Task?
To build effective general purpose systems that learn from offline data, we need an algorithm that can generalize to different tasks. So far, we evaluated on goal-reaching tasks. In this experiment, we test generalization to a different task in an environment with the same dynamics: avoiding a chasing agent the same environment. We compare PLDM and HIIP, using models trained on optimal data from Section 4.2 without any further training. In this task, the chaser follows an expert policy along the
7

---

330
ArD dstanne 1R ch
[o 80
80
KChaser
Max chaser
Eploste
(b)
(c)
Figure 5: Zero-shot generalization to the chasing task. (a) In the chase environment, the blue agent is tasked with avoiding the red chaser. The chaser follows the shortest path the agent. The observations of the agent remain unchanged: wc pass the chaser state as the goal state. The agent has to avoid the specified state instead of reaching it. (b) Left: Performance of the tested methods on the chasing task across different chaser speeds, with faster chaser making the task harder. Baselines include agents that take action ("Zero") and random actions ("Random') ) (b) Right: Average distance between the agent and chaser agent throughout the episode when chaser speed 1.0. (c) Visualization ant-umaze environment. The 4-legged ant is tasked with reaching a randomly sampled goal within a u-shaped room.
shortest path to the agent, and we vary its speed to adjust difficulty. The goal of the controlled agent to avoid being caught. Goal-conditioned methods are excluded from evaluation, as they cannot avoid specific states by design. At each step, the agent observes the chaser's state and selects actions to maintain distance. In PLDM, we invert the sign of the planning objective to maximize distance in the latent space to the goal state. In HILP, defined as maintaining distance 2 1.4 pixels over 100 steps. The results are shown in Figure 5b. We also plot average distance between the agents over time in Figure 5b. We see that PLDM performs better than HILP, and is able to evade the chaser more effectively by maintaining greater separation by episode end.
invert the skill direction. We evaluate the success rate-
Takeaway: Assuming the environment dynamics remain fixed, PLDM can generalize to tasks other than goal-reaching simply by changing the planning objective.
4.7 Extending to a Hligher-Dimensione Control Environment.
So far, we have focused on environments with simple control dynamics over a point-mass agent, where actions are 2D displacement or acceleration vectors. We now investigate whether the same trend holds in setting with more complex control. We choose Ant-U-Mazz a standard state-based environment with 29-dimensional state space and 8-dimensional action space (Figure 5c). Solving this task with PLDM requires learning non-trivial dynamics that better resemble real-world control.
We collect dataset using a pretrained directional expert policy from Park et al. [51]. generating 5M transitions by resampling new direction every 10
CRL
GCBC
steps and adding Gaussian noise with standard devia- tion 1.0 to each action. For evaluation, the quadruped is initialized at the bottom left or right corner, with
Success rate
75
GCIQL
50
HILP
HIQL
25
PLDM
the goal at the opposite diagonal. Each method is evaluated on 10 trials.
50
250
500
As in 4.4, we test the methods' abilities stitch short training trajectories by using datasets with trajectory lengths 25, 50, 100, 250, and 500 (during evaluation, the start and goal are approximately 200 steps apart).
Sequence length
Figure 6: Success rates in Ant-U-Maze for agents trained on trajectories of varying lengths.
PLDM, HIQL, and HIIPP outperform other GCRL baselines in trajectory stitching, achieving 100% success rates while other methods fail with shorter trajectories. In contrast to the Two-Rooms experiments, HIQL outperforms GCIQL in this setting. We hypothesize that HIQL's hicrarchical structure may particularly benefit high-dimensional ant control: the low-level policy can manage fine-grained joint movements, while the high-level policy can govern overall navigation.
Takeaway: Planning with a latent dynamics model maintains good performance in a standard quadruped maze task, indicating promising generalization to higher control complexity.

---

4.8 What Methods Can Generalize to Unseen Environment Layouts?
In this experiment, we test whether methods can generalize new obstacle confgurations/layous a key requirement for general purpose agents, since collecting data for every scenario is infeasible. We introduce a new navigation environment with slightly more complex dynamics and configurable layouts (Figure 2). Building on top of Mujoco PointMaze, [66] layouts are generated by randomly permuting wall locations. Data is collected by randomly sampling actions at each step. Observation includes an RGB top-down view of the maze and agent velocity; actions are 2D accelerations. The goal is to reach randomly sampled location. See Appendix C.2 for details.
To test generalization to new obstacle configurations. we vary the number of training maze layouts (5, 10, 20, 40) and evaluate on held-out unseen layouts. For models trained on 5 maps, we further analyze how test-time performance degrades as layouts diverge from the training distribution. Figure shows that PLDM generalizes best even when trained on just maps while other methods fail. As test layouts become more out-of-distributie
CL
all methods except PLDM degrade in performance. We also evaluate all methods on a single fixed layout and observe 100% success rate across the board (Table 4). Figure 7 and 8 show PLDM's inferred plans and trajectories from different agents. We also investigate HILP'ss failure to generalize in Appendix I, and show that HILP's learned representation space successfully captures distance between states in mazes during training. but fails on unseen mazes.
PLDM
PLDM
HILP
HIQL
CRL
Goal
Plan
Figure 7: Left: Plans generated by PLDM test time. Right: Actual agent trajectories for the tested methods. PLDM is the only method that reliably succeeds on held-out mazes.
Takeaway: The model based approach enables better generalization to unseen obstacle layouts than model-free methods.
Conclusion
In this work, conducted comprehensive study of existing methods for learning from offline data without rewards, spanning both reinforcement learning and optimal control, aiming to identify the most promising approaches for leveraging suboptimal trajectories. We focus on of navigation tasks that present unique challenges due to the need for spatial reasoning, generalization to new layouts, and trajectory stitching. Our findings highlight HILP and PLDM the strongest candidates, with PLDM demonstrating the best generalization to new obstacle layouts and to a state-avoidance task. We aggregate our experimental results in Table 1. Overall, we draw three main conclusions:
PLDM exhibits robustness data quality, high level of data efficiency, best-of-class generalization to new layouts, and excels at adapting to tasks beyond goal-reaching:
Learning a well-structured latent-space (e.g. using HILP) enables trajectory stitching and robustness to data quality, although is more data hungry than other methods;
3. Model-free GCRL methods are great choice when the data is plentiful and good quality.
Future work. Our findings indicate that learning and planning with latent dynamics models is a promising direction for building general autonomous agents. There are many promising areas for exploration: extending PLDM to more complex domains such as robotic manipulation and partially observable environments; 2) investigating improved dynamics learning methods to mitigate issues like accumulating prediction errors for tasks involving long-horizon reasoning [37]; and 3) improving test-time efficiency, either by backpropagating gradients through the forward model [8] or via amortized planning.

---

Acknowledgments
This work was supported through the NYU IT High Performance Computing resources, services, and staff expertise, by the Institute of Information & Communications Technology Planning & Evaluation (IITP) with grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research, by the Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AT) and the National Science Foundation (under NSF Award 1922658), and in part by AFOSR grant FA9550-23-1-0133 "World Models and Autonomous Machine Intelligence" " and by ONR MURI N00014-22-1-2733 "Self-Leaming Perception through Interaction with the Real World".
References
[1] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[2] R. Balestriero and Y. LeCun. Learning by reconstruction produces uninformative features for perception. arXiv preprint 1rxiv:2442.1133// 2024.
[3] E. Banijamali, R. Shu, M. Ghavamzadeh, H. Bui, and A. Ghodsi. Robust locally-linear controllable embedding. KurXiv:1710.05371 Feb. 2018. doi: 10445500/XXx.1170..037...
URL http: http::///aavv../a org/ httg//b//1700 05373. arXiv:1710.05373 [es].
[4] A. Bardes, J. Ponce, and Y. LeCun. Vicreg: self-supervised learning. arXiv preprint arXiv:2105.04906 2021.
regularization for
[5] A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver. Successor features for transfer reinforcement learning. Advances in neural information processing systems, 30, 2017.
[6] D. Bertsekas. Dynamic programming and optimal control: Volume I, volume 4. Athena scientific, 2012.
[7] D. P. Bertsekas. Reinforcement Learning and Optimal Control. Athena Scientific, Belmont, MA, 2019. ISBN 978-1-886529-46-6 URL http::////ww...it.eee/tttiaiaa hwt.diiiitttttboobooeoooo html.
[8] H. Bharadhwaj. K. Xie, and F. Shkurti. Model-predictive control via cross-entropy and gradient- based optimization. In Learning for Dynamics and Control, pages 277-286. PMLR, 2020.
[9] K. Black, N. Brown, D. Driess, Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, Hausman, B. Ichter, et al. no: A vision-language- language-acttion flow model for general robot control. arXiv preprint 24X1v:2410.24164, 2024.
[10] A. Brohan, N. Brown, Carbajal, Chebotar, Chen, K. Choromanski, Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-ace models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023.
[11] A. E. Bryson. Optimal control-1950 to 1985. IEEE Control Systems Magazine, 16(3):26-33. 1996.
[12] Buckman, C. Gelada, and M. G. Bellemare. The importance of pessimism in fixed-datase policy optimization. arXiv preprint arXiv:2009.06799 2020.
[13] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. Prooeee Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[14) K. Cho. On the properties neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1409 1259, 2014.
10

---

[15] O. X.-EE Collaboration, A. O'Neill A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar,
A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung. A. Bewley, A. Herzog, A. Irpan. A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie,
A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid, B. Burgess- Limerick, B. Kim, B. Schölkopf, B. Wulfe, B. Ichter, C. Lu, C. Xu, C. Le, C. Finn, C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalashnikov. D. Sadigh, Liu, F. Ceola, F. Xia, F. Zhao, F. V. Frujeri, F. Stulp. G. Zhou, G. Sukhatme, G. Salhotra, G. Yan, G. Feng. G. Schiavi, G. Berseth, G. Kahn, G. Yang, G. Wang, H. Su, H.-S. Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Furuta, H. Bharadhwaj. H. Walke, H. Fang, H. Ha, I. Mordatch, I. Radosavovic, I. Leal, J. Liang, J. Abou-Chakra, J. Kim, J. Drake, J. Peters, J. Schneider, J. Hsu, J. Vakil, J. Bohg. J. Bingham, J. Wu, J. Gao, J. Hu, J. Wu, J. Wu, J. Sun, J. Luo, J. Gu, J. Tan, J. Oh, J. Wu, J. Lu, J. Yang. J. Malik, J. Silvério, J. Hejna, J. Booher, J. Tompson, J. Yang. J. Salvador, J. Lim, J. Han, K. Wang, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Bymee K. Oslund, K. Kawaharazuka,
E. Johns, E. Foster,
K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. Singh, K.-H. Zeng, K. Hatch, K. Hsu, L. Itti, L. Y. Chen, L. Pinto, L. Fei-Fei, L. Tan, L. J. Fan, L. Ott, L. Lee, L. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Irkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. C. Yip. M. Zhang, M. Ding, M. Heo. M. K. Srirama, M. Sharma, M. J. Kim, M. Z. Irshad, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. Liu, N. D. Palo. N. M. M. Shafiullah. O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. T. Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, P. Sermanet, Abbeel, Sundaresan, Q. Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Mart'in- Mart'in, in, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang, R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani,
Levine, S. Lin, S. Moore, Bahl, Dass, S. Sonawani, S. Tulsiani, S. Song, S. Xu, S. Haldar, S. Karamcheti, S. Adebola, S. Guist, Nasiriany, S. Schaal, S. Welker,
Tian, Ramamoorthy, S. Dasari, S. Belkhale, S. Park, S. Nair, S. Mirchandani, T. Osa, T. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding. T. Davchev, T. Z. Zhao. T. Armstrong. T. Darrell, T. Chung, V. Jain, V. Kumar, V. Vanhoucke, V. Guizilini. W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang. X. Zhu, X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang, Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Y. Dou, Y. Cho, Y. Lee, Y. Cui, Y. Cao, Y.-H. Wu, Y. Tang, Y. Zhu, Y. Zhang, Y. Jiang. Y. Li, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Ma, Z. Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin. Open X-Embodiment: Robotic learning datasets and RT-X models. https:/ /ttp//aavvvv//a//////eee/eaaaa/a org/ abs/ 22100 08864, 2023.
[16] S. Dasari, Ebert, S. Tian, S. Nair, Bucher, K. Schmeckpeper, S. Singh, Levine, and C. Finn. Robonet: Large-scale multi-robot learning. (11.iv:1100.12155) Jan. 2020. doi: 10448550arXi..1110.1122 URL http://arxiv wtwpi/i.rrg/bbb/////eee 11215. 1rXiv:1910.12215 [cs].
[17] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. (arXiv:1812.000663, Dec. 2018. doi: 1105500/rxxi.1111200065 URL httt:://arxii htgg/bbs/1811 00568. [cs].
[18] D. Ernst, P. Geurts, and L. Wehenkel. Trec-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
[19] B. Eysenbach, T. Zhang, Levine, and R. R. Salakhutdinov. Contrastive learning as goal- conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35: 35603-35622 2022.
[20] C. Finn and S. Levine. Deep visual foresight for planning robot motion. (arXiv: (wtXXi::660000000 1610.0069). Mar. 2017. doi:
1...000arXiv.110..0666. URL httpp://aaxiii.rr//bbb/////// 1610.00696.666.. arXiv:1610.0069: [cs].
[21] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. 15rXiv:1509.00 061 13), Mar. 2016. doi: 10.48550/arXiv. 1509.0611 13. URI. http://arxice wwwpe/ii..rg/abb////ooo/ 06113. arXiv:1509.061 111.50090161133 [cs].
11

---

[22] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.0722 2020.
[23] D. Ghosh, A. Gupta, A. Reddy, J. Fu, C. Devin, B. Eysenbach, and S. Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088. 2019.
[24] R. Ghugare, M. Geist, G. Berseth, and B. Eysenbach. Closing the gap between td learning and supervised learning-a generalisation point of view. arXiv preprint arXiv:2401.11237, 2024.
[25] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent 423 dynamics for planning from pixels. arXiv preprint arXi::1811.0455/ 424, 2018.
[26] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models, arXiv preprint arXiv:2301.0411 2023.
[27] N. Hansen, X. Wang, and H. Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.00 04955, 2022.
[28] N. Hansen, H. Su, and X. Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint urXiv:2310.16822 2023.
[29] K. Hatch, T. Yu, R. Rafailov, and C. Finn. Example-based offline reinforcement learning without rewards. Proceedings of Machine Learning Research vol, 144:1-17, 2022.
[30] H. Hu, Y. Yang, Q Zhao, and C. Zhang. The provable benefits of unsupervised data sharing for offline reinforcement learning. arXiv preprint arXiv:2302.13449. 2023.
[31] A. Khazatsky, K. Pertsch, S. Nair. A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, Hejna, M. Itkina, M. Lepert, Y. J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, Mercat, A. Rehman, P. R. Sanketi. A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe. T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, Chung, J. Drake, E. P. Foster, J. Gao, D. A. Herrera, M. Heo, K. Hsu, J. Hu, D. Jackson, C. Le, Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O' O'Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, Bohg. K. Goldberg, A. Gupta, Gupta, D. Jayaraman, J. Lim, J. Malik, R. Martín-Martin, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip, Zhu, T. Kollar, S. Levine, and C. Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024.
[32] Kim, S. Park, and S. Levine. Unsupervised-to-onliee reinforcement learning. arXiv preprint arXiv:2408. /4785, 2024.
[33] Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprini arXiv:21 2021.
[34] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-leaminng In International Conference on Learning Representutions, 2022. URL https://aaaat org/ abs/ abs/2110. 06169.
[35] S. Kuindersma, R. Deits, M. Fallon, A. Valenzuela, H. Dai, F. Permenter, T. Koolen, P. Marion, and R. Tedrake. Optimization- based locomotion planning, estimation, and control design for the atlas humanoid robot. Autonomous robots, 40:429 2016.
[36] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-larrinn for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 1179-1191, 2020. URL https: //tpx/vaxvvv///////////ee//aaav. orgo abs//006....//////////////// 047799.
[37] N. Lambert, K. Pister, and R. Calandra. Investigating compounding prediction errors in learned dynamies models. (arXiv:2003.06637), Mar. 2022, doi: 10448500/2XX1.220000033.., URL [cs].
wwtpe/iwvvvrraa/////x.oooae 09637. arXiv:2203.09637
12

---

[38] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pages 5639-5650. PMLR, 2020.
[39] Y. LeCun. A path towards autonomous machine intelligence version 0.9. 2022-06-27. Open Review, 62(1):1-62, 2022.
[40] T. Lesort, N. Diaz-Rodriguess J.-F. Goudou, and D. Filliat. State representation learning for control: An overview. Nearal Networks, 108:379-392, 2018.
[41] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[42] E. Littwin, O. Saremi, M. Advani, V. Thilak, P. Nakkiran, C. Huang, and J. Susskind. How jepa avoids noisy features: The implicit bias of deep linear self distillation networks. arXiv preprint arXiv:2407.03475 2024.
[43] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132. PMLR, 2020.
[44] V. Mnih. Playing atari with deep reinforcement learning, arXiv preprint arXiv: 2013.
[45] M. Morari and J. H. Lee. Model predictive control: past, present and future. Computers & chemical engineering, 23(4-5):667-683 1999.
[46] S. Nair, S. Savarese, and C. Finn. Goal-aware prediction: Learning to model what matters, In International Conference Machine Learning, pages 7207-7219, PMLR, 2020.
[47] S. Nair, A. Rajeswaran, Kumar, C. Finn, and A. Gupta. R3m: universal visual representation for robot manipulation. arXiv preprint arXiv:2203. /2601, 2022.
[48] Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, L. Y. Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
[49] M. A. OpenAI, B. Baker, M. Chociej, R. J6zefowicz, B. McGrew, J. W. Pachocki, J. Pachocki, A. Petron, M. Plappert, G. Powell, et al. Learning dexterous in-hand manipulation. corr abs/1 05/11808.0017 (2018), arXiv preprint 18xv:8808.00177 2018.
[50] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
[51] S. Park, K. Frans, B. Eysenbach, and S. Levine. Ogbench: Benchmarking offline goal- conditioned rl. arXiv preprint arXiv:2410.20092, 2024.
[52] S. Park, D. Ghosh, B. Eysenbach, and S. Levine. Higl: Offline goal-conditioned rl with latent states as actions. Advances Neural Information Processing Systems, 36, 2024.
[53] S. Park, T. Kreiman, and S. Levine. Foundation policies with hilbert representations. arXiv preprint 24Xiv:240..15567 2024.
[54] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[55] K. Pertsch, O. Rybkin, F. Ebert, S. Zhou, D. Jayaraman, C. Finn, and S. Levine. Long- horizhorizon visual planning with goal-conditioned hierarchical predictors. Advances Neural Information Processing Systems, 33:17321-1733: 2020.
[56] O. Rybkin, K. Pertsch, K. Derpanis, K. Daniilidis, and A. Jaegle. Learning what you can do before doing anything, arXiv preprint 1006.0188009666 2018.
13

---

[57] G. Schultz and K. Mombaur. Modeling and optimal control of human- running. JEEE/ASME Transactions on mechatronics, 15(5):783-792, 2009.
[58] M. Schwarzer, N. Rajkumar, M. Noukhovitch, A. Anand, L. Charlin, R. D. Hjelm, P. Bachman, and A. C. Courville. Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, 2222222669199, 2021.
[59] R. Shu, T. Nguyen, Y. Chow, T. Pham, K. Than, M. Ghavamzadeh, S. Ermon, and H. H. Bui. Predictive coding for locally-linear control. (arXiv:2003.00086, Mar. 2020. doi: 10.48550/ arXiv.2003.0108 URL http: //axivv http:/aawwwiaara..a///// wwg//b//////aeooeo att//bbaa////0 http:/203....00 01086. arXiv:2003.0100 [cs].
[60] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanetot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[61] V. Sobal, J. SV, S. Jalagam, N. Carion, K. Cho, and Y. LeCun. Joint embedding predictive architectures focus on slow features. arXiv preprint arXiv:2211. 2022.
[62] R. S. Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
[63] Y. Tassa, T. Erez, and W. Smart. Receding horizon differential dynamic programming. J. Platt, D. Koller, Singer, and S. Roweis, editors, Advances in Neural Information Processing Sys- tems, volume 20. Curran Associates, Inc., 2007. URL https: neurips cc/ http:///pwpppoooooo.ooooo/aaaaaa
Paper pdf.
[64] E. Todorov and W. Li. A generalized iterative leg method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005., pages 300-306 vol. 1, 2005. doi: 10.1 A0000000. 1469949.
[65] E. Todoroy and W. Li. A generalized iterative lęg method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005., pages 300-306. IEFE, 2005.
[66] E. Todorov, T. Erez, and Y. Tassa Mujoco: A physics engine for model-based control. In IROS, pages 5026-5033, IEEE, 2012. ISBN 978-1-4673-1737-55 URL http://bdd http://bblppuiittiieee. uni-trier. de/db/cont warros/iror22111 htoo##TooorooEE11..
[67] A. Touati and Y. Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34:13-23, 2021.
[68] A. Touati, J. Rapin, and Y. Ollivier. Does zero-shot reinforcement learning exist? arXiv preprint arXiv: 22X9::2209.1493: 14935, 2022.
[69] K. Wang, K. Zhou, Q Zhang, J. Shao, B. Hooi, and J. Feng. Towards better laplacian represen- tation in reinforcement learning with generalized graph drawing. In International Conference on Machine Learning, pages 1 11003-11012. PMLR, 2021.
[70] K. Wang, K. Zhou, J. Feng, B. Hooi, and Wang. Reachability aware laplacian representation in reinforcement learning. arXiv preprint arXiv:2210.13155: 2022.
[71] M. Watter, J. T. Springenberg. J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. (arXivv1500.007665,, Nov. 2015. doi: 4445550arii..15000777550 URL http: org/abs//106/ 07365. 1:X.:35506.0736 [cs].
[72] G. Williams, A. Aldrich, and E. Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprint 1rX 2015.
[73] Y. Wu, G. Tucker, and O. Nachum. The laplacian in rl: Learning representations with efficient approximations. arXiv preprint arXivv1810.04580 2018.
[74] R. Yang, L. Yong. X. Ma. H. Hu, C. Zhang, and T. Zhang. What is essential for unseen goal generalization of offline goal-conditioned rl? In International Conference on Machine Learning, pages 19541139571. PMLR, 2023.
14

---

[75] S. Yang, O. Nachum, Y. Du, J. Wei, P. Abbeel, and D. Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129 2023.
[76] D. Yarats, D. Brandffonbrener, H. Liu, M. Laskin, P. Abbeel, A. Lazaric, and L. Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXi::2201. 1325, 2022.
[77] L. Yen-Chen, M. Bauza, and P. Isola. Experience-embeadee visual foresight. 19rXiv:111100000 Nov. 2019. doi: 191 URL http:// org/abs/1911 15071. arXiv:1911.00 1.00071 [es].
[78] Yu, G. Thomas, L. Yu, X. Ma, Ermon, J. Zou, and C. Finn. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 15:14129-141443 2020. URL https: //tp://////vvv///// org/ aaaabb//a abs/2005. 13239.
[79] Yu, A. Kumar, Chebotar, K. Hausman, C. Finn, and Levine. How leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning, pages 2561 1-6635. PMLR, 2022.
[80] M. Zare, P. M. Kebria, A. Khosravi, and S. Nahavandi. A survey imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions Cybernetics, 2024.
[81] M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. J. Johnson, and S. Levine. Solar: Deep structured representations for model-based reinforcement learning. (arXiv: 808 09000) June 2019. doi: 10.445550arXxi.1188.00m [es].
wwtpaa////.////////////////e///a 09105. urXiv:1808.099 190. URL http://aaaxx...r////b////0000
[82] W. Zhang, A. GX-Chen, V. Sobal, Y. LeCun, and N. Carion. Light-weight probing of unsuper- vised representations for reinforcement learning. arXiv preprint arXiv:2208. /2345, 2022.
[83] G. Zhou, H. Pan, Y. LeCun, and Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983, 2024.
15

---

Appendix
Table of Contents
Reproducibility
17
Visualization of Plans and Trajectories for Diverse PointMaze
17
Environments and Datasets
18
C.I Two-Rooms Environment
18
C.2 Diverse PointMaze
18
C,3 Ant U-Maze
19
D Models
19
D.I PLDM
19
E Effects of Uncertainty Regularization via Ensembles
22
Analyzing planning time of PLDM
22
E1 Evaluating PLDM Performance With Adjusted Inference Compute Budget
23
G Extended Baselines: Input Reconstruction Learning and TD-MPC2
23
Analyzing Statistical Significance of Results
24
I Analyzing HILP's Out-of-Distrihutton Generalization
25
J Hyperparameters
25
J.1 CRL, GCBC, GCIQL, HIQL, HILP
25
J.2 PLDM
28
J.3 Further related work
29
K Discussion of Computational Costs
30
Limitations
30
16

---

A Reproducibility
Code is available at htttps///ggttuu...m///vaaaaaaae
An implementation of our method is provided in the GitHub repository above, which includes environments, data generation, training, and evaluation scripts. Detailed hyperparameters for our method and the baselines are provided in Appendix J.
Visualization of Plans and Trajectories for Diverse PointMaze
FAPC4
Training Layouts
PLDM Plan
PLDM
CRL
GCBC
GCIQL
HIQL
HILP
In distribution
Out-of-
distribution
The training layouts used in the 5 layout setting. Middle: Trajectories of different agents navigating unseen maze layout towards goal test time. As the layouts become incrcasingly out-of- distribution, only PLDN consistently succeeds. Layouts can represented either wall or empty space. The distribution shift is quantified by the minimum edit distance between given test layout and the closest training layout. The top row corresponds to in-distribution layout with minimum edit distance of 0, and with each subsequent row, the minimum edit distance increases by I incrementally.
Figure 8: Top:
4x4 array, with each value being
17

---

C Environments and Datasets
C.1 Two-Rooms Environment
We build own top-down navigation environment. It implemented in PyTorch [54], and supports GPU acceleration. The environment does not model momentum, i.e. the agent does not have velocity, and is moved by the specified action vector each step. When the action takes the agent through wall, the agent is moved to the intersection point between the action vector and the wall. We generate the specified datasets and save them to disk for our experiments. The datasets generation takes under 30 minutes.
C.2 Diverse PointMaze
Here, build upon the Mujoco PointMaze environment [66], which contains point mass agent with 4D state vector (global x, global y, U2, vy), where v is the agent velocity. To allow our models to perceive the different maze layouts, use model input top down view of the maze rendered as (64, 64, 3) RGB image tensor instead of relying on (global x, global y) directly.
Mujoco PointMaze allows for customization of the maze layout via a grid structure, where grid cell can either be a wall or space. We opt for 4 x 4 grid (excluding outer wall). Maze layouts generated randomly. Only the following constraints are enforced: 1) all the space cells are interconnected, 2) percentage of space cells range from 50% to 75%.
We set action repeat to 4 for our version of the environment.
C.2.1 Dataset Generation
We produce four training datasets with the following parameters:
# episodes
episode length
# Transitions
# layouts
per layout
1000000
5
2000
100
1000000
10
1000
100
1000000
20
500
100
1000000
40
250
100
Table 3: Details for Diverse PointMaze datasets
Each episode is collected by setting the (global global y) random location in the maze, and agent velocity (Vas vyy) by randomly sampling a 2D vector with ||o|]| S 5, given that Us and by are clipped within range of [-5, 5] in the environment.
C.2.2 Evaluation
All the test layouts during evaluation are disjoint from the training layouts. For each layout, trials are created by randomly sampling start and goal position guaranteed to be at least 3 cells away on the maze. The same set of layouts and trials are used evaluate agents for given experimental setting.
We evaluate agents in two scenarios: How agents perform on test layouts when trained on various number of train layouts; 2) Given a constant number of training layouts, how agents perform on test maps with varying degrees of distribution shift from the training layouts.
For scenario 1), we evaluate the agents on 40 randomly generated test layouts, trial per layout.
For scenario 2), we randomly generate test layouts and partition them into groups of 5, where all the layouts each group have the same degree of distribution shift from train layout as defined by metric D Dmin defined the following:
18

---

Given train layouts { { Ligin ... ...L ) test layout Lust and let d(L1, L2) represents the edit distance between two layouts L1 and L2 binary grid representation. We quantify the distribution shift of List as Dmin (1, d( Lest, u.ain)
In this second scenario evaluate trials per layout, thus total of 5 x 25 per group.
C.2.3 Results for Single Maze Setting
Table 4: Results averaged over 3 seeds + std
Method
Success rate
PLDM
0,990 = 0.001
CRL
0.980 0.001
GCBC
0,970 = 0.024
GCIQL
1,000 0.000
HIQL
1,000 = 0.000
HILP
1.000 0.000
C.3 Ant U-Maze
To investigate whether our findings generalize to enviromments with more complicated control dynamics, we test the methods the Ant U-Maze environment [66] with 8-dimensional action space and 29-dimensional state space. Similar to our previous analysis on Two-Rooms, we showcase the trajectory stitching capabilities of different methods.
C.3.1 Dataset Generation
To collect the dataset, we use pretrained expert directional policy from Park et al. [51] to generate 5M transitions of exploratory data, where resample new direction every 10 steps, and apply Gaussian noise with a standard deviation of 1.0 to every action.
C.3.2 Evaluation
For evaluation, the quadruped is randomly initialized at either the left bottom right bottom corner, while the goal location is at the opposite diagonal corner, thus requiring the ant to make turn. Each method is evaluated 10 trials.
D Models
For CRL, GCBC, GCIQL, and HIQL the implementations from the repository' of OGBench [51]. Likewise, for HILP we use the official implementation" from its authors.
For the Diverse PointMaze environment, to keep things consistent with our implementation of PL.DM (D.1.4), instead of using frame stacking, we append the agent velocity directly to the encoder output.
D.I
PLDM
D.1.1
Objective for collapse prevention
To prevent collapse, we introduce VICReg- based [4] objective. We modify it to apply variance objective across the time dimension to encourage features to capture information that changes, as
htmp::/gittaa..mm/ccoooogarrggrd https://ithhbb.co//cccooonaaaaap
19

---

opposed to information that stays fixed [61]. The objective to prevent collapse is defined as follows:
H
Cvar
1
m0,
/Var /Va((Zz (Ze,:) +C)
HD
1=0 1=
N
((Z,) = N
(7 - T Zo), 7
Zd
Loov =
1
Σ
H 1=0
D
it
H
N
LIDM
01.6 - MLP(( ( 70+1,)))
N
1=
0=0
We also apply a tunable objective enforce the temporal smoothness of learned representations:
N
Ltime -sim
Ze,b
Ze+1.
N
I=O
b=U
The combined objective is a weighted sum above:
LIEPA
L-si + alovar + BLeax + ò Leime--sin + w CIDM
D.1.2 Ablations of Objective Components
We conduct a careful ablation study over each loss component by setting its coefficient to zero. Two-Room ablations are performed in the optimal setting with sequence length 90, dataset size 3M, and all expert data. Diverse Maze ablations are performed in the training maps setting.
Ablation
Success rate (Two-Rooms)
Success rate (Diverse Maze)
98.0 + 1.5
98.7 + 2.8
var coeff (ar)
13.4 = 9.2
11.4 + 6.5
cov coeff (33
29.2 + 4.4
7.8 + 4.1
time sim coeff (6)
71.0 + 3.0
95.6 + 3.2
IDM coeff (w)
98.0 + 1.5
75.5 + 8.2
D.1.3 Model Details for Two-Rooms
We use the same Impala Small Encoder used by the other methods from OGBench [51]. For predictor, we use the a 2-layer Gated recurrent unit [14] with 512 hidden dimensions: the predictor input at timestep is 2D displacement vector representing agent action at timestep while the initial hidden state is (5p)),, or the encoded state at timestep 0. single layer normalization layer is applied to the encoder and predictor outputs across all timesteps. Parameter counts the following:
total params: 2218672
encoder params: 1426096
predictor params: 793600
D.1.4 Model Details for Diverse PointMaze Enviromment
For the Diverse PointMaze environment, we use convolutional networks for both the encoder and predictor. To fully capture the agent's state at timestep t, we first encode the top down view of
the maze to get spatial representation of the environment hg R3364464 + R 16xx66x26 zen ho() We incorporate the agent velocity by first transforming it into planes Expander2D R2 + 26x226 " Expander2D(s")) where each slice [**[]] is filled with s'(i]. Then, we
concatenate the expanded velocity tensor with spatial representation along the channel dimension to get overall representation: concat(s" genv dim 0) I 16226x2
20

---

For the predictor input, we concatenate the state E R 18x226x2 with the expanded action (xpander2D(a.) 22x26x26 along the channel dimension. The predictor output has the same 1812266226 Both the encodings and predictions are flattened
dimension as the representation:
for computing the VicReg and IDM objectives.
We set the planning-frequenge (Section 3.3) in MPPI to k = 4 for this environment.
The full model architecture is summarized using PyTocc--like notations.
total params: 53666
encoder params: 33296
predictor parama: 20370
PLDM(
(backbone): MeNet6(
(layers): Sequential(
(0): Conv2d(3, 16, kernel_size=(6, 5), (1, 1))
(1) : GroupNorm(4, 16, eps-1e-05, affine-True)
(2) ReLU()
(3) : Conv2d(16, 32, sernel_size=(f 5), stride= 2))
(4) : GroupNormm 32, eps=1e-05, affine=True)
(6) : ReLU()
(6) Conv2d (32, 32, kernel_size==3, 3), stride=(1, 1))
(7): GroupNorm 32, eps-1e-05, affine-True)
(8) ReLU()
(9): (32, 32, (3rnel_sizo--3, 3), stride-(1, 1), padding-(1, 1))
(10) GroupNormm (8, 32, eps=1e-05, affine=True)
(11): ReLU()
(12): (32, 16, kernel_size=(1, 1), stride=(1, 1))
(propio_encoder):
2xpander2D()
(predictor): ConvPredictor
(layers): Sequential(
(0): Conv2d(20, 32, kernel_size-(C 3), stride=(1, 1), padding= (1, 1))
(1) GroupNorm(4, 32, eps-1e-05, affine-True)
(2) ReLU()
(3) : C3nv2d(32, 32, kernel_size-de 3), stride=(1, 1), (1, 1))
(4) : Grouplorrm(4, 32, eps=1e-05, affine=True)
(5) : ReLU()
(6): C32v2d(32, 18, kernel_size-(3 (3, 3), stride-(1, 1), padding- (1, 1))
(action_encoder): Expander:2
)
D.1.5 Model Details for Ant-U-Maze
We encode the global (z, y) position using 2-layer MLP into 256 dimensional embedding, and concatenate it with the rest of the raw proprioceptive state to make our overall state representation. Our predictor is a 3-layer MLP with ensemble size of During training, variance and covariance regularization is only applied on the part of the representation for (z, y) (first 256 dimensions), since the rest of the proprioceptive state are not encoded and therefore do not collapse.
total params: 1080615
encoder params: 9120
predictor params: 1072007
PLDM
(backbone): MLPEncoder
(globa_xy_enl eecoodec): Sequential
(0): Linear (in_aiinteeeeeee-22 fut_feature--32, bias-True)
(1): (oerrlorm((32.)) eps-1e-05,
(2) (is(lalaac--Trre))
(3) Linear (inear(i__fatures-te out features-te,5, bias-True)
21

---

(4) : LayerNorm (256,), eps-1e-05, elementtusss__fffieerrrr
encoder)
Identity()
(predictor): MLPPredictor(
(layers): Sequential(
(0): Lineur(in,f featuresss,,,, bias=True)
(1) LayerNorm((256,)) eps=1e-05, elementuriee
(2) (ish(innlace==Irue)
(3) Linear(in_featurrt
out_featuree=266 bias=True)
(4) Layer)oorm((266.), eps-1e-05, Alementwisse_afffne-ritee
(6) : (ish(anplaco-frre)
(6) : (ineariin_eauuueee=2555 aut_feaauuee=283, bias-True)
(7): Layerlorr
(283,), (228)))
eps-1e-05, elementwise, (re
E Effects of Uncertainty Regularization via Ensembles
100
100
Success rate (9))
80
80
80
60
60
60
40
40
40
0.00
0.05
0.10
0.15
20
40
60
80
10a
100
10
10a
Fraction non-random data
Sequence length
Dataset size
100
80
40
20
50
250
500
Sequence length
PLDM w/ Cun:
PLDM w/o Cunc
Figure 9: Top Row: environment. Bottom Row: AntMaze environment.
Analyzing planning time of PLDM
In order to estimate how computationally expensive it is to run planning with latent dynamics model, we evaluate PLDM, GCIQL, and HIQL on 25 episodes the Two-Rooms environment. Each episode consists of 200 steps. We record the average time per episode and the standard deviation. omit HILP, GCBC, or CRL because the resulting policy architecture is the same, making evaluation time identical to that of GCIQL. HIQL takes more time due to the hierarchy of policies. When replanning every step, PLDM slower than the policies. However, PL.MM can match the latencies of policies by replanning less frequently with negligible performance drop.
22

---

100
100
Success rate (9)
75
75
75
50
50
50
0.00
0.05
0.10
0.15
20
40
60
80
10a
10
10a
10
Fraction of nen-randem data
Sequence length
Dataset size
CRL
GCBC
GCIQL
HILP
HIQL
PLDM
PLDM (replan every 4 steps)
Comparing PLDM's performance under fixed inference time compute budget on two-rooms. We see that across wooms experiments, PLDM performs only slightly worse when replanning every 4 steps compared to replanning every step.
Figure 10:
100
100
Success rate (96)
50
50
25
25
0
20
5.0
7.5
Number of training maps
In-distribution
Out-of-distribttion
GCBC
GCIQL
HILP
HIQL
PLDM (replan every steps)
Figure Il: Evaluating PLDM's performance under fixed inference time compute budget on diverse maze.
Similarly to two-rooms, we only see a minor degradation in performnace
Table 5: Time of evaluation on one episode in Two-Rooms environment. Time is averaged across 25 episodes. PLDM success rates are normalized against the setting that replans every step. PLDM can match the latencies of GCIQL and HIQL by replanning less frequently with neligible cost to performance.
Method
Replan Every
Time per episode (seconds)
Normalized Success Rate
PLDM
16.0 + 0.13
1.00
PLDM
4.8 0.09
0.95
PLDM
16
2.6 0.07
0.90
PLDM
0.07
0.62
GCIQL
3.6 0.10
HIQL
4.0 0.08
F.1
Evaluating PLDM Performance With Adjusted Inference Compute Budget
As we see in Table 5, replanning every 4 steps brings PLDM close to other methods' inference speed. In order further compare how PLDM compares other methods when inference time compute fixed between methods, we run additional evaluations, and report them in Figure 10 and Figure 11.
G Extended Baselines: Input Reconstruction Learning and TD-MPC2
In this section, we evaluate reconstruction- based objectives for training the encoder and dynamics model, as well as TD-MPC2 [28]. For reconstruction, we compare two approaches: one based on Dreamer [25], and one that replaces our VICReg objective with pixel reconstruction.
Dreamer
We adapt DreamerV3 [26] to our setting by:
Removing rewards during training and using only the reconstruction loss;
Omitting policy learning, instead planning via latent representation distance, just like we did for PLDM. Since the representations DreamerV3 uses are discrete, we use KI. divergence instead of L2 distance. We note that this not what DreamerV3 architecture was designed
23

---

Table Comparing reconstruction- based latent-dynamics learning and TDMPC-2 to other baselines. We test the methods on good-quality data in the two-rooms environment. We see that reconstruction-bant methods perform significantly worse than PLDM and other baselines. TD-MPC2 fails to learn altogether due to collapse, and performs poorly even with the added IDM objective.
Method
Success rate
CRL
89.3 + 1.2
GCBC
86.0 t 4.5
GCIQL
98.0 + 0.9
HILP
100.0 0.0
HIQL
96.4 t 3.0
PLDM
97.4 = 1.3
DreamerV3
24.0 + 6.9
Reconstruction
26.2 = 13.9
TD-MPC2
0,0
+ 0.0
TD-MPC2 + IDM
35.0 - 0.0
for, making this comparison flawed. However, we believe serves to highlight that compared to Dreamer V3, PLDM is designed for very different setting in terms of available data.
Reconstruction As opposed to Dreamer, this baseline uses the same architecture the encoder and dynamics as PLDM instead of RSSM, and only replaces the VICReg objective with reconstruction term. The architecture of the decoder mirrors that of the encoder.
TD-MPC2 Like Dreamer models, TD-MPC2 [28] also relies on rewards to learn its representions. In order to adapt TD-MPC2 to setting, we remove the reward prediction component, and only keep the objective enforcing consistency between the predictor and encoder outputs.
Results are shown in Table 6. We test all methods on the two-rooms environment described in Section 4.1. We use good-quality data, with long trajectories and good transition coverage. We find that pixel observation reconstruction is not good objective to learn representations, and using it results in poor planning performance. TD-MPC2 collapses altogether, achieving 0% success rate. To prevent collpase, we add inverse dynamics modelling (IDM), which somewhat improves performnace, although is still far behind other methods. We only tested one seed for TD-MPC2 + IDM.
H Analyzing Statistical Significance of Results
In order to analyze whether the results in this paper are statistically significant, we perform Welch's I-test to compare the performance of PLDM to other methods. Because 5 seeds is not enough for statistical tests, we pool results across settings and show results in Table 7. We also run additional seeds for certain selected settings to get a total 10 seeds per method, and show results of statistical analysis in Table 8. Overall, we see that the results are significant, except for certain settings comparing to HILP and GCIQL, which aligns with our findings.
Table 7: Statistical significance of results (pooled). / means that Welch's t-test showed that PLDM is better than the corresponding method when pooling results across seeds and dataset parameters.
Fraction of non-random data
Number of training maps
In-distribution Out-of-distribution
Sequence length
Dataset size
CRL
(p=1 (6=-166)-03) (p=4.22 25c--25)
(p=8.466-07) 1144666-08)
(p=11.13c-00
(p==3.43e-10 10)
(p=4.02e-00
GCBC
(p==7.260-03
(p=3.71e-20)
/ (p=1119cc0
19944-05
GCIQL
X
/ (p=1.29e-12)
/ /p==122ee-03
IIILP
X / (p=2.22e--0)
1e=2211
/ (p=6.54e-06)
(0==.10ee)5)
IIIQL
(p=
(p=2.39e-02)
(p=2.84e-06)
(p=9.0le-03)
24

---

HILP Table Statistical significance of results for selected datasets. means that Welch's t-test showed that PLDM better than the corresponding method results across 10 seeds.
Fraction of non-random data 0%
Sequence length 17
Dataset size 20312
CRL
(p=3.82e-1)
(p=1. 13c-07) (p=5.75e-10)
/ (p= (1)
GCBC
(p=1.54e-12)
(p=1.888-144
GCIQL X
X
X
(p==2449e- 19) (p==2.65e-11 11)
HIQL
(p=8.89c-05)
/ (p=2.83e-07)
I Analyzing HILP's
Generalization
To understand HILP's poor generalization to -iitt---iitiittioon (OOD) maze layouts, we visualize the distance in HILP's learned latent representation space. HILP learns latent representation é(s), such that [|Ø(s) (s,,) 2 2/2 equal the lowest number of transitions needed traverse from to sy- We hypothesize that fails to generalize
out-of-distribution maze layouts, resulting in incorrect predicted distances and in the failure of the goal-conditioned policy. We visualize the distances on in-distribution and outtof-distribbttion layouts for an encoder trained on 5 different layouts in Appendix I. We see that HILP distances are meaningful only on in-distribution layouts, and are very noisy -uittf-diitributtin layouts. This failure of the latent-space distance to generalize to out-of-distribution layouts confirms our hypothesis, and highlights the strength of PLDM.
80
(a) HILP distances on in-distribution maze layouts
(b) HILP distances on out-of-distribution maze layouts
Figure 12: HILP learns representations of states such that distance in representation space between pair of states is equal to the number of steps needed go between steps. These plots visualize the distance from the state denoted with red dot other states distances increase smoothly and mostly reflect the number steps needed to go between states. (b) We see that on an out-of distribution maze, the representation space distances no longer make sense.
the maze. We see that an in-distribution maze, the
Hyperparameters
J.1 CRL, GCBC, GCIQL, HIQL, HILP
Unless listed below, hyperparameters remain consistent with default values from OGBench! and HILP2 repositories
25

---

J.1.1 Two-Rooms
To tune hyperparameters, we ran grid search for good quality data, then another grid search for sequence length 17, then tested both on all settings. grid searches, we searched over learning rate, expectiles (for GCIQL, HILP, and HIQL), and the probability of sampling random goal. For CRL and GCBC, we ended up using default OGBench parameters. The other parameters are described below:
H.L.P We used learning rate of 3e-4, expectile of 0.7, and skill expectile 0.7 in all settings.
GCIQL.
For settings with dataset size of 634 as well as for good quality data, we used expectile 0.7, learning rate 3e-4, BC coefficient of 0.3, and probability of sampling random goal. For all other settings, used learning rate 3e-4, expectile of 0.7, BC coefficient of 0.003, and probability of sampling a random goal 0.3.
HIQI, For sequence length 17 and 33, we used learning rate of 3e-5, expectile 0.7, AWR temperature of 3.0 (both high and low levels), and probability of sampling random goal of 0.6. For all other settings, we used learning rate of 3e-4, expectile of 0.7, AWR temperature of 3.0 (both high and low levels), and probability of sampling random goal of O.
J.1.2 Diverse PointMaze
Table 9: Dataset-specific hyperparameters CRL, GCBC, and HILP for the Diverse PointMaze environment. For HII.P, set the same value for expectile and skill expectile.
CRL
GCBC
HILP
Dataset
LR
Expectile
5 layouts
0.0003
0.0003
0.0001
0.9
10 layouts
0.0003
0.0001
0.0001
0.9
20 layouts
0.0003
0.0001
0.0001
0.9
40 layouts 0.0003
0.0001
0.0001
0.9
Table 10: Dataset-specific hyperparameters of GCIQL and HIQL for the Diverse PointMaze environment.
GCIQL
HIQL
Dataset
BC Coeff.
Prob Rand Goal
AWR Temp.
Prob Rand Goal
LR
Expectile
LR
Expectile
5 layouts
0.0003
0.6
0.3
0.15
0.00003
0.6
0.6
10 layouts
0.0003
0.6
0.3
0.15
0.0001
0.7
3.0
0.0
20 layouts
0.0003
0.6
0.3
0.15
0,00003
0.6
1.0
0.6
40 layouts
0.0003
0.6
0.3
0.15
0.0001
0.7
3.0
0.0
J.1.3 Ant U-Maze
Table 11: Ant-Umaze: dataset-specific hyperparameters for CRL, GCBC, and HIIP,, parsed from names.
CRL
GCBC
HILP
Seq Len
BC Coeff.
Prob Rand Goal
Expectile (skill / actor)
LR
LR
LR
25
3e-5
0.1
0.3
3e-3
3e-4
0.6 0.8
50
3c-5
0.1
3e-3
3c-4
0.6
100
3e-4
0.1
3e-3
3e-4
0.6/ 0.8
250
3e-4
0.1
0.0
3e-3
3e-4
0.6/ 0.8
500
3c-4
0.1
0.0
3e-3
3c-4
0.6/ 0.8
26

---

Table 12: Ant-Umaze: dataset-specific hyperparameters for GCIQL and HIQL parsed from names.
GCIQL
HIQL
Seq Len
BC Coeff.
Prob Rand Goal
AWR Temp.
Prob
LR
Expectile
LR
Expectile
Rand Goal
25
3e-5
0.9
0.15
0.15
3e-4
0.6
3.0
0.6
3c-5
0.15
0.15
3e-4
3.0
100
3e-5
0.15
0.15
3e-4
3.0
250
3c-4
0.3
0.0
3e-4
0.6
3.0
500
3e-4
0.9
0.3
0.0
3e-4
0.6
3.0
0.6
27

---

J.2
PLDM
J.2.1
Two-Rooms
The best case setting sequence length 91, dataset size 3M, non-random % 100, wall crossing % x 35. For our experiments we vary each of the above parameters individually.
Table 13: Datasct-agnostic hyperparameters for Two-Rooms
Hyperparameter
Value
Batch Size
64
Predictor Horizon (H)
16
Optimizer
Adam
Scheduler
Cosine
Ensemble size K
MPPI noise
MPPI # samples
500
MPPI X
0.005
Planner Cuncertainty coeff 5
0.0001
Planner Cnncetaintty coeff y
0.9
For the dataset specific hyperparameters, we tune the following parameters from Appendix D.1.1:
Table 14: Dataset specitic hyperparameters for two-Rooms
Dataset
LR
8
Sequence length 91
0.0007
4.0
6.9
0.75
Sequence length 65
0.0003
5.0
6.9
0.75
Sequence length 33
0.0014
3.5
6.9
0.75
Sequence length = 17
0,0028
3.0
6.9
0.75
Dataset size
0.0030
13.0
0.50
Dataset size 1269
0.0010
2.2
13.0
0.50
Dataset size 5078
0.0005
2.2
13.0
0.90
Dataset size 20312
0.0030
2.2
13.0
0.50
Dataset size 81250
0.0010
2.2
13.0
0.50
Dataset size 325k
0.0010
4.0
6.9
0.75
Dataset size 1500k
0.0010
4.0
6.9
0.75
Non-random % 0.001
0.0007
3.9
6.9
0.74
Non-random % 0.01
0.0007
3.9
6.5
0.19
Non-random % 0.02
0,0007
3,9
6.5
0.72
Non-random % = 0.04
0.0007
3.9
6.5
0.65
Non-random % 0.08
0.0007
3.9
6.5
0.24
Non-random % 0.08
0.0007
6.5
0.24
Wall crossing %
0.0007
4.0
6.9
0.75
28

---

J.2.2 Diverse PointMaze
Table 13: Datasct-agnostic hyperparameters for Diverse PointMaze
Hyperparameter
Value
Epochs
Batch Size
128
Predictor Horizon (H)
Optimizer
Adam
Scheduler
Cosine
Ensemble size K
MPPI noise o
MPPI # samples MPPI
500
0.0025
Table 16: Dataset specific hyperparameters Diverse PointMaze
Dataset
LR
8
map layouts 5
0.04
35.0
12.0
0.1
5.4
map layouts 5
0.04
35.0
12.0
0.1
5.4
map layouts 0.05
54.5
15.5
0.1
5.2
map layouts
0.05
54.5
15.5
J.2.3 Ant-U-Maze
Table 17: Dataset-agnostic hyperparameters for Ant-U-Maze
Table 18: Dataset-specific hyperparameters for Ant-U-Maze
Hyperparameter
Value
Dataset
LR
Epochs Batch Size
64
Sequence length
0.006
Predictor Horizon (H)
16
Sequence length 50
0.004
Optimizer
Adam
Sequence length 100
0.003
Scheduler
Cosine
Sequence length 250
0.001
Ensemble size K
5
Sequence length 500
0.001
a
26.2
B
0.5
8.1
0.58
MPPI noise MPPI # samples
5
500
MPPI A
0.0025
Planner Cuncertaityy Coeff
1
Planner Cuncertainty coeff
0.9
J.3 Further related work
Offtine RL. This field aims to learn behaviors purely from offline data without online interactions. As opposed to imitation learning [80], offline RL is capable of learning policies that are better than the policy collecting the data. However, big challenge preventing the policy from selecting actions that were not seen in the dataset. CQL [36] relies on model conservatism to prevent the learned policy from being overly optimistic about trajectories not observed the data. IQL [33] introduces an objective that avoids evaluating the Q-function on state-action pairs not seen in the data to prevent value overestimation. MOPO [78] is model-based approach to learning from offline data, and uses model disagreement constrain the policy, See [41] for more in-depth survey.
29

---

Foundation models in RL. Recently, following the success of NLP, the RL community put lot of effort into training large sequence models, which sparked dataset collection efforts like Open-X -- Embodiment [15] and DROID [31]. Large datasets have enabled training models such as RT-2 [10] and Octo [48]. See [75] for a more extensive survey on the topie.
Training representations for RL. Another way to use large amounts of data to improve RL agents using self-supervised learning (SSL). CURL [38] introduce an SSL objective in addition the standard RL objectives. Later works also explore using separate pre-training stage [47, 58, 82]. Zhou et [83] show that pre-trained visual representations from DINO [13, 50] can be used to learn a word model for planning.
K Discussion of Computational Costs
All experiments require only a single GPU, and take up to day when using an Nvidia V100 GPU. We estimate the total computational cost of the experiments included in the paper and in the research process to be between 500 and 2000 GPU days.
L Limitations
All experiments conducted in navigation environments, excluding robot manipulation or partially observable settings. However, we argue that the conceptual understanding of the effects of data quality the investigated methods will hold, as even the relatively simple setting, many recent methods break down in surprising ways. Another limitation lies in the fact that PLDM is about 4 times slower during inference, although in Appendix we find that even when limited the same budget of inference compute as model-free methods, PLDM retains most of its performnace.
30

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main claims are supported by our experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have a limitations section in the conclusion.
31

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper is empirical in nature, no theoretical results were provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide hyperparameters and code to reproduce results.
Guidelines:
The answer NA means that the paper does not include experiments.
32

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, see latent-plannin ppllannngggithuu github..iii io.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr cc/ gguides/CodeSuuiissee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nipss /tipsscccppubllli http:ccccppppppcccccee//cocppcce http://dddd//eeuuubmimiiieee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
33

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting//etails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experiments report standard error. In addition, we conduct tests verify statistical significance of results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
34

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tttupip//nurrps..... wwterris.ccppuuuue
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper conducts research the area of optimal control and RL, which do have societal impact. However, this specific research paper does not have substative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
35

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
36

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to LLM policy (https: //t/enurips..// wwc/oooffeeecce///00000 for what should or should not be described.
37


Reviews:
[{"review_id": "WJTNlnjy2W", "reviewer": "Reviewer_3zum", "summary": "The authors focus on the problem of learning control from a set of reward-free trajectories (i.e. offline data). In particular, the authors focus on a comparison of offline reward-free RL algorithms with the optimal control (OC) method, utilizing a learned JEPA model that the authors refer to as PLDM. The authors compare the algorithms along the following axes: transfer to new environments, transfer to new tasks, data efficiency, best-case performance, learning from a random policy, and trajectory stitching. The experiments are performed on a custom \"two-room\" task, AntMaze and PointMaze tasks. The authors show results in which that the proposed model either outperforms or performs similarly to the best baseline in all tested categories.", "questions": "- Any hypothesis how would the model-based method perform with a smaller action repeat?\n\n- How central is JEPA to the performance here? What would happen with the TD-MPC2 model instead of JEPA?", "limitations": "Yes, though I believe that the authors should tone down their interpretation of the results", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Overall I believe this is an interesting and valuable manuscript. My biggest issue is with the description of the \"generalization\" experiments and with a strong wording of the claims.\n\nStrengths:\n\n- Reward-free learning is an important direction\n- The paper is clearly written\n- Experiments on both vision and proprioceptive environments\n\nWeaknesses:\n\n- I think that the authors should be more explicit about what \"new task\" and \"new environment\" actually mean in terms of the dynamics of the MDP. For example, changing robot embodiment (e.g. ant -> humanoid or even ant -> ant with longer legs) is different than displacing walls in a maze. I think that this is especially important considering that the method proposed explicitly learns a dynamics model. To the best of my understanding, these perturbations have local impact on the dynamics model.\n\n- In general, I think that a lot of the presented experiments favor methods that explicitly learn a dynamics model (which is policy agnostic) rather than a Q-function (which is conditioned on a policy). Performing experiments where there are limited/local perturbations to the dynamics is fine, but in my opinion the authors should tone down their interpretation of the results (e.g. \"(...) model-based planning demonstrates superior generalization to novel layouts, better trajectory stitching, and greater data efficiency\")\n\n- To the best of my understanding, the hyperparameters for the baselines are not tuned (the authors use the default values from OGBench), while the proposed method uses separate hyperparameter settings per environment. If the baselines were tuned via the same procedure as the proposed method the results would be more convincing \n\n- Why not use any model-based RL baselines? For example, TD-MPC2 [1] was shown to work in an offline setup and also uses a learned model, but there are a variety of other papers\n\nNitpicks:\n\n- I would highly prefer to have metrics instead of colored stars in Table 1 - it is hard to say what \"average performance\" or other actually mean\n\n- There is a lot going on in Figure 1\n\n- I do not like that the authors motivate their work with a broad comparison of RL and OC in the offline setting, but end up using a single model learning OC algorithm which is also a method proposed in this work. I think that the introduction would work better if authors just said that they propose an OC/model learning method which they will benchmark against RL\n\n- The first contribution relating to the proposed environments and data generating process could be documented better. Where is the \"granular control\" described?", "was_revised": true, "final_justification": "This paper offers a joint analysis of OC and RL in the context of reward-free RL. Although the technical novelty is limited, I find the number of experiments satisfactory, and most of my reservations concern the description of experiments and the use of strong language in describing the results - the authors have promised to adjust the camera-ready version accordingly. As such, I recommend acceptance.", "rebuttal": "### Weaknesses\n\n> **I think that the authors should be more explicit about what \"new task\" and \"new environment\" actually mean in terms of the dynamics of the MDP.**\n---\n\nThank you for pointing this out. We will add this description to the paper. \nBy new task, we mean any task defined on the same environment, with the same state/action space and dynamics. We picked avoidance task as it's a natural task for navigation, and showcases flexibility of PLDM which GCRL methods don't have.\nIn terms of new environment, we only consider layout perturbations, and assume that there exists a distribution of layouts, some samples from which are seen during training. During inference, unseen samples are used for evaluation.\n\n> **In general, I think that a lot of the presented experiments favor methods that explicitly learn a dynamics model (which is policy agnostic) rather than a Q-function (which is conditioned on a policy).**\n---\n\nIn this work, we aim to investigate what methods can learn well from reward-free offline data. To do well with this data, methods need to be able to generalize to new tasks. We disagree that this favors dynamics models specifically, but we do agree that this setting points out an important limitation in most commonly used value-based methods in RL: the policy collecting the data may be completely irrelevant to the downstream task, which impedes Q-learning. In this paper, we argue that, instead of Q-learning, a promising way to tap into reward-free offline data is learning a dynamics model and planning (PLDM), as it's better at handling low-quality data, and can be used to solve different tasks after training.\n\n> **To the best of my understanding, the hyperparameters for the baselines are not tuned (the authors use the default values from OGBench), while the proposed method uses separate hyperparameter settings per environment. If the baselines were tuned via the same procedure as the proposed method the results would be more convincing **\n---\n\nWe do tune baseline hyperparameters for diverse maze, see appendix I.1.2 for the exact valus.\nFor two rooms, we tuned hyperparameters for the good-quality data, and used them throughout the rest of the experimetns. We are currently running hyperparameter tuning for experiments with sequence length set to 17, as tuning all methods hyperparameters for each individual setting will be prohibitively expensive. The experiments are still running for GCBC, HIQL, CRL. We didn't tune HILP as it already achieves perfect performance. We did observe a boost in performance for GCIQL. We will update results of GCIQL and other methods as the experimetns complete.\n\n| Method   | Success Rate Before Tuning | Success Rate After Tuning (single seed) |\n| -------- | -------------------------- | --------------------------------------- |\n| GCIQL    | 34.8  +-  4.4              | 48.0                                    |\n| PLDM<br> | 83.5  +- 3.7               | x                                       |\n\n> **Why not use any model-based RL baselines? For example, TD-MPC2 [1] was shown to work in an offline setup and also uses a learned model, but there are a variety of other papers**\n---\n\nWe ran additional experiments to test TDMPC2 performance in our setting. Since we are in the reward-free offline setting, we removed the reward and q-function learning components from TDMPC2, but kept the rest of the architecture unchanged. We found that TDMPC2 collapses without additional reward supervision and achieves 0% success rate.\n\n\n### Nitpicks\n\n> **I would highly prefer to have metrics instead of colored stars in Table 1 - it is hard to say what \"average performance\" or other actually mean**\n---\n\nThank you for pointing this out. We will redo Table 1 to have stars denote percentile performance of the methods.\n\n> **There is a lot going on in Figure 1**\n---\n\nThank you for pointing this out, we will simplify it by removing the results part of the figure.\n\n> **I do not like that the authors motivate their work with a broad comparison of RL and OC in the offline setting, but end up using a single model learning OC algorithm which is also a method proposed in this work. I think that the introduction would work better if authors just said that they propose an OC/model learning method which they will benchmark against RL**\n---\n\nThank you for this thoughtful comment. We agree that clarifying the framing in the introduction would improve the paper and will revise it to more clearly state that we are introducing a novel OC/model-learning method. That said, the broader motivation of the work remains to explore the relative strengths and weaknesses of RL and OC in the offline reward-free setting. PLDM was proposed specifically because existing OC methods do not work under these conditions. To support this comparison, we have included Dreamer as an additional OC baseline and added results for TD-MPC2 in this rebuttal.\n\n> **The first contribution relating to the proposed environments and data generating process could be documented better. Where is the \"granular control\" described?** \n---\n\nThank you for pointing this out. By granular control, we mean the flexibility the proposed environment affords in adjusting data collection properties, such as agent behavior (purely random or biased for one direction), state coverage (limit trajectories to one room), episode lengths, etc. The TwoRooms environment is also GPU-accelerated, meaning that we can quickly collect new datasets and experiment with many variations of the datasets. We will add this discussion to the paper in the sections describing the environments.\n### Questions\n\n> **Any hypothesis how would the model-based method perform with a smaller action repeat?**\n---\n\nSmaller action repeat, e.g. 2 instead of 4, will mean that covering the same distance will require twice the number of steps. This will require a longer planning horizon, which will in turn mean that the model error will accumulate more, leading to worse performance. On the other hand, we hypothesize that long-horizon tasks can be solved more easily with planning if action repeat is increased.\n\n> **How central is JEPA to the performance here? What would happen with the TD-MPC2 model instead of JEPA?**\n---\n\nWhile PLDM and TD-MPC2 use different methods for learning the representation, TD-MPC2 can be considered to be a JEPA, as it learns representations and the dynamics by predicting in the latent space and does not use reconstruction.\nAs other reviewers also pointed out the relevance of TDMPC2, we ran additional experiments to test its performance. Since we are in the reward-free offline setting, we removed the reward and q-function learning components from TDMPC2, but kept the rest of the architecture unchanged. We found that TDMPC2 collapses without additional reward supervision and achieves 0% success rate."}, {"review_id": "GdjfQ4GoIG", "reviewer": "Reviewer_XygG", "summary": "In this paper, different RL and control-based methods are evaluated on a suite of navigation tasks with offline dataset of varying quality and a latent dynamics model using JEPA is proposed for planning. Extensive experiments are conducted to show the advantages and disadvantages of different approaches, especially the ability of the latent dynamics model in handling suboptimal offline data and adapting to diverse environment configurations.", "questions": "1. While in the literature, many benchmarks with varying data quality, e.g., D4RL, have been proposed, it is unclear why these two new tasks were proposed as an alternative testbed, especially considering these two navigation tasks are actually simple. Can D4RL also be used in the experiments as the testbed? \n2. As can be seen from the results, except for generalization, the performance of PLDM is not competitive, and in many cases, HILP even performs better. Then what are the advantages of PLDM compared with HILP? Moreover, as mentioned, in the literature, the use of latent dynamics model for planning is actually not new, and more discussions and comparisons among these latent-state based approaches should be reported. \n3. As is well-known, the performances of model-based approaches rely heavily on the accuracy of the learned model, then how can we guarantee the accuracy of the model of PLDM?  Is it also the reason that these approaches are only compared in the two simple navigation tasks as more complex the task is, the less accuracy of the learned model may be. \n4. Experiments on some large-scale environments are welcome to show the scalability of PLDM.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nExtensive experiments are conducted to show the effectiveness of the proposed latent dynamics model along with a full discussion of the advantages and disadvantages of different RL and optimal control-based methods. Furthermore, two new navigation environments with granular control over the data generation process are proposed as the testbed. \n\nWeakness:\n1) Many works have been proposed to show relationships between RL and optimal-control based approaches, and latent dynamics models are also commonly used for planning in the literature, the contribution of the paper is incremental; \n2) In many cases, the performances, except for the generalization ability of the proposed latent dynamics, are not competitive compared with existing methods;\n3) These approaches are only verified on two simple navigation tasks, the scalability of the proposed model in some real-world applications is unclear.", "was_revised": true, "final_justification": "All the authors' claims on not using some more complex environments to verify PLDM are that the adopted simple tasks can be used to better generate data with various qualities. It is obviously true that the simple tasks can facilitate the stress-test of specific properties of the existing algorithms, but a better performance in simple tasks does not mean it can still achieve good results on more complex domains. Without the results on tasks such as the Kitchen tasks and the manipulation environments, where HILP performs good, it is hard to say that planning with a latent dynamics model proves to be a strong approach for handling suboptimal offline data and adapting to diverse environment configurations. So, my evaluation and the score of the paper are unchanged.", "rebuttal": "### Weaknesses:\n\n> **1. Many works have been proposed to show relationships between RL and optimal-control based approaches, and latent dynamics models are also commonly used for planning in the literature, the contribution of the paper is incremental;**\n---\n\nWe agree that the relationship between RL and optimal control has been explored since the very inception of RL, and we do not claim novelty in either being the first to investigate the relationship between RL and control, or to propose planning. However, we argue that our proposed approach is novel because it considers the planning with latent-dynamics in the reward-free setting. Previous works that use planning, such as TD-MPC2, use reward signal. PLDM on the other hand doesn't use rewards and therefore is more flexible in terms of what data it can use, and what tasks it can be applied to downstream. Furthermore, our work includes comprehensive empirical evaluations across multiple datasets with varying data quality, which, as reviewers saB6, 2Ba1, and WZKp noted, provide valuable insights into the strengths and limitations of existing methods in the reward-free offline regime.\nWe would be happy to clarify or expand on any part of our contribution, and if there are specific related works we’ve overlooked, we would gladly cite them upon suggestion.\n\n> **2.  In many cases, the performances, except for the generalization ability of the proposed latent dynamics, are not competitive compared with existing methods;**\n---\n\nAside from generalization, PLDM exhibits stronger robustness to data quality that GCRL methods across all experiments. Additionally, although HILP has competitive performance with PLDM on sequence length and data quality, PLDM is significantly more data efficient.\n    \n> **3. These approaches are only verified on two simple navigation tasks, the scalability of the proposed model in some real-world applications is unclear.**\n---\n\nWhile we agree that extending our expeirments to more complicated environments would be valuable, we believe we already provide strong enough evidence with experiments spanning both visual and proprioceptive domains, as also mentioned by reviewer 3zum. These simple environments are designed in a way to have a minimally complex setting to stress-test methods for offline reward-free data, and we believe these environments achieve this goal -- we see existing methods fail in unexpected ways. Additionally, granular control of data generation afforded by the simple environments we consider would be much harder in more complicated environments like manipulation. \n\n### Questions:\n\n> **1. Why another benchmark, and not D4RL?**\n---\n\nWhile D4RL is indeed a great benchmark for offline RL, it is not suitable for the reward-free context, as each environment there only has one task. For example, the navigation tasks considered in D4RL only have one goal state in the maze. Our Two-Rooms and Diverse Maze environments, while simple, sample the goal location randomly, making it much more challenging.\n\n> **2. As can be seen from the results, except for generalization, the performance of PLDM is not competitive, and in many cases, HILP even performs better. Then what are the advantages of PLDM compared with HILP? Moreover, as mentioned, in the literature, the use of latent dynamics model for planning is actually not new, and more discussions and comparisons among these latent-state based approaches should be reported.**\n---\n\nWe appreciate the reviewer’s question. While HILP demonstrates strong performance in stitching and on low-quality data, PLDM performs competitively in these settings and brings distinct advantages that make it a valuable contribution. In particular, PLDM is significantly more data-efficient (Section 4.3), and it shows stronger generalization to both unseen tasks (Section 4.6) and unseen layouts (Section 4.8), which is crucial for scalable offline learning.\nWe agree that latent dynamics models for planning have been explored in prior work. However, to the best of our knowledge, PLDM is the first end-to-end method to systematically apply this paradigm in the offline, reward-free setting. We appreciate the reviewer’s suggestion to expand the discussion of related latent-state planning approaches and will include a more detailed comparison in the final version of the paper.\n\n>  **3. As is well-known, the performances of model-based approaches rely heavily on the accuracy of the learned model, then how can we guarantee the accuracy of the model of PLDM? Is it also the reason that these approaches are only compared in the two simple navigation tasks as more complex the task is, the less accuracy of the learned model may be.**\n---\n\nPLDM relies on model ensembles to measure dynamics model uncertainty (see Eq. 6 in the paper). While this is not a perfect measure of model accuracy, it can serve as its proxy during planning, and limit considered actions to ones where the dynamics model is accurate. We already show this can work on ant navigation (see section 4.7), an environment with relative complex dynamics. Overall, we believe that uncertainty estimation will enable PLDM to scale to even more complicated dynamics.\n\n> **4. Experiments on some large-scale environments are welcome to show the scalability of PLDM.**\n---\nWe agree that further experiments are helpful to confirm scalability of PLDM to other environments. However, we would like to reiterate that current experiments with simple environments are design to stress-test specific properties of the existing algorithms for learning from offline reward-free data. While the proposed point-mass environments are simple, they showcase strengths and weaknesses of the existing methods, and uncover unexpected trends. In addition, our results on AntMaze—where the agent is a quadruped rather than a point mass—offer promising early evidence that PLDM can extend to settings with more complex control dynamics."}, {"review_id": "B5Z8N3dJLb", "reviewer": "Reviewer_WZKp", "summary": "This paper conducts an empirical study to compare model-based planning with goal-conditioned offline RL on learning from offline reward-free datasets. The main method being studied is Planning with Latent Dynamics Models (PLDM), which applies MPPI to a JEPA-style latent dynamics models, using latent distance as cost function to plan for goal-reaching behaviors. The baselines being compared to include GC-IQL, HIQL, HILP and contrastive RL. The main comparisons are done in a proprietary Two-Rooms environment, where a point agent needs to reach a goal. The paper analyzes the behaviors of each method under various levels of data assumptions. Other claims regarding performances in high-dimensional action spaces and generalization to new environments are evaluated on AntMaze and PointMaze respectively. The paper makes numerous findings, a few notable ones listed below:\n\n1. All methods perform well given rich data with good coverage.\n2. HILP performs best where significant stitching of trajectories is required, but it is not data efficient.\n3. PLDM is data efficient and generalizes best to new tasks and environment designs.\n\nThese findings provide valuable insights for researchers and practitioners to advance the field of planning with world models.", "questions": "1. In the dreamer baseline in Appendix F, what's the planning cost? Is it pixel distance?\n2. Can you include comparisons on more complex domains such as manipulation? An example could be franka kitchen from D4RL, which provides datasets of varying quality and can be cast as a goal-reaching problem.", "limitations": "The main limitation of this paper, as the authors addressed at the end, is the simplicity of the environments. All the evaluations are in navigation environments with fully observed states. Having results on manipulation problems could be of interest to the broader research community.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper conducts extensive experiments using simple, representative methods of each algorithmic class on simple domains. \n2. The results are summarized into clear messages and include positive and negative results. This provides a good guideline for researcher and practitioners alike.\n3. Specifically, the finding that model-based planning generalizes well to unseen environment designs (e.g. new maze structures), whereas HILP fail to do so, is interesting and opens up potential future research directions\n\n**Weaknesses**\n1. The evaluation domains have limited difficulty, and it is unclear how much the claims extend to more complex domains.\n2. Some claims presented in the paper are overstated. For example, in Section 4.6, the paper studies generalization to new tasks by inverting the sign of the cost function, converting goal-reaching tasks to goal-avoidance tasks. From this one experiment, the paper claims that PLDM \"can generalize well to a new task.\" This is vastly overstated, as the results might stem from the idiosyncrasies of the environment dynamics.\n3. The paper only compares to a single instantiation of model-based planning with a JEPA-style model (a dreamer comparison is included in the appendix in limited domains). Again, it is unclear how much the results transfer to model-based planning with other model architectures.", "was_revised": true, "final_justification": "The authors have addressed my questions regarding the model-based RL baseline, but they have not added experiments in more complex environments. In its current form, the paper offers bits of useful insights into model-based planning. However, it lacks a clear central claim, and the experiments are confined to toy domains. I therefore recommend borderline acceptance, though I would not object to a rejection.", "rebuttal": "### Weaknesses\n\n> **1. The evaluation domains have limited difficulty, and it is unclear how much the claims extend to more complex domains.**\n---\n\nWe agree that further experiments are helpful to confirm scalability of PLDM to other environments. However, we would like to reiterate that current experiments with simple environments are design to stress-test specific properties of the existing algorithms for learning from offline reward-free data. While the proposed point-mass environments are simple, they showcase strengths and weaknesses of the existing methods, and uncover unexpected trends. In addition, our results on AntMaze—where the agent is a quadruped rather than a point mass—offer promising early evidence that PLDM can extend to settings with more complex control dynamics.\n\n>  **2. Some claims presented in the paper are overstated. For example, in Section 4.6, the paper studies generalization to new tasks by inverting the sign of the cost function, converting goal-reaching tasks to goal-avoidance tasks. From this one experiment, the paper claims that PLDM \"can generalize well to a new task.\" This is vastly overstated, as the results might stem from the idiosyncrasies of the environment dynamics.**\n---\n\nThank you for pointing this out. We will tone down this claim. However, we'd like to emphasize that this experiment serves to showcase that PLDM can be adapted to an entirely different task just by changing the planning objective -- an ability that GCRL methods lack, no matter how simple the task is. \n\n\n>  **3. The paper only compares to a single instantiation of model-based planning with a JEPA-style model (a dreamer comparison is included in the appendix in limited domains). Again, it is unclear how much the results transfer to model-based planning with other model architectures.**\n---\n\nWhile it's true that we only consider a single instantiation of a model-based algorithm, we did try Dreamer-style reconstruction, and found performance very poor even with good-quality data. Additionally, prompted by this review, we conducted experiments with TDMPC2, and found that it performs poorly or collapses completely.\nExisting model-based methods like Dreamer and TDMPC2 heavily rely on the provided reward information in the data, and fail when the data is not available. Thus, comparing PLDM to these methods is not an apples to apples comparison. We are considering the reward-free offline setting, which, while very important, hasn't been explored in the literature as much as learning from rewards was.\n\n### Questions\n\n> **In the dreamer baseline in Appendix F, what's the planning cost? Is it pixel distance?**\n---\n\nWe used the latent representation space distance, just like we did for PLDM.\n\n>  **Can you include comparisons on more complex domains such as manipulation? An example could be franka kitchen from D4RL, which provides datasets of varying quality and can be cast as a goal-reaching problem.**\n---\n\nWe are running experiments with manipulation, but we are afraid that this will take longer than the duration of the rebuttal. Also, as mentioned above, we believe the existing experiments already support the message we are conveying in this paper."}, {"review_id": "djJ2Znqkc9", "reviewer": "Reviewer_2Ba1", "summary": "This paper investigates the comparative strengths of reinforcement learning (RL) and optimal control methods in the offline setting, specifically when agents must learn from trajectories only. The authors systematically evaluate goal-conditioned RL approaches against a control-based method that trains a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) for planning. Through experiments on a suite of navigation tasks with varying offline dataset qualities, the study reveals that goal-conditioned model-free algorithms performs well with abundant, high-quality data (especially HILP), while model-based planning exhibits superior generalization to novel environments, enhanced trajectory stitching capabilities, and greater data efficiency. The work provides an empirical analysis of these paradigms under different data and environment conditions, contributing to the understanding of their applicability in reward-free offline learning.", "questions": "* Currently, PLDM replans every step, leading to increased computational cost. Could the authors present an experimental analysis demonstrating how performance (e.g., success rate, trajectory quality) varies with different replanning horizons (e.g., replanning every 2nd, 4th, or 8th step)? This investigation would provide crucial insights into the practical deployability and computational trade-offs of the method\n* In Sec. 4.5 the authors come to the conclusion that HILP and PLDM generalize better to out of distribution goals. Did the authors investigate the reasons why this is the case?\n* Can the authors explain the difference between PLDM's inferred plan and the actual trajectory in Sec. 4.8?", "limitations": "The authors already describe that their navigation setups limit the overall results of the paper despite arguing directly that this will somehow not be the case. I would add that their choice of baseline algorithms to which PLDM is compared to further limits the generality of the arguments presented.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n* The paper addresses a large number of questions regarding the performance of different paradigms in goal-conditioned offline learning. The empirical setup explores various factors such as dataset size, data diversity, trajectory stitching, and environment variability, contributing to a comprehensive experimental analysis.\n* The paper offers a valuable benchmark of goal-conditioned RL algorithms and a systematic comparison with model-based planning, which is relevant to the community. The insights into the strengths of model-based planning in generalization and data efficiency for these offline settings are impactful and can guide future research.\n* The paper is generally well-written and organized, enhancing readability (except Sec. 4.8; see below). The inclusion of section summaries at the end of each major section significantly improves comprehension, making the content accessible to the reader.\n\n### Weaknesses\n* Despite answering numerous questions regarding performance, the paper does not adequately delve into why a particular method performs better under specific conditions, limiting the depth of insight. Further investigation into the underlying mechanisms driving observed performance differences would strengthen the analysis.\n* The baseline comparisons are somewhat narrow, focusing exclusively on model-free GCRL algorithms (which in my opinion all use an implicit reward with the given goal). I do not share the opinion that goal-conditioned algorithms are the only \"reward-free\" algorithms and by the title of the paper I would expect a broader range of baseline methods. In Sec. 3.2 the authors mention that they compare to zero-shot methods despite the fact that all baseline methods are goal-conditioned. The absence of other relevant approaches, such as skill-learning methods (e.g., Choreographer) or other unsupervised RL approaches, is a notable limitation. Furthermore, given that the paper's examined method is model-based and emphasizes the advantages of test-time planning, a comparison against a Dreamer-style MBRL algorithm like Director (from Hafner et al., Deep Hierarchical Planning from Pixels) or at least another simple MBRL baseline would have provided a more complete and fair evaluation.\n* The tasks utilized in the experiments lack sufficient variation, as acknowledged in the limitations section. While reaching goals may be feasible for MPC in these relatively simple navigation environments, the generalizability of the findings to more complex tasks with larger search spaces (e.g., reaching a diamond in Minecraft) remains questionable. Integrating more diverse and challenging tasks would substantially strengthen the claim of generality.\n* The proposed PLDM (Planning with Latent Dynamics Models) replans at every step, which, as shown in Appendix E, results in approximately four times slower computation. The paper would benefit significantly from an investigation into how performance changes with larger replanning horizons (e.g., replanning every 4th step), addressing a practical efficiency concern.\n* Section 4.8 exhibits an unusual arrangement. It refers to Figure 2, which is placed at the very beginning of the paper without prior mention or context. Additionally, Section 4.8 relies heavily on Figure 8 and Table 4, both located in the Appendix, to such an extent that they are crucial for understanding the section, not merely supplementary results. This organization impedes flow and clarity.", "was_revised": true, "final_justification": "I will update my score since the paper is not a clear reject for me and the authors provided good explanations during the rebuttal. Thus, I update my score to borderline reject. But in my opinion, for such a paper the baselines are all of a similar type with not much variation. Furthermore, the simple setting (i.e. navigation task) cannot really show the strengths and weaknesses of PLDM as good as the authors claim as navigation tasks with such small horizons and in which not much exploration is needed are usually more beneficial for planning-based methods. Therefore, I am not convinced that the experiments \"stress-test\" PLDM. Thus, I cannot really recommend accepting the paper. But I will not be against a final accept if the AC and the reviewers agree to accept the paper.", "rebuttal": "### Weaknesses\n\n> **1. Despite answering numerous questions regarding performance, the paper does not adequately delve into why a particular method performs better under specific conditions, limiting the depth of insight. Further investigation into the underlying mechanisms driving observed performance differences would strengthen the analysis.**\n---\n\nThank you for pointing that out. As you mentioned, the paper is empirical in nature, with limited analysis of particular reasons for failure. We believe that such analysis is beyond the scope of this work. However, we do analyze why HILP fails on layout generalization, and show that the learned distance function used to train the policy does not generalize well to new layouts (section H).\n\n> **2. The baseline comparisons are somewhat narrow, focusing exclusively on model-free GCRL algorithms.**\n---\n\nWe do not solely focus on model-free GCRL algorithms, as HILP is not a goal-conditioned method, and is instead an unsupervised RL method. We also tested Dreamer, and reconstruction-based representation learning, and got poor results even with good data, and therefore didn't test on more settings (see Table 6). Additionally, prompted by questions in this review, we conducted experiments with TDMPC2, and found that due to lack of reward information, TDMPC2 representations completely collapse, leading to 0% success rate.\n\n> **3. The tasks utilized in the experiments lack sufficient variation, as acknowledged in the limitations section.**\n---\n\nWhile we agree that further experiments are needed to confirm scalability of PLDM, we would like to reiterate that current experiments with simple environments are design to stress-test specific properties of the existing algorithms for learning from offline reward-free data. While the proposed point-mass environments are simple, they showcase strengths and weaknesses of the existing methods, and uncover unexpected trends. Additionally, we test on ant navigation, an environment that has much more complicated dynamics due to the agent being not a point mass but a 4-legged ant, which showcases PLDM's scalability to complex dynamics.\n\n> **4. The proposed PLDM (Planning with Latent Dynamics Models) replans at every step, which, as shown in Appendix E, results in approximately four times slower computation. The paper would benefit significantly from an investigation into how performance changes with larger replanning horizons (e.g., replanning every 4th step), addressing a practical efficiency concern.**\n---\n\nWe believe there must have been an understanding. We would like to point out that Appendix E tests exactly that, and shows that replanning every 4th step as opposed to every step, retains 95% of the original performance while being only 20% slower than policy-based methods. Please let us know if this is not what you were looking for, we are happy to answer any further questions.\n\n> **5. Section 4.8 exhibits an unusual arrangement.**\n---\n\nWhile we agree the figures in the appendix aid in understanding this section, we disagree that they are crucial:\n- Table 4 essentially shows that all the methods get 100% performance, which is exactly what is said in the main text, making the table redundant. \n- Figure 7 is a summarized version of Figure 8 and shows just one layout example. Figure 8 shows trajectories for each method on a separate image, and shows more layouts with different edit distances. While Figure 8 helps in understanding the experiment, we believe Figure 7 is enough to convey the main point. Additionally, Figure 8 is way too large to be able to fit in the main text.\n\nFigure 2 is shown in the beginning of the paper because we believe this is the most interesting result, but we agree that placing it directly in the sction discussing it will be more convenient for the reader. We will move this figure to section 4.8.\n\n### Questions\n\n> **Currently, PLDM replans every step, leading to increased computational cost. Could the authors present an experimental analysis demonstrating how performance (e.g., success rate, trajectory quality) varies with different replanning horizons (e.g., replanning every 2nd, 4th, or 8th step)? This investigation would provide crucial insights into the practical deployability and computational trade-offs of the method**\n---\nSee response to Weakness 4.\n\n> **In Sec. 4.5 the authors come to the conclusion that HILP and PLDM generalize better to out of distribution goals. Did the authors investigate the reasons why this is the case?**\n---\n\nWe hypothesize that GCRL methods fail on out-of-distribution goals due to trajectory stitching. PLDM can easily stitch trajectories from the training data as the dynamics model predicts one step ahead, and thereofre will learn transitions starting from the same state s but with different actions, e.g. $(s, a_1, s_1')$ and $(s, a_2, s'_2)$ if such transitions are present within any two trajectories in the dataset. HILP works well because it is not technically goal-conditioned, but instead uses latent-space direction to condition the policy (see Figure 1 in [HILP paper](https://arxiv.org/abs/2402.15567)). Covering all existing directions in latent space is easier than all existing goals in the observation space, therefore HILP can reach out of distribution goals as long as they can be reached by following known directions in the latent space. Offline GCRL on the other hand struggles with stitching, as in order to enable stitching, these methods rely on random goal sampling and dynamic programming. The larger the environment, the harder it is to sample reasonable goals to encourage value and policy learning, making such methods fail.\n\n> **Can the authors explain the difference between PLDM's inferred plan and the actual trajectory in Sec. 4.8?**\n---\n\nThe inferred plan is constructed at timestep t=0. We construct a new plan every time we take a step, therefore the resulting executed trajectory may differ from the first planned trajectory due to randomness in the planning algorithm (MPPI samples candidate trajectories randomly) and due to small differences between model predictions and actual environment observations."}, {"review_id": "BGYP4yBNVK", "reviewer": "Reviewer_saB6", "summary": "The paper studies how to learn general purpose control policies from offline, reward-free trajectories (focusing specifically on navigation tasks). It compares five recent model-free reinforcement learning approaches (HIQL, GCIQL, CRL, GCBC, HILP) with a new method, Planning with a Latent Dynamics Model (PLDM).  PLDM trains a JEPA-like latent dynamics model from pixel observations using self-supervised objectives, then performs model-predictive control in the latent space. Experiments demonstrate that PLDM is sample efficient, can handle stitching of random trajectories, and generalized best to unseen layouts.", "questions": "1. Besides the decoder-based and Dreamer-style ablations, did you test other reconstruction-free SSL objectives (e.g., BYOL, VICReg) within the same planning loop?\n2. Can you quantify how much of PLDM’s gain derives from the JEPA encoder versus from having any learned latent model at all? (or provide a rough heuristic as to what you'd expect?)\n3. How sensitive is performance to small visual perturbations (Gaussian noise, random crops, mild viewpoint changes)?\n4. What are training wall-clock times and peak GPU memory for PLDM versus HILP and HIQL on Two-Rooms and Ant-UMaze?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nSystematic analysis of reward-free offline learning is timely and useful for the community; the dataset generator and evaluation protocol will likely be reused. PLDM provides a compelling demonstration that model-based planning can outperform strong model-free methods when data are sparse or misaligned.\n\nWeaknesses:\nThis paper focuses on navigation tasks. While these tasks are challenging, my main concern is to what extent do these studies replicate to other tasks. Also quite recently there have been other latent planning style architectures for example:\n\nKong et al. Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference: https://proceedings.neurips.cc/paper_files/paper/2024/hash/df22a19686a558e74f038e6277a51f68-Abstract-Conference.html\n\nNoh et al. Latent Adaptive Planner for Dynamic Manipulation: https://arxiv.org/abs/2505.03077\n\nThese approaches lean more into variational-bayes (MLE-style) approaches for incorporating the latent parameters for planning.\nIt's unclear if the advantage due to the PLDM method is because of the incorporation of JEPA or if this is an advantage we'd expect to see when incorporating latent parameters into these models in general.", "was_revised": true, "final_justification": "The author's rebuttal were able to properly address my concerns and make me more confident in accepting this work into NeurIPS.", "rebuttal": "### Weaknesses\n  \n> **This paper focuses on navigation tasks. While these tasks are challenging, my main concern is to what extent do these studies replicate to other tasks.**\n---\n\nWhile we agree that further experiments are needed to confirm scalability of PLDM to other environments, we would like to reiterate that current experiments with simple environments are design to stress-test specific properties of the existing algorithms for learning from offline reward-free data. While the proposed point-mass environments are simple, they showcase strengths and weaknesses of the existing methods, and uncover unexpected trends. Additionally, we test on ant navigation, an environment that has much more complicated dynamics due to the agent being not a point mass but a 4-legged ant, which showcases PLDM's scalability to complex dynamics.\n\n \n> **Also quite recently there have been other latent planning style architectures for example...**\n---\n\nThank you for pointing out these works. We have added them to the related works section.\nHowever, we couldn't directly compare these to PLDM as [Latent Adaptive Planner for Dynamic Manipulation](https://arxiv.org/abs/2505.03077) does not provide the implementation, and [Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference](https://proceedings.neurips.cc/paper_files/paper/2024/hash/df22a19686a558e74f038e6277a51f68-Abstract-Conference.html) requires reward signal, although only per-trajectory, making direct comparison impossible.\n\n### Questions\n\n> **Besides the decoder-based and Dreamer-style ablations, did you test other reconstruction-free SSL objectives (e.g., BYOL, VICReg) within the same planning loop?**\n---\n\nIn addition to the experiments in the paper, we conducted an additional experiment with TDMPC2 architecture and representation learning objective. Because TDMPC2 relies on rewards to learn a Q-function and a policy, we removed those components, but kept the rest unchanged. We found that TDMPC2 without rewards completely collapses, leading to 0% success rate even with good-quality data. We added inverse dynamics learning to TDMPC2, and found that, while avoiding collapse, such model still achieves 0% success rate with good data.\n\n> **Can you quantify how much of PLDM’s gain derives from the JEPA encoder versus from having any learned latent model at all? (or provide a rough heuristic as to what you'd expect?)**\n---\n\nJEPA architecture only stipulates that the representations and dynamics are learned without reconstruction, and therefore encapsulates TDMPC2 as well. As discussed above, we found that VICReg-based objective performs much better, with TDMPC2 objective failing completely. If we compare JEPA vs non-JEPA approaches, we can look at the comparison between PLDM and reconstruction-based representation learning in Table 6 in the paper. We see that reconstruction-free approaches get less than 30% success rate with good-quality data, while PLDM achieves ~97%.\n\n>  **How sensitive is performance to small visual perturbations (Gaussian noise, random crops, mild viewpoint changes)?**\n---\n\nWe hypothesize that unless the models are trained with such perturbation, they will fail. We are currently running experiments to test if PLDM can work with these perturbations without seeing these transformations during training, we will report the results as soon as they are available.\n### Formatting issues\nThank you for pointing out the typos, we have updated the manuscript accordingly."}]

Author rebuttal:

Meta-review: This paper presents a systematic comparison between model-free reinforcement learning (RL) and model-based planning for solving tasks from reward-free offline data. The authors propose Planning with Latent Dynamics Models (PLDM), a method that learns a latent dynamics model with a Joint Embedding Predictive Architecture (JEPA) and uses it for planning. The study evaluates various approaches on a suite of navigation tasks, analyzing performance across different factors like data quality, data efficiency, and generalization to new environments and tasks.

Strengths: 
- The reviewers praised the paper for tackling an important and timely problem in offline RL. There was a strong consensus that the experimental evaluation is extensive and valuable, providing a comprehensive analysis that is useful to the community. 
- The clarity of the writing and the organization of the paper were also highlighted as significant strengths, with reviewers noting that the results are summarized into clear messages .

Weaknesses: 
- The primary concern, shared by multiple reviewers, is the limited complexity and variety of the evaluation domains. The experiments are largely confined to simple navigation tasks, which raises questions about the scalability of the findings and the proposed method (PLDM) to more complex, real-world problems 

Recommendation: This paper has a mixed reception/ While the concerns about the limited evaluation environments are valid, the paper provides a well-executed and valuable empirical study on a very relevant problem. The authors' careful, controlled experiments successfully "stress test" different methods and yield clear insights. Their commitment to refining their claims in the final version significantly strengthens the paper. Balancing the valuable empirical contributions against the limited scope, the paper leans towards acceptance.
Decision: Accept (poster)