Title: Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: self-supervised learning

representation learning

dynamic learning

offline rl

reward-free RL

goal-conditoined RL

zero-shot RL
Abstract: A long-standing goal in AI is to develop agents capable of solving diverse tasks across a range of environments, including those never seen during training. Two dominant paradigms address this challenge: (i) reinforcement learning (RL), which learns policies via trial and error, and (ii) optimal control, which plans actions using a known or learned dynamics model. However, their comparative strengths in the offline setting—where agents must learn from reward-free trajectories—remain underexplored. In this work, we systematically evaluate RL and control-based methods on a suite of navigation tasks, using offline datasets of varying quality. On the RL side, we consider goal-conditioned and zero-shot methods. On the control side, we train a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning. We investigate how factors such as data diversity, trajectory quality, and environment variability influence the performance of these approaches. Our results show that model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts and is more data-efficient, while achieving trajectory stitching performance comparable to leading model-free methods. Notably, planning with a latent dynamics model proves to be a strong approach for handling suboptimal offline data and adapting to diverse environments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how to learn general purpose control policies from offline, reward-free trajectories (focusing specifically on navigation tasks). It compares five recent model-free reinforcement learning approaches (HIQL, GCIQL, CRL, GCBC, HILP) with a new method, Planning with a Latent Dynamics Model (PLDM).  PLDM trains a JEPA-like latent dynamics model from pixel observations using self-supervised objectives, then performs model-predictive control in the latent space. Experiments demonstrate that PLDM is sample efficient, can handle stitching of random trajectories, and generalized best to unseen layouts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nSystematic analysis of reward-free offline learning is timely and useful for the community; the dataset generator and evaluation protocol will likely be reused. PLDM provides a compelling demonstration that model-based planning can outperform strong model-free methods when data are sparse or misaligned.\n\nWeaknesses:\nThis paper focuses on navigation tasks. While these tasks are challenging, my main concern is to what extent do these studies replicate to other tasks. Also quite recently there have been other latent planning style architectures for example:\n\nKong et al. Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference: https://proceedings.neurips.cc/paper_files/paper/2024/hash/df22a19686a558e74f038e6277a51f68-Abstract-Conference.html\n\nNoh et al. Latent Adaptive Planner for Dynamic Manipulation: https://arxiv.org/abs/2505.03077\n\nThese approaches lean more into variational-bayes (MLE-style) approaches for incorporating the latent parameters for planning.\nIt's unclear if the advantage due to the PLDM method is because of the incorporation of JEPA or if this is an advantage we'd expect to see when incorporating latent parameters into these models in general.\n\nQUESTIONS:\n1. Besides the decoder-based and Dreamer-style ablations, did you test other reconstruction-free SSL objectives (e.g., BYOL, VICReg) within the same planning loop?\n2. Can you quantify how much of PLDM’s gain derives from the JEPA encoder versus from having any learned latent model at all? (or provide a rough heuristic as to what you'd expect?)\n3. How sensitive is performance to small visual perturbations (Gaussian noise, random crops, mild viewpoint changes)?\n4. What are training wall-clock times and peak GPU memory for PLDM versus HILP and HIQL on Two-Rooms and Ant-UMaze?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the comparative strengths of reinforcement learning (RL) and optimal control methods in the offline setting, specifically when agents must learn from trajectories only. The authors systematically evaluate goal-conditioned RL approaches against a control-based method that trains a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) for planning. Through experiments on a suite of navigation tasks with varying offline dataset qualities, the study reveals that goal-conditioned model-free algorithms performs well with abundant, high-quality data (especially HILP), while model-based planning exhibits superior generalization to novel environments, enhanced trajectory stitching capabilities, and greater data efficiency. The work provides an empirical analysis of these paradigms under different data and environment conditions, contributing to the understanding of their applicability in reward-free offline learning.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* The paper addresses a large number of questions regarding the performance of different paradigms in goal-conditioned offline learning. The empirical setup explores various factors such as dataset size, data diversity, trajectory stitching, and environment variability, contributing to a comprehensive experimental analysis.\n* The paper offers a valuable benchmark of goal-conditioned RL algorithms and a systematic comparison with model-based planning, which is relevant to the community. The insights into the strengths of model-based planning in generalization and data efficiency for these offline settings are impactful and can guide future research.\n* The paper is generally well-written and organized, enhancing readability (except Sec. 4.8; see below). The inclusion of section summaries at the end of each major section significantly improves comprehension, making the content accessible to the reader.\n\n### Weaknesses\n* Despite answering numerous questions regarding performance, the paper does not adequately delve into why a particular method performs better under specific conditions, limiting the depth of insight. Further investigation into the underlying mechanisms driving observed performance differences would strengthen the analysis.\n* The baseline comparisons are somewhat narrow, focusing exclusively on model-free GCRL algorithms (which in my opinion all use an implicit reward with the given goal). I do not share the opinion that goal-conditioned algorithms are the only \"reward-free\" algorithms and by the title of the paper I would expect a broader range of baseline methods. In Sec. 3.2 the authors mention that they compare to zero-shot methods despite the fact that all baseline methods are goal-conditioned. The absence of other relevant approaches, such as skill-learning methods (e.g., Choreographer) or other unsupervised RL approaches, is a notable limitation. Furthermore, given that the paper's examined method is model-based and emphasizes the advantages of test-time planning, a comparison against a Dreamer-style MBRL algorithm like Director (from Hafner et al., Deep Hierarchical Planning from Pixels) or at least another simple MBRL baseline would have provided a more complete and fair evaluation.\n* The tasks utilized in the experiments lack sufficient variation, as acknowledged in the limitations section. While reaching goals may be feasible for MPC in these relatively simple navigation environments, the generalizability of the findings to more complex tasks with larger search spaces (e.g., reaching a diamond in Minecraft) remains questionable. Integrating more diverse and challenging tasks would substantially strengthen the claim of generality.\n* The proposed PLDM (Planning with Latent Dynamics Models) replans at every step, which, as shown in Appendix E, results in approximately four times slower computation. The paper would benefit significantly from an investigation into how performance changes with larger replanning horizons (e.g., replanning every 4th step), addressing a practical efficiency concern.\n* Section 4.8 exhibits an unusual arrangement. It refers to Figure 2, which is placed at the very beginning of the paper without prior mention or context. Additionally, Section 4.8 relies heavily on Figure 8 and Table 4, both located in the Appendix, to such an extent that they are crucial for understanding the section, not merely supplementary results. This organization impedes flow and clarity.\n\nQUESTIONS:\n* Currently, PLDM replans every step, leading to increased computational cost. Could the authors present an experimental analysis demonstrating how performance (e.g., success rate, trajectory quality) varies with different replanning horizons (e.g., replanning every 2nd, 4th, or 8th step)? This investigation would provide crucial insights into the practical deployability and computational trade-offs of the method\n* In Sec. 4.5 the authors come to the conclusion that HILP and PLDM generalize better to out of distribution goals. Did the authors investigate the reasons why this is the case?\n* Can the authors explain the difference between PLDM's inferred plan and the actual trajectory in Sec. 4.8?\n\nLIMITATIONS:\nThe authors already describe that their navigation setups limit the overall results of the paper despite arguing directly that this will somehow not be the case. I would add that their choice of baseline algorithms to which PLDM is compared to further limits the generality of the arguments presented.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper conducts an empirical study to compare model-based planning with goal-conditioned offline RL on learning from offline reward-free datasets. The main method being studied is Planning with Latent Dynamics Models (PLDM), which applies MPPI to a JEPA-style latent dynamics models, using latent distance as cost function to plan for goal-reaching behaviors. The baselines being compared to include GC-IQL, HIQL, HILP and contrastive RL. The main comparisons are done in a proprietary Two-Rooms environment, where a point agent needs to reach a goal. The paper analyzes the behaviors of each method under various levels of data assumptions. Other claims regarding performances in high-dimensional action spaces and generalization to new environments are evaluated on AntMaze and PointMaze respectively. The paper makes numerous findings, a few notable ones listed below:\n\n1. All methods perform well given rich data with good coverage.\n2. HILP performs best where significant stitching of trajectories is required, but it is not data efficient.\n3. PLDM is data efficient and generalizes best to new tasks and environment designs.\n\nThese findings provide valuable insights for researchers and practitioners to advance the field of planning with world models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper conducts extensive experiments using simple, representative methods of each algorithmic class on simple domains. \n2. The results are summarized into clear messages and include positive and negative results. This provides a good guideline for researcher and practitioners alike.\n3. Specifically, the finding that model-based planning generalizes well to unseen environment designs (e.g. new maze structures), whereas HILP fail to do so, is interesting and opens up potential future research directions\n\n**Weaknesses**\n1. The evaluation domains have limited difficulty, and it is unclear how much the claims extend to more complex domains.\n2. Some claims presented in the paper are overstated. For example, in Section 4.6, the paper studies generalization to new tasks by inverting the sign of the cost function, converting goal-reaching tasks to goal-avoidance tasks. From this one experiment, the paper claims that PLDM \"can generalize well to a new task.\" This is vastly overstated, as the results might stem from the idiosyncrasies of the environment dynamics.\n3. The paper only compares to a single instantiation of model-based planning with a JEPA-style model (a dreamer comparison is included in the appendix in limited domains). Again, it is unclear how much the results transfer to model-based planning with other model architectures.\n\nQUESTIONS:\n1. In the dreamer baseline in Appendix F, what's the planning cost? Is it pixel distance?\n2. Can you include comparisons on more complex domains such as manipulation? An example could be franka kitchen from D4RL, which provides datasets of varying quality and can be cast as a goal-reaching problem.\n\nLIMITATIONS:\nThe main limitation of this paper, as the authors addressed at the end, is the simplicity of the environments. All the evaluations are in navigation environments with fully observed states. Having results on manipulation problems could be of interest to the broader research community.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, different RL and control-based methods are evaluated on a suite of navigation tasks with offline dataset of varying quality and a latent dynamics model using JEPA is proposed for planning. Extensive experiments are conducted to show the advantages and disadvantages of different approaches, especially the ability of the latent dynamics model in handling suboptimal offline data and adapting to diverse environment configurations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nExtensive experiments are conducted to show the effectiveness of the proposed latent dynamics model along with a full discussion of the advantages and disadvantages of different RL and optimal control-based methods. Furthermore, two new navigation environments with granular control over the data generation process are proposed as the testbed. \n\nWeakness:\n1) Many works have been proposed to show relationships between RL and optimal-control based approaches, and latent dynamics models are also commonly used for planning in the literature, the contribution of the paper is incremental; \n2) In many cases, the performances, except for the generalization ability of the proposed latent dynamics, are not competitive compared with existing methods;\n3) These approaches are only verified on two simple navigation tasks, the scalability of the proposed model in some real-world applications is unclear.\n\nQUESTIONS:\n1. While in the literature, many benchmarks with varying data quality, e.g., D4RL, have been proposed, it is unclear why these two new tasks were proposed as an alternative testbed, especially considering these two navigation tasks are actually simple. Can D4RL also be used in the experiments as the testbed? \n2. As can be seen from the results, except for generalization, the performance of PLDM is not competitive, and in many cases, HILP even performs better. Then what are the advantages of PLDM compared with HILP? Moreover, as mentioned, in the literature, the use of latent dynamics model for planning is actually not new, and more discussions and comparisons among these latent-state based approaches should be reported. \n3. As is well-known, the performances of model-based approaches rely heavily on the accuracy of the learned model, then how can we guarantee the accuracy of the model of PLDM?  Is it also the reason that these approaches are only compared in the two simple navigation tasks as more complex the task is, the less accuracy of the learned model may be. \n4. Experiments on some large-scale environments are welcome to show the scalability of PLDM.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors focus on the problem of learning control from a set of reward-free trajectories (i.e. offline data). In particular, the authors focus on a comparison of offline reward-free RL algorithms with the optimal control (OC) method, utilizing a learned JEPA model that the authors refer to as PLDM. The authors compare the algorithms along the following axes: transfer to new environments, transfer to new tasks, data efficiency, best-case performance, learning from a random policy, and trajectory stitching. The experiments are performed on a custom \"two-room\" task, AntMaze and PointMaze tasks. The authors show results in which that the proposed model either outperforms or performs similarly to the best baseline in all tested categories.\n\nSTRENGTHS AND WEAKNESSES:\nOverall I believe this is an interesting and valuable manuscript. My biggest issue is with the description of the \"generalization\" experiments and with a strong wording of the claims.\n\nStrengths:\n\n- Reward-free learning is an important direction\n- The paper is clearly written\n- Experiments on both vision and proprioceptive environments\n\nWeaknesses:\n\n- I think that the authors should be more explicit about what \"new task\" and \"new environment\" actually mean in terms of the dynamics of the MDP. For example, changing robot embodiment (e.g. ant -> humanoid or even ant -> ant with longer legs) is different than displacing walls in a maze. I think that this is especially important considering that the method proposed explicitly learns a dynamics model. To the best of my understanding, these perturbations have local impact on the dynamics model.\n\n- In general, I think that a lot of the presented experiments favor methods that explicitly learn a dynamics model (which is policy agnostic) rather than a Q-function (which is conditioned on a policy). Performing experiments where there are limited/local perturbations to the dynamics is fine, but in my opinion the authors should tone down their interpretation of the results (e.g. \"(...) model-based planning demonstrates superior generalization to novel layouts, better trajectory stitching, and greater data efficiency\")\n\n- To the best of my understanding, the hyperparameters for the baselines are not tuned (the authors use the default values from OGBench), while the proposed method uses separate hyperparameter settings per environment. If the baselines were tuned via the same procedure as the proposed method the results would be more convincing \n\n- Why not use any model-based RL baselines? For example, TD-MPC2 [1] was shown to work in an offline setup and also uses a learned model, but there are a variety of other papers\n\nNitpicks:\n\n- I would highly prefer to have metrics instead of colored stars in Table 1 - it is hard to say what \"average performance\" or other actually mean\n\n- There is a lot going on in Figure 1\n\n- I do not like that the authors motivate their work with a broad comparison of RL and OC in the offline setting, but end up using a single model learning OC algorithm which is also a method proposed in this work. I think that the introduction would work better if authors just said that they propose an OC/model learning method which they will benchmark against RL\n\n- The first contribution relating to the proposed environments and data generating process could be documented better. Where is the \"granular control\" described?\n\nQUESTIONS:\n- Any hypothesis how would the model-based method perform with a smaller action repeat?\n\n- How central is JEPA to the performance here? What would happen with the TD-MPC2 model instead of JEPA?\n\nLIMITATIONS:\nYes, though I believe that the authors should tone down their interpretation of the results", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}