Title: Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Offline Goal-Conditioned RL

Physics-informed Neural Networks
Abstract: Offline Goal-Conditioned Reinforcement Learning (GCRL) holds great promise for domains such as autonomous navigation and locomotion, where collecting interactive data is costly and unsafe. However, it remains challenging in practice due to the need to learn from datasets with limited coverage of the state-action space and to generalize across long-horizon tasks. To improve on these challenges, we propose a Physics-informed (Pi) regularized loss for value learning, derived from the Eikonal Partial Differential Equation (PDE) and which induces a geometric inductive bias in the learned value function. Unlike generic gradient penalties that are primarily used to stabilize training, our formulation is grounded in continuous-time optimal control and encourages value functions to align with cost-to-go structures. The proposed regularizer is broadly compatible with temporal-difference-based value learning and can be integrated into existing Offline GCRL algorithms. When combined with Hierarchical Implicit Q-Learning (HIQL), the resulting method, Eikonal-regularized HIQL (Eik-HIQL), yields significant improvements in both performance and generalization, with pronounced gains in stitching regimes and large-scale navigation tasks. Code is available at https://github.com/VittorioGiammarino/Eik-HIQL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nValue estimation is a hard problem that can directly affect the performance and generalization of policies learned via offline RL, especially in the goal-conditioned (GC) setting. Prior work has explored using a physics-informed (PI) regularizer, via the Hamilton-Jacobi-Bellman (HJB) equation, which requires explicit access to system dynamics. Instead, this paper proposes the use of the Eikonal PDE, which is model free and induces a distance field structure in the estimated value function. The Eikonal PDE relates a speed profile function to a travel time function between a state and goal, via its spatial gradient. A high speed profile results in low travel time, and vice versa. The authors justify the use of the Eikonal PDE by relating it to the HJB equation. Specifically, they show that the corresponding Hamiltonian in the HJB equation is upper-bounded by an Eikonal residual, with the travel time function set to the value function and making a specific choice of local speed profile involving the dynamics and cost functions. Using this connection, they propose a general regularization term inspired by the Eikonal residual. It leaves the speed profile as a general function which can be chosen by the practitioner. \n\nThey implement this regularized value function estimation protocol in the context of hierarchical implicit Q-learning (HIQL), an algorithm for offline GCRL, and evaluate the approach on environments from OGbench, which contains multiple goal-conditioned tasks. They consider three variants of their algorithm, making different choices for the speed profile term, including a constant, exponential, and linear function of distance to obstacles and goals. Additionally, they compare to a regularization term inspired by the HJB equation, as used in prior work. On a specific pointmaze environment, across two types of datasets and four maze sizes, they find their approach, Pi-HIQL with a constant speed profile, outperforms the other variants and the HJB baseline. They then compare Pi-HIQL with a constant speed profile to standard HIQL across multiple environments. We see Pi-HIQL making substantial gains across multiple environments, especially in the case where it must stitch together trajectories from the dataset. However, they find in contact-rich domains that their approach underperforms standard HIQL. Finally, they compare to other offline GCRL algorithms and show that their method does better in long-horizon and large-scale environments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Offline RL and goal-conditioned RL are both important frameworks where improved value function estimation can have a big impact, and this is well-motivated in the introduction of the paper.\n- The evaluations clearly show that their method outperforms both the unregularized version of HIQL and an HJB-inspired regularization term on many environments. They also consider three variants of their approach which make different choices for the speed profile function. And they compare to other choices of offline GCRL algorithms as well, making their evaluations fairly rigorous.\n- In cases where their method underperforms, it is well justified by the assumptions made with their regularization strategy. The authors are upfront about these limitations, showing clearly when this approach is most applicable and when it may not be as useful.\n- The connection to the HJB equation is well-formulated, and the evaluation does a good job comparing to an HJB-based regularization strategy.\n- Figure 1 provides a nice visualization of how the physics-informed regularizer can improve value function estimation in the goal-conditioned setting.\n\nWeaknesses:\n- Modifying the choice of regularization term is a fairly minor, straight-forward change, especially in the context of prior work that regularize with the HJB equation. However, it is well-motivated in the paper and evaluated well.\n- In section 5, the different choices of speed profiles make use of a distance function, d(s), which is never clearly defined. It is only visualized for the pointmaze environment in Figure 4. It would be helpful to have these terms clearly defined somewhere, preferably in an environment-specific way.\n- While Figure 5 provides a nice visualization of the learned value function contours, I find it hard to see how \"Pi_HIQL Exp and Pi-HIQL Lin display artifcats even near the goal\". Some annotations of the plots to help illustrate this would be helpful. The main thing I can see is how Pi-HIQL HJB fails to capture maze geometry, which is crucial too.\n- A minor point: I would suggest expanding acronyms in the introduction of the paper, even if they were defined in the abstract for clarity.\n\nQUESTIONS:\n- Is the choice of a constant S(s) = 1 motivated by the goal-conditioned reward discussed in Section 3, which has a maximum norm of 1? Do you think the more complex alternatives would outperform the constant speed profile with more complex reward functions?\n- What are some strategies for overcoming the limitations in environments with discontinuities? Would this primarily occur in the choice of distance function used in the speed profile?\n\nLIMITATIONS:\n- The authors clearly state that this regularization is especially well-suited to environments without sharp discontinuities. They also show this in their evaluations, with their approach underperforming in contact-rich tasks.\n- They also fully acknowledge that this regularization term requires access to obstacle locations, which may be a strong assumption in some settings.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents **Pi-HIQL**, a physics-informed regularization technique for offline goal-conditioned reinforcement learning. The method introduces a geometric inductive bias derived from the Eikonal partial differential equation (PDE), aiming to improve value learning and generalization in long-horizon and sparse-reward tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The paper offers a compelling intuition and theoretical foundation through the use of an Eikonal-based regularizer.\n\n**Weaknesses:**\n\n- Certain aspects of the method require further clarification. For example, the proof and derivation of Proposition 4.1 are incomplete; specifically, the simplification to the HJB equation is omitted. The proposition's correctness is questionable, as minimizing a sum does not necessarily yield a result smaller than minimizing the terms independently.\n- Some critical design choices are left unmotivated — in particular, the selection of the speed function S(s)S(s)S(s). Using a constant speed seems oversimplified, and it's unclear how this choice affects the regularization dynamics.\n- In practice, the proposed method reduces to regularizing the norm of the value function's gradient. While the Eikonal perspective provides one interpretation, alternative formulations exist, and the benefits of adopting this particular regularizer over others are not fully justified.\n\nQUESTIONS:\n1. In Equation 5, why must the expression equal zero? Could delta_t be zero in some cases to satisfy the equation trivially, and if so, what would that imply for learning?\n2. In the navigation task, the robot must handle contact dynamics with the ground, which can induce discontinuities in state transitions. Why is the proposed regularizer not affected by such discontinuities?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Physics-Informed HIQL (PI-HIQL), an offline goal-conditioned reinforcement learning (GCRL) approach that integrates a physics-informed (PI) regularizer derived from the Eikonal partial differential equation (PDE). The authors emphasize that existing offline GCRL methods struggle with value function estimation due to limited data coverage, particularly in sparse environments. To address this, PI-HIQL augments Hierarchical Implicit Q-Learning (HIQL) with an Eikonal regularizer that enforces a distance-like structure on the goal-conditioned value function (GCVF). The experiment results demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Pros:**\n1. The paper is well-structured and effectively presents the main idea.\n\n2. Incorporating physical principles (via the Eikonal PDE) into offline RL is a conceptually novel and promising direction. Prior work has shown that physics-informed biases can enhance representation learning and regularization in decision-making, and this work extends this paradigm to value function estimation in GCRL.\n\n3. The method demonstrates that a simple, physics-inspired regularization term, when added to an SOTA GCRL algorithm (HIQL), substantially improves performance across diverse navigation tasks. This underscores the regularizer’s utility as a lightweight yet impactful modification for offline RL frameworks.\n\n**Cons:**\n\nFrom my perspective, there are no significant drawbacks. But one minor clarification problem in the content:\n* In line 73, the phrase \"in model-based or Koopman inspired frameworks\" could be more precise. It is suggested to explicitly state how these frameworks leverage underlying dynamical properties: ​​\"utilizing the Koopman operator framework and incorporating Time-reversal symmetry information of the dynamics\"​​ would enhance technical clarity.\n\nQUESTIONS:\n1. Have you examined the sample efficiency of the PI-HIQL? Since including PI during learning could essentially include more informative information during training the policy. Note that I'm not asking for additional results, just for discussion purpose.\n\nLIMITATIONS:\nYes, the paper includes thorough discussions of the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe papers proposes a new smoothness-inducing regularizer designed for Offline GCRL based on the Eikonal PDE. It implements HIQL extended with the Eikonal regularizer and demonstrates that it achieves SoTA on some robotic benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. Paper is well written with properly defined background terminology.\n2. The method created with Eikonal regularizer achieves SoTA on the inspected tasks.\n3. The method nicely utilizes hierarchical nature of high-level policy to induce a structural bias with the regularizer that uses subgoals.\n4. The limitations are discussed explicitly.\n\n## Weaknesses\n\n> The proposed regularizer is broadly compatible with temporal-difference-based value learning and can be integrated into existing Offline GCRL algorithms.\n> \n1. The regularizer is evaluated only when combined with the HIQL, for such a general claims it should be evaluated on other GCRL methods. For example table 2 could also have Pi-QRL, Pi-CRL. Even if they don’t reach SoTA, it is an important information whether they improve their baselines.\n2. I’m not sure if the proposed regularizer can even be applied to any GCRL method. It requires an action to be a state (a subgoal), hence requires high-level policy and therefore hierarchical reinforcement learning. (See question no.1)\n3. The evaluation is quite limited in terms of benchmarks\n4. The benefit of the regularizer seem to dissipate in more complex environments, which poses risks to scalability and applicability of the method in real-life scenarios.\n5. I think “Physics-informed” is a very general name that does not sound very specific to the regularizer. For example HJB regularized HIQL could also be called physics-informed. Maybe put “Eikonal” in the name?\n\nQUESTIONS:\n1. Application of Eikonal regularizer to all GCRL: What about GCRL that is not hierarchical? How can this regularizer be applied there?\n2. It would be interesting see an example - even artificially made up - in which S(s) other than 1 induces useful structural bias and helps in some way with the training.\n3. I suppose the weird behaviour of HIQL on the Figure 1 is because it is not fully trained yet and the value function resembles expected, optimal value function at the end. How can authors guarantee that the effect observed in Figure 1 is not a consequence of suboptimal hyperparameters? \n4. Training curves would be interesting to see to understand the Figure 1 in more details.\n5. It would also be interesting to consider the performance of the eikonal regularizer from the perspective of sample efficiency. \n6. For the non-Lipschitz environments mentioned in the limitations. Does Eikonal regularizer improve the performance at the beginning of the training?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}