Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
Authors: Vittorio Giammarino
Ruiqi Ni
Ahmed H Qureshi
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Offline Goal-Conditioned RL
Physics-informed Neural Networks
Primary area: reinforcement_learning

Abstract: Offline Goal-Conditioned Reinforcement Learning (GCRL) holds great promise for domains such as autonomous navigation and locomotion, where collecting interactive data is costly and unsafe. However, it remains challenging in practice due to the need to learn from datasets with limited coverage of the state-action space and to generalize across long-horizon tasks. To improve on these challenges, we propose a Physics-informed (Pi) regularized loss for value learning, derived from the Eikonal Partial Differential Equation (PDE) and which induces a geometric inductive bias in the learned value function. Unlike generic gradient penalties that are primarily used to stabilize training, our formulation is grounded in continuous-time optimal control and encourages value functions to align with cost-to-go structures. The proposed regularizer is broadly compatible with temporal-difference-based value learning and can be integrated into existing Offline GCRL algorithms. When combined with Hierarchical Implicit Q-Learning (HIQL), the resulting method, Eikonal-regularized HIQL (Eik-HIQL), yields significant improvements in both performance and generalization, with pronounced gains in stitching regimes and large-scale navigation tasks. Code is available at https://github.com/VittorioGiammarino/Eik-HIQL.

Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
Vittorio Giammarino, Ruiqi Ni and Ahmed H. Qureshi
Department of Computer Science Purdue University
{vgiammar , hhuurarhrrrsssapuuuuuee.ee
Abstract
Offline Goal-Cone Conl-Conditioned Reinforcement Learning (GCRL) holds great promise for domains such as autonomous navigation and locomotion, where collecting in- teractive data is costly and unsafe. However, it remains challenging in practice due to the need to learn from datasets with limited coverage of the state aattin space and to generalize across long-horizon tasks. To improve these challenges, we propose a Physics-informed (Pi) regularized loss for value learning, derived from the Fikonal Partial Differential Equation (PDH) and which induces geometric in- ductive bias in the learned value function. Unlike generic gradient penalties that are primarily used stabilize training, our formulation grounded in continuous time optimal control and encourages value functions to align with cost-to-go structures. The proposed regularizer is broadly compatible with pipporallddiffeenee-bassss value learning and can be integrated into existing Offline GCRL algorithms. When combined with Hierarchical Implicit Q-Learning (HIQL), the resulting method, Nikoonalreeglarizze HIQL (Eik-HIQL) yields significant improvements in both performance and generalization, with pronounced gains in stitching regimes and large-scale navigation tasks. Code is available at link
1
Introduction
In recent years, many of the most effective machine learning paradigms have capitalized on vast amounts of unlabeled or weakly labeled data. Similarly, dynamic systems learning, Offline Goal- Conditioned Reinforcement Learning (GCRL) has emerged pivotal framework, enabling the use of large-scale, multitask datasets without requiring explicit reward annotations. Specifically, Offline RL 1 leverages passively collected trajectories to learn control policies, offering great promise for applications such as autonomous navigation, locomotion, and manipulation, where interactive training usually costly and unsafe. GCRL [ 4 extends this capability by enabling learning across diverse datasets without explicit rewards. Despite its potential, Offline GCRL faces significant challenges, including accurate Condiioodd Value Function (GCVF) estimation from limited data, policy extraction from imperfect value functions, and generalization to unseen state-goal pairs [5) Among these issues, GCVF estimation remains the most fundamental, as improvements in this area enhance both policy extraction and generalization, ultimately advancing the entire field of GCRL.
Physics-informed (Pi) inductive biases, defined as structural constraints grounded in physical laws such as symmetry, conservation principles, or consistency with Partial Differential Equations (PDEs), provide promising direction for enhancing GCVF estimation in the offline setting. As demonstrated in prior work (6), methods can introduce physically or geometrically meaningful structure into the learned value function, enhancing both sample efficiency and generalization. In Fig. I illustrate
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Goal-Conditoned Value Function
Goal-Condltioned Value Function
FLi
34
(a) Eik- HIK-HIQL (ours)
(b) HIQL
in [1] learned after 100,000 training steps by our Physics-informed algorithm Eik-HIQI, and the standard HIQL. The plots are generated by varying the agent's s center of mass x-y coordinates while keeping all other states fixed. Recall that the policy я trained to move the agent in the direction that maximizes the GCVF. The effects of the Eikonal regularizer are evident in Fig. Da where the contour plot closely follows the maze structure, in contrast to Fig. Db. where the learned GCVF ignores the maze structure.
Figure I: Contour plots of the GCVF for wetimazz-giaattnaviieeee
representative GCRL task which an agent must navigate from various starting positions in a maze to specified goal. Fig. D shows the contour plot of GCVF learned by non-Pi state-of-the-ar (SOTA) algorithm. The resulting value function fails to robustly encode obstacle constraints, leading to suboptimal policies that often fail reach the goal. These limitations motivate the use of Pi regularizers as a principled means to incorporate structural priors into value learning, and thus improve GCVF estimation in complex environments.
The primary contribution of this work the introduction of an Fikonal regularizer for GCVF estimation in Offline GCRL. tasks. Inspired by the Fikonal PDE (7). this regularizer imposes distance-like cost-to-go structure on the learned GCVF, serving as an effective inductive bias during training. By enforcing this structure, the regularizer improves value estimation accuracy and promotes generalization to unseen states, while also reducing the number of required training steps compared to non-Pi approaches (see Fig. [T). In contrast to Mamillon-JacobiBellee PDE-based methods [6]. which require explicit system dynamics and often suffer from numerical instability [8.9 99, our method is model-free and easy implement. Empirically, it outperforms both HJB-regularized and unregularized baselines while adding only minimal computational overhead.
(HJB)
To validate the effectiveness of our Eikonal regularizer, we integrate it into the Hierarchical Implicit Q-Learning (HIQL.) framework [00, a SOTA algorithm for Offline GCRL. We refer to this variant as EiK-HIQL. This choice is motivated by HIQL's strong baseline performance, making it an ideal candidate to highlight the benefits of our approach. Importantly, the Eikonal regularizer broadly compatible with other cimpoorl-ddifferenee-beee in Appendix[ D Our evaluation, conducted on the challenging OGbench benchmark [Л compares Eik-HIQL against Quasimetric RL (QRL) [12]. Contrastive RL (CRL) [13, and the standard HIQL baseline. Eik-HIQL consistently outperforms or matches the baselines, achieving SOTA results in large-scale navigation and trajectory stitching scenarios. These gains underscore the utility of the Eikonal regularizer in enhancing GCVF estimation and overall Offline GCRL performance, with limited exceptions in tasks involving complex object interactions.
algorithms, as we further demonstrate empirically
2 Related work
To the best of our knowledge, regularization for value estimation has only recently been explored. Notably, Lien et al. [6] propose an Offline RL objective derived from the HJB equation in continuous- time optimal control
91 aiming to enforce first-order derivative consistency within the critic network. In contrast, we introduce simpler, model-free Pi regularizer for GCVF learning, based on the residual of the Fikonal PDE. We show that this regularizer induces a distance-like structure the value function and integrates naturally into standard temporal-difee diffreeeeeefeee--bbeeeee GCRL pipelines.

---

While gradient norm penalties, closely related to the Eikonal PDE residual, have been employed in generative modeling [4 and, more recently, in model-based RL to regularize Q-functions and mitigate overfitting 15 their application in GCRL. remains, to our knowledge, unexplored. In contrast to these prior methods, which primarily aim to stabilize training, our regularizer is designed to inject structural inductive bias into the GCVF, thereby improving sample efficiency and generalization. To the best of our knowledge, this work presents the first use of the Eikonal PDE as regularization objective in value-based RL, and its first practical deployment in the Offline GCRL setting. More broadly, there has been a growing interest in incorporating physical priors and geometric, distance- like structures into RL algorithms, especially in model-based or Koopman- inspired frameworks [16] 17 These approaches typically leverage the Koopman operator, which assumes access to (or approximations of) the underlying system dynamics, and may incorporate structural information such as reversibility or symmetry of the dynamics. In contrast, our method operates directly on the GCVF in fully model-free setting.
In parallel, the use of non-Pi constraints for value learning has been extensively studied in Offline RL.. Many approaches constrain learned policies to remain close to the behavior policy, either through ex- plicit density modeling [8420] or implicit divergence constraints [2I Others directly regularize
the Q-function to assign low values to outtof-distribution actions and improve robustness [23 24).
Our work is also closely related to the literature on GCRL 4 Eik-HIQL extends HIQL [0] by
integrating the Eikonal regularizer into the GCVF estimation loss. HIQL itself combines hierarchical
actors [2 26 with Implicit Q-Learning 23 Other GCRL methods include hindsight relabeling (27]
contrastive representation learning 13]. state-occupancy matching [28), and quasimetric RI. 112
Offiin GCRL has also been studied through the lens of goal-conditioned supervised learning (GCSL) [29 30) where goal-reaching policies are trained via conditional imitation or regression.
Recent work has analyzed the out-of-distribution goal generalization problem [3]] and addressed
GCSL via self-supervised reward shaping (32] and occupancy- -ased score modeling 33
Beyond RL, Pi losses and neural networks (NNs) have been widely applied to learn parameterizations PDEs such as Burgers, Schrödinger, and Kavier-Stokes equations [34 These methods leverage automatic differentiation estimate derivatives with respect to NN inputs and solve nigh-dimensional PDEs. The Eikonal PDE, in particular, has been used in seismology (36] and motion planning [37] 38), where distance fields provide essential geometric structure. In this work, we extend the use of the Eikonal PDE to the Offline GCRL domain, demonstrating how its distance preeervvig properties enhance GCVF estimation, especially in large environments and when data stitching is required.
3 Preliminaries
Offtine GCRL We model the decision process as a finite-horizon discounted Markov Decision Process (MDP) described by the tuple A, R, Pgy Pu, 7) where S is the set of states, G is the set of goals, .A is the set of actions, T : S x A is the transition probability function where P(S) denotes the space of probability distributions over S, R : S x G + R is conditond reward function defined (s. -1 when s # g and R(s, 9) = otherwise, P3 E P(G) is the goal distribution, Po E P(S) is the initial state distribution and 7 E (0, is the discount factor. In this work, we assume the goal space G to be equivalent to the state space, i.e., G = S, and the goal g is sampled according to Pa (g Pg) at the beginning of each episode. The learning agent's objective is to maximize the expected sum of discounted rewards R(s,.)) g) to successfully reach Formally, this objective is expressed as J(T) = Er o 7'R(s, g)] where Tx (g) = (g. 80, ag 81, a1,..., ) and π : S x 9 + P(AA). Additionally, we define the GCVF induced by the policy as (*(sg)) = Er.,1) EE 2 at, 9)180 s, G 9) and the goal-conditioned
state-action value function as QT a, g) = (.r(s) IE'o 7 Rs,,,, a, SS, = 8, Ag = a, G = g]. In the offline setting, the learning agent must optimize J(T) using only a static, offline dataset D, which comprises trajectories of the form r = (so, (o, 81, .2,..,,.777 87). Finally, note that we write wo when a function is parameterized with parameters 0 E θ C R
Our algorithm, Eik-HIQL, extends the Offline GCRL algorithm HIQL, as introduced by Park et al. [10]. HIQI. incorporates two key components: a GCVF estimation process that is robust to -iittriiittiiitoo actions and a hierarchical actor. The hierarchical actor comprises a high-level policy, To.."
Hierarchical IQL
hs
S x G + S, which predicts subgoals, and low-level policy, T/e S X S -
3

---

P(A), which generates actions to achieve those subgoals. Robustness in the GCVF estimation step is
enabled by an action-free variant of implicit Q-learning [23. inspired by 39 40
(v(Ov) -B("..")"DD.op | [2(R(,,9)
+ Ve,, (5',9) - Ver (s,
(I)
where Oy denotes the parameters of the target GCVF network 41 and (-(-) represents the expectile
loss function with c E [0.5, 1]: (i( = | - 1(x < 0) x² In Eq. the expectile regression induced by L5(-) replaces the max operator in the Bellman equation
with the goal of avoiding queries of actions. The ability to properly handle overestimated values for disttiibittitttoonn actions is critical challenge in Offline RL, since, unlike in online RL, erroncous estimates cannot be corrected through environment interactions. The estimated GCVF is subsequently used to train the hierarchical actor where -hi (8t+* St; g) and z (a]8t, Se+k) aim maximize Vo, (St++ 9) and Vo, (St+1, S(+k). respectively. It is shown in Park et al. [10] that this hierarchy, compared the flat formulation using a single policy no [a) St. 9), can better address low signal-to-nn to-isse ratios in the estimated GCVE.
The Eikonal equation The Fikonal equation non-linear first-order PDE that describes wave propagation in heterogeneous media 7 It is expressed in its general form as:
VsT(,,
(2)
S(s)
where || denotes the Euclidean norm, T -S G - IR represents the travel-time through the medium from the state to a goal location 9 and V.T(s.g) g) is the partial derivative of the travel-time T with respect to s. The function S : S - R defines the speed profile of the medium in the state location s. As described in [37]. higher values for S(s) lead a low travel-time T (s. g) from s and therefore to preferable paths (sp, 81 ..... 87) compared to those with lower S(s). Consequently, the solution to the Fikonal PDE in (2), represented by the travel-time T(s,)) g). reflects cost-to-go structure. Minimizing ((s,9) 9) yields the shortest travel time from to g. as determined by the speed profile Ss)) In other words, (s, g) encodes a GCVF for specific class of optimal control problems. We formally establish this connection in the next section. In method, we propose regularizer, inspired by the Eikonal PDE in (2). with the goal of providing an additional distance-like structure to the learned GCVE. Under smooth dynamics assumptions, demonstrate that our regularizer significantly improves performance over the SOTA baselines while adding minimal complexity.
Physics-iforrmed Eikonal regularizer
In this section, we first relate the Eikonal PDE in ② to the HJB equation 8 9 for continuous-time undiscounted optimal control. This connection draws parallels to prior studies such as Lien et al. (6). offering additional motivation for our regularizer and insights into its empirical effectiveness. We then introduce the riguaae-regllarized loss for GCVF learning, which, when combined with hierarchical actor, forms the core of our Eik-HIQL algorithm. Finally, summarize the main components of Eik-HIQL.
Optimal control perspective on the Eikonal PDE following continuous-time dynamical system:
We start our analysis by considering the
s(t) (s((t, a(t)), t f20. 0,
where s(t) denotes the state of the system at time s(t) its derivative and a(t) the control action. The function f(-) represents the system dynamics, determining how the state evolves in response to the control action. As common in the literature, we assume f(-) to be a Lipschitz continuous function (3) Given the initial conditions s(0) sp, a(0) = ag and the goal s(T) g, the undiscounted optimal control problem seeks minimize
J= J= l' a(t))d,
o
where c(s(t), ( is the instantaneous cost function. The optimal value function V(s.g) associated with (3) defined as
V(s.g) = inf
c(s(t),((t)dtt
e(.)
4

---

and satisfies the principle of optimality
((s.9) = inf inf + ((s(t + At), aEA .EA
where At is small time step. Note that, unless otherwise specified, throughout this section we keep the standard continuous-time optimal control theory notation, where V(s,g) () is used in place of (*(s,g) g) to denote the optimal value function [4].
Approximating V(s(t + At). g) in (4) with its first-order Taylor expansion. V(s(t + At),
V(s. g) + (s..)) + O(At)), and taking the limit At + 0, we derive the following
HJB equation:
inf [c(s, a) V.V(s..) = 0. "EA
(5)
which holds true at optimality. Refer to Appendix for the step-by-step derivations. The HJB equation in (5) encodes the relationship between the system dynamics, the cost function, and the infoea[c(s, a) + (s. a)]
value function; where the left-hand side (s, 9. V.V(s.g) is referred to as the Hamiltonian. The following proposition establishes the connection between the Hamiltonian and the Eikonal PDE in 2 highlighting their equivalence under specific conditions.
Proposition 4.1.
Given the Hamiltonian H(s, g. V.V(s, g)), the following inequality holds H(, V V.V (n([ V. (8. [(s a)) S d' (s) [V.V(s,)))I)'())
(6)
REA
where (s)) = infard c(s, a) and (s) = supae A (s, [s,a))- In the special case in which f() = a, ||a|| and c(s, a) is constant over ||a|| the Hamiltonian simplifies to
Hss V.V(s.))) (s) (s,
Proof. The inequality in (6) follows from using the Pauchy-Schwarz inequality on the inner prod- uct V.V(s. g)t f(s. a) in (5). Using the definitions (s) supac (l, a)ll and c*((s inf a), we obtain the upper bound in (6). For the equality in 77. when c(s, a) is constant over ||a|| = 1, the inner product V. ()) f(s, a) attains its minimal value when points in the direc- tion opposite to VVV(,,)). Specifically, this occurs when a" = arg inf |all=1 V. V.V(s, (A,g)TTC [(x a) =
Vs, /V/lVVVVV gl Substituting fs, a) = a" into the Hamiltonian in (6) and simplifying yields the result in 7 Refer to Appendix C for the full proof.
Remark 4.2 (Connection between HJB and Eikonal residuals). Proposition 4.1 shows that, even without assumptions on the dynamics, the Hamiltonian H(s, g. V.V(s. Fikonal-like residual, where the ratio F* f=(ss/cc(()
(V(ssg)) g)) is upper-bounded by an (s) defines local speed profile S(s) as in ( In the
special case of isotropic dynamics with [(s, a) a, |||||| = 1, and constant cost, the Hamiltonian reduces exactly to the Eikonal PDE with S(s)= = 1/c² Thus, while the HJB PDE formalizes cost-to-go under known dynamics, the Eikonal PDE captures a related spatial structure through S(s). making it natural approximation when dynamics are unknown.
Remark 4.3 (Why the Eikonal residual helps in Offline GCRL). Temporal-difference learning (e.g.. (T) is known to converge to the optimal GCVF V under ideal conditions: namely, on-policy data and infinite sample budget 42 However, these assumptions are often violated in the offline setting, where biased datasets and long-horizon tasks exacerbate extrapolation eтror and limit generalization [10]. In this context, the Eikonal residual in Eq. (7) can play a crucial role by introducing a geometric inductive bias that encourages the learned value function to behave like a distance field, through the constraint VVV.Vo(s, g)ll s0 c(s). This regularization is particularly effective when the true V is Lipschitz continuous, i.e., in environments where the dynamics do not induce sharp discontinuities 45). Under such conditions, the Eikonal residual shapes the gradient norm of the learned GCVF Vo to match the local structure of V*, up to a scaling factor. This effect supported by Rademacher's 's theorem 46), which guarantees that Lipschitz continuous functions are differentiable almost everywhere, with || (*(s,g)) g)ll || < J., where T. is the minimal Lipschitz constant. As a result, when V* is Lipschitz continuous, the Eikonal residual in Eq. 7 provides a principled inductive bias that, as we empirically demonstrate, improves both sample efficiency and generalization, particularly in long-horizon tasks with smooth dynamics.
Furthermore, practice, the Fikonal residual offers a tractable alternative to HJB regularization in model-free settings where the dynamics f(s, a) are unknown. Prior work (6) approximates the HJB
5

---

term using finite differences, i.e., f(s,a) =' but our experiments show no clear advantage over the simpler Eikonal residual regularization. Moreover, note that, while the HJB equation in Eq. (5) holds only at optimality, the inequality in Proposition remains valid throughout training. providing consistent structural guidance even when V is suboptimal.
Fikooal-regularizeed implicit V-learning Based on the upper-bound in Proposition 4.1 and the discussion in Remark (..
we propose the following Eikonal-regularized implicit V V-learning loss for GCVF estimation:
(v( =B ( (2()(s,)) + +7Vo, (s', (5',g) - Ver (s.
(8)
+ (IV (s.9)l S(s) -()))
(9)
where the term in 8 corresponds to the expectile regression in ① (cf. 10) and 2 is our Eikonal regularizer.
( V Vo, (x,g)] is the Euclidean norm of the GCVF gradient with respect to its
input and ((x) (8) is pre-specified speed profile that maps states to scalar values (see Preliminaries in Section B The term V. V. (s. g) is computed via automatic differentiation, following standard approaches in PiNNs (sce Algorithm? 2 in Appendix D The speed profile S(s) is designed such that encapsulates additional biases or task-specific information into the Eikonal regularizer [37]. We demonstrate in experiments that the simple choice of S(s) I works best in practice; however, we believe that more interesting designs might further improve collision avoidance in cluttered environments and consequently enhance both safety and performance. We defer this direction on Pi Safe GCRL to future work. Finally, note that in (9), with respect to the upper bound in (6), we set (*(s) -1 and opt to multiply ((s) with |V, (8,g))] rather than using c² (s) 1/S(()) in This in order to ensure numerical stability.
As discussed in Remark 4.3 the effectiveness of the regularization in Eq. (9) stems from the geometric structure of the Eikonal PDE, whose solution defines a signed distance field [36] 37.. In our final formulation, we uniformly set the speed profile to S(s) 1, corresponding to the constraint Ve (s, I) & across the feasible state space. By Rademacher's theorem [46), this enforces i-Lipsehitz continuity almost everywhere, encouraging smoother and more generalizable value estimates even under limited offline data coverage. Although the true optimal value function V may not be exactly 1- 1-Lipschitz over the entire feasible space, we show in our experiments (Section 5 that this regularization remains effective in practice. Note that using a different constant S(x) > O everywhere in the feasible space would simply uniformly rescale the value function without affecting
its shape. Consequently, since policy gradients are invariant to such rescaling [4 47 48 choice has no impact on the induced policy.
this design
In the following, we provide a brief summary of our algorithm, Eik- HIQL.. During training, Bik-HIQL performs an Fikonal-regularized value estimation step, where the loss in 8 9 is minimized to learn GCVE This is followed by a policy extraction step, in which the hicrarchical actor introduced in Park et al. [0] (see Preliminaries in Section
Eikonal- HIQL
3 is trained based on the estimated GCVF. The full pseudo-code for Eik-HIQL well as a JAX [49] implementation showing how to compute the gradient V. Vey in D are provided in Appendix D respectively Algorithm
and Algorithm
2
For the full implementation of Eik-HIQL refer to our GitHub repository
5 Experiments
In this section, we analyze the effects of the Eikonal regularizer in on the GCVF estimation problem. Specifically, we will perform an ablation over different designs of speed profiles Sss), compare the Eikonal regularizer with an HJB regularizer, and analyze value functions learned with and without our Eikonal term. Then, we compare the performance obtained by our Fikonal-regularizet algorithm, Eik-HIQL,, against the SOTA algorithms for Offline GCRL. Finally, we will also present the limitations of our approach. The experiments in this section are conducted on the environments in Fig. 2 A table summarizing the most relevant hyperparameter values is provided in Appendix D
Speed profiles and HJB regularizer comparison Recall that our algorithm, Eik-HIQL, estimates the GCVF using the loss defined Eqs. (8)(9) where the speed profile 5ss encodes task-specific structure into the Eikonal regularizer. We perform ablation over different choices of S(s) and

---

pointmaze
(b) antmaze
humanoid-
(d) antsoccer
(c) cube
(f) scene
maze
Figure 2: Environments from OGhench 11 used in our experiments. These include a variety of goal- conditioned tasks spanning navigation and locomotion (e.g., pointmaze, antmaze, humanoidmazer contact-rich locomotion (antsoccer), and contact-rich manipulation (cube, scene). The environ- ments differ significantly dynamics complexity, dimensionality, and task structure, providing a comprehensive testbed for evaluating Offline GCRL algorithms.
Table 1: Summary of the speed profiles ablation. All agents are trained for 100,000 training steps using 10 seeds. We report the mean and standard deviation across seeds for the best evaluation achieved during training. For each seed, evaluations are conducted over different random goals, as designed in Park et al. M. with the learned policy tested for 50 episodes per goal. Results within 95% of the best value are written in bold.
Environment
Dataset Type
Maze Dimension
Eik EIk-HIQL
Eik- Eik-HIQL Exp (T0
Eik-HIQL Lin
11
HJB-HIQU 12
medium large
94 + 4
95 +
90 +
83 + 9
61 4
60
53 +
navigate
giant teleport
79
42 +
9
17 = 10
39 7
10 10:00
18 1
pointmaze
medium
97 +
90 9
85 + 10
64
large
73 +
33 17
41 11
6:7
stitch
giant
teloport
44 1 9
43 +:
38 7
18 1:
find the simple constant setting S(s) = 1 to be particularly effective. It outperforms more complex alternatives that require explicit knowledge of obstacle coordinates, which may not be available in practice. Specifically, we compare S(s) = against the following two speed profiles:
(x) =Smin + Smin) exp (-> dmxx d(s)
(10)
dmax dani
d((s) dnin 1.0
Sin =clip
(11)
dmax
dyna
where À is a decay rate, Smin represents the minimum tolerable speed, dmin and dmax are respectively the minimum and maximum tolerable distances of state from its nearest obstacle and d(s) is a function d S - R describing the Euclidean distance of the state s from its nearest obstacle (see Appendix B for more information). We refer Sexp() in (I0) as the exponential speed profile (Eik-HIQL Exp in Table T and to (-m(-) in (TT). originally introduced in [37], as the linear speed profile HIQL Lin in Table I Both (TO) and [T] assign high speed values to states s far from obstacles, and low speed values to states near obstacles. This encourages the agent to avoid obstacles, as proximity them results longer travel-time from s to g. Moreover, due the use of the exp(.) function, when compared to Sin(.); (-) ensures smoother decay of the speed value the distance from the obstacles decreases [50]. Finally, note that, in both these speed profiles, computing d(s) requires knowledge of the obstacles' coordinates which represents strong assumption in some settings. addition to this ablation, we also compare our Eikonal regularization term in g with an HJB regularizer derived from the HJB PDE in (5)
CUB (0v) B(s."")DD (V. (s. (s' s) - 1)>
(12)
In (12), the dynamics f(s, a). originally required by the HIB PDE in (5), is replaced by a finite difference term (s' 3) as proposed by Lien al. (6). The term 12) is then used in place of Eikonal regularizer in (9) for GCVF estimation in Eik-HIQL. We refer to this approach as HJB- HIQL. Table The results for these experiments are summarized in Table I and all the learning curves available in Appendix

---

Two main observations explain the superior performance of the simple choice S(s) = 1. First, in Offline GCRL, sampled trajectories are already constrained the feasible space, and there is no explicit penalty for colliding with obstacles during learning. As such, complex speed profiles incorporating obstacle proximity provide limited additional benefit over the uniform case. Second, the simplicity of (s)) enables more efficient and stable learning of the GCVF, whereas profiles requiring privileged information (e.g., obstacle maps) may introduce unnecessary complexity. Our choice of (s)) 1 also aligns with standard practice in the literature, which commonly adopts uniform speed profiles to model wavefront propagation in the feasible state space 57) This also ensures a fair comparison with the Offline GCRL baselines, information. Furthermore, we provide visualizations for the learned GCVF in Appendix B where contour plots for pointmaze giant- stitch- vv illustrate that Eik-HIQL with S(s) Teams more structured and accurate value function, closely aligning with the maze layout. In contrast, Eik- HIQI. Exp and Fik-HIQI. Lin display artifacts even the goal, while Fik-HIQL. HJB fails to recover the maze geometry altogether. Based on this empirical evidence, and to ensure a fair comparison with baselines, we adopt the constant speed profile S(s) 1 in all subsequent experiments. Nonetheless, we highlight that in the context of Safe GCRL, where value functions must encode both safety and task performance, investigating how different choices of S(s) influence the shape and behavior of the GCVF represents an interesting direction for future work.
none of them leverage privileged
Eik-HIQL vs HIQL We compare Eik-HIQL with its non-regularized counterpart, HIQL [10] to isolate the effect of the Eikonal regularizer. This comparison is conducted under tightly controlled conditions: both methods use identical network architectures, hyperpa- rameters, and training pipelines, ensuring that the only differ- ence lies in the presence of the regularizer. As shown in Table 2 Eik-HIQL consistently outperforms HIQL on navigation tasks, where the value function is typically Lipschitz continuous.
Table Complete comparison between Fik-HiQRI. and the Offline GCRL baselines. Agents are trained for 100,000 steps on pointmaze tasks and million steps on the remaining tasks, each using 10 seeds. The evaluation follows the methodology described in Table ! We report the mean and standard deviation across seeds for the best evaluation achieved during training. Results within 95% the best value are written in bold, and rows are highlighted when the Fikonal regularizer improves performance by 100% or more compared to the non-regularized HIQL performance.
Environment
Datasel
Dim
Eik-IIIQL
IIQL
QRL
CRL
nediux
98 d: 6 92 1:2
83 3
54 19
large
83 + 9
49 13
90 +
56 4
37
teloport
29
34
pointEaze
pediux
The benefits are particularly pronounced in large mazes and stitching regimes, with improvements exceeding 100% in out of 31 evaluated tasks.
stitch
giant
Large
13 1 6
+ 10
I=4 19:17
85 + 11
56 -
0 0:0
1
teloport
1
38 1
12 1
nedius
96 +
96 +
87
94 +
d
80
navigate
giant
+
14
Ieleport
62
43 d 3
39
66 +
medium
2:22
9.5 + 14
68
Given the magnitude of these gains relative to the standard de- viations across 10 random seeds,
stitch
large
84 3
48 48 11
3
74 1:
24 5
2
0:0
13
teleport
47
35
29
44
nedius
33
the improvements are both prac- tically meaningful and statisti- cally significant. Full learn- ing curves for these experiments are provided in Appendix B These results highlight the ro- bustness of the Eikonal regu- larizer in smooth environments and demonstrate its ability to en- hance generalization in settings where HIQL struggles to scale. As illustrated in Fig. [ for the antmaze- caamaaze-aviggatee seeeegiars giant-itttto- task, Fik-HIQL, produces a GCVF that reflects the underlying maze geometry, whereas HIQL fails to capture this structure, leading to poor goal-directed performance.
explorw
teloport
large
15 + 10
45 :
6=
0
2
22 + 6
0+0
aediux
86 +
80 +
22 2:2:2
61 1:
navigate
large
64 #
50 1
71:
22 1.9
husano idnias
68 2
18 E
1
4 1
medium
79 +
88 +
22 4
40 +
stitch
28 +
giant
10
3
0
0 0::0
navigate
arena pedius
3
13 #
+
212
24 2:2
1::2
etitch
nedius
arena
1 + 0
6
+
2
0
0 0::0
manipulation
cube www.ee-..eee....... play
25 1:
31 + 4
11 11:3
32 +
scose-play
7
d:
35
8

---

By contrast, in interactive, contact-rich domains such as antsoccer and manipulation, where the dynamics (and consequendly the value function) exhibit discontinuities, Eik-HIQL offers limited advantage. We defer detailed discussion of these cases to the Limitations section below.
Offline GCRL We extend our comparison sate--f-theean Offline GCRL algorithms, including QRL [12 and CRL [13,, alongside HIQL. Results across all 31 tasks are summarized in Table 2 with full learning curves in Appendix D
Eik-HIQL consistently outperforms all baselines in the most challenging settings, most notably in the antmaze-stitch tasks, which combine complex locomotion with, composite datasets, and in humanoidmaze, where high-dimensional states and unstable control amplify learning difficulty.
QRL performs well in simpler domains such as pointmaze, where its quasimetric structure aids goal-conditioned estimation. CRL is competitive on several navigate variants, but struggles in high-dimensional and stitched tasks such as humanoidma dmaze. contrast, Eik-HIQL demonstrates strong generalization and planning performance in both long-horizon and large-scale environments.
In antsoccer and manipulationn however, Eik-HIQL performs comparably to the baselines. As mentioned, these domains involve discontinuous or contact-rich dynamics, and we do not observe consistent benefits from the Eikonal regularizer due to the fact that the imposed gradient condition does not hold globally at optimality throughout the entire state space.
Limitations We conclude by discussing some limitations of our approach, particularly in contact-rich tasks that involve interactions with external objects. In the antsoccer domain, for instance, an agent must not only navigate but also interact with soccer ball to reach specified goal. Similarly, in manipulation tasks such as sube-single-p -plly, the agent must co- ordinate precise contact interactions with ob- jects in the scene. These tasks introduce hybrid, non-Lipschitz dynamics due to discontinuities in contact states, often modeled as categorical vari- ables (e.g.. in-contact vs. free motion), which pose challenge for regularizers grounded in smoothness and Lipschitz continuity assump- tions [43], such our Eikonal term. As visu- alized Fig. for the antsoccer experiments, Eik-HIQL learns a GCVF that aligns well with the environment geometry, particularly for the navigational components. However, this structured value function does not consistently yield better performance (cf. Table as these tasks also require complex interactions with objects in the environment. Similar trends observed in manipulation tasks, where Eik-HIQL performs comparably to the baselines but does not show marked gains. These experiments included for completeness and highlight that, while Eik-HIQL excels in davigation-domintted dominated domains, additional mechanisms, such as task- adakttde speed profiles or repre- sentation learning tailored contact dynamics, may be necessary to extend its benefits interactive. contact-rich environments. We leave this direction for future work.
+
-40
-60
(a) Eik-HIQL
(b) HIQL
Figure 3: Countour plots of the GCVF on wewoocer-medium-nviiiii
[I], learned after million steps by Eik-HIQL and HIQL re- spectively. These plots are generated following the same methodology in Fig.
6 Conclusion
We introduced Eik-HIQL, a novel approach to Offline GCRL that integrates an Eikonal PDE- baseb regularizer for GCVF estimation. Our analysis demonstrated that the Eikonal regularizer effectively introduces a useful distance-like inductive bias, which promotes consistent gradient magnitudes and improves value estimation in high-dimensional spaces (cf. Fig. D This property mitigates irregularities from the limited coverage of offline datasets, resulting in robust, globally consistent GCVFs that accurately capture the underlying structure of the environment. Consequently, Hik- outperformed SOTA baselines across diverse tasks, excelling in complex scenarios such as large-scale mazes and trajectory stitching, where traditional methods often fail to generalize.
However, our experiments also highlighted limitations in interactive tasks. We discuss how our Eikonal regularizer induces excessive smoothness in the learned GCVFs which does hinder perfor-

---

mance in interactive tasks where mon-smoothnsss, or at least non- global smoothness, is required. These findings suggest that, while the Eikonal regularizer significantly enhances navigation tasks, future work should incorporate mechanisms that better capture task-specific dynamics, such as object interaction, to improve applicability across diverse domains.
Overall, this work underscores the potential of Pi methods to address fundamental challenges in Offline GCRL. By enhancing scalability and generalization of value estimation, the Eikonal regularizer provides foundation for leveraging domain knowledge in RL. Future research could expand on this foundation by exploring multi-agents settings and integrating task specific biases for interactive environments.
References
[1] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-art, pages 45-73. Springer, 2012.
[2] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[3] Leslie Pack Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence, volume 2, pages 1094 8. Citeseer, 1993.
[4] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In International Conference on Machine Learning, pages 1312-1320. PMLR, 2015.
[5] Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? arXiv preprint arXiv:2406.0993 2024.
[6] Yun-Hsuan Lien, Ping-Chun Hsieh, Tzu-Mao Li, and Yu-Shuen Wang. Enhancing value function estimation through first-order state-action dynamics in offline reinforcement learning. In International Conference on Machine Learning, 2024.
[7] Marcus M Noack and Stuart Clark. Acoustic wave and eikonal equations in a transformed metric space for various types of anisotropy. Heliyon, 3(3), 2017.
[8] Rémi Munos. A convergent reinforcement learning algorithm in the continuous case based on finite difference method. In International Joint Conference on Artificial Intelligence, pages 021-831,10 1997.
[9] Rémi Munos and Paul Bourgine. Reinforcement learning for continuous stochastic control problems. Advances in Neural Information Processing Systems, 10, 1997.
[10] Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal- conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024.
[11] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. arXiv preprint arXiv:2410.20092, 2024.
[12] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning, pages 36411-36430 PMLR, 2023.
[13] Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Process- ing Systems, 35:35603-35620. 2022.
[14] Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in Neural Information Processing Systems, 30, 2017.
10

---

[15] Ruijie Zheng, Xiyao Wang, Huazhe Xu, and Furong Huang. Is model ensemble necessary? model-based rl via a single model with lipschitz regularized value function. arXiv preprint arXiv:2302.01244, 2023.
[16] Matthias Weissenbacher, Samarth Sinha, Animesh Garg, and Kawahara Yoshinobu. Koopman q-learning: Offfine reinforcement learning via symmetries of dynamics. International Conference Muchine Leurning, pages 23645-23667. PMLR, 2022.
[17] Peng Cheng, Xianyuan Zhan, Wenjia Zhang. Youfang Lin, Han Wang, Li Jiang, et al. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl. Advances in Neural Information Processing Systems, 36:7612-7633 2023.
[18] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint ar1iv: 9.. 1361, 2019.
[19] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[20] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[21] Ziyu Wang. Alexander Novikov, Konrad Zoina, Josh S Merel, Jost Tobias Springenberg. Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
[22] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weiggtte regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177) 2019.
[23] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learnigg arXiv preprint arXiv:2110.061166 2021.
[24] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
[25] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2): 181-211, 1999.
[26] Vittorio Giammarino and Ioannis Ch Paschalidis. Online baum-welch algorithm for hierarchical imitation learning. In 2021 60th IEEE Conference Decision and Control (CDC), pages 3717-3722. IEEE, 2021.
[27] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in Neural Information Processing Systems, 30, 2017.
[28] Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned reinforcement learning via f-advantage regression. Advances in Neural Information Processing Systems, 35:310-323 2022.
[29] Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.00 07749,, 2021.
[30] Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking conl-cond supervised learning and its connection to offline rl. arXiv preprint arXiv:2202.04477 2022.
[31] Rui Yang, Lin Yong. Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang What is essential for unseen goal generalization of offline goal-conditioned rl? In International Conference on Machine Learning, pages 39543-39571. PMLR, 2023.
11

---

[32] Lina Mezghani, Sainbayar Sukhbaatar, Piotr Bojanowski, Alessandro Lazaric, and Karteek Alahari. Learning coal-conditioned policies offline with self-supervised reward shaping. In Conference Robot Learning, pages 1401-1410. PMLR, 2023.
[33] Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Niekum. Smore: Score models for offline goal-conditioned reinforcement learning. arXiv preprint arXiv:2311.02013 2023.
[34] Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Data-driven discovery of partial differential equations. Science Advances, 364))e1602614, 2017.
[35] Maziar Raissi, Paris Perdikaris, and George E. Karniadakis. Physics- informeed neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[36] Jonathan D Smith, Kamyar Azizzadenesheli, and Zachary E Ross. Eikonet: Solving the eikonal equation with deep neural networks. IEEE Transactions on Geoscience and Remote Sensing, 59 (12):10655-1066, 2020.
[37] Ruiqi Ni and Ahmed H Qureshi. Ntfields: Neural time fields for physics-informed robot motion planning. In The Eleventh International Conference on Learning Representations, 2023.
[38] Ruiqi Ni, Zherong Pan, and Ahmed H Qureshi. Physics-informed temporal difference metric learning for robot motion planning. In The Thirteenth International Conference on Learning Representations, 2025. URL /tttps://openrevtiiii wntp///orrriiddddoaaoee///// dd-iiiageVoOiiO
puet/fooumm?
[39] Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 4085-4098, 2022.
[40] Dibya Ghosh, Chethan Anand Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. In International Conference on Machine Learning, pages 11321-11339. PMLR, 2023.
[41] Volodymyr Mnih. Playing atari with deep reinforcement learning. arXiv:1312.560,, 2013.
arXiv preprint
[42] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[43] Jeongho Kim, Jaeuk Shin, and Insoon Yang. Hamilton-jacobi deep q-learning for deterministic continuous-time systems with lipschitz continuous controls. Journal of Machine Learning Research, 22(206):1-34, 2021.
[44] Wendell H Fleming and Halil Mete Soner. Controlled Markov processes and viscosity solutions, volume 25. Springer Science & Business Media, 2006.
[45] Bora S Banjanin and Samuel A Burden. Nonsmooth optimal value and policy functions in mechanical systems subject to unilateral constraints. IEEE Control Systems Letters, 4(2): 506-511, 2019.
[46] Juha Heinonen. Lectures on Lipschitz analysis. Number 100. University of Jyväskylä, 2005.
[47] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999.
[48] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference Machine Learning, pages 1889-1897. PMLR, 2015.
[49] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milen and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL
http: / htt///ggtttbbtt com/ caan/jaxmmm/jja
12

---

[50] Ruiqi Ni and Ahmed H Qureshi. Physics-informed neural motion planning on constraint manifolds. arXiv preprint arXiv:2403.05765, 2024.
[51] James A Sethian. A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences, 1514)11591-1555, 1996.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: we have limitations paragraph.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
the proof is included in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the details are reported between main text and the appendix. Furthermore, we have released the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Code has been released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7/nips.com wttp:ccpppbccccceee guwde//dde/eeee Cooeeooodeesssseeeeeee http:p/iiiiigiieeeeeeeo/eaaasses
siiiisisccyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
[Yes]
The most important details are available in the main text and the rest is reported in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: yes, we report the standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///warrp.ccccaaaaaa//aaaaaa
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

A Impact Statement
This paper aims to advance the field of machine learning for autonomous decision-making and control robotic systems. We achieved this goal through the development of physics-informed methods for offline goal-conditioned reinforcement learning. While our work has potential societal implications, we do not identify any that require specific emphasis here.
B Speed profiles and HJB comparison: additional details and contour plots
In this section, we provide additional details related to the Speed Profiles and HJB Regularizer Comparison paragraph in Section 5 Specifically, we illustrate how the distance function d(s), used to define the speed profiles in (TO) and IT is computed in Fig. 4
Additionally; Fig. 5 presents contour plots of the GCVFs learned by the algorithms evaluated in Table | These visualizations provide qualitative support for the quantitative results Table [ showing that higher returns tend correlate with smoother, artifact-free value functions that more closely follow the structure of the underlying maze.
(a) d(s).
(b) Stnn (8)
Figure 4: Fig. 4a ialluusttssts the computation of the distance function d(ss) used in (10) and .TI. Let the state be represented by its spatial coordinates = (x, y) E R22 and let O = (or,.... OM denote the set of obstacle coordinates in the maze. We define d(s) min ||s o||2, i.e., the Fuclidean distance from 8 to the nearest obstacle. Fig. 4b) reports the resulting speed profile obtained using Stin (s) in (II for the pointmaze petee-mmdiunn redduumnaee wam---avvatteeyy dataset.
AFA
TP
TA
(a) Eik-HIQL
(b) Eik-HIQL Exp 10 (c) Eik-HIQL Lin 11
(d) HUB-HIQL I2
Figure 5: Contour plots of the GCVF pointmaze- wwwpaaazeeiiiaantttth-----eeeeee
[1], learned after 100,000
training steps by the algorithms in Table [] We observe that Eik-HIQL with constant speed profile S(s) 1 provides the most accurate GCVF estimation, resulting in the highest score for this task. In contrast, the contour plot for HJB-HIQL in (d) fails to effectively capture the maze layout, Furthermore, in (b) and (c) we annotate, with red circles, examples of artifacts arising in Eik- HIQL Exp and Eik-HIQL Lin, respectively.
C HJB PDE step-by-step derivations and Proof of Proposition 4.1
In the following we provide the step-by-step derivations for the HJB PDE (5) and the full proof for Proposition 4.
21

---

C.1 HJB PDE derivations
The optimal value function ((s,g g) associated with the undiscounted optimal control problem in satisfies the following principle of optimality
((s, g) = inf inf [c),, V(s(t + At).
(13)
aEA
where At is a small time step. By substituting the Taylor expansion
V(s(t + g) ((o.))))TVV).)))((///)))+++-
into 13
we obtain
V(s,g) aed inf (ot
Subtracting (ss, g) on both sides and dividing by At gives:
inf aEA
[c(s, a) V.V(s.g) f(s,a)+ +..
Taking the limit At -> we recover Eq. 5
inf aEA
inf V.V.. 0.
C.2 Proof of Proposition 4.1
Proof. Consider the HJB PDE in 55. By applying the Cauchy-Schwwarz inequality the argument of the minimization we obtain
c(s,a) + (.V ((s)) s -
Then, by defining
(s) = sup [((
aeA
can further upper bound the right- figh-hand-ssde and obtain:
c(s, + v.V(s, [(,, S c(s, IV.
Finally, applying the infimum over A both sides yields:
inf (c(s,a) a) + (sV(s.g) [(s, a)) S inf [c(s, a) [I.V((s.)) aed
aEd
where the result in Eq. (6) is obtained by defining (x) = inf.e.A
For the equality in note that for the isotropic dynamics [(s a) a and given c(s, a) constant over ||||| 1, the inner product V, V,V(s, f(s, a) attains its minimal value when points in the direction opposite to V. V (s. Specifically, this occurs when
a" = arg inf V.V(g,) (.V(s.)) (v.V(s
||a|==1
Substituting (() a" into
H(s,9, V.V(s,g))
inf [c((s V.V(s.) f(s. aeA
and simplifying yields
= (3 - (V.Vss))) H(s.g. (...g.V.V(s.)
22

---

D Psuedocode and Hyperparameters
During training, Eik-HIQL performs an Eikonal-regularized value estimation step followed by hierarchical policy extraction step. During the value estimation step, the following loss, as provided in is minimized:
(v(vv) Pa
L3 (R(s).)) + ~VV (s44119) Vov (st,g)
(14)
+ (1IV, Vo. (s.. S(s,) - 1)²
The hierarchical policy extraction step follows Park al. [T] and leverages, for both Tin. and mb..
an advantage-weighted regression-style objective:
exp(3 Ad (st, Setk g)) log mb.. (5t+ (2. 9))
(On:) ( (-..
(15)
(O6)) (s (( exp(( ( ar s(++)) log né (arlse, 8(+6)
(16)
where / is an inverse temperature hyperparameter and hi (st, Sttk. 9) and ( ae 88+k) are respectively approximated as Vo, (8t+*: g) Ve, (St, g) and Vo, ($++1, 8t-k) -Vo, (St, 81+k). The full pseudocode for Eik-HIQL is provided in Algorithm Furthermore, a function written in JAX on how to compute the gradient V. in (14) is summarized in Algorithm 2 Finally, Table reports the hyperparameter values most commonly used in our experiments. For more implementation details, refer our GitHuu repository
Algorithm 1 EGikonal-regulariin Hierarchical Implicit Q-Learning (Eik-HIQU
Input: Offline dataset D, value function Vev. target value function Voy, high-level policy hhi low-level policy T speed profile S, expectile factor discount factor inverse temperature parameter 8, learning rates ay, Cths, Otoo target update rate
T
while not converged do (St, 8,41,9) ~ D
Update Vo, minimizing (vv) in 14 with learning rate ay
Oy + (1 - T 0v + T0v
end while
while not converged do
(st, St+k. D Update Té.. maximizing Jun (0nx) in 15 with learning rate ons
hi end while
while not converged do
(st, a, S1+1, St-k) ~ D
Update wlo maximizing Ju (0)o) in 16 with learning rate ano end while
Algorithm 2 Compute V.Ve,
Input: states goals g. network parameters 0y
Define FORWARD(s, g. )y:: return SELECT (s, 9, params 0y)
grad_s <- JAX. WAX.VMAP(AXX.GRADD (FORWRRD, argnums = 0), in_axes = (0,0, (0,0,Nov None)) (s, return grad_s
23

---

Paintmazz-MMdllum Mew.-m------ee Medium-Navvgaate
Paintmaze
Pointmaze-Largo-lavvie
Pointmaze- Seaiize--Ginnt--te naatt--aiigg Navigate
Pointtm aww.rmaazze-ee Sointmmaz--Tlleparrt pelptriillot..
Success Percentage
80
60
62-
40
20
0.50
1.0
9.25
0.00
Steps (x10
Steps (x10")
Steps
Steps (x10')
Polittmaze-Medlum-5501
Pointmaze-Large-tlicc
Pointmaze-GiattS000
Pointnaaz-Teleporrt-tttc
Success Percentage
100
80
30-
23
60
40-
20
20-
10
0.00
0.25
2.50
0.75
1.00
0.00
0.25
0.30
5.75
1:00
0.00
0.25
0.30
0.75
1.50
0.00
0.25
0.50
0.75
1.03
Steps (x100)
Steps (x10")
Steps (x10*)
Steps (x107)
Eik-HIQL
Eik-HIQL Exp
Eik-HIQL Lin
HJB-HIQL
Figure 6: Learning curves for the speed profile ablation and the comparison with an HJB regularizer in Table Plots show the average success percentage per evaluation across seeds as a function of training steps.
Table 3: Hyperparameter values for Eik HIk-HIQ..
Hyperparameter Name
Value
Decay rate (A)
1.0
Minimum speed (Smin)
0.1
Discount factor (7)
0.99
Batch size (B)
1024
Optimizer
Adam
Learning rates ov, Ohhic Olo
3 . 10
Target update rate (r)
0.005
Expectile factor (t)
0.7
Inverse temperature parameter (8)
3.0
E
Learning Curves
Fig. shows the complete learning curves, plotted a function of training steps, for the experiments reported in Table T Figs. g 10, and disppla the learning curves for the pointmaze, antmaze,
humanoidmaze antsoccer, and mani canipulation experiments in Table 2 respectively.
All experiments were conducted on a single NVIDIA RTX 3090 GPU (24 GB VRAM), using local server equipped with a 12th Gen Intel i7-12700F CPU, 32 GB RAM. No cloud services or compute clusters were used. Each individual experimental run required approximately hours of compute time the GPU.
F Additional Experiments
In the following, present additional experiments demonstrating that Eikonal regularizer be seamlessly integrated with broad range of temporal-differee bbsed GCRL algorithms. In particular, apply it Goal-Conditioned variants of IQL 23 and IVL 39 40). yiclding Eik-GCIQL and -CIVL,,,, respectively. The corresponding results are summarized in Table with learning curves provided in Fig. 12 13] These experiments confirm the same conclusions drawn in the main paper from the comparison between Eik-HIQL and HIQL (Table and further support our claim that the Eikonal regularizer can be successfully combined with diverse TD-based algorithms.
24

---

Pointmazee Medium---------- Navigate Medium-Naviistee
Seintmazze-Mddium-i
Pointmaae--Ginn-ieee Navigte
Pointrn weaennazzeeeeeeeeeee Nelipor--Naviistee Navigate
1.00
Success Percentagg
60
60
40-
40
20-
20
20-
0-
Steps (x10P)
Steps (x10P)
Steps (x10P)
Steps (x10F)
Sointmaze-Medie Mediim--ttchhh
Dointmaze-Large-550
Pointmaze-Gioe Sertmmaze-Ginn-Sttloe
Poiniinsze-Teleport-
Success Percentage
1.00
50
80-
50-
40-
40-
20
10
0.00
0.25
3.50
0.75
1.00
0.50
0.25
0.50
5.75
1.50
0.00
0.25
0.50
0.75
1.50
0.00
0.25
0.50
0.75
1.00
Steps (x
Steps (x10*)
Steps (X10P)
Steps (x10F)
Eik-HIQL
HIQL
QRL
CRL
Figure 7: Learning curves for the pointmaze experiments in Table 2 Plots show the average success percentage per evaluation across seeds as function of training steps.
Ardmaaze-Medium--Maiies
Artmaze-Larg-NNavigtee
Glant-Navigate
Antrmaze-Telgort-aavagiet
Success Percentage
1.00
60
50
40
20
20
20
20
5.50
0.75
0.50
6.75
1.00
0.25
0.50
1.00
0.25
0.50
0.75
1.00
Steps (x109)
Steps (x10%)
Steps (x105)
Steps
Saarmaze-Medium Stitch
AntmazzeLarrg-Sttine
Antinaze Giant-Stitch
Antrnaze-Teleport. Seepot--Sttchh
Success Percentage
1.00
50
40
60-
40-
20
26
20
Steps (x109)
Steps (x10%)
Steps (x10%)
Steps (x10F)
Antmazee Medium-Err Explore
Laanaaze-Larg--eeee Explore
Antmaze heimaazz-illepar--Explorr
Success Percentage
20
40
15
10
20-
01
0.00
5.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.30
0.75
1.00
Steps (x10f)
Steps (x10')
Steps (x10")
Eik-HIQL
HIQL
QRL
CRL
Figure 8: Learning curves for the antmaze experiments in Table Plots show the average success percentage per evaluation across seeds as a function of training steps.
25

---

Tvvmanidmazz-MMdiiu--Wvigggg
Hummaroddmaze--Lrgg-avvvvva
Numaroidmaze-Guiii Gaan---aaannntnntnteee
Success Percentage
60
60
40-
20
0.2
0.6
1.0
0.2
0.6
1.0
0.2
0.6
1.0
Steps (x10")
Steps (x100))
Steps (x10%)
Humanaidmaze Stitch
Humaneidm Large- aarge-Stitch
Humaneidm Giant-Stitch
Success Percentage
50
15
0
1.0
0.2
1.0
Steps (x10%)
Steps (x109)
Steps (x10%)
Eik-HIQL
HIQL
QRL
CRL
Figure 9: Learning curves for the humanoi dmaze experiments in Table 2 Plots show the average success percentage per evaluation across seeds function of training steps.
AnsooccerArenn-Navviigee
Antsoccer-Arena Stilch
Messoccee-MMediung Stitch
Success Percentage
60
14
20
50
12
15
30
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.30
0.00
0.25
0.50
0.75
1.00
0.50
0.25
0.50
0.75
1000
Steps (x10%)
Steps (x109)
Steps
Steps (x109)
Eik-HIQL
HIQL
QRL
CRL
Figure 10: Learning curves for the antsoccer experiments in Table 2 Plots show the average
success percentage per evaluation across seeds function of training steps.
Sube-Single-Play
Scene-Play
Success Percentage
60
50
20
30
20
10
10
0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Steps (x (x10%)
Steps (x106)
Eik-HIQL
HIQL
QRL
CRL
Figure 11: Learning curves for the manipulati iatiion on experiments in Table 2 Plots show the average success percentage per evaluation across seeds function of training steps.
26

---

Table 4: Summary of the experiments with different TD-based GCRI. algorithms. All agents are trained for 100,000 training steps using seeds. We report the mean and standard deviation across seeds for the best evaluation achieved during training. For cach seed, evaluations conducted over different random goals, as designed Park et al. [Л with the learned policy tested for 50 episodes per goal. Results within 95% of the best value are written in bold.
Environment
Type
Dimension
GCIQL
Eik-GCIOL
GCIVL
Eik-GCIVL
medium
59 =
63 6346
90
large
39 +
60 y
38 +
82 39
navigate
glant
0
2
0
86 11
teleport
+
38
49 -
pointmaze
medium
41
57
95
large
25 =
22 =
11
67 =
glant
01:
04:0
0
23 10
aport
28
25 1
41
38 -
large
9
7
16 4
15 =
navigate
glant
0+0
0
tel teleport
24 1
23
32 5
30 +
modium
19
21
large
611
313
12
712
stitch
glant
0+0
0
teleport
18
23
30
28 =
PointmazzLLare--Navigtee
Baaalmaze-Giant-Navigee
Dointmaze-Teleporr Niaia
Normalized Score
0.8
0.8
0.6
0.4
0.2
0.2
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.5
1.00
(x10*)
Steps (x10*)
Steps (x10*)
Steps (x10*)
Pointmaze: Medium Stitch
Paintmaze-Large- Stitch
Pointmaze Giant-Slitch
Seintmazz-Teleporree Stitch
Normalized Score
1.0
0.
0.6
9.6
0.4
0.2
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.55
150
Steps
(x109)
Steps (x10²)
Steps (x10")
Steps (x10²)
GCIQL
GCIQL
Figure 12: Learning curves for the pointmaze experiments in Table 4 Plots show the average success percentage per evaluation across seeds function of training steps.
27

---

Antmare--eedlm--Naviaa
Antmaze-Large-Naawaaee
Antmaze-Gian-Naaiiate
Antmaze-Tllepor-Naalaaee
Normalized aeed Score
0.20
0.0100
0.4
0.15
0.077
M
0000
0.2
0.05
6.0
0.0025
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50 0.75 1.00
0.00
025
0.50 0.75
1.00
Steps (x10*)
Steps (x10%)
Steps (x109)
Steps (x109)
Antmaze-Medium-Sticc
Antmaze-Large-Stie
Antmaze-Gian-Stiich
Antmaze-Teleport-Stte
Normalized Score
0.3
0.04
0.02
0.2
-0.02
-0.04
0.0
0.00
0.0
0.00
0.5
1.50
0.25
1.00
0.00
1.00
0.00
0.25
0.75
1.00
Steps (x10%)
Steps (x10º)
Steps (x10")
Steps (x109)
Eik GCIQL
GCIOL
EI--CCIL
GCIVL
Figure 13: Learning curves for the antmaze experiments in Table 2 Plots show the average success percentage per evaluation across seeds as a function training steps.
28


Reviews:
[{"review_id": "ToVYf5fvJU", "reviewer": "Reviewer_zAXA", "summary": "The papers proposes a new smoothness-inducing regularizer designed for Offline GCRL based on the Eikonal PDE. It implements HIQL extended with the Eikonal regularizer and demonstrates that it achieves SoTA on some robotic benchmarks.", "questions": "1. Application of Eikonal regularizer to all GCRL: What about GCRL that is not hierarchical? How can this regularizer be applied there?\n2. It would be interesting see an example - even artificially made up - in which S(s) other than 1 induces useful structural bias and helps in some way with the training.\n3. I suppose the weird behaviour of HIQL on the Figure 1 is because it is not fully trained yet and the value function resembles expected, optimal value function at the end. How can authors guarantee that the effect observed in Figure 1 is not a consequence of suboptimal hyperparameters? \n4. Training curves would be interesting to see to understand the Figure 1 in more details.\n5. It would also be interesting to consider the performance of the eikonal regularizer from the perspective of sample efficiency. \n6. For the non-Lipschitz environments mentioned in the limitations. Does Eikonal regularizer improve the performance at the beginning of the training?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. Paper is well written with properly defined background terminology.\n2. The method created with Eikonal regularizer achieves SoTA on the inspected tasks.\n3. The method nicely utilizes hierarchical nature of high-level policy to induce a structural bias with the regularizer that uses subgoals.\n4. The limitations are discussed explicitly.\n\n## Weaknesses\n\n> The proposed regularizer is broadly compatible with temporal-difference-based value learning and can be integrated into existing Offline GCRL algorithms.\n> \n1. The regularizer is evaluated only when combined with the HIQL, for such a general claims it should be evaluated on other GCRL methods. For example table 2 could also have Pi-QRL, Pi-CRL. Even if they don’t reach SoTA, it is an important information whether they improve their baselines.\n2. I’m not sure if the proposed regularizer can even be applied to any GCRL method. It requires an action to be a state (a subgoal), hence requires high-level policy and therefore hierarchical reinforcement learning. (See question no.1)\n3. The evaluation is quite limited in terms of benchmarks\n4. The benefit of the regularizer seem to dissipate in more complex environments, which poses risks to scalability and applicability of the method in real-life scenarios.\n5. I think “Physics-informed” is a very general name that does not sound very specific to the regularizer. For example HJB regularized HIQL could also be called physics-informed. Maybe put “Eikonal” in the name?", "was_revised": true, "final_justification": "My initial concerns centered around the generality and applicability of the regularizer, its evaluation scope, and the interpretation of results. The authors have addressed these in a thorough and convincing manner:\n\nOn generality: The authors clarified that the regularizer is not limited to hierarchical methods. They provided new empirical results demonstrating its applicability to non-hierarchical TD-based methods (GCIQL and GCIVL), confirming its broader compatibility across the GCRL landscape. This directly resolves my concern (W1, W2, Q1).\n\nOn benchmark diversity: While I initially noted the evaluation as limited (W3), the authors justifiably argue that OGBench is a comprehensive benchmark spanning various environments and challenges. The added experiments further strengthen this claim.\n\nThe authors also acknowledge the limitations in non-Lipschitz environments and outline this as an avenue for future work.\n\nTaken together, these clarifications and additional experiments improve the strength and clarity of the paper. The work is methodologically sound, well-written, and demonstrates clear potential for impact in offline RL, especially in the under-explored space of structure-inducing regularizers.", "rebuttal": "Thank you for reviewing our work. We appreciate your constructive feedback and the time you dedicated to evaluating our submission.\n\nWe have carefully addressed each of your comments and clarified the points you raised in our rebuttal. If our responses have satisfactorily resolved your concerns, we would be grateful if you could consider raising your score.\n\nIf there are still aspects that remain unclear, please feel free to reach out, we are happy to provide further clarifications. Thank you again for your review and for helping us improve our work.\n\n## W1, W2, Q1\n\n> The regularizer is evaluated only when combined with the HIQL, for such a general claim it should be evaluated on other GCRL methods.\n\n> I’m not sure if the proposed regularizer can even be applied to any GCRL method. It requires an action to be a state (a subgoal), hence requires high-level policy and therefore hierarchical reinforcement learning. (See question no.1)\n\nThank you for raising this important point. We would like to clarify that the proposed Eikonal regularizer, as defined in Eq. (9), is compatible with any Temporal Difference (TD)-based Goal-Conditioned value learning method, such as the formulation shown in Eq. (8). \n\nImportantly, it does not require a hierarchical actor or subgoal-based high-level policy, and is therefore applicable beyond just HIQL. In our work, we apply it to HIQL because it represents a strong baseline and serves as an ideal candidate for demonstrating the benefits of our approach ( as mentioned in lines 48–51).\n\nTo reinforce this point, we have conducted additional experiments using a Goal-Conditioned adaptation of two non-hierarchical algorithms:\n\n1. Goal-Conditioned Implicit Q-Learning (GCIQL) [1]\n2. Goal-Conditioned Implicit V-Learning (GCIVL) [2,3]\n\nIn both cases, we applied our Eikonal regularizer (Pi-GCIQL and Pi-GCIVL, respectively), without any hierarchical structure or high-level policy. For these experiments we follow the same procedure described in the caption of Table 1. We will include the following results in the final version of the paper. \n\n| Environment | GCIQL | Pi-GCIQL | GCIVL | Pi-GCIVL |\n| ------------------ | --------- | ------------ | --------- | ------------- |\n| *pointmaze-medium-navigate* | 60 +- 1 | 59 +- 9 | 63 +- 6 | **90 +- 5** |\n| *pointmaze-large-navigate* | 39 +- 1 | 60 +- 9 | 38 +- 5 | **82 +- 39** |\n| *pointmaze-giant-navigate* | 0 +- 0 | 2 +- 4 | 0 +- 0 | **86 +- 11** |\n| *pointmaze-teleport-navigate* | 29 +- 5 | 25 +- 12 | 38 +- 5 | **49 +- 4** |\n| *pointmaze-medium-stitch* | 41 +- 11 | 56 +- 6 | 57 +- 9 | **95 +- 4** |\n| *pointmaze-large-stitch* | 25 +- 8 | 22 +- 3 | 11 +- 8 | **67 +- 9** |\n| *pointmaze-giant-stitch* | 0 +- 0 | 0 +- 0 | 0 +- 0 | **23 +- 10** |\n| *pointmaze-teleport-stitch* | 28 +- 5 | 25 +- 3 | **41 +- 5** | **38 +- 3** |\n| *antmaze-medium-navigate* | 27 +- 4 | 25 +- 6 | 36 +- 5 | **50 +- 5** |\n| *antmaze-large-navigate* | 9 +- 3 | 7 +- 2 | **16 +- 4** | **15 +- 3** |\n| *antmaze-giant-navigate* | 0 +- 0 | 0 +- 0 | 0 +- 0 | 0 +- 0 |\n| *antmaze-teleport-navigate* | 24 +- 3 | 23 +- 2 | **32 +- 5** | **30 +- 3** |\n| *antmaze-medium-stitch* | 19 +- 4 | 21 +- 5 | **25 +- 4** | **27 +- 6** |\n| *antmaze-large-stitch* | 6 +- 3 | 3 +- 3 | **12 +- 3** | 7 +- 2 |\n| *antmaze-giant-stitch* | 0 +- 0 | 0 +- 0 | 0 +- 0 | 0 +- 0 |\n| *antmaze-teleport-stitch* | 18 +- 5 | 23 +- 3 | **30 +- 3** | **28 +- 3** |\n\nFurthermore, we have also worked on Physics-informed extensions of QRL and CRL. However, not being these two algorithms purely based on TD learning, more considerations are required and we defer this to future work.\n\n## W3\n> The evaluation is quite limited in terms of benchmarks\n\nWe would like to emphasize that the OGBench benchmark [4] is a recently introduced and comprehensive suite that spans a diverse set of agents, environments, and offline datasets. \n\nWhile we acknowledge the importance of broad evaluation, we believe that incorporating additional benchmarks would not have substantially enhanced the insights provided. The chosen tasks already allowed us to highlight both the strengths and limitations of our approach across varied scenarios, offering a meaningful and representative assessment.\n\n## W4\n> The benefit of the regularizer seem to dissipate in more complex environments\n\nWe respectfully disagree with this observation. In our experiments, environment complexity can be understood along three key axes: (1) agent complexity, (2) maze size, and (3) dataset type. \n\nAs shown in Table 2, our proposed Pi-HIQL consistently outperforms HIQL across all three dimensions. For instance, in high-complexity scenarios such as *humanoid-giant-navigate* and *antmaze-giant-stitch*, Pi-HIQL demonstrates clear advantages over HIQL. These results highlight that the benefits of our regularizer not only persist but become more pronounced as the tasks grow more challenging.\n\n## W5\n> “Physics-informed” is a very general name that does not sound very specific to the regularizer. \n\nWe appreciate the reviewer’s feedback and understand the concern about the generality of the term. While “Physics-informed” is indeed broad, we chose it to reflect the fact that our regularizer is grounded in principles from physics-based optimal control, specifically through PDE-inspired constraints.\n\nThat said, we agree it is important to avoid confusion, and we will clarify the terminology in the final version. \n\n## Q2\n> It would be interesting see an example - even artificially made up - in which S(s) other than 1 induces useful structural bias and helps in some way with the training.\n\nWe agree this is a very relevant point. Unfortunately, in our current experimental setup, safety violations such as collisions with obstacles are not directly penalized. We conjecture that this lack of penalty is the primary reason why the structural bias introduced by varying $S(s)$ does not yield meaningful performance gains over the baseline choice of $S(s)=1$.\n\nAs noted in lines 290-293, we are actively working to extend our framework to a safe Goal-Conditioned RL setting, where safety constraints are explicitly enforced. In such a context, a state-dependent speed term $S(s)$ is expected to play a more significant role in safety-aware value function learning and influence downstream evaluation metrics. We look forward to exploring this direction in future work.\n\n## Q3\n> I suppose the weird behaviour of HIQL on the Figure 1 is because it is not fully trained yet and the value function resembles expected, optimal value function at the end. How can authors guarantee that the effect observed in Figure 1 is not a consequence of suboptimal hyperparameters? \n\nThank you for the question. We would like to clarify that Fig. 1 is intended as an illustrative example to demonstrate the practical effect of our Eikonal regularizer on the learned value function.\nIn the specific case of *antmaze-giant-navigate-v0* (shown in Fig. 1), we observe that with longer training, as reported in Table 2, the performance gap between Pi-HIQL and HIQL becomes negligible. However, this is not the case for all environments included in Table 2.\n\nThe key takeaway is that while extended training may sometimes reduce the performance difference, the regularizer consistently introduces a useful inductive bias that improves sample efficiency and helps guide value learning early in training. This effect is particularly beneficial in settings with large environments, complex dynamics, and in the stitch regime.\n\n## Q4, Q5, Q6\n> Training curves would be interesting to see to understand the Figure 1 in more details.\n\nWe agree, and we have included all training curves in Appendix D, as mentioned in lines 321-323. The curve corresponding to Fig. 1, for the *antmaze-giant-navigate-v0* environment, is shown in Fig. 8.\n\n> It would also be interesting to consider the performance of the eikonal regularizer from the perspective of sample efficiency. \n\nIf we define sample efficiency as the number of gradient steps required to reach satisfactory performance, our results suggest that the Eikonal regularizer improves efficiency. This is evident from the training curves in Appendix D. While the regularizer requires computing an additional term (Eq. 9) alongside the standard TD loss (Eq. 8), the computation is relatively efficient and adds only moderate overhead per training step. We believe this added complexity is justified by the performance gains observed across several benchmarks.\n\n> For the non-Lipschitz environments mentioned in the limitations. Does Eikonal regularizer improve the performance at the beginning of the training?\n\nAs shown in the training curves in Appendix D, this is not the case. In highly non-Lipschitz environments, such as those involving contact-rich dynamics, we observe limited early-stage benefits. Addressing this limitation is an important direction for future work, and we are actively exploring ways to adapt our method to better handle such challenging settings.\n\n**References**\n\n[1] Kostrikov I, Nair A, Levine S. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169. 2021 Oct 12.\n\n[2] Xu H, Jiang L, Jianxiong L, Zhan X. A policy-guided imitation approach for offline reinforcement learning. Advances in Neural Information Processing Systems. 2022 Dec 6;35:4085-98.\n\n[3] Ghosh D, Bhateja CA, Levine S. Reinforcement learning from passive data via latent intentions. InInternational Conference on Machine Learning 2023 Jul 3 (pp. 11321-11339). PMLR.\n\n[4] Park S, Frans K, Eysenbach B, Levine S. Ogbench: Benchmarking offline goal-conditioned rl. arXiv preprint arXiv:2410.20092. 2024 Oct 26."}, {"review_id": "70ftX0O6uN", "reviewer": "Reviewer_XkBr", "summary": "This paper introduces Physics-Informed HIQL (PI-HIQL), an offline goal-conditioned reinforcement learning (GCRL) approach that integrates a physics-informed (PI) regularizer derived from the Eikonal partial differential equation (PDE). The authors emphasize that existing offline GCRL methods struggle with value function estimation due to limited data coverage, particularly in sparse environments. To address this, PI-HIQL augments Hierarchical Implicit Q-Learning (HIQL) with an Eikonal regularizer that enforces a distance-like structure on the goal-conditioned value function (GCVF). The experiment results demonstrate the effectiveness of the proposed approach.", "questions": "1. Have you examined the sample efficiency of the PI-HIQL? Since including PI during learning could essentially include more informative information during training the policy. Note that I'm not asking for additional results, just for discussion purpose.", "limitations": "Yes, the paper includes thorough discussions of the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Pros:**\n1. The paper is well-structured and effectively presents the main idea.\n\n2. Incorporating physical principles (via the Eikonal PDE) into offline RL is a conceptually novel and promising direction. Prior work has shown that physics-informed biases can enhance representation learning and regularization in decision-making, and this work extends this paradigm to value function estimation in GCRL.\n\n3. The method demonstrates that a simple, physics-inspired regularization term, when added to an SOTA GCRL algorithm (HIQL), substantially improves performance across diverse navigation tasks. This underscores the regularizer’s utility as a lightweight yet impactful modification for offline RL frameworks.\n\n**Cons:**\n\nFrom my perspective, there are no significant drawbacks. But one minor clarification problem in the content:\n* In line 73, the phrase \"in model-based or Koopman inspired frameworks\" could be more precise. It is suggested to explicitly state how these frameworks leverage underlying dynamical properties: ​​\"utilizing the Koopman operator framework and incorporating Time-reversal symmetry information of the dynamics\"​​ would enhance technical clarity.", "was_revised": true, "final_justification": "The paper is well-structured and effectively presents the main idea. Incorporating physical principles into offline RL is a conceptually novel and promising direction. Prior work has shown that physics-informed biases can enhance representation learning and regularization in decision-making, and this work extends this paradigm to value function estimation in GCRL. There are no explicit limitations overall, thus I keep the score as 5", "rebuttal": "Thank you very much for your review. We truly appreciate the positive evaluation and the time you took to provide constructive feedback.\n\nWe have carefully addressed your comments and questions in the rebuttal, and we are grateful for your suggestions, which helped us improve the clarity and quality of our work.\n\n## W1\n> In line 73, the phrase \"in model-based or Koopman inspired frameworks\" could be more precise.\n\nThank you for pointing this out. We agree with the reviewer’s suggestion and will revise the phrasing in line 73 to improve technical clarity. \n\n## Q1\n> Have you examined the sample efficiency of the PI-HIQL?\n\nThank you for raising this important point. While we did not include a dedicated sample efficiency analysis, we refer the reviewer to the learning curves in Appendix D, and in particular to the results on the AntMaze benchmark in Fig. 8, where both Pi-HIQL and HIQL eventually succeed. In these plots, the impact of the regularizer on sample efficiency can be better appreciated.\n\nFor example, in tasks such as *antmaze-giant-navigate* and *antmaze-large-stitch*, we observe that Pi-HIQL converges significantly earlier than HIQL, indicating improved sample efficiency. In other tasks, such as *antmaze-giant-stitch* and in the humanoid setting, we observe that the Pi regularizer not only improves efficiency but also enables learning altogether, where HIQL alone fails to make progress."}, {"review_id": "UdrgW1rt7D", "reviewer": "Reviewer_3kEX", "summary": "This paper presents **Pi-HIQL**, a physics-informed regularization technique for offline goal-conditioned reinforcement learning. The method introduces a geometric inductive bias derived from the Eikonal partial differential equation (PDE), aiming to improve value learning and generalization in long-horizon and sparse-reward tasks.", "questions": "1. In Equation 5, why must the expression equal zero? Could delta_t be zero in some cases to satisfy the equation trivially, and if so, what would that imply for learning?\n2. In the navigation task, the robot must handle contact dynamics with the ground, which can induce discontinuities in state transitions. Why is the proposed regularizer not affected by such discontinuities?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper offers a compelling intuition and theoretical foundation through the use of an Eikonal-based regularizer.\n\n**Weaknesses:**\n\n- Certain aspects of the method require further clarification. For example, the proof and derivation of Proposition 4.1 are incomplete; specifically, the simplification to the HJB equation is omitted. The proposition's correctness is questionable, as minimizing a sum does not necessarily yield a result smaller than minimizing the terms independently.\n- Some critical design choices are left unmotivated — in particular, the selection of the speed function S(s)S(s)S(s). Using a constant speed seems oversimplified, and it's unclear how this choice affects the regularization dynamics.\n- In practice, the proposed method reduces to regularizing the norm of the value function's gradient. While the Eikonal perspective provides one interpretation, alternative formulations exist, and the benefits of adopting this particular regularizer over others are not fully justified.", "was_revised": true, "final_justification": "The rebuttal solved my concerns about the theoretical parts.", "rebuttal": "Thank you for your detailed review. We appreciate your consideration of our submission and the feedback you provided.\n\nWe have taken your comments seriously and responded to each of your concerns in detail. If our explanations address your doubts and clarify the issues you raised, we kindly ask you to consider revisiting your score.\n\nPlease do not hesitate to reach out if there are any remaining uncertainties or further questions. Thank you again for your time and for giving us the opportunity to improve our work.\n\n## W1, Q1\n> The proof and derivation of Proposition 4.1 are incomplete; specifically, the simplification to the HJB equation is omitted. \n\n> In Equation 5, why must the expression equal zero? Could delta_t be zero in some cases to satisfy the equation trivially, and if so, what would that imply for learning?\n\nThank you for pointing this out. We will revise the final version of the paper to include additional steps leading to the HJB equation in Eq. (5). Briefly, by substituting the Taylor expansion $V(s(t+\\Delta t), g) = V(s, g) + \\nabla_s V(s,g)^{\\intercal}f(s,a)\\Delta t + O(\\Delta t^2)$ into the principle of optimality in Eq.(4), we obtain:\n\n$$V(s,g) = \\inf_{a \\in \\mathcal{A}}[c(s,a)\\Delta t + V(s,g) + \\nabla_s V(s,g)^{\\intercal}f(s,a)\\Delta t  + O(\\Delta t^2)].$$\n\nSubtracting $V(s,g)$ on both sides and dividing by $\\Delta t$ gives:\n\n$$0 = \\inf_{a \\in \\mathcal{A}}[c(s,a) + \\nabla_s V(s,g)^{\\intercal}f(s,a) + O(\\Delta t)].$$\n\nTaking the limit as $\\Delta t \\to 0$, we recover Eq. (5):\n\n$$ \\inf_{a \\in \\mathcal{A}} [c(s,a) + \\nabla_s V(s,a)^{\\intercal}f(s,a)]=0.$$\n\nRegarding the second question: $\\Delta t$ serves as a small positive quantity in a first-order approximation, and the derivation considers the limiting behavior as $\\Delta t \\to 0$. This is standard in continuous-time control theory.\n\n>The proposition's correctness is questionable, as minimizing a sum does not necessarily yield a result smaller than minimizing the terms independently.\n\nWe believe there may have been a misunderstanding. As outlined in the proof, we first apply the Cauchy-Schwarz inequality to the argument of the minimization:\n\n$$ c(s,a) + \\nabla_s V(s,g)^{\\intercal}f(s,a) \\leq c(s,a) + ||\\nabla_s V(s,g)||||f(s,a)||.$$\n\nThen, by defining \n\n$$F^*(s)=\\sup_{a \\in \\mathcal{A}}||f(s,a)||,$$\n\nwe can further upper bound the right-hand-side and obtain: \n\n$$ c(s,a) + \\nabla_s V(s,g)^{\\intercal}f(s,a) \\leq c(s,a) + ||\\nabla_s V(s,g)||F^*(s).$$\n\nFinally, applying the infimum over $a \\in \\mathcal{A}$ on both sides yields:\n\n$$\\inf_{a \\in \\mathcal{A}}[c(s,a) + \\nabla_s V(s,g)^{\\intercal}f(s,a)] \\leq \\inf_{a \\in \\mathcal{A}} [c(s,a) + ||V(s,g)||F^*(s)],$$\n\nwhere the result in Eq. (6) is obtained by defining $c^*(s) = \\inf_{a \\in \\mathcal{A}} c(s,a)$. \n\nWe hope this clarifies the derivation. Please also refer to [1] for a study that highlights the Eikonal PDE as a special case of HJB PDE . To avoid any confusion, we will include the full step-by-step derivation in the final version of the paper.\n\n## W2\n> The selection of the speed function $S(s)$. Using a constant speed seems oversimplified, and it's unclear how this choice affects the regularization dynamics.\n\nWe respectfully disagree with this assessment. The choice of a constant speed function has been carefully motivated throughout the paper. In particular, lines 225-235 and 275-288 discuss the rationale behind this design choice, emphasizing its simplicity and effectiveness. \n\nMoreover, Table 1 provides empirical evidence that supports its use, showing consistent improvements when the regularizer is applied.\n\nFinally, Fig. 1 illustrates the impact of the regularizer on the learned value function, demonstrating that even with a constant $S(s)$, the regularization yields meaningful structural benefits.\n\n## W3\n> In practice, the proposed method reduces to regularizing the norm of the value function's gradient. While the Eikonal perspective provides one interpretation, alternative formulations exist, and the benefits of adopting this particular regularizer over others are not fully justified.\n\nTo the best of our knowledge, no prior work has proposed regularizers specifically designed for value function learning in the Goal-Conditioned RL setting, particularly those grounded in physics-informed principles such as the Eikonal formulation.\n\nWe have reviewed and summarized the most relevant literature in our Related Work section. If there are additional approaches we may have overlooked, we would greatly appreciate any specific references and will be glad to acknowledge and discuss them in the final version of the paper.\n\n## Q2\n> In the navigation task, the robot must handle contact dynamics with the ground, which can induce discontinuities in state transitions. Why is the proposed regularizer not affected by such discontinuities?\n\nThank you for asking this question. We are actively studying this phenomenon and hope to provide a more comprehensive answer in future extensions of our work.\n\nTo the best of our understanding, in tasks such as AntSoccer and robotic manipulation, the presence of external objects induces more severe and non-local discontinuities in the state space. These arise from contact events, such as collisions and object interactions, and are directly encoded in the state representation. As a result, the value function exhibits more pronounced discontinuities compared to standard locomotion tasks.\n\nIn contrast, in locomotion tasks, although contact dynamics are present, the transitions tend to be internally consistent and more predictable. This leads to smoother value functions, which align better with the assumptions of our regularizer. These observations explain its stronger performance in locomotion settings and highlight the need for more structured representations or multi-body modeling when applying similar techniques to manipulation-like domains.\n\n**References**\n\n[1] Cacace S, Cristiani E, Falcone M. Can Local Single-Pass Methods Solve Any Stationary Hamilton--Jacobi--Bellman Equation?. SIAM Journal on Scientific Computing. 2014;36(2):A570-87."}, {"review_id": "V5qBoyyYSJ", "reviewer": "Reviewer_zhNd", "summary": "Value estimation is a hard problem that can directly affect the performance and generalization of policies learned via offline RL, especially in the goal-conditioned (GC) setting. Prior work has explored using a physics-informed (PI) regularizer, via the Hamilton-Jacobi-Bellman (HJB) equation, which requires explicit access to system dynamics. Instead, this paper proposes the use of the Eikonal PDE, which is model free and induces a distance field structure in the estimated value function. The Eikonal PDE relates a speed profile function to a travel time function between a state and goal, via its spatial gradient. A high speed profile results in low travel time, and vice versa. The authors justify the use of the Eikonal PDE by relating it to the HJB equation. Specifically, they show that the corresponding Hamiltonian in the HJB equation is upper-bounded by an Eikonal residual, with the travel time function set to the value function and making a specific choice of local speed profile involving the dynamics and cost functions. Using this connection, they propose a general regularization term inspired by the Eikonal residual. It leaves the speed profile as a general function which can be chosen by the practitioner. \n\nThey implement this regularized value function estimation protocol in the context of hierarchical implicit Q-learning (HIQL), an algorithm for offline GCRL, and evaluate the approach on environments from OGbench, which contains multiple goal-conditioned tasks. They consider three variants of their algorithm, making different choices for the speed profile term, including a constant, exponential, and linear function of distance to obstacles and goals. Additionally, they compare to a regularization term inspired by the HJB equation, as used in prior work. On a specific pointmaze environment, across two types of datasets and four maze sizes, they find their approach, Pi-HIQL with a constant speed profile, outperforms the other variants and the HJB baseline. They then compare Pi-HIQL with a constant speed profile to standard HIQL across multiple environments. We see Pi-HIQL making substantial gains across multiple environments, especially in the case where it must stitch together trajectories from the dataset. However, they find in contact-rich domains that their approach underperforms standard HIQL. Finally, they compare to other offline GCRL algorithms and show that their method does better in long-horizon and large-scale environments.", "questions": "- Is the choice of a constant S(s) = 1 motivated by the goal-conditioned reward discussed in Section 3, which has a maximum norm of 1? Do you think the more complex alternatives would outperform the constant speed profile with more complex reward functions?\n- What are some strategies for overcoming the limitations in environments with discontinuities? Would this primarily occur in the choice of distance function used in the speed profile?", "limitations": "- The authors clearly state that this regularization is especially well-suited to environments without sharp discontinuities. They also show this in their evaluations, with their approach underperforming in contact-rich tasks.\n- They also fully acknowledge that this regularization term requires access to obstacle locations, which may be a strong assumption in some settings.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Offline RL and goal-conditioned RL are both important frameworks where improved value function estimation can have a big impact, and this is well-motivated in the introduction of the paper.\n- The evaluations clearly show that their method outperforms both the unregularized version of HIQL and an HJB-inspired regularization term on many environments. They also consider three variants of their approach which make different choices for the speed profile function. And they compare to other choices of offline GCRL algorithms as well, making their evaluations fairly rigorous.\n- In cases where their method underperforms, it is well justified by the assumptions made with their regularization strategy. The authors are upfront about these limitations, showing clearly when this approach is most applicable and when it may not be as useful.\n- The connection to the HJB equation is well-formulated, and the evaluation does a good job comparing to an HJB-based regularization strategy.\n- Figure 1 provides a nice visualization of how the physics-informed regularizer can improve value function estimation in the goal-conditioned setting.\n\nWeaknesses:\n- Modifying the choice of regularization term is a fairly minor, straight-forward change, especially in the context of prior work that regularize with the HJB equation. However, it is well-motivated in the paper and evaluated well.\n- In section 5, the different choices of speed profiles make use of a distance function, d(s), which is never clearly defined. It is only visualized for the pointmaze environment in Figure 4. It would be helpful to have these terms clearly defined somewhere, preferably in an environment-specific way.\n- While Figure 5 provides a nice visualization of the learned value function contours, I find it hard to see how \"Pi_HIQL Exp and Pi-HIQL Lin display artifcats even near the goal\". Some annotations of the plots to help illustrate this would be helpful. The main thing I can see is how Pi-HIQL HJB fails to capture maze geometry, which is crucial too.\n- A minor point: I would suggest expanding acronyms in the introduction of the paper, even if they were defined in the abstract for clarity.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed summary and thoughtful review. We truly appreciate your evaluation and the time you dedicated to providing constructive feedback.\n\nWe have carefully addressed your comments and questions in the rebuttal, and we are grateful for your suggestions, which have helped improve the clarity and overall quality of our work.\n\n## W2, W3, W4\n> In section 5, the different choices of speed profiles make use of a distance function, d(s), which is never clearly defined. \n\n> While Figure 5 provides a nice visualization of the learned value function contours, I find it hard to see how \"Pi_HIQL Exp and Pi-HIQL Lin display artifcats even near the goal\". Some annotations of the plots to help illustrate this would be helpful.\n\n> I would suggest expanding acronyms in the introduction of the paper, even if they were defined in the abstract for clarity.\n\n\nThank you for pointing out these issues. We will revise the final version of the paper accordingly to address all the raised concerns.\n\nSpecifically, we will explicitly define the distance function $d(s)$ in Appendix B, add annotations to Fig. 5 to highlight the observed artifacts in Pi_HIQL Exp and Pi-HIQL Lin, and expand all acronyms upon first use in the introduction to improve clarity for the reader.\n\n## Q1\n> Is the choice of a constant S(s) = 1 motivated by the goal-conditioned reward discussed in Section 3, which has a maximum norm of 1? Do you think the more complex alternatives would outperform the constant speed profile with more complex reward functions?\n\nThank you for this insightful question. We do believe that more expressive, state-dependent speed profiles $S(s)$ have the potential to outperform the constant choice $S(s)=1$ in settings with more complex requirements and reward functions. This is primarily because $S(s)$ enables direct shaping of the value function, introducing useful inductive biases into the structure of $V(s,g)$. Such biases can lead to a more efficient approximation of the optimal value function $V^*(s,g)$.\n\nWhile the constant profile performs well in our current setting, we are actively working on extending our framework to richer environments, where the advantages of structured speed functions can be more clearly observed. We look forward to sharing results in this direction in future work.\n\n## Q2\n> What are some strategies for overcoming the limitations in environments with discontinuities? Would this primarily occur in the choice of distance function used in the speed profile?\n\nThank you for asking this important question. We agree that the choice of the speed profile can help mitigate some of the challenges posed by discontinuities in the value function, although it may not provide a general solution to the problem.\n\nIn environments where the optimal value function $V^*(s, g)$ exhibits discontinuities and is not globally differentiable, our Eikonal regularizer, which relies on gradient information, should ideally be applied only on differentiable regions of the state space. One possible strategy is to design a speed profile that acts as a switching mechanism, effectively disabling the regularization term in regions where discontinuities are likely to occur.\n\nAn alternative direction involves using representation learning to map the state into a space where the value function is smoother or globally differentiable. We consider this to be a promising avenue for future work and are currently exploring this direction in ongoing research."}]

Author rebuttal:

Meta-review: (a) The paper introduces Pi HIQL, which extends HIQL for offline goal-conditioned RL. The method adds an Eikonal PDE–based regularizer to the value-learning objective. This regularizer enforces a distance-like structure by encouraging the gradient of the goal-conditioned value function (GCVF) to have constant magnitude. In practice, the authors augment HIQL’s expectile TD loss with an additive Eikonal penalty. This eikonal PDE prior is most conducive for domains with distance-like value/reward landscape (e.g. navigation). This formulation connects naturally to the hamilton jacobi bellman equation from optimal control and is a special case (more restrictive assumption). The resulting regularizer provides a model-free geometric prior that improves robustness and generalization when data coverage is limited. On OGBench, HIQL achieves large gains, especially in large mazes and stitching tasks. The improvements are more limited in contact-rich tasks where the value function is not globally smooth.

(b) The approach is a simple, grounded in PDEs and optimal control, and it is model-free and easy to integrate into TD-style learning. The theoretical motivation is clear, with a derivation from the HJB equation and a formal link to the Eikonal residual.
The empirical results are strong, with pronounced improvements on OGBench navigation and stitching benchmarks. Although it would have been good to characterize perfiormance and limitations on a broader range of more realistic problem domains (in semi contact rich environments). 

(c) The scope of gains is mostly in navigation-dominated tasks, with limited benefits in contact-heavy domains like ant soccer and manipulation. The baseline coverage in the original submission was narrow, since the Eikonal regularizer was shown primarily with HIQL.
Although QRL and CRL were included as external baselines, there was no initial evidence that the regularizer helps other TD-based GCRL learners. The authors later committed to adding results with GCIQL and GCIVL. Some clarifications are missing in the submission, such as the exact definition of the distance function, annotations in figures, and fuller proofs and derivations. There is also a terminology issue: physics-informed is too broad, and reviewers suggested using Eikonal more explicitly.

(d) This is a technically sound paper. It delivers a low overhead inductive bias for offline GCRL. The paper shows clear wins where the hypothesis fits, i.e. smooth distance-like landscapes. The authors are also transparent about where it does not fit, namely contact-rich tasks. The connection to HJB is clear. The proposed loss is minimalistic, and the empirical gains in long-horizon navigation and stitching are both substantial and statistically meaningful.

(e) zAXA raised concerns about generality and evaluation scope, since the method was initially only shown with HIQL. The authors clarified that the regularizer is not tied to hierarchy or subgoals. Writing and clarity: XkBr asked for more precise terminology around Koopman/model-based phrasing and raised questions about sample efficiency. Reviewer 3kEX questioned parts of proposition 4.1 and the design choice of S(s) (resolved during rebuttal).
Decision: Accept (poster)