Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learning
Authors: Vittorio Giammarino
Ahmed H Qureshi
Venue: ICLR 2026 Poster (2026)
Keywords: Goal-conditioned Reinforcement Learning
Quasimetric RL
Eikonal Partial Differential Equation
Primary area: reinforcement learning
TL;DR: A novel, Eikonal PDE-constrained algorithm for Goal-conditioned Reinforcement Learning

Abstract: Goal-Conditioned Reinforcement Learning (GCRL) mitigates the difficulty of reward design by framing tasks as goal reaching rather than maximizing hand-crafted reward signals. In this setting, the optimal goal-conditioned value function naturally forms a quasimetric, motivating Quasimetric RL (QRL), which constrains value learning to quasimetric mappings and enforces local consistency through discrete, trajectory-based constraints. We propose Eikonal-Constrained Quasimetric RL (Eik-QRL), a continuous-time reformulation of QRL based on the Eikonal Partial Differential Equation (PDE). This PDE-based structure makes Eik-QRL trajectory-free, requiring only sampled states and goals, while improving out-of-distribution generalization. We provide theoretical guarantees for Eik-QRL and identify limitations that arise under complex dynamics. To address these challenges, we introduce Eik-Hierarchical QRL (Eik-HiQRL), which integrates Eik-QRL into a hierarchical decomposition. Empirically, Eik-HiQRL achieves state-of-the-art performance in offline goal-conditioned navigation and yields consistent gains over QRL in manipulation tasks, matching temporal-difference methods.

Published as a conference paper at ICLR 2026
GOAL REACHING WITH EIKONAL-CONSTRAINE HIER- ARCHICAL QUASIMETRIC REINFORCEMENT LEARNING
Vittorio Giammarino, Ahmed H. Qureshi
Purdue University
(vgiammar, ahqureshi Spuuruueeds
ABSTRACT
-ondiCondtiioeed Reinforcement Learning (GCRL) mitigates the difficulty of re- ward design by framing tasks as goal reaching rather than maximizing hand-crafted reward signals. In this setting, the optimal goal-conditioned value function naturally forms quasimetric, motivating Quasimetric RL (QRL), which constrains value learning to quasimetric mappings and enforces local consistency through discrete, trajectory-basee constraints. We propose Eikonal Constrained Quasimetric RL (Eik-QRL), a continuous-time reformulation of QRL. based on the Eikonal Partial Differential Equation (PDE). This PDF-bbsedd structure makes Fik-QRI. trajectory- free, requiring only sampled states and goals, while improving out-of-distribuit generalization. We provide theoretical guarantees for Eik-QRL and identify limita- tions that arise under complex dynamics. To address these challenges, introduce Eik- Hieearerrrhlaal QRL (Eik-Hi HHiRL., which integrates Eik-QRL into a hierarchical decomposition. Empirically, Eik- Eik-HiQRL achieves state-of- rfte-of-theeart performance in offline goal-conditioned navigation and yields consistent gains over QRL in manipulation tasks, matching temporal-differene methods.
INTRODUCTION
In recent years, Reinforcement Learning (RI.) has emerged powerful paradigm for solving complex decision-making tasks from experience, achieving success in domains such as game play- ing (Mnih et al.] 20155) Silver et al. (017), robotics (Tang et al. (025), autonomous driving (Kiran) (ct al. 2021 Sabouni et al. 2024] Ahmad et al. 2025), and industrial control (Mirhoseini et al.) 2020). Despite these achievements, the aforementioned solutions fundamentally rely on human-designed rewards, a process that is notoriously difficult and time consuming. Reward engineering introduces hand-crafted structures, additional hyperparameters, and assumptions about task solutions, which ultimately undermines RL's promise to transcend human design and ability.
Gonl-Conditioned RL (GCRL) provides elegant way to address this challenge by framing problems in terms of reaching arbitrary goals (Schaul et al.] 2015) Andrychowicz et al.) 2017). By conditioning policies on goals, GCRL offers scalable route toward flexible, goal-directed behavior without the need for manually designed reward functions. Beyond this practical advantage, recent work has uncovered fundamental geometric property of GCRL: the optimal goal-conditioned value function V*(s, g) corresponds to the length of the shortest feasible path from a state to a goal g, and therefore naturally defines quasimetric (Sontag) 1995
Liu et 2023).
This perspective underpins Quasimetric RL (QRL) (Wang et al. 2023), recently proposed GCRL algorithm that restricts value functions to quasimetric mappings, reducing the hypothesis space from arbitrary functions to a structured subset aligned with goal-reaching tasks. In this framework, GCRL is recast as an optimization problem over quasimetrics, providing an alternative to the classical Teempoal-DDiffreence (TD) recursion (Sutton & Barto 2018). To accurately represent the optimal value function, Waan et al. (002)) show that the formulated optimization problem must preserve local distances, I.c., the costs of transitions between successive states, while simultancously pushing apart distant states. Consequently, QRL is framed as the maximization of a quasimetric function where local consistency is enforced through trajectory-based constraints.

---

Published as a conference paper at ICLR 2026
Building on these foundations, we propose novel formulation of QRL that enforces local constraints directly in continuous time rather than through discrete transitions. We argue that for many interesting cases the discrete-time perspective offers no fundamental benefit beyond practical convenience, as the continuous-time formulation naturally leads to Partial Differential Equations (PDEs). Historically, the difficulties of solving PDEs has limited their role in RI., but recent advances in Physics-Informen Neural Networks (PINNs) (Raissi et al., 2019) have begun to overcome this barrier. By exploiting automatic differentiation, PINNS embed PDE constraints directly into the training objectíve, enabling neural networks to approximate PDE solutions.
This progress opens the door to a continuous- time perspective on QRL that brings new structural advantages. Most notably, our PDE formulation becomes trajectory-free: rather than relying on short rollouts, it only requires sampling random states and goals from the feasible space. Furthermore, beyond enforcing local consistency, the PDE acts as an implicit regularizer on the value function (Gi ammarino et al., 2025), improving both learning stability and out-of-distribution estimation accuracy.
Contributions We summarize the contributions of our work as follows:
1. Our main contribution is the introduction of novel PDE-constrained formulation of QRL that derives continuous local constraints from the Eikonal PDE (Noack & Clark] 2017), leading to Fik-QRL. We provide theoretical guarantees for Fik-QRL. and highlight inherent limitations in complex dynamical settings (ef. Section 4).
2. Building on Eik- Eik-QRL, we propose Eik- Hikerrrrrcchcal QRL (Eik-HiQRL), hicrarchical algorithm that mitigates these limitations while preserving the strengths of Fik-QRL (ef. Section (5. Eik- HiQRL achieves state-of-the-art (SOTA) performance on the OGbench navigation suite (Park)
al. 2024a) and delivers consistent improvements over QRL in manipulation tasks, while also matching the performance of TD-based algorithms.
3. Other minor contributions include an additional theoretical analysis supporting the hierarchical design of Eik-HiQRL (ef. Appendix B and an experimental evaluation protocol that accounts not only for goal-reaching success but also for collision avoidance, an aspect often overlooked in the RL literature.
RELATED WORK
Goal-Conditioned RL GCRL was introduced to address two longstanding challenges in classical
RL (Sutton & Barto 2018). The first is the difficulty of designing reward functions, a time consuming process, prone to unintended side effects. Reward shaping (Ng et al.] 1999) has been proposed to ease this burden, but it remains subject to reward hacking, where agents exploit loopholes maximize rewards without achieving the intended behavior (Amodei et al. 2016 Everitt et al., 2017] (021 Pan et al. 2022; Di Langosco al. 2022). Sparse rewards offer alternative hand-crafted shaping, yet they introduce complementary challenges due to the weakness of the learning signal. To this end, GCRL provides natural way to cope with this difficulty by framing tasks in terms of goal reaching, which enables learning directly from sparse success indicators (Kaelbling 1993 Schaul et al. 2013). and has motivated specialized algorithms to improve efficiency under sparse feedback (Andrychowicz et al. (017). A second limitation of classical RL is its focus on single-task settings, which restricts generalization across objectives. By conditioning policies on goals, GCRL extends naturally to
multi-goal learning and supports more general-purpose behaviors. Recent advances in this direction include Contrastive RI. (Eysenbach et al.) 2022), state-occupancy matching (Ma et al. 2022), and
QRL Waag et al. 2023).
Offline GCRL Offline RL trains policies from pre-collected datasets without further interaction
with the environment (Levine et al. 2020). This paradigm is appealing in domains where online data collection is expensive, risky, or impractical, but it suffers from distributional shift and extrapolation error when policies select actions outside the dataset support. Extending goal conditioning to this setting has given rise to Offfine GCRL, which inherits the challenges of both paradigms: agents must generalize to unseen state-gou -oal pairs while learning entirely from fixed data and without corrective feedback. These compounded difficulties have motivated range of specialized methods (Chebotar jet al. 2021] Yang et al., 2022 2023 Mezghani et al.) 2023 Sikchi et al.) 2023 Park et al. 2024b a) aimed at improving stability and generalization. Our work contributes to this line by introducing PDE- based constraints that improve generalization to dustr--distribution state-goal pairs, with particular benefits in large environments and in scenarios that require trajectory stitching.
2

---

Published as a conference paper at ICLR 2026
Quasimetries in GCRL A key advance in GCRL is recognizing that the optimal goal-conditioned value function corresponds to the shortest feasible path between a state and a goal, and therefore naturally defines quasimetric (Sontag 1995 Liu et al. 2023). As mentioned above, QRL (Wang et al. 2023) builds on this observation by restricting value functions to quasimetric mappings, aligning the hypothesis space with shortest-path goal reaching. This perspective connects to earlier work on quasimetries in representation learning (Sontag 1995; Pitis et al., Durugkar et al.
2021) and
has inspired the development of differentiable architectures (Wang & Isolat Liu et al. 2023 that
provide universal approximation guarantees for quasimetric parameterized functions. Our work establishes PDE-constrained QRL as a new direction, reformulating value learning in continuous time and revealing structural properties that extend beyond existing formulations.
Continuous-ting Optimal Control and PINNs The continuous-time view of optimal control is traditionally formulated through PDEs, most notably the Hamilton-JacobiiBB Belobi- (HJB) equa- tion (Munos 1997| Munos & Bourgine 1997). Although these formulations capture both local and global properties of value functions, they are notoriously difficult to solve in practice, which has long limited their impact in RL. More recently, PINNs (Rudy et al. 2017 Raissi et al. 2019 Bansal & Tomlin (02T) have leveraged automatic differentiation to embed PDE constraints directly into neural training, enabling progress in areas such as system identification and scientific computing. This paradigm has also been applied to RL: (Shilova et al. (2023) used PINNs to approximate HJB solutions in continuous-timg RL, though their results were limited to simple dynamics and low-dimensional state spaces, while other works have employed PDE- PDE-inspired regularizers to improve value estimation accuracy in Offline RI. (Lien et al. 2024) and Offline GCRL (Giammarino et al. 2025). In contrast, our approach incorporates PDE constraints into the QRI. framework, yielding continuous-time formulation that offers structural benefits not explored in earlier work.
Hierarchy in RL Temporal abstraction is a long-standing idea in RL, where policies are decom- posed into high- and low-level components to improve exploration and credit assignment (Sutton) et al. 1999b Vezhnevets et al.) 2017 Nachum et al. 2018 Giammarino & Paschalidis 2021). Recent work has shown that this structure particularly effective in GCRL, where subgoal decomposition mitigates the challenges of long horizons (Park et al. 2224b).. In our framework, hierarchy is essential, as it both mitigates the practical limitations of Eik -RL and amplifies its strengths, thereby further improving value learning and overall performance in long-horizon tasks.
3 PRELIMINARIES
Coal-Conditioned RL We consider a finite-horizon discounted Markov Decision Process (MDP) defined by (S, G.A. T, P.. Pu. where S is the state space, G the goal space, A the action space, T:S x - PS)) the transition map with P(S) the set of distributions over S, and R : S x G - R goal-conditioned reward function such that R( g) -1 when s * g and R(s, 9) = 0 otherwise. Furthermore, Pg E (() is the goal distribution, Po E P(S) is the initial state distribution, and y E (0, is the discount factor. Unless specified, we assume g S and g P, the start of each episode. The agent aims to maximize
((xx) = E,, (g) IE y'R(st, g)], where T (g) = (g. s0. ag. $1, a,...., ST) are trajectories generated by the decision policy π : x G + P(A). The value function induced by æ is defind as V*((s.)) g) = Er,(g) |E r'R(s..g) S, = G = g). In the offline setting, the agent must optimize J(T) using only a fixed dataset D of trajectories (So, On, 81; 82,-.., 8T). Finally, we denote parameterized functions no where A O c R
Quasimetric RL Given X C Idd compact subset with non-empty interior, a quasimetric function defined as d X x X E20 such that V2,, I2, 23 the triangle inequality holds:
d(11, I x2) + d(z2, 23);
and Vx, d(x, x)- = 0. We denote by Qmet(X() the space of quasimetrics over X' x X. QRL is formulated on the observation that, in markovian environments, the optimal goal-conditioned value function, (* = -d" (s. belongs to the space of quasimetrics. Hence, -V* E Qmet(S), As a consequence, GCRL can be re-framed as an optimization problem over Qmet(S). Wang et al. (2023) argue that, in order to capture both local and global relationships between states, this optimization should preserve local distances (i.e., transition costs) between nearby states while maximally spreading out distant ones. This motivates the following formulation:
max E R.ee panl (S (dd((s.g))
s.t. Fe Doss))<D [max(do (s, 8') cost, 0) S e2
(1)
Glohal Relationships (GRs)
Local Relationships (I.ss)
3

---

Published as a conference paper at ICLR 2026
where: cost = -R(ss (R(sss'), s'), Pstate and Pgoal de- note respectively the marginal state and goal distributions induced by the trajectories dataset, ç is monotonically increasing transformation that dampens large distances do (s. g). and do is a parameterized quasimetric model, i.c., dg E (S) A diagram for QRL is provided in Fig. The optimization problem in ( guaran- tees optimal value recovery under the assump- tion that quasimetric models are universal func- tion approximators, i.e., capable representing any function in Qmet(S) up to an >> 0.
IQE
n
+0
d(s,g) V(s,g)
Figure 1: Diagrams for QRL (Wang al. 2020).
QRL consists of a quasimetric value function pa- rameterized by an Interval Quasimetric Embedding (IQE) model (Wang & Isola) and an actor π.
Continuous-time Optimal Control defined by 8(I) - ((()), a(t), with 2 0 and where s(t) denotes the state the system at time i, a(l) its time derivative and a(t) the control action. Given the initial conditions x(0) 80, a(O) do and a goal x(T) = g. the undiscounted optimal control problem seeks to minimize the cumulative cost J for c(s(t), a({))dt where c(s(t), a(t) is the instantaneous cost function. The optimal value function is defined as d' (s, g) = infa()) So c(s(t), a(t)),t, and, for sufficiently small At, it satisfies the principle of optimality
When in continuous-time, we consider dynamical system
d' (8.9) inf lcf, a) + d' (s(t At). aEA
Here we intentionally reuse the notation d' (s. g). as in the quasimetric setting above, and will later establish a direct connection between the two. Under standard regularity conditions (Assump- tions
4.2 4.3 and 4.4). d' (s(t + At). in (2) can be approximated by its first-order Taylor
expansion Giammarino al. 2025). yielding
inf [c(s,a) V.d" (s.g) a)] aEA
(3)
Eq. 43p is the HJB PDE for the undiscounted problem defined above and highlights the relationship between the dynamics, cost, and value function in continuous time.
EIKONAI-CCONTTRIIN QUASIMETRIC RL
In this section, we introduce and discuss our Eik-QRL algorithm. We begin by stating the assumptions underlying our formulation, then derive continuous-time variants of quasimetric value learning, including Eik- MIKLQRII as novel perspective on QRI. We conclude with theoretical analysis that highlights Eik- Fik-QRL structural properties and limitations motivating hierarchical extension.
Assumptions We begin by stating the assumptions underlying our formulation.
Assumption 4.1 (Compact state and action spaces). Let the state space S and the action space AA be compact, and let K. C S denote the feasible (obstacle-free) subset.
Assumption 4.2 (Lipschitz dynamics). The transition map T : A + S is deterministic and Lipschitz continuous in s. That is, there exists a constant LT > O such that for every s, s E IC and E A: [ a) T(s', a)l < LT ||s s'll. Equivalently. in the continuous-time limit At - we have as = 8 + f(s, At + of At), where f(, a) is Lipschitz continuous in (uniformly in a).
Assumption 4.3 (Cost regularity). We consider a continuous, bounded, nonnegative running cost c: S:Sx - R20 where c(s, g) denotes the instantancous S S while pursuing goal E G. Specifically, we assume a unit running cost away from the goal, that is, c(s,g) 9) = if g and c(s. g) if A g, for all g G. The value function d(s. g) is obtained by accumulating this cost along the trajectory until the goal is reached.
incurred when the agent visits state
Assumption 4.4 (Value function regularity). For each fixed goal E KC, the optimal value function d' (:.g) exists, is finite on X, and is locally Lipschitz.
These assumptions are standard optimal-contin regularity conditions that ensure sound HJB formulation MMuos & Bourgine| 1997, Munos 1997] Kim et al., (202I). Assumption 4.1 g.gaptionnn.guuaatttttte guarantees that the minimization over actions in Eq. (3) well posed, i.e., the arg min in the HJB operator exists.
4

---

Published as a conference paper at ICLR 2026
ensures that the system dynamics admit unique and well-defined trajectories since Lipschitz continuity of f(s a) guarantees existence and uniqueness of ODE solutions (Coddington & Levinsonn [955). This in turn makes the value function and the associated HJB PDE well posed, and provides stability for local expansions and numerical approximations such as ((x a) + f At + o(At). Determinism is imposed mainly for analytical convenience and is consistent with prior work (Kumar et al. 2019 Wang et al., 2023); importantly our method still applies in stochastic settings, as illustrated by our teleport experiments in Section Finally, Assumption 4.. ensures finite path costs, and Assumption 4.4 permits point-wiss wise Taylor expansions of value functions.
Assumption 4.2
Formulation Let the dynamics be T(s, a) = s' = s f(s,, a) At + of ((t)) and the instantaneous cost be c(s, 9). such that the one-step cost is equal to c(s,g) g) Át + o())). Note that, following GCRL, the cost does not depend on actions. QRL in Eq. (I) imposes local consistency through the constraint
(( max (do(s, 5') 0)²
e
where, compared Eq. we set the cost equal to 1 aligned with the GCRL framework. This constraint penalizes violations of the inequality d(s, I c(s,g) At + of At) along observed transitions (s,s'), s'). We now establish connection between Eq. and the HJB PDE in 3
Consider the triangle inequality 9) I d(s,s') + d(s', 9). by replacing d(s, s') S c(s,g) At o(At), we get
d(s,g) S c(s,g) At + d(s',g) + o(tt).
(5)
Assuming d((,g) is differentiable at s, the first-order Taylor expansion d(ss, qgbeeeeeee becomes d(s' ((''.)))
d(s,g) + v.d(s,g) f(s,a) At + o(At). Replacing d(s', .)) in Eq. (5) yields d(s, g) <
c(s g) At + d(s.g) + v.d(s s, 9) f(s, a) A + o(D(). By subtracting d(,,g) from both sides,
dividing by At and taking the limit At +
get
0 S (ss + v.d(s,g) f(s,a).
(6)
which is the static HJB inequality and holds true for every a E A and d(-). Under the assumptions stated above, by minimizing over and assuming d(-) d' Eq. (6) tightens to the HJB PDE
0 = c(s.g)= + (s,") f(s.a))
where at arg min, Vad (s,g) g) (s)
In the hypothetical case that a" were available, it becomes natural to view the HJB residual in (6 as surrogate for the discrete local constraint in (4). In practice, if the goal g is chosen as a state visited along the same trajectory as the transition (s, s') (Andrychowicz et al. 2017). one can reasonably approximate f(s, a²) =
s' 8, which leads to the following HJB-QRL formulation:
max E..g (c(do(s 9)]]
s.t. (v,do(s.9) ((s,s)) (V
s) s) +)) (
S 2.
(7)
Global Relationships (GRs)
IJBB-Local Relationships (IUB LRs)
This formulation offers PINN-style perspective on goal-conditioned value learning, which extends beyond purely additive regularizers explored in prior work (Lien et al. 2024 Giammarino et al. 2025). However, the problem in (7) becomes difficult to optimize in practice, as the inner product Voda(e (s.g) g) (s' - s) is often ill-conditioned in high- nigh-dimensional state spaces, a challenge also noted in many PINN- approaches to HJB (Shilova et al. 2023 Bansal & Tomlin 2021). Furthermore, retains dependence on transition tuples (s, 8') reflecting QRL's reliance on trajectories.
As result, in order to tackle the aforementioned practical challenges, we propose to refine HJB-QRL by imposing explicit assumptions on the system dynamics f(s, a). Specifically, we simplify the HJB PDE to an Eikonal PDE (Noack & Clark 2017) by leveraging unit-speed, isotropic dynamics f(s, a) = a with |a|| < 1. In this setting. it follows that a" = v.d(s, g)/llV,dd ///VV, (s. g) reducing the local constraint to unit-slope condition. This yields the following Eikonal- constal-co QRL (Eik-QRL) formulation:
max Es,y y ((do(,
S.I. EgrPaue Par (V).do(s 1) 1)²]
< e
(8)
Glohal Relationships (GRs)
Eikonal-Local Relationships (Eik-LRs)
5

---

Published as a conference paper at ICLR 2026
Remark 4.5 (Benefits). Relative to QRL and HJB-QRL, Eik-QRL becomes trajectory- it requires only i.dd. samples of states and goals (s, 9) rather than full rollouts or transition pairs (s, s'). This property is particularly appealing in applications such as (i) navigation with maps, where and g can be drawn uniformly from free poses of an occupancy map; (ii) robotic manipulation, where target object end-effector poses are sampled within a collision-free workspace; (Iéi) autonomous driving and motion planning, where waypoints are sampled along lane centerlines or roadgraph nodes; and (iv) industrial or process control, where operating setpoints are sampled within certified envelope. In addition. Eik-QRL improves state-space coverage: for S C R*, each sampled pair (s.g) 9) contributes a full gradient vector Vedo(s, (s. g) E R* thereby coupling all coordinate directions at 8. Aggregating such gradients across diverse pairs has a regularizing effect, promoting global consistency, whereas QRL's constraints act only along observed transitions in D. Our experiments confirm these advantages, particularly in large environments and in stitched-data regimes.
Remark 4.6 (Limitations). A natural concern arises from assuming isotropic dynamies f(s, a) with |a|| S 1, which may appear restrictive. We emphasize that this choice primarily simplifies the problem numerically, and empirical evidence show that Eik-QRL outperforms HJB-QRL and QRL even under complex dynamics. We acknowledge, however, that this assumption commits us to a class of solutions that might not work optimally for all MDPs and, consequently, envision future work which extend beyond isotropic dynamics, while preserving similar numerical advantages, More broadly, we view both Eik-QR -RIQ and HJB-QRI. as providing a hybrid regime in which simplified dynamical models impose explicit value-learning constraints that complement model-free RL. Unlike model-based RL (Hafner et al. 2019), where models are mainly used to generate imaginary rollouts, this perspective offers potential bridge between model-free and model-based approaches.
Theoretical Analysis We now turn to guarantees for Eik-QRI. Formal statements and proofs for these results are deferred to Appendix A Optimal value recovery guarantees follow from the existence of quasimetric approximator dg. E Qmet(S) that satisfies - (s*(s))) S. dg (s. < -(1+ c) (s. g) for some small c > 0. Given such an approximator, a standard high-probabillty recovery bound can be derived. However, unlike classical QRL, establishing existence in Eik-QRL more delicate due to the gradient constraint enforced through the Eik-LRs term in (). To drive 0 while respecting this constraint, the target- (s, g) must itself obey a matching gradient regularity (Anil et al. 2019). The next lemma formalizes this regularity, ensuring that compatible dø- exists and closing the argument for optimal value recovery.
Lemma 4.7 (1-Lipschitz optimal value). Let K C K. be compuct and corvex, fix any E K. which d' ((gg) is fimite on K, and suppose the dynamics on K are the unit- anit-speed integrator s(1) = a(l) with ||a(t]|| < 1 (equivalently, T(s, a) + At + At) on K). Assume the ranning cost is constant, c(s,g) = 1 on KC \ {6} and c(g. 9) = O. Then, all s, E K, d' (.,9) is L-Lipschitz on K.
By Lemma 4.7 we have that d' (. g) (:.g) is 1-Lipschitz on K. This implies the Eikonal condition d (s.gg)l = 1 on K \ }g). so d' is feasible for the Eik-LRs constraint in (8). This alignment between the true geometry and the Eik-LRs allows a universal quasimetric approximator optimized by (8) to recover d² We leverage this fact to state our approximation guarantee.
Theorem 4.8 (Approximate value recovery). Let the conditions of Lemma 4.77 hold, and (do be the set of universal approximators of Qmct(K) in the Loe norm. Suppose that dg- E {do is solution of Eq. (8) such that dg- E c(le) == (do (o. S 11+0 +++c) c) for small > 0. Then, for any 17 0 and s ~ Patate. 9 Pgout. it holds with probability at least 1 - O (-E(()))) that Idg. (s.g) 9) + (1+e)V*(s)
I-n,
Thus, Theorem 4.8 establishes that a universal quasimetric approximator trained with Eik- Gik-QRL in 8 recovers V (s. approximately with high probability, under the sufficient conditions of Lemma 4.7 While these conditions provide clean guarantees, they are not strictly required in practice. As shown in our experiments (Section ④ Eik- QRL, remains competitive even when 1-L.ischitz property cannot be verified, in the antmaze ze tasks. This observation aligns with the fact that if the target d' is Lipschitz, then approximating it with Lipschitz function, possibly with a different constant, preserves shortest-pate geometry up a positive scaling factor (Sethian 1996), which is typically sufficient for policy gradient methods given their ialee-ivvrianne (Sutton et al. 1999a Schulman et al. 2015).
In summary, Eik-QRL offers clear structural benefits, including trajectory free estimation and PDE- based regularization, but at the cost of stronger regularity assumptions on the target value, in particular local Lipschitz continuity of the optimal value function. These assumptions are standard in continuous-

---

Published as a conference paper at ICLR 2026
High-level
Low-lovel
ICE
MLP
8:
+ ((scia
V
( 8cl)
(x = (x,
(5
('(s,))
Figure 2: Diagram for Eik- HiQRL. The high-level module consists of a quasimetric model dh, parameterized
IQE (Wang& & Isola), which is trained to optimize the Eik-QRL objective in Eq. 8). The low-level module comprises a value function V and a goal representation network Φ MLPs and trained via TD-based recursion. Both high-level policy 77" and the low-level policy n/ are trained via advantage-weighted regression Peng
S G Z (Park al. 2024b), both instantiated
al. 2019). Full details are provided in Appendix C
time optimal control, yet they are not guaranteed in general MDPs. This raises practical question: can we retain the udvantages of Eik-QRL while miligaling violations of these regularity conditions?
5 ELKONAL-CONSTRAIII
HIERARCHICAL QRL (EIK-HIQRL)
To retain the advantages of Eik-QRL while mitigating its limitations, we propose a hierarchical architecture, yielding the Eik-HiQRL algorithm illustrated in Fig. [2] This design is motivated by the following considerations. First, highlighted in previous work (Prk et al. 2024b). key challenge in GCRL is the low signal-to signal-to-noies to-soose ratio in value estimates, especially in ang-horizon tasks where the initial state and goal are far apart. Park et al. (2024b) propose hierarchical solution in which high-level actor outputs subgoals and low-level controller attempts to reach them. Building on this, value functions can have even stronger effect than bierarchy alone on mitigating this signal-to-noise problem, further justifying the use of quasimetric value learning strategies such as Eik-QRL. BRB, and QRL together with hierarchy.
show in Appendix that muasimmtrri-parameteeree
Given this result, a natural question is whether one could rely solely on a single quasimetric value function for both high- and low-level policies. Unfortunately, enforcing quasimetric structures in high- dithemmmnnii state spaces is itself challenging as the approximation error of quasimetric parameterizations grows exponentially with the state-space dimension[] A natural way to address this problem is to reduce the state dimensionality before applying quasimetric projection, either by projecting states into an embedding space or by selecting a subset of task-relevant state variables on which to define subgoals. If, in this process, approximately isotropic dynamics, we can then leverage not only the benefits of the quasimetric structure but also the advantages of the Eikonal formulation (Remark 4.5)
ensure that the resulting abstract space evolves under
Eik-HiQRL is designed precisely to reconcile these aspects: (1) quasimetric projection for the high-level value in a diwdeeensiooall abstract space, (2) PDE- based regularization via the Eikonal formulation, and (3) a hierarchical structure that improves the signal-to-noise ratio. At the high level, we define lnver-dimensional abstract space S (e.g., the positions of the agent or task-relevant objects) where the regularity assumptions of Eik-QRL hold and the quasimetric projection is tractable. This enables learning a high-level value function that improves subgoal generation, going beyond the single-value design in HIQL (Park et al. 2024b). These subgoals then guide the low-level controller more effectively, further mitigating the signal-to-noll ratio issue in value function learning.
In the following section, we provide empirical results that demonstrate the benefits of this design. For all the implementation details for both Eik-HiQRL and Eik- Fik-QRL refer to Appendix C
EXPERIMENTS
Our experiments are primarily conducted in the Offline settnng which offers a more controlled testbed than the online counterpart for evaluating value function learning and out-of-distribution generalization. In the offline case, the dataset remains fixed throughout training, making it possible to directly assess algorithm's 's ability to generalize beyond the training distribution. By contrast, in the online setting it is difficult to disentangle whether performance fluctuations arise from fortunate
'This result can he readily derived by extending Wang & Isola Theorem 3 and combining it with standard results covering numbers (e.g..(Shalev-S5 (Shalev-Shwart Ben-David 2014 Ex. 27.1)).
7

---

Published as a conference paper at ICLR 2026
(a) peintmaze
antmaze
(c) humanoid- (d)
(e) cube
(f) scene
maze
ancceccer
Figure Environments from OGbench Park al. 2024a) used in our experiments.
exploration or genuine improvements in generalization. this end, we base our experiments on the tasks from OGbench (Park et al. 2024a), summarized in Fig. 3 We provide in Appendix F more details on these environments, including a discussion of how closely their dynamics satisfy the assumptions stated in Section
This section is organized as follows. First, Table provides controlled comparison between Eik-QRL in (8. HJB-QRL in (7). and QRL in (T) highlighting the pros and cons of introducing PDE-based constraints into quasimetric value learning. The same table also includes our hierarchical formulation Fik-HiQRL, showing how the proposed hierarchical design overcomes the limitations of Eik-QRL on the antmaze ze suite. Next, we compare Fik-HiQRL against strong Offline GCRL baselines on OGbench tasks in Fig. and further stress-test it in non-regular environments (where our assumptions do not hold) in Table Appendix H complements these main results with additional analyses and ablations. Specifically, Appendix H.. and H.2 investigate the effects of different actors and optimization pipelines on empirical performance; Appendix HH.3 reports complete comparison between Eik HiQRL and Offline GCRL baselines; Appendix H.4]isolatee the impact of the Eikonal constraint within our hierarchy by comparing Eik-HiQRL to HiQRL (without the Eikonal term in the high-level value function). Finally, Appendix H.. [.. | explore how to exploit trajectory- -fee property of Eik-QRL in practice and evaluate its effectiveness in online RL setting.
Empirical Comparison of Quasimetric Formulations We begin with controlled comparison of QRL, HJB-QRL, Eik-QRL, and Eik-HiQRL, intended to provide empirical support for the theo- retical results established in the previous sections. We evaluate these algorithms on two benchmark suites: pointmaze ze and antmaze. The pointmaze ze environment offers an idealized setting with isotropic dynamics, so that the main assumptions in our analysis are satisfied. In contrast, the antmaze suite involves higher-dimenn nigher-dimensional state spaces and complex dynamics, where discontinu- ities contact points challenge the Lipschitz continuity assumption on the dynamics (Assumption (4.2). All algorithms compared across different maze dimensions and dataset variants. Performance is reported using two main metrics. The first the success rate R. defined as the percentage of episodes in which randomly sampled goal is reached. The second collision rate к, measured as the percentage of time steps during which the agent is colliding with obstacles. These results are and learning curves provided in Appendix G
summarized in Table and
few notes on implementations are in order. Eik-QRL and HJB-QRL are both implemented with a hierarchical actor, consistent with analysis in Appendix B Whereas QRL follows the original implementation of Wang et al. (2023) as presented in Fig. As mentioned, Appendix H. illustrates an additional comparison where QRL combined with a hierarchical actor (denoted QRL:hi) in order to isolate the contribution of PDE constraints versus hierarchy. All the details for these variants are provided Appendix D Finally, Eik-HiQRL follows the design introduced Section where, for all the presented navigation tasks, the abstract space is defined as the agent's coordinate space.
Turning to the results in Table in the pointmaze setting, we observe no tangible differences among the three PDE-constrained formulations, consistent with our theoretical analysis. All of them achieve improved accuracy the estimated value function, which translates into lower collision rates and stronger performance compared to QRL. Refer for instance giant mazes for both navigate and stitch ch datasets. QRL achieves competitive success rates, but the learned policies heavily rely collisions with walls, behavior that undermines performance in larger environments.
For the antmaze suite, we highlight the following observations. Compared to the idealized pointmaze setting, the performance of all purely quasimetric algorithms (Eik-QRL, HJB HJB-QRL, and QRL) deteriorates, with particularly pronounced drops the giant variants. This reflects the increased difficulty of learning accurate quasimetrics in higher-dimensional state space and with
8

---

Published as a conference paper at ICLR 2026
Table 1: Summary of the comparison among the QRL formulations. All agents are trained for 10s steps using 10 seeds and evaluated every 10* steps. We report the mean and standard deviation across seeds for the last evaluation. For each seed, evaluations are conducted over different random goals, with the learned policy tested for 50 episodes per goal. For success rate (R), results within 95% of the best value are written in bold. For collision avoidance (x) the lowest average is colored.
Eavironment
Datase
Dām
R()
Ea-HIQRL
(()
R (1)
ERL-RRL
6 (4)
R (1)
IJB-QRL
< (c)
R R())
QRL
K. (L)
nedium
2
87 +
4
88 + A
10 3
79 = 4
46 +
navigate
giant
Large
82 12
4 9
542
82=12
90 +
18 12
11 1
88 -
91 +
11 + 6
89 +
59 + 3
te lelortt
46117
3
40
49
40
46
20
po Ininnaee
nodium
10:1:2
5
10:::1
78
Large
75 +
2
78 1:
5 1
73
9
84
5
giant
62 + 22
28 + 14
13#16
19 +
70+:17
19#122
51 + 13
61#10
talcport
501:1
5 5
37
d:
29 1:9
31::22
33
78 7
median
93 + 2
18 +
84 + 8
26 +
31 31
35 3
82
25 4
navigate
giant
large
+ 7
28 3
8 #
8
2
0
telegort
49 +
40 - 3
45 1 3
38 2
26 +
42 1 2
32 = 6
52 5
cntmazo
nndiun
94 +: 1
19 2
70 + 4
32 4
37 +
38 2
66
26 +-2
wtitth
giant
Large
47::12
81 +
23 2
4-2
23 1
31 3
1
13 13.
36 1 3
15 22 6
39 1
1
colepozt
51
28 4-2
28
13 4.3
27
complex and non-Lipschitz dynamics. Despite this degradation. Eik-QRL consistently outperforms HJB-QRL, underscoring the numerical challenges of optimizing (7). At the same time, Eik-QRL remains on par with QRL (and with QRL:hi in Table indicating that the Eikonal PDE constraint does not excessively compromise effectiveness in this regime. Finally, Eik- Hik-HiQRL& achieves the strongest performance across the antmaze tasks, highlighting the benefits of our hierarchical design.
Evaluation in Challenging and Non-Regular Environments This second set of experiments is divided into two parts. The first evaluates Eik- against established Offline GCRL algorithms in the humano no dmaze navigation tasks. The second examines scenarios where the regularity assumptions do not hold, including MDPs with third-party objects and categorical variables in the state space. These experiments are designed to test performance in settings where our theoretical analysis is difficult to apply. The baselines considered in this section include Hierarchical Implicit
Q-Learning (HIQL) (Park et al. 22244). Contrastive RL (CRL) (Eysenbach et al. [2022). QRL (Wang
al. (023). and Eikonal-regularized HIQL (Eik-HIQL) (Giammarino et al. (025). More details
about these baselines, including how they differ from our algorithm, are provided in Appendix F
For the humanoi i dmaze tasks, the results in Fig. provide strong evidence for the effectiveness of our PDE-constrained hierarchical design. Hik-HIQRL consistently outperforms all baselines, with especially large gains in the long- horizon large and giant settings. The advantage most evident on large-st stitch and giant stlant-siitch where Eik-HiQRL achieves statistically significant improvements over the best-performing baseline according to Welch's t-test (1 = 11.7, p R 10-9 and = 22, p x 10-14 respectively). As also highlighted in Table when data stitching is required the PDE-constrained algorithms exhibit their clearest benefits due to the regularizing effect of the PDEs. To the best of our knowledge, Fik-HiQRL. achieves SOTA performance For additional experiments, as well as an ablation comparing Fik-HiQRL. with HIQRL (which uses QRL instead of Bik-QRL in the high-level policy), see respectively Appendices H.3 and H.4
this benchmark.
For environments involving external object interactions, we evaluate the soccer suite, where an ant agent must move a ball to designated target, as well as the cube and scene robotic manipulation tasks. In these settings, the abstract space S is defined as the concatenation of the ant and ball coordinates for ant soccer, and as a latent space learned end-to-end for the two manipulaa caanipulation in tasks. Specifically, to learn this abstract latent space we define an MLP v/ S + Z, which is trained end-to-end as part of the objective in Eq. B During training, we feed v(s)) and ((g) to the quasimetric value network, and the gradients from both the Global Relationships loss and the Eikonal Local Relationships term in Eq. (8) are baokpgaaa through v. Importantly, we do not introduce any additional auxiliary loss to explicitly enforce our geometric assumptions in the embedding space. The results, reported in Table show that Eik-HiQRL achieves performance comparable to the baselines. Unlike in the navigation domain, however, we do not observe the same level of performance gains. This reflects the non-trivial characteristies of dynamics in complex manipulation, where interactions with objects introduce additional challenges. In particular, contact
9

---

Published as a conference paper at ICLR 2026
Nomnal hed Score
0.75
0.50
0.50
0.25
0.0
0.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1,0
Steps (x10%)
Steps (x10*)
Steps (x10*)
Hurranaidazze Medium- Stitch
Humaniidmaze-Larggeee
Humanoidmaze Giant Siath-Setch
Normalzzdd Score
0.75
0.6
0.50
0.5
0.2
0.0
0.0
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Steps (x10f)
Steps (x10f)
Steps (x10f)
Eik-HiQRL (ours)
Eik-HIQL
HIQL
QRL
CRL
Figure 4:
Learning curves for the humano i dmmae experiments. All agents are trained for 10 steps using 10 seeds and evaluated every 109 steps. For each seed, evaluations conducted in Table Plots show the average success and standard deviation across seeds.
Table Summary of the comparison on non-regular environments. All agents are trained for 106 steps using 10 seeds and evaluated every 10 steps. For each seed, evaluations are conducted as in Table and we report the best evaluation. Results within 95% of the best value are written in bold.
Eaviremment
Dntaset
Dimension
Eik HIRHH (ours)
Eik-HIQL
HIQL
QRL
CRL
navigate
modiun
azera
61 +
312
19
13 $ 3
60 +
10 3
212
24 I:
4
stitch
acera
32
7
2
1
17 17
2
0
1
manipalacion
ciae-singio-piay
12 1:3
25 151:1
31 +
11 11::3
32 +
scene-ples
55 + 14
52 + 7
52 +
8 8
35 d
events often represented through categorical mode-switching variables (e.g.. in-contact vs. out- of-contact), which induce sharp discontinuities and hybrid structure the underlying value function. These discontinuities are odds with the smooth topology implicitly assumed by PDE-based constraints (Assumption) [4.4), so that enforcing Fikonal constraints in this regime introduces a bias that does not perfectly match the true optimal value landscape. We believe that these peculiarities make robotic manipulation a promising direction for follow-up work, with significant opportunities design PDE-constrained algorithms explicitly tailored contact-rich, hybrid dynamics. Learning curves for these experiments are provided in Appendix
7 CONCLUSION
In this work, we derived and analyzed Eik-QRI,, PDE-constrained algorithm that introduces a PINN- based perspective on value learning for model-free GCRL. Our analysis highlights both the strengths and limitations of this formulation, motivating its hierarchical extension. Eik-HiQRL, and outlining several avenues for future work. We complement this analysis with extensive experiments that provide empirical support for claims. These results demonstrate the effectiveness of Eik Hik-H while also underscoring the challenges of applying PDE-constrained algorithms in non-regular environments, pointing to practical and impactful directions for further research. An especially promising direction to leverage our analysis more directly for representation design. By explicitly characterizing which geometric and regularity properties are beneficial for PDE-based value learning, results clarify what structural features an embedding space should satisfy for PDE-based algorithms to be applicable. mechanisms that realize such embeddings while still supporting
Designing representation-leamii effective control remains an open challenge. Progress along this line could offer a compelling path to further bridge the gap between formal guarantees and real-world deployment. Taken together, our contributions establish both theoretical novelty and practical relevance, and we hope this work will serve as a solid foundation for subsequent advances in PDE-informed RL.
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
We have taken extensive measures to ensure the reproducibility of our results, The main algorithms are presented concisely in the paper, with design choices, loss formulations, and training procedures clearly specified. Additional implementation details, hyperparameters, and specifications are provided Appendix
C for Eik-HiQRL and in Appendix [D] for the other variants. To further support reproducibility, we include a preliminary version of our code with this submission, and we will release a cleaned-up and fully documented version with the camera-ready paper.
REFERENCES
HM Ahmad, Ehsan Sabouni, Alexander Wasilkoff, Param Budhraja, Zijian Guo. Songyuan Zhang, Chuchu Fan, Christos Cassandras, and Wenchao Li. Hierarchical multi-agent reinforcement learning with control barrier functions for safety-critical autonomous systems. arXiv preprint arXiv:2507. 14850, 2025.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, arXiv preprint arXiv:1606.06563 2016.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in Neural Information Processing Systems, 30, 2017.
Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. In International Conference Machine Learning, pp. 291-301. PMLR, 2019.
Somil Bansal and Claire Tomlin. Deepreach: A deep learning approach to high-dimensional reachability, In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 1817-1824. IEEE, 2021.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07749, 2021.
Earl A Coddington and Norman Levinson. Theory of ordinary differential equations. McGraw-Hill New York, 1955.
Lauro Langosco Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger, Goal misgeneralizatton in deep reinforcement learning. In International Conference on Machine Learning, pp. 12004-120111 PMLR, 2022.
Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. Advances Neural Information Processing Systems, 1188622-8636 2021.
Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417 2017.
Tom Everitt, Mareus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, 198 (Suppl 27):6435 6467, 2021.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35:35603-35620, 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
Vittorio Giammarino and Ioannis Ch Paschalidis. Online Daum-welch algorithm for hierarchical imitation learning. In 2021 60th IEEE Conference on Decision and Control (CDC), pp. 3717-3722 IEFE, 2021.
11

---

Published as a conference paper at ICLR 2026
Vittorio Giammarino, Ruiqi Ni, and Ahmed H Qureshi. Physics-informen value learner for offline goal-conditioned reinforcement learning. arXiv preprint (exXiv:250900678222 2025.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2565. PMLR, 2019.
Leslie Pack Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence, volume 2. pp. 1094-8. 8. Citeseer, 1993.
Jeongho Kim, Jaeuk Shin, and Insoon Yang. Hamilton-jacohi deep q-learning for deterministic continuous-time systems with lipschitz continuous controls. Journal of Machine Learning Research, 22(206):1-34, 2021.
Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909 4926, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-laarning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0166 2020.
Yun-Hsuan Lien, Ping-Chup Chun Hsieh, Tzu-Mao Li, and Yu-Shuen Wang. Enhancing value function esti- mation through first-order state- sate-action dynamics in offline reinforcement learning. In International Conference on Machine Learning, 2024.
Bo Liu, Yihao Feng, Qiang Liu, and Peter Stone. Metric residual network for sample efficient goal-conditioned reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8799 $799-8806, 2023.
Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned reinforcement learning via f-advantage regression. Advances in Neural Information Processing Systems, 10:310-323, 2022.
Lina Mezghani, Sainbayar Sukhbaatar, Piotr Bojanowski, Alessandro Lazaric, and Karteck Alahari. Learning goal-conditioned policies offline with self-supervised reward shaping. In Conference on Robot Learning, pp. 1401-1410. PMLR, 2023.
Azalia Mirhoseini. Anna Goldie, Mustafa Yazgan, Joe Jiang, Ebrahim Songhori, Shen Wang. Young- Joon Lee, Eric Johnson, Omkar Pathak, Sungmin Bae, et al. Chip placement with deep reinforce- ment learning. arXiv preprint arXiv:2004.10744 2020.
Volodymyr Mnih. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1oe.:3322.00.. 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
Rémi Munos. A convergent reinforcement learning algorithm in the continuous case based on finite difference method. In International Joint Conference on Artificial Intelligence, pp. 826-831, 1997.
Rémi Munos and Paul Bourgine. Reinforcement learning for continuous stochastic control problems. Advances in Neural Information Processing Systems, 10, 1997.
Ofr Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, volume 99, pp. 278-287. Citeseer, 1999.
12

---

Published as a conference paper at ICLR 2026
Marcus M Noack and Stuart Clark. Acoustic wave and eikonal equations in transformed metric space for various types of anisotropy. Heliyon, 3(3), 2017.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping
and mitigating misaligned models. arXiv preprint arXiv:2201.0333 2022.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking
offline goal-conditioned rl. arXiv preprint arXiv:2410.20092 20244
Scohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Higl: Offline goal-conditioned with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024b.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weiighte regression: Simple and scalable off- oof-cy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
Silviu Pitis, Harris Chan, Kiarash Jamali, and Jimmy Ba. An inductive bias for distances: Neural nets that respect the triangle inequality. In International Conference on Learning Representutions.
Matthias Plappert, Marein Andrychowiez, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Data-driven discovery of partial differential equations. Science Advances, ((4):e1602611 2017.
Ehsan Sabouni, HM Sabbir Ahmad, Vittorio Giammarino, Christos G Cassandras, Ioannis Ch Paschalidis, and Wenchao Li. Reinforcement learning- based receding horizon control using adaptive control barrier functions for safety- crriticl systems. In 2024 IEEE 63rd Conference on Decision and Control (CDC), pp. 1014001-440 IEEE, 2024.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning, pp. 1312-1320. PMLR, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897. PMLR, 2015.
James A Sethian. fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences, 93(4):1591-1555, 1996.
Shai Shalev-Shwartz and Shai Ben-David Understanding machine learning: From theory to algorithms. Cambridge University Press, 2014.
Alena Shilova, Thomas Delliaux, Philippe Preux, and Bruno Raffin. Revisiting continuous- time reinforcement learning. a study hjb solvers based on pinns and fems. In European Workshop on Reinforcement Learning (EWRL) 2023 Workshop. 2023.
Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Nickum. Smore: Score models for offline goal-conditioned reinforcement learning. arXiv preprint arXiv:2311.02013, 2023.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3554 359, 2017.
Eduardo Sontag. An abstract approach to dissipation. In Proceedings of 1995 34th IEEE Conference Decision and Control, volume 3, pp. 2702-2703. IEEE, 1995.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
13

---

Published as a conference paper at ICLR 2026
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999a.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):81-2111 1999b.
Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martín- Martin,Martin, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 28694-28698, 2025.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pp. 3540-3549, PMLR, 2017.
Tongzhou Wang and Phillip Isola. Improved representation of asymmetrical distances with interval quasimetric embeddings. In NeurlPS 2022 Workshop on Symmetry and Geometry in Neural Representations.
Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforce- ment learning via quasimetric learning. In International Conference on Machine Learning, pp. 36411-36430. PMLR, 2023.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. arXiv preprint arXiv:2202.04477, 2022.
Rui Yang. Lin Yong. Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline goal-conditioned ri? In International Conference on Machine Learning, pp. 39543-39571. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Related Work
Preliminaries
4
Eikonal-Constranned Quasimetric RL
5
Constal-Cnsstraneed Hierarchical QRL (Eik-HiQ (iLL))
7
6
Experiments
Conclusion
10
8
Reproducibility Statement
11
Theoretical Results for Section
16
Assumptions restated
16
A.2
Lemmal4.7 restated
16
Theorem restated
16
Didactic example: joint benefits of hierarchy and quasimetric
19
Fik-HiQRL
21
Overview
22
Implementation details
22
Pseudo-code and Hyperparameters
23
D Eik-QRL, Fik-QRL-> and QRLahi
24
Offline GCRL Environments used in Section 6
24
Offfine GCRL Baselines used Section 6
25
Learning curves for experiments in Section 6
26
H
Additional Experiments
29
H.I
Comparison with QRL:hi
29
Additional Ablation
30
H.3
Full Comparison Eik- HiQRL vs Baselines
31
H.4
Comparison Eik-HiQRL HiQRL
34
H.5
Trajectory-free Experiments
36
H.6
Online RI. Experiments)
36
15

---

Published as a conference paper at ICLR 2026
H.6.1
Pointmaze
37
H.6.2 PetchReach
37
A THEORETICAL RESULTS FOR SECTION
A.1
ASSUMPTIONS RESTATED
We begin by restating the assumptions introduced in Section 4 Unless otherwise specified, these assumptions are assumed to hold throughout this section.
Assumption A.1 (Compact state and action spaces). Let the state space and the action space be compact, and let K S S denote the feasible (obstacle-free) subset.
Assumption A.2 (Lipschitz dynamics). The transition map T S x A- S is deterministic and Lipschitz continuous s. That is, there exists a constant LT such that for every 8, s' KC and E A: [ a) T(s', a)l I LT ||s - I'Il. Equivalently, in the continuous-time limit At -> 0, we have (s, = s f(s, a) At + ((S)), where f(,, a) is Lipschitz continuous in s (uniformly in a). Assumption A.3 (Cost regularity). The cost с 8 + Rz0 is continuous, bounded, and nonnegative. Moreover, c(g.g) = 0 for all E G, consistent with the shortest-path interpretation.
Assumption A.4 (Value function regularity). For each fixed goal E I, the optimal value function d" (:gg) exists, is finite on IC, and is locally Lipschitz.
A.2 LEMMA RESTATED
Lemma A.5 (1 (1-Lipschitz property of the optimal value function), Let K C K. be compact and convex, and fix any g E K. such that d' (:,g) unit-speed integrator
g) is finite on K. Suppose the dynamics on K are given by the
8(1) = a(I), [|a(t)| 1
equivalently, T(s, a) sta s+aAttol (At) on K. Assume the running cost is constant, with c(s, g) for s # and c(g.g) = O. Then d' (..g) is 1-Lipschitz on K, i.e.,for all s' E K,
(d' g) - d' (s', 9)1 I || ss sl'l.
Proof. By the triangle inequality for d", d' (s. (s',g)) < max(d" d' (s, d' d'ss'ss)
Since K is convex and the dynamics are å = with ||||| < 1, the straight- -inee trajectory from to is feasible at unit speed and requires time ||s s'Il. Hence d' d'(s, s') S ||s - s'll and d' (s', 8) S I|s s'|l- Substituting into the inequality above proves the claim.
A.3 THEOREM 4.8 RESTATED
Before restating and proving Theorem 4.8 we introduce the following auxiliary lemma. Lemma A.6 (Bound for Gradient- Regularz-e Quasimetries). Define the feasible region
F (s,g) x (*(*.)) > -xc).
Suppose the conditions of Lemma A.5 hold, and therefore, we can assume at the same time:
(Al) Gradient constraint: dg E Qmet(S) is continuously differentiable in s and satisfies
[[V.do(s, 11+,, V8,9EK
(A2) Approximation of relaxed value: dg is universal approximator of Qmet(KC in the Loc norm. Therefore, approximates the value function V of relaxed-cost MDP with reward R(s, a) = -1 - -/2,, up to additive error e/2:
(do(s.)) + V-' (s. S Vs.gES.
16

---

Published as a conference paper at ICLR 2026
Then, for all (s.9) E F. the following bound holds:
(v) I do(s,g) I =1++
Proof. Lower bound. This part of the proof follows from Lemma in Wang et al. (2023). In our case, however, assumptions (A1) and (A22) can only be satisfied simultaneously under the conditions stated in Lemma A.5
Let V denote the value function of the relaxed-cost MDP with reward R(s, 8) = -1 - e/2. Then:
-
By assumption (A2),
do(s,g) (0959)2-V'(660
E - (V
-
If * g. then (*(sgg) -1,
e/2) - c
- 2 ~V
hence
d -V's.
If = then V (s. = 0, and by definition of a quasimetric and goal-reaching cost, we also have (s = (s. g) = 0. Thus, the inequality holds with equality.
Upper bound. Let so = s + 81
ST = be an optimal path of length = - (s,
Using the triangle inequality:
T-1
do (s
Σ 1=0
do (8t, 81+1).
Define the interpolation (e( = 8 + u($141 8t), for u E [0, 1]. Note that the whole one-step
interpolation path lies within the feasible space K. Then:
do d0 81+1) L' v=do(T((u))9) (81+1
88) (((4)) - 8t) du & / | (V=do(T(()))))) ||8e+1 s du.
Since V S 1 and for At small enough ||s241 sell S get: do(s S1+1) 5 (1+
Summing over the T' steps we obtain:
de < T (1
Leading to the expected bound:
< (1.
Provided Lemma A.6 we can now formally restate and prove Theorem 4.8
Theorem A.7 (Approximate Value Recovery under Gradient Constraint). Define the feasible region
F {(.9)) EK K. V (s.g) > -x).
Suppose the conditions of Lemma hold and furthermore
(i) ( is family of quasimetric models, i.e.. set of universal approximators of Qmet(S) in the Loo norm;
(ii) The model dž E (o is a solution to the constrained optimization problem:
min max Es,g (-((do(8,))) A ABs. (()V)d((>.9)))) 1)²] Ae? XSO
for some strictly increasing conver function S. such that dī E grad
((e)
(do : ||V,do(s,g)) - +1 E
17

---

Published as a conference paper at ICLR 2026
Then for all (8,g) F, it holds that:
- (8,9) S do- (8, < + =
Moreover, any v) > O, and Pstates! 9 Pgoat have:
with probability least 1 1- 0 ( (-Es.o
dg (s:g))
+ +1 +1+ Sn
Proof. By Lemma A.6 and under the assumptions of the theorem, we have for all (s,g) E F
(s. - dg- (s,g) - (1+ <V)
(9)
This bound means the learned quasimetric overestimates the cost (i.e., is more pessimistic than -V*), but not more than factor + E.
We now quantify how close dg- (s, g) is to the upper bound -(1 + <>)* with high probability.
Define the bad event. Let r7 > 0 be small tolerance. Define the event:
= {(s g) do- (s.g) < -11+ - -/)
This is the "bad" event where the quasimetric overshoots the pessimistic bound by more than 1y. Let p be the bad event probability p P for all ~ Psuse and g ~ Paoal.
Upper-bound the expectation. Split the expectation of dg- over the event and its complement E*: Ejdg-1 = E[dg-
E] p E(dg-
E (1-p).
On F,, by definition of the event:
do- (s,g) g) S (1 c) (s.
n
Bldg-
E
[VL] n I -(1
On E by the upper bound from equation 9 dg. (s. g) < -1 =
Eldg.
ET < +1 e
Putting these together:
E E(dg-] S p. 1(-
= --(1+))B[V pr/.
Lower-bound the expectation. From the lower bound of equation 9
dg- (8, g) V (v,.
E(do- -V(.
Conclusion. Putting both bounds on E[dg-] together:
- -11+ - pn 2 E[V]
= Ps " (-E(V")).
With probability at least
c (-E(V*)),
the bad event E does not occur. That is.
dg-(s.g) 2 - (1+ 17-
Combined with the always-valid upper bound dg- (s.g) S -(1 + (sV"(s,)) g). we conclude:
[dg- (s.g) + eL E I- 0] with probability at least 1 1 ( (-EVV")
18

---

Published as a conference paper at ICLR 2026
B DIDACTIC EXAMPLE: JOINT BENEFITS OF HIERARCHY AND QUASIMETRIC
In the following we provide additional theoretical results that justiff the use of a quasimetric value function together with hierarchical actor.
We consider goal-conditioned value function V(s.g) the ID environment in Fig. 5 where agent moves toward a fixed goal state from an initial state s, making decisions based on gs. The true optimal value function (assuming a step cost of -1) is given by:
Figure 5: Toy environment.
V (s.g) s gl TT
The learned (unprojected) noisy value function follows multiplicative Gaussian noise model:
V(s. g) = (s. + +Zzgg) Zs.g NO,,
(10)
with noise magnitude > 0.
We then apply an approximate quasimetric projection to enforce triangle inequality constraints:
( 2 (88 81) + (81. g), Vs, 81.91
This projection modifies the noise structure by decomposing ong-horizon estimates into 1-step segments:
((s Σ 8(0), 8(0) = -11 -1, ~~~100111
(11)
In the following, we state Proposition B.I which provides novel bound on the probability of actor's eтror under quasimetric value function parameterization. We provide complete proof for Proposition B.T] and include a brief comparison with the corresponding results obtained without quasimetric projection in Park et al. (2024b).
Proposition B.1. In the environment of Fig. 5 after applying the quasimetric projection described
above, the error probabilities of the flat policy a and the hierarchical policy n' n/hh are respectively bounded by:
V2
E(T) S +
0
/T(T
T(1
Pfatt
V2k
V2
E(xl x
/T/T(1 Phigh)
0v/k(k Ptow)
where T is the full horizon, the number of steps after which Th selects a subgoal, & denotes the cumulative distribution function of the standard normal distribution, ((x) = Piz < x]
/2x I e dt, and E [0, 1] is the correlation coefficient between the total noise accumulated along the two neighboring values that the policy compares.
Proof. The proof follows similarly to (Park et al. 2024b Proposition 4.1), however under the approximate quasimetric projection model.
Flat policy. The flat policy chooses between (ss 1,g) and (s 1,g), so define the difference:
A nat = (( + 1,g) V(s - 1,,)) = a(22 - Si):
where S -T Z, S2 Σ1- Z%. and Prit = Corr(S, S2). Then:
T-1
Var(Ana) = 0² [T + 1) + (T - 1) - 2ppa / (T + 1T 1)] 20 2T - pua),
yielding:
E(T) = P||Aaa < <0 5
a 02/2(1 Pna
19

---

Published as a conference paper at ICLR 2026
Hierarchical policy (high-level). Let the high-level policy compare subgoals at k and + k.
Define:
nigh ((s + k. V(s k. 2/: o -(),,
Let S1 = Eiet Zi. S2 = -i=1 Z; and assume Corr(S1, (1,5,, S2) Phigh- Then:
Var Aighh) = o² Var(S² - 51) = o² (T k) + (T - k) - Phiigh / (T' k) k)
Var (nigh)) o² 21 2high v VT2 k2
When 7' >> ki we use vT2 - 12 NT T 27 yielding:
Var Var(Anigh) 20 Phigh).
S0:
E(T²)) S +b
0 /
phigh
Hierarchical policy (low-level). For reaching chosen subgoal (horizon k), define:
Aoo V(s + 1 k) V(s 1, + k) o S1),
where each sum is over l + terms. Then:
Var(A R 202k(o1 -low),
and:
2
E(x') S +
22:(1 Plaw)
Total hierarchical error. Using union bound:
E( o x²) s E(Th) ) + E(T'),
which concludes the result.
In summary, under quasimetric projection we get from Proposition B.1
v2
E(T) $ $
(12)
0 / T(1 [1-
Pfa
22k
V2
É n S + Without quasimetric projection from Park al.
(13)
T T(1
Phigh
J. k:(1
Plow)
2024b
Proposition 4.1) we get:
2
E(T) < Φ
(14)
Tv +
V2k
V2
E( o x < $
+ ①
(15)
TvT +
o
+
Eik-HiQRL gets
22k
v2 ov 12
ÊER-HIQRL (T' o
x²) < +
+
(16)
0 T T(1 Phigh
+
From Proposition B.1 the tightest upper bound is obtained when quasimetric projection is used for both x' and T² corresponding to the bound in Eq. (13). This strategy is implemented in our Eik- Eik-QRL and QRL:hi variants (see Appendix(D) D). and it explains the strong empirical results observed in the idealized poi ntmaze setup (Table in Appendix H). In higher-dimensional spaces such
20

---

Published as a conference paper at ICLR 2026
Tw 50, gw0.03
T- 50, gm
T- T=1000, g=0.01
T=1000, dm0.03
0.4
0.3-
Polcy Error
0.1
20
6
40
250
500
750
1000
250 subgoal steps K
500
750
1000
subgoal steps
subgoal steps k
subgoal steps k
F-actor
H-actor
Quasimetric-F-attor
Quasimetric-H-actor
Eik-HIQRL
Figure 6: Comparison of policy errors in the didactic environment in Fig. F-actor corresponds to the bound in (14), H-actor to (15). Quasimetric- F-ctoo to 12). Quasimetric-
In the figure legend,
H-actor to (13) and Eik-HiQRL to (16)
High-level
Low-level
IQE
MLP
2h
SLiki sel)
Vt
61 s2])
x
ae
9
9
d' (8.9) (8,9)
(2
(*
nt (allss
Figure 7: Diagram of the Eik-HiQRL architecture. The high-level planning module consists of quasimetric model d', parameterized as an IQE (Wang & Isola), which trained to optimize the physics-informed objective in 17). and high-level policy n" trained via the advantage-weighen objective in [1)). The low-level module comprises value function V and goal representation network Φ, both instantiated as MLPs and trained to minimize the Implicit V-Learning loss in [8 The low-level policy T' is trained to maximize the advantage-weightd regression objective in 20
as entmaze, however, accurate quasimetric projections become substantially harder to obtain. This difficulty manifests as increased noise a in Eq. (IT), which in turn explains the challenges faced by all purely quasimetric-bant algorithms, whether PDE-constrained or not, when applied to high-dimensional state spaces.
In this analysis, the correlation coefficient is modeling parameter, and, similarly to in Eqs. (TO) (IT). it typically estimated from data in practical environments. Proposition [.]]] we introduce for correctness: after quasimetric projection, the noise terms on neighboring segments become correlated, and this correlation must reflected the variance of the value difference. In the illustrative example of Fig. 6 we simply fix 0.01. Choosing small correlation coef- ficient makes the bound more conservative, corresponding to harder regime where shared noise between neighboring estimates is limited, and random fluctuations have stronger impact on value comparisons.
Finally, Fig. 6 compares the bounds derived above. Provided appropriately chosen subgoal step k. Eik- Hik-HHQRL... achieves performance comparable to fully quasimetric agents. Moreover, hierarchi- cal policies consistently outperform their flat counterparts, underscoring the benefits of temporal abstraction long-horizon tasks.
C EIK-HIQRL
In the following, provide comprehensive overview of the Eik-HiQRL algorithm, including full implementation details, complete pseudo-code, and the main hyperparameters used in our experiments. Additional resources and extended documentation available in our GitHub repository
21

---

Published as a conference paper at ICLR 2026
C.I OVERVIEW
Refer to Fig- 7 for graphical illustration of the Eik-HiQRL architecture. The high-level planning module consists of a quasimetric model parameterized by IQE (Wang & Isola), together with a high-level policy that outputs a subgoal embedding 2 <([[<tkk> 5t)). The low-level control module includes a value function V parameterized by an MLP, as well as goal representation network 6, also instantiated as an MLP with length normalization applied to its final layer. The low-level policy takes the current state and the subgoal embedding as input and outputs an action to interact with the environment.
Recall that the value decomposition described above enables the construction of high-level abstract space S. Accordingly, in Fig. 7 we denote the inputs to the high-level module by 8. / S.
The high-level value function de (denoted as dh in Fig. trained to solve the following Fik-QRI. optimization problem:
max Es.g (s
s.t. EssPane Pad
(1, -) S e
6
(17)
Global Relationships (GRs)
Eikonal-Local Relationships (Eik-LRss)
where Pate and Paral denote the marginal state and goal distributions induced by the offline dataset, and ( a monotonically increasing transformation that dampens large distances (o(s.9) g)-
The low-level value function Vo, (referred to as V in Fig. 7 and the goal representation network so (ø in Fig. 7 are trained by minimizing the following Implicit V-L.earning objective:
Cv 0g) ( p~P, (R(.9))) + ,Va, (s', Vo, (s, =
(18)
where 2g and z/3 are the goal embeddings produced by da.- Oy are the parameters of the target value network Mnih 2013). and (x) = |e- - 1(x < 0) is the expectile loss with E [0.5, 1].
Policy extraction is performed via advantage-weighted regression for both the high-level policy T and the low-level policy The using the following objectives:
(0r) Fe p- exp (3. A" (se, S1+k, 9)) log nb (se+k (0.99).
(19)
(O:) (1 exp (B. A' ar St+k)) log Té (at St, stmx)
(20)
where 8 is an inverse temperature hyperparameter, and the advantages are approximated as Á g) g) do(st. - do and Å'(st, az, 8144) = Vev (8t+1, BC+R) Ve, (8t, 81+4).
C.2 IMPLEMENTATION DETAILS
In this section, we describe how the optimization problem in T7 is solved in practice. We propose two main variants: the first, which refer to as Eik-QRL-X and simpler unconstrained version, Eik-QRL.
The Eik- FILL-QR-A variant adopts constrained optimization approach based on the Lagrangian relax- ation of the original problem. Specifically, it formulates a minimax objective that introduces a dual variable to explicitly enforce the Eikonal constraint:
min max EgwPan, grrP ((do(s, g)) + A 0 X
- -x22
(21)
This formulation ensures principled trade-off between maximizing global value alignment (through the first term) and enforcing local Lipschitz continuity via the constraint. However, it requires careful tuning of dual ascent steps and often leads to unstable optimization dynamics due to the interplay between the primal and dual variables.
By contrast, the Eik- Eik-QRL variant simplifies the training process by removing the explicit constraint and instead incorporating the local regularization term directly into the objective as a soft penalty:
min EgnPan gruP
((do(s, ([V..0(s.))) 1)2
(22)
22

---

Published as a conference paper at ICLR 2026
Table 3: Hyperparameter values for Eik-HiQRL.
Hyperparameter Name
Value
Actor (mr), me,)) hidden dims
(512, 512, 512)
Values (dg. Vo,) hidden dims
(512, 512, 512)
Goal representation network (do,)) hidden dims
(512, 512, 512)
Quasimetric type
IQE
Quasimetric latent dim (m)
512
Subgoal representation dim (dim(2))
10
Discount factor (7)
0.99
Batch size (B)
1024
Optimizer
Adam
Learning rates Odd av, Oh. Oi
3 10-4
Target update rate (T)
0.005
Expectile factor (t)
0.7
Inverse temperature parameter (8)
3.0
This relaxed formulation eliminates the need for dual variable updates and simplifies implementation, while still encouraging the learned quasimetric satisfy the Eikonal constraint approximately. Although it lacks explicit constraint satisfaction guarantees, we find it to be more robust in practice and easier to tune.
We evaluate both implementations in the ablation study reported in Section H.2 Our final algorithm adopts the simpler formulation in (22). as it demonstrated superior training stability and consistently
higher performance with both flat and hierarchical actors. Refer to Table 6 for summary of these results.
C.3 PSEUDO-CODE AND HYPERPARAMETERS
In this section, we provide implementation-Ieved details necessary to reproduce our results. We first present, in Algorithm | the full pseudo-code for Eik- Eik-HiQRL, covering both the high-level planning and low-level control modules. This includes training routines for the quasimetric model, value functions, and policies. We then summarize in Table [3] the key hyperparameters used in our experiments. These details reflect the final configuration that achieved the best performance across our benchmarks. For further information and code, please refer to our GitHub repository
Algorithm Contranl Hierarchical Quasimetric RL (Eik-HiQRL)
Input: Offline dataset D, high-level quasimetric model de, low-level value function Vey, target value function Vor. goal representation network Φo,, high-level policy Te. low-level policy T expectile factor discount factor y, inverse temperature parameter learning rates aas av, Oh or, target update rate while not converged do (st, 8(+1,9) ~
Update de optimizing the problem in (2 with learning rate ad
Update Ve, and 6o, minimizing Lv (0v vv.C 0%) in (18) with learning rate ay
Oy - + 70v
end while
while not converged do
((+k,g)) ~ D
Update Tे. maximizing (0,) in 19 with learning rate on
end while
while not converged do
(St, at, 3t-1, D
Update Tép maximizing n (0,) in 20 with learning rate ay end while
23

---

Published as a conference paper at ICLR 2026
5, g
Length Norm
h
8t
π
h
9
IQE
h (hr k/ 9)
d
9
3t
π
at
heth
d(s,g) = -
x' (as [82, hltk)
Figure Diagram for the the hierarchical Eik-QRL, Eik-QRL-A and QRL:hi used for the experiments in Table and Table 6
D EIK-QRL, EIK-ERIIIR QRL-À AND QRL:HI
In this section, we describe the hierarchical variants Eik- Gik-QRL, Eik- Eik-QRL-À and QRL:hi, which main- tain single value function but incorporate hierarchical actors that operate subgoals. Introducing hierarchical policies requires additional care, primarily due to the need for subgoal encoding to enable effective high-level planning. An illustrative diagram of this hierarchical architecture is provided in Fig. 8
Each of these variants employs single quasimetric value function trained to optimize different ob-
jective: QRL:hi minimizes (T). Eik-QRL. minimizes (22). and Eik-QRL-A minimizes the constrained form 21.. Subgoal representations are implemented using length normalization, and both levels
of the bierarchical actor are trained via Seaattgg--wightee regression using the objectives in (19 and (20). The corresponding advantages are computed as
A (st, 8144. 9) (s( g) do(s g), (' at S(+k) (st, 81+k) do (00+ St4R).
Note that when Eik-QRL is not paired with a hierarchical actor, it follows the standard QRL pipeline described in the preliminaries (Fig. in Section B optimizing the unconstrained objective (22). while Eik-QRL-X optimizes the constrained formulation in (II). We denote these flat variants as Eik- Fik-QRL:flai and Eik-QRL-XAfflat respectively and experimental results for these versions are reported
in Table
E OFFLINE GCRL ENVIRONMENTS USED IN SECTION 6
(a)
(b) antmaze
(c) humanoid- (d)
(c) cube
scene
pointtaze
antscccer
Figure 9: Environments from OGbench Park et al. 2024a used in our experiments in Section 6
In the following, we provide brief description of the agents used in experiments (Fig. Our goal is to highlight to what extent their dynamics comply with the regularity assumptions Section From experimental standpoint, two assumptions particularly relevant: Assumption 4.2 which requires Lipschitz continuity of the dynamics, and Assumption 4.4 which prescribes Lipschitz continuity of the optimal goal-conditioned value function.
24

---

Published as a conference paper at ICLR 2026
pointmaze This task involves position control of a 2-D point mass. The state dimension for this agent is 2 (the (z. y) coordinates), and the action space is also 2-dimensional. The dynamics in this environment are isotropic and therefore Lipschitz continuous, Assumption 4.2 holds. Moreover, in this setting Assumption 4.2 is a sufficient condition for Assumption 4.4, and thus both assumptions satisfied.
antmaze This task involves controlling quadrupedal ant agent with 8 degrees of freedom. The state dimension for this agent is 29, and the action dimension is 8. The dynamics involve intermittent contact between the feet and the ground for locomotion, so Assumption 4.2 is not strictly satisfied. However, due to the absence of explicit categorical variables in the state space and external manipulable objects, expect the optimal goal-conditioned value function remain reasonably regular in practice, making Assumption 4.4 a plausible approximation.
humanoidmaze This task involves controlling a humanoid agent with 21 degrees of freedom. The state dimension for this agent is 69, and the action dimension 21. As in antmaze, the dynamics involve intermittent contact with the ground, so Assumption 4.2 does not strictly hold. Nonetheless, the absence of categorical state variables and external manipulable objects suggests that the optimal goal-conditioned value function is still relatively smooth, and Assumption 4.4 is again a reasonable approximation.
antsoccer This task involves controlling dimension is 42 (29 for the ant and 13 for the ball: 7 position//orientation coordinates and 6 linear and angular velocity coordinates), and the action dimension 8. The dynamics involve both ground contacts and interactions with separate ball object, which introduce discontinuities and non-smooth transitions. Consequently, Assumption 4.2 does not hold this setting. Moreover, due these contact-rich interactions with an external object, we do not expect the optimal goal-conditioned value function to be globally Lipschitz, and Assumption 4.4 is generally violated.
agent dribble soccer ball. The overall state
cube This task involves pick-and-place manipulation of cube blocks, where robot arm must arrange cubes into designated configurations. The state dimension for this agent is 28, and the action dimension is The dynamics involve contact-rich interactions between the end-effector and the cubes, which break Lipschitz continuity, Assumption [.2 does not hold. In addition, the presence of multiple interacting objects and discrete mode switches (e.g., grasp no-grasp) leads to value function discontinuities, Assumption 4.4 is also violated in practice.
This task involves manipulating diverse everyday objects, such as cube block, window, drawer, and two button locks, where pressing a button toggles the lock status of the corresponding object (the drawer or window). The state dimension for this agent is 40, and the action dimension is As cube, the dynamics are contact-rich, and button presses induce discrete mode switches, so Assumption 4.2 does not hold. The combination of external objects and categorical state variables (e.g.. lock status) introduces sharp discontinuities in the optimal coal-conditioned value function, leading to violations of Assumption 4.4
scene
F OFFLINE GCRL BASELINES USED IN SECTION 6
Table 4: Structural comparison of the considered goal-conditioned RL methods. A checkmark (V) indicates that the corresponding component is present in the method, while a cross (X) indicates its absence.
Eik-HiQRL.
Bik-QRL
QRL
HIQL
Eik-HIQL
CRL
Quasimetric Value Fikonal Regularization Hierarchical Actor Hierarchical Value
X
X
X
X Xx
X
/
X
/
X
X
X
X
25

---

Published as a conference paper at ICLR 2026
Table 4
summarizes the main structural differences among Eik-
QRL
8
QRL
(Wang al. 2023).
HIQL (Park et al. 2024b). Eik-HIQL (Gammmarioo et al. 2023), CRL Eysenbach et al. 2022) and
Eik- Eik-HiQRL (Fig.
learns a quasimetric goal-conditione value function without any PDE- based constraints. Eik- QRL builds QRL by adding Eikonal constraints, which provides similar theoretical guarantees under our regularity assumptions and a smoother value landscape, while also introducing a hierarchical actor. HIQL, in contrast, uses hierarchical actor but learns standard TD-based value (rather than a quasimetric) and does not impose any PDE constraints. Eik-HIQL augments HIQL with Eikonal regularization, yet maintains -oasquaaimttric value and does not decompose the value hierarchically. Our Eik-HiQRL combines the strengths of these approaches: it uses quasimetric value constrained by the Eikonal PDE at the high level, together with both hierarchical actor and hierarchical value structure. This design preserves the theoretical benefits of Fik-QRL where its assumptions are most plausible, while mitigating its practical limitations through hierarchy and enabling improved scaling to long-horizon tasks. Finally, CRL (Eysenbach et al. (022) is substantially different method compared to the previous algorithms which learns a critic based on contrastive learning principles. We include it here for the sake of completeness.
G LEARNING CURVES FOR EXPERIMENTS IN SECTION
In this section, present the full set of learning curves corresponding the experiments in the main text. Fig. II 12
14 show success rate and collision rate as functions of training steps for the
results summarized Table [
covering respectively Fik-HiQRI, Eik-QRI,, HJB-QRI,, and QRI.
Fig. 15 and report the learning curves for the antsoccer and man hanippllee pul at on experiments in Table
All experiments were conducted on a single NVIDIA RTX 3090 GPU (24 GB VRAM), using a local server equipped with 12th Gen Intel i7-12700F CPU, 32 GB RAM. No cloud services compute clusters were used. Each individual experimental run required approximately hours of compute time on the GPU,
A natural concern is the additional computa- tional overhead introduced by the Eikonal con- straint, which requires automatic differentiation with respect the inputs the value network. In our implementation. this amounts to one extra backward pass per minibatch, which can be fully
600
500
Wallclock time
400
300
parallelized on modern hardware. To quantify the overhead, we compare wall-clock training time for Eik-QRL and QRL over 100k gradient steps with batch size 1024. The average time for 100 training steps (mean + standard deviation across training) is 6.37 + 5.69 ms for Eik-QRL and 6.20 + 5.55 for QRL, i.e., difference of less than 3%, well within the observed vari- ability. In practice, we also observe comparable memory usage for the two methods. These re- sults indicate that the Eikonal regularization has negligible impact on training efficiency in our setting (Cf. Fig. 10).
200
100
0.0
0.2
0.4
0.6
0.8
1.0
Steps (x10))
Eik-QRL
QRL
Figure 10: Wall-clock training time as a function of training steps for Eik-QRL and QRL over 100k gradient steps with batch size 1024. The curves closely overlap, indicating that the Eikonal term in Eik-QRL introduces negligible computational overhead compared to QRL.
26

---

Published as a conference paper at ICLR 2026
Pointmaze Medium Navigate
Pointmaze-Large Navigate
Peintmaaz--Gaan Giant-Naviaatt
Pointmaze Paintmaaze Teleport- Navigate
Success Rates
1.0
1.0
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Steps (x10')
Stops (x10")
Steps (x10*)
Stops (x10')
Pointmaze Madimm--Stiiieeee Sticch
Pointmaze-Large--e
Paintmaze Saapaazz---iann--iee Glant-Stitch
Centtmaze-Tllportt.eo
Success Rate
1.0
1.0
Collision
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Steps (x10*)
Steps
Steps (x10+)
Steps (x10²)
Antmaze-La Lag---aa-g---giaaaa
Antmaze Navit-aa
Antmaze Teleport Talpooot-aooootaaaaatt
Success Rate
Collision
0.5
0.5
0.4
0.5
0.5
0.5
0.2
0.0
0.0
0.0
0.5
0.5
0.0
Steps (x10*)
Steps (x10*)
Steps (x10+)
Steps (x10P)
Antmaze Medium b
Antmaaee Large Stitch
Antmaze Giant Stitch
Antr aze Teleport Stitch
Success Rate
0.50
0.500.50
0.50
Collision Pate
0.250.25
0.25
0.0
0.0
0.0
0.00
0.000.00
0.00
0.5
0.5
1.0
Steps (x10*)
Steps (x10*)
Steps (x10*)
Steps (x109)
Figure 11: Learning curves for Eik-HiQRL. in the experiments in Table 1 Plots show the average success rate and collision rate per evaluation across seeds as function of training steps.
2jjjtmazz-Meelun--NWWvaa
Painttaaze-arne--elllltee
Printmaze-Gaann--aalllae
Pointmaaze-lleeeot-aavvaaa
Success Rate
1.0
Collision Rate
0.5
0..
0.5
0.5
0.0
0.0
0.0
0.0
Steps (x109)
Steps (x109)
Steps (x10')
Steps (x10f)
Pointmaze- Mediuu-th
Psintmaze-Large-S000
PtintraazeGianntstttchh
Pointmare-Teleport-te
Success Rate
1.0
1.0
Cellision Rabe
0.5
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
10
0.0
0.5
1.0
(x10%)
(x10*)
(x10')
(x10*)
Antmaze MeviiatNaviggae
Antmaaze-Larg-Navteee
Antmaze Nlvii
Antmaze Senmmaze-Teleorttooo Navieportt-Naigtttt
Success Rate
Colission Rote
0.4
0.4
0.5
0.5
0.2
0.2
0.0
0.0
0.0
0.0
0.0
Steps (x10*)
Steps (x10²)
Steps (x10*)
Steps (x10*)
Antmaze Medium-Stthh
Antmaaz-LLaggeSütte
Antmaze Giant-Stäch
Antmaze Teleport-Stikch
Succes Rate
0.4
0.4
0.4
0.4
0.4
Callision Rate
0.2
0.2
0.2
0.2
0.2
0.0
0.0
0.5
1.0
0.0
0.0
0.5
1.0
0.0
0.0
0.5 Steps (x10')
1.0
0.0
0.0
0.0
0.5
1.0
0.0
Steps (x10%)
Steps (x10²)
Steps (x (x10')
Figure 12: Learning curves for Eik-QRL the experiments in Table
Plots show the average
success rate and collision rate per evaluation across seeds function of training steps.
27

---

Published as a conference paper at ICLR 2026
Success Rate
1.0
PaintmazeeLarge-eevvaaa
1.0
1.0
PyttomaeeGiant-avviaas
Pointmaae-iileeort-Wvvage
Collision Rate
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Steps (x109)
Steps (x10*)
Steps (x10*)
Steps (x10f)
Pointmaze Modium-Stter
Pointmaze-1 aaga--rontttn
Pointmaze Glant-Stitch
Pointmaze Teleport-
Success Rate
1.0
Callision Rate
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
Steps (x10f)
Steps (x10f)
Steps (x10*)
Steps (x10f)
Antmaze Antmaze.Mediuma Medium Navinate
Antmaze-Large Navigate
Antmaze Bataazz--iant-avvaaeae Navigatt
Antmaze Talmaaz--eleprtt.ooo Teleport Nallaart Navigate
Success Rate
0.4
0.4
0.4
0.4
Callision Rate
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Steps (x109)
Steps (x10*)
Steps (x10')
Steps (x10')
Antmaze- Manmaze-Medlum-555 Stitch
Antmaze-large--50n
AntmaazeGGaan Matrmaze-Glan--S0ki
Antmaze Cotmaae-Tlloport-toe Telsport-sttoe
Success Rate
0.4
0.4
0.4
0.4
Callision Rate
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Steps (x109)
Steps (x109)
Steps (x10')
Steps (x10f)
Figure 13: Learning curves for HJB-QRL in the experiments in Table I Plots show the average success rate and collision rate per evaluation across seeds as a function of training steps.
Sointmaaze-Mediune Navigate
Paintmaaz--argg--aavvvaa
Navii-Navigtee
Seintmaaz-Telepetttteee
Success Rate
Collision
0.5
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
Steps (x10*)
Steps (x10F)
Steps (x10²)
Pointmaze Medium Stitch
Pointmaze Large Stitch
Paintmaze Giant Stint-Stithh
Pointmaze weleeaeee--eeerttSSc Seeereeeeteeeoor--Ssshhhh Seirch
Success Rate
Collis
0.5
0.5
0.5
0.5
0.0
0.0
Steps (x10*)
Steps (x10*)
Steps (x10*)
Steps (x10*)
Antmaze- Medium Novigate
Antmaze Large Navigate
Antmaze Giant Navigate
Antmaze Teleport Navigate
Success Rate
Collision Robe
0.5
0.50
0.500.50
0.50
0.5
0.5
0.25
0.250.25
0.25
0.0
0.0
0.0
0.0
0.000.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Steps (x10*)
Steps (x10F)
Steps (x10*)
Steps (x10*)
Antmaze Medium-Stithh
Antmaze- Large- Stith
Antmaze Giant- Stitch
Antma wwwpe-eeeeepeeeeeeee
Success Rate
0.5
0.5
0.50
0.50
0.50
0.4
Collision
0.25
0.25
0.25
0.2
0.0
0.0
0.00
0.00
0.00
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Steps (x10
Steps
Steps (x102)
Steps (x10?)
Figure 14: Learning curves for QRL in the experiments Table Plots show the average success rate and collision rate per evaluation across seeds as function of training steps.
28

---

Published as a conference paper at ICLR 2026
Artsoccer-Arenn-Naawaaeee
Ansoocce-Aree--Sttcch
Antsocce--eedum-Sit00
0.35
Normalized Score
0.30
0.08
0.5
0.4
0.08
0.10
0.20
0.3
0.06
0.04
0.10
0.05
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.25
0.25
0.50
0.75
1.00
Steps (x10f)
Steps (x10f)
Steps (x109)
Steps (x109)
Elk-HIQRL
Eik-HIQL
HIQL
QRL
CRL
Figure 15: Learning curves for the soccer experiments in Table 2 Plots show the average success percentage per evaluation across seeds function of training steps.
Cube-Sinee Siig-e-nggeePe Play
Scene-Play
Normalized Score
0.3
0.2
0.4
0.2
0.0
0.0
0.50
0.25
0.50
1.00
Steps (x10f)
Steps (x10%)
Eik-HIQRL
Elk-HIQL
HIQL
QRL
CRL
Figure 16: Learning curves for the mani manipulate ion on experiments in Table 2 Plots show the average success percentage per evaluation across seeds function of training steps.
H ADDITIONAL EXPERIMENTS
In the following. present additional experi-
ments and ablations that complement the results the main paper. Section H.T reports a comparison analogous Table but with QRL replaced by QRL:hi as introduced in Section
Section presents an ablation over actors and optimization losses to validate our design choices.
Section provides full comparison between Eik-HiQRL and Offline GCRL baselines, extending the experiments from the main text.
Section H.4 reports an ablation between Eik-HiQRL and HiQRL, contrasting the Eikonal-constraieed
optimization pipeline in Eq. ( with the standard QRL formulation in Eq. within our hierarchical framework.
Section H.5 investigates the trajectory-free setting and presents an empirical study demonstrating the potential of Eik-QRL in this regime.
Finally, Section H.6 concludes this Section by showing that Eik-QRL can also be applied in the online RL setting.
H.1 COMPARISON WITH QRL::I
In the following, we report a comparison analogous to Table but with QRL replaced by QRL:hi as presented in Appendix D We include these experiments for completeness and to better disentangle the contributions of hierarchy from those of the PDE-based constraints. The results, summarized in Table[5] emphasize the strong regularizing effect of PDE-based constraints, which yield tangible improvements in the wointmaze-giane environment under both the navigate and st itth data regimes. In the antma setting, the considerations discussed in the main text remain valid
29

---

Published as a conference paper at ICLR 2026
Table Summary of the comparison among the QRL formulations. All agents are trained for 10s steps using 10 seeds and evaluated every 10* steps. We report the mean and standard deviation across seeds for the last evaluation. For each seed, evaluations are conducted over different random goals, with the learned policy tested for 50 episodes per goal. For success rate (R), results within 95% of the best value are written in bold. For collision avoidance (x) the lowest average is colored.
Eavironment
Dataset
Dim
R (f)
Fa-HIQRI
R. (t)
FILQQRI.
HII-QRIL
R (*)
QRI.hi
undiun
$
7
88
3
+:
navigate
glant
Large
73
14
12::2
18:2
91
83 =
17 1:6
69 1 13
telepozt
46417
3G 6
40 11
49 14
40 8
48=11
28 8
18#11
nediium
96 +
10:22
86 +
10
86
10 2
80 9
2
Large
76 +
5 #
78 + 7
73
9
91 +
11 1
giant
70117
te legortt
501:10
37
29
31
28
452
modium
3
nav navigat
giant
lazqe
59 +
86 + 2
25 +-
28 1
8 2
74 4
35 d:
28
34 8
365 22
23
8G
32 +
telegort
+
40
45 1:3
28 28:10
2
40
2
nediuum
94 +
19
70 +
+
37 07:55
38 : 2
83
31 +
large
4
13:44
giant
47::12
2
0
telegort
51 +
28 19:22
28 #
35 3
13 1
37 1 4
38 3
34 1 3
Pojjtmaaz--eiliu-eeeiiite
Peintmazzelarge-Nealaate
PointaazeGGian-Nalllte
Pointmaze-Tdeeporr-Wvvvaaa
Success Rate
2.0
1.0
1.0
Collision Rate
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.0
0.5
0.5
0.5 Steps (x10')
0.0
0.5
1.0
Steps (x109)
Steps (x109)
Steps (x10*)
Poin wwwtmmaaee-eiddmeen Stttch
Pointmaze-Large--50
Pointmazz-Giantssttch
Pointmaze Tinntmace-Taleport-Sttthh
Success Rate
1.0
1.0
Collision Rate
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Steps (x10*)
Steps (x10*)
Steps (x10²)
Steps (x10²)
Anttmaze-Large-aaviiete
Antmaze- Glana
Antmazz-Teeeeoot--ee -eviaat
Success Rate
0.4
0.4
Collsiinn
0.4
0.5
0.2
0.2
0.2
0.0
0.0
0.5
0.5
0.0
0.5
1.
0.0
0.5
1.0
Steps (x109)
Steps (x10a)
Steps (x10')
Steps (x10
Antmaaee Medium- Stitch
Antmaze- Sttth
Antmaae Giant-Sticch
Antmaze Teleport-Sttch
Success Rate
0.4
0.4
0.4
Callision Rate
0.4
0.4
0.5
0.5
0.2
0.2
0.2
0.0
0.0
0.5
0.0
0.0
0.0
0.5
0.0
0.0
0.5 Steps (x107)
0.0
0.0
0.5
0.0
Steps (x10%)
Steps (x109)
Steps (x10F)
Figure 17: Learning curves for QRL:hi in the experiments in Table rate and collision rate per evaluation across seeds as a function of training steps.
Plots show the average success
here as well. The corresponding learning curves are shown in Fig. Eik-QRL HJB-QRL, and QRL:hi, respectively.
22
13 and 17 for Eik- Eik-HIQRL,
H.2 ADDITIONAL ABLATION
This section reports an additional ablation comparing quasimetric-baae algorithms across actor types (flat vs. hierarchical) and optimization variants (constrained in Eq. 21 vs. unconstrained in Eq. 22). We include this experiment to further justify our design choices and to provide a more compact comparison across algorithms. The results, summarized in Table confirm the benefits of combining
30

---

Published as a conference paper at ICLR 2026
Table Additional ablation. All agents are trained for 10 steps using 10 seeds and evaluated every 10 steps. We report the mean and standard deviation across seeds for the best evaluation across training. For each seed, evaluations are conducted over 5 different random goals, with the learned policy tested for 50 episodes per goal. Results within 95% of the best value are written in bold.
ElkHHOQL
QRL
ORLibi
Ataa
EIk-QRL
Elk-QRL-X
mocium
nevigate
g:an7
large
89 +
90 :
72 7
732 10
95
63 + 21
68 :
98
90 =
57 :
teloport
53 I
34 31277
25 1:
49
27
pol
99 +
80 10
96 +
98 +
98 +
94 +
8G +
11
75
g:an:
teleport
62
42 t:
39 +
29 +
32
35 #
mes1um
61 6
69
nevigate
lazge
61 = 13
84 +
0
62 1
84 7
1
0=0
9
$
0
1344
9
68 1 10
6
teloport
antmaze
mediun
1+1
stitch
granz
lasge
51 10
0200
0
0=0
0::
0.
0:::0
24
Ieleport
58 =
31 #
43 43
14
17
3
38 #
Pointmaze Mecire--Medmmmmimd Naviggt
Pointmaze-Larg Lingtaaze-
Pointmaze Penttmaze-Giant-ine Giant-Navigate
Pointmaze SenimmazzeTeellpeee Telepprt-Navigiae
Narmalized Score
1.0
1.-
0.6
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.0
0.0
0.0
0.50
180
Steps (X10P)
Steps (x10º)
Steps (x10')
Steps (x10*)
Polntmaze-Medium-Stiich
Pointmaze-Large-Sae
Mlnttazze-Gian--Stooe
Pointmaae-TeleporttStthh
Normalized Score
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.0
0.0
0.0
0.55
0.75
0.25
0.00
0.30
1.0
Steps (x10*)
Steps (x10*)
Steps (x1)
Steps (x10*)
Eik HIQRL
QRL:hi
Eik QRL-A:flat
EIk-QRL-A
QRL
Eik-ORL:fiac
Eik-QRL
Figure 18: Learning curves for the poi nointmaze experiments in Table 6 Plots show the average success percentage per evaluation across seeds as function of training steps.
hicrarchical actors with PDE-based constraints. The corresponding learning curves are shown in Fig.
18 and Fig.
19
H.3 FULL COMPARISON EIK-HIQRL VS BASELINES
This section provides a thorough evaluation of Eik Hik-HiQRL against the Offline GCRL baselines intro- duced in the main text, complementing and summarizing the earlier comparisons. The experiments are reported Table and the corresponding learning curves are shown in Fig. 20 16 for the pointmaze antmaze, antmaze explore, humano cmaaz antsoccer, and manipulation environments, respectively. Consistent with the conclusions of the main text, Eik- HiQRL comfortably outperforms matches all baselines across tasks. As also noted, performance gains are smaller in the manipulate puwptiiiii.... in setting, which represents an interesting testbed for future research.
21 22
[5 and
31

---

Published as a conference paper at ICLR 2026
Nntmaze-Medduu-Naaiieee
Antmazz-Large-Nawigerr
Antmaze-GientNaviigtte
Tetpare-Teleport- Navia
Normalized Score
6.8
0.6
0.4
0.4
0.2
0.2
0.2
6.0
0.0
0.0
0.00
0,50
0.75
0.25
1.00
0.00
1.00
Steps (x10%)
Steps (x10")
Steps (x10%)
Steps (x109)
Antmaze-Medlum-Sticc
Antmaze-Large-Sttca
Antmaze-Giant-stun
Antmaze-Teleport-Stte
0.6
Normalized Score
0.2
0.2
6.0
0.0
0.0
0.88
0.50
0.25
1.50
0.00
0.25
0.7
1.80
Steps (x10*)
Steps (x10*)
Steps (x10*)
Steps (x10'²
Elk-HIQRL
QRL:hl
Eik ARf-lat-
Eik OR-QRR-A
QRL
fik-QRLiflat
Eik-QRL
Figure 19: Learning curves for the antmaze experiments in Table 6 Plots show the average success percentage per evaluation across seeds as a function of training steps.
Table 7: Summary of our experimental results. All agents are trained for 10t steps using 10 seeds and evaluated every 10 steps. We report the mean and standard deviation across seeds for the best evaluation. For each seed, evaluations are conducted over 5 different random goals, as designed in Park et al. (2024a). with the learned policy tested for 50 episodes per goal. Results within 95% of the best value are written in bold.
Enviromment
Dataset
Dimension
EIk-HOOL
Eik-HIQL
IIQL
QRL
CRL
mediuum
09 +
98 +
92 +
83 +
+ 19 +
d:
90 90
navigate
gi ant
+
79
72
Leluport
68 +
47 10
7
34
60
poinzmaze
medium
+: 2
8
80
3
stitch
large
57 + 14
95 +
73 6
22
19 7
4
85 11
56 +
0 0
4 = 6
teleport
38
d:
+:
06:1111
87
04422
navigate
giant
lazye
66 +:
67
69 1
80
1446
telaport
53 +
52 + 4
43 43
39
66 #
antrate
sLitch
mediuum
88 4:
94
84 +
94 +
95 + 14
6
24
68 1:6
54 14
1314
giant
teleport
60 +
47
1
29
34
medium
explrr
large
16
18 +
6
0
0 0=0
Lelsport
7
15
46 1od50
2
22
medium
91
86 +
90 90:53
22
61 +
navigate
74 4:
61
4:44
7
22..9
66
hume dineee
medium lange
d:
d.
22
+
stitch
28
2
gi ani
69 +
19
3 ==
0
0
navigate
arana med:um
13 4:
61 +:
19
3.2.2
60 6
13 13:3
2:2
10
4 4
stitch
medicm
a190a
82 4
7
2
12:
17 17.88
6
0.:
241
0 0=0 0
1
manipclation
cubo-singlo-play
12:1: 1:3
25
31 31 4
11:13
32 1:
SUOCOM play
66 4 14
62 +
62 12.d 8
8
35 +
32

---

Published as a conference paper at ICLR 2026
solitmazze-Mekuu--laaaage
Pointmaze-Lange-ävviigee
Pointmaze-Giant-aaiiaee Pointmaze- Nooooe-aaalle
Normalized Score
1.0
10
0.6
0.6
0.4
0.00
8.30
1.00
0.25
1.00
1.0
Steps (x10*)
Steps (x10*)
Steps (x100')
Steps (x10')
Pointmaze-Meeium-Sttee
Pointmaze-Large-Stiice
Dointmaze-Giante-ee
Peintmaze-lelepor-Sttth
Normalized lized Score.
1.0
1.0
0.8
0.8
0.4
0.4
0.2
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Steps (x109)
Steps (x10F)
Steps (x (x10
Steps (x109)
EIL-HIQLL
Eik-HIQL
HIQL
QRL
CRL
Figure 20: Learning curves for the poi ntmaze ze experiments in Table 7 Plots show the average
success percentage per evaluation across seeds function of training steps.
Antmaze Antmaze-Medium Mediimmdaaiiiiitteeee
Antmaze-Lappe LariaazeeLarg---eee Marge-avvigate
Barmazz--Gaat-Naaane
1.0
0.6
Normalized Score
0.8
0.6
0.2
0.0
Steps (x10%)
Steps (x10%)
Steps (x10%)
Steps (x10%)
Antmaze-Medium-Stice
Antmaze-Large-Stttee
Antmaze-GGant-Stttch
Antmaze-feleport-Stee.
Normalized Score
0.8
0.6
0.6
0.6-
6.6
0.2
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Steps
(x10f)
Steps (x10%)
Steps (x10%)
Steps (x109)
EIK-HQQRL
ElK-HIOL
HIQL
QRL
CRL
Figure 21: Learning curves for the maze experiments in Table Plots show the average success percentage per evaluation across seeds as a function of training steps.
Antmaae--eeduum-Exae
Antmaaz-Large-Explen
Antim Seatmaae-Teeepor--EEpoooo
0.7
5.5
0.5
0.20
2.4
Cormalized
2.4
0.3
0.2
5.2
0.2
6.3
2.00
5.5
2.4
Steps (x10%)
Steps (x10*)
Steps (x10")
Eik-HIORL
Eik-HIQL
HIOL
QRL
CRL
Figure 22: Learning curves for the antma ze eepeeeepllle ore experiments in Table 7 Plots show the average success percentage per evaluation across seeds function of training steps.
33

---

Published as a conference paper at ICLR 2026
Table 8: Second ablation. Training and evaluation procedures follow Table Results within 95% of the best value are written in bold.
Enstronment
Dataset
Dim
Elkk
HIQRL
nadiun
95
nevigate
Large
glant
66
taloport
53 $3-44
42 +
antmaze
mediium um
94 14:22 2
94 +
stitch
Lazge
giant
88 + 8
$1::0
$2
88 +
Celeport
60
exploce
lazqu
16:: 11
13 +
teloport
7: 4
17 + 11
medlun
91 +
92 +
nevigate
large
74
74 #
giant
88
humanaidhczo
nedium
85 35:53
87 +
larga giant
69
40
noviçate
acena nedi um
13
antsoSOGE
stitch
zzena medium
32 4 7
30
8 =
Antmaze Medium Navigate
Large Navigate
Gaant-Ginnnt-aaaae Naviggee
Antmaze-Teleper Naviggtte
1.0
Normalized Score
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.30
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Steps (x10%)
Steps (x10%)
(x10%)
Steps (x10f)
Antmaze--ed Seaize-Meeilu--Sttee
Antmaze-Large-Sune
Antmaze Glant-Stitch
AntmazeTeleeprt Stitch
Normalized Score
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.00
0.5
0.75
1.00
0.25
0.50
1.00
0.00
0.75
1.00
0.25
0.50
0.75
1.00
Steps (x10%)
Steps (x10%)
Steps (x10%)
Steps (x109)
Elk-HIORL
HIQRL
Figure 23: Learning curves for the antmaze experiments in Table g Plots show the average success percentage per evaluation across seeds as a function training steps.
H.4 COMPARISON EIK-HIQRL VS HIQRL
This section complements the ablation in Section and examines the impact of Eikonal constraints within our hierarchical framework, thereby closing the loop of our analysis. To this end, we compare Eik-HiQRL and HiQRI.: the former employs the Eikonal-constrainee optimization pipeline in Eq. for high-level value function lcarning, while the latter relies on the standard QRL formulation Eq.
I Results are summarized in Table 8 with learning curves reported in Fig. 23 24 25 and 26 The conclusions align with those of the previous sections, once again showing substantial gains in large environments and in the stit witc setting, underscoring the benefits of our PDE-based constraints.
34

---

Published as a conference paper at ICLR 2026
Antraaze Beaiazze-Meiuu---xppeee
Antmaaz--arge--Eppae
Antmaze-Teelporr Explore
0.5-
2.4
5.3
0.10
0.35
0.0
0.2
2.4
0.6
0.5
0.5
0.2
644
05
5.8
300
5.
0.2
2.4
25
0.8
25
Steps (x10%)
Steps (x109)
(x109
Eik-HIQRL
HIQRL
Figure 24: Learning curves for the antmaze-exet ore experiments in Table 8 Plots show the
average success percentage per evaluation across seeds as a function training steps.
Hummanidmaaee-eediin-avvvis
Humanoidmaze-Large-aaaaa
Normalized Score
0.8
0.0
0.6
0.6
0.4
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Steps (x10%)
Steps (x10%)
Steps (x109)
Hurmanoidmaze-Mediuuu Siithh
Baamaaniddaaze-Lrgg-SSuuse
Humanaidmaze-Go Giant-Stttch
Normalized Score
0.6-
0.4
0.4
0.2
0.0
0.2
0.4
0.6 (x10%)
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Steps
Steps (x10%)
Steps (x10*)
Eik-HiQRL
HIQRL
Figure 25: Learning curves for the humano imaze experiments in Table 8 Plots show the average success percentage per evaluation across seeds function of training steps.
Antsocee-Aren---avvlaaee
Mtsoccer-Meduu--aavieee..
AntsoccerArea-SStich
Antsocce:Medlum-Siich
0.35
Normallzed Score
0.6
0.08
0.12
0.25
0.100
0.20
0.3
0.04
0.10
0.023
0.0
0.000
0.00
0.00
Steps (x10f)
Stops (x10%)
Steps (x10f)
Steps (x10%)
Eik-HiQRL
HiQRL
Figure 26: Learning curves for the soccer experiments in Table 8 Plots show the average success percentage per evaluation across seeds function of training steps.
35

---

Published as a conference paper at ICLR 2026
H.S
TRAJECTORY- EXPERIMENTS
FetchReach State (Trajectory Free)
0.4
Normalized Score
0.3
0.2
0.1
-0.1
0.0
0.5
Steps (x10*)
Eik-QRL (Value Only)
(a) Learning curve Eik-QRL in trajectory- (b) FetchReach environment in Plappert et al. (2018).
free setting.
Figure 27: Summary of the results in the fully trajectory-free setting. We train goal-conditioned value function Vo(s, g) -do(s, using Eik-QRL as defined in Eq. (8) Training relies solely on random samples of end-effector (z, y, -)oordiiisss from the robot workspace. The end-effector is then controlled via inverse kinematics, with its position updated by following the negative value gradient, i.e., 8t+1 8 Vo (8t: g) for a suitable stepsize >> 0. The corresponding learning
curve is shown in Fig. 27a
We report mean and standard deviation over random seeds.
To investigate the trajectory-free regime, we evaluate Eik-QRL on robotic goal-reaching task based on the Fet chReach environment (Plappert et al. 2018). The setup, illustrated in Fig. 27b uses 7-DoF Fetch mobile manipulator equipped with two-finger parallel gripper; the underlying environment controls the robot Cartesian space by specifying small displacements of the end- effector, while inverse kinematics are handled internally by the simulator. To make this setting fully trajectory-free, we restrict the state space the end-effector Cartesian coordinates (ar, y, =), and randomly sample 100k end-effector positions from the robot workspace, aggregating them into replay buffer without storing any transition trajectories.
Value learning is performed with Eik-QRL as in Eq. (8), where minibatches of random coordinates are independently sampled as state s and goal g. Every 10k training steps, we evaluate the resulting controller over 50 rollouts. We do not train separate policy network in this experiment; instead, control is obtained by directly updating the end-effector position using the gradient of the learned goal-conditioned value function,
St+1 St - Vo.
for suitable stepsize > (we use in this experiment). The robot arm then commanded to the updated end-efffector position via inverse kinematics.
Results for this experiment are reported in Fig. and show that, despite the very weak form of super- vision, the method reaches average success rate of about 40% after 100k training steps. We view this as an initial settings with smooth, approximately isotropic dynamics. Note that do not perform any architecture or hyperparameter tuning in these experiments, but simply reuse the same configuration as in the previous tasks (see 3).le Future work will focus leveraging the learned coal-conditioned value function as initialization for subsequent value-learning or control schemes, effectively bootstrapping learning in more complex data-scarce regimes.
demonstration of the potential of Eik- LRL in fully trajectory-free
H.6 ONLINE RL EXPERIMENTS
In this section, we evaluate QRL, Eik- -RL, and Eik-HiQRL in an online GCRL setting on two continuous-contie benchmarks: Pointmaze and FetchReach. For both domains, iden- tical network architectures, hyperparameters, and training budgets across methods, and we report mean and standard deviation over random seeds.
36

---

Published as a conference paper at ICLR 2026
H.6.1
POINTMAZE
Open
Open
Diverse
Medium
150
150
100
Avg Reward
50
50
50
50
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Stops (x10%)
Steps (x10")
Steps (x10')
Steps (x10*)
Stops (x110))
Medium Diverse
60
Medium Diverse GR
Large
Large Diverse
Large Diverse GR
150
200
Aug Reward
1.00
40
100
100
20
100
0-
0.5
0.5 Steps (x109)
Steps (x10%)
Steps (x10*)
Steps (x10*)
Steps (x10%)
Eik-QRL
QRL
Ek-HiQRL (Ours)
Figure 28: Online GCRL results Pointmaze. We evaluate each algorithm every 10k steps 50 evaluation episodes and report mean and standard deviation over 3 random seeds.
This Pointmaze environment consists of 2-DoF point mass (a "ball") that Cartesian directions and and must reach target goal within closed maze (Fu et al. 2020). The environment can instantiated with different maze layouts and increasing levels of difficulty, adapted from Fu et al. 2020). We consider multiple maze sizes and configurations, including single-goal and multi-goal variants, where the latter are referred to as diverse mazes. A further group of variations instantiates another class of diverse mazes in which both goal locations and initial agent states randomly sampled at each reset. These environments share the same base identifiers as their default counterparts, with the suffix Diverse_cr GR (GR stands for Goal and Reset) appended.
in the
Because the agent is controlled through forces, the underlying dynamics are closer to a double- integrator model in free space, but collisions with the effective dynamics highly anisotropic and introduce strong non-smoothness in the value function. As result, Assumption 4.2 and the induced Lipschitz regularity in Assumption 4.4 are strictly satisfied. This setting therefore provides stress test for Eik-QRL outside its ideal isotropic regime.
walls and contact effects make the
We compare QRL, Eik-QRL, and Eik-HiQRL on these online tasks in Fig. 28 All methods learn meaningful goal-reaching policies across the different maze configurations. The results highlight that Fik-QRI, remains competitive even when its modeling assumptions are vio- lated, while Eik-HiQRL is designed to better
FetchReach
Normalized Score
1.0
0.8
0.6
0.4
cope with such settings and indeed outperforms both QRL and Eik-QRL.
0.00
0.01
0.02
0.03
0.04
0.05
H.6.2 FETCHREACH
Steps (x10²)
Eik-QRL
QRL
We further evaluate QRL and Eik-QRL in an online GCRL setting on the Feach en- vironment introduced in |Plappert et al. (201)). The task is to move the end effector 7-DoF Fetch manipulator, equipped with two-finger parallel gripper, to a randomly sampled target position within the robot workspace. The robot controlled in Cartesian space: each action specifies small displacement (Ax, Ay, Az)
Figure 29: Learning eurves for the online RL periments on Pet chReach. We evaluate each al- gorithm every 5k steps over 50 evaluation episodes and report mean and standard deviation over 5 ran- dom seeds.
37

---

Published as a conference paper at ICLR 2026
the end effector together with scalar command that opens closes the gripper. while inverse kinematics are handled internally by the simulator. The resulting continuous dynamics are smooth and approximately isotropic in the control space, both Assumption 4.2 (Lipschitz dynamies) and, consequently, Assumption 4.4 (Lipschitz goal-conditioned value function) are expected to hold reasonably well in this setting.
In Fig. 29 We report online training curves comparing QRL and Eik- Hil-QRRL under identical architectures, hyperparameters, and training budgets P Both methods rapidly achieve high success rates, and their convergence characteristics are very similar: the learning curves closely overlap, with only minor differences in transient behavior. These experiments show that, under modeling conditions that match its ussumptions, Eik-QRL can be successfully extended to an online robotic control task with performance comparable QRI..
follow the original QRI. implementation provided at https://gitae httpg/ggthhbco.o.oo
quasimen muasimetric-io aaaaaiigggaaaaaaaaaiaaaaaaaaaaia
38


Reviews:
[{"review_id": "ZIwxpxloMS", "reviewer": "Reviewer_dSDx", "summary": "The paper considers the problem of goal conditioned quasimetric reinforcement learning (GCQRL). The novel contribution of the paper is to introduce a (Eikonal) PDE constrained formulation. This Eikonal constrained formulation can be viewed as a refinement of the preexisting HJB PDE constrained GCQRL problem, derived by assuming the continuous-time system dynamics to be isotropic. Although this dynamics assumption is simplistic the formulation trades off this with a very desirable trajectory-free nature, requiring only samples from state, goal distributions. The paper establishes that under some Lipschitz dynamics, cost and value function assumptions, for compact state-action spaces, it is possible for a universal quasimetric approximator to approximate the optimal value function under this Eikonal-constrained formulation. Further, to alleviate issues rising from the complex dynamics that most RL problems have and break the assumptions of the Eikonal formulation, the authors propose a hierarchical version of the Eikonal constrained formulation, where a Eikonal GCQRL operates on an abstract (higher level) state space, and a more traditional TD style RL algorithm operates in the actual (lower level) state space. The authors complement their theoretical contributions with empirical evidence demonstrating that their Eikonal formulation consistently outperforms other QRL methods on a variety of robotics tasks.", "questions": "Besides point 2. in the weakness section, please address:\n1. The running cost $c(s,a)$ is introduced in the usual state-action sense in Section 3, but is used in the state-goal sense $c(s,g)$ in Section 4. Could you define what $c(s,g)$ means more clearly like you did for the rewards in goal conditioned RL?\n2. Could you give some intuition on whether (local) Lipschitzness of the optimal value function in Assumption 4.4 is a reasonable assumption in practice or not?\n3. (Minor formatting point) In table 1, in the row ant maze navigate large, the algorithm giving the lowest $\\kappa$ is not colored.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The presented formulation is trajectory-free and only requires one to sample (state, goals) rather than complete trajectory rollouts which I really appreciate.\n2. The core part of the presented Eikonal approach boils down to a constrained optimization problem which can be readily solved through a wide suite of existing physics informed ML methods.\n3. The experiments in the paper specifically highlight that the formulation performs well in settings where no theoretical statements can be made currently (i.e., complex, non-Lipschitz dynamics).", "weaknesses": "1. The main weakness is the relatively simplistic dynamics assumption of the formulation (i.e., Lipschitzness, and that the continuous time counterpart is unit speed isotropic). However, I feel that this weakness is adequately acknowledged and addressed by the authors.\n2. One of the main assumptions for Lemma 4.7 and Theorem 4.8, stated in line 301 says $c(s,g)=1$ on $\\mathcal{K}\\setminus g$. Does this mean $g$ can be the only suitable goal in $\\mathcal{K}$? If so, I feel that is a limitation as for continuous control tasks, several points in a continuous neighborhood can be also be goal states reaching which leads to success for an RL policy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8IMFF3kkma", "reviewer": "Reviewer_9KNx", "summary": "- This paper introduces Eikonal-Constrained Quasimetric RL (Eik-QRL), which uses a continuous Eikonal constraint instead of the standard local Quasimetric RL local constraint.\n- Eik-QRL is derived and and formulated. This involves: (1) moving from QRL to a HJB-QRL continuous time variant, including bye assuming smooth dynamics and a smooth value function; (2) Moving from HJB-QRL to Eik-QRL, which is easier to optimize in practice, by assuming unit-speed, isotropic dynamics.\n- Theoretical guarantees are then proved: Under unit-speed isotropic dynamics in a convex space, the optimal value function has unit gradient norm everywhere, and a universal approximator that satisfies the unit gradient constraint will recover the optimal value function.\n- The paper notes the benefits of Eik-QRL include trajectory-free estimation and PDE-based regularization, at the cost of strong assumptions (including the 1-Lipschitz property) which may not hold in general MDPs\n- Acknowleging the potential limitations in general MDPs, they propose a hierarchical approach that can use a high-level policy to exploit the benefits of Eik-QRL in a “a lightweight, dynamics-agnostic state-space”, while the low-level policy uses standard TD-learning that does not have restrictive assumptions.\n- Their methods are empirically tested in the offline GCRL setting. In the simple pointmaze environment, they show improvements in collisions, in more complex antmaze, they show Eik-QRL is on par with QRL despite Eikonal constraints, and show EikHiQRL achieves the strongest results. In humanoidmaze with more complex dynamics still, they also outperform baselines\n- In environments where 3rd party objects and categorical variables mean regularity assumptions do not hold, they observe comparable performance to baselines. They identify designing PDE-constrained algorithms for these non-regular, non-isotropic robotic settings as follow up direction.\n- Overall, the paper contributes novel theory which enables a practical method that performs very well in certain settings", "questions": "- Could you explain why exactly PDE acts as an implicit regularizer?\n- Could you propose an experiment to demonstrate why “trajectory-free” could be useful in practice?\n- How correlated is the success rate metric with the collision avoidance metric? Is the collision avoidance metric providing much extra information?\n- Perhaps you could better highlight your strong performance improvements over QRL in the challenging and non-regular environments?\n- Could you provide more environment details in the appendix? Ideally including details regarding the extent to which the Eik assumptions hold in each environment.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The theory and formulation of Eik-QRL is novel and is an interesting perspective on QRL.\n- The clarity and writing in the paper is generally very good. The theory is well written, the writing is well-balanced, limitations are well-acknowledged.\n- While Eik-QRL does make strong assumptions, it seems plausible these assumptions could hold to some extent in certain scenarios.\n- Eik-QRL often shows strong performance on benchmarks versus baselines, including state-of-the art humanoid maze performance. This seems to establish Eik-HiQRL as a SOTA method in the navigation benchmarks.\n- Empirically, performance is reasonable in settings where the strong assumptions do not hold. This seems to mitigate some concerns regarding the limitations of the restrictive assumptions of Eik-QRL.", "weaknesses": "**Assumptions of Eik-QRL may limit real-world use-cases.**\n\nThe authors have well-acknowledged the limitations of some assumptions made by Eik-QRL. My general concern is that these assumptions could limit the significance and wider impact of this method. While the paper has evaluated their method on a wide range of environments, including some where their assumptions break down, many of these environments seem to be navigation-based - it seems plausible the method is overfitting to these specific toy navigation benchmarks which may not be representative of many realistic settings.\n\n**Experiments and results could be performed and presented in a more systematic manner to help us better understand the strengths and weaknesses of Eik-QRL**\n\nI would like to see improved benchmarking and systematic experiments that better decompose and provide evidence for exactly where and how  Eik-QRL adds value and where and how it struggles. This could add more clarity on the practical value of the method. I will give some examples below.\n\nThe authors state three main benefits that Eik-QRL provides. They state trajectory-free as a strong advantage in the introduction, but I cannot see any clear experiments validating this. While they do have evidence of the “improved state coverage” benefit, this evidence is hard to parse in the results.\n\nThe baseline methods they compare against all seem to use slightly different components, making it hard to fairly compare and hard to determine what are the core valuable contributions of Eik-HiQRL (is the performance gain simply due to the hierarchy, or is it due to the Eik method? Is Eik particularly suited to a high-level abstract-space, versus standard TD methods or standard QRL?). Ideally, we could easily compare Eik-QRL and Eik-HiQRL to an equivalent QRL and HiQRL to better distinguish where the value add is. If Eik-HiQRL is proposed as their SOTA method, it could be useful to better compare the Eik high-level policy to other standard high-level policies, while keeping the low-level policy method constant.\n\nIn general, presenting aggregated may make it easier to parse overall performance differences between methods.\n\nPerhaps a more thorough and systematic approach could be taken to “red-teaming” the Eik method to finding and understand cases where the assumptions are limiting in practice. This could involve: (i) adding another non-regular environment such as CALVIN, or (ii) better determining the extent to which Eik performance gains are mainly due to the ‘navigation’ component in the navigation environments.\n\n**Other**\n\nAfter strong theory to introduce Eik-QRL, there is minimal theory or discussion to properly justify the choice to use Eik-HiQRL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vxMVxDZmXj", "reviewer": "Reviewer_oo25", "summary": "This paper introduces Eikonal-Constrained Quasimetric RL (Eik-QRL), a novel approach to Goal-Conditioned Reinforcement Learning (GCRL). It builds upon Quasimetric RL (QRL), which frames GCRL as learning a quasimetric (a shortest-path distance function) $d(s, g)$. The key insight of this paper is to reformulate QRL's discrete, trajectory-based local consistency constraint (i.e., $d(s, s') \\le \\text{cost}$) into a continuous-time Partial Differential Equation (PDE) constraint.\n\nBy assuming simplified unit-speed, isotropic dynamics ($f(s,a) = a$), this PDE constraint reduces to the Eikonal equation: $||\\nabla_s d_\\theta(s, g)|| = 1$. This new formulation, Eik-QRL, is trajectory-free, meaning it only requires sampling states ($s$) and goals ($g$) rather than full state-action-next-state transitions. This makes it an effective regularizer and highly suitable for offline RL.\n\nTo address the limitations of the isotropic dynamics assumption in complex environments, the authors propose Eik-Hierarchical QRL (Eik-HiQRL). This method uses the efficient, trajectory-free Eik-QRL as a high-level planner to propose subgoals in a simple abstract space (e.g., $(x, y)$ coordinates) where the Eikonal assumption holds. A separate, standard low-level TD-learning policy is then trained to reach these subgoals in the full, complex state space.", "questions": "1.  The performance gains are most pronounced in navigation tasks. In manipulation and antsoccer, where the abstract space $\\overline{\\mathcal{S}}$ is a \"latent space learned end-to-end\" (for manipulation), the method is only on par with baselines. Could you elaborate on how this latent space is learned? How do you ensure this learned space adheres to the 1-Lipschitz / unit-speed properties required for the Eik-QRL high-level planner to be effective?\n2.  Following on from Q1: How sensitive is Eik-HiQRL to the quality of this abstract space? If the abstract space does not accurately reflect the geometry of the underlying state space (e.g., it thinks two states are close when they are separated by a wall), does the Eikonal constraint still provide a useful learning signal?\n3.  The jump from HJB-QRL (Eq. 7) to Eik-QRL (Eq. 8) is a key step, motivated by numerical stability and removing the reliance on transition tuples. Did the authors experiment with a middle ground? For example, using the HJB-QRL formulation but with a *learned* local dynamics model to estimate $f(s, a)$ (or $s' - s$), rather than simplifying the dynamics to $f(s,a)=a$?\n4.  In Appendix C.2, the paper discusses two optimization methods: the constrained Lagrangian (Eq. 21) and the soft penalty (Eq. 22), ultimately choosing the soft penalty for stability. Does using a soft penalty risk the Eikonal constraint not being fully enforced (i.e., $||\\nabla_s d_\\theta|| \\neq 1$)? If so, how much does this \"approximate\" enforcement weaken the theoretical guarantees of Theorem 4.8, which rely on the constraint being met?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.  The core idea of connecting quasimetric learning's local consistency to the Eikonal PDE is a creative, insightful, and theoretically sound contribution.\n2.  A major practical strength of Eik-QRL is that it is trajectory-free, only requiring i.i.d. state and goal samples. This makes it more data-efficient and better suited for offline learning from unstructured datasets than the original QRL, which requires transition tuples.\n3.  The paper clearly identifies the main weakness of its own Eik-QRL formulation (the strong isotropic dynamics assumption) and proposes a very logical and effective solution: use Eik-QRL as a high-level planner in a simple abstract space where the assumption *does* hold, and use a standard model-free controller for the complex low-level dynamics.", "weaknesses": "1.  The paper's strongest results are in navigation tasks (pointmaze, antmaze, humanoidmaze). In the antsoccer and manipulation tasks (Table 2), the performance gains vanish, and Eik-HiQRL is only \"comparable\" to baselines. The paper acknowledges this but it suggests the method's applicability is currently best suited for tasks where a simple Cartesian abstract space is available.\n2.  The success of Eik-HiQRL appears to be highly dependent on the choice of the high-level abstract space $\\overline{\\mathcal{S}}$, which must be one where the Eikonal (unit-speed) assumption holds. For navigation, this is intuitively the agent's $(x, y)$ coordinates. For manipulation, the paper states this is a \"latent space learned end-to-end\", but details on *how* this space is learned and *how* it is constrained to satisfy the Eikonal properties are not in the main paper. This choice seems critical and non-trivial for applying the method to new domains.\n3.  The paper derives HJB-QRL (Eq. 7) as a general PDE constraint, but quickly simplifies it to Eik-QRL by assuming $f(s,a)=a$. The justification is that HJB-QRL is \"ill-conditioned\" and still relies on transitions. This simplification is what limits the method to isotropic dynamics. An alternative, underexplored direction would be to keep the HJB-QRL formulation and use a learned local dynamics model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tnOHjEdhlH", "reviewer": "Reviewer_nmvA", "summary": "This paper proposes Eikonal-Constrained Quasimetric Reinforcement Learning (Eik-QRL), a novel goal-conditioned reinforcement learning framework that integrates constraints derived from partial differential equations (PDEs), along with its hierarchical extension, Eik-HiQRL. The method reformulates the discrete trajectory constraints in Quasimetric Reinforcement Learning (QRL) into continuous-time constraints governed by the Eikonal PDE, thereby providing a more theoretically grounded and smooth representation of distance-based objectives. The effectiveness of the proposed approach is empirically validated on the ogebnch benchmark.", "questions": "1.Regarding the fundamental issue of the isotropy assumption: Can the authors quantify how much the optimal value function deviates from the 1-Lipschitz property under realistic robotic dynamics? Can they provide a theoretical analysis or bound—even a relaxed one—for cases with non-isotropic dynamics? Since HJB-QRL is theoretically more general, why does it perform poorly in practice, and is there room for improvement?\n\n2.Proposition B.1 — Hierarchical Analysis:The analysis is based on a 1D toy environment (Fig. 5). Can this be extended to higher-dimensional settings?How is the correlation coefficient ρ estimated or controlled in practical environments?\n\n3.Why does the unconstrained version (Eq. 22) outperform the constrained version (Eq. 21)? Table 5 shows that Eik-QRL outperforms Eik-QRL-λ, suggesting that a soft penalty may be more effective than a hard constraint. Does this observation contradict the theoretical motivation of the method?\n\n4.Figure 6 shows Quasimetric-H-actor has the tightest bound. Why should we use Eik-HiQRL instead?\n\n5.The authors consider a continuous formulation of QRL. To my knowledge, a major limitation of QRL lies in its deterministic dynamics assumption. Could the authors clarify whether Eik-QRL is applicable under stochastic dynamics? If not, how could QRL be extended to handle stochastic environments?\n\n6.Statistical significance: Many results exhibit large standard deviations (e.g., Table 1, pointmaze-giant-stitch: 62 ± 22). Were any statistical tests (such as a t-test) conducted to verify the significance of the reported improvements?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well-written, with a clear motivation and a high level of completeness in both theoretical and experimental aspects.\n\n2.The paper presents clear theoretical contributions with a rigorous derivation chain from QRL to HJB and finally to the Eikonal equation. Lemma 4.7 establishes the 1-Lipschitz property of the optimal value function, while Theorem 4.8 provides a high-probability value recovery guarantee, ensuring the theoretical soundness of the framework.\n\n3.In terms of algorithmic design, the trajectory-free nature of the method represents a key advantage, particularly evident on stitching datasets, where traditional approaches rely heavily on trajectory reconstruction. The PDE constraint serves as an implicit regularizer, enhancing out-of-distribution estimation. Moreover, the hierarchical design is well-structured: the high-level abstraction operates in a simplified latent space to avoid complex dynamics, while the low-level temporal difference (TD) module compensates for the limitations of Eik-QRL.\n\n4.The experiments are highly comprehensive, covering six categories of environments—PointMaze, AntMaze, HumanoidMaze, AntSoccer, and Manipulation tasks. The paper also introduces an innovative evaluation metric, the collision rate, which addresses a notable gap in the existing literature. Furthermore, extensive ablation studies are conducted, including five additional experiments in the appendix, providing thorough empirical validation of the proposed approach.", "weaknesses": "1.The assumptions are somewhat overly idealized—for instance, the requirement of unit-speed isotropic dynamics (i.e., f(s,a)=a) does not hold in real-world robotic systems, where manipulators typically exhibit complex, nonlinear dynamics. This limits the direct applicability of the theoretical model to practical robotic control scenarios.\n\n2.On the manipulation tasks in the ogebnch benchmark, the proposed method does not outperform previous state-of-the-art algorithms, indicating that its advantages may be less pronounced in environments requiring fine-grained control and complex dynamics.\n\n3.Although Eik-QRL represents a practical compromise that replaces full quasimetric projection with PDE-based constraints for handling high-dimensional and complex dynamical systems, the authors do not provide results on high-dimensional visual environments. As a result, the generality and robustness of their claims remain uncertain without further empirical validation in such settings.\n\n4.The authors are encouraged to clearly specify the differences and relationships among Eik-QRL, QRL, HIQL, and the closely related Eik-HIQL, and to explicitly discuss the advantages and limitations of Eik-QRL relative to these methods. It is recommended to include a comparative table in the related work section to enhance clarity.\n\n5.To the best of my knowledge, QRL has demonstrated experimental effectiveness in online settings; however, it remains unclear how Eik-QRL performs in online training scenarios.\n\n6.Computational efficiency is not reported—for example, there is no comparison of training time or memory consumption. The PDE constraint requires automatic differentiation to compute gradients ‖∇_s d_θ(s,g)‖, which may be more expensive than discrete constraints. Appendix E only states “4 hours on RTX 3090” without providing comparisons to baseline methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)