Conservative Offline Goal-Conditioned Implicit V-Learning
Authors: Kaiqiang Ke
Qian Lin
Zongkai Liu
Shenghong He
Chao Yu
Venue: ICML 2025 poster (2025)
Keywords: Offline Goal-Conditioned Reinforcement Learning; Goal-Stitching Tasks; Value Function Overestimation;
Primary area: reinforcement_learning->batchoffline
TL;DR: We propose Conservative Goal-Conditioned Implicit Value Learning (CGCIVL), which mitigates value overestimation in cross-trajectory sampling by penalizing unconnected state-goal pairs.

Abstract: Offline goal-conditioned reinforcement learning (GCRL) learns a goal-conditioned value function to train policies for diverse goals with pre-collected datasets. Hindsight experience replay addresses the issue of sparse rewards by treating intermediate states as goals but fails to complete goal-stitching tasks where achieving goals requires stitching different trajectories. While cross-trajectory sampling is a potential solution that associates states and goals belonging to different trajectories, we demonstrate that this direct method degrades performance in goal-conditioned tasks due to the overestimation of values on unconnected pairs. To this end, we propose Conservative Goal-Conditioned Implicit Value Learning (CGCIVL), a novel algorithm that introduces a penalty term to penalize value estimation for unconnected state-goal pairs and leverages the quasimetric framework to accurately estimate values for connected pairs. Evaluations on OGBench, a benchmark for offline GCRL, demonstrate that CGCIVL consistently surpasses state-of-the-art methods across diverse tasks.

Conservative Offline Goal-Conditioned Implicit V-Learning
Kaiqiang Ke' Qian Lin
Zongkai Liu"
Shenghong He Chao Yu"
Abstract
2020). Consequently, offline GCRL has attracted significant attention in recent years, enabling the learning of skillful agents from pre-collected datasets while circumventing the challenges of online data collection (Ma et al., 2022b; Park et al., 2023; Kim et al., 2024).
Offline condioondi reinforcement learning (GCRL) learns a goal-conditioned value func- tion to train policies for diverse goals with pre- collected datasets. Hindsight experience replay addresses the issue of sparse rewards by treating intermediate states as goals but fails to complete goal-stitching tasks where achieving goals re- quires stitching different trajectories. While cross- trajectory sampling a potential solution that as- sociates states and goals belonging different tra- jectories, we demonstrate that this direct method degrades performance in conliconditioned tasks due to the overestimation of values on uncon- nected pairs. To this end, we propose Conser- vative Goal-Conditioned Implicit Value Learning (CGCIVL), a novel algorithm that introduces a penalty term penalize value estimation for un- connected state-goal pairs and leverages the quasi- metric framework to accurately estimate values for connected pairs. Evaluations on OGBench, a benchmark for offline GCRL, demonstrate that CGCIVL consistently surpasses state-of-the-art methods across diverse tasks.
A common approach in offline GCRL to train a goal- conditioned value function V(s,g) g) from a static dataset via temporal difference (TD) learning (Sutton & Barto, 2018; Schaul al., 2015; Ghosh et al., 2023). One of the primary challenges in value learning is the low learning efficiency issue caused by sparse rewards. Existing methods utilize Hindsight Experience Replay (HER) (Andrychowicz et al., 2017), which samples states and goals within a single tra- jectory, tackle this problem. However, such approaches ignore the issue of the cross-trajectory state-goal pairs, i.e., pairs that can only be connected by stitching multiple trajectories. Such cross-trajectory pairs are essential in goal- stitching tasks, where the initial state and the goal could belong to different trajectories.
However, directly incorporating cross-trajectory state-goal pairs into data sampling can lead to value estimation er- rors, particularly when some of these pairs are uncon- nected-than is, when no valid sequence of transitions exists within the dataset linking the state to the goal. Therefore, their values cannot be estimated accurately using TD learn- ing and tend to be consistently overestimated during training. Moreover, these overestimated values propagate to neighbor- ing states through the bootstrapping process of TD learning, thus misleading the policy to select suboptimal actions.
1. Introduction
Coal-Conditioned Reinforcement Learning (GCRL) aims to train RL agents that are capable of mastering a diverse range of skills, each defined by a distinct goal (Schaul et al., 2015; Liu et al., 2022). In offline settings, these goals are typically identified as states already contained in the dataset, eliminating the need for manual goal design and facilitating the acquisition of general-purpose policies without active en- vironment interactions. Such setup is especially beneficial for real-world applications, where direct interactions can be costly, time-consuning, or even hazardous (Kumar et al.,
In this paper, we investigate the above issue in depth. First, by theoretically analyzing the discrepancy between the esti- mated and optimal values, we show that the overestima- tion of values on unconnected state-goal pairs leads to propagation errors in adjacent states. Then, we propose Conservative Goal-Conditionel Implicit Value Learning (CGCIVI.), novel algorithm designed learn conserva- tive estimate of the goal-conditioned value function. Specif- ically, a regularization term is incorporated to prevent over- estimated values of unconnected pairs and quasimetric model is employed to prevent potential under-estimation on cross-trajectory but connected state-goal pairs. Theoretical guarantees further show that the proposed method allevi- ates overestimation of values on unconnected pairs while
'School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China *Shanghai Innovation Institute, Shanghai, China PPenghhnng Laboratory, Shenzhen, China. Corre- spondence to: Chao Yu <yuchao3 mail.sysu.coo syu...du.cc........
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Conservative Offline Goal-Conditioned Implicit V-Learning
maintaining accurate value estimates for all connected pairs. Empirically, experiments OGBench (Park 2024), benchmark specifically designed for offline GCRL, demon- strate that our algorithm consistently matches or surpasses state-of-the-art methods across distinct environments with varying configurations.
Quasimetrics generalize metries by relaxing the symmetry requirement. The space of all quasimetrics over S is denoted as Q(S), and its negation is defined as - (S) == { {-dld e ((S)).
The quasimetric model do is a parameterized model that sat- isfies the properties of the quasimetric, where 0 is the param- eter to be optimized. Specifically, a quasimetrie model de typically consists of (1) deep encoder mapping two states $1, $2 E S to æ 22 in the latent space Rd and (2) a differen- tiable latent quasimetric head dhaten E Q(S) that computes the quasimetric distance such as ||| - x2ll2. for two input latents. In this work, we implement quasimetric models using Interval Quasimetric Embeddings (IQE) (Wang & Isola, 2022a), with details provided Appendix C.I.
2. Preliminaries
Problem setting The problem of offline GCRL can be formulated as Goal-Augmented Markov Decision Process (GA-MDP) MM- = A, T,, R, G. 7) and dataset D, where S denotes the state space, A denotes the action space, T S x A + S denotes the transition function, R:Sx 9 - [Fmin: 0] denotes the reward function, 9 denotes the goal space, and ? E (0, is the discount factor. The dataset D consists of trajectories T = (So, Go, $1, 01...... ST). We consider any state in the dataset as a potential goal and define the goal space G as {s| 8 E D).
The optimal goal-conditioned value function under GA- MDP can be represented by quasimetric models as it sat- isfies triangle inequality, which has been proved in prior works (Pitis et al., 2020; Wang & Isola, 2022a; Liu et al., 2023):
The objective of offline GCRL is to learn a policy т that maximizes the expected cumulative reward:
Vs1. $2, 83, V (21: + V (82) S (*1 (7)
Equation (7) shows that V E Q (8). Therefore, we can use da to fit goal-conditioned value functions.
(TT = Egrg.enrmoot) (15.7)
Σ r' R(st,g)]-
(1)
1=0
Off-policy value estimation in GCRL In GCRL, the noal-conditioned value function is learned through an itera- tive process based on the optimal Bellman operator T:
3. Motivation: Value Overestimation in Cross-trajectory Sampling
In this section, we consider value learning with static dataset D, where sampled state-goal pairs fall into the fol- lowing two categories:
VK+1 = min E
DND, [V - TV&(s, (2)
g
where T is defined as:
Definition 3.1. Given a state set S, an action set A, and dataset D {T, | E {I...., n.]), the state-goal pair (s.g) E S x S is in-trajectory if there exists trajectory {s0, do. 81....,, 8T - 8T) in the dataset and indices i,j E 1....... T) such that: 8;
(s)) R(s,g) + max
v( (3)
a.s'm((()..0))
The mixture distribution pmn (g] ()) is defined as:
8j g. <<) j. Otherwise, the state-goal pair (s, g) is called cross- trajectory.
(g 8) := asPonj (g s) + (1 - a) pan
where a E [0, 1] is the weight of the mixture and the com-
As shown in Equation (4), when a 1, only in-rrajectory state-goal pairs are sampled, akin to HER. As a decreases, arbitrary states are more likely to be sampled as targets, lead- ing to higher probability of selecting cross-trajectory pairs. To demonstrate the challenge of value function learning for cross cross-trajectory pairs, we conduct experiments to evaluate the performance under different values of the parameter a. As shown in Figure HER performs well on goal- navigating tasks (left), where pairs of initial states and goals are in trajeeecctory, but fails on goal-stitching tasks involving cross-trajectory pairs (right). Moreover, incorporating cross- trajectory pairs into value learning with a lower a degrades performance on both tasks. To further clarify this issue, we first provide a definition to categorize cross- trajs-tra pairs into the following two types:
ponents are defined as follows:
Ptraj (g s): Given a state 8 at time t in trajectory (sp, dus .1,..., ,,). ). a future state g 9. is sampled by selecting mn uniformly at random from the discrete interval {t, t
Prand A uniform probability distribution over all states in the dataset D.
Quasimetric model tion d: S x S R20 satisfying the following properties:
Given set S, a quasimetric is func-
ssi, 82, 83 S, 82) + d(s2, 8z) 2 ((s1)ss), (5)
Vs 8, d(s,s) 0.
2

---

Conservative Offline Goal-Conditioned Implicit V-Learning
Resinogo leneanagstt
Prstiaaee Sigt-7tt
Traj
S1
1
S2
a
0.2
0.2
Traj2
S2
1
91
3880 +.20
20pp K30
V Ver
0
Figure 1. Performance of the GCIVL, on two different tasks under varying a. In Maa-aMaae--arg--aviiiieee (left). pairs of initial states and goals are in-trajectory in the dataset. The algorithm achieves optimal performance at 1.0. In PointMaze-large-ttiee (right), pairs of initial states and goals are cross-traeectory in the dataset. The policy fails to complete this task across all settings.
Trajs
S1
1
S3
a
V = -
V39.81 = 0
Figure 2. Illustration of unconnected state-goal pair in offline GCRL. The reward function is defined as -1 for each transition and upon reaching the goal and the discount factor is set y = 1. Solid black lines represent transitions present in the datasct. The target goal 9i is represented by the green cirele. The state-goal pair (s3, 9n) is unconnected, while other state-goal pairs are connected. The values the sı and 82 can be overestimated via the optimal Bellman operator, represented by the yellow circle.
Definition 3.2. Given a state set S, an action set A, the static dataset can be represented as D = {(s;, as. Si+1) Si, 8i+1 E S,a; E A). A state-goal pair (x,g) E S x S is connected if there exists a finite sequence {sp, ao $1,...,8p-1 On } such that: &n = 8, 8n = g. (sk, ak: sk-1) E D for all E {0, I,.., n - 1}. Otherwise, the state-goal pair (s,g) g) is called unconnected.
The unconnected state-goal pairs, whose values cannot be estimated accurately, may induce the policy to incorrect actions. Furthermore, the estimation error be propa- gated to other states through TD learning. We consider the example illustrated in Figure 2 to provide an easy-to- understand explanation. Assume the discount factor y = and the reward function is - for every transition, except when the goal is reached, where the reward is O. The value function is updated via the optimal Bellman operator. Ac- cording to Definition 3.2, the state-gou gotte-goalllg pair (53, 91) is un- connected, with its value V(53. 91) assumed to be randomly initialized to O. As there is no valid path in the dataset DD that connects 83 to 91. the value V(*3,91 cannot be updated and could remain its initial overestimated value during TD learning. Then, V(53, 91) remains fixed at its initial overestimated value. This overestimation affects pol- icy extraction $1, where the policy π (811, 91) incorrectly favors action a= which leads to 83, over the correct action a, since V(32291) = 2 < V(s,,g1 = -1. Furthermore, based on the optimal Bellman iteration, the connected state- goal pair ($1, 91) is incorrectly estimated to -1 instead of the expected value 22 according to the valid trajectory (s1 - 52 - g1).
Learning).
(ls.9)552(ss.) +7 max Gk-1 (s', a.''N
Proof. See proof in Appendix B.2.
For any unconnected state-goal pair (s', its error (s', tends to be large due to the difficulty in esti- mating its true value from the dataset D. Consequently, for connected state-goal pair (s, g) adjacent to (s', g) (i.e., there exists an action a such that (s, a, s') D), Theorem 3.3 im- plies that the error (s (8) may also be large, as it inherits the propagated error from ( (s',g))
To mitigate this issue, it is desirable to prevent the selection of actions a that lead to unconnected state-goal pairs (s', g) during the iterative process. This requires introducing addi- tional mechanisms underestimate values for unconnected state-goal pairs, which we describe in the next section.
4. Conservative Goal-Conditioned Implicit V-Learning
|Ve(s. (s. denote the total error at iteration h of V-learning, and let d,(s,g) [V((s.g) - TV- (s, g)l denote the Bellman iteration er- ror (Kumar et al., 2019), where T is the optimal Bellman operator. Then, we derive the following bound on the total error at iteration l:
Let (x(
In this section, we develop novel algorithm, Conservative Implicit V-learning (GCIVL), to avoid the overestimation of value on unconnected state-goal pairs in offline settings. The core idea is to penalize the value of unconnected pairs while ensuring accurate value estimation for connected pairs through a quasimetric framework as dis- cussed in Section 4.1 and Section 4.2. We detail the practical
Theorem 3.3 (Error Propagation in Goal-Condittonel Value
3

---

Conservative Offline Goal-Conditioned Implicit V-Learning
implementation of the proposed algorithm and highlights key techniques that ensure its efficiency in Section 4.3.
is unbiased, the estimated value function is given by:
(=(s.))=VV(e. -n [1 -1- ( -
Puni
(s. g). (9)
4.1. Conservative Goal-Conditioned Off-Policy Evaluation
Pm
We then provide formal proof showing that, under suit- able conditions, the value function accurately estimates in- trajectory state-goal pairs while preserving conservative estimates on cross-trajectory pairs.
We aim to estimate the value function (s( for tar- get policy π given access to dataset D generated by behavior policy Tg. In order to prevent overestimation of the value function for unconnected state-goal pairs, we in- troduce a penalty term that minimizes the value of state- goal pairs sampled from a specific distribution p, g). Since the value function training process does not query the value function at unobserved states, we restrict Pμ to match the marginal state distribution in the dataset, such that (s) = d (glss, where p(g(s) denotes an arbitrary distribution which satisfies supp μ C supp Pm (s) is the state marginal distribution of the dataset and related to gg. The resulting update formula, as a function of a tradeoff factor 17, is as follows:
Proposition 4.2. For any e >> 0, there exists a penalty factor n such that the learned value fanction Equation (9) satisfies:
(=((tt :9) (=* ,g)
where (s+,9) denotes any in-trajectory state-godt pairs in the dataset D.
Proof. See proof in Appendix B.4.
Proposition 4.2 demonstrates that in-trajectory state-goal pairs can be accurately estimated. Subsequently, it is shown that cross-trajectory pairs are assigned substantially lower value estimates compared to in-trajectory pairs.
V+t1 + min (nE (V(sg))
V gralgl)))
8 VB (V(.())))V((so)))))
Proposition 4.3. For any >> O with static n, n >> 0, there exists an a such that the learned value function satisfies:
+B +B (nd (gls)
(= .9) (7* -
where V represents an empirical estimate of the true value function V. B denotes the empirical Bellman operator, which the sample-based counterpart of the theoretical Bellman operator. In Theorem 4.1, we demonstrate that the learned value function V=, defined as V= := lima VT. serves conservative lower bound for the V at all state- goal pairs.
where (st .9) denotes any in-trajectory state-goal pairs, and (s :9) denotes any cross-trajectory state-goal pairs.
Proof. See proof in Appendix B.5
4.2. Quasimetric-Based Value Estimation for Cross- Trajectory State-Goal Pairs
Theorem 4.1. For any p(g[s) with supp μ C supp pm and 5 (0, 1), with probability 2 1 8, the V-function n obtained by iterating Equation (8) satisfies:
As shown in Proposition 4.3, the values of trajestra state-goal pairs are underestimated, encompassing all un- connected pairs. However, cross- cross-rraecttrry pairs that are connected can also be underestimated. As a result, we lever- age a quasimetric model represent the value function so that the value of such pair can be bounded via triangle in- equality. As illustrated in Figure 2, $1 and 91 are connected but belong to different trajectories with intermediate nodes $2. Since V (s11,2, and V(s2, 91) can be accurately esti- mated given Proposition 4.2, the value of ($1.91) can be bounded by the triangle incquality.
(na00)-nn(10 Vs E D, P" S (* (1 (1-7P=)-1 /
po
Br., (8,9) + (I (1->p=)-1 (1-7) /1->)D/)
(8,9).
Proof. See proof Appendix B.3.
Proposition 4.4. Suppose the learned value function V satisfies the properties of quasimetric. For any > and the connected state state-goal pair (s.g) with intermediate states
Given Theorem 4. it follows that the value 7aa is under- estimated across all state-goal pairs with appropriate choice of parameter 1). However, it is important to avoid underestimation on values of in-trajectory state-goal pairs. In practice we replace μ[gls) with a uniform random dis- tribution (g), which equally samples across all goals in the dataset. Assuming that the empirical Bellman operator
$1, $2,..., Sn observed in the dataset, there exists penalty factor 1) such that:
(T((s 2 Σ (s1
0=0

---

Conservative Offline Goal-Conditioeed Implicit V-Learning
where 50 8, 8n+11 = 9-
4.3.1. QUASIMETRIC DISTILLATION
In goal-conditioned problems, the optimal value function actually obeys the triangle inequality. However, during TD learning, intermediate values may not naturally adhere to this property, so directly enforcing the quasimetric con- straint can negatively impact learning performance (Wang et al., 2020; 2021). To address this issue, we use dual- network approach: an unrestricted network 0, for TD learn- ing and a quasimetric constrriied network 0a E 2 (S)) The expectile regression loss for O,, is defined as:
Proof. The proof follows directly from the properties of the quasimetric and Proposition 4.2. The triangle inequality ensures that the accumulated values of sub-segments (s, 8.) and (st. 9) provide valid lower bound for the total value (s, g). Adjusting for small through appropriate tuning of 7 completes the proof.
Proposition 4.4 guarantees that the value of any connected state-goal pairs are bounded by the sum of sub-segment values no matter whether they belong to the same trajectory. Building on this property, we further leverage the quasimet- ric, in conjunction with the penalty-based -based conservative value estimation, to achieve a clear separation between connected and unconnected state-goal pairs.
La. = BBsdd g- 1 (r(8,g) + 7Va (v((s, of s'wI(5'(n.g)
(11)
V(s, g))l
Then, we distill values from 0, into 02 and add a penalty term to underestimate values of unconnected state-goal pairs. The loss for 0a is defined as:
Proposition 4.5. For any ŋ > O and c > O, there exists hyperparameter a such that the learned value function V satisfies:
Lox = Esnd' Bardd"gggppp. .gmpp m o d' (9.9)
(Vo. (8.9) - (s,
(Vo (12)
(= S ( c,
+ nB and g' p'mpenals)
where (st .9) represents any connected state-goal pairs, and (s ,9) are any unconnected pairs.
Finally, to align 0g with the conservative estimates of 02, we introduce supervised term in the loss function of 9.. This ensures that O, incorporates the conservative estimated of Vg, on unconnected pairs while retaining flexibility during TD updates. The final loss function for 0, is:
Proof. See proof in Appendix B.6.
LI
By combining Proposition 4.4 and Proposition 4.5, we show that incorporating quasimetric constraints into value func- tion estimation ensures reasonable lower bounds for con- nected state-goal pairs while maintaining underestimation for unconnected pairs.
Co. = Eg gope. ( + 7Val's's ,9) - V(s,
d'N((e)l.g)) + pB sndta, (V. - (s.
(13)
where p a tunable hyperparameter. The proposed dual- network framework effectively decouples value iteration (via 0.) from quasimetric constraints (via 92). Ablation
4.3. A Practical Algorithm
Our algorithm is implemented based on Goal-Conditioned Implicit V-Learning (Park et al., 2023), which is a variant of Implicit Q-Learning (IQL) (Kostrikov et al., 2021) designed for handling goal-conditioned tasks in the offline setting. As stated in Appendix A, at any time step k GCIVI can be interpreted as performing policy evaluation for specific policy T.. Given this observation, we incorporate penalty term into the original loss function, leading to the following updated formula:
studies in Section 5.3 demonstrate that this structure im- proves the efficiency of value learning.
4.3.2. HIERARCHICAL FRAMEWORK
As highlighted in prior works, existing offline reinforce- ment learning algorithms often struggle with long-horizon tasks where the goal is far away, primarily due to the signal- to-noise ratio challenge (Park al., 2023). An effective approach to addressing this issue is to leverage hierar- chical policy framework. Specifically, we extract both a high-level policy Tb, (Se+* st,9) and a low-level pol- icy oh. (a st, 8144), aiming to maximize V(8+4 g) and V(s+11, St+k), respectively. Here, 81+1 is treated as a way- point or sub-goal, where k represents the step size.
VK+1 min (ols), (r(s (V) s'>p(e'la.aa
V(ssg)) + n| Bnd (o) 11. (s.
g
(10)
where V E Q (S) ensures that the value function satisfies the property of quasimetric.
The high-level policy outputs sub-goals $t4k based on the current state 8, and the final goal g. The low-level policy
5

---

Conservative Offline Goal-Conditioned Implicit V-Learning
Algorithm 1 CGCIVL
average success rate computed over 50 trials per goal. In experiments, we report 95% confidence intervals shaded regions in figures or standard deviations in tables, unless oth- erwise specified. We provide more details of environments and datasets in Appendix D.
Input: offline dataset D
initialize unconstrained value function Vo, (s. g), quasi- metric value function model (8, high-level policy Tă, ($++*|st, 9). low-level policy The (a|st, S(+b))
while not converged do
Our algorithm is compared with the following approaches: 1)
0g + 0. Xu Vo. (0v) {# Equation (13)
Behavioral Cloning (GCBC) (Lynch et al., 2020; Ghosh et al., 2021), a straightforward goal- conditioned method that learns policies by mimicking the trajectories in the dataset; 2) V-Learning (GCIVL) (Park et al., 2023), an offline GCRL algorithm that uses expectile regression to approximate the optimal value functions; 3) Goal-Conditiones Implicit Q- Learning (GCIQL) (Kostrikov et al., 2021), which, like GCIVL, applies expectile regression but focuses on approx- imating Q-values instead; 4) Quasimetric Reinforcement Learning (QRL) (Wang et al., 2023), a non-traditional GCRL method that fits a quasimetric value function with a dual-objective framework; 5) Contrastive Reinforcement Learning (CRL) (Eysenbach et al., 2022). a one-step RL algorithm that learns value functions through contrastive learning and performs single-step policy improvement; and 6) Hierarchical Implicit Q-Learning (HIQL) (Park et al., 2023), hierarchical RL algorithm that derives a two-level policy based on GCIVL
0 + 0g- - Vo, (0a) (# Equation (12))
0, < On- Vo, La (0n) (# Equation (14))
0g+ + 0g- Ve Ce (Be) [# Equation (15)}
Implicit
end while
then outputs appropriate actions a conditioned on the current state and the sub-goal, The loss functions for the high-level and low-level policies are as follows:
Co. -- B( exp(3 - A (1t,,)4)))))
(14)
log mb ($1+* s),g),.
Co.- = Be - exp(B. 1.B.A. A' at, (s St+x))
(15)
log Té, (at s14k)],
where we approximate A (st, $14k: 9) as Vo, (8c4k.g) (st, and (s; az S(++) as Vo, (S141, St4k) Vo. (st, 8144). The pseudocode of CGCIVL is shown in Algorithm
5.2. Performance on Offfine Goal-Conditioned Tasks
5. Experiments
The results in Table demonstrate that CGCIVL consis- tently outperforms or matches all baseline algorithms across most tasks in every environment. In locomotion environ- ments, our algorithm achieves state-of-the-an art performance and particularly excels in long-horizon and goal-stitching tasks. For instance, on PointMaze searre-gian---tsteee CGCIVL achieves 81% success rate, surpassing the second-best method (50%) by substantial margin. Similarly, on both AntMaze-giant--sine and aarraanidaaa--giia-ttiiiiii CIVL exhibits clear advantage. In addition, in goal- navigating tasks, the proposed algorithm also demonstrates improvements over prior methods. This advantage arises from the fact that trajectories in the dataset are suboptimal, and CGCIVL effectively leverages the strengths of different trajectories to construct more efficient goal-reaching strate- gies. Moreover, our algorithm also achieves nearly optimal results in the manipulation environment, particularly in the Cube and Scene. This indicates that our proposed algo- rithm is capable of learning diverse trajectory- skills to accomplish more complex goals.
We conduct a series of experiments to evaluate the effec- tiveness of the proposed algorithm in addressing offline goal-conditioned tasks. In Section 5.1, we describe the ex- perimental setup, including the environments with different configurations and the dataset types used for training. In Sec- tion 5.2, we compare CGCIVI! with several strong baseline methods across diverse environments and datasets, demon- strating its superior performance in both goal-navigating and goal-stitching tasks. In Section 5.3, we analyze ablation studies to evaluate the sensitivity of the cross-trajectory goal- sampling ratio and the penalty coefficient. Furthermore, we discuss the contribution of the quasimetric distillation and the hierarchical policy structure to the overall performance.
CG-
5.1. Experimental Setup
We evaluate the proposed algorithm OGbench (Park et al., 2024), benchmark designed to evaluate algorithms in offline GCRL across different tasks and datasets. The ex- periments are conducted in three locomotion environments (PointMaze, AntMaze, HumanoidMaze) and three manip- ulation environments (Cube, Puzzle, Scene). In each task, the policy evaluated across five different goals, with an
The significant effectiveness of the proposed algorithm stems from its more accurate value function estimation. As shown in Figure 3, in GCIVL, the value function shows little variation for states far from the goal, remaining around 200. Specifically, with the goal located in the bettom-left,
Implementation details can be found in Appendix C.2
6

---

Conservative Offline Goal-Conditineed Implicit V-Learning
values for states in the bottom-right corner, despite being closer to the goal, are noticeably similar to those in the top- right corner. In contrast, CGCIVL achieves more accurate value estimates, especially for states far from the goal.
Ablation As shown in Figure evaluate the perfor- mance of CGCIVL under different penalty coefficients 7) The results indicate that both excessively small and large degrade the performance. Specifically, when = 0.0, the conservative regularization is entirely removed, and the algorithm struggles on both navigation and stitching tasks. This suggests that the algorithm fails to learn an accurate value function due to the overestimation of values on uncon- nected state-goal pairs. Conversely, excessively large " (e.g. 17 1.0) also harms performance, the optimization and limits the ability of the model to fit the true value effectively. In practice, we determine the optimal through empirical validation across multiple candidate values.
COOML
200
over-constrains
160
20
175
Sh00-0003000
Slap-120-000
paintmaze stitch
1.0
Figure 3. Estimated value functions of GCIVL and CGCIVL in the
caanMaaz--iaat-stinnee environment (a 0.7). The goal location is marked with red star. The magnitude of the value function represented by the color scale, with warmer colors indicating higher values. Arrows depict the actions chosen by the policy.
Success Rate
5.3. Ablation Study
Step
Step
0=0000
0=
Ablation As shown in Figure 4, we analyze the per- formance of CGCIVL under the varying cross-traeectory goal-sampling ratio, controlled by the parameter a. In goal- navigating tasks, increasing achieves higher success rates by encouraging the algorithm to focus more on in-trajectory goals during training. However, as reaches higher values (e.g., 0.7, 0.9, 1.0), the algorithm's performance remains relatively stable, largely due to its accurate value function estimation, even with a certain probability of random goal sampling. In contrast, in goal-stitching tasks, reducing a brings significant improvements in experimental results, sampled state-goal pairs in the test often span across multi- ple trajectories in the dataset, requiring the agent to prioritize cross-trajectory goals during the training process.
Figure 5. Performance of CGCIVL on two different environments under varying 7). The success is shown across different training steps, with shaded regions representing the standard deviation.
The results in Figure 6 highlight the signifi- cant contributions of both the hierarchical policy framework and the quasimetric distillation mechanism to CGCIVL's performance. The figure shows that hierarchical architec- tures excel in long-horizon tasks through sub-goa selection and long-term planning but offer limited benefits in simpler environments. In addition, the quasimetric distillation mech- anism is crucial for accurate value estimation, especially for cross-trajectory goals. Its removal severely impacts success rates, with the most pronounced effects observed in stitching tasks. These findings emphasize that the hierarchical policy framework enables efficient planning, while the quasimetric distillation mechanism ensures robust value estimation for goal stitching and long-horizon reasoning.
Ablation 3
Pointwaze-large-nn naviigateee
Pointmaze-large-te
Rato
0.4
0.4
0.2
0.2
6. Related Work
0.0
0.0
Stop
Goal-Conditioned RL GCRL focuses on learning uni- versal value function (Schaul et al., 2015), enabling agents to achieve various goals. Prior work RL has explored a diverse range of techniques, such as contrastive learning (Eysenbach et al., 2021; 2022), state- occupancy matching (Durugkar et al., 2021; Ma et al., 2022a) and successor features (Borsa al., 2018; Touati
0.5
d- 0.7
0.9
1.0
goal-conditioned
Figure Performance CGCIVL on two different environments under varying a. The success rate is shown across different training steps, with shaded regions representing standard deviation.

---

Conservative Offline Goal-Conditioned Implicit V-Learning
Table Experimental results for PointMaze, AntMaze, HumanoidMaze, Cube, Scene and Puzzle across diverse datasetsets, The table reports the average binary success rate (%) across five test-time goals for each task, averaged over seeds. Standard deviations are indicated by the + symbol. Entries within 95% of the best pes-orrrfonnnng value in each row are highlighted bold.
Environment
Dataset Type
GCBC
GCIVL
GCIQL
QRL
CRL
I
CGCIVL
pointmaze-mediing medium-navvgate--
9
63 1
53 1
82 5
= 7
79 1 15
87 +
pointmaze-large large-llaa--g--tt------
29 = 6
45
3143
86
39
58 = 15
92
navigate
pointmaze-giane saatizzeggintttaae glant-navigate-v0
1
0
0
68 +7
27 10
46
80 = 12
average performance
13.00
36.00
29.00
78.67
31.67
61.00
86.33
PointMaze
centtmazz-mmdium -itichu--ttich-v
23 18
14
+
80 12
10 8
74
89 +
stitch
hanntmaze-lagg-stttche
7+5
12
31+2
84
0+0
13++6
98
pointmaze-gimml-siceee
0:::0
0::00
0::00
50 8
0:::0
0
81 1:
average performance
10.00
27.33
17.33
71.33
3.33
29.00
89.33
untmaze-mediumm antmaze-meeiiu-nnvvggl -iaiddiiiinnnniin-eeee
71 14444
3
navigate
mutmaze-lareennvvigae
2
16
34
=
91
0::0
0
0
16
16 3
73
average performance
17.67
29.33
35.00
59.67
64.67
84.00
86.33
AntMaze
mumaaee-mediun-sticch--
45 11
44
29 :
59 7
53 =
94 1
91
antmaze-large-stttenn
3
18
7
18 2
11
67 :
79 +
stitch
milmaize-gian--stitch-v
0
0
0
0 0=
0::0
25 7
36
average performance
16,00
20.67
12.00
25.67
21.33
62.00
68.67
8++2
24
27
21 +
60
89 +
91 +
humanoidmazz-largee law.:eddddnnnelaaaeaeaiiiiieee navigate-vv
1
2+11
2+1
5=1
+
58
navigate
humanoidmaze-ge hatrrrrr--iian--nviatee----
0:1:0
0::0
0
1=
3:1:
1::
average performance
3.00
8.67
9.67
9.00
29.00
54.00
59.33
HumanoidMaze
humanoidmaze meeiinnzzzeeeiuu------ mediim--sitthh---eeeeeeee-
=
12 2
12433
2
1:2
90
61:
0
3=1
4
4:3
stitch
humanoidmazz-gian -iant-stitch---
0::
0::0
0
0=
0
6
34
average performance
11.67
4.33
4.00
7.00
13.33
40.67
52.00
cuhe-single
6++2
53 +
68 +
5 5-1
19 + 2
15 15++3
84 +
cube-double-plee doublee-lay-vO
1++1
36 +
40 +:
1
10 -:
6
46
Cube
play
tiabbe-rrple-pla--O-
1
1:::
3
0=0
4
3
5
babbe--uadrule-plaa--..
01:
0 0
010
0=
0
0
0
average performance
2.00
22.50
27.75
1,50
8.25
6.00
33.75
Scene
play
scene-play-v0
5
51 #
=
77:15
95 +:
3
12422
muzzle-4x4-play-x
01:
1312
26
0=0
0::
0.00
Puzzle
play
puzzle-4x5-play--
0:0
7+1
1411
0=0
140
411
1513
puzzle-4x6-play-v0
1241
0=0
4
3
94
average performance
0.05
9.00
36.75
0.25
2.00
4.75
17.25
Ollivier, 2021; Ghosh et al., 2023). One of the primary challenges GCRL the low learning efficiency caused by sparse rewards. HER (Andrychowicz al., 2017) is proposed address this issue, which treats the intermedi- ate states in trajectories new goals, thereby providing additional learning signals to the agent. Other works have explored alternative approaches, such as hierarchical frame- works (Kulkarni et al., 2016; Nachum et al., 2018; :; Chane- Sane al., 2021), 1), which decompose tasks into manageable subgoals, and model-based planning (Zhu al., 2021; Men- donca et al., 2021), which utilizes predictive models to guide agents toward their goals.
as quasimetric. Based on this insight, various network architectures have been proposed to effectively represent quasimetric functions (Wang & Isola, 2022b;a; Liu al., 2023). Utilizing these architectures represent the value function helps constrain the function space, thereby acceler- ating value function convergence (Wang et al., 2023).
Offfine GCRL Offline GCRL aims to learn goal-reaching policies from pre-collected datasets. Due to the lack of inter- actions with the environment, most GCRL algorithms strug- gle to perform well in offline settings. Some approaches adopt imitation learning methods, directly fitting policies to the trajectories in the dataset (Ghosh et al., 2021; Yang et al., 2022). However, these methods heavily depend on the quality of the collected trajectories. Other approaches adapt traditional offline reinforcement learning algorithms (Ku- mar et al., 2020: Kostrikov et al., 2021) to goal-conditioned
In addition, several studies have discovered that the opti- mal value function in GCRL satisfies the triangle inequal- ity (Pitis et al., 2020; Wang & Isola, 2022a; Liu et al., 2023), offering novel perspective that the optimal value function be interpreted generalized metric function, known
8

---

Conservative Offline Goal-Conditioned Implicit V-Learning
paintmaze-lane laree-lrrg--aa navigate vö
paintmaze- htt.maaae.iintttaaaaaaae..om
peintmaze large stitch-v0
pointmaze giant-atitch v0
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.2
Step
Step
Step
Step
(wlo QD)
Figure 6. Ablation study results on four tasks: pointmace-large-naviiee pointmaze- siieanazz--gat---avvttt---- wttp://wwwwggaa....oooooeeeeeeee pointmaze-large-stiih-he and pointmaze ciannnzzegiaat--eee giant-stitch--v stitch-vO. Each plot compares full CGCIVL algorithm against ablated versions. CGCIVL (blue) represents the complete algorithm. CGCIVL (w/o HF) (green) removes the hierarchical policy framework, and CGCIVL (w/o QD) (red) removes the quasimetric distillation mechanism.
scenarios, avoiding the issue of distribution shift (Park et al., 2023). key challenge in offline GCRL. is goal stitch- ing (Park et al., 2024). Previous works have utilized genera- tive models to stitch different trajectories the dataset (Kim al., 2024; Li al., 2024). However, these methods de- pend on the quality of the trained generative model, making the reliability of the generated trajectories difficult to en- sure. In contrast, we focus on enabling the agent to learn and integrate knowledge from existing trajectories without augmenting the dataset.
Acknowledgments
We gratefully acknowledge the support from the Dis- tinguished Young Scholars Project funded by the Nat- ural Science Foundation of Guangdong Province (No. 2025B (2SB1155020060), the Basic and Applied Basic Research Program the Guangzhou Science and Technology Plan (No. 2025A04J7141) and the Xiaomi Young Talents Pro- gram.
Impact Statement
7. Conclusion
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential socictal consequences of our work, none of which we feel must be specifically highlighted here.
In this work, classify state-goal pairs in offline settings and identify a key challenge in value estimation for cross- trajectory pairs-some of these pairs may be unconnected. Values of those unconnected state-goal pairs cannot be es- timated accurately, and errors may be propagated to other states, thus misleading policy to extract suboptimal actions. To address this problem, propose novel algorithm that incorporates regularization term and quasimetric con- straints during the value function learning process. This approach ensures the underestimation of unconnected state- goal pairs while providing accurate value estimates for con- nected pairs. Experiments across multiple environments and datasets demonstrate that our method significantly im- proves performance on conl-conditioned tasks, especially in goal-stitching scenarios.
References
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong. R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. In Advances in Neural Information Processing Systems, vol- ume 30, 2017.
Borsa, D., Barreto, A., Quan, J., Mankowitz, D., Munos, R., Van Hasselt, H., Silver, D., and Schaul, T. Uni- versal successor features approximators. arXiv preprint arXiv:1812.07626, 2018.
Future research could focus on developing more robust and generalizable framework for learning reduced goal spaces from offline datasets, thereby improving the algorithm's 's scalability and adaptability to diverse and complex envi- ronments. Additionally, an interesting direction could be exploring strategies to identify more suitable distributions for underestimating the value function of state- goal pairs, thereby improving the stability of the learning process.
Chane-Sane, E., Schmid, C., and Laptev, I. Goal- conditioned reinforcement learning with imagined sub- goals. In International Conference on Machine Learning, pp. 1430-1440, 2021.
Durugkar, I., Tec, M., Nickum, S., and Stone, P. Adver- sarial intrinsic motivation for reinforcement learning. In

---

Conservative Offline Goal-Conditioned Implicit V-Learning
Advances in Neural Information Processing Systems, vol- ume 34, pp. 8622-8636, 2021.
Liu, B., Feng, Y., Liu, Q. and Stone, P. Metric residual network for sample efficient goal-conditioned reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8799 88999 2023.
Eysenbach, B., Salakhutdinov, R., and Levine, S. C- learning: Learning to achieve goals via recursive clas- sification. In International Conference on Learning Rep- resentations, 2021.
Liu, M., Zhu, M., and Zhang. W. Goal-conditioned rein- forcement learning: Problems and solutions. In Inter- national Joint Conference on Artificial Intelligence, pp. 5502-5511 2022.
Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. R. Contrastive learning goal-conditioned reinforce- ment learning. In Advances in Neural Information Pro- 2022.
Lynch, C., Khansari, M., Xiao, T., Kumar, V., Tompson, J., Levine, S., and Sermanet, P. Learning latent plans from play. In Proceedings of the Conference on Robot Learning, pp. 1113-1132, 2020.
cessing Systems, volume 35, pp.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.0721 07211 2020.
Ma, J. Y., Yan, J., Jayaraman, D., and Bastani, O. Offline goal-conditioned reinforcement learning via f-advantage regression. In Advances in Neural Information Processing Systems, volume 35, pp. 310-323, 2022a.
Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C. M., Eysenbach, B., and Levine, S. Learning reach goals via iterated supervised learning. In International Conference on Learning Representations, 2021.
Ma, J. Y., Yan, J., Jayaraman, D., and Bastani, O. Offline goal-conditioned reinforcement learning via f-advantage regression. In Advances in Neural Information Processing Systems, volume 35, pp. 310-323 323, 2022b.
Ghosh, D., Bhateja, C. A., and Levine, S. Reinforcement learning from passive data via latent intentions. In Inter- national Conference on Machine Learning, volume 202, pp. 11321-11332 11339, 2023.
Mendonca, R., Rybkin, O., Daniilidis, K., Hafner, D., and Pathak, D. Discovering and achieving goals via world models. In Advances in Neural Information Processing Systems, volume 34, pp. 14379-24399, 2021.
Kim, S., Choi, Y., Matsunaga, D. E., and Kim, K.-E. Stitch- ing sub-trajectories with conditional diffusion model for goal-conditioned offline rl. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13160-13167, 2024.
Nachum, O., Gu, S. S., Lee, H., and Levine, S. Data- efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, volume 31, 2018.
Kostrikov, Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. In International Conference on Learning Representations, 2021.
Newey, W. and Powell, J. Asymmetric least squares estima- tion and testing. Econometrica: Journal of the Economet- ric Society, pp. 819-847, 1987.
Kulkarni, T. D., Narasimhan, K., Saeedi, A., and Tenen- baum, J. Hierarchical deep reinforcement learning: In- tegrating temporal abstraction and intrinsic motivation. In Advances in Neural Information Processing Systems, volume 29, 2016.
Park, S., Ghosh, D., Eysenbach, B., and Levine, S. Hiql: Offline goal-conditioned rl with latent states as actions. In Advances in Neural Information Processing Systems, volume 36, pp. 34866-34891, 2023.
Park, S., Frans, K., Eysenbach, B., and Levine, S. Ogbench: Benchmarking offline goal-conditioned rl. arXiv preprint arXiv:2410.20092, 2024.
Kumar, A., Fu. J.. Soh, M., Tucker, G., and Levine, S. Stabilizing off- ooolcy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, volume 32, 2019.
Pitis, S., Chan, H., Jamali, K., and Ba, J. An inductive bias for distances: Neural nets that respect the triangle inequality. In International Conference on Learning Rep- resentations, 2020.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative g-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp. 1179-1191, 2020.
Schaul, T., Horgan, D., Gregor,
K., and Silver, D. Uni- versal value function approximators. International Conference on Machine Learning, pp. 1312-1320, 2015.
Li, G., Shan, Y., Zhu, Z., Long, T., and Zhang, W. Diffstiich: Boosting offline reinforcement learning with diffusion- based trajectory stitching. In International Conference on Machine Learning, volume 235. pp. 28597-28609, 2024.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
10

---

Conservative Offline Goal-Conditineed Implicit V-Learning
Touati, A. and Ollivier, Y. Learning one representation optimize all rewards. In Advances in Neural Information
Processing Systems, volume 34, pp. 13-23, 2021.
Wang, R., Foster, D. P., and Kakade, S. M. What are the statistical limits of offline rl with linear function approxi- mation? arXiv preprini arXiv:2010. 1895, 2020.
Wang, R., Wu, Y., Salakhutdinov, R., and Kakade, S. Insta- bilities offline with pre-trained neural representation. In International Conference on Machine Learning, pp. 10948-10960, 2021.
Wang, T. and Isola, Improved representation of asym- metrical distances with interval quasimetric embeddings. arXiv preprint arXiv:221..55100 2022a.
Wang. T. and Isola, P. On the learning and learnability of quasimetrics. arXiv preprint 25X4::2206,15778, 2022b.
Wang, T., Torralba, A., Isola, P. and Zhang. A. Opti- mal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learn ing, pp. 3641 1-36430, 2023.
Yang. R., Lu, Y., Li, W., Sun, H., Fang, M., Du Y.. Li, X., Han, L., and Zhang, C. Rethinking goal-conditioned supervised learning and its connection to offline rl. In International Conference on Learning Representations, 2022.
Zhang, T., Janner, M., Li, Y., Rocktäschel, T., Grefenstette, E., and Tian, Y. Efficient planning in a compact latent action space. In International Conference on Learning Representations, 2023.
Zhu, M., Liu, M., Shen, J., Zhang, Z., Chen, S., Zhang, W., Ye, D., Yu, Y., Fu, Q., and Yang. W. Mapgo: Mode- lassisted policy optimization for goal-oriented tasks. In International Joint Conference on Artificial Intelligence, pp. 3484-3491, 2021.

---

Conservative Offline Goal-Conditioned Implicit V-Learning
A. Analysis of GCIVL
GCIVL a variant of Implicit Q-Learning (IQL) (Kostrikov et al., 2021) designed for handling goal-conditioned tasks in the offline setting. GCIVL leverages expectile regression (Newey Powell, 1987) learn the goal-conditioned value function, updated iteratively as follows:
Va+1 = min E 10
V(s,
(16)
V d'mpp''' (..g)
where - (xco) x² denotes the expectile loss with an expectile T E [0.5,1).
Lemma A.1. In the deterministic environment described above, there exists policy TH such that the value function update Equation (16) can be equivalently expressed as:
V4+1 arg min Baw grop (gla) ((((s,)) + >Va(6' V(s,9)) anay of' (+(,,)))
(17)
where T- 8,0) represents the deterministic transition dynamics, and m. is policy that depends on the current value function vi.-
Based on the above lemma, GCIVL be interpreted composite process of value function evaluation and policy improvement. Specifically, Vi+1 is computed via policy evaluation by iterating the Bellman operator:
B V/ = r 7 Vk.
(18)
where the transition operator pr: Vn is defined as:
= p= = (a As + 1, the operator converges to the optimal Bellman operator T (Ghosh al., 2023).
(((b))))))) (V(ss .)]-
(19)
B. Proofs
B.1. Proof of Lemma A.1
Proof.
V&+1 min DD (C(((s)) + vVi((' V(s,g)))
(20)
min Baotn (6 (1 (r(s,)) >Va V(s,g))),
(21)
where PD(s) (s) denotes uniformly sampling state from the dataset, and Pre (s'ls, represents the probability of s' being adjacent to s in the dataset, conditioned on s and goal We assume:
U. {s' pPg 8,9) > DD,
(22)
U++ {s' r(s,g) + +VVa(s' > V(s.g), s' U.))
(23)
Ug = Ug - UU+
(24)
Thus:
Vk+1 = min E [o))
Σ T(r(8,g) ( +VV(''')) -(s,g)) Pxo 8,9)
s'EU²
(25)
(ss,g))² Pag (s' |
s'EUT
By setting the derivative of Equation (25) to 0, we obtain the following expression for Vk+1:
Vi+1 (8. Eveu, TP (s'(s.g) (s. s') + EveV, TP7 (s' (s g) BB g') 7 EveUt Pa, (8' 9. + T/. Eveuy Pr, (s.18888,
(26)
12

---

Conservative Offline Goal-Conditioned Implicit V-Learning
where 5) r(),g) VVi('', g). We denote:
P p (8'1sg)) p(s' | s.g) = Evven (.0)+))))++111- Eveu, prg(d))))) s E
(27)
(1 (1-1) (5' s.
<.vey (1)( Eveu, Pao (1 s' EU;
where Ev, p(s' s, 1. We could express Vk+1 (3. EvEU, 0(s' (. Bs 8'). Since the environment is deterministic, we first define generalized inverse mapping:
(s' s.g) = a, where T(a s.9) s'.
(28)
Then we could construct policy (a s.
pss |,,g), ff= s.g).
(a 8,
(29)
0.
otherwise.
B.2. Proof of Theorem 3.3
Proof.
(x(s,a) a)
[V((s,g) ('(sg))
(30)
Vi((, 9) IV TV-- (8,
(31)
(V((,,9) TV +
(*
(32)
[V<(s,g) (s. + max V/-i('',g) max V (s's
(33)
S (8, - TV (s. g)و + 7 max V2- (5',g) - V'(''
(34)
(, + max 1
(35)
where the inequality in (33) holds due to the triangle inequality and the inequality in (35) holds because the absolute difference between two maxima bounded by the maximum of the absolute differences.
B.3. Proof of Theorem 4.1
Proof. By setting the derivative Equation (8) to O, we obtain the following expression for V441 in terms of Vu:
vs.g E D, (s.t = (s. 1( (s)
(36)
ph (gls)
Since, p(g]s) 2 0, 0.p% 2 conclude that each iteration, the updated V-value remains underestimated, i.e.
pik+1= < B= As proved in prior work (Kumar et al., 2020), if the reward function r(s.g) g) and the transition function T(s'(s.9) satisfy concentration properties, then with high probability(w.h.p) 2 1 5, E (0, 1), the empirical Backup operator is bounded:
&
W,S.gED, -BV (1 (1->)
(37)
So the fixed point of Equation (36) is given by:
() s (I 11-7Pp 7-pP)-11 R n /
Cr.T, Rymax
por
(1 -1-9 1->)VD
(38)
=V*(s,g) 17 [1 (1-7pka-1
(s,g)
(1-7PT) (Pn--
Cr. Rmax
(s,g),
pet
(1 - 7) (
thus proving the underestimation of the value iterated via Equation (8).
13

---

Conservative Offline Goal-Conditioned Implicit V-Learning
Furthermore, when p(g]s) (g) meaning the goals are sample uniformally from dataset D and the emprical Bellman operator is unbiased, the estimated value function V= is as follows:
D
(=(ss))= 1 (T (P------1
P unni
(x.g)-
(39)
a'
(1 a) Peni
B.4. Proof of Proposition 4.2
Proof. For the state-goal pair (st .) which within the same trajectory, we have:
(S 9) = V=
(1 - -
(s,g).
(40)
Pag a p
Obviously, Plyj 2 Pumi then:
V= .9) 2 (= .)) 17 [11 --p=)- (st.g)
(41)
To guarantee
(= g) 2 (=*(st -
it suffices to ensure, for the given pair (s+,g). .g).
(1 (1- (8+ :)) S e.
Hence, can choose
s
(I- [1] (st
If the inequality must hold for all state-goal pairs (s,g), then
e [| (II~PPy)
7
S
sup (s.9)
(I pp
(8,g)
B.5. Proof of Proposition 4.3
Proof. For the state-goal pair (st, .)) which within the same trajectory, have:
(=5(s (s+ g) 17 [(1 - -P=) (s+,g). .g).
(42)
For the state-goal pair (st,g) which is unconnected, pinj = 0, so we have:
(= ,9) = (=*(s (I (
(s.g).
(43)
We want to show that there exists choice of a E (0,1) such that
(x(() < (*((tgg) -
Recall the two key inequalitiessequalitis::
(1) VJT ( .9) 2 V=(st.g) "7 "n(11 - --p") -1 (st ,9),
(2) (8 (9)) (*(xg) 11 (1 -1- [+t ]] (s.9).
14

---

Conservative Offline Goal-Conditioned Implicit V-Learning
For brevity, define:
e = (1-
(0t.)) c2(a) = - (8,9).
Then from (1) and (2) we get:
v=(**.g) 2 (=( n11, (=((sT = (=(s,g) - ne2(a).
Subtracting the second from the first,
(**(st - -V 2 ( (909)-nc1 ma] - -)(ssg)) -
i.e.,
V*(st 9) - 2 ( + [cz(a)
a).
As - term I-a grows unboundedly. Under mild assumptions (nonnegative transitions, < 1, etc.), the operator (I (/-77 -P-)))--[[] is monotone with respect its input. Hence c2(a) be made arbitrarily large by choosing close enough to 1. Therefore, 7) (e2 ci| can exceed any fixed constant (including (V* (V=tss..0) - V* (8888))) if necessary). Consequently, we can select sufficiently close to 1 such that:
(= .g) - (= .9) > e.
Equivalently,
(=*(s < (** - c.
Thus, for any fixed 17, there always exists that satisfies the desired inequality.
B.6. Proof of Proposition 4.5
Proof. Any connected state-goal pair can be categorized into two cases:
Case 1: In-trajectory state-goal pairs.
According to Proposition 4.3, the proposition holds directly.
Case 2: Cross-trajectory state-goal pairs. Assume s and g in different trajectories within the dataset D. By definition, (s).g) = 0, and thus we have:
(= .g) (*(s,9) [a (1--7pp (U--pp)-11 TLmn (5,
On the other hand, suppose there exist n intermediate states $1, $2...., Sn connecting some s to the same goal g. where each pair (8,, 8441) is in-trajectory. By the properties of the quasimetric, we know:
(s* 9) 2 (** + (s),))) + ... (*((n.)).
Therefore, to establish a strict value separation, it suffices to show:
(** .9 < (**(t, s1) (TT( .... (T*())) e.
(44)
According to the proof in Appendix B.5, it is straightforward to show that there exists some a such that the inequality in Equation (44) holds. Combining Case and Case 2, the proposition is proved.
15

---

Conservative Offline Goal-Conditioned Implicit V-Learning
C. Implementation Details
C.1. Interval Quasimetric Embeddings
IQE trains an encoder f(æ; () that maps states s S to latent vectors 2n E 2, alongside latent quasimetric head
(2,, Zy: 16) to estimate the distance between state s and goal g in the latent space. Together, these components define parameterized quasimetric over the state space S:
d(s. 0, xb) == diaem (s; f(: ():
(45)
IQE structures input latents as two- do-ddmmeniioall matrices by reshaping. Given latents u, v R IQE quantifies the Lebesgue measure of unions of multiple intervals along the real line:
Vi 1.2...... k, du v) &
U 1=1
Uus
Juy. max(uy; [iy]]
(46)
fundamental IQE is obtained by summing over all components:
dIQE-sum (2, v)
(u(u.))
(47)
more flexible alternative, IQE- maxmean,, introduces an additional parameter a, blending maximum and mean reductions:
dIQE= mammmenn (u, a) maxmean(d) (u, o),...,,,dd du v); a max(di(u, v).... (u, v)) + (1 - a)) mean(d1(u, u).... da (u, e)).
(48)
(49)
C.2. Implementation Details of CGCIVL
Our algorithm implementation is based on the reproduction of HIQL in the open-source OGbench and can be found https//githubb.oomkkka222 8/CGCIVL..it. On this basis, we implemented the quasimetric model based IQE, denoted as dg. The underestimation of the value function is represented as max de,. In order to stabilize the maximization process of dg., a function & is introduced, which a monotonically increasing convex function. The final penalty term can be written as max ((de). J. At the same time, in the process of distilling the value function, a masking mechanism is employed to exclude certain loss contributions when the error between the predicted and target values falls within specified range.
C.3. Hyperparameter Settings
The detailed hyperparameter settings are shown in Table The training process employed a batch size 1024, with the policy and value networks designed as MLPs of dimensions (256, 256) and (512, 512, 512), respectively. The GELU activation function was used to ensure smooth gradient flow, while the Adam optimizer, configured with learning rate of 0.0003, facilitated efficient parameter updates. To further stabilize the training process, the target network smoothing coefficient is set to 0.005.
Table 2. Hyperparameters.
Hyperparameter
Value
Batch size
1024
Policy MLP dimensions
(256, 256)
Value MIP dimensions
(512, 512, 512)
Nonlinearity
GELU
Optimizer
Adam
Learning rate
0.0003
Target network smoothing coefficient
0.005
Penalty coefficient
0.01
Some hyperparameters in the training process are tuned based on the specific task and environment. Specifically:
16

---

Conservative Offline Goal-Conditioned Implicit V-Learning
In the antmaaz--gian--sticc---o0 environment, the algorithm was trained for 2,000,000 steps.
In the bumaaoidmazzziiaa--nvvigggl and humanoidmaaz---aa-ssticc--- environments, the algorithm was trained for 3,000,000 steps.
For all other environments, the training steps set to 1,000,000, consistent with the settings in OGbench.
In the goal-stitching task, a was set to 0.5 to emphasize sampling from cross-trajectory pairs. In contrast, for the goal- navigating task, a was set to 0.9 to prioritize sampling from in-trajectory pairs. During the quasimetric distillation phase, another parameter, Og, was introduced to control sampling, ensuring the quasimetric focuses more on in-trajectory pair estimation. ag is typically set 0.8.
D. Details of Environments and Datasets
We evaluate the proposed algorithm on OGbench, recently introduced benchmark designed to assess offline goal- conditioned reinforcement learning algorithms across diverse tasks and datasets.
Environments
Our experiments involve six representative environments:
PointMaze: A 2D navigation task where point mass agent must traverse maze to reach specified goal locations.
AntMaze: A more challenging task requiring quadrupedal ant agent with 8 degrees of freedom (DoF) to navigate complex environments.
HumanoidMaze: The most complex task, involving humanoid agent with 21 DoF, which requires high-dimensional control and advanced planning to solve.
Cube: A pick-and-place manipulation task involving 1-4 cube blocks. The agent must learn to move, stack, swap, and permute cubes using a robot arm. Training data consists of play-style demonstrations with random pick-and-place actions. The task requires generalizable multi-object manipulation and long-horizon planning from unstructured data.
Scene: long-horizon manipulation task involving diverse objects such as a cube, drawer, window, and button locks. The agent must execute complex object interactions (e.g.. unlocking and using containers) to reach goal configurations. It demands sequential reasoning and composition of atomic manipulation skills.
Puzzle: A combinatorial reasoning task where robot arm solves the "Lights Out" puzzle by pressing buttons on a grid (3x3 to 4x6). Pressing toggles adjacent buttons' states, and the goal is to match target pattern. The agent must learn precise control and combinatorial generalization over a vast discrete state space, based random-play trajectories.
Each environment is evaluated across three maze configurations:
Medium: small maze layout identical to the medium maze in D4RL (Fu et al., 2020).
Large: A larger maze with increased navigation complexity, matching the large maze in D4RL.
Giant: A significantly larger maze, featuring paths up to 3000 steps, adapted from Zhang al. (2023) with added layout complexity.
Datasets Three dataset types are used to evaluate different capabilities:
Navigate:
The dataset collected using noisy expert policy that navigates the maze by sequentially reaching randomly sampled goals. This dataset is used to evaluate navigation performance.
Stitch: The dataset composed of short goal-reaching trajectories. Completing tasks with this dataset requires stitching multiple short trajectories to reach the goal, testing long- horizon reasoning capabilities.
Play: The dataset consists of "slay"-style data collected by non-Markovian expert policies with temporally correlated
noise. It features unstructured, exploratory interactions with the environment, where the agent performs random actions without specific task goals.
17


Reviews:
[{"review_id": "P1BlNgtYCD", "reviewer": "Reviewer_Q24N", "summary": "This paper proposes an algorithm for goal-conditioned offline RL called Conservative Goal-Conditioned Implicit V-Learning (CGCIVL). CGCIVL improves upon Hierarchical Implicit Q-Learning (Park et al., 2024b) by introducing two techniques. First, it adopts a regularizer similar to CQL (Kumar et al., 2020) to penalize values for unconnected state-goal pairs. Then, based on the observation that a goal-conditioned value function is a pseudometric, it models the value function with Interval Quasimetric Embeddings to prevent over-penalization of values for connected state-goal pairs. CGCIVL outperforms existing baselines on the OGbench(Park et al., 2024a) benchmark containing various goal-reaching tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s constructive feedback. Below, we respond to each concern point-by-point.\n\n### R1: Methods and Evaluation Criteria\n\nThe choice between using $V_{\\theta_v}$ or the distilled $V_{\\theta_d}$ for advantage estimation ($\\tilde{A}^h$ and $\\tilde{A}^h$) is flexible, as both approaches are empirically valid, surpassing all baselines. Our experiments confirm that $V_{\\theta_d}$ achieves comparable performance when used for policy extraction, suggesting either value function can be adopted without compromising results. We will clarify this point in the final paper.\n\n|                         | pointmaze-large-navigate | pointmaze-giant-navigate | pointmaze-large-stitch | pointmaze-giant-stitch |\n| ----------------------- | ------------------------ | ------------------------ | ---------------------- | ---------------------- |\n| CGCIVL(with $\\theta_v$) | $92 \\pm 4$               | $80 \\pm 12$              | $98 \\pm 2$             | $81 \\pm 17$            |\n| CGCIVL(with $\\theta_d$) | $98 \\pm 2 $              | $78 \\pm 14$              | $96 \\pm 6$             | $82 \\pm 15$            |\n\n### R2: Theoretical Claims\n\nThank you for pointing out this question. Here we briefly explain why there exists  an $\\alpha$ satisfies condition for all state-goal pairs.  Proof of Proposition 4.3 demonstrates that for any fixed $\\epsilon > 0$ and arbitrary tuple $x=(s^+,s^-,g)$ sampled from dataset, where $(s^+,g)$ is in-trajectory and $(s^-,g)$ is cross-trajectory, there exists an $\\alpha_x\\in(0,1)$, such that inequality holds when $1>\\alpha>\\alpha_x$.  Consequently, let $\\tilde{\\alpha}=\\sup_x{\\alpha_x}$, then we could find a static $\\alpha\\in (\\tilde{\\alpha},1)$ , which satisfies the condition for all state-goal pairs. We will provide further clarification in the revised version of the paper.\n\n### R3: Experimental Designs or Analyses\n\n As suggested, we have conducted additional ablation studies to analyze the sensitivity of the penalty coefficient $\\eta$, and results are presented in **Figure 1** (see the linked [PDF](https://anonymous.4open.science/r/additional-experiment-DDFC/icml_2025_rebuttal.pdf) for details).  The results indicate that both excessively small $\\eta $ (causing insufficient regularization) and excessively large $\\eta$ (over-constraining the optimization) degrade the performance. In practice, we determine the optimal $\\eta$ through empirical validation across multiple candidate values.\n\n### R4: Other Comments or Suggestions\n\nMaximizing values for connected state-goal pairs is indeed an interesting direction. However, we might need to carefully address several practical considerations: 1) directly sampling from the distribution of connected state-goal pairs in the dataset is difficult, and 2) further analysis is required to establish appropriate theoretical bounds, similar to those in CQL, when incorporating this approach. We plan to thoroughly explore these open issues in future work.", "claims_and_evidence": "Most of the claims made in the submission are supported by clear and convincing evidence. For those that are problematic, refer to the following sections.", "methods_and_evaluation_criteria": "It is unclear why the authors use $V_{\\theta_v}$ instead of the distilled $V_{\\theta_d}$ to estimate the advantage functions $\\tilde{A}_h$ and $\\tilde{A}_l$. Aside from that, the proposed methods and the evaluation criteria make sense for the problem.", "theoretical_claims": "The $\\alpha$ in Proposition 4.3 depends on the choice of the state-goal pairs, which means there might be no $\\alpha$ that satisfies the condition for all state-goal pairs. The proposition becomes irrelevant since $\\alpha$ is fixed for the entire training process. As Propositions 4.4 and 4.5 are all based on Proposition 4.3, the two propositions are also irrelevant.", "experimental_designs_or_analyses": "The penalty coefficient $\\eta$ also seems to play an essential role in the algorithm, but the authors have not conducted a sensitivity analysis on it.", "relation_to_broader_scientific_literature": "The proposed algorithm is mainly based on HIQL(Park et al., 2024b). The penalization term for unconnected state-goal pairs was inspired by CQL(Kumar et al., 2020). The observation that an optimal goal-conditioned value function is a quasimetric was proved by Liu et al. (2023). Finally, the authors modeled their value function using IQE(Wang & Isola, 2022a).", "essential_references_not_discussed": "To the best of my knowledge, the paper has cited all of the essential references.", "other_strengths_and_weaknesses": "Trajectory stitching is necessary for real-world problems because collecting high-quality data is challenging. This paper proposes an interesting method of applying HER for cross-trajectory state-goal pairs.", "other_comments_or_suggestions": "CQL adds a term to the loss function that maximizes the values for in-distribution data so that the regularizer is canceled out for in-distribution data. Similarly, adding a loss function term that maximizes the values for connected state-goal pairs might be helpful.", "questions_for_authors": "I do not have any additional questions for the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f78b1LLr8K", "reviewer": "Reviewer_96R5", "summary": "This paper proposes a method for offline goal conditioned reinforcement learning with a penalty term to penalize the value function for unconnected state-goal pairs and does evaluation on OGBench. The results suggest the method outperforms previous methods on goal conditioned tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful feedback on our work. Please refer to detailed responses below to each of the raised concerns.\n\n### R1: Weaknesses\n\nIn order to provide more comprehensive evaluation of the performance, we have conducted additional experiments in three manipulation environments (Cube, Scene, Puzzle)  \\- comprising a total of 8 manipulation tasks of varying complexity. The results, presented in **Table 1** (see the linked [PDF](https://anonymous.4open.science/r/additional-experiment-DDFC/icml_2025_rebuttal.pdf) for details), demonstrate that CGCIVL achieves superior performance in all manipulation tasks, particularly in scene and elementary cube tasks, consistent with results in maze environments. We will include these additional experimental results in the revised version of the paper.\n\n### R2: Questions\n\n1. In the original paper, we aimed to validate the algorithm in solving the goal stitching task. However, OGbench currently only provides the stitch dataset in maze environments. Nevertheless, we have supplemented our evaluation with additional experiments in manipulation environments to further verify the algorithm’s performance (See our response in **R1**). We will expand the experimental scope and incorporate these additional results into the final version of the paper.\n2. Besides the analysis of $\\alpha$, we have also performed sensitivity studies on both the penalty coefficient $\\eta$ and the subgoal interval $k$. The analysis of $\\eta$ and related experiments can be found in our response R2 to Reviewer [Q24N](https://openreview.net/forum?id=5ryn8tYWHL&noteId=56lsvAsJb7). **Figure 2** (see the linked [PDF](https://anonymous.4open.science/r/additional-experiment-DDFC/icml_2025_rebuttal.pdf) for details) shows the the performance of CGCIVL across different subgoal interval sizes. Results indicate that CGCIVL achieves the optimal performance with $k$ between $25$ and $50$. Overly small values of $k$ lead to the \"signal-to-noise\" issue in the value functions, as identified in the HIQL paper[1], while excessively large values of $k$ makes subgoals difficult to be achieved.\n\n### R3:  Other Comments or Suggestions\n\n We will include the mean score for each environment in the revised version of our paper to provide more comprehensive results. \n\n[1] Park, S. , Ghosh, D. , Eysenbach, B. , and Levine, S. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems,36, 2024b.", "claims_and_evidence": "The paper makes several claims.\n\n1. Offline goal conditioned reinforcement learning suffers from value over estimation on unconnected state action pairs.\n\nSupport for this claim is presented through theoretical analysis in Theorem 3.3 and experimental evidence both in table 1 and figure 3.\n\n2. The method proposed in this paper addresses the value over estimation and achieves better performance.\n\nGenerally this claim is supported from the main results in Table 1, but it only includes a subset of tasks from OGBench. The evidence would be more convincing if results are shown for all of the OGBench experiments.", "methods_and_evaluation_criteria": "The method and evaluation criteria appears to be well suited for the problem. The proposed method directly addresses the problem and the provides theoretical motivation. The benchmark selection is appropriate.", "theoretical_claims": "The proofs appear to be correct.", "experimental_designs_or_analyses": "The experimental design is valid and the selection of benchmark is good. There are ablation studies to back up claims and compares against the other state of the art methods in offline goal conditioned reinforcement learning. The experimental section can be improved by providing results on the entire OGBench.", "relation_to_broader_scientific_literature": "The key contributions of the paper is related to the advancement of offline goal conditioned reinforcement learning. It proposes a new method that addresses an important problem in this direction of research.", "essential_references_not_discussed": "Other related works that are essential are included in the paper.", "other_strengths_and_weaknesses": "The paper is original and combines ideas in conservative value estimation to offline goal conditioned reinforcement learning. It is well motivated with its approach and achieves higher performance compared to previous algorithms.\n\nA weakness of the paper is that it only compares in maze navigation tasks and is unclear how it would scale to other domains.", "other_comments_or_suggestions": "It would be helpful to include a mean for each of the environments.", "questions_for_authors": "1. Is there a reason the method is not run for all of the environments on OGBench?\n\n2. The method has a lot of important hyperparameters. What are the sensitivity to other hyperparameters besides alpha?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kSjjyBK5J3", "reviewer": "Reviewer_uWae", "summary": "This paper introduces conservatism to prevent overestimation in unconnected state-goal pairs and uses a quasimetric value network to prevent underestimation in connected cross-trajectory state-goal pairs. Theoretical analysis is provided for the idealized version of the algorithm, and the practical implementation of the algorithm outperforms offline goal-conditioned RL baselines on OGBench.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing our work. We have addressed each concern you raised below.\n\n### R1: Claims and Evidence\n\nThank you for the reviewer’s comment. The theoretical guarantees mentioned in our paper indeed refer to the algorithm prototype based on Eq. (8), rather than the practical algorithm. However, in Lemma 1 (Appendix A), we prove that the expectile regression in the practical algorithm is equivalent to the Bellman operator in Eq. (8). Additionally, compared to Eq. (8), the practical algorithm incorporates two key techniques: 1) hierarchical learning to address long-horizon tasks, and 2) quasimetric distillation to improve the efficiency of value learning. These techniques do not fundamentally alter the core components of the CQL-inspired penalty term and quasimetric, which form the foundation of the theoretical analysis. Therefore, while Eq. (8) does not exactly match the practical algorithm, we believe the practical algorithm still benefits from the theoretical guarantees established by the analysis.\n\n### R2: Theoretical Claims\n\n1. We clarify notations you mentioned as follows:\n   - $\\mu(g|s)$ denotes an arbitrary distribution which satisfies $\\operatorname{supp} \\mu \\subset \\operatorname{supp} p_{m}^{\\alpha}$. \n   - $\\hat{V}$ represents an empirical estimate of the true value function $V$ during iteration.\n   - $\\hat{B}$ denotes the empirical Bellman operator, which is the sample-based counterpart of the theoretical Bellman operator.\n2. Yes. The correct statement of Proposition 4.5 should be: For any $\\epsilon>0$ and $\\eta>0$, there exists a hyperparameter $\\alpha$ such that the inequality holds. \n\n### R3: Relation to Broader Scientific Literature\n\nWe would like to clarify that our work extends beyond a mere combination of existing techniques. The key contributions are:\n\n- Problem identification: \n\n  To the best of our knowledge, we are the first to formalize the critical issue of value overestimation for unconnected state-goal pairs in offline GCRL.\n\n- Feasible solutions: \n\n  Our solution penalizes the values of all cross-trajectory state-goal pairs while ensuring that values on connected pairs are not excessively under-estimated. We introduce a CQL-inspired regularization term for the first goal and use a quasimetric model for accurate value estimation of connected pairs to achieve the second. Both methods are supported by theoretical guarantees.\n\n- Difference with original methods\n\n  Unlike CQL, which penalizes OOD actions, we introduce a penalty term tailored for state-goal pairs. Unlike QRL, which trains value functions without value iteration, our approach incorporates quasimetric properties into the value iteration process to ensure accurate value estimation for connected state-goal pairs.\n\nThe novelty of our method lies in re-engineering these components to address a new problem in offline GCRL, rather than merely combining them. \n\n### R4: Weaknesses\n\n1. The originality of our work is discussed in R3.\n2. In R2.1, we provide explanations for any undefined notations and will include these details in the revised version of the paper. As defined in Sec 2, in Algorithm 1, states are randomly sampled, and goals are sampled in two ways: 1) $p_{rand}^{\\mathcal{D}}(g)$ samples uniformly from all states in $\\mathcal{D}$, and 2) $p_m^{\\alpha}(g|s)$ samples from the same trajectory as state $s$ with probability $\\alpha$, otherwise using $p_{rand}^{\\mathcal{D}}(g)$.\n\n### R5: Other Comments or Suggestions\n\n1. See R2.1. \n2. See R4.2. \n3. Future revisions will implement distinct notation per proposition for clarity.\n\n### R6: Questions\n\n1. Although the proof in the original paper is based on discrete state and action spaces, its key components can also be  extended to continuous settings. Non-negative penalty terms for underestimation can be generalized to density-based terms, and concentration bounds for the empirical Bellman operator $\\hat{B}^{\\pi}$ do not require discretization. The Neumann series ensures that  $(I - \\gamma P^{\\pi})^{-1}$ remains well-defined in continuous spaces when $\\gamma < 1$. Therefore, while Theorem 4.1 has not been strictly proven for continuous settings, our algorithm still benefits from the theoretical analysis in continuous environments, as further supported by experimental results.\n2. As described in Sec 2 of our paper, the distance between state and goal should satisfies properties of quasimetric (Eq. (6)). However, the value function should exhibit an inverse relationship with the distance away from the goal (Eq. (7)). Thus we have $V(A;B)\\geq V(A;C)+V(C;B)$, which is hold when $V(A;B)=0, V(A,C)<0,V(C,B)<0$.\n3. Please refer to R1 where we discuss the differences between the practical algorithm and Eq. (8).\n4. Proposition 4.5 is based on Theorem 4.1 by incorporating the quasimetric and replacing $\\mu$ with the uniform distribution $p_{rand}^{\\mathcal{D}}$. Consequently, Proposition 4.5 cannot hold if the conservatism is not included.", "claims_and_evidence": "The performance of the proposed CGCIVL algorithm is demonstrated through abundant experiments, which are convincing evidence. \n\nHowever, the connection between theoretical analysis and the practical CGCIVL algorithm is weak, as the theory is based on an idealized version of CGCIVL (Eq. (8)). The theorems serve therefore more as a motivation than a guarantee.", "methods_and_evaluation_criteria": "Conservatism or regularization is a standard technique in reinforcement learning. The Quasimetric framework serves specifically for the goal-conditioned RL problem. Therefore, the proposed method is overall appropriate for the discussed problem.\n\nThe benchmark OGBench is suitable for goal-conditioned reinforcement learning.", "theoretical_claims": "Many notations are used without formal definition, thus hindering the understanding of the theorems along with their proofs. For instance, $\\mu(g\\mid s)$, $\\hat V$ and $\\hat{\\mathcal{B}}$ in Eq. (8).\n\nThe formulation of Proposition 4.5 is problematic. As claimed in the proposition, the inequality should hold for any $\\epsilon>0$, then this will imply that the $\\hat V^\\pi(s^-,g)$ has to be $-\\infty$. However, this seems to be a minor typo. \n\nI believe the theoretical claims are sound after the above issues are addressed.", "experimental_designs_or_analyses": "Experiments are solid to support the claim.", "relation_to_broader_scientific_literature": "The two key components of CGCIVL, conservatism and quasimetric, are not novel in RL literature. The former is standard in RL algorithms e.g. CQL (Kumar et al., 2020), COMBO (Yu et al., 2021), and the latter is also proposed in https://arxiv.org/abs/2304.01203. The paper is only investigating the effect of combining these two techniques. Nonetheless, the successful combination of these two methods reveal the contribution of this paper.", "essential_references_not_discussed": "No missing reference found.", "other_strengths_and_weaknesses": "Strength: The paper investigates the advantage of combining conservatism and quasimetric. \n\nWeakness: \n\n1) Both conservatism and quasimetric are existing techniques in RL literature, although this does not severely harm originality, as the methods are tailored specifically for GCRL.\n2) Lack of clarity. Many notations are used without formal definition (already discussed above). In addition, the main algorithm (Algorithm 1) needs detailed description. For instance, in Eq. (12), how do we estimate the expectation, and how to sample from $g$ from $p_m^\\alpha(g|s)$?", "other_comments_or_suggestions": "1. All notations should be defined before using. Many notations are not standard across literatures, so they will cause confusion to readers without clear definition. \n2. Algorithm 1 should be described in details to convince the readers that it can be implemented in practice. For instance, we need to discuss how to sample $g$. \n3. Both Proposition 4.3 and 4.5 use $s^-$ and $s^+$, but they stand for different meanings in the two propositions. Therefore, we should consider using different notations.", "questions_for_authors": "1. Does Theorem 4.1 hold for continuous state space or discrete state space? Similarly, does it require discrete action space or continuous action space also holds? This question reflects how the theoretical analysis aligns with practice.\n2. It seems unnatural to require the value function to be a quasimetric, because sometimes the ground truth value function might not be a quasimetric. For instance, Suppose states A,B are connected, but both (A,C) and (C,B) are unconnected. Then we should have $V(A,B)=0$, $V(A,C)<0$, $V(C,B)<0$. This violates the quasimetric property. \n3. Can CGCIVL fit in the framework of Eq. (8), assuming that we do not use any function approximation? This question determines the relationship between the theory and the practical algorithm.\n4. Does Proposition 4.5 still hold if conservatism (regularization) is not added in the algorithm? This is relevant to the novelty of this paper. \n\nI am willing to raise the score if the above concerns are resolved and the comments in the previous part are handled.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E83qteSfim", "reviewer": "Reviewer_W7HN", "summary": "This paper proposes conservative goal-conditioned implicit V-learning (CGCIVL). The main insight of CGCIVL is to penalize cross-trajectory goal-conditioned values, which may potentially be overestimated, with a conservative regularizer. To improve the empirical performance of CGCIVL, the authors additionally employ other techniques (e.g., quasimetric value functions, hierarchical policy extraction, etc.) from the literature. They evaluate CGCIVL on OGBench, showing that it outperforms the previous methods on navigation environments, including those that require stitching.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable suggestions. We have carefully addressed each of your concerns in the responses below.\n\n### R1: Methods and Evaluation Criteria\n\nWe have extended our evaluation to manipulation environments (see Table 1 in the linked [PDF](https://anonymous.4open.science/r/additional-experiment-DDFC/icml_2025_rebuttal.pdf) for details). Detailed analysis is provided in our response R1 to Reviewer [96R5](https://openreview.net/forum?id=5ryn8tYWHL&noteId=wMyrrzRg7V).\n\n### R2: Supplementary Material\n\nWe would like to polish the README file to ensure our algorithm can be easily reproduced.\n\n### R3: Weaknesses\n\n1. We have conducted additional experiments comparing CGCIVL’s performance with and without the conservative regularization term (i.e., $\\eta\\neq 0$ and $\\eta=0$ respectively). As shown in **Figure 1** (see the linked [PDF](https://anonymous.4open.science/r/additional-experiment-DDFC/icml_2025_rebuttal.pdf) for details), removing this component leads to a significant performance drop, confirming its critical role. Furthermore, we observe that the performance is robust within a suitable range of the regularization coefficient, but both excessively small and large values degrade the results.\n2. See our response in R1.\n3. In our paper, all algorithms were evaluated under the same number of training steps to ensure a fair comparison. Different from results in OGbench where baselines were trained with 1M steps, we trained all algorithms for longer steps in complicated environments (See Appendix C.3 for details). **Figure 3** (see the linked [PDF](https://anonymous.4open.science/r/additional-experiment-DDFC/icml_2025_rebuttal.pdf) for details) shows training curves for all algorithms in these complex environments.\n4. We would like to clarify that CQL and quasimetric are the key techniques of our algorithm, addressing value overestimation on unconnected state-goal pairs. IQL is a necessary policy improvement technique, which could be replaced with other methods. The hierarchical structure is a common approach to handle long-horizon tasks and may be omitted in non-long-horizon environments.\n\n### R4: Comments\n\n1. $\\mu(g|s)$ denotes an arbitrary distribution which satisfies $\\operatorname{supp} \\mu \\subset \\operatorname{supp} p_{m}^{\\alpha}$. Trajectory truncation only alters the goal associated with states in different segments, without directly changing the distribution of states in the dataset. Therefore, we can approximate sampling states from $\\mathcal{D}$ as sampling from $d^{\\pi_{\\beta}}$.\n2. We'll mention $V_{\\theta_d}\\in\\mathcal{Q}^-(S)$ near Eq. (13) for clarification in the revised manuscript.\n3. The parameter $\\beta$ in Equations (14)-(15) serves as the temperature coefficient for both the high-level and low-level policy extraction. We empirically set $\\beta=3.0$ across all experiments.", "claims_and_evidence": "The claims are empirically supported to some degree, but I do have several questions (see below).", "methods_and_evaluation_criteria": "Their evaluation criteria are reasonable in general, but the tasks are limited to (similar) navigation environments, and it'd have been more convincing if the authors had shown CGCIVL's performance on manipulation environments as well.", "theoretical_claims": "I briefly reviewed the theoretical results (though I haven't thoroughly gone through the Appendix), and at least they look believable to me. The theoretical results are largely based on standard proof techniques about conservative value estimation.", "experimental_designs_or_analyses": "I don't have particular concerns about experimental designs or analyses other than the ones I listed in the weakness section below.", "relation_to_broader_scientific_literature": "CGCIVL is built upon several existing methods --- IQL, GCIVL, HIQL, CQL, and QRL. Although the \"novelty\" of CGCIVL is not necessarily extremely prominent, I think the paper does have a reasonable degree of contribution (given that the claims are fully empirically supported).", "essential_references_not_discussed": "I don't see any particular missing work.", "other_strengths_and_weaknesses": "### Strengths\n* Figure 5 is quite convincing to me (especially in comparison with Figure 1). It is nice to see that $\\alpha < 1$ improves performance on \"stitch\" datasets with the proposed techniques.\n* CGCIVL achieves the best performance on almost all tasks employed in the paper.\n\n### Weaknesses\n* The paper omits a key ablation result -- how does CGCIVL's conservative regularization affect performance? This is the supposed key ingredient of the method, so I believe it is crucial to show how this design choice affects performance. In Figure 4 (in its current form), most of the performance gains are seemingly from quasimetric value functions and hierarchical policy extraction.\n* The authors only evaluate CGCIVL on maze navigation environments. While the authors employ many datasets from OGBench, it'd have been much more informative if the authors had shown how CGCIVL works on other types of environments as well (e.g., manipulation). Does CGCIVL also work well on OGBench manipulation environments? If not, why?\n* The authors use more training steps (e.g., 3M) for some challenging tasks (e.g., humanoidmaze), whereas the baseline results are obtained at 1M steps. Is CGCIVL still better than the baselines when they are trained with the same number of epochs?\n* The proposed method is fairly complicated. It combines a number of different ingredients from previous methods -- quasimetric value functions, hierarchical policy extraction, implicit Q-learning, conservative Q-learning, etc. Hence, to some degree, their method is somewhat expected to work better than the baselines, because the baselines are usually more \"atomic\" (in the sense that they mostly employ one or two key techniques). While I don't think this is a major limitation, it would have been a great plus if their method had been simpler.\n\nOverall, I'm not entirely convinced by the empirical results, mainly due to the lack of ablations and the limited types of environments. I'd be happy to adjust my score if these points are addressed.", "other_comments_or_suggestions": "* $\\mu$ is never formally defined (it is instead somewhat implicitly defined around L180). Relatedly, is $d^{\\pi_\\beta}$ correct? I suspect $s$ is sampled from the dataset distribution, not $d^{\\pi_\\beta}$ (note that they are different when dataset trajectories are truncated).\n* I'd explicitly mention that $V_{\\theta_d} \\in \\mathcal{Q}^-(\\mathcal{S})$ around Equation (13). This is not explicitly stated in the current draft. \n* What is the value of $\\beta$ used for the experiments?", "questions_for_authors": "I don't have any questions other than the ones I asked above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Authors present a way to incorporate stitching into hindsight experience replay in offline goal-conditioned RL with conservatism. This is a really interesting problem in goal-conditioned RL where there should be stitching in the goal space as well as the state space. While standard RL takes care of stitching in state space, hindsight experience replay does not allow for stitching in goal space. The authors propose to use quasimetric structure and conservatism to generalize to unconnected state-goal pairs.

There were concerns about completeness of the ablations, experiments, and incomplete definitions in the theorems. However, the authors provided additional results and clarifications in the rebuttal that assuaged most of the concerns. I recommend the authors make the suggested edits in the camera ready.
Decision: Accept (poster)