Goal-conditioned Offline Planning from Curious Exploration
Authors: Marco Bagatella
Georg Martius
Venue: NeurIPS 2023 poster (2023)
Keywords: deep reinforcement learning
unsupervised reinforcement learning
goal-conditioned reinforcement learning
model-based planning
TL;DR: We study and address a class of value artifacts when learning from the outcomes of a curious exploration phase.

Abstract: Curiosity has established itself as a powerful exploration strategy in deep reinforcement learning. Notably, leveraging expected future novelty as intrinsic motivation has been shown to efficiently generate exploratory trajectories, as well as a robust dynamics model. We consider the challenge of extracting goal-conditioned behavior from the products of such unsupervised exploration techniques, without any additional environment interaction. We find that conventional goal-conditioned reinforcement learning approaches for extracting a value function and policy fall short in this difficult offline setting. By analyzing the geometry of optimal goal-conditioned value functions, we relate this issue to a specific class of estimation artifacts in learned values. In order to mitigate their occurrence, we propose to combine model-based planning over learned value landscapes with a graph-based value aggregation scheme. We show how this combination can correct both local and global artifacts, obtaining significant improvements in zero-shot goal-reaching performance across diverse simulated environments.

Goal-conditioned Offline Planning from Curious Exploration
Marco Bagatella
ETH Zürich & Max Planck Institute for Intelligent Systems Tübingen, Germany
abagatella@ethz..c
Georg Martius
University Tübingen & Max Planck Institute for Intelligent Systems
Tübingen, Germany
Abstract
Curiosity has established itself as powerful exploration strategy in deep reinforce- ment learning. Notably, leveraging expected future novelty as intrinsic motivation has been shown to efficiently generate exploratory trajectories, well robust dynamics model. We consider the challenge of extracting goal-conditioned behav- ior from the products of such unsupervised exploration techniques, without any additional environment interaction. We find that conventional goal-conditioned reinforcement learning approaches for extracting value function and policy fall short in this difficult offline setting. By analyzing the geometry of optimal goal- conditioned value functions, relate this issue specific class of estimation artifacts in learned values. order to mitigate their occurrence, we propose to combine model-based planning over learned value landscapes with graph-based value aggregation scheme. We show how this combination can correct both local and global artifacts, obtaining significant improvements in zero-shot goal-reaching performance across diverse simulated environments.
Introduction
While the standard paradigm in reinforcement learning (RL) is aimed maximizing a specific reward signal, the focus of the field has gradually shifted towards the pursuit of generally capable agents [1, 20, 40]. This work focuses on a promising framework for learning diverse behaviors with minimal supervision, by decomposing the problem into consecutive phases. The first phase, referred to as exploratory [28, 43] or intrinsic [38], consists unsupervised exploration of an environment, aiming collect diverse experiences. These can be leveraged during the second phase, called exploitatory or extrinsic, to distill family of specific behaviors without further environment interaction. This paradigm unifies several existing frameworks, including zero-shot deep reinforcement learning [28, 43], Explore20flline [23], EXORL [52] and model-based zero-shot planning [38].
An established technique for the intrinsic phase is curiosity- exploration, which leverages some formulation surprise as a self-supervised reward signal. practice, this often done by estimating the uncertainty of forward dynamic predictions of learned model [49]. At the end of the exploratory phase, this method produces a sequence of collected trajectories, as well as trained dynamics model. At its core, our work focuses on the subsequent extrinsic phase, and is concerned with extracting strong goal-conditioned hehavior from these two products, without further access to the environment.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Exploration
Learning
2 Planning for local artifacts
3 Aggregation for global artifacts
Dataset
gs:
(s; g)
V(s; g)
1
3
s
Dyuamics Model
P
P
Figure 1: This work focuses on extracting goal-conditioned behavior from exploratory trajectories 'D and a trained dynamics model P. We propose to learn a value function through goal-conditioned RL, plan with zero-order trajectory optimization to mitigate local estimation artifacts, and aggregate value estimates along graph in order to correct global estimation artifacts.
In its simplest form, this corresponds to learning to control the agent's future state distribution in order achieve given goal or, more concretely, visit particular state. Existing methods tackle this problem without additional learning through model-based online planning [38]. or rely on learning an amortized policy, either during exploration [28, 43] or fully offline manner [23, 52]. The former class of approaches requires the definition of hand-crafted cost function, without which long-horizon, sparse tasks would remain out of reach. On the other hand, the latter class does not rely a well-shaped cost function, but may suffer from myopic behavior in the absence of explicit planning, and from estimation errors the learned value function.
As we do not assume access to external task-specific cost functions, we first investigate the effective- ness of goal-conditioned RL algorithms for learning value function. In doing so, we empirically extend the results obtained by Yarats et al. [52] to previously unstudied model-based and goal- conditioned settings. We thus confirm that RL methods not involving offline corrections can be competitive when trained on offline exploratory data. Additionally, by formal analysis of the geometry optimal goal-conditioned value functions, we pinpoint geometric feature of inaccurate value landscapes, which we refer to as Tlocaal optima. After showing the detrimental effects of such learning artifacts to the performance of goal-conditioned RL, we propose to mitigate them through a novel combination of model-based planning with practical graph-based aggregation scheme for learned value estimates. Crucially, we show how these two components address the same estimation issue on different scales, that is, respectively, locally and globally. We provide graphical overview in Figure for simple maze environment.
The contributions of this work can be organized as follows:
We provide empirical evaluation of goal-conditioned reinforcement learning algorithms when trained on the outcomes of an unsupervised exploration phase.
We pinpoint class of estimation artifacts in learned conl-conditi value functions, which explains the suboptimality of naive, model-free reinforcement learning approaches this setting.
We propose to combine model-based planning with graph-based value aggregation address the occurrence of both local and global value artifacts.
We provide the first fully offline method for achieving goals after a curious exploration phase.
After the relating our work to existing literature, we evaluate coal-conditioned RL methods when learning from curious exploration (Section 4). We then pinpoint class of estimation artifacts that not compatible with optimal goal-conditioned value functions and analyze them through the lens of Tllocal optima (Section 5). Equipped with these insights, show how our practical method can mitigate such artifacts (Section 6). A compact description of the resulting method, highlighting its intrinsic and extrinsic phases, be found in Appendix H.
2
Related Work
Unsupervised Exploration The challenge of exploration has been at the core of reinforcement learning research since its inception. Within deep reinforcement learning, successful strategies

---

often derived from count-based [2]. prediction-basse [41] or direct exploration techniques [10], with the first two traditionally relying on intrinsic motivation [7] computed from state visitation distributions or prediction errors (i.e., surprise), respectively. Depending on the entity on which prediction error is computed, methods can focus on estimating forward dynamics [32, 38, 43, 44), inverse dynamics [33] or possibly random features of the state space [3]. Another important distinction falls between methods that retroactively compute surprise [33] or methods that compute future expected disagreement [38, 43], and proactively seek out uncertain areas in the state space. Overall, such techniques compute an additional intrinsic reward signal, which can be summed to the extrinsic environment reward signal, used alone for fully unsupervised exploration. In this work, build upon efficient exploration methods that are proactive, and rely on prediction disagreement of a forward dynamics model [38, 43, 44J. Nevertheless, our method could be easily applicable in conjunction with any aforementioned strategy, as long as it can produce exploratory data, and a dynamics model, which could potentially also be learned offline.
Zero-shot Frameworks Unsupervised exploration techniques are capable of efficiently sampling diverse trajectories, without extrinsic reward signals. As a result, they are a promising source of data to learn flexible behaviors. Several works in the literature have proposed that task-agnostic unsupervised exploration phase be followed by a second phase, optimizing a task-specific reward function only provided a posteriori. This has been formalized as zero-shot reinforcement learning [9, 17,7,,4 483. Notably, this framework conventionally forbids task-specific fine-tuning, explicit planning or computation during the second phase. On the applied side, agents endowed with learned world models [28, 43] have shown promise for zero-shot generalization without additional learning or planning. Other works are less strict in their constraints: Sancaktar et al. [38] also propose an intrinsic exploration phase followed by zero-shot generalization, but the latter relies on task-specific online planning. Furthermore, two concurrent works [23, 52] study the application of offline deep reinforcement learning for the second phase. However, these methods are not designed for goal- conditioned tasks, and thus require an external task-specific reward function to perform relabeling during training. Our framework fits within this general research direction, in that it also involves task- agnostic and task-specific components. The initial phase is task-agnostic: we rely on unsupervised exploration, and train universal value estimator [40] completely offline, without access to an external task-specific reward signal. In the second phase, we perform model-based planning via zero-order trajectory optimization to control the agent. We remark that, while this requires significant computation, existing algorithms are fast enough to allow real-time inference [35].
mondionndt Reinforcement Learning The pursuit of generally capable agents has been a ubiquitous effort in artificial intelligence research [36, 42, 45]. Goal-conditonned RL represents practical framework for solving multiple tasks, by describing each task through goal embedding. which can then condition reward and value functions. Deep RL methods can then be used to estimate values by providing the goal as an additional input to their (often neural) function approximators [40]. A ubiquitous technique in these settings is that of goal relabeling [1], which substitutes trajectory's goal with one sampled a posteriori. Goal-conditioned methods have been shown to be successful in the offline setting [46], often when coupled with negative sampling methods [5], but have been mostly evaluated on standard offfine datasets. An alternative line of work frames the problem as supervised learning instead [11], and shows connections the original reinforcement learning objective [51] Additionally, several works have explored the combination of goal-conditioned learning with model- based planning [4, 25, 31]. This paper studies the natural application of goal-conditioned RL after an unsupervised exploration phase, which allows straightforward definition of the multiple tasks to be learned. Moreover, our main formal insights in Section build on top of common assumptions in goal-conditioned RL.
Graph-based Value Estimation Several existing works rely on graph abstractions in order to retrieve global structure and address value estimation issues. In particular, Eysenbach et al. [12] and Savinov et al. [39] propose high-level subgoal selection procedures based on shortest paths between states sampled from the replay buffer. Another promising approach proposes to build graph during training, and to leverage its structure when distilling a policy [30]. In contrast, our method relies on discrete structures only at inference, and directly integrates these structures into model-based planning, thus removing the need for subgoal selection routines.
3

---

3 Preliminaries
3.1 Background and Notation
The environments of interest can be modeled as Markov decision processes (MDP) .M
{S,A, P. Rt po, 7). where S and A are (possibly continuous) state and action spaces, P: : S x A +
((S) is the transition probability distribution 1, R: 8 A R is reward function, po ((S) is
an initial state distribution, and 7 is discount factor.
Gond-conditioned reinforcement learning augments this conventional formulation by sampling goal g ~ Hig, where Pg is a distribution over a goal space g. While goal abstraction is possible, for simplicity of notation we consider [ S (i.e., goal-conditioned behavior corresponds to reaching given states). The reward function is then defined as R(s, (:g) = R(s; g) = d(s.))
for some 2 O, and under some distance metric d(.). We additionally assume that episodes terminate when a goal is achieved, and the maximum undiscounted return within
episode is
thus one. It is now possible to define a condiconndt policy x : S x G + (2(A) and its Ep., 3 g), with 8; ~ P( a,-1), ae ~ (st:g). The
value function V ())(s))) objective of a goal-conditioned RL agent can then be expressed as finding a stochastic policy r argmax Egg-po.s-pp V* V=
When learning from curious exploration, we additionally assume the availability of a set D {50, do, $1, "1,..,,,,, ST, ): of N exploratory state-action trajectories of possibly different lengths Tú well access to estimate of the probabilistic transition function P Rs P, i.e., dynamics model.
3.2 Data Collection and Experimental Framework
As our contributions are motivated by empirical observations, we also introduce our experimental pipeline early on. Algorithms evaluated in four MDP instantiations, namely maze 1 rge and kitchen from D4RL [14], peatch__ustt from gymnasiun-robottcs [13] and an adapted p npad environment [18]. These environments were chosen drastically differ dynamics, dimensionality. and task duration. In particular, maze_large involves the navigation to distant goals in 2D maze by controlling the acceleration of point mass, kitchen requires controlling 9 DOF robot through diverse tasks, including manipulating a kettle and operating switches, fetch, Setch_push is manipulation task controlled through operation space control, and pinpad requires navigating a room and pressing a sequence of buttons in the right order.
For each enviromment, to ~22 hours of real-time interaction at 30Hz, which is an order of magnitude less than existing benchmarks [14, 24]. We stress that this amount of data is realistically collectible in the real world with minimal supervision, thus providing strong benchmark for data-efficient learning of general agents. Moreover, the nature of the data is significantly different from standard offline datasets, which often contain high-quality demonstrations [14] that can be distilled into policies through naive imitation.
collect 200k exploratory transitions by curious exploration, corresponding
The environment's 's dynamics are estimated through an ensemble of stochastic neural networks, similar to the one proposed in Chua et al. [8]. The intrinsic exploration signal is ensemble disagreement [38, 43, 49], as described in Appendix 1.2. All algorithms are evaluated by success rate, that is, the ratio of test episodes in which the commanded goal is achieved. Details can be found in Appendix I.
4 Value Learning from Curious Exploration
Curious exploration produces a learned dynamics model P and set D of trajectories, lacking both goal and reward labels. In order to achieve arbitrary (and potentially distant) goals at inference time, and considering that a task-specific cost function is not externally provided, it is natural to resort to learning and optimizing a value function. This section evaluates established methods for value learning and studies the quality of their estimates, thus motivating the analysis and method proposed in Sections 5 and 6.
We use !(S) to represent the space of probability distributions over S.

---

A standard approach to train goal-conditioned value function from unlabeled data involves two steps. The first step consists of a goal-relabeling procedure, which associates each training sample with goal selected posteriori, and computes self- self-supervised reward signal from it, as briefly explained in Section 4.1. The second step consists of extracting goal-conditioned value function by applying off-policy RI. algorithms on the relabeled data, analyzed in Section 4.2. While this work focuses on data collected through curious exploration, the methods described can also be applied to a general offline goal-conditioned reinforcement learning settings, for which provide minimal study in Appendix
4.1 Relabeling Curious Exploration through Self-supervised Temporal Distances
We follow the framework proposed by Tian et al. [46] to relabel exploration trajectories, and compute self-supervised reward signals by leveraging temporal distances, without assuming access to external task-specific reward functions. Training samples (st, Ot, 9t+-1) are sampled uniformly from the replay buffer D, and need to be assigned a goal and reward scalar r. Each training sample is assigned positive goal with probability Py. and negative goal otherwise. Positive goals sampled future states from the same trajectory: = 8t4T. where T ~ Geo(pgeo follows a geometric distribution with parameter Paeo- Negative goals are instead sampled uniformly from the entire dataset, while Tian et al. [46] rely on prior knowledge on the semantics of the state space to extract challenging goals. Rewards for all state-goal pairs are simply computed using the reward function r R[ (t51111 9) 1 1. This corresponds to minorizer for the class of reward functions R($ g) 1 Sg as setting
the reward is non-zero only for positive goals sampled with
0 effectively removes the necessity to define specific distance function d over state and goal spaces. We remark that while its extreme sparsity prevents its optimization via zero-shot model based planning, remains functional as reward signal for training value function in hindsight.
4.2
Value Estimation and Suboptimality
Now that sampling and relabeling procedures have been defined, we focus our attention on which methods can effectively retrieve conditondt value function V(s; g). In particular, evaluate off-policy actor-critic algorithms, which are in principle capable of extracting the optimal value func- tion (and policy) from trajectories generated by a suboptimal policy (as the one used for exploration). We thus ask the following questions:
1. Which off- off-policy actor-critic algorithum most suitable for learning goal-conditioned value function. purely offline, from data collected through unsupervised exploration?
2. How close to optimality is the learned value function?
To answer the first question, we consider several off-policy actor-critic algorithms, and evaluate the learned value function through the success rate of an agent which seeks to optimize it. The choice of algorithms should adhere to the offline setting, as well as the availability of an estimated dynamics model P. We thus evaluate TD3 [15] as default model-free actor-critic algorithm, and
Large_arge
pinpad
fetch_push
kitchen
0.4
0.35
0.5
0.30
0.8
Suocesss Rate
0.3
0.25
Shocess Rate
Rate
0.4
0.20
0.2
0.15
0.4
0.2
0.10
0.2
0.1
CRR MEPO MORO TD3
CRR МAРОМИРО TD3
CRR MBPO MOPO
TD3
CRR MBPO MOPO
TD3
Actor Network
Model-based Planning
Figure Performance of actor networks (blue) and model-based planning (orange) on value functions trained through various actor-critic algorithms from the same curious exploration datascts.
5

---

compare its performance to CRR [50], state-of-the-arr offline method [16]. We also evaluate two model-based methods, namely MBPO [19] and MOPO [53]. the latter of which introduces an uncertainty-based penalty for offline learnin.. Further value-learning baselines are presented in Appendix C. All methods have access to the same offline exploration dataset D; model-based method additionally query the dynamics model P order to generate training trajectories. Algorithm-specific hyperparameters were tuned separately for each method through grid search, and kept constant across environments. Each algorithm is allowed the same number of gradient steps (sufficient to ensure convergence for all methods). We implement two strategies for optimizing the value functions: querying the jointly learned actor network, and employing model- planning and using zero- order trajectory optimization (e.g., CEM [37]iCEM(35) with the learned model Í to optimize n-step look ahead value estimates [46]. Average success rates over seeds are presented with 90% simple bootstrap confidence intervals in Figure
To answer the first question, we consider the performance of the best optimizer (often model-based planning, in orange) for each of the value learning methods. We observe that simpler actor-criti algorithms (i.c., TD3) are overall competitive with the remaining value learning methods. MBPO, MOPO and CRR are not consistently better in the considered environments, despite involving offline corrections or relying on the learned model P to generate training trajectories. We hypothesize that this phenomenon can be attributed, at least partially, the exploratory nature of the data distribution, as exemplified in Figure 3. Additionally, we note that our findings and hypothesis are consistent with those reported by Yarats al. [52] for non-goal-connitionee tasks.
We then turn to examining the performance gap between actor networks and model-based planning. We observe tendency for model-based planning to outperform actor networks, with the largest improvements in fet setchh_psstt h,, as all learned policies similarly fail in this environment. While this stands true for almost all considered algorithms, CRR poses exception to the success of model-based planning for two environments, which we hypothesize is due to its reliance on weighted behavior cloning rather than on backpropagation through learned critic.
We remark that possible to show that, under common assumptions, policy which exactly optimizes an optimal goal-conditioned value function is guaranteed to achieve its goals (see Appendix A.2). The low success rate of actor networks, assuming their objective is well-optimized, is thus symptom of suboptimality for learned value func- tions. In Section 5, we argue how this suboptimality can be partially traced back to a class of estimation artifacts; moreover, we show how model-based planning is intrin- sically more robust to such artifacts than actor networks (see Section 1).
E
Figure 3: 200k transitions from the exploration dataset D (left), and from the standard offline dataset provided by D4RL (right). Unsupervised exploration leads to a more uniform state coverage, and eases out-of-disttibutton issues.
Due to its strong empirical performance, TD3 is adopted for value learning through the rest of this work. We how- ever remark that the analysis and the method proposed in Sections and 6 remain valid for each of the RL meth- ods considered. Further details on methods and results presented in this section are reported in Appendix I.
Pathologies of Goal-Conditioned Value Functions
The performance gap between model- planning and actor networks described in the previous section a symptom of estimation issues in learned value functions. In this section, we pinpoint a particular class of geometric features which are not compatible with optimal goal-conditioned value functions. Such features represent a significant failure case for an agent optimizing a learned value function, as they effectively trap it within manifold of the state space. Intuitively, this happens when the estimated value of the current state exceeds that of all other reachable states, despite the goal not being achieved.
We now formalize this intuition by defining such geometric features as T-local optima. We then show that 7 -local optima are not present in optimal goal-conditioned value functions, and are therefore

---

estimation artifacts. This justifies the method proposed in Section 6, which is designed to mitigate this class of artifacts.
Let us define the set operator T :P(S) - TP(S) that returns the set of states reachable in one step
from given initial set of states So C
T(So)
U {s/ E 3a E A: P(s' 80, a) > 0). SoES.
We now formally introduce class of geometric features, which refer T-local optima: Definition 5.1. state E S is a T-local optimum point for a value function (ss) if and only if max E'ES
(s')) > V(s*)= >
max S'ET((="MMM'>
V(s)).
Intuitively, a T-local optimum is hill in the value landscape over the state space, where proximity defined by the transition function of the MDP. The first inequality ensures that s* does not maximize the value function globally. The second inequality guarantees that, if there is an action à E AA: P(s* | a) = 1. this action would be chosen by greedily optimizing the value of the next-state (i.e.. argmax J ((s') aa), thus trapping the agent in s*. By extending the previous definition to the multistep case, we can also conveniently define depth as the minimum number of steps required reach a state with higher value (i.e., escape the T-local optimum). Definition 5.2. T-local optimum point s* has depth & 2 k is the smallest integer such that max V(s*) **ET*t(((se) ([s+))
(ss)..
Intuitively, T-local optimum of depth 1s dominates the value of states reachable in up to k steps.
These definitions prepare the introduction of a core obser- vation. Within the setting of cond-conditioned RI, T-local optima are not an intrinsic feature of the value landscape, but rather an estimation artifact.
Proposition 5.3. Given a goal 9 G. an optimal goal- conditioned value function (*:gg) has no T-local op-
Figure 4: Learned value as function of z-y coordinates in maze_ large. Choosing actions that maximize the value of the next state traps the agent in a corner, i.e. a T-local optimum point.
tima.
The proof follows directly from Bellman optimality of the value function under standard assumptions of goal- conditioned RL introduced in Section 3.1. We report it in its entirety in Appendix A.1.
We remark that, in principle, showing that state is T-local optimum point requires evaluating the value function over the support of the next state distribution, which would require access to the true dynamics of the MDP. Moreover, exhaustive evaluation is unfeasible in continuous state spaces. However, one can bypass these issues by instead estimating, rather than proving, the presence of T-local optima. Given a candidate T-local optimum point s, V(: g) can be evaluated (i) random states s, sampled from the exploration data D instead of the state space S and (ii) samples s' from the next state distribution, accessed through the dynamic model P(s" s, a) with ~ U(A). The maximum operators in Definition 5.1 can then be estimated through sampling; if both inequalities hold, can be estimated to be a -lcaa optimum point.
Having defined T-local optima, and shown that they are learning artifacts, we now investigate their impact on goal-reaching performance. We train n = 30 bootstrapped value estimators for all environments, and for each of them we generate trajectories through model-based planning with horizon H. We then estimate the occurrence of T -loaal optima of depth A: 2 H by testing each state along each trajectory, as described above. This is reported in Figure 5, separately for successful and unsuccessful trajectories. We observe that, in most enviromments, trajectories which avoid -loca optima generally tend to be more likely to succeed; this tendency tends to fade in environments which only require short-term planning (e.g., fetch hutchh_hhhh.... We note that these estimates are sampled- based, and thus noisy by nature, especially in low-data regimes; we thus expand on this analysis in Appendix B. For illustrative purposes, we also visualize -local optimum in the value landscape over ma ze maaze_la large Figure 4.
P)p()) represents power set.
7

---

naze ___arge
pinpad
Ietch_push
xitchen
Occunence of (T-lcaal Optima
0.04
Occunence of 7-local Optina
0.14
Occunence of (T-lcall Optina
Occurrence of T-local Optima
0.5
0.12
0.03
0.15
0.30
0.02
0.08
0.10
0.3
0.06
0.2
0.01
0.05
0.02
0.00
0.00
0.00
0.0
Fiilure
Succcss
Fielare
Failace
Sucocss
Fillare
Suooess
Figure 5: Occurrence of T- optima of depth 2 H along successful and unsuccessful trajectories generated through model-based planning with horizon H 15, estimated via sampling. An horizontal bar marks the median.
Mitigating Local and Global Value Artifacts
Having shown that T-local optima are estimation artifacts, we now focus on categorizing and mitigating their occurrence. A first observation, which confirms the results in Section 4, is that model-based planning is robust to T-local optima of small depth. This further argued in Section 6.1. However, this does not address the occurrence of global estimation artifacts (i.e. T-local optima of large depth). We thus propose graph-based value aggregation scheme for this purpose in Section 6.2. Finally, in Section 6.3 we empirically show how combining these two components can address estimation artifacts both locally and globally, and consistently improves goal-reaching performance.
6.1
Model-based Planning for Local Value Artifacts
We will now provide further arguments for the effectiveness of model-based planning over inaccurate value landscapes, supported by Figure 2. Model-based planning a model-predictive control (MPC) algorithm, which finds an action sequence (ao. aH-1) by optimizing the RL objective through H H-step value estimate for a given horizon H, dynamics model P, state 80, and goal g:
H-1
(4o, -H--))=
argmax (ag.* ag 1)EAA
r R(ss,g) ( (V(HH:)) with 8:41 P()): (2)
1=0
and executes the first action ag- We remark that in our setting, due to the extreme sparsity of R(s;g) = 1s=g in continuous spaces, the objective in Equation 2 simplifies to the maximization of V(SH: g) alone. For the same reason, it is easy to show that an actor network trained through policy gradient is optimizing the same objective model- planning with horizon H 1:
40 = argmax Q(50, ao: g) = argmax R(so: g) + (s11g)) = argmax V(s1g)) a.EAA
(3)
agEA
a.EA
with 81 ~ P(-|su, ag). Under certain assumptions, such as exact dynamics estimation (i.e., P= P), exact optimization for Equation 2 and existence of self-loops in the MDP, it is possible to show that (i) model-based planning from a -locaa optimum point depth and (ii) model-based planning is guaranteed to reach state with a higher value if the horizon H is greater than the depth of the largest -loca optima over the value landscape. For a complete argument, Appendix A.3. As consequence, actor networks be trapped in all local optima, while model-based planning with horizon H can potentially escape local optima of depth & < H, and is therefore potentially more robust to such estimation artifacts, as shown empirically in Figure 2.
guaranteed to fail
S
6.2 Graph-based Aggregation for Global Value Artifacts
While we have shown that model-based planning robust to local artifacts, the correction of global estimation artifacts, which span beyond the effective planning horizon H, remains a significant challenge. We propose to address by grounding the value function in states observed during exploration, such that long-horize value estimates computed by aggregating short-horizon (i.c.,

---

local) estimates. Given the current state s E S and the goal 9 E G, we build sparse graph by sampling vertices from the empirical state distribution D, from the replay buffer, and connecting them with edges weighted by the learned goal-conditioned value V:
G (V,E) 8) with C D. {s, 9) and {(s', s" (ss ( VVVVV,
(4)
Crucially, we then prune the graph by removing all edges that encode long-horizon estimates, i.e., edges whose weight is below a threshold Vinin- In practice, we set Vmin to the maximum value which still ensures connectivity between the agent's 's state s and the goal along the graph. This can be computed efficiently, as detailed in Appendix D. Once the graph has been constructed and pruned, we simply compute value estimates by minimizing the product of value estimates along all paths to the goal.
T-1 II Vis; sc+1) where (su...., 8T) is path between and g on (5) (S0....AT))
V(sgg)= = min
This value aggregation scheme differentiates itself from existing methods [12] in its vertex sampling phase, which relies on density estimation to address skewed state distributions, and in its pruning phase, which does not depend on hyperparameters. Crucially, the final value aggregate can be directly adapted as cost function for model-bases planning, and not used as a proxy for subgoal selection. These differences and their consequences are described extensively in Appendix D.
In practice, this scheme can mitigate global T-local optima by replacing long-horizen value estimates through an aggregation of short-horizon estimates. Such short-horizon estimates have been shown to be generally more accurate, and useful in containing value overestimation issues [12], which can be considered a generalization of T -local optima.
For reference, we now provide visual intuition for the effect of the graph-based value aggregation scheme we propose. We highlight this in maze_large, as prone to global estimation arti- facts, and allows straightforward visualization. The blue column in Figure plots value functions {V(s; 9s) Y learned through TD3 against x-y coordinates for three different goals. The green column reproduces the same plots, but estimates values through graph- based value aggregation, as described in this Section, resulting in corrected estimates (V(s; 9: H We further compute expert trajectories for all three goals, in red, and plot V and V along them, resulting the three plots on the right. When comparing value estimates with optimal values (V*, in orange, computed in closed form), we observe that graph-base based aggregation results in more accurate estimates. Crucially, is able to address global estimation artifacts, which result in a "bleeding" effect for Vss; go) (on the left), or in large "hills" in Vss: go) (on the right, in blue).
R9
PL
V*a go)
06-
0.4
08-
100
130
209
R
R9
06-
0.4
(x
((:02)
100
233
R9
69
0.
08-
Figure 6: Learned values (15 g:) as a function z-y coordinates in maze Laaze_laarge (blue column). Graph-based value aggregation results in corrected estimates V(;gg, (green column). Values along optimal trajectories are compared to optimal value functions / (6)) on the right.

---

maze_large
pinpad
fetch_puak
kitchen
0.7
0.4
0.4
0.6
Succoss Rate
0.6
0.3
0.4
0.2
0.3
0.2
0.2
0.1
0.2
0.0
0.0
0.0
TD3
TD3+MBP
TDS+MBPPAAgergattoo
Shaped Cost+MBP
Figure 7: Performance of actor network (TD3) against model-based planning without (TD3+MBP) and with wraph-based value aggregation (TD3 MPDS++MAgggggatiion Model-based planning on aggregated values addresses both local and global estimation artifacts, and performs well across diverse environments.
6.3 Experimental Evaluation
Having evaluated diverse value learning algorithms, and described a class of harmful estimation artifacts, now set out evaluate how such artifacts can be addressed both locally and globally. Figure 7 compares the performance of actor networks with that of model-based planning, with and without graph-based value aggregation. We observe that model-based planning improves performance in most tasks, as it addresses T -local optima of limited depth. However, it fails to correct global artifacts in long-horizon tasks (i.e., maze maze_large and pinpac).. In this case, we observe that graph-based value aggregation increases success rates significantly. Furthermore, we observe that our method, which combines these two components, consistently achieves strong performance across diverse environments.
In each plot, dashed line represents the performance of model-based planning with external sparse cost (for some e > 0). On one hand, its strong performance in environments with shorter optimal trajectories confirms the accuracy of the learned model. On the other, its failure for long-horizon tasks shows the limitation of planning in the absence of a learned value function, which has been an important motivation for this work.
7 Discussion
This work is concerned with the challenge of achieving goals after unsupervised exploration phase, without any further interaction with the environment. In this challenging setting, we observe that simple actor-critic methods are surprisingly competitive with model-based or offline algorithms, but their actor networks are often suboptimal due to estimation artifacts in the learned value function. After characterizing these artifacts by leveraging the coaliconditooned nature of the problem, show how model-based planning can address this problem locally, graph- based value aggregation can apply global corrections, and their combination works consistently across diverse environments.
Limitations With respect to simply querying actor networks, this method comes at a cost. In particular, it increases computational requirements and risks introducing further artifacts due to model exploitation, or the discretization involved in graph-based aggregation (see Appendix D). Furthermore, the proposed method is only applied at inference and, while itss able to mitigate learning artifacts, does not directly correct them by updating the value estimator. Nevertheless, within our settings, we find the impact of these limitations to be, in practice, modest.
Our method represents strong practical approach for goal- noal-nea after curious exploration, and sheds light on interesting properties of both offline and goal-conditee conditioned problems. An exciting future avenue would further leverage these formal insights, possibly through a tighter integration of value learning and value correction. We thus believe that further exploring the interplay between theoretically sound analysis of the properties of goal-conditioned value function, and the design of practical algorithms, represents rich research opportunity.
10

---

Acknowledgments and Disclosure of Funding
We thank Andreas Krause, Nico Gürtler, Pierre Schumacher, Núria Armengol Urpí and Cansu Sancaktar for their help throughout the project, and the anonymous reviewers for their valuable feedback. Marco Bagatella is supported by the Max Planck ETH Center for Learning Systems. Georg Martius is member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 011S18039B).
References
[1] M. Andrychowiez, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abheel, and W. Zaremba. Hindsight experience replay. Advances in Neural Information Processing Systems, 2017.
[2] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, 2016.
[3] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. In International Conference Learning Representations, 2019.
[4] H. Charlesworth and G. Montana. Plangan: Model-based planning with sparse rewards and multiple goals. In Advances in Neural Information Processing Systems, 2020.
[5] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach, R. C. Julian, C. Finn, al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In International Conference on Machine Learning, 2021.
[6] X. Chen, A. Ghadirzadeh, T. Yu, J. Wang, A. Gao, W. Li, Bin, C. Finn, and C. Zhang. Lapo: Latent-variable dddvanttae-weighheee policy optimization for offline reinforcement learning. Advances Neural Information Processing Systems, 2022.
[7] N. Chentanez, A. Barto, and S. Singh. Intrinsicllly motivated reinforcement learning. Advances in Neural Information Processing Systems, 2004.
[8] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 2018
[9] P. Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5(4):613-224 1993.
[10] A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return, then explore. Nature, 590(7847):580-558 586, 2021.
[11] S. Emmons, B. Eysenbach, Kostrikov, and S. Levine. Rys: What essential for offline RL via supervised learning? In International Conference on Learning Representations, 2022.
[12] B. Eysenbach, R. R. Salakhutdinov, and S. Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems, 2019.
[13]
Gymnasium-robotics
https://github. com/
F Farama-Fa -Fonnaat ion Gaon///mnaaaaa -uluumu-oooo Robotics 2023.
[14] Fu, A. Kumar, O. Nachum, Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[15] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference Machine Learning, 2018.
11

---

[16] N. Gürtler, S. Blaes, P. Kolev, F. Widmaier, M. Wuthrich, S. Bauer, B. Schölkopf, and G. Mar- tius. Benchmarking offline reinforcement learning on real-robot hardware. In The Eleventh International Conference on Learning Representations, 2023.
[17] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018.
[18] D. Hafner, K.-H. Lee, I. Fischer, and P. Abbeel. Deep hierarchical planning from pixels. In Advances in Neural Information Processing Systems, 2022.
[19] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 2019.
[20] L. P. Kaelbling. Learning to achieve goals. In IJCAI, volume 2. pages 1094 8, 1993.
[21] D. P. K. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[22] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
[23] N. Lambert, M. Wulfmeier, W. Whitney, A. Byravan, M. Bloesch, V. Dasagi, T. Hertweck, and M. Riedmiller. The challenges of exploration for offline reinforcement learning. arXiv preprint arXiv:2201.1188 2022.
[24] M. Laskin, Yarats, H. Liu, K. Lee, A. Zhan, K. Lu, C. Cang, L. Pinto, and Abbeel. URLB: Unsupervised reinforcement learning benchmark. In Deep RL Workshop NeurIPS 2021, 2021.
[25] J. Li, C. Tang, M. Tomizuka, and W. Zhan. Hierarchical planning through goal-conditioned offline reinforcement learning. IEEE Robotics and Automation Letters, 2022.
[26] T. P. Lillicrap, J. J. Hunt, A. Pritzel, Heess, T. Erez, Tassa, D. Silver, and Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[27] K. Lowrey, A. Rajeswaran, Kakade, E. Todorov, and Mordatch. Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control. In International Conference Learning Representations (ICLR), 2019.
[28] R. Mendonca, O. Rybkin, K. Danilidis, D. Hafner, and D. Pathak. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems, 2021.
[29] R. Mendonca, S. Bahl, and D. Pathak. Alan: Autonomously exploring robotic agents in the real world. arXiv preprint ar X33:2332.00600 2023.
[30] L. Mezghani, S. Sukhbaatar, P. Bojanowski, A. Lazaric, and K. Alahari. Learning goal- conditioned policies offline with self-supervised reward shaping. In 6th Annual Conference Robot Learning, 2022.
[31] S. Nasiriany, V. Pong, S. Lin, and S. Levine. Planning with goal-conditioned policies. Advances in Neural Information Processing Systems, 32, 2019.
[32] P.-Y. Oudeyer, Kaplan, and V. V. Hafner. Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation, 11(2):265-22) 286, 2007.
[33] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. International Conference on Machine Learning, 2017.
[34] L. Pineda, B. Amos, A. Zhang, N. O. Lambert, and R. Calandra. Mbrl-lib: A modular library for basel reinforcement learning. Arxiv, 2021.
[35] C. Pinneri, S. Sawant, S. Blaes, J. Achterhold, J. Stueckler, M. Rolinek, and G. Martius. Sample- efficient cross-entropy method for real-time planning. Conference on Robot Learning, 2021.
12

---

[36] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-maron, M. Giménez, Sulsky, J. Kay, T. Springenberg, T. Eccles, J. Bruce. A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent. Transactions on Machine Learning Research, 2022.
[37] R. Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1(2): 1999.
[38] C. Sancaktar, S. Blaes, and G. Martius. Curious exploration via structured world models yields zero-shot object manipulation. In Advances in Neural Information Processing Systems. Curran Associates, Inc., Dec. 2022.
[39] N. Savinov, A. Dosovitskiy, and V. Koltun. Semi-parametric topological memory for navigation. In International Conference on Learning Representations, 2018.
[40] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International Conference on Machine Learning, 2015.
[41] J. Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international Conference simulation of adaptive behavior: From animals to animats, pages 222-227, 1991.
[42] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604-009 2020.
[43] R. Sekar, O. Rybkin, K. Danilliiis, P. Abbeel, D. Hafner, and D. Pathak. Planning to explore via sule-suue world models. In International Conference on Machine Learning, 2020.
[44] B. C. Stadie, S. Levine, and Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. In International Conference Learning Representations, 2016.
[45] O. E. L. Team, A. Stooke, A. Mahajan, C. Barros, C. Deck, J. Bauer, J. Sygnowski, M. Trebacz, M. Jaderberg. M. Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint 1rX.v:2107.128000
2021.
[46] S. Tian, S. Nair, F. Ebert, S. Dasari, B. Eysenbach, C. Finn, and S. Levine Model-based visual planning with self-supervised functional distances. In International Conference on Learning Representations, 2021.
[47] A. Touati and Y. Ollivier: Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 2021.
[48] A. Touati, J. Rapin, and Y. Ollivier. Does zero-shot reinforcement learning exist? In 3rd Offline RL Workshop: Offline RL as a "Launchpad", 2022.
[49] M. Vlastelica, S. Blaes, C. Pinneri, and G. Martius. Risk-averse zero-order trajectory optimiza- tion. In 5th Annual Conference on Robot Learning, 2021.
[50] Z. Wang, A. Novikov, K. Zolna, J. S. Merel, J. T. Springenberg, S. Reed, B. Shahriari, N. Siegel, C. Gulcehre, N. Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 2020.
[51] R. Yang, Y. Lu, W. Li, Sun, M. Fang, Y. Du, Li, L. Han, and C. Zhang. Rethinking goal- conditioned supervised learning and its connection to offline RI. In International Conference on Learning Representations, 2022.
[52] D. Yarats, D. Brandfoobrener, H. Liu, M. Laskin, P. Abbeel, A. Lazaric, and L. Pinto. Don't ' change the algorithm, change the data: Exploratory data for offline reinforeement learning. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World, 2022.
[53] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model- based offline policy optimization. Advances in Neural Information Processing Systems, 2020.
13

---

Supplementary Material to Goal-conditioned Offline Planning from Curious Exploration
A Proofs
A.1 Proposition 5.3
Let us consider an arbitrary goal g E 9 and the optimal goal-conditioned value function (s;s; g). Proposition 5.3 states that (s g) has no T-local optima. The proof follows directly from the Bellman optimality condition, assuming that (1) R(s;g) g) =d(s.)) for some 2 and distance function -), and that (2) the episode terminates when the goal is achieved.
For candidate state 8, us assume that the first inequality of the condition for being a T-local optimum point is met: max (s(:sg > V=(=g)) It is now sufficient to show that the second
max inequality (i.e., (* (
V*(s'gg) cannot hold, and therefore & cannot be a T-local optimum point. According to Bellman optimality, we have that
V (3; g) R(8;g) + (B;gg) max E aed d'mP(-la.3)
(*(:ig)
(S6)
where we use ((:g) = 7(1 ((s;)(())))))))))))))))))))))R (1 R)s;g) acts as the termination signal, to enforce
that value propagation stops when reaching the goal (and the episode ends). If R(5; g) then V*(5g) g) = 1, which would break the first inequality, since max V (s':g)
I 1. This implies that
x'ES
R($: g) = O. As consequence, Equation S6 simplifies
g5: max E acd s'MP(Ha.5))
V"(s':9)
(S7)
S max (EE((18)
V (s::g)
We now differentiate two cases. If max (=(5': 0 (i.e., the goal cannot be reached), then also have that V ($: ((
max
(s's': = 0. and is not Tolocal optimum point.
If the goal instead reachable from at least one state in the next state distribution (i.e., max)) V (s):g)) >> 0), we have that
(*( S ((:
V*(l'g
max NET((3))
V"($'g
max (EE((())
(S8)
V*($':9)-
(E
The second inequality follows from the goal-conditioned assumption that 0 < < 1, and the max V*(s'gg) implies that V* g) *
final equality is due to the fact that V V* g)
e'ET((8))
max (*(:'g)). Thus, (5: ((E)(())) optimum point.
max ((TE(M))
('':g)) does not hold, and is not Tloccal
A.2 Optimization of Optimal Goal-conditioned Value Functions
Section 4 argues that, assuming that the objective for actor networks is well optimized, the failure of naive actor-critic algorithms is due to estimation errors in the critic network. Accordingly,
'While supremum operator sup(-) would be required in open domains, for simplicity use max(.) throughout this work.
14

---

we show that, under common assumptions, state-space trajectory which optimizes an optimal goal-conditioned value function V* (s g) is guaranteed to achieve its goal g G. Furthermore, minimization of the actor loss results in an actor network which produces this exact trajectory.
Given goal 9 9, and an initial state so E S, we make the following assumptions
1. (*(s g) > 0, i... the goal can be reached from the initial state.
2. For all s E S, a E A. 3 s' : P(s'ls, a) = 1, i.c., the MDP transitions deterministically.
We now consider the state-space trajectory (st (si) which optimizes V under the dynamies of the MDP, that is
8141 argmax V* VOSi<T.
(S9)
((E)(()
We will show that, for certain finite T, ST achieves the goal, i... R(ST: 1 and R(s;; g) 0 < < T. We note that, since the episode terminates when the goal is achieved and (s:g) y(1 R)s; Bellman optimality implies that R(sr:g) = (*(srig) = 1. as shown in Equation $6.
From Equation S6 we can then derive
V (s; R(ss 9) + max
E
V'(s':g)
aEd (NPPaa.))
=ymax aEA s/MP(aa,))))
V*(s':g)
(S10)
= =V max (s':gg =y (E
>VV*(s++1g))
where the third equality follows from deterministic transitions, and the last from the objective in
Equation S9, By applying this equality recursively, we can write V* (s(gg g) = y V (so: g). After T = [log,, (*(soi [)] steps,
(*(sT:g) =7 > ( " 1.
(S11)
Since V* (::g) S therefore V' (*T: = and 87 achieves the goal.
An actor network is trained to output
((; g) argmax Q" (s,, a;g)
aEd
argmax R(s;; g) +
E
V*(8'ig)
(S12)
aeA
s(mP((aa.)))
argmax
E acA 8'-P(-)a.s.)
V*(8'g)
Under deterministic dynamics, the optimal policy is therefore producing the trajectory which opti- mizes the objective in Equation S9, and thus reaches the goal.
A.3 Model-based Planning and -local Optima
Section 5 pinpoints T-local optima class of estimation artifacts in learned value functions. Following from Section 1, we now present two properties arising from the interaction between model-based planning, and suboptimal value landscape, depending the depth of Tloccal optima, and the planning horizon.
First, we show that, given a goal G. model-based planning with horizon H from T-local optimum point s* of depth Section 3. and additionally assume:
fails to achieve the goal. We make the assumptions presented in
1. p i.e. exact dynamics estimation.
2. Model-based planning optimizes its objective exactly (see Equation S13).
*On top of those listed in Section 3.1.
15

---

3. For every state E S and goal G. (s: 2 ET(())(()) V(s:gg), then 3 a E A. P(s s, a) 1. This implies the existence of self-loops for a set of states including T -local optimum points.
We recall from Section 6.1 that, this setting, model-based planning optimizes the following objective:
H 00
argmax V(sHig) with 8,41 P(I)s,,)))
(S13)
EEAH
We note that, if 80 = is T-local optima, then by assumption (3) an action á is available such that Pss"ss" a) 1. If a; for 0 Si < H. then the objective in Equation S13 simply evaluates to V V(st:g) g). Any other action sequence will reach a state within the H-step future state distribution T (1s" " }), and will evaluate to lower equal value per the definition of T-local optima of depth k: :9) 2 max v(s':g). If ties are broken favorably an MPC model-basee based planning
#'ET(((="))
algorithm will therefore execute a, which deterministically transitions to the T-local optimum point at each step. Thus, the goal is never reached.
Second, show that, if the planning horizon H is greater than the depth of all T-local optima over the state space S, open-loop execution of the action sequence returned through model- planning successfully optimizes the value function (i.e., reaches a state with higher value).
This property holds under additional assumptions, namely:
4. For all s E S, E A. 3 E S P(s'ls, a) 1, i.e, the MDP transitions deterministically.
5. The action sequence optimizing the objective for model-based planning (Equation S13) executed in its entirety before replanning, i.e., open-loop execution of the plan.
6. V(s:gg < max V(8;; g). i.e., the initial state does not maximize the value function.
Since there are no Tloocal optima of depth k 2 H, then there must exist a state 81 E T (fso such that (sigg) V(so g) and HI' S H. Now, let consider the state reachable in up H steps with the highest value:
V(s1:))
max H'SH S'ET" ({so))
max V(s':g)
(S14)
As shown in Equation S13, model-based planning only optimizes over states that can be reached in exactly H steps. thus sufficient to show that 81 TT as this would ensure that
((snig) < v((1g)) v
max V(snig)=
= max V(sn:g) with 8,41 ~ P(-(s;. a;).
s ("so"((so))
d 1-1 Ee""
(S15)
Thus, open-loop execution of the action sequence d -1 optimizing Equation S13 would reach state with higher value (in this case, 8H)).
We recall that 8 E TH' (1so3) for some H' << H. We will now show that 8 E T (1 by contradiction. Let us assume that s1 & ((so))) We thus have that
V(sig) = max H'SH WET ((so)) ((so)
max V(sigg)
max
max (
V(s'gg)
H*HHSS
(S16)
max W/ HCH"SH WETH" (1s))
V(s':g)
IV
max V(s':g)
*ET((sH)
'Given two action sequences that both maximize the objective in Equation S13, we need to assume that the agent breaks the tie by maximizing the value of the first state in the trajectory V (s1;g)) This assumption would not be necessary in case of loop execution of the action sequence.
16

---

Following from assumption (4), self-loop is thus available. Therefore $1 E T((s1)). and more importantly $1 T({so))** for any H" 2 H²', thus contradicting the initial assumption and completing the proof.
B Occurrence of Artifacts and Downstream Performance
Sections pinpoints T-local optima as a class of estimation artifacts learned value functions. This section empirically validates the harmfulness of such artifacts for agent which attempts to optimize value function (s;g))
In order to so, we train n = 30 value functions for each environment. We then optimize each of them through model-based planning with horizon H for different goals, and obtain set of trajectories. We split all evaluation trajectories two groups: successes (those that achieve the goal and obtain return greater than 0) and failures. We then estimate the occurrence of T-local optima of depth 2 H along each trajectory, i.e., the ratio of states in the trajectory which are likely to be T-local optimum points, and plot its distribution for each group.
In practice, evaluating whether a candidate state E S is T-local optima is unfeasible, requires the evaluation of maximum operators over continuous spaces, A sampling-basee estimation of T-local optima remains possible, as described in Section but will necessarily introduce a significant amount of noise. Nonetheless, we present the results of this procedure in Figure S8 (identical to Figure in the main paper).
We note that the distributions of occurrences present long tails. We observe a general tendency for successful trajectories to contain fewer estimated T-local optima, the median occurrence is zero in three environments, and in any case never greater than the median occurrence for unsuccessful trajectories. This tendency is unsurprisingly the strongest in ma ze_ large, which observed to be prone to the occurrence of global estimation artifacts with depth k 2 H 15.
We additionally report analysis of the same trajectories with indirect procedure, which does not rely on sampling. Appendix A.3 argues that, under several assumptions, model-based planning with horizon H from T-local optimum point of depth & 2 H is incapable of reaching the goal, any state with a larger value. The reverse does not necessarily hold: lack of value improvement does not imply the presence of artifacts, but might result from suboptimal optimization procedure. Moreover, in practical settings, not all necessary assumptions hold. Nevertheless, it is still possible to compute the ratio of states 8; in evaluation trajectories for which V(s;gg 2 (ss+ Hig): this metric should tend to zero in the absence of estimation artifacts in the value function V(: g), assuming that the action selection procedure optimizes it correctly. We report similar plots for this metric, which we refer to as occurrence of non-monotonicities, in Figure $9. We observe that the patterns occurring Figure $8 arise more evidently, as non-monotonicities tend to occur less frequently in successful trajectories.
__large
pinpad
Ietch_push
kitchen
0.04
Occurrence of (T-local Optina
Occurence of T-local Optina
Occurrence of T --ocaal Optina
Occurrence of local Optima
0.5
0.03
0.10
00.15
0.4
0.02
0.08
0.10
0.3
0.06
0.2
0.01
0.04
0.05
0.1
0.02
0.00
0.00
0.0
Failure
Sucness
Failur
Sucoess
Failare
Sucoess
Fullure
Success
Figure S8: Occurrence of -local optima along successful and unsuccessful trajectories, estimated via sampling. An horizontal bar marks the median.
17

---

saaze_aagee
pinpad
fetch_pusk
kitchen
Occureence of Mon--Moattoniittee
Occurrence [Noo-Monetoniittae
Occureence of of Noon-Monttonicitiis
Occurrence of Non Moln-Moottoniittiss
0.5
0.5
0.4
0.3
000000 1000000
0.4
0.4
0.2
0.2
0.0
0.0
Failure
Failure
Figure S9: Occurrence of non-monotonicities along successful and unsuccessful trajectories. An
horizontal bar marks the median.
C Comparison to Additional Reinforcement Learning Algorithms
In this Section we benchmark three additional value-learning methods for offline goal-conditioned reinforcement learning, namely [QL[22], LAPO [6] and WGCSL [51]. Additional hyperparameters tuned with limited budget algorithms (actor network, in blue), and in combination with model-based planning (in orange). Results are averaged over 12 seeds and reported Figure $10. The three additional methods perform within the range of those evaluated in the main paper, suggesting that they do not avoid value estimation artifacts. Moreover, they do not consistently outperform TD3, confirming the competitiveness of simple actor-critic algorithms. Finally, we observe that these additional methods generally improve their performance when coupled with model-based planning, as shown in Section 4.
ze_ arge. Performances
reported for the original
D Comparison to Graph-based Methods
Discrete structures such as graphs have been explored in order to model long horizhn temporal distances over MDPs. Eysenbach et al. [12] have shown how shortest paths on graph constructed by uniformly sampling replay buffer can effectively be used to select subgoals in goal-conditioned RL, introducing Search on the Replay Buffer (SORB). Our graph-based value aggregation scheme generally follows this framework, but with several differences, which we find to be crucial when learning from exploration trajectories. this section, we expand the method description from Section 6.2, and outline these differences.
The first, and perhaps most crucial, difference lies in how shortest paths are leveraged guide goal-conditioned agent. In Eysenbach et al. [12], shortest paths are only used to retrieve the second state along them, which is then commanded to the agent subgoal. In practice, found this to potentially result in the selection of subgoals which are too close, and therefore cause myopic
maze_large
pinpad
fetch_push
kitchen
0.5
0.5
0.4
Success Rate
Success Rate
0.4
0.3
0.3
0.6
0.4
0.1
0.1
0.0
0.0
0.0
0.0
IQL
1ANO
WOCSL
CRR
MOPO
MOPO
TDN
IQL
1AAO
WOCSL
CRR
MOPO
TD3
JUL
LANO
WGCSL
CRR
MIUNO
MOPO
TD3
IQ0.
LAPO
WOCSL
CRR
MOPO
TD3
Actor Network
Model-based Planning
Figure S10: Extension of Figure 2 to additional value learning methods. We confirm that TD3
remains, average, competitive with offline and model-basee baselines when trained curious exploration data.
18

---

behavior in the agent. For instance, in maze large, the agent is encouraged to move at lower velocities to avoid overshooting its closest subgoal. While this could be addressed by conditioning the agent on subgoals further along the shortest path, this solution can be counterproductive, and force the agent to follow unreliable long-horizon value estimates. In contrast, we employ the entirety of the shortest path in order to compute oong-horizon estimates by aggregating short-horizon estimates along this path. Conveniently, aggregates can be directly optimized through model-based planning, without needing to command intermediate subgoals, or to retrain the policy to match corrected value estimates.
A second difference with respect to existing works [12] lies in the sampling procedure for vertices in the graph. Instead of adopting uniform sampling, we sample states according to the inverse of a learned density estimate. In practice, we use an Exponential Kernel Density Estimator with bandwidth h = 20, where the distance between two states is simply the temporal distance d(s,, s') = log, V(s; 8'). This makes the density estimates independent of the parameterization of the state space S, thus avoiding the strong assumption that Euclidean distances in the state space are meaningful.
Third and last, once the graph has been constructed, a pruning threshold Vminn is usually enforced to remove all edges encoding long-horizon distance estimates, which have low value estimates ss"; < Vuni- For this reason, in principle, this threshold should be as large as possible. However, an excessively high threshold would result in a disconnected graph, in which shortest paths may not exist. While this threshold is conventionally hand-tuned, we propose to select it automatically, by choosing the largest value Vinin that ensures connectivity from the agent's state to the goal. This can be efficiently computed through Dijkstra-like algorithm. We refer to our codebase for its implementation.
Having outlined three key differences with respect to previous methods, we now provide an empirical evaluation of their effects on downstream performance in Figure $11. All success rates reported are obtained by optimizing value functions learned through TD3 with model-based planning and default hyperparameters.
maag__laree
pinpad
fetch_push
kitchen
0.8
0.4
0.4
Success Rate
0.6
Sucoess Rate
Succoss Rate
0.6
Succoss Rate
0.3
0.4
0.2
0.2
0.2
0.1
0.2
0.1
0.0
0.0
0.0
0.0
Our aggregation
Our aggregation w/ uniform sampling
SORB w/ Vinin 75
SORB w/ Vmin = 10
SORB w/ Vmin =03
SORB Vnnn = -00
Figure S11: Ablation and comparison of graph- methods. Our value aggregation scheme largely outperforms naive application of SORB to this setting, while removing the need to tune the graph pruning threshold Vmin. We find that density-aware sampling of vertices in the graph is crucial for succeeding in long-horizon tasks.
In particular, compare our graph- value aggregation scheme (Our aggregation) with an ablation that samples graph vertices uniformly from the replay buffer. We observe that this results in sharp decrease in performance in long-horizon tasks. We also replace our value aggregation scheme with a SORB-like subgoal selection procedure for different pruning thresholds Vuin- We find that naive application of SORB to this setting is not effective, and can be detrimental with respect to simple model- based planning.
Removing the necessity to tune Vmin leaves a single hyperparameter to be chosen for graph- based value aggregation, namely the size of the graph. This number is limited by the complexity of the shortest path search, and thus use the default value of 1000 from the literature. However,
19

---

found 100 vertices to already achieve sufficient coverage in pinpad, which therefore adopts this reduced hyperparameter.
In terms of limitations, as the graph aggregation procedure involves learned value estimates, it remains prone to failure due compounding errors. However, two design choice introduce degree of robustness. First, value estimates are only computed over states and goals that appear in the dataset the value function was trained on, which alleviates OOD issues. Second, the pruning step removes all edges encoding long-horizon estimates, which are empirically found to be less accurate [12]. Moreover, the tuning procedure we propose for the pruning threshold effectively minimizes the number of paths in the graph, while ensuring the existence of a solution to the optimization problem in Equation 5. Despite this, for particularly sparse datasets, the pruning can fail to remove inaccurate estimates, and the graph-based aggregates may reproduce the artifacts of the underlying value estimator. Further limitations of the graph search include increasing computational costs, and artifacts due to the effective discretization of the state space: while increasing the sample size can address the latter, it aggravates the former in a tradeoff.
E
Sample Efficiency
All value functions in previous experiments are trained on 200k transitions gathered through curious exploration. This amount of data is around an order of magnitude lower than popular offline datasets [14], and has been chosen to be realistically collectable on real hardware. In practice, 200k timesteps at 30 Hz amount to little under 2 hours of real time interaction. Nonetheless, from Figure 7 (main paper) on smaller datasets, namely with 50k and 100k transitions. We visualize results in Figure S12, and find them to remain consistent, while success rates decrease on smaller datasets, as expected.
also evaluate methods
parge_
pinpad
fetch_push
kiten
Success Rate
0.4
0.75
0.4
0.3
0:25
02
0.0
100%
200
50k
100%
200
50
100
150
200k
100%
150
200
Exploration Steps
Exploration Steps
Exploration Steps
Exploration Steps
TD3
TD3+MBP
TD3+MBBP+Agrrgatoon
Figure S12: Performance of an actor network (TD3) against model-based planning without (TD3+MBP) and with graph-based value aggregation TDD+-MBP+Aggregation reported for value functions trained on datasets with 50k, 100k, and 200k samples.
F Applicability to General Offline Goal-Conditioned Settings
The motivation of this work is to address the challenge of extracting goal-conditioned behavior after curious exploration. Focusing on data gathered through unsupervised exploration has value in itself, arguably, as this requires minimal supervision and can enable real-world robot learning [29]. However, the proposed method can in principle be applied to general offline coal-conditioned settings. This section explores this possibility and provides empirical evidence.
Throughout this work, the exploratory nature of the data distribution is leveraged mainly in two points. First, argued in Section 4.2, when learning from curious exploration data, simple actor-critic methods are, on average, competitive with model-based or offline methods. In standard offline settings with near-expert data, offline algorithms should be expected be more effective [14]. However, within the realm of curious exploration, Section 2 suggests that TD3 is equally performant, and thus the proposed method uses it as core algorithm for value learning. Second, the proposed method relies on model-basee planning, whose performance is heavily dependent on the quality and robustness of the learned model. Model-based planning is thus particularly promising when learning from curious exploration, as the unsupervised exploration phase already produces dynamics model, which does not need to trained from scratch. Moreover, as data has been collected to maximize the information gain of this dynamics model, it is potentially more robust [38]. When learning from offline datasets, a model is generally not available: while can also be learned offline, training it to high accuracy can be difficult.
20

---

maze _aage
meze__ait naze_large
fetch_push
fetch_push
1.00
1.00
Success Rate
0.8
Success Rate
0.8
Success Rate
0.75
Rate
0.75
0.50
0.50
0.4
0.4
0.25
0.25
0.00
0.00
0.0
1.0
0.0
0.5
& Expert Duta (a)
Uniform Data
Expert Data
% Uniform Data
(b)
(c)
(d)
Dynamies model learned during exploration
Dynamics model learned offline
Figure S13: Ablation mixing the original dataset collected through curious exploration with expert or random uniform datasets. Our method can in practice be applied different data sources, although may it suffer from weaker exploration in long-horizon environments (c.g. maze _are___aagee..._ Environments with more complex dynamics (e.g. Fetch_pushh largely benefit from the dynamics model learned during exploration (with active data collection).
We provide minimal empirical study of the applicability of the proposed method different data sources in Figure S13. Each subplot presents the performance of our full method (TD3 + model-based planning graph-based value aggregation) when mixing the original curious exploration datasets with data collected from different sources (the size is fixed at 200k samples). In particular, a fraction of the trajectories is replaced with transitions from expert policy (a,c), and from uniformly random policy (b,d). Performance curves are recorded when using the dynamics model learned during curious exploration (green), and a dynamics model learned offline on the mixed data (violet); offline model learning was not tuned specifically. Due to time and compute constraints, we also present this analysis over representative environments (maze_larg) in (a,b) and fetch_push in (c,d)), and average over 9 random seeds.
In these settings, observe that access expert data does not hinder success rates (Figure S13a,c): interestingly, theproposed method shows robustness to shrinking data support. On the other hand, when introducing data collected by uniform random policy (Figure $13b,d), performance de- creases accordingly to the reduced exploration: while collecting random trajectories is sufficient in fetch_push, it is not comparable curiosity-driven exploration for long-horizon tasks (i.e., maze_largee Furthermore, we note that models trained offline (in violet) achieve generally success when the complexity of the environment's 's dynamics increases (fetch cht_cshh)) confirming that online training via disagreement maximization returns more robust model [38].
G Comparison to Value-driven Exploration
In this Section, we provide an informative comparison with related framework: POLO [27]. POLO also combines value learning and model-based planning. However, there are fundamental differences:
POLO explores by maximizing the uncertainty of the value function; our framework lever- ages the uncertainty of the dynamics model.
POLO assumes the availability of ground truth models for planning; our framework does not and adopts learned dynamic model.
POLO does not rely on graph-based components for long-horizon planning.
While official implementation of POLO is not available. can loosely reproduce it by integrating POLO's exploration signal in our framework, replacing TD3 with its value learning scheme, and disabling graph-based value aggregation, while still relying on iCEM for model-based planning and learning a dynamics model. We name the resulting method POLO*. We find that POLO* suffers from model inaccuracy when computing its value targets via MPPI optimization. We thus introduce variant which uses on-policy value targets, which we refer to as POLO**
Figure S14 reports success rates for POLO* and POLO** against our method (TD3 + model-based planning + graph-based aggregation) in all of our benchmark environments. Plots are averaged over 12 seeds, except for POLO* which uses 5 due to higher computational cost. Moreover,
21

---

large
pinpad
fatch fetch_puush
kitchen
0.80
1,00
0.50
0.80
0.40
Success Rate
0.40
Suocess Ratz
Success Rate
Success
0.40
0.30
0,40
0.20
0.20
0.20
0.20
0.00
0.00
*
TD3+ AD2+MMBPAAggregatioo
Figure $14: Comparison with adapted implementations of POLO a privileged baseline that value-driven exploration and online value learning.
plot exploration trajectories in maze_larga in Figure S15. We note that POLO privileged baseline, as it requires goals to be specified during training, and its value function is trained online by optimizing against its uncertainty. Our method, on the other hand, learns its value function offline from unlabeled data, and is therefore more widely applicable.
While the closer implementation (POLO*) is less performant, the improved version (POLO**) performs comparably or better than our method in three out of four environment. Despite successfully leveraging online value learning, we find potential for long-horizon exploration to limited, as shown by a large drop in performance in maze_ large. We confirm this by visualizing trajectories gathered by POLO** (value-driven), and by the unsupervised exploration phase used our work (dynamics-driven) in Figure S15.
H
Compact Algorithmic Description
We report the full method (TD3 + MDP + Aggregation) in a compact algorithmic description box S1, well in short verbal summary below.
Our work deals with extracting a goal-reaching controller from the outcomes of curious exploration. Exploration data is collected while learning a dynamic model ensemble by maximizing its disagree- ment through model-based planning with zero-order trajectory optimization. This constitutes the intrinsic phase, which returns exploration dataset, and a trained dynamics model. We then train value function offline on the exploration dataset with TD3. During inference, we are given a goal. We select actions through model-based planning with the learned dynamics model. This time, actions
Uniform
Valuo-driven
canamics-drived
steps
RA
200k steps
Random uniform, value-driven (as in POLO) and iinamics-driven (default in method) exploration in maze_large Lighter colors represent high exploration incentive. Uniform exploration is insufficient, and dynamics-driven exploration achieves dense coverage of the state space while being task-agnostic.
Figure S15:
22

---

sequences are selected by maximizing the aggregated value of the last state in imagined trajectories, where aggregation is performed with graph search over the value estimates from TD3.
Algorithm S1 TD3 + MDP + Aggregation
Intrinsic Phase 2: Collect a trajectory by maximizing the intrinsic reward Fint (S17, S18) with iCEM and the
1:
for each exploration episode do
dynamics model IP.
3: Add the trajectory the replay buffer D.
4: Train the dynamies model P via MLE over the replay buffer D.
5: end for
6: for each offline training epoch do
7: Train goal-conditioner value function V with TD3 on the replay buffer D, relabeled as
described in Subsection 4.1.
8: end for
9: for each evaluation goal do
D Extrinsic Phase
10: Build graph G as described Subsection 6.2.
11: Act by maximizing the aggregated value V (5, S19) with iCEM and the dynamics model P..
12: end for
Implementation Details
This section provides an in-depth description of methods and hyperparameters. Our codebase builds upon mbrl- -lbb [34], and adapts it to implement unsupervised exploration and goal-conditioned value-learning algorithms. To ensure reproducibility, we make publicly available
Figure S16: Environments used for evaluation: from left to right, faaeed_llarge pinpad, fetch_push and kitchen.
1.1 Environments
Empirical evaluations rely on four simulated environments. maze_ Laze_largee is adapted from D4RL. [14], with the sole modification of fixing the initial position of the agent in corner of the maze, in order to make exploration more challenging. The goal distribution includes 3 goals, placed on far-away corners of the maze, displayed in Figure 6 (main paper). pinpad is implemented on top of D4RL [14] as a continuous, state-based version of the environment introduced in Hafner et al. [18]. The agent is initialized in room with 4 buttons at its corners, and tasked with pressing 3 of them in the correct order. Observations are thus 7 dimensional, and include 2D position and velocity of the agent, as well as d-dimensional history of the last buttons pressed. The agent is controlled through acceleration, as in maze large. kitchen is adapted from gymnasium -nobotiirooooscsss [13] in order to allow coal-conditioning. The goal distribution includes controlling each of the seven objects individually: sliding door, cabinet, a light switch, two burners, teapot and a microwave oven. We use a frame skip value of 50 to encourage exploration. fetch_pust is one of the benchmarks presented in Andrychowicz et al. [1], and we use it in its unmodified form, provided in gymnasium [13]. A summary of further environment parameters can be found in Table S1.
*Code available sites.google.com/// wwwgllll...o//e/////o/pcccccccce
23

---

[
9
[A]
Episode Length
?
maze_large
4
2
2
600
0.99
pinpad
7
3
2
300
0,99
fetch__uss push
25
3
4
50
0.95
kitchen
30
17
8
280
0.99
Table S1: Environment parameters.
1.2 Unsupervised Exploration
Unsupervised exploration of the environment relies curious model-based planning [38, 43]. We will thus describe the exploration algorithm, followed by the architecture of the dynamics model P and the computation of the intrinsic cost signal.
Unsupervised exploration alternates enviromment interaction with model training. until sufficient experience is collected. We warm-start exploration by executing uniformly random policy for 3000 steps, and saving its trajectories in the replay buffer D) Then, before each episode, the dynamics model P is trained on data from the replay buffer 'D via supervised learning for 12 epochs. Adam [21] used an optimizer with batch size 512, learning rate "ne 0.00028 and weight decay parameter À 0.0001.
The dynamics model P is learned as an ensemble of Gaussian neural networks, as suggested in Chua al. [8]. Given state E S and an action S, each of the M ensemble mem- ber outputs a Gaussian distribution parameterized by its mean and its variance: (s, a) = [pm (s,a), a), (arm m)])m In practice, inputs and outputs are normalized, and the model pre- dicts state differences. By default, the ensemble contains 7 members; during inference, only the best members according likeliiidood on validation set are queried. Implementation and hyperparameters, reported in Table S2, are the default from mbrl llib [34].
Exploration episodes are controlled through finite-horizon model-based planning with zero-order trajectory optimization (i.e., iCEM with the learned model P).. Given state-action pair (s, a). an intrinsic reward signal can be computed as the trace of the covariance matrix over the ensemble's 's predictions [38, 49]
Hyperparameter
Value
# of ensemble members # of elites
5
# of layers
# of units per layer
200
Activation function
SiLU
Tim (s, a) Tr(Coo((JJ aM)
(S17)
Propagation method
TS1 [8]
# of particles
20
This reward function can be aggregated into cost function for trajectory optimization, as described in Section I.4. Overall. our implementation of unsupervised exploration runs 5K steps per hour on single Nvidia RTX3060 GPU or equivalent.
Table S2: Hyperparameters for the dynamies model P.
I.3
Value Learning
The relabeling procedure for exploration data is adapted from Tian al. [46], and described in
Section 4. We implement TD3 [14] and CRR [50] for value learning, and build upon TD3 for two
model-based algorithms: MBPO [19] and MOPO [53]. We found DDPG [26] and SAC [17] to be slightly less performant in comparison to TD3. Each value learning algorithm trained for 200 epochs, which, for buffer size of 200K, corresponds
wall-clock time less than 3 hours single NVIDIA RTX3060 GPU. Hyperparameters are listed in Table $3. Those tagged as (MBPO) are adopted by both MBPO and MOPO, while those tagged (MOPO) (CRR) algorithn-speeffie. Remaining hyperparameters are shared by all algorithms. All hyperparameters are shared across environments, and they were tuned through grid search across maze large and fetch__push
1.4 Model-based Planning
Model-based planning used both during the exploration phase, and during goal-conditioned evaluation. During exploration, the cost of each trajectory (so, ag. ... -H aH-1) - is computed as:
24

---

H-1 Σ 1=
(sa((s do..... S00,---.,
Tim St, 0e)
(S18)
During goal-conditioned evaluation, value function V(s;g) is optimized, and the cost of each trajectory directly adapted from the reinforcement learning objective:
Cexe((0,, 1,
Σ * 0e) + (0n"-((sn-igg))
(S19)
where the equality holds due to the extreme sparsity of R[s; g) 1==9 in continuous spaces. Hyper- parameters are fixed across environments, except for the planning horizon H. We find exploration H 30. On the other hand, we found goal- benefit from longer horizon, which is thus exploitation and hallucination. For this reason, conditioned evaluation to be more prone to model during the extrinsic phase, we set H 10 for pinpad and 15 for the remaining environments. For the same reason, we also compute the variance of particles for each trajectory, and discard those exceeding a set threshold.
While vanilla CEM [37] also performs well as zero-order trajectory optimizer, we found iCEM [35] to reach similar performances when using a reduced number of samples. Hyperparameters are reported in Table S4.
1,5 Metries
Unless specified, all plots show the mean estimate from 9 seeds, which are the combination of seeds for collecting exploration data, and 3 seeds for value learning and planning for each dataset. Each mean estimate is accompanied by 90% simple bootstrap confidence interval.
L6 Computational Costs
Our method's training costs (TD3 model-based planning + aggregation) are those TD3 (~70 minutes single NVIDIA RTX3060 GPU), comparable to other value-learning baselines (~500-90 minutes). Inference costs differ greatly: actor networks can be queried at >100Hz; model-based planning with and without aggregation at 3Hz and 0.3Hz respectively. An optimized implementation would bring significant speedups, e.g. through precomputation and parallelization of graph search, which could address the significant slowdown.
Open Challenges and Negative Results
Our method provides insights on how to mitigate the occurrence of estimation artifacts in learned value functions. However, it does not correct them by finetuning the neural value estimator. This possibility is interesting, as it would bring no computational overhead, and was broadly explored. TD finetuning with CQL-like penalties on the value of -local optima was attemped, but unsuccessful. In this context, we found such correction strategies to be sensible to hyperparameters, and often resulting overly pessimistic estimates. We, however, stress the significance of direct value correction as an exciting direction for future research.
K
Numerical Results
Table S5 reports success rates from Figures 2 and (main paper) in numerical form.
25

---

Hyperparameter
Value
of critic layers
2
# of units per critic layer
512
of actor layers
2
of units per actor layer
512
Batch size
512
Critic learning rate
le-5
Actor learning rate
le-5
Activation function
ReLU
Polyak coefficient
0.995
Hyperparameter
Value
Target noise
0.2
Target noise clipping threshold
0.5
# of iterations Population size Elite ratio
3
Policy delay
400
Policy Squashing
True
0.01
Pg
0.75
0.1
Pgco
0.2
Population decay
False
Colored noise exponent
3.0
(MBPO) # updates before rollouts
500
Fraction of kept elites
0.3
(MBPO) of rollouts
5000
(MBPO) Rollout horizon
Table S4: Hyperparameters for model- based planning.
(MBPO) Buffer capacity
1M
(MBPO) Real data ratio
0.0
(MOPO) Penalty coefficient
0.1
(CRR) /
1.0
(CRR) of action samples
(CRR) Advantage type
mean
(CRR) Weight type
exponential
(CRR) Maximum weight
20.0
Table S3: Hyperparameters for value learning.
ma maazelarrge
pinpad
ce_phhuush
kitchen
CRR
0.173 <0255
0..55
0.083 0.083
0.5 <0331
MBPO
8,00 0.000 < 6.000
0017
c0.000
0,83 c 0.083 0.033
0.111 20144
20.774
MOPO
0.023 < 0,049 c0.079
0.03
< 000072
0.83 0003<0.05 0.083 0.083
0015 0..550004666 0.099
TD3
0,079 0. 144 <0.205
n2Mx-02267
0.2299
0,800 0.083 0,083
- 0.142
CRR+MBP
0.031 <0.063
0.321
0.190 0.231
0.174 0.2460
0.238
MBPO+MBP
0.024 < 0.060 0.098
0.04 0.068
0.09
0,638 < 0.719 0.791
0.174 0.222 0.285
MOPO+MBP
0.000 0.025 <0.09
0.025
0.038
0.24 0.884 0.935
0.444 0.503
TD3+MBP
0.206 < 0.301 c0.412
0.190 0.232
0.227
0.38 0.693 0.745
0.333 0.379 00,22
"DS+MMBPAAgregation
0,627 < 0.705 0.814
0.314 0.410 c.5.00
0.587 < 0.647 0.700
0.301
0.365 0.48
MBP+Sparse
0.030
0.100
0.730
0.430
Table S5: Average success rates with 90% bootstrap confidence intervals from Figures and (main
paper).
26


Reviews:
[{"review_id": "CwhbD0WNyp", "reviewer": "Reviewer_cdVD", "summary": "This paper addresses the challenge of extracting goal-conditioned behavior without the need for continued environment interaction. The authors tackle this challenge by leveraging unsupervised exploration to collect diverse experiences. The study explores the suitability of different off-policy actor-critic algorithms and investigates the performance gap between utilizing network-based and model-based planning approaches. To overcome global value estimation artifacts, the authors propose a graph-based value aggregation method that aggregates short-horizon estimates. This technique aims to correct potential inaccuracies in global value estimation. The proposed framework is evaluated using various simulated environments.\n", "questions": "- in section 4.1, why do we need sampling negative goals? \n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Analysis of the performance gap between model-based planning and actor networks is interesting.\n- Extensive experiments\n", "weaknesses": "One notable weakness of this paper is the limited technical contribution. The two main components proposed, model-based planning and graph-based aggregation, are existing algorithmic components commonly used in the field. As a result, the paper lacks novelty in terms of introducing new techniques or approaches. \n\nFurthermore, the proposed framework of using model-based planning and graph-based aggregation does not appear to be specifically tailored to the setup of achieving goals after an unsupervised exploration phase without further interaction with the environment. It would be beneficial to clarify whether certain components or analyses of the proposed framework are specifically relevant to this setup. If not, it might be more appropriate to broaden the scope to a generic goal-conditioned RL setup. Additionally, it would be valuable to see an evaluation of the proposed framework in the context of this broader setup.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback and clear assessment. In this response we further clarify our technical contributions, provide a discussion and experiments regarding the generality of our method and answer the reviewer's question.\n\n# Technical contributions\nWhile both model-based planning and graph-based techniques have been explored in the literature, we are not aware of previous works that analyze them under the common lens of robustness to local minima, or propose their combination. In particular, graph search has been mostly leveraged for subgoal selection [1] or policy distillation [2], and not embedded in the planning procedure.\n\nMoreover, from a strictly technical perspective, we propose significant modifications to the graph search procedure introduced in [1], as we find that a naive combination of existing methods is not effective. Namely, we propose importance sampling with learned density estimates for the vertices of the graph, and automatic threshold selection for graph pruning. This guarantees that the graph will be pruned as aggressively as possible, thus reducing reliance on inaccurate long-horizon estimates, while ensuring the existence of a solution to the graph search problem. A detailed discussion and ablation of these features can be found in Appendix D.\n\n# Applicability to general offline GC RL\nWe agree that the scope of the method is not strictly limited to learning from curious exploration data. In fact, it can in principle be applied to general goal-conditioned settings. However, certain components of our analysis and method are specific to curious exploration or are designed to leverage the properties of exploratory data. We thank the reviewer for encouraging the exploration of this direction and refer them to the common response for a discussion and new experimental results.\n\n# Sampling of negative goals\nFinally, we would like to address the question about Section 4.1. If negative goals were not sampled, the value function would only be trained on goals that were actually reached from each given state. On the one hand, this would hinder trajectory stitching. On the other hand, the value function would systematically overestimate values, as any goal which is not easily achievable from a given state would never be observed during training. At inference, when a challenging goal is set, the value network would then be queried on a state-goal pair that is significantly different from those observed during training. Empirically, this results in a drop in performance when the probability of sampling negative (random) goals is set to zero.\n\n# References\n1) Eysenbach et al., Search on the Replay Buffer: Bridging Planning and Reinforcement Learning, NeurIPS 2019\n2) Mezghani et al., Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping, CoRL 2022"}, {"review_id": "CEf60uJxcC", "reviewer": "Reviewer_7YsN", "summary": "This paper investigates the issue of mitigating estimation artifacts in the value function of off-policy RL. The authors adopt an unsupervised scheme to collect exploratory data, which is subsequently used to train both off-policy and offline RL algorithms. In their analysis, the authors identify a geometric characteristic known as $\\mathcal{T}$-local optima, which represents inaccurate value landscapes. To address this issue, the authors propose a combination of model-based planning and graph-based aggregation, resulting in enhanced performance on long-horizon goal-conditioned tasks.", "questions": "Overall, I appreciate the authors for identifying the problem and proposing a novel model-based planning method to address it. I would like to raise my score if the authors can address the following concerns mentioned in the above section on weaknesses.\n* The performance is not comparable to that of prior works in several environments.\n* It is recommended to compare the results with additional baselines and report the computational costs for both training and testing.\n* Please consider supplementing the literature review of related works.\n* It would be beneficial to provide additional evidence from other benchmarks, such as AntMaze or FetchSlide/Pick/HandReach.\n* Additionally, improving the presentation and organization of the paper would be helpful.\n* Minor: Line 125 appears to contain a typo where $p_{G}$ is mentioned.\n", "limitations": "This paper highlights one limitation: the method does not address estimation artifacts during training. However, another significant limitation is the negative impact imposed by the computational costs associated with training the ensemble model, graph building, and test-time planning.\n", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "* The finding in this paper that learned value landscapes can hinder achieving long-horizon goals is novel and interesting. \n* The proposed method, which combines model-based planning with graph-based value aggregation, makes sense and is demonstrated to enhance the performance of vanilla TD3.\n* The empirical evidence provided in Appendix C through visualization effectively demonstrates the presence of estimation artifacts in the value landscapes.", "weaknesses": "* I have concern regarding the performance of the proposed method, as it fails to attain comparable results with previously published works, such as [1][2] on Maze2D, [3][4] on FetchPush, and [5][6] on Kitchen.\n* Furthermore, the planning method incurs higher training and testing costs than prior offline goal-conditioned works [4][6], yet its performance does not meet the same standards. I recommend that the authors check this issue by conducting a comprehensive comparison, and elucidate the advantages of the proposed method over other offline goal-conditioned RL baselines and model-based planning baselines.\n* The literature review in this work is inadequate as it fails to cite and discuss important prior offline goal-conditioned works [4][6][7][8]. Notably, the work by [8], which involves the creation of a graph from offline dataset, is particularly relevant. Furthermore, this paper overlooks prior goal-conditioned methods [7][9][10] that employ model-based planning.\n* The evaluation benchmark appears to be limited, especially two environments are simple point-based tasks. To conduct a comprehensive study, it is essential to incorporate additional benchmarks such as AntMaze in D4RL, as well as the environments FetchSlide, FetchPick, and FetchHandReach used in [4].\n* The paper would benefit from an improvement in its presentation and organization. A significant portion of the paper is devoted to discussing value learning, leaving only Section 6.3 for the main experimental results. To address this, I recommend compressing or relocating part of Section 2, Section 3, Section 4, and Section 5, perhaps to the Appendix. Additionally, it would be helpful to move some of the interesting evidence from Appendix C to the main paper.\n\n\n\n[1] Chen X, Ghadirzadeh A, Yu T, et al. Lapo: Latent-variable advantage-weighted policy optimization for offline reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 36902-36913.\n\n[2] Janner M, Du Y, Tenenbaum J B, et al. Planning with diffusion for flexible behavior synthesis[J]. arXiv preprint arXiv:2205.09991, 2022.\n\n[3] Charlesworth H, Montana G. Plangan: Model-based planning with sparse rewards and multiple goals[J]. Advances in Neural Information Processing Systems, 2020, 33: 8532-8542.\n\n[4] Yang R, Lu Y, Li W, et al. Rethinking goal-conditioned supervised learning and its connection to offline rl[J]. arXiv preprint arXiv:2202.04478, 2022.\n\n[5] Kostrikov I, Nair A, Levine S. Offline reinforcement learning with implicit q-learning[J]. arXiv preprint arXiv:2110.06169, 2021.\n\n[6] Emmons S, Eysenbach B, Kostrikov I, et al. RvS: What is Essential for Offline RL via Supervised Learning?[J]. arXiv preprint arXiv:2112.10751, 2021.\n\n[7] Li J, Tang C, Tomizuka M, et al. Hierarchical planning through goal-conditioned offline reinforcement learning[J]. IEEE Robotics and Automation Letters, 2022, 7(4): 10216-10223.\n\n[8] Mezghani L, Sukhbaatar S, Bojanowski P, et al. Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping[J]. arXiv preprint arXiv:2301.02099, 2023.\n\n[9] Nasiriany S, Pong V, Lin S, et al. Planning with goal-conditioned policies[J]. Advances in Neural Information Processing Systems, 2019, 32.\n\n[10] Charlesworth H, Montana G. Plangan: Model-based planning with sparse rewards and multiple goals[J]. Advances in Neural Information Processing Systems, 2020, 33: 8532-8542.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the detailed comments and clear listing of points, which we address sequentially.\nPlease let us know if further clarification or evidence is needed. \n\n# Performance comparison \nAs noted by the reviewer, performance in the four environments is not comparable to that reported in previous work. The main reason is that our experiments do not use off-the-shelf offline datasets (e.g., D4RL [1]): we deliberately focus on datasets collected with minimal supervision and in realistic timeframes (see Section 3.2). The datasets are collected through curious exploration (using model disagreement as intrinsic motivation), which is fundamentally different from near-expert demonstrations, or uninformed exploration contained in standard datasets. While curious exploration attains better exploration compared to uniformly collected datasets [2], it does not generally contain high-quality, task-specific behavior (unlike most D4RL datasets).\n\nThis naturally impacts downstream task performance. For instance, on maze_large, unsupervised exploration manages to visit all sections of the maze, albeit hard-to-reach ones are visited less often compared to the D4RL dataset (see Fig. 3), and we observe success rates above 70% (Figure 6). In more complex environments, we find curious exploration to be less performant. For instance, in kitchen, 2/7 objects are not interacted with during exploration, and downstream agents fail to control them.\n\nData quantity is also limited: we focus on datasets that could be collected within a few hours at 30 Hz, totaling 200K transitions. Compared to the literature, our datasets are up to an order of magnitude smaller [1,3]. Our evaluation also slightly differs from the D4RL protocol: we assess multiple goals per environment, reporting success rates for interpretability, not returns (see Appendix E).\n\nFor these reasons, we include multiple baselines, to which we added those suggested by the reviewer.\n\n# Additional baselines\nWe implemented and benchmarked 3 of the baselines suggested, namely IQL, LAPO and WGCSL. We have prioritized them over the three remaining works, which are not value-based and are not reported to significantly outperform the three implemented ones. Additional hyperparameters were tuned with a limited budget on maze_large. We ran each baseline in its original configuration (actor network, in blue), and in combination with model-based planning (in orange). Performance is averaged over 12 seeds and reported in Fig. B2. The three baselines perform within the range of existing ones, suggesting that they do not avoid value estimation artifacts. Moreover, they do not consistently outperform TD3, confirming the competitiveness of simple actor-critic algorithms, as we hypothesized.\nFinally, the new baselines generally improve their performance when coupled with model-based planning, as shown for the existing ones.\n\n# Computational costs\nWe have extended Appendix E.4 to include more precise computational costs, including baselines. Our method's training costs (TD3 + model-based planning + aggregation) are those of TD3 (\\~70 minutes on RTX3060), comparable to other value-learning baselines (\\~50-90 minutes). Inference costs differ greatly: actor networks can be queried at >100Hz; model-based planning with and without aggregation at \\~3Hz and \\~0.3Hz respectively (Appendix E.4). An optimized implementation would bring significant speedups, e.g. through precomputation and parallelization of graph search. The intrinsic phase also incurs computational costs (Appendix E.2).\n\n# Literature review\nWe thank the reviewer for the pointers to relevant literature. We will rework and expand our related work section to include informative context on goal-conditioned reinforcement learning. We agree that [4] is particularly relevant, and it was in fact already included (see line 115), albeit the reference points to a different venue.\n\n# Additional Environments\nAs suggested by the reviewer, we have applied our method to the four proposed environments.\n- antmaze is an interesting benchmark, involving low-level control, and high-level planning. However, the performance of our method, as well as baselines, is constrained by the unsupervised exploration phase. While being a state-of-the-art exploration signal, ensemble disagreement of the dynamics model is not able to make significant progress in this environment: the exploration algorithm focuses on exploring the dynamics of the robot, and does not explore beyond the initial corridor in the first 200k steps. All baselines, as well as our method, do not make progress in this task.\n- hand_reach is also constrained by the exploration data: the chaotic nature of disagreement-based curiosity results in not observing fine-grained motor control. As a result, we find that no method reaches the precision required by the original benchmark (~5 mm average error in fingertip positions). However, both our full method and a simple TD3 policy are able to solve the task for larger tolerances. We opt not to include it to avoid confusion. \n- we found exploration to be sufficient in fetch_pickandplace and fetch_slide. We thus extended the plots in Fig. 6 by Fig. B3. The results are consistent with those observed in fetch_push: model-based planning is crucial, while graph-based aggregation is not as beneficial, as any goal can already be reached within the planning horizon.\n\n# Presentation\nWe have spatially compressed the initial discussion and added one example from Appendix C to the main paper to more clearly demonstrate learning artifacts, as suggested.\n\n# References\n1) Fu et al., D4RL: Datasets for Deep Data-Driven RL, arXiv:2004.07219\n2) Lampert et al., The Challenges of Exploration for Offline RL, arXiv:2201.11861\n3) Yang et al. Rethinking Goal-conditioned Supervised Learning and its Connection to Offline RL, arXiv:2202.04478\n4) Mezghani et al., Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping, CoRL 2022"}, {"review_id": "ahvfRWX8nb", "reviewer": "Reviewer_nGpW", "summary": "This paper tackles the problem of performing goal-conditioned control using data collected from curious exploration. They first perform an empirical evaluation of several methods for learning value functions from this offline data, and then identify an issue with estimated value functions called $\\mathcal{T}$-local optima, and then a value aggregation scheme that grounds the value function in states observed during exploration, which essentially replaces a long-horizon value function estimate with a combination of shorter-horizon estimates. The paper finds that this aggregation strategy combined with model-predictive control is able to improve performance on tasks including maze solving, pinpad, fetch push, and Franka kitchen.", "questions": "- How does the choice of path length in equation 5 affect performance? \n- As Equation 5 minimizes over a number of paths composed of learned value estimates, does this result in compounding errors and model exploitation as well?\n- Following my comment in “weaknesses”, are there specific properties of the curious exploration setting that make this method more effective? Will the performance of this method change significantly if presented with data from a random policy or near-expert policy? \n\n** Update after rebuttal : I have read the authors' rebuttals. They address my concerns reasonably well, and I have chosen to maintain my rating. **", "limitations": "The authors discuss some of the limitations to their methods, particularly the computational requirements compared to policy learning. I don’t see direct potential negative societal impact for this work, so I think the discussion is sufficient, but would appreciate a further discussion of failure cases or limitations particularly of their graph aggregation method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-motivated and clearly written. Each section was easy to follow and the hypotheses and experimental settings very sensible.\n- The observation of $\\mathcal{T}$-local optima as an estimation artifact and a cause of issues in goal-conditioned value function learning is very interesting and to my knowledge, has not been discussed before. While the authors’ empirical analysis in showing that $\\mathcal{T}$-local optima indeed are the source of pathologies across multiple environments in the supplementary material is a bit less conclusive than Figure 5 would lead one to believe, I believe that this is a solid contribution that can help to direct future research to look into this further. \n- The authors’ proposed solution to address these local optima via a combination of model-based planning and a (to my knowledge) novel graph aggregation scheme seems sensible and effective. It achieves improved (on maze_large and pinpad) or similar (on fetch_push and kitchen) performance. \n- The empirical analysis of actor networks versus model-based planning helps to solidify some observations that have been made in prior work.", "weaknesses": "- I would recommend being more transparent with the results for an experiment like Figure 5 for the other environments, where the trend is not as apparent. This would be more helpful for readers to understand how well this hypothesis holds up empirically, and while I understand there is a significant amount of noise, that itself is also useful to know about.\n- While the manuscript places significant emphasis on the unsupervised exploration/curiosity component of the problem setting, it doesn’t appear that any component of the method is actually specific to the curious exploration setting, nor does it leverage any particular properties of curious exploration. It seems like all of the arguments that the paper makes would be equally valid when using data from a random policy or an expert policy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their assessment and detailed feedback. In this response, we comment on Figure 5, on the generality of the method, and provide clarifications on graph-based aggregation.\n\n# Figure 5\nWe appreciate the suggestion for extending Figure 5 to include other environments from the Appendix, and have applied this change. \nAs the reviewer notes, these metrics are not designed to exactly quantify the existence of value artifacts, but only to estimate their presence and effects; they are sampled-based, and thus noisy by nature, especially in low-data regimes. We will add these remarks directly at the end of Section 5 to ensure clarity. Furthermore, we would like to encourage the development of more robust probing methods as a further direction of research.\n\n# Generality of the method\nWe agree that our method can be applied beyond the curious exploration setting, and potentially learn from different data sources. Nevertheless, the empirical evaluation in Section 4.2 and certain components of our method are motivated by curious exploration. We agree that certain ideas and components can be effective in broader settings, and we thus perform an empirical investigation on the applicability of our method to data collected from an expert, or a random actor. We refer the reviewer to the common response for a broader discussion and experimental results.\n\n# Graph-based aggregation\nThe choice of path length in Equation 5 is not necessary: all paths over the (pruned) graph are considered (as proposed in [1]), and the minimization is in practice solved through Dijkstra's algorithm. As the minimization is over products of learned value estimates, it is indeed prone to failure due to compounding errors. There are, however, two design choices that introduce robustness.\n\nFirst, value estimates are only computed over states and goals that appear in the dataset the value function was trained on, which alleviates OOD issues. Most importantly, the pruning step (see line 159 in the Appendix) effectively removes all edges encoding long-horizon estimates, which are empirically found to be less accurate [1]. Overall, the minimization is therefore solved over a smaller number of paths with respect to those existing in the fully-connected graph.\n\nMoreover, the tuning procedure we propose for the pruning threshold minimizes the number of candidate paths, while ensuring the existence of a solution. Despite this, for particularly sparse datasets, the pruning can fail to remove inaccurate estimates, and the graph-based aggregates may reproduce the artifacts of the underlying value estimator.\n\nFurther limitations of the graph search include increasing computational costs, and artifacts due to the effective discretization of the state space: while increasing the sample size can address the latter, it aggravates the former in a tradeoff. We have added a dedicated discussion of failure cases and limitations, with a particular focus on graph-based aggregation to Appendix D, thus extending the existing discussion on the details of this component.\n\n# References\n1) Eysenbach et al., Search on the Replay Buffer: Bridging Planning and RL, NeurIPS 2019"}, {"review_id": "vqNNm9AOvQ", "reviewer": "Reviewer_yLbp", "summary": "The paper presents a method that using a dataset of rollouts and a dynamic model create a graph like structure in-order to train goal conditioned policies. The nodes of the graph are selected based on it high local value function, and connected using a model based mechanism offline that avoid the myopic behavior of other methods.", "questions": "I would be interesting to see the comparison mentioned in the previous section. ", "limitations": "none", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Solid theoretical analysis of challenges on graph based models that create local and global minima.\nIn-depth comparison with other techniques, and prolific appendix with theoretical support, more results, and value function analysis.\n", "weaknesses": "Even if I see the intense amount of effort on the paper, its hard to piece all the contributions together. I'm mean more in the sense of clarity. One way of addressing this for example could be adding pseudocode of the algorithm.\nThe idea is an improvement of other graph based efforts of the past, as well pointed it out by the authors, Salinov et al , and others.\nI believe is missing a comparison with the line of work of Lowrey et al 2019 ICRL, \"https://sites.google.com/view/polo-mpc\"\"Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control\" , Hnsen et al 2022 , Neurips, and related work where a value function analysis it also performed, and the mixed of value function and model based planning is done.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions, which we address sequentially as follows.\n\n# Contributions\nWe have added to the Appendix a detailed algorithm box describing the practical algorithm, and how the different components are combined. In the context of this response, we would like to also report a verbal equivalent.\n\nOur work deals with extracting a goal-reaching controller from the outcomes of curious exploration. Exploration data is collected while learning a dynamic model ensemble by maximizing its disagreement through model-based planning with zero-order trajectory optimization. This constitutes the intrinsic phase, which returns an exploration dataset, and a trained dynamics model.\n\nWe then train a value function offline on the exploration dataset with TD3. During inference, we are given a goal. We select actions through model-based planning with the learned dynamics model. This time, actions sequences are selected by maximizing the aggregated value of the last state in imagined trajectories, where aggregation is performed with graph search over the value estimates from TD3.\n\nWe will reword the introduction to provide a similar description.\n\n# POLO\nWe thank the reviewer for pointing us in the direction of POLO [1]. We believe that the method represents an interesting privileged baseline. The main differences between our method and POLO are as follows:\n- POLO explores by maximizing the uncertainty of the value function; our framework leverages the uncertainty of the dynamics model.\n- POLO assumes the availability of ground truth models for planning; our framework does not and adopts a learned dynamic model.\n- POLO does not rely on graph-based components for long-horizon planning.\n\nWhile an official implementation of POLO is not available, we can loosely reproduce it by integrating POLO's exploration signal in our framework, replacing TD3 with its value learning scheme, and disabling graph-based value aggregation, while still relying on iCEM for model-based planning and learning a dynamics model. We name the resulting method POLO*. We find that POLO* suffers from model inaccuracy when computing its value targets via MPPI optimization. We thus introduce a variant which uses on-policy value targets, which we refer to as POLO**. \n\nFigure B4 report success rates for POLO* and POLO** against our method (TD3 + model-based planning + graph-based aggregation) in all of our benchmark environments. Plots are averaged over 12 seeds, except for POLO* which uses 5 due to higher computational cost. Moreover, we plot exploration trajectories in maze_large in Figure B5.\nWe note that POLO is a privileged baseline, as it requires goals to be specified during training, and its value function is trained online by optimizing against its uncertainty. Our method, on the other hand, learns its value function offline from unlabeled data, and is therefore more widely applicable.\n\nWhile the closer implementation (POLO*) is less performant, the improved version (POLO**) performs comparably or better than our method in three out of four environment. Despite successfully leveraging online value learning, we find its potential for long-horizon exploration to be limited, as shown by a large drop in performance in maze_large. We confirm this by visualizing trajectories gathered by POLO** (value-driven), and by the unsupervised exploration phase used in our work (dynamics-driven) in Figure B5.\n\n# References\n1) Lowrey et al., Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control, ICLR 2019"}]

Author rebuttal:
We thank all reviewers for their constructive feedback, which helped us to improve our paper. We ran additional baselines, added evaluations on more environments, and provided individual responses.
Here, we summarize the added validation, and then address the applicability of our method in general offline goal-conditioned settings, with arbitrary data sources.
We look forward to further discussing our work and providing clarifications or additional results.

# Summary
We gathered additional results which confirm our contributions, and provide further insights into our method.
- We add a study on the applicability of our method to general offline goal-conditioned settings (see Fig. B1 in the rebuttal PDF, and discussion below).
- We extend the experimental results on value estimation of Section 4.2 by introducing 3 additional baselines, i.e., IQL [4], LAPO [5] and WGCSL [6]. These methods perform similarly to benchmarked offline RL algorithms, thus confirming the effectiveness of standard methods such as TD3 in this setting. (see Fig. B2 and response to Reviewer 7YsN)
- We evaluate our method on two additional environments: fetch_pickandplace and fetch_slide. Results are consistent with those reported in fetch_push. (see Fig. B3 and response to Reviewer 7YsN)
- We study POLO [7] as an alternative (but not task-agnostic) value-driven exploration algorithm. We had to adapt the method as it assumes ground-truth models. By training its value function online, POLO achieves good performance in most environments. Our method, despite training its value function completely offline, is often able to match its performance and surpasses it when long-horizon exploration is necessary. (see Fig. B4 and response to Reviewer yLbp)

# Applicability to general offline goal-conditioned settings
The motivation of our work is to address the challenge of extracting goal-conditioned behavior after curious exploration. We believe that focusing on data gathered through unsupervised exploration has value in itself, as this requires minimal supervision and can enable real-world robot learning [1]. However, as reviewers noted, our method can in principle be applied to general offline goal-conditioned settings. In this section, we comment on this possibility and provide empirical evidence.

## Specificity to unsupervised curious exploration
Throughout this work, the nature of the data distribution is leveraged mainly in two points.
- As argued in Section 4.2, we find that, when learning from curious exploration data, simple actor-critic methods are, on average, competitive with model-based or offline methods. In standard offline settings with near-expert data, we expect offline algorithms to be more effective [2]. However, within the realm of curious exploration, we found TD3 to be equally performant, and thus use it as core algorithm for value learning.
- Our method relies on model-based planning, whose performance is heavily dependent on the quality and robustness of the learned model. Model-based planning is thus particularly promising when learning from curious exploration, as the unsupervised exploration phase already produces a dynamics model, which does not need to be trained from scratch. Moreover, as data has been collected to maximize the information gain of this dynamics model, it is potentially more robust [3]. When learning from offline datasets, a model is generally not available: while it can also be learned offline, training it to high accuracy can be difficult.

## Evaluation on different data sources
We provide an empirical study of the applicability of our method on different data sources in Fig. B1. Each subplot presents the performance of our full method (TD3 + model-based planning + graph-based value aggregation) when mixing the original curious exploration datasets with data collected from different sources (the size is fixed at 200k samples). In particular, we replace a fraction of the trajectories with transitions from an expert policy (a,c), and from a uniformly random policy (b,d). We include performance curves when using the dynamics model learned during curious exploration (green), and a dynamics model learned offline on the mixed data (violet); offline model learning was not tuned specifically. Due to time and compute constraints, we also present this analysis over two representative environments (maze_large in (a,b) and fetch_push in (c,d), and average over 9 random seeds.

In these settings, we observe that access to expert data does not hinder success rates (Fig. B1a,c): interestingly, our method shows robustness to shrinking data support. On the other hand, when introducing data collected by a uniform random policy (Fig. B1b,d), performance decreases accordingly to the reduced exploration: while collecting random trajectories is sufficient in fetch_push, it is not comparable to curiosity-driven exploration for long-horizon tasks (i.e. maze_large). Furthermore, we note that models trained offline (in violet) achieve generally success when the complexity of the environment's dynamics increases (fetch_push), confirming that online training via disagreement maximization returns a more robust model [3].

We have added these results to the Appendix and will refer to them in Section 4 to provide more context.

# References
1) Mendonca et al., ALAN: Autonomously Exploring Robotic Agents in the Real World, ICRA 2023
2) Fu et al., D4RL: Datasets for Deep Data-Driven RL, arXiv:2004.07219
3) Sancaktar et al., Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation, NeurIPS 2022
4) Kostrikov et al., Offline RL with Implicit Q-Learning, ICLR 2022
5) Chen et al., Lapo: Latent-variable Advantage-weighted Policy Optimization for Offline RL, NeurIPS 2022
6) Yang et al., Rethinking Goal-conditioned Supervised Learning and its Connection to Offline RL, ICLR 2022
7) Lowrey et al., Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control, ICLR 2019

Meta-review: The paper tackles an important problem of extracting goal-conditioned policies from data collected by using curioisty-driven explorarion. Reviewers unanimously but mildly vote to accept the paper, I concur. Authors should consider including responses to reviewer questions and additional experiments in the final version of the paper.
Decision: Accept (poster)