SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We will show that, for a certain finite  $T$ ,  $s_T$  achieves the goal, i.e.,  $R(s_T; g) = 1$  and  $R(s_i; g) = 0 \forall 0 \leq i < T$ . We note that, since the episode terminates when the goal is achieved and  $\gamma(s; g) = \gamma(1 - R(s; g))$ , Bellman optimality implies that  $R(s_T; g) = 1 \iff V^*(s_T; g) = 1$ , as shown in Equation S6.
From Equation S6 we can then derive
$$
\begin{array}{l} V^{*}(s_{i};g) = R(s_{i},g) + \gamma \max_{a\in \mathcal{A}}\underset {s^{\prime}\sim \mathcal{P}(\cdot |a,s_{i})}{\mathbb{E}}V^{*}(s^{\prime};g) \\ = \gamma \max  _ {a \in \mathcal {A}} \underset {s ^ {\prime} \sim \mathcal {P} (\cdot | a, s _ {i})} {\mathbb {E}} V ^ {*} \left(s ^ {\prime}; g\right) \tag {S10} \\ = \gamma \max  _ {s ^ {\prime} \in \mathcal {T} (\{s _ {i} \})} V ^ {*} (s ^ {\prime}; g) \\ = \gamma V ^ {*} (s _ {i + 1}; g), \\ \end{array}
$$
where the third equality follows from deterministic transitions, and the last from the objective in Equation S9. By applying this equality recursively, we can write  $V^{*}(s_{i};g) = \gamma^{-i}V^{*}(s_{0};g)$ . After  $T = \lceil \log_{\gamma}V^{*}(s_{0};g)\rceil$  steps,
$$
V ^ {*} \left(s _ {T}; g\right) = \gamma^ {- T} V ^ {*} \left(s _ {0}; g\right) \geq 1. \tag {S11}
$$
Since  $V^{*}(\cdot ;g)\leq 1$ , therefore  $V^{*}(s_{T};g) = 1$  and  $s_T$  achieves the goal.
An actor network is trained to output
$$
\begin{array}{l} \pi (s_{i};g) = \operatorname *{argmax}_{a\in \mathcal{A}}Q^{*}(s_{i},a;g) \\ = \underset {a \in \mathcal {A}} {\operatorname {a r g m a x}} R \left(s _ {i}; g\right) + \gamma \underset {s ^ {\prime} \sim \mathcal {P} (\cdot | a, s _ {i})} {\mathbb {E}} V ^ {*} \left(s ^ {\prime}; g\right) \tag {S12} \\ = \operatorname * {a r g m a x} _ {a \in \mathcal {A}} \underset {s ^ {\prime} \sim \mathcal {P} (\cdot | a, s _ {i})} {\mathbb {E}} V ^ {*} (s ^ {\prime}; g) \\ \end{array}
$$
Under deterministic dynamics, the optimal policy is therefore producing the trajectory which optimizes the objective in Equation S9, and thus reaches the goal.
A.3 Model-based Planning and  $\mathcal{T}$ -local Optima
Section 5 pinpoints  $\mathcal{T}$ -local optima as a class of estimation artifacts in learned value functions. Following from Section 6.1, we now present two properties arising from the interaction between model-based planning, and a suboptimal value landscape, depending on the depth of  $\mathcal{T}$ -local optima, and the planning horizon.
First, we show that, given a goal  $g \in \mathcal{G}$ , model-based planning with horizon  $H$  from a  $\mathcal{T}$ -local optimum point  $s^*$  of depth  $k \geq H$  fails to achieve the goal. We make the assumptions presented in Section 3.1, and additionally assume:
1.  $\hat{\mathcal{P}} = \mathcal{P}$ , i.e., exact dynamics estimation.
2. Model-based planning optimizes its objective exactly (see Equation S13).
4On top of those listed in Section 3.1.
3. For every state  $s \in S$  and goal  $g \in \mathcal{G}$ , if  $V(s; g) \geq \max_{s' \in \mathcal{T}(\{s\})} V(s'; g)$ , then  $\exists \bar{a} \in \mathcal{A}: P(s|s, a) = 1$ . This implies the existence of self-loops for a set of states including  $\mathcal{T}$ -local optimum points.
We recall from Section 6.1 that, in this setting, model-based planning optimizes the following objective:
$$
a _ {0} ^ {H - 1} = \underset {a _ {0} ^ {H - 1} \in \mathcal {A} ^ {H}} {\operatorname {a r g m a x}} V (s _ {H}; g) \quad \text {w i t h} s _ {i + 1} \sim \hat {P} (\cdot | s _ {i}, a _ {i}), \tag {S13}
$$
We note that, if  $s_0 = s^*$  is a  $\mathcal{T}$ -local optima, then by assumption (3) an action  $\bar{a}$  is available such that  $P(s^* | s^*, \bar{a}) = 1$ . If  $a_i = \bar{a}$  for  $0 \leq i < H$ , then the objective in Equation S13 simply evaluates to  $V(s^*; g)$ . Any other action sequence will reach a state within the  $H$ -step future state distribution  $\mathcal{T}^H(\{s^*\})$ , and will evaluate to a lower or equal value as per the definition of  $\mathcal{T}$ -local optima of depth  $k$ :  $V(s^*; g) \geq \max_{s' \in \mathcal{T}^H(\{s^*\})} V(s'; g)$ . If ties are broken favorably  $^5$ , an MPC model-based planning algorithm will therefore execute  $\bar{a}$ , which deterministically transitions to the  $\mathcal{T}$ -local optimum point at each step. Thus, the goal is never reached.
Second, we show that, if the planning horizon  $H$  is greater than the depth of all  $\mathcal{T}$ -local optima over the state space  $S$ , open-loop execution of the action sequence returned through model-based planning successfully optimizes the value function (i.e., reaches a state with a higher value).
This property holds under additional assumptions, namely:
4. For all  $s \in S, a \in A, \exists s' \in S: P(s'|s, a) = 1$ , i.e., the MDP transitions deterministically.
5. The action sequence optimizing the objective for model-based planning (Equation S13) is executed in its entirety before replanning, i.e., open-loop execution of the plan.
6.  $V(s_0; g) < \max_{s' \in S} V(s'; g)$ , i.e., the initial state does not maximize the value function.
Since there are no  $\mathcal{T}$ -local optima of depth  $k\geq H$ , then there must exist a state  $s_1\in \mathcal{T}^{H'}(\{s_0\})$  such that  $V(s_{1};g) > V(s_{0};g)$  and  $H^{\prime}\leq H$ . Now, let us consider the state reachable in up to  $H$  steps with the highest value:
$$
V \left(s _ {1}; g\right) = \max  _ {H ^ {\prime} \leq H} \max  _ {s ^ {\prime} \in \mathcal {T} ^ {H ^ {\prime}} \left(\left\{s _ {0} \right\}\right)} V \left(s ^ {\prime}; g\right) \tag {S14}
$$
As shown in Equation S13, model-based planning only optimizes over states that can be reached in exactly  $H$  steps. It is thus sufficient to show that  $s_1 \in \mathcal{T}^H(\{s_0\})$ , as this would ensure that
$$
V \left(s _ {0}; g\right) <   V \left(s _ {1}; g\right) \leq \max  _ {s ^ {H} \in \mathcal {T} ^ {H} \left(\left\{s _ {0} \right\}\right)} V \left(s _ {H}; g\right) = \max  _ {a _ {0} ^ {H - 1} \in \mathcal {A} ^ {H}} V \left(s _ {H}; g\right) \text {w i t h} s _ {i + 1} \sim \hat {P} (\cdot | s _ {i}, a _ {i}). \tag {S15}
$$
Thus, open-loop execution of the action sequence  $a_0^{H - 1}$  optimizing Equation S13 would reach a state with a higher value (in this case,  $s_H$ ).
We recall that  $s_1 \in \mathcal{T}^{H'}(\{s_0\})$  for some  $H' \leq < H$ . We will now show that  $s_1 \in \mathcal{T}^H(\{s_0\})$  by contradiction. Let us assume that  $s_1 \notin \mathcal{T}^H(\{s_0\})$ . We thus have that
$$
\begin{array}{l} V(s_{1};g) = \max_{H^{\prime}\leq H}\max_{s^{\prime}\in \mathcal{T}^{H^{\prime}}(\{s_{0}\})}V(s^{\prime};g) \\ \geq \max  _ {H ^ {\prime} <   H ^ {\prime \prime} \leq H} \max  _ {s ^ {\prime} \in \mathcal {T} ^ {H ^ {\prime \prime}} \left\{\left. s _ {0} \right\}\right)} V \left(s ^ {\prime}; g\right) \tag {S16} \\ \geq \max  _ {H ^ {\prime} <   H ^ {\prime \prime} \leq H} \max  _ {s ^ {\prime} \in \mathcal {T} ^ {H ^ {\prime \prime} - H ^ {\prime}}} V (s ^ {\prime}; g) \\ \geq \max  _ {s ^ {\prime} \in \mathcal {T} (\{s _ {1} \})} V (s ^ {\prime}; g) \\ \end{array}
$$
Given two action sequences that both maximize the objective in Equation S13, we need to assume that the agent breaks the tie by maximizing the value of the first state in the trajectory  $V(s_{1};g)$ . This assumption would not be necessary in case of open-loop execution of the action sequence.
Following from assumption (4), a self-loop is thus available. Therefore  $s_1 \in \mathcal{T}(\{s_1\})$ , and more importantly  $s_1 \in \mathcal{T}(\{s_0\})^{H''}$  for any  $H'' \geq H'$ , thus contradicting the initial assumption and completing the proof.
B Occurrence of Artifacts and Downstream Performance
Sections 5 pinpoints  $\mathcal{T}$ -local optima as a class of estimation artifacts in learned value functions. This section empirically validates the harmfulness of such artifacts for an agent which attempts to optimize a value function  $\hat{V}(s;g)$ .
In order to do so, we train  $n = 30$  value functions for each environment. We then optimize each of them through model-based planning with horizon  $H$  for different goals, and obtain a set of trajectories. We split all evaluation trajectories in two groups: successes (those that achieve the goal and obtain a return greater than 0) and failures. We then estimate the occurrence of  $\mathcal{T}$ -local optima of depth  $k \geq H$  along each trajectory, i.e., the ratio of states in the trajectory which are likely to be  $\mathcal{T}$ -local optimum points, and plot its distribution for each group.
In practice, evaluating whether a candidate state  $s \in S$  is a  $\mathcal{T}$ -local optima is unfeasible, as it requires the evaluation of maximum operators over continuous spaces. A sampling-based estimation of  $\mathcal{T}$ -local optima remains possible, as described in Section 5, but will necessarily introduce a significant amount of noise. Nonetheless, we present the results of this procedure in Figure S8 (identical to Figure 5 in the main paper).
We note that the distributions of occurrences present long tails. We observe a general tendency for successful trajectories to contain fewer estimated  $\mathcal{T}$ -local optima, as the median occurrence is zero in three environments, and in any case never greater than the median occurrence for unsuccessful trajectories. This tendency is unsurprisingly the strongest in maze_large, which was observed to be prone to the occurrence of global estimation artifacts with depth  $k \geq H = 15$ .
We additionally report an analysis of the same trajectories with an indirect procedure, which does not rely on sampling. Appendix A.3 argues that, under several assumptions, model-based planning with horizon  $H$  from a  $T$ -local optimum point of depth  $k \geq H$  is incapable of reaching the goal, or any state with a larger value. The reverse does not necessarily hold: lack of value improvement does not imply the presence of artifacts, but might result from a suboptimal optimization procedure. Moreover, in practical settings, not all necessary assumptions hold. Nevertheless, it is still possible to compute the ratio of states  $s_i$  in evaluation trajectories for which  $\hat{V}(s_i; g) \geq \hat{V}(s_{i+H}; g)$ : this metric should tend to zero in the absence of estimation artifacts in the value function  $\hat{V}(\cdot; g)$ , assuming that the action selection procedure optimizes it correctly. We report similar plots for this metric, which we refer to as occurrence of non-monotonicities, in Figure S9. We observe that the patterns occurring in Figure S8 arise more evidently, as non-monotonicities tend to occur less frequently in successful trajectories.
Figure S8: Occurrence of  $\mathcal{T}$ -local optima along successful and unsuccessful trajectories, estimated via sampling. An horizontal bar marks the median.
Figure S9: Occurrence of non-monotonicities along successful and unsuccessful trajectories. An horizontal bar marks the median.
C Comparison to Additional Reinforcement Learning Algorithms
In this Section we benchmark three additional value-learning methods for offline goal-conditioned reinforcement learning, namely IQL[22], LAPO [6] and WGCSL [51]. Additional hyperparameters are tuned with a limited budget on maze_large. Performances are reported for the original algorithms (actor network, in blue), and in combination with model-based planning (in orange). Results are averaged over 12 seeds and reported in Figure S10. The three additional methods perform within the range of those evaluated in the main paper, suggesting that they do not avoid value estimation artifacts. Moreover, they do not consistently outperform TD3, confirming the competitiveness of simple actor-critic algorithms. Finally, we observe that these additional methods generally improve their performance when coupled with model-based planning, as shown in Section 4.
D Comparison to Graph-based Methods
Discrete structures such as graphs have been explored in order to model long-horizon temporal distances over MDPs. Eysenbach et al. [12] have shown how shortest paths on a graph constructed by uniformly sampling a replay buffer can effectively be used to select subgoals in goal-conditioned RL, introducing Search on the Replay Buffer (SORB). Our graph-based value aggregation scheme generally follows this framework, but with several differences, which we find to be crucial when learning from exploration trajectories. In this section, we expand the method description from Section 6.2, and outline these differences.
The first, and perhaps most crucial, difference lies in how shortest paths are leveraged to guide a goal-conditioned agent. In Eysenbach et al. [12], shortest paths are only used to retrieve the second state along them, which is then commanded to the agent as a subgoal. In practice, we found this to potentially result in the selection of subgoals which are too close, and therefore cause myopic
Figure S10: Extension of Figure 2 to additional value learning methods. We confirm that TD3 remains, on average, competitive with offline and model-based baselines when trained on curious exploration data.
behavior in the agent. For instance, in maze_large, the agent is encouraged to move at lower velocities to avoid overshooting its closest subgoal. While this could be addressed by conditioning the agent on subgoals further along the shortest path, this solution can be counterproductive, and force the agent to follow unreliable long-horizon value estimates. In contrast, we employ the entirety of the shortest path in order to compute long-horizon estimates by aggregating short-horizon estimates along this path. Conveniently, aggregates can be directly optimized through model-based planning, without needing to command intermediate subgoals, or to retrain the policy to match corrected value estimates.
A second difference with respect to existing works [12] lies in the sampling procedure for vertices in the graph. Instead of adopting uniform sampling, we sample states according to the inverse of a learned density estimate. In practice, we use an Exponential Kernel Density Estimator with bandwidth  $h = 20$ , where the distance between two states is simply the temporal distance  $d(s, s') = \log_{\gamma} V(s; s')$ . This makes the density estimates independent of the parameterization of the state space  $S$ , thus avoiding the strong assumption that Euclidean distances in the state space are meaningful.
Third and last, once the graph has been constructed, a pruning threshold  $V_{\mathrm{min}}$  is usually enforced to remove all edges encoding long-horizon distance estimates, which have low value estimates  $V(s'; s'') < V_{\mathrm{min}}$ . For this reason, in principle, this threshold should be as large as possible. However, an excessively high threshold would result in a disconnected graph, in which shortest paths may not exist. While this threshold is conventionally hand-tuned, we propose to select it automatically, by choosing the largest value  $V_{\mathrm{min}}$  that ensures connectivity from the agent's state to the goal. This can be efficiently computed through a Dijkstra-like algorithm. We refer to our codebase for its implementation.
Having outlined three key differences with respect to previous methods, we now provide an empirical evaluation of their effects on downstream performance in Figure S11. All success rates reported are obtained by optimizing value functions learned through TD3 with model-based planning and default hyperparameters.
Figure S11: Ablation and comparison of graph-based methods. Our value aggregation scheme largely outperforms a naive application of SORB to this setting, while removing the need to tune the graph pruning threshold  $V_{\mathrm{min}}$ . We find that density-aware sampling of vertices in the graph is crucial for succeeding in long-horizon tasks.
In particular, we compare our graph-based value aggregation scheme (Our aggregation) with an ablation that samples graph vertices uniformly from the replay buffer. We observe that this results in a sharp decrease in performance in long-horizon tasks. We also replace our value aggregation scheme with a SORB-like subgoal selection procedure for different pruning thresholds  $V_{\mathrm{min}}$ . We find that a naive application of SORB to this setting is not effective, and can be detrimental with respect to simple model-based planning.
Removing the necessity to tune  $V_{\mathrm{min}}$  leaves a single hyperparameter to be chosen for graph-based value aggregation, namely the size of the graph. This number is limited by the complexity of the shortest path search, and thus we use the default value of 1000 from the literature. However, we
found 100 vertices to already achieve sufficient coverage in pinpad, which therefore adopts this reduced hyperparameter.
In terms of limitations, as the graph aggregation procedure involves learned value estimates, it remains prone to failure due to compounding errors. However, two design choice introduce a degree of robustness. First, value estimates are only computed over states and goals that appear in the dataset the value function was trained on, which alleviates OOD issues. Second, the pruning step removes all edges encoding long-horizon estimates, which are empirically found to be less accurate [12]. Moreover, the tuning procedure we propose for the pruning threshold effectively minimizes the number of paths in the graph, while ensuring the existence of a solution to the optimization problem in Equation 5. Despite this, for particularly sparse datasets, the pruning can fail to remove inaccurate estimates, and the graph-based aggregates may reproduce the artifacts of the underlying value estimator. Further limitations of the graph search include increasing computational costs, and artifacts due to the effective discretization of the state space: while increasing the sample size can address the latter, it aggravates the former in a tradeoff.
E Sample Efficiency
All value functions in previous experiments are trained on 200k transitions gathered through curious exploration. This amount of data is around an order of magnitude lower than popular offline datasets [14], and has been chosen to be realistically collectable on real hardware. In practice, 200k timesteps at  $30\mathrm{Hz}$  amount to little under 2 hours of real time interaction. Nonetheless, we also evaluate methods from Figure 7 (main paper) on smaller datasets, namely with 50k and 100k transitions. We visualize results in Figure S12, and find them to remain consistent, while success rates decrease on smaller datasets, as expected.
Figure S12: Performance of an actor network (TD3) against model-based planning without (TD3+MBP) and with graph-based value aggregation (TD3+MBP+Aggregation), reported for value functions trained on datasets with 50k, 100k, and 200k samples.
F Applicability to General Offline Goal-Conditioned Settings
The motivation of this work is to address the challenge of extracting goal-conditioned behavior after curious exploration. Focusing on data gathered through unsupervised exploration has value in itself, arguably, as this requires minimal supervision and can enable real-world robot learning [29]. However, the proposed method can in principle be applied to general offline goal-conditioned settings. This section explores this possibility and provides empirical evidence.
Throughout this work, the exploratory nature of the data distribution is leveraged mainly in two points. First, as argued in Section 4.2, when learning from curious exploration data, simple actor-critic methods are, on average, competitive with model-based or offline methods. In standard offline settings with near-expert data, offline algorithms should be expected to be more effective [14]. However, within the realm of curious exploration, Section 2 suggests that TD3 is equally performant, and thus the proposed method uses it as core algorithm for value learning. Second, the proposed method relies on model-based planning, whose performance is heavily dependent on the quality and robustness of the learned model. Model-based planning is thus particularly promising when learning from curious exploration, as the unsupervised exploration phase already produces a dynamics model, which does not need to be trained from scratch. Moreover, as data has been collected to maximize the information gain of this dynamics model, it is potentially more robust [38]. When learning from offline datasets, a model is generally not available: while it can also be learned offline, training it to high accuracy can be difficult.
(c)
Figure S13: Ablation mixing the original dataset collected through curious exploration with expert or random uniform datasets. Our method can in practice be applied to different data sources, although may it suffer from weaker exploration in long-horizon environments (e.g. maze_large). Environments with more complex dynamics (e.g. fetch.push) largely benefit from the dynamics model learned during exploration (with active data collection).
We provide a minimal empirical study of the applicability of the proposed method on different data sources in Figure S13. Each subplot presents the performance of our full method (TD3 + model-based planning + graph-based value aggregation) when mixing the original curious exploration datasets with data collected from different sources (the size is fixed at 200k samples). In particular, a fraction of the trajectories is replaced with transitions from an expert policy (a,c), and from a uniformly random policy (b,d). Performance curves are recorded when using the dynamics model learned during curious exploration (green), and a dynamics model learned offline on the mixed data (violet); offline model learning was not tuned specifically. Due to time and compute constraints, we also present this analysis over two representative environments (maze_large in (a,b) and fetch.Push in (c,d)), and average over 9 random seeds.
In these settings, we observe that access to expert data does not hinder success rates (Figure S13a,c): interestingly, the proposed method shows robustness to shrinking data support. On the other hand, when introducing data collected by a uniform random policy (Figure S13b,d), performance decreases accordingly to the reduced exploration: while collecting random trajectories is sufficient in fetch.push, it is not comparable to curiosity-driven exploration for long-horizon tasks (i.e., maze_large). Furthermore, we note that models trained offline (in violet) achieve generally success when the complexity of the environment's dynamics increases (fetch.push), confirming that online training via disagreement maximization returns a more robust model [38].
G Comparison to Value-driven Exploration
In this Section, we provide an informative comparison with a related framework: POLO [27]. POLO also combines value learning and model-based planning. However, there are fundamental differences:
- POLO explores by maximizing the uncertainty of the value function; our framework leverages the uncertainty of the dynamics model.
- POLO assumes the availability of ground truth models for planning; our framework does not and adopts a learned dynamic model.
- POLO does not rely on graph-based components for long-horizon planning.
While an official implementation of POLO is not available, we can loosely reproduce it by integrating POLO's exploration signal in our framework, replacing TD3 with its value learning scheme, and disabling graph-based value aggregation, while still relying on iCEM for model-based planning and learning a dynamics model. We name the resulting method POLO*. We find that POLO* suffers from model inaccuracy when computing its value targets via MPPI optimization. We thus introduce a variant which uses on-policy value targets, which we refer to as POLO**.
Figure S14 reports success rates for POLO* and POLO** against our method (TD3 + model-based planning + graph-based aggregation) in all of our benchmark environments. Plots are averaged over 12 seeds, except for POLO* which uses 5 due to higher computational cost. Moreover, we
Figure S14: Comparison with adapted implementations of POLO - a privileged baseline that uses value-driven exploration and online value learning.