SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now formalize this intuition by defining such geometric features as  $\mathcal{T}$ -local optima. We then show that  $\mathcal{T}$ -local optima are not present in optimal goal-conditioned value functions, and are therefore
estimation artifacts. This justifies the method proposed in Section 6, which is designed to mitigate this class of artifacts.
Let us define the set operator  $\mathcal{T}:\mathbb{P}(\mathcal{S})\to \mathbb{P}(\mathcal{S})^2$  that returns the set of states reachable in one step from a given initial set of states  $S_0\subseteq S$ :
$$
\mathcal {T} \left(\mathcal {S} _ {0}\right) = \bigcup_ {s _ {o} \in \mathcal {S} _ {0}} \left\{s ^ {\prime} \in \mathcal {S} \mid \exists a \in \mathcal {A}: P \left(s ^ {\prime} \mid s _ {0}, a\right) > 0 \right\}. \tag {1}
$$
We can now formally introduce a class of geometric features, which we refer to as  $\mathcal{T}$ -local optima:
Definition 5.1. A state  $s^* \in S$  is a  $\mathcal{T}$ -local optimum point for a value function  $V(s)$  if and only if
$$
\max _ {s ^ {\prime} \in \mathcal {S}} V (s ^ {\prime}) > V (s ^ {*}) > \max _ {s ^ {\prime} \in \mathcal {T} (\{s ^ {*} \}) \backslash \{s ^ {*} \}} V (s ^ {\prime}).
$$
Intuitively, a  $\mathcal{T}$ -local optimum is a hill in the value landscape over the state space, where proximity is defined by the transition function of the MDP. The first inequality ensures that  $s^*$  does not maximize the value function globally. The second inequality guarantees that, if there is an action  $\bar{a} \in \mathcal{A}: P(s^*|s^*, a) = 1$ , this action would be chosen by greedily optimizing the value of the next-state (i.e.,  $\bar{a} = \operatorname{argmax}_{a \in \mathcal{A}} \int_{s' \in \mathcal{S}} V(s') P(s'|s^*, a)$ ), thus trapping the agent in  $s^*$ . By extending the previous definition to the multistep case, we can also conveniently define depth as the minimum number of steps required to reach a state with a higher value (i.e., escape the  $\mathcal{T}$ -local optimum).
Definition 5.2. A  $\mathcal{T}$ -local optimum point  $s^*$  has depth  $k \geq 1$  if  $k$  is the smallest integer such that
$$
V(s^{*}) <   \max_{s^{\prime}\in \mathcal{T}^{k + 1}(\{s^{*}\})}V(s^{\prime}).
$$
Intuitively, a  $\mathcal{T}$ -local optimum of depth  $k$  dominates the value of states reachable in up to  $k$  steps.
These definitions prepare the introduction of a core observation. Within the setting of goal-conditioned RL,  $\mathcal{T}$ -local optima are not an intrinsic feature of the value landscape, but rather an estimation artifact.
Proposition 5.3. Given a goal  $g \in \mathcal{G}$ , an optimal goal-conditioned value function  $V^{*}(s;g)$  has no  $\mathcal{T}$ -local optima.
The proof follows directly from Bellman optimality of the value function under standard assumptions of goal-conditioned RL introduced in Section 3.1. We report it in its entirety in Appendix A.1.
Figure 4: Learned value as a function of  $x - y$  coordinates in maze_large. Choosing actions that maximize the value of the next state traps the agent in a corner, i.e. a  $\mathcal{T}$ -local optimum point.
We remark that, in principle, showing that a state  $s$  is a  $\mathcal{T}$ -local optimum point requires evaluating the value function over the support of the next state distribution, which would require access to the true dynamics of the MDP. Moreover, exhaustive evaluation is unfeasible in continuous state spaces. However, one can bypass these issues by instead estimating, rather than proving, the presence of  $\mathcal{T}$ -local optima. Given a candidate  $\mathcal{T}$ -local optimum point  $s$ ,  $V(\cdot; g)$  can be evaluated on (i) random states  $s_r$  sampled from the exploration data  $\mathcal{D}$  instead of the state space  $S$  and (ii) samples  $s'$  from the next state distribution, accessed through the dynamic model  $\hat{P}(s'|s, a)$  with  $a \sim \mathcal{U}(\mathcal{A})$ . The maximum operators in Definition 5.1 can then be estimated through sampling; if both inequalities hold,  $s$  can be estimated to be a  $\mathcal{T}$ -local optimum point.
Having defined  $\mathcal{T}$ -local optima, and shown that they are learning artifacts, we now investigate their impact on goal-reaching performance. We train  $n = 30$  bootstrapped value estimators for all environments, and for each of them we generate trajectories through model-based planning with horizon  $H$ . We then estimate the occurrence of  $\mathcal{T}$ -local optima of depth  $k \geq H$  by testing each state along each trajectory, as described above. This is reported in Figure 5, separately for successful and unsuccessful trajectories. We observe that, in most environments, trajectories which avoid  $\mathcal{T}$ -local optima generally tend to be more likely to succeed; this tendency tends to fade in environments which only require short-term planning (e.g., fetch.push). We note that these estimates are sampled-based, and thus noisy by nature, especially in low-data regimes; we thus expand on this analysis in Appendix B. For illustrative purposes, we also visualize a  $\mathcal{T}$ -local optimum in the value landscape over maze_large in Figure 4.
$\mathbb{P}(\cdot)$  represents a power set.
Figure 5: Occurrence of  $\mathcal{T}$ -local optima of depth  $k \geq H$  along successful and unsuccessful trajectories generated through model-based planning with horizon  $H = 15$ , estimated via sampling. An horizontal bar marks the median.
6 Mitigating Local and Global Value Artifacts
Having shown that  $\mathcal{T}$ -local optima are estimation artifacts, we now focus on categorizing and mitigating their occurrence. A first observation, which confirms the results in Section 4, is that model-based planning is robust to  $\mathcal{T}$ -local optima of small depth. This is further argued in Section 6.1. However, this does not address the occurrence of global estimation artifacts (i.e.  $\mathcal{T}$ -local optima of large depth). We thus propose a graph-based value aggregation scheme for this purpose in Section 6.2. Finally, in Section 6.3 we empirically show how combining these two components can address estimation artifacts both locally and globally, and consistently improves goal-reaching performance.
6.1 Model-based Planning for Local Value Artifacts
We will now provide further arguments for the effectiveness of model-based planning over inaccurate value landscapes, supported by Figure 2. Model-based planning is a model-predictive control (MPC) algorithm, which finds an action sequence  $(a_0,\dots a_{H - 1})$  by optimizing the RL objective through a  $H$ -step value estimate for a given horizon  $H$ , dynamics model  $\hat{P}$ , state  $s_0$ , and goal  $g$ :
$$
(a _ {0}, \dots , a _ {H - 1}) = \underset {(a _ {0}, \dots , a _ {H - 1}) \in \mathcal {A} ^ {H}} {\operatorname {a r g m a x}} \sum_ {i = 0} ^ {H - 1} \gamma^ {i} R (s _ {i}, g) + \gamma^ {H} V (s _ {H}; g) \quad \text {w i t h} s _ {i + 1} \sim \hat {P} (\cdot | s _ {i}, a _ {i}), \tag {2}
$$
and executes the first action  $a_0$ . We remark that in our setting, due to the extreme sparsity of  $R(s;g) = \mathbf{1}_{s = g}$  in continuous spaces, the objective in Equation 2 simplifies to the maximization of  $V(s_H;g)$  alone. For the same reason, it is easy to show that an actor network trained through policy gradient is optimizing the same objective as model-based planning with horizon  $H = 1$ :
$$
a _ {0} = \underset {a _ {0} \in \mathcal {A}} {\operatorname {a r g m a x}} Q (s _ {0}, a _ {0}; g) = \underset {a _ {0} \in \mathcal {A}} {\operatorname {a r g m a x}} R (s _ {0}; g) + \gamma V (s _ {1}; g) = \underset {a _ {0} \in \mathcal {A}} {\operatorname {a r g m a x}} V (s _ {1}; g), \tag {3}
$$
with  $s_1 \sim P(\cdot | s_0, a_0)$ . Under certain assumptions, such as exact dynamics estimation (i.e.,  $\hat{P} = P$ ), exact optimization for Equation 2 and existence of self-loops in the MDP, it is possible to show that (i) model-based planning from a  $\mathcal{T}$ -local optimum point of depth  $k$  is guaranteed to fail if  $H \leq k$  and (ii) model-based planning is guaranteed to reach a state with a higher value if the horizon  $H$  is greater than the depth of the largest  $\mathcal{T}$ -local optima over the value landscape. For a complete argument, see Appendix A.3. As a consequence, actor networks can be trapped in all local optima, while model-based planning with horizon  $H > 1$  can potentially escape local optima of depth  $k < H$ , and is therefore potentially more robust to such estimation artifacts, as shown empirically in Figure 2.
6.2 Graph-based Aggregation for Global Value Artifacts
While we have shown that model-based planning is robust to local artifacts, the correction of global estimation artifacts, which span beyond the effective planning horizon  $H$ , remains a significant challenge. We propose to address it by grounding the value function in states observed during exploration, such that long-horizon value estimates are computed by aggregating short-horizon (i.e.,
local) estimates. Given the current state  $s \in S$  and the goal  $g \in G$ , we build a sparse graph by sampling vertices from the empirical state distribution  $\mathcal{D}_s$  from the replay buffer, and connecting them with edges weighted by the learned goal-conditioned value  $\hat{V}$ :
$$
\mathbf {G} = (\mathcal {V}, \mathcal {E}) \text {w i t h} \mathcal {V} \subset \mathcal {D} _ {s} \cup \{s, g \} \text {a n d} \mathcal {E} = \left\{\left(s ^ {\prime}, s ^ {\prime \prime}, \hat {V} \left(s ^ {\prime}; s ^ {\prime \prime}\right)\right) \forall \left(s ^ {\prime}, s ^ {\prime \prime}\right) \in \mathcal {V} \times \mathcal {V} \right\}. \tag {4}
$$
Crucially, we then prune the graph by removing all edges that encode long-horizon estimates, i.e., edges whose weight is below a threshold  $V_{\mathrm{min}}$ . In practice, we set  $V_{\mathrm{min}}$  to the maximum value which still ensures connectivity between the agent's state  $s$  and the goal  $g$  along the graph. This can be computed efficiently, as detailed in Appendix D. Once the graph has been constructed and pruned, we simply compute value estimates by minimizing the product of value estimates along all paths to the goal.
$$
\bar {V} (s; g) = \min  _ {\left(s _ {0}, \dots , s _ {T}\right)} \prod_ {i = 0} ^ {T - 1} \hat {V} \left(s _ {i}; s _ {i + 1}\right) \text {w h e r e} \left(s _ {0}, \dots , s _ {T}\right) \text {i s a p a t h b e t w e e n} s \text {a n d} g \text {o n} \mathbf {G} \tag {5}
$$
This value aggregation scheme differentiates itself from existing methods [12] in its vertex sampling phase, which relies on density estimation to address skewed state distributions, and in its pruning phase, which does not depend on hyperparameters. Crucially, the final value aggregate can be directly adapted as a cost function for model-based planning, and is not used as a proxy for subgoal selection. These differences and their consequences are described extensively in Appendix D.
In practice, this scheme can mitigate global  $\mathcal{T}$ -local optima by replacing long-horizon value estimates through an aggregation of short-horizon estimates. Such short-horizon estimates have been shown to be generally more accurate, and useful in containing value overestimation issues [12], which can be considered a generalization of  $\mathcal{T}$ -local optima.
For reference, we now provide a visual intuition for the effect of the graph-based value aggregation scheme we propose. We highlight this in maze_large, as it is prone to global estimation artifacts, and allows straightforward visualization. The blue column in Figure 6 plots value functions  $\{\hat{V}(s; g_i)\}_{i=0}^2$  learned through TD3 against  $x-y$  coordinates for three different goals. The green column reproduces the same plots, but estimates values through graph-based value aggregation, as described in this Section, resulting in corrected estimates  $\{\bar{V}(s; g_i)\}_{i=0}^2$ . We further compute expert trajectories for all three goals, in red, and plot  $\hat{V}$  and  $\bar{V}$  along them, resulting in the three plots on the right. When comparing value estimates with optimal values ( $V^*$ , in orange, computed in closed form), we observe that graph-based aggregation results in more accurate estimates. Crucially, it is able to address global estimation artifacts, which result in a "bleeding" effect for  $\hat{V}(s; g_0)$  (on the left), or in large "hills" in  $\hat{V}(s_t; g_0)$  (on the right, in blue).
Figure 6: Learned values  $\hat{V}(s; g_i)$  as a function of  $x - y$  coordinates in maze_large (blue column). Graph-based value aggregation results in corrected estimates  $\hat{V}(s; g_i)$  (green column). Values along optimal trajectories are compared to optimal value functions  $V^*(s_t; g_i)$  on the right.
Figure 7: Performance of an actor network (TD3) against model-based planning without (TD3+MBP) and with graph-based value aggregation (TD3+MBP+Aggregation). Model-based planning on aggregated values addresses both local and global estimation artifacts, and performs well across diverse environments.
6.3 Experimental Evaluation
Having evaluated diverse value learning algorithms, and described a class of harmful estimation artifacts, we now set out to evaluate how such artifacts can be addressed both locally and globally. Figure 7 compares the performance of actor networks with that of model-based planning, with and without graph-based value aggregation. We observe that model-based planning improves performance in most tasks, as it addresses  $\mathcal{T}$ -local optima of limited depth. However, it fails to correct global artifacts in long-horizon tasks (i.e., maze_large and pinpad). In this case, we observe that graph-based value aggregation increases success rates significantly. Furthermore, we observe that our method, which combines these two components, consistently achieves strong performance across diverse environments.
In each plot, a dashed line represents the performance of model-based planning with an external sparse cost (for some  $\epsilon > 0$ ). On one hand, its strong performance in environments with shorter optimal trajectories confirms the accuracy of the learned model. On the other, its failure for long-horizon tasks shows the limitation of planning in the absence of a learned value function, which has been an important motivation for this work.
This work is concerned with the challenge of achieving goals after an unsupervised exploration phase, without any further interaction with the environment. In this challenging setting, we observe that simple actor-critic methods are surprisingly competitive with model-based, or offline algorithms, but their actor networks are often suboptimal due to estimation artifacts in the learned value function. After characterizing these artifacts by leveraging the goal-conditioned nature of the problem, we show how model-based planning can address this problem locally, graph-based value aggregation can apply global corrections, and their combination works consistently across diverse environments.
Limitations With respect to simply querying actor networks, this method comes at a cost. In particular, it increases computational requirements and risks introducing further artifacts due to model exploitation, or the discretization involved in graph-based aggregation (see Appendix D). Furthermore, the proposed method is only applied at inference and, while it's able to mitigate learning artifacts, it does not directly correct them by updating the value estimator. Nevertheless, within our settings, we find the impact of these limitations to be, in practice, modest.
Our method represents a strong practical approach for goal-reaching after curious exploration, and sheds light on interesting properties of both offline and goal-conditioned problems. An exciting future avenue would further leverage these formal insights, possibly through a tighter integration of value learning and value correction. We thus believe that further exploring the interplay between a theoretically sound analysis of the properties of goal-conditioned value function, and the design of practical algorithms, represents a rich research opportunity.
We thank Andreas Krause, Nico Gurtler, Pierre Schumacher, Núria Armengol Urpí and Cansu Sancaktar for their help throughout the project, and the anonymous reviewers for their valuable feedback. Marco Bagatella is supported by the Max Planck ETH Center for Learning Systems. Georg Martius is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645. We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01IS18039B).
[1] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in Neural Information Processing Systems, 2017.
[2] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, 2016.
[3] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
[4] H. Charlesworth and G. Montana. Plangan: Model-based planning with sparse rewards and multiple goals. In Advances in Neural Information Processing Systems, 2020.
[5] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach, R. C. Julian, C. Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In International Conference on Machine Learning, 2021.
[6] X. Chen, A. Ghadirzadeh, T. Yu, J. Wang, A. Y. Gao, W. Li, L. Bin, C. Finn, and C. Zhang. Lapo: Latent-variable advantage-weighted policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 2022.
[7] N. Chentanez, A. Barto, and S. Singh. Intrinsically motivated reinforcement learning. Advances in Neural Information Processing Systems, 2004.
[8] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 2018.
[9] P. Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613-624, 1993.
[10] A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return, then explore. Nature, 590(7847):580-586, 2021.
[11] S. Emmons, B. Eysenbach, I. Kostrikov, and S. Levine. Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations, 2022.
[12] B. Eysenbach, R. R. Salakhutdinov, and S. Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems, 2019.
[13] Farama-Foundation. Gymnasium-robotics. https://github.com/Farama-Foundation/Gymnasium-Robotics, 2023.
[14] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[15] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, 2018.
Supplementary Material to Goal-conditioned Offline Planning from Curious Exploration
A.1 Proposition 5.3
Let us consider an arbitrary goal  $g \in \mathcal{G}$  and the optimal goal-conditioned value function  $V^{*}(s;g)$ . Proposition 5.3 states that  $V^{*}(s;g)$  has no  $\mathcal{T}$ -local optima. The proof follows directly from the Bellman optimality condition, assuming that (1)  $R(s;g) = \mathbf{1}_{d(s,g) \leq \epsilon}$  for some  $\epsilon \geq 0$  and distance function  $d(\cdot, \cdot)$ , and that (2) the episode terminates when the goal is achieved.
For a candidate state  $\tilde{s}$ , let us assume that the first inequality of the condition for being a  $\mathcal{T}$ -local optimum point is met:  $\max_{s' \in S} V^{*}(s'; g) > V^{*}(\tilde{s}; g)^{3}$ . It is now sufficient to show that the second inequality (i.e.,  $V^{*}(\tilde{s}; g) > \max_{s' \in \mathcal{T}(\{\tilde{s}\}) \setminus \{\tilde{s}\}} V^{*}(s'; g)$ ) cannot hold, and therefore  $\tilde{s}$  cannot be a  $\mathcal{T}$ -local optimum point. According to Bellman optimality, we have that
$$
V ^ {*} (\tilde {s}; g) = R (\tilde {s}; g) + \gamma (\tilde {s}; g) \max  _ {a \in \mathcal {A}} \underset {s ^ {\prime} \sim \mathcal {P} (\cdot | a, \tilde {s})} {\mathbb {E}} V ^ {*} (s ^ {\prime}; g) \tag {S6}
$$
where we use  $\gamma(s;g) = \gamma(1 - R(s;g))$ :  $(1 - R(s;g))$  acts as the termination signal, to enforce that value propagation stops when reaching the goal (and the episode ends). If  $R(\tilde{s};g) = 1$ , then  $V^{*}(\tilde{s};g) = 1$ , which would break the first inequality, since  $\max_{s' \in S} V^{*}(s';g) \leq 1$ . This implies that  $R(\tilde{s};g) = 0$ . As a consequence, Equation S6 simplifies to
$$
\begin{array}{l} V^{*}(\tilde{s};g) = \gamma \max_{a\in \mathcal{A}}\underset {s^{\prime}\sim \mathcal{P}(\cdot |a,\tilde{s})}{\mathbb{E}}V^{*}(s^{\prime};g) \\ \leq \gamma \max  _ {s ^ {\prime} \in \mathcal {T} (\{\tilde {s} \})} V ^ {*} (s ^ {\prime}; g) \tag {S7} \\ \end{array}
$$
We can now differentiate two cases. If  $\max_{s' \in \mathcal{T}(\{\tilde{s}\})} V^{*}(s'; g) = 0$  (i.e., the goal cannot be reached), then we also have that  $V^{*}(\tilde{s}; g) = \max_{s' \in \mathcal{T}(\{\tilde{s}\}) \setminus \{\tilde{s}\}} V^{*}(s'; g) = 0$ , and  $\tilde{s}$  is not a  $\mathcal{T}$ -local optimum point.
If the goal is instead reachable from at least one state in the next state distribution (i.e.,  $\max_{s' \in \mathcal{T}(\{\tilde{s}\})} V^{*}(s'; g) > 0$ ), we have that
$$
\begin{array}{l} V ^ {*} (\tilde {s}; g) \leq \gamma (\tilde {s}; g) \max  _ {s ^ {\prime} \in \mathcal {T} (\{\tilde {s} \})} V ^ {*} (s ^ {\prime}; g) \\ <   \max  _ {s ^ {\prime} \in \mathcal {T} (\{\tilde {s} \})} V ^ {*} (s ^ {\prime}; g) \tag {S8} \\ = \max _ {s ^ {\prime} \in \mathcal {T} (\{\hat {s} \}) \backslash \{\hat {s} \}} V ^ {*} (s ^ {\prime}; g). \\ \end{array}
$$
The second inequality follows from the goal-conditioned assumption that  $0 < \gamma < 1$ , and the final equality is due to the fact that  $V^{*}(\tilde{s};g) < \max_{s^{\prime}\in \mathcal{T}(\{\tilde{s}\})}V^{*}(s^{\prime};g)$  implies that  $V^{*}(\tilde{s};g)\neq \max_{s^{\prime}\in \mathcal{T}(\{\tilde{s}\})}V^{*}(s^{\prime};g)$ . Thus,  $V^{*}(\tilde{s};g) > \max_{s^{\prime}\in \mathcal{T}(\{\tilde{s}\})\setminus \{\tilde{s}\}}V^{*}(s^{\prime};g)$  does not hold, and  $\tilde{s}$  is not a  $\mathcal{T}$ -local optimum point.
A.2 Optimization of Optimal Goal-conditioned Value Functions
Section 4 argues that, assuming that the objective for actor networks is well optimized, the failure of naive actor-critic algorithms is due to estimation errors in the critic network. Accordingly,
3While a supremum operator  $\sup (\cdot)$  would be required in open domains, for simplicity we use  $\max (\cdot)$  throughout this work.
we show that, under common assumptions, a state-space trajectory which optimizes an optimal goal-conditioned value function  $V^{*}(s;g)$  is guaranteed to achieve its goal  $g\in \mathcal{G}$ . Furthermore, minimization of the actor loss results in an actor network which produces this exact trajectory.
Given a goal  $g \in \mathcal{G}$ , and an initial state  $s_0 \in S$ , we make the following assumptions:
1.  $V^{*}(s_{0};g) > 0$ , i.e., the goal can be reached from the initial state.
2. For all  $s \in S, a \in \mathcal{A}, \exists s' \in S : P(s'|s, a) = 1$ , i.e., the MDP transitions deterministically.
We now consider the state-space trajectory  $(s_i)_{i=0}^T$  which optimizes  $V^*$  under the dynamics of the MDP, that is
$$
s _ {i + 1} = \underset {s ^ {\prime} \in \tilde {T} (\{s _ {i} \})} {\operatorname {a r g m a x}} V ^ {*} \left(s ^ {\prime}; g\right) \quad \forall 0 \leq i <   T. \tag {S9}
$$