SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Marco Bagatella
ETH Zürich & Max Planck Institute for Intelligent Systems
mbagatella@ethz.ch
Georg Martius
University of Tubingen & Max Planck Institute for Intelligent Systems
georg.martius@uni-tuebingen.de
Curiosity has established itself as a powerful exploration strategy in deep reinforcement learning. Notably, leveraging expected future novelty as intrinsic motivation has been shown to efficiently generate exploratory trajectories, as well as a robust dynamics model. We consider the challenge of extracting goal-conditioned behavior from the products of such unsupervised exploration techniques, without any additional environment interaction. We find that conventional goal-conditioned reinforcement learning approaches for extracting a value function and policy fall short in this difficult offline setting. By analyzing the geometry of optimal goal-conditioned value functions, we relate this issue to a specific class of estimation artifacts in learned values. In order to mitigate their occurrence, we propose to combine model-based planning over learned value landscapes with a graph-based value aggregation scheme. We show how this combination can correct both local and global artifacts, obtaining significant improvements in zero-shot goal-reaching performance across diverse simulated environments.
While the standard paradigm in reinforcement learning (RL) is aimed at maximizing a specific reward signal, the focus of the field has gradually shifted towards the pursuit of generally capable agents [1, 20, 40]. This work focuses on a promising framework for learning diverse behaviors with minimal supervision, by decomposing the problem into two consecutive phases. The first phase, referred to as exploratory [28, 43] or intrinsic [38], consists of unsupervised exploration of an environment, aiming to collect diverse experiences. These can be leveraged during the second phase, called exploitative or extrinsic, to distill a family of specific behaviors without further environment interaction. This paradigm unifies several existing frameworks, including zero-shot deep reinforcement learning [28, 43], Explore2Offline [23], ExORL [52] and model-based zero-shot planning [38].
An established technique for the intrinsic phase is curiosity-based exploration, which leverages some formulation of surprise as a self-supervised reward signal. In practice, this is often done by estimating the uncertainty of forward dynamic predictions of a learned model [49]. At the end of the exploratory phase, this method produces a sequence of collected trajectories, as well as a trained dynamics model. At its core, our work focuses on the subsequent extrinsic phase, and is concerned with extracting strong goal-conditioned behavior from these two products, without further access to the environment.
Figure 1: This work focuses on extracting goal-conditioned behavior from exploratory trajectories  $\mathcal{D}$  and a trained dynamics model  $\hat{P}$ . We propose to learn a value function through goal-conditioned RL, plan with zero-order trajectory optimization to mitigate local estimation artifacts, and aggregate value estimates along a graph in order to correct global estimation artifacts.
In its simplest form, this corresponds to learning to control the agent's future state distribution in order to achieve a given goal or, more concretely, to visit a particular state. Existing methods tackle this problem without additional learning through model-based online planning [38], or rely on learning an amortized policy, either during exploration [28, 43] or in a fully offline manner [23, 52]. The former class of approaches requires the definition of a hand-crafted cost function, without which long-horizon, sparse tasks would remain out of reach. On the other hand, the latter class does not rely on a well-shaped cost function, but may suffer from myopic behavior in the absence of explicit planning, and from estimation errors in the learned value function.
As we do not assume access to external task-specific cost functions, we first investigate the effectiveness of goal-conditioned RL algorithms for learning a value function. In doing so, we empirically extend the results obtained by Yarats et al. [52] to previously unstudied model-based and goal-conditioned settings. We thus confirm that RL methods not involving offline corrections can be competitive when trained on offline exploratory data. Additionally, by a formal analysis of the geometry of optimal goal-conditioned value functions, we pinpoint a geometric feature of inaccurate value landscapes, which we refer to as  $\mathcal{T}$ -local optima. After showing the detrimental effects of such learning artifacts to the performance of goal-conditioned RL, we propose to mitigate them through a novel combination of model-based planning with a practical graph-based aggregation scheme for learned value estimates. Crucially, we show how these two components address the same estimation issue on different scales, that is, respectively, locally and globally. We provide a graphical overview in Figure 1 for a simple maze environment.
The contributions of this work can be organized as follows:
- We provide an empirical evaluation of goal-conditioned reinforcement learning algorithms when trained on the outcomes of an unsupervised exploration phase.
- We pinpoint a class of estimation artifacts in learned goal-conditioned value functions, which explains the suboptimality of naive, model-free reinforcement learning approaches in this setting.
- We propose to combine model-based planning with graph-based value aggregation to address the occurrence of both local and global value artifacts.
- We provide the first fully offline method for achieving goals after a curious exploration phase.
After the relating our work to existing literature, we evaluate goal-conditioned RL methods when learning from curious exploration (Section 4). We then pinpoint a class of estimation artifacts that are not compatible with optimal goal-conditioned value functions and analyze them through the lens of  $\mathcal{T}$ -local optima (Section 5). Equipped with these insights, we show how our practical method can mitigate such artifacts (Section 6). A compact description of the resulting method, highlighting its intrinsic and extrinsic phases, can be found in Appendix H.
Unsupervised Exploration The challenge of exploration has been at the core of reinforcement learning research since its inception. Within deep reinforcement learning, successful strategies are
often derived from count-based [2], prediction-based [41] or direct exploration techniques [10], with the first two traditionally relying on intrinsic motivation [7] computed from state visitation distributions or prediction errors (i.e., surprise), respectively. Depending on the entity on which prediction error is computed, methods can focus on estimating forward dynamics [32, 38, 43, 44], inverse dynamics [33] or possibly random features of the state space [3]. Another important distinction falls between methods that retroactively compute surprise [33] or methods that compute future expected disagreement [38, 43], and proactively seek out uncertain areas in the state space. Overall, such techniques compute an additional intrinsic reward signal, which can be summed to the extrinsic environment reward signal, or used alone for fully unsupervised exploration. In this work, we build upon efficient exploration methods that are proactive, and rely on prediction disagreement of a forward dynamics model [38, 43, 44]. Nevertheless, our method could be easily applicable in conjunction with any aforementioned strategy, as long as it can produce exploratory data, and a dynamics model, which could potentially also be learned offline.
Zero-shot Frameworks Unsupervised exploration techniques are capable of efficiently sampling diverse trajectories, without extrinsic reward signals. As a result, they are a promising source of data to learn flexible behaviors. Several works in the literature have proposed that a task-agnostic unsupervised exploration phase be followed by a second phase, optimizing a task-specific reward function only provided a posteriori. This has been formalized as zero-shot reinforcement learning [9, 47, 48]. Notably, this framework conventionally forbids task-specific fine-tuning, explicit planning or computation during the second phase. On the applied side, agents endowed with learned world models [28, 43] have shown promise for zero-shot generalization without additional learning or planning. Other works are less strict in their constraints: Sancaktar et al. [38] also propose an intrinsic exploration phase followed by zero-shot generalization, but the latter relies on task-specific online planning. Furthermore, two concurrent works [23, 52] study the application of offline deep reinforcement learning for the second phase. However, these methods are not designed for goal-conditioned tasks, and thus require an external task-specific reward function to perform relabeling during training. Our framework fits within this general research direction, in that it also involves task-agnostic and task-specific components. The initial phase is task-agnostic: we rely on unsupervised exploration, and train a universal value estimator [40] completely offline, without access to an external task-specific reward signal. In the second phase, we perform model-based planning via zero-order trajectory optimization to control the agent. We remark that, while this requires significant computation, existing algorithms are fast enough to allow real-time inference [35].
Goal-Conditioned Reinforcement Learning The pursuit of generally capable agents has been a ubiquitous effort in artificial intelligence research [36, 42, 45]. Goal-conditioned RL represents a practical framework for solving multiple tasks, by describing each task through a goal embedding, which can then condition reward and value functions. Deep RL methods can then be used to estimate values by providing the goal as an additional input to their (often neural) function approximators [40]. A ubiquitous technique in these settings is that of goal relabeling [1], which substitutes a trajectory's goal with one sampled a posteriori. Goal-conditioned methods have been shown to be successful in the offline setting [46], often when coupled with negative sampling methods [5], but have been mostly evaluated on standard offline datasets. An alternative line of work frames the problem as supervised learning instead [11], and shows connections to the original reinforcement learning objective [51]. Additionally, several works have explored the combination of goal-conditioned learning with model-based planning [4, 25, 31]. This paper studies the natural application of goal-conditioned RL after an unsupervised exploration phase, which allows a straightforward definition of the multiple tasks to be learned. Moreover, our main formal insights in Section 5 build on top of common assumptions in goal-conditioned RL.
Graph-based Value Estimation Several existing works rely on graph abstractions in order to retrieve global structure and address value estimation issues. In particular, Eysenbach et al. [12] and Savinov et al. [39] propose high-level subgoal selection procedures based on shortest paths between states sampled from the replay buffer. Another promising approach proposes to build a graph during training, and to leverage its structure when distilling a policy [30]. In contrast, our method relies on discrete structures only at inference, and directly integrates these structures into model-based planning, thus removing the need for subgoal selection routines.
3.1 Background and Notation
The environments of interest can be modeled as Markov decision processes (MDP)  $\mathcal{M} = \{\mathcal{S},\mathcal{A},P,R,\mu_0,\gamma \}$ , where  $\mathcal{S}$  and  $\mathcal{A}$  are (possibly continuous) state and action spaces,  $P:\mathcal{S}\times \mathcal{A}\to \Omega (\mathcal{S})$  is the transition probability distribution $^1$ ,  $R:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$  is a reward function,  $\mu_0\in \Omega (\mathcal{S})$  is an initial state distribution, and  $\gamma$  is a discount factor.
Goal-conditioned reinforcement learning augments this conventional formulation by sampling a goal  $g \sim \mu_{\mathcal{G}}$ , where  $p_{\mathcal{G}}$  is a distribution over a goal space  $\mathcal{G}$ . While goal abstraction is possible, for simplicity of notation we consider  $\mathcal{G} \subseteq S$  (i.e., goal-conditioned behavior corresponds to reaching given states). The reward function is then defined as  $R(s,a;g) = R(s;g) = \mathbf{1}_{d(s,g) \leq \epsilon}$  for some  $\epsilon \geq 0$ , and under some distance metric  $d(\cdot)$ . We additionally assume that episodes terminate when a goal is achieved, and the maximum undiscounted return within an episode is thus one. It is now possible to define a goal-conditioned policy  $\pi : S \times \mathcal{G} \to \Omega(\mathcal{A})$  and its value function  $V^{\pi}(s_0;g) = \mathbb{E}_{P,\pi} \sum_{t=0}^{\infty} \gamma^t R(s_t, g)$ , with  $s_t \sim P(\cdot | s_{t-1}, a_{t-1})$ ,  $a_t \sim \pi(s_t;g)$ . The objective of a goal-conditioned RL agent can then be expressed as finding a stochastic policy  $\pi^{\star} = \operatorname{argmax}_{\pi} \mathbb{E}_{g \sim p_{\mathcal{G}}, s \sim p_0} V^{\pi}(s;g)$ .
When learning from curious exploration, we additionally assume the availability of a set  $\mathcal{D} = \{(s_0,a_0,s_1,a_1,\ldots ,s_{T_i})_i\}_{i = 1}^N$  of  $N$  exploratory state-action trajectories of possibly different lengths  $T_{i}$ , as well as access to an estimate of the probabilistic transition function  $\hat{P}\approx P$ , i.e., a dynamics model.
3.2 Data Collection and Experimental Framework
As our contributions are motivated by empirical observations, we also introduce our experimental pipeline early on. Algorithms are evaluated in four MDP instantiations, namely maze_large and kitchen from D4RL [14], fetch.push from gymnasium-robotics [13] and an adapted pinpad environment [18]. These environments were chosen to drastically differ in dynamics, dimensionality, and task duration. In particular, maze_large involves the navigation to distant goals in a 2D maze by controlling the acceleration of a point mass, kitchen requires controlling a 9 DOF robot through diverse tasks, including manipulating a kettle and operating switches, fetch.push is a manipulation task controlled through operation space control, and pinpad requires navigating a room and pressing a sequence of buttons in the right order.
For each environment, we collect 200k exploratory transitions by curious exploration, corresponding to  $\sim 2$  hours of real-time interaction at  $30\mathrm{Hz}$ , which is an order of magnitude less than existing benchmarks [14, 24]. We stress that this amount of data is realistically collectible in the real world with minimal supervision, thus providing a strong benchmark for data-efficient learning of general agents. Moreover, the nature of the data is significantly different from standard offline datasets, which often contain high-quality demonstrations [14] that can be distilled into policies through naive imitation.
The environment's dynamics are estimated through an ensemble of stochastic neural networks, similar to the one proposed in Chua et al. [8]. The intrinsic exploration signal is ensemble disagreement [38, 43, 49], as described in Appendix I.2. All algorithms are evaluated by success rate, that is, the ratio of test episodes in which the commanded goal is achieved. Details can be found in Appendix I.
4 Value Learning from Curious Exploration
Curious exploration produces a learned dynamics model  $\hat{\mathcal{P}}$  and a set  $\mathcal{D}$  of trajectories, lacking both goal and reward labels. In order to achieve arbitrary (and potentially distant) goals at inference time, and considering that a task-specific cost function is not externally provided, it is natural to resort to learning and optimizing a value function. This section evaluates established methods for value learning and studies the quality of their estimates, thus motivating the analysis and method proposed in Sections 5 and 6.
We use  $\Omega(S)$  to represent the space of probability distributions over  $S$ .
A standard approach to train a goal-conditioned value function from unlabeled data involves two steps. The first step consists of a goal-relabeling procedure, which associates each training sample with a goal selected a posteriori, and computes a self-supervised reward signal from it, as briefly explained in Section 4.1. The second step consists of extracting a goal-conditioned value function by applying off-policy RL algorithms on the relabeled data, as analyzed in Section 4.2. While this work focuses on data collected through curious exploration, the methods described can also be applied to a general offline goal-conditioned reinforcement learning settings, for which we provide a minimal study in Appendix F.
4.1 Relabeling Curious Exploration through Self-supervised Temporal Distances
We follow the framework proposed by Tian et al. [46] to relabel exploration trajectories, and compute self-supervised reward signals by leveraging temporal distances, without assuming access to external task-specific reward functions. Training samples  $(s_t, a_t, s_{t+1})$  are sampled uniformly from the replay buffer  $\mathcal{D}$ , and need to be assigned a goal  $g$  and a reward scalar  $r$ . Each training sample is assigned a positive goal with probability  $p_g$ , and a negative goal otherwise. Positive goals are sampled as future states from the same trajectory:  $g = s_{t+\tau}$ , where  $\tau \sim \mathrm{Geo}(p_{\mathrm{geo}})$  follows a geometric distribution with parameter  $p_{\mathrm{geo}}$ . Negative goals are instead sampled uniformly from the entire dataset, while Tian et al. [46] rely on prior knowledge on the semantics of the state space to extract challenging goals. Rewards for all state-goal pairs are simply computed using the reward function  $r = R(s_{t+1}; g) = \mathbf{1}_{s_{t+1} = g}$ : the reward is non-zero only for positive goals sampled with  $\tau = 1$ . This corresponds to a minorizer for the class of reward functions  $R(s_{t+1}; g) = \mathbf{1}_{d(s_{t+1}, g) \leq \epsilon}$ , as setting  $\epsilon = 0$  effectively removes the necessity to define a specific distance function  $d$  over state and goal spaces. We remark that while its extreme sparsity prevents its optimization via zero-shot model based planning, it remains functional as a reward signal for training a value function in hindsight.
4.2 Value Estimation and Suboptimality
Now that sampling and relabeling procedures have been defined, we focus our attention on which methods can effectively retrieve a goal-conditioned value function  $V(s; g)$ . In particular, we evaluate off-policy actor-critic algorithms, which are in principle capable of extracting the optimal value function (and policy) from trajectories generated by a suboptimal policy (as the one used for exploration). We thus ask the following questions:
1. Which off-policy actor-critic algorithm is most suitable for learning a goal-conditioned value function, purely offline, from data collected through unsupervised exploration?
2. How close to optimality is the learned value function?
To answer the first question, we consider several off-policy actor-critic algorithms, and evaluate the learned value function through the success rate of an agent which seeks to optimize it. The choice of algorithms should adhere to the offline setting, as well as the availability of an estimated dynamics model  $\hat{P}$ . We thus evaluate TD3 [15] as default model-free actor-critic algorithm, and
Figure 2: Performance of actor networks (blue) and model-based planning (orange) on value functions trained through various actor-critic algorithms from the same curious exploration datasets.
compare its performance to CRR [50], a state-of-the-art offline method [16]. We also evaluate two model-based methods, namely MBPO [19] and MOPO [53], the latter of which introduces an uncertainty-based penalty for offline learning. Further value-learning baselines are presented in Appendix C. All methods have access to the same offline exploration dataset  $\mathcal{D}$ ; model-based method additionally query the dynamics model  $\hat{\mathcal{P}}$  in order to generate training trajectories. Algorithm-specific hyperparameters were tuned separately for each method through grid search, and kept constant across environments. Each algorithm is allowed the same number of gradient steps (sufficient to ensure convergence for all methods). We implement two strategies for optimizing the value functions: querying the jointly learned actor network, and employing model-based planning and using zero-order trajectory optimization (e.g., CEM [37]/iCEM[35]) with the learned model  $\hat{P}$  to optimize  $n$ -step look ahead value estimates [46]. Average success rates over 9 seeds are presented with  $90\%$  simple bootstrap confidence intervals in Figure 2.
To answer the first question, we consider the performance of the best optimizer (often model-based planning, in orange) for each of the value learning methods. We observe that simpler actor-critic algorithms (i.e., TD3) are overall competitive with the remaining value learning methods. MBPO, MOPO and CRR are not consistently better in the considered environments, despite involving offline corrections or relying on the learned model  $\hat{P}$  to generate training trajectories. We hypothesize that this phenomenon can be attributed, at least partially, to the exploratory nature of the data distribution, as exemplified in Figure 3. Additionally, we note that our findings and hypothesis are consistent with those reported by Yarats et al. [52] for non-goal-conditioned tasks.
We then turn to examining the performance gap between actor networks and model-based planning. We observe a tendency for model-based planning to outperform actor networks, with the largest improvements in fetch.push, as all learned policies similarly fail in this environment. While this stands true for almost all considered algorithms, CRR poses an exception to the success of model-based planning for two environments, which we hypothesize is due to its reliance on weighted behavior cloning rather than on backpropagation through a learned critic.
We remark that it is possible to show that, under common assumptions, a policy which exactly optimizes an optimal goal-conditioned value function is guaranteed to achieve its goals (see Appendix A.2). The low success rate of actor networks, assuming their objective is well-optimized, is thus a symptom of suboptimality for learned value functions. In Section 5, we argue how this suboptimality can be partially traced back to a class of estimation artifacts; moreover, we show how model-based planning is intrinsically more robust to such artifacts than actor networks (see Section 6.1).
Due to its strong empirical performance, TD3 is adopted for value learning through the rest of this work. We however remark that the analysis and the method proposed in Sections 5 and 6 remain valid for each of the RL methods considered. Further details on methods and results presented in this section are reported in Appendix I.
Figure 3: 200k transitions from the exploration dataset  $\mathcal{D}$  (left), and from the standard offline dataset provided by D4RL (right). Unsupervised exploration leads to a more uniform state coverage, and eases out-of-distribution issues.
5 Pathologies of Goal-Conditioned Value Functions
The performance gap between model-based planning and actor networks described in the previous section is a symptom of estimation issues in learned value functions. In this section, we pinpoint a particular class of geometric features which are not compatible with optimal goal-conditioned value functions. Such features represent a significant failure case for an agent optimizing a learned value function, as they can effectively trap it within a manifold of the state space. Intuitively, this happens when the estimated value of the current state exceeds that of all other reachable states, despite the goal not being achieved.