SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Seohong Park1 Dibya Ghosh1 Benjamin Eysenbach2 Sergey Levine1
<sup>1</sup>University of California, Berkeley <sup>2</sup>Princeton University
seohong@berkeley.edu
Unsupervised pre-training has recently become the bedrock for computer vision and natural language processing. In reinforcement learning (RL), goal-conditioned RL can potentially provide an analogous self-supervised approach for making use of large quantities of unlabeled (reward-free) data. However, building effective algorithms for goal-conditioned RL that can learn directly from diverse offline data is challenging, because it is hard to accurately estimate the exact value function for faraway goals. Nonetheless, goal-reaching problems exhibit structure, such that reaching distant goals entails first passing through closer subgoals. This structure can be very useful, as assessing the quality of actions for nearby goals is typically easier than for more distant goals. Based on this idea, we propose a hierarchical algorithm for goal-conditioned RL from offline data. Using one action-free value function, we learn two policies that allow us to exploit this structure: a high-level policy that treats states as actions and predicts (a latent representation of) a subgoal and a low-level policy that predicts the action for reaching this subgoal. Through analysis and didactic examples, we show how this hierarchical decomposition makes our method robust to noise in the estimated value function. We then apply our method to offline goal-reaching benchmarks, showing that our method can solve long-horizon tasks that stymie prior methods, can scale to high-dimensional image observations, and can readily make use of action-free data. Our code is available at https://seohong.me/projects/hiql/
Many of the most successful machine learning systems for computer vision [15, 36] and natural language processing [10, 18] leverage large amounts of unlabeled or weakly-labeled data. In the reinforcement learning (RL) setting, offline goal-conditioned RL provides an analogous way to potentially leverage large amounts of multi-task data without reward labels or video data without action labels: offline learning [54, 55] enables leveraging previously collected and passively observed data, and goal-conditioned RL [44, 79] enables learning from unlabeled, reward-free data. However, offline goal-conditioned RL poses major challenges. First, learning an accurate goal-conditioned value function for any state and goal pair is challenging when considering very broad and long-horizon goal-reaching tasks. This often results in a noisy value function and thus potentially an erroneous policy. Second, while the offline setting unlocks the potential for using previously collected data, it is not straightforward to incorporate vast quantities of existing action-free video data into standard RL methods. In this work, we aim to address these challenges by developing an effective offline goal-conditioned RL method that can learn to reach distant goals, readily make use of data without reward labels, and even utilize data without actions.
One straightforward approach to offline goal-conditioned RL is to first train a goal-conditioned value function and then train a policy that leads to states with high values. However, many prior papers have observed that goal-conditioned RL is very difficult, particularly when combined with offline training and distant goals [35, 38, 102]. We observe that part of this difficulty stems from the "signal-to-noise"
(a) Three components of HIQL.
(b) Hierarchical policies get clearer learning signals.
Figure 1: (left) We train a value function parameterized as  $V(s, \phi(g))$ , where  $\phi(g)$  corresponds to the subgoal representation. The high-level policy predicts the representation of a subgoal  $z_{t+k} = \phi(s_{t+k})$ . The low-level policy takes this representation as input to produce actions to reach the subgoal. (right) In contrast to many prior works on hierarchical RL, we extract both policies from the same value function. Nonetheless, this hierarchical structure yields a better "signal-to-noise" ratio than a flat, non-hierarchical policy, due to the improved relative differences between values.
ratio in value functions for faraway goals: when the goal is far away, the optimal action may be only slightly better than suboptimal actions, because a transition in the wrong direction can simply be corrected at the next time step. Thus, when the value function is learned imperfectly and has small errors, these errors can drown out the signal for distant goals, potentially leading to an erroneous policy. This issue is further exacerbated with the offline RL setting, as erroneous predictions from the value function are not corrected when those actions are taken and their consequences observed.
To address this challenge, we separate policy extraction into two levels. We first train a goal-conditioned value function from offline data with implicit Q-learning (IQL) [49] and then we extract two-level policies from it. Our high-level policy produces intermediate waypoint states, or subgoals, as actions. Because predicting high-dimensional states can be challenging, we will propose a method that only requires the high-level policy to product representations of the subgoals, with the representations learned end-to-end from the value function. Our low-level policy takes this subgoal representation as input and produces actions to reach the subgoal (Figure 1a). Here, in contrast to previous hierarchical methods [56, 65], we extract both policies from the same value function. Nonetheless, this hierarchical decomposition enables the value function to provide clearer learning signals for both policies (Figure 1b). For the high-level policy, the value difference between various subgoals is much larger than that between different low-level actions. For the low-level policy, the value difference between actions becomes relatively larger because the low-level policy only needs to reach nearby subgoals. Moreover, the value function and high-level policy do not require action labels, so this hierarchical scheme provides a way to leverage a potentially large amount of passive, action-free data. Training the low-level policy does require some data labeled with actions.
To summarize, our main contribution in this paper is to propose Hierarchical Implicit Q-Learning (HIQL), a simple hierarchical method for offline goal-conditioned RL. HIQL extracts all the necessary components—a representation function, a high-level policy, and a low-level policy—from a single goal-conditioned value function. Through our experiments on six types of state-based and pixel-based offline goal-conditioned RL benchmarks, we demonstrate that HIQL significantly outperforms previous offline goal-conditioned RL methods, especially in complex, long-horizon tasks, scales to high-dimensional observations, and is capable of incorporating action-free data.
Our method draws on concepts from offline RL [54, 55], goal-conditioned RL [4, 44, 79], hierarchical RL [6, 62, 77, 85, 86, 96], and action-free RL [7, 12, 34, 80, 88, 105], providing a way to effectively train general-purpose goal-conditioned policies from previously collected offline data. Prior work on goal-conditioned RL has introduced algorithms based on a variety of techniques, such as hindsight relabeling [4, 13, 27, 56, 57, 75, 100], contrastive learning [23, 24, 102], and state-occupancy matching [20, 60].
However, directly solving goal-reaching tasks is often challenging in complex, long-horizon environments [35, 56, 65]. To address this issue, several goal-conditioned RL methods have been proposed based on hierarchical RL [11, 17, 51, 56, 65, 66, 81, 91, 103] or graph-based subgoal planning [22, 38, 40, 45, 46, 69, 78, 101]. Like these prior methods, our algorithm will use higher-level subgoals in a hierarchical policy structure, but we will focus on solving goal-reaching tasks from offline data. We use an offline RL algorithm [49] to train a goal-conditioned value function from the dataset, which allows us to simply extract the hierarchical policies in a decoupled manner with no need for potentially complex graph-based planning procedures. Another important difference from prior work is that we only train a single goal-conditioned value function, unlike previous hierarchical methods that train multiple hierarchical value functions [56, 65]. Perhaps surprisingly, we show that this can still significantly improve the performance of the hierarchical policies, due to an improved "signal-to-noise" ratio (Section 4).
Our method is most closely related to previous works on hierarchical offline skill extraction and hierarchical offline (goal-conditioned) RL. Offline skill extraction methods [2, 43, 50, 72, 76, 84] encode trajectory segments into a latent skill space, and learn to combine these skills to solve downstream tasks. The primary challenge in this setting is deciding how trajectories should be decomposed hierarchically, which can be sidestepped in our goal-conditioned setting since subgoals provide a natural decomposition. Among goal-conditioned approaches, hierarchical imitation learning [35, 59] jointly learns subgoals and low-level controllers from optimal demonstrations. These methods have two drawbacks: they predict subgoals in the raw observation space, and they require expert trajectories; our observation is that a value function can alleviate both challenges, as it provides a way to use suboptimal data and stitch across trajectories, as well as providing a latent goal representation in which subgoals may be predicted. Another class of methods plans through a graph or model to generate subgoals [25, 26, 58, 83]; our method simply extracts all levels of the hierarchy from a single unified value function, avoiding the high computational overhead of planning. Finally, our method is closely related to POR [97], which predicts the immediate next state as a subgoal; this can be seen as one extreme of our method without representations, although we show that more long-horizon subgoal prediction can be advantageous both in theory and practice.
Problem setting. We consider the problem of offline goal-conditioned RL, defined by a Markov decision process  $\mathcal{M} = (\mathcal{S},\mathcal{A},\mu ,p,r)$  and a dataset  $\mathcal{D}$ , where  $\mathcal{S}$  denotes the state space,  $\mathcal{A}$  denotes the action space,  $\mu \in \mathcal{P}(\mathcal{S})$  denotes an initial state distribution,  $p\in S\times \mathcal{A}\to \mathcal{P}(\mathcal{S})$  denotes a transition dynamics distribution, and  $r(s,g)$  denotes a goal-conditioned reward function. The dataset  $\mathcal{D}$  consists of trajectories  $\tau = (s_0,a_0,s_1,a_1,\dots ,s_T)$ . In some experiments, we assume that we have an additional action-free dataset  $\mathcal{D}_S$  that consists of state-only trajectories  $\tau_{s} = (s_{0},s_{1},\ldots ,s_{T})$ . Unlike some prior work [4, 40, 46, 65, 101], we assume that the goal space  $\mathcal{G}$  is the same as the state space (i.e.,  $\mathcal{G} = \mathcal{S}$ ). Our goal is to learn from  $\mathcal{D}\cup \mathcal{D}_{\mathcal{S}}$  an optimal goal-conditioned policy  $\pi (a|s,g)$  that maximizes  $J(\pi) = \mathbb{E}_{g\sim p(g),\tau \sim p^{\pi}(\tau)}[\sum_{t = 0}^{T}\gamma^{t}r(s_{t},g)]$  with  $p^\pi (\tau) = \mu (s_0)\prod_{t = 0}^{T - 1}\pi (a_t\mid s_t,g)p(s_{t + 1}\mid s_t,a_t)$ , where  $\gamma$  is a discount factor and  $p(g)$  is a goal distribution.
Implicit Q-learning (IQL). One of the main challenges with offline RL is that a policy can exploit overestimated values for out-of-distribution actions [55], as we cannot correct erroneous policies and values via environment interactions, unlike in online RL. To tackle this issue, Kostrikov et al. [49] proposed implicit Q-learning (IQL), which avoids querying out-of-sample actions by converting the max operator in the Bellman optimal equation into expectile regression. Specifically, IQL trains an action-value function  $Q_{\theta_Q}(s,a)$  and a state-value function  $V_{\theta_V}(s)$  with the following loss:
$$
\mathcal {L} _ {V} \left(\theta_ {V}\right) = \mathbb {E} _ {(s, a) \sim \mathcal {D}} \left[ L _ {2} ^ {\tau} \left(Q _ {\bar {\theta} _ {Q}} (s, a) - V _ {\theta_ {V}} (s)\right) \right], \tag {1}
$$
$$
\mathcal {L} _ {Q} \left(\theta_ {Q}\right) = \mathbb {E} _ {\left(s, a, s ^ {\prime}\right) \sim \mathcal {D}} \left[ \left(r _ {\text {t a s k}} (s, a) + \gamma V _ {\theta_ {V}} \left(s ^ {\prime}\right) - Q _ {\theta_ {Q}} (s, a)\right) ^ {2} \right], \tag {2}
$$
where  $r_{\mathrm{task}}(s,a)$  denotes the task reward function,  $\bar{\theta}_Q$  denotes the parameters of the target Q network [64], and  $L_2^\tau$  is the expectile loss with a parameter  $\tau \in [0.5,1)$ :  $L_2^\tau (x) = |\tau -\mathbb{1}(x < 0)|x^2$ . Intuitively, expectile regression can be interpreted as an asymmetric square loss that penalizes positive values more than negative ones. As a result, when  $\tau$  tends to 1,  $V_{\theta_V}(s)$  gets closer to  $\max_aQ_{\bar{\theta}_Q}(s,a)$  (Equation (1)). Thus, we can use the value function to estimate the TD target  $(r_{\mathrm{task}}(s,a) + \gamma \max_{a'}Q_{\bar{\theta}_Q}(s',a'))$  as  $(r_{\mathrm{task}}(s,a) + \gamma V_{\theta_V}(s'))$  without having to sample actions  $a'$ .
After training the value function with Equations (1) and (2), IQL extracts the policy with advantage-weighted regression (AWR) [67, 70, 71, 73, 74, 94]:
$$
J _ {\pi} \left(\theta_ {\pi}\right) = \mathbb {E} _ {\left(s, a, s ^ {\prime}\right) \sim \mathcal {D}} \left[ \exp \left(\beta \cdot \left(Q _ {\bar {\theta} _ {Q}} (s, a) - V _ {\theta_ {V}} (s)\right)\right) \log \pi_ {\theta_ {\pi}} (a \mid s) \right], \tag {3}
$$
where  $\beta \in \mathbb{R}_0^+$  denotes an inverse temperature parameter. Intuitively, Equation (3) encourages the policy to select actions that lead to large  $Q$  values while not deviating far from the data collection policy [71].
Action-free goal-conditioned IQL. The original IQL method described above requires both reward and action labels in the offline data to train the value functions via Equations (1) and (2). However, in real-world scenarios, offline data might not contain task information or action labels, as in the case of task-agnostic demonstrations or videos. As such, we focus on the setting of offline goal-conditioned RL, which does not require task rewards, and provides us with a way to incorporate state-only trajectories into value learning. We can use the following action-free variant [34, 97] of IQL to learn an offline goal-conditioned value function  $V_{\theta_V}(s,g)$ :
$$
\mathcal {L} _ {V} \left(\theta_ {V}\right) = \mathbb {E} _ {\left(s, s ^ {\prime}\right) \sim \mathcal {D} _ {S}, g \sim p (g | \tau)} \left[ L _ {2} ^ {\tau} \left(r (s, g) + \gamma V _ {\bar {\theta} _ {V}} \left(s ^ {\prime}, g\right) - V _ {\theta_ {V}} (s, g)\right) \right]. \tag {4}
$$
Unlike Equations (1) and (2), this objective does not require actions when fitting the value function, as it directly takes backups from the values of the next states.
Action-labeled data is only needed when extracting the policy. With the goal-conditioned value function learned by Equation (4), we can extract the policy with the following variant of AWR:
$$
J _ {\pi} \left(\theta_ {\pi}\right) = \mathbb {E} _ {\left(s, a, s ^ {\prime}\right) \sim \mathcal {D}, g \sim p (g | \tau)} \left[ \exp \left(\beta \cdot A (s, a, g)\right) \log \pi_ {\theta_ {\pi}} (a \mid s, g) \right], \tag {5}
$$
where we approximate  $A(s, a, g)$  as  $\gamma V_{\theta_V}(s', g) + r(s, g) - V_{\theta_V}(s, g)$ . Intuitively, Equation (5) encourages the policy to select the actions that lead to the states having high values. With this action-free variant of IQL, we can train an optimal goal-conditioned value function only using action-free data and extract the policy from action-labeled data that may be different from the passive dataset.
We note that this action-free variant of IQL is unbiased when the environment dynamics are deterministic [34], but it may overestimate values in stochastic environments. This deterministic environment assumption is inevitable for learning an unbiased value function solely from state trajectories. The reason is subtle but important: in stochastic environments, it is impossible to tell whether a good outcome was caused by taking a good action or because of noise in the environment. As a result, applying action-free IQL to stochastic environments will typically result in overestimating the value function, implicitly assuming that all noise is controllable. While we will build our method upon Equation (4) in this work for simplicity, in line with many prior works on offline RL that employ similar assumptions [14, 33, 34, 41, 42, 93, 97], we believe correctly handling stochastic environments with advanced techniques (e.g., by identifying controllable parts of the environment [92, 99]) is an interesting direction for future work.
4 Hierarchical policy structure for offline goal-conditioned RL
Goal-conditioned offline RL provides a general framework for learning flexible policies from data, but the goal-conditioned setting also presents an especially difficult multi-task learning problem for RL algorithms, particularly for long-horizon tasks where the goal is far away. In Section 4.1, we discuss some possible reasons for this difficulty, from the perspective of the "signal-to-noise" ratio in the learned goal-conditioned value function. We then propose hierarchical policy extraction as a solution (Section 4.2) and compare the performances of hierarchical and flat policies in a didactic environment, based on our theoretical analysis (Section 4.3).
4.1 Motivation: why non-hierarchical policies might struggle
One common strategy in offline RL is to first fit a value function and then extract a policy that takes actions leading to high values [3, 8, 29-31, 49, 52, 67, 71, 95, 97, 98, 100]. This strategy can be directly applied to offline goal-conditioned RL by learning a goal-conditioned policy  $\pi(a \mid s_t, g)$  that aims to maximize the learned goal-conditioned value function  $V(s_{t+1}, g)$ , as in Equation (5). However, when the goal  $g$  is far from the current state  $s$ , the learned goal-conditioned value function may not provide a clear learning signal for a flat, non-hierarchical policy. There are two reasons for this. First, the differences between the values of different next states  $(V(s_{t+1}, g))$  may be small, as bad outcomes by taking suboptimal actions may be simply corrected in the next few steps, causing
(a) Optimal value function
(b) Noisy value function
Figure 2: Hierarchies allow us to better make use of noisy value estimates. (a) In this gridworld environment, the optimal value function predicts higher values for states  $s$  that are closer to the goal  $g(\bullet)$ . (b, c) However, a noisy value function results in selecting incorrect actions  $(\rightarrow)$ . (d) Our method uses this same noisy value function to first predict an intermediate subgoal, and then select an action for reaching this subgoal. Actions selected in this way correctly lead to the goal.
(c) Flat policy
(d) Hierarchical policy
only relatively minor costs. Second, these small differences can be overshadowed by the noise present in the learned value function (due to, for example, sampling error or approximation error), especially when the goal is distant from the current state, in which case the magnitude of the goal-conditioned value (and thus the magnitude of its noise or errors) is large. In other words, the "signal-to-noise" ratio in the next time step values  $V(s_{t + 1},g)$  can be small, not providing sufficiently clear learning signals for the flat policy. Figure 2 illustrates this problem. Figure 2a shows the ground-truth optimal value function  $V^{*}(s,g)$  for a given goal at each state, which can guide the agent to reach the goal. However, when noise is present in the learned value function  $\hat{V} (s,g)$  (Figure 2b), the flat policy  $\pi (a\mid s,g)$  becomes erroneous, especially at states far from the goal (Figure 2c).
4.2 Our hierarchical policy structure
To address this issue, our main idea in this work, which we present fully in Section 5, is to separate policy extraction into two levels. Instead of directly learning a single, flat, goal-conditioned policy  $\pi(a \mid s_t, g)$  that aims to maximize  $V(s_{t+1}, g)$ , we extract both a high-level policy  $\pi^h(s_{t+k} \mid s_t, g)$  and a low-level policy  $\pi^\ell(a \mid s_t, s_{t+k})$ , which aims to maximize  $V(s_{t+k}, g)$  and  $V(s_{t+1}, s_{t+k})$ , respectively. Here,  $s_{t+k}$  can be viewed as a waypoint or subgoal. The high-level policy outputs intermediate subgoal states that are  $k$  steps away from  $s$ , while the low-level policy produces primitive actions to reach these subgoals. Although we extract both policies from the same learned value function in this way, this hierarchical scheme provides clearer learning signals for both policies. Intuitively, the high-level policy receives a more reliable learning signal because different subgoals lead to more dissimilar values than primitive actions. The low-level policy also gets a clear signal (from the same value function) since it queries the value function with only nearby states, for which the value function is relatively more accurate (Figure 1b). As a result, the overall hierarchical policy can be more robust to noise and errors in the value function (Figure 2d).
4.3 Didactic example: hierarchical policies mitigate the signal-to-noise ratio challenge
To further understand the benefits of hierarchical policies, we study a toy example with one-dimensional state space (Figure 3). In this environment, the agent can move one unit to the left or right at each time step. The agent gets a reward of 0 when it reaches the goal; otherwise, it always gets  $-1$ . The optimal goal-conditioned
value function is hence given as  $V^{*}(s,g) = -|s - g|$  (assuming  $\gamma = 1$ ). We assume that the noise in the learned value function  $\hat{V} (s,g)$  is proportional to the optimal value: i.e.,  $\hat{V} (s,g) = V^{*}(s,g) + \sigma z_{s,g}V^{*}(s,g)$ , where  $z_{s,g}$  is sampled independently from the standard normal distribution and  $\sigma$  is its standard deviation. This indicates that as the goal becomes more distant, the noise generally increases, a trend we observed in our experiments (see Figure 8).
Figure 3: 1-D toy environment.
In this scenario, we compare the probabilities of choosing incorrect actions under the flat and hierarchical policies. We assume that the distance between  $s$  and  $g$  is  $T$  (i.e.,  $g = s + T$  and  $T > 1$ ). Both the flat policy and the low-level policy of the hierarchical approach consider the goal-conditioned values at  $s \pm 1$ . The high-level policy evaluates the values at  $s \pm k$ , using  $k$ -step away subgoals. For the hierarchical approach, we query both the high- and low-level policies at every step. Given these settings, we can bound the error probabilities of both approaches as follows:
Figure 4: Comparison of policy errors in flat vs. hierarchical policies in didactic environments. The hierarchical policy, with an appropriate subgoal step, often yields significantly lower errors than the flat policy.
Proposition 4.1. In the environment described in Figure 3, the probability of the flat policy  $\pi$  selecting an incorrect action is given as  $\mathcal{E}(\pi) = \Phi\left(-\frac{\sqrt{2}}{\sigma\sqrt{T^2 + 1}}\right)$  and the probability of the hierarchical policy  $\pi^\ell \circ \pi^h$  selecting an incorrect action is bounded as  $\mathcal{E}(\pi^\ell \circ \pi^h) \leq \Phi\left(-\frac{\sqrt{2}}{\sigma\sqrt{(T / k)^2 + 1}}\right) + \Phi\left(-\frac{\sqrt{2}}{\sigma\sqrt{k^2 + 1}}\right)$ , where  $\Phi$  denotes the cumulative distribution function of the standard normal distribution,  $\Phi(x) = \mathbb{P}[z \leq x] = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{x} e^{-t^2 / 2} \mathrm{d}t$ .