SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proof can be found in Appendix E.1. We first note that each of the error terms in the hierarchical policy bound is always no larger than the error in the flat policy, implying that both the high- and low-level policies are more accurate than the flat policy. To compare the total errors,  $\mathcal{E}(\pi)$  and  $\mathcal{E}(\pi^{\ell}\circ \pi^{h})$ , we perform a numerical analysis. Figure 4 shows the hierarchical policy's error bound for varying subgoal steps in five different  $(T,\sigma)$  settings. The results indicate that the flat policy's error can be significantly reduced by employing a hierarchical policy with an appropriate choice of  $k$ , suggesting that splitting policy extraction into two levels can be beneficial.
5 Hierarchical Implicit Q-Learning (HIQL)
Based on the hierarchical policy structure in Section 4, we now present a practical algorithm, which we call Hierarchical Implicit Q-Learning (HIQL), to extract hierarchical policies that are robust to the noise present in the learned goal-conditioned value function. We first explain how to train a subgoal policy (Section 5.1) and then extend this policy to predict representations (learned via the value function), which will enable HIQL to scale to image-based environments (Section 5.2).
5.1 Hierarchical policy extraction
As motivated in Section 4.2, we split policy learning into two levels, with a high-level policy generating intermediate subgoals and a low-level policy producing primitive actions to reach the subgoals. In this way, the learned goal-conditioned value function can provide clearer signals for both policies, effectively reducing the total policy error. Our method, HIQL, extracts the hierarchical policies from the same value function learned by action-free IQL (Equation (4)) using AWR-style objectives. While we choose to use action-free IQL in this work, we note that our hierarchical policy extraction scheme is orthogonal to the choice of the underlying offline RL algorithm used to train a goal-conditioned value function.
HIQL trains both a high-level policy  $\pi_{\theta_h}^h (s_{t + k}\mid s_t,g)$ , which produces optimal  $k$ -step subgoals  $s_{t + k}$ , and a low-level policy  $\pi_{\theta_\ell}^\ell (a\mid s_t,s_{t + k})$ , which outputs primitive actions, with the following objectives:
$$
J _ {\pi^ {h}} \left(\theta_ {h}\right) = \mathbb {E} _ {\left(s _ {t}, s _ {t + k}, g\right)} \left[ \exp \left(\beta \cdot \tilde {A} ^ {h} \left(s _ {t}, s _ {t + k}, g\right)\right) \log \pi_ {\theta_ {h}} ^ {h} \left(s _ {t + k} \mid s _ {t}, g\right) \right], \tag {6}
$$
$$
J _ {\pi^ {\ell}} \left(\theta_ {\ell}\right) = \mathbb {E} _ {\left(s _ {t}, a _ {t}, s _ {t + 1}, s _ {t + k}\right)} \left[ \exp \left(\beta \cdot \tilde {A} ^ {\ell} \left(s _ {t}, a _ {t}, s _ {t + k}\right)\right) \log \pi_ {\theta_ {\ell}} ^ {\ell} \left(a _ {t} \mid s _ {t}, s _ {t + k}\right) \right], \tag {7}
$$
where  $\beta$  denotes the inverse temperature hyperparameter and we approximate  $\tilde{A}^h (s_t,s_{t + k},g)$  as  $V_{\theta_V}(s_{t + k},g) - V_{\theta_V}(s_t,g)$  and  $\tilde{A}^{\ell}(s_t,a_t,s_{t + k})$  as  $V_{\theta_V}(s_{t + 1},s_{t + k}) - V_{\theta_V}(s_t,s_{t + k})$ . We do not include rewards and discount factors in these advantage estimates for simplicity, as they are (mostly) constants or can be subsumed into the temperature  $\beta$  (see Appendix A for further discussion). Similarly to vanilla AWR (Equation (5)), our high-level objective (Equation (6)) performs a weighted regression over subgoals to reach the goal, and the low-level objective (Equation (7)) carries out a weighted regression over primitive actions to reach the subgoals.
Algorithm 1 Hierarchical Implicit Q-Learning (HIQL)
1: Input: offline dataset  $\mathcal{D}$ , action-free dataset  $\mathcal{D}_S$  (optional,  $\mathcal{D}_S = \mathcal{D}$  otherwise)  
2: Initialize value function  $V_{\theta_V}(s,\phi(g))$  with built-in representation  $\phi(g)$ , high-level policy  $\pi_{\theta_h}^h (z_{t + k}\mid s_t,g)$ , low-level policy  $\pi_{\theta_\ell}^\ell (a\mid s_t,z_{t + k})$ , learning rates  $\lambda_V,\lambda_h,\lambda_\ell$   
3: while not converged do  
4:  $\theta_V\gets \theta_V - \lambda_V\nabla_{\theta_V}\mathcal{L}_V(\theta_V)$  with  $(s_t,s_{t + 1},g)\sim \mathcal{D}_S$  # Train value function, Equation (4)  
5: end while  
6: while not converged do  
7:  $\theta_h\gets \theta_h + \lambda_h\nabla_{\theta_h}J_{\pi^h}(\theta_h)$  with  $(s_t,s_{t + k},g)\sim \mathcal{D}_S$  # Extract high-level policy, Equation (6)  
8: end while  
9: while not converged do  
10:  $\theta_\ell \gets \theta_\ell +\lambda_\ell \nabla_{\theta_\ell}J_{\pi^\ell}(\theta_\ell)$  with  $(s_t,a_t,s_{t + 1},s_{t + k})\sim \mathcal{D}$  # Extract low-level policy, Equation (7)  
11: end while
We note that Equation (6) and Equation (7) are completely separated from one another, and only the low-level objective requires action labels. As a result, we can leverage action-free data for both the value function and high-level policy of HIQL, by further training them with a potentially large amount of additional passive data. Moreover, the low-level policy is relatively easy to learn compared to the other components, as it only needs to reach local subgoals without the need for learning the complete global structure. This enables HIQL to work well even with a limited amount of action information, as we will demonstrate in Section 6.4.
5.2 Representations for subgoals
In high-dimensional domains, such as pixel-based environments, directly predicting subgoals can be prohibitive or infeasible for the high-level policy. To resolve this issue, we incorporate representation learning into HIQL, letting the high-level policy produce more compact representations of subgoals. While one can employ existing action-free representation learning methods [34, 61, 68, 82] to learn state representations, HIQL simply uses an intermediate layer of the value function as a goal representation, which can be proven to be sufficient for control. Specifically, we parameterize the goal-conditioned value function  $V(s,g)$  with  $V(s,\phi (g))$ , and use  $\phi (g)$  as the representation of the goal. Using this representation, the high-level policy  $\pi^h (z_{t + k}\mid s_t,g)$  produces  $z_{t + k} = \phi (s_{t + k})$  instead of  $s_{t + k}$ , which the low-level policy  $\pi^{\ell}(a\mid s_t,z_{t + k})$  takes as input to output actions (Figure 1a). In this way, we can simply learn compact goal representations that are sufficient for control with no separate training objectives or components. Formally, we prove that the representations from the value function are sufficient for action selection:
Proposition 5.1 (Goal representations from the value function are sufficient for action selection). Let  $V^{*}(s,g)$  be the value function for the optimal reward-maximizing policy  $\pi^{*}(a\mid s,g)$  in a deterministic MDP. Let a representation function  $\phi (g)$  be given. If this same value function can be represented in terms of goal representations  $\phi (g)$ , then the reward-maximizing policy can also be represented in terms of goal representations  $\phi (g)$ :
$$
\begin{array}{l} \exists V _ {\phi} (s, \phi (g)) s. t. V _ {\phi} (s, \phi (g)) = V ^ {*} (s, g) f o r a l l s, g \Longrightarrow \\ \exists \pi_ {\phi} (a \mid s, \phi (g)) \text {s . t .} \pi_ {\phi} (a \mid s, \phi (g)) = \pi^ {*} (a \mid s, g) f o r a l l s, g. \\ \end{array}
$$
While Proposition 5.1 shows that the parameterized value function  $V(s, \phi(g))$  provides a sufficient goal representation  $\phi$ , we found that additionally concatenating  $s$  to the input to  $\phi$  (i.e., using  $\phi([g, s])$  instead of  $\phi(g)$ ) [39] leads to better empirical performance (see Appendix A for details), and thus we use the concatenated variant of value function parameterization in our experiments. We provide a pseudocode for HIQL in Algorithm 1 and the full training details in Appendices A and D.
Our experiments will use six offline goal-conditioned tasks, aiming to answer the following questions:
1. How well does HIQL perform on a variety of goal-conditioned tasks, compared to prior methods?
2. Can HIQL solve image-based tasks, and are goal representations important for good performance?
3. Can HIQL utilize action-free data to accelerate learning?
4. Does HIQL mitigate policy errors caused by noisy and imperfect value functions in practice?
(a) AntMaze
(b) Kitchen
(c) CALVIN
Figure 5: State-based benchmark environments.
(a) Procgen Maze
Figure 6: Pixel-based benchmark environments.
(b) Visual AntMaze
(c) Roboverse
We first describe our evaluation environments, shown in Figure 5 (state-based) and Figure 6 (pixel-based). AntMaze [9, 87] is a class of challenging long-horizon navigation tasks, where the goal is to control an 8-DoF Ant robot to reach a given goal location from the initial position. We use the four medium and large maze datasets from the original D4RL benchmark [28]. While the large mazes already present a significant challenge for long-horizon reasoning, we also include two even larger mazes (AntMaze-Ultra) proposed by Jiang et al. [43]. Kitchen [35] is a long-horizon manipulation domain, in which the goal is to complete four subtasks (e.g., open the microwave or move the kettle) with a 9-DoF Franka robot. We employ two datasets consisting of diverse behaviors ('-partial' and '-mixed') from the D4RL benchmark [28]. CALVIN [63], another long-horizon manipulation environment, also features four target subtasks similar to Kitchen. However, the dataset accompanying CALVIN [84] consists of a much larger number of task-agnostic trajectories from 34 different subtasks, which makes it challenging for the agent to learn relevant behaviors for the goal. Procgen Maze [16] is a pixel-based maze navigation environment. We train agents on an offline dataset consisting of 500 or 1000 different maze levels with a variety of sizes, colors, and difficulties, and test them on both the same and different sets of levels to evaluate their generalization capabilities. Visual AntMaze is a vision-based variant of the AntMaze-Large environment [28]. We provide only a  $64 \times 64 \times 3$  camera image (as shown in the bottom row of Figure 6b) and the agent's proprioceptive states, excluding the global coordinates. As such, the agent must learn to navigate the maze based on the wall structure and floor color from the image. Roboverse [25, 104] is a pixel-based, goal-conditioned robotic manipulation environment. The dataset consists of  $48 \times 48 \times 3$  images of diverse sequential manipulation behaviors, starting from randomized initial object poses. We evaluate the agent's performance across five unseen goal-reaching tasks that require multi-stage reasoning and generalization. To train goal-conditioned policies in these benchmark environments, during training, we replace the original rewards with a sparse goal-conditioned reward function,  $r(s, g) = 0$  (if  $s = g$ ),  $-1$  (otherwise).
We compare the performance of HIQL with six previous behavioral cloning and offline RL methods. For behavioral cloning methods, we consider flat goal-conditioned behavioral cloning (GCBC) [19, 33] and hierarchical goal-conditioned behavioral cloning (HGCBC) with two-level policies [35, 59]. For offline goal-conditioned RL methods, we evaluate a goal-conditioned variant of IQL [49] ("GC-IQL") (Section 3), which does not use hierarchy, and POR [97] ("GC-POR"), which uses hierarchy but does not use temporal abstraction (i.e., similar to  $k = 1$  in HIQL) nor representation learning. In AntMaze, we additionally compare HIQL with two model-based approaches that studied this domain in prior work: Trajectory Transformer (TT) [41], which models entire trajectories with a Transformer [90], and TAP [43], which encodes trajectory segments with VQ-VAE [89] and performs model-based planning over latent vectors in a hierarchical manner. We use the performance reported by Jiang et al. [43] for comparisons with TT and TAP. In our experiments, we use 8 random seeds and represent  $95\%$  confidence intervals with shaded regions (in figures) or standard deviations (in tables), unless otherwise stated. We provide full details of environments and baselines in Appendix D.
6.2 Results on state-based environments
We first evaluate HIQL in the five state-based environments (AntMaze-{Medium, Large, Ultra}, Kitchen, and CALVIN) using nine offline datasets. We evaluate the performance of the learned policies by commanding them with the evaluation goal state  $g$  (i.e., the benchmark task target position in AntMaze, or the state that corresponds to completing all four subtasks in Kitchen and CALVIN), and measuring the average return with respect to the original benchmark task reward function. We test two versions of HIQL (without and with representations) in state-based environments. Table 1 and Figure 7a show the results on the nine offline datasets, indicating that HIQL mostly achieves the best performance in our experiments. Notably, HIQL attains an  $88\%$  success rate on AntMaze-Large and  $53\%$  on AntMaze-Ultra, which is, to the best of our knowledge, better than any previously reported
Table 1: Evaluating HIQL on state-based offline goal-conditioned RL. HIQL mostly outperforms six baselines on a variety of benchmark tasks, including on different types of data. We show the standard deviations across 8 random seeds and refer to Appendix B for the full training curves. Baselines: GCBC [33], HGCBC [35], GC-IQL [49], GC-POR [97], TAP [43], TT [41].
Dataset	GCBC	HGCBC	GC-IQL	GC-POR	TAP	TT	HIQL (ours)	HIQL (w/o repr.)
antmaze-medium-diverse	67.3 ±10.1	71.6 ±8.9	63.5 ±14.6	74.8 ±11.9	85.0	100.0	86.8 ±4.6	89.9 ±3.5
antmaze-medium-play	71.9 ±16.2	66.3 ±9.2	70.9 ±11.2	71.4 ±10.9	78.0	93.3	84.1 ±10.8	87.0 ±8.4
antmaze-large-diverse	20.2 ±9.1	63.9 ±10.4	50.7 ±18.8	49.0 ±17.2	82.0	60.0	88.2 ±5.3	87.3 ±3.7
antmaze-large-play	23.1 ±15.6	64.7 ±14.5	56.5 ±14.4	63.2 ±16.1	74.0	66.7	86.1 ±7.5	81.2 ±6.6
antmaze-ultra-diverse	14.4 ±9.7	39.4 ±20.6	21.6 ±15.2	29.8 ±13.6	26.0	33.3	52.9 ±17.4	52.6 ±8.7
antmaze-ultra-play	20.7 ±9.7	38.2 ±18.1	29.8 ±12.4	31.0 ±19.4	22.0	20.0	39.2 ±14.8	56.0 ±12.4
kitchen-partial	38.5 ±11.8	32.0 ±16.7	39.2 ±13.5	18.4 ±14.3	-	-	65.0 ±9.2	46.3 ±8.6
kitchen-mixed	46.7 ±20.1	46.8 ±17.6	51.3 ±12.8	27.9 ±17.9	-	-	67.7 ±6.8	36.8 ±20.1
calvin	17.3 ±14.8	3.1 ±8.8	7.8 ±17.6	12.4 ±18.6	-	-	43.8 ±39.5	23.4 ±27.1
Table 2: Evaluating HIQL on pixel-based offline goal-conditioned RL. HIQL scales to high-dimensional pixel-based environments with latent subgoal representations, achieving the best performance across the environments. We refer to Appendix B for the full training curves.
Dataset	GCBC	HGCBC (+ repr.)	GC-IQL	GC-POR (+ repr.)	HIQL (ours)
procgen-maze-500-train	16.8 ±2.8	14.3 ±4.1	72.5 ±10.0	75.8 ±12.1	82.5 ±6.0
procgen-maze-500-test	14.5 ±5.0	11.2 ±3.7	49.5 ±9.8	53.8 ±14.5	64.5 ±13.2
procgen-maze-1000-train	27.2 ±8.9	15.0 ±5.7	78.2 ±7.2	82.0 ±6.5	87.0 ±13.9
procgen-maze-1000-test	12.0 ±5.9	14.5 ±5.0	60.0 ±10.6	69.8 ±7.4	78.2 ±17.9
visual-antmaze-diverse	71.4 ±6.0	35.1 ±12.0	72.6 ±5.9	47.4 ±17.6	80.5 ±9.4
visual-antmaze-play	64.4 ±6.3	23.8 ±8.5	70.4 ±26.6	57.0 ±8.1	78.4 ±4.6
visual-antmaze-navigate	33.2 ±7.9	21.4 ±4.6	22.1 ±14.1	16.1 ±15.2	45.7 ±18.1
roboverse	26.2 ±4.5	26.4 ±6.4	31.2 ±8.7	46.6 ±7.4	61.5 ±5.3
(a) State-based environments
(b) Pixel-based environments
Figure 7: Performance comparison. Following the protocol proposed by Agarwal et al. [1], we report the interquartile mean (IQM) metrics to evaluate the statistical significance of HIQL's performance on offline goal-conditioned RL benchmarks (Tables 1 and 2). We refer to Appendix B for the full Riable plots.
result on these datasets. In manipulation domains, we find that having latent subgoal representations in HIQL is important for enabling good performance. In CALVIN, while other methods often fail to achieve any of the subtasks due to the high diversity in the data, HIQL completes approximately two subtasks on average.
6.3 Results on pixel-based environments
Next, to verify whether HIQL can scale to high-dimensional environments using goal representations, we evaluate our method on three pixel-based domains (Procgen Maze, Visual AntMaze, and Robo-verse) with image observations. For the prior hierarchical approaches that generate raw subgoals (HGCBC and GC-POR), we apply HIQL's value-based representation learning scheme to enable them to handle the high-dimensional observation space. Table 2 and Figure 7b present the results, showing that our hierarchical policy extraction scheme, combined with representation learning, improves performance in these image-based environments as well. Notably, in Procgen Maze, HIQL exhibits larger gaps compared to the previous methods on the test sets. This is likely because the high-level policy can generalize better than the flat policy, as it can focus on the long-term direction toward the goal rather than the maze's detailed layout. In Robo-verse, HIQL is capable of generalizing to solve unseen robotic manipulation tasks purely from images, achieving an average success rate of  $62\%$ .
6.4 Results with action-free data
As mentioned in Section 5.1, one of the advantages of HIQL is its ability to leverage a potentially large amount of passive (action-free) data. To empirically verify this capability, we train HIQL on action-limited datasets, where we provide action labels for just  $25\%$  of the trajectories and use state-only trajectories for the remaining  $75\%$ . Table 3 shows the results from six different tasks, demonstrating
Table 3: HIQL can leverage passive, action-free data. Since our method requires action information only for the low-level policy, which is relatively easier to learn, HIQL mostly achieves comparable performance with just  $25\%$  of action-labeled data, outperforming even baselines trained on full datasets.
Dataset	GC-IQL (full)	GC-POR (full)	HIQL (full)	HIQL (action-limited)	vs. HIQL (full)	vs. Prev. best (full)
antmaze-large-diverse	50.7 ±18.8	49.0 ±17.2	88.2 ±5.3	88.9 ±6.4	+0.7	+38.2
antmaze-ultra-diverse	21.6 ±15.2	29.8 ±13.6	52.9 ±17.4	38.2 ±15.4	-14.7	+8.4
kitchen-mixed	51.3 ±12.8	27.9 ±17.9	67.7 ±6.8	59.1 ±9.6	-8.6	+7.8
calvin	7.8 ±17.6	12.4 ±18.6	43.8 ±39.5	35.8 ±30.7	-8.0	+23.4
procgen-maze-500-train	72.5 ±10.0	75.8 ±12.1	82.5 ±6.0	77.0 ±12.5	-5.5	+1.2
procgen-maze-500-test	49.5 ±9.8	53.8 ±14.5	64.5 ±13.2	65.5 ±16.4	+1.0	+11.7
Figure 8: Value and policy errors in Progen Maze: (left) As the distance between the state and the goal increases, the learned value function becomes noisier. (middle) We measure the accuracies of learned policies. (right) Thanks to our hierarchical policy extraction scheme (Section 4.2), HIQL exhibits the best policy accuracy, especially when the goal is far away from the state. The blue numbers denote the accuracy differences between HIQL and the second-best methods.
Policy accuracy metric	GC-IQL	GC-POR (+ repr.)	HIQL (ours)
All goals (train)	61.9 ±1.9	64.5 ±2.7	66.6 ±2.1 (+2.1)
All goals (test)	60.3 ±2.8	63.6 ±3.2	68.0 ±4.1 (+4.4)
Distant goals (train)	49.3 ±3.3	48.1 ±4.5	56.8 ±8.9 (+7.5)
Distant goals (test)	47.5 ±8.6	47.2 ±3.7	59.9 ±10.4 (+12.4)
that HIQL, even with a limited amount of action information, can mostly maintain its original performance. Notably, action-limited HIQL still outperforms previous offline RL methods (GC-IQL and GC-POR) trained with the full action-labeled data. We believe this is because HIQL learns a majority of the knowledge through hierarchical subgoal prediction from state-only trajectories.
6.5 Does HIQL mitigate policy errors caused by noisy value functions?
To empirically verify whether our two-level policy architecture is more robust to errors in the learned value function (i.e., the "signal-to-noise" ratio argument in Section 4), we compare the policy accuracies of GC-IQL (flat policy), GC-POR (hierarchy without temporal abstraction), and HIQL (ours) in Progen Maze, by evaluating the ratio at which the ground-truth actions match the learned actions. We also measure the noisiness (i.e., standard deviation) of the learned value function with respect to the ground-truth distance between the state and the goal. Figure 8 shows the results. We first observe that the noise in the value function generally becomes larger as the state-goal distance increases. Consequently, HIQL achieves the best policy accuracy, especially for distant goals  $(\mathrm{dist}(s,g)\geq 50)$ , as its hierarchical policy extraction scheme provides the policies with clearer learning signals (Section 4.2).
We refer to Appendix C for further analyses, including subgoal visualizations and an ablation study on subgoal steps and design choices for representations.
We proposed HIQL as a simple yet effective hierarchical algorithm for offline goal-conditioned RL. While hierarchical RL methods tend to be complex, involving many different components and objectives, HIQL shows that it is possible to build a method where a single value function simultaneously drives the learning of the low-level policy, the high-level policy, and the representations in a relatively simple and easy-to-train framework. We showed that HIQL not only exhibits strong performance in various challenging goal-conditioned tasks, but also can leverage action-free data and enjoy the benefits of built-in representation learning for image-based tasks.
Limitations. One limitation of HIQL is that the objective for its action-free value function (Equation (4)) is unbiased only when the environment dynamics are deterministic. As discussed in Section 3, HIQL (and other prior methods that use action-free videos) may overestimate the value function in partially observed or stochastic settings. To mitigate the optimism bias of HIQL in stochastic environments, we believe disentangling controllable parts from uncontrollable parts of the environment can be one possible solution [92, 99], which we leave for future work. Another limitation of our work is that we assume the noise for each  $V(s,g)$  is independent in our theoretical analysis (Proposition 4.1). While Figure 8 shows that the "signal-to-noise" argument empirically holds in our experiments, the independence assumption in our theorem might not hold in environments with continuous state spaces, especially when the value function is modeled by a smooth function approximator.
We would like to thank Aviral Kumar for an informative discussion about the initial theoretical results, Chongyi Zheng, Kuan Fang, and Fangchen Liu for helping set up the Roboverse environment, and RAIL members and anonymous reviewers for their helpful comments. This work was supported by the Korea Foundation for Advanced Studies (KFAS), the Fannie and John Hertz Foundation, the NSF GRFP (DGE2140739), AFOSR (FA9550-22-1-0273), and ONR (N00014-21-1-2838). This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley.