HIQL: Offline Goal-Conditioned RL with Latent States as Actions
Authors: Seohong Park
Dibya Ghosh
Benjamin Eysenbach
Sergey Levine
Venue: NeurIPS 2023 spotlight (2023)
Keywords: reinforcement learning
TL;DR: We propose a simple hierarchical offline goal-conditioned RL method that extracts both high- and low-level policies from a single value function.

Abstract: Unsupervised pre-training has recently become the bedrock for computer vision and natural language processing. In reinforcement learning (RL), goal-conditioned RL can potentially provide an analogous self-supervised approach for making use of large quantities of unlabeled (reward-free) data. However, building effective algorithms for goal-conditioned RL that can learn directly from diverse offline data is challenging, because it is hard to accurately estimate the exact value function for faraway goals. Nonetheless, goal-reaching problems exhibit structure, such that reaching distant goals entails first passing through closer subgoals. This structure can be very useful, as assessing the quality of actions for nearby goals is typically easier than for more distant goals. Based on this idea, we propose a hierarchical algorithm for goal-conditioned RL from offline data. Using one action-free value function, we learn two policies that allow us to exploit this structure: a high-level policy that treats states as actions and predicts (a latent representation of) a subgoal and a low-level policy that predicts the action for reaching this subgoal. Through analysis and didactic examples, we show how this hierarchical decomposition makes our method robust to noise in the estimated value function. We then apply our method to offline goal-reaching benchmarks, showing that our method can solve long-horizon tasks that stymie prior methods, can scale to high-dimensional image observations, and can readily make use of action-free data. Our code is available at https://seohong.me/projects/hiql/

HIQL: Offline Goal-Conditioned RL with Latent States as Actions
Seohong Park'
Dibya Ghosh
Benjamin Eysenbach²
Sergey Levine'
UUiveeriity of California, Berkeley "Princeton University
sechong@berkeley.
Abstract
Unsupervised pre-training has recently become the bedrock for computer vision and natural language processing. In reinforcement learning (RL), goal-conditioned RL can potentially provide analogous self-supervised approach for making of large quantities of unlabeled (reward-free) data. However, building effective algorithms for goal-conditioned RL that learn directly from diverse offline data is challenging, because it is hard to accurately estimate the exact value function for faraway goals. Nonetheless, goal-reaching problems exhibit structure, such that reaching distant goals entails first passing through closer subgoals. This structure can be very useful, as assessing the quality of actions for nearby goals is typically easier than for more distant goals, Based on this idca, we propose a hierarchical algorithm for goal-cone conciiontt RL from offline data. Using one action-fres value function, we learn two policies that allow us to exploit this structure: high-level policy that treats states as actions and predicts (a latent representation of) subgoal and low-level policy that predicts the action for reaching this subgoal. Through analysis and didactic examples, we show how this hierarchical decomposition makes our method robust to noise in the estimated value function. We then apply our method offline goal-reaching benchmarks, showing that our method can solve long-horiee monghoorioon tasks that stymie prior methods, can scale to high-dimensional image observations, and can readily make use of action-fii ffrion-free data. Our code is available https: / /eohhong...... ttt prop/ecsssshill...
Introduction
Many of the most successful machine learning systems for computer vision [15, 36] and natural language processing [10, 18] leverage large amounts of unlabeled or weakly-labeled data. In the reinforcement learning (RL) setting, offline goal-conditioned RL provides an analogous way to potentially leverage large amounts of multi-task data without reward labels or video data without action labels: offline learning [54, 55] enables leveraging previously collected and passively observed data, and goal-conditioned RL [44, 79] enables learnin from unlabeled, reward- -free data. However, offline coal-conditioned RI. poses major challenges. First, learning an accurate coal-conditioned value function for any state and goal pair is challenging when considering very broad and long- horgzoorizon geal-reaching tasks. This often results in noisy value function and thus potentially an erroneous policy. Second, while the offline setting unlocks the potential for using previously collected data, it not straightforward to incorporate vast quantities of existing action-free video data into standard RL methods. In this work, we aim to address these challenges by developing an effective offline goal-conditioned RL method that can learn to reach distant goals, readily make use of data without reward labels, and even utilize data without actions.
One straightforward approach offline goal-conditioned RL to first train a goal-conditioned value function and then train policy that leads to states with high values. However, many prior papers have observed that conlicondi RL very difficult, partieularly when combined with offline training and distant goals [35, 38, 102). We observe that part of this difficulty stems from the "signal-to-noise"
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

V(vg)
V(s ((5)))))
a
V
V(s.g)
By St+1
0(244))
ae
Flat policy
Hierarchical policy
x' (ayls, F1y44)
xt max V(e4a. ((
max V(544 (os..))
(i Bi4c) ((c.
(a) Three components of HIQL.
(b) Hierarchical policies get clearer learning signals.
Figure 1: (left) We train value function parameterized as V(s, o(g)). where ((g) corresponds the subgoal representation. The high-level policy predicts the representation of a subgoal 21-4A = (. The low-level policy takes this representation input to produce actions to reach the subgoal. (right) In contrast to many prior works hierarchical RL, we extract both policies from the same value function. Nonetheless, this hierarchical structure yields better "signal-to-nooss" ratio than flat, oon-hierarchica policy, due to the improved relative differences between values.
ratio in value functions for faraway goals: when the goal is far away, the optimal action may be only slightly better than suboptimal actions, because transition in the wrong direction can simply be corrected at the next time step. Thus, when the value function is learned imperfectly and has small errors, these errors can drown out the signal for distant goals, potentially leading to an erroneous policy. This issue is further exacerbated with the offline RL setting, as erroneous predictions from the value function
corrected when those actions taken and their consequences observed.
To address this challenge, we separate policy extraction into two levels. We first train a goal- conditioned value function from offline data with implicit Q-learning (IQL) [49] and then we extract two-level policies from it. Our high-level policy produces intermediate waypoint states, or subgoals, actions. Because predicting nigh-dimensiooal states be challenging, we will propose method that only requires the high-level policy to product representations of the subgoals, with the representations learned end-to-end from the value function. Our low-level policy takes this subgoal representation as input and produces actions to reach the subgoal (Figure Ia). Here, in contrast previous hierarchical methods [56, 65], we extract both policies from the same value function. Nonetheless, this hierarchical decomposition enables the value function to provide clearer learning signals for both policies (Figure 1b). For the high-level policy, the value difference between various subgoals is much larger than that between different low-level actions. For the low-level policy, the value difference between actions becomes relatively larger because the low-level policy only needs to reach nearby subgoals. Moreover, the value function and high-level policy do not require action labels, so this hierarchical scheme provides a way to leverage a potentially large amount of passive, action-fire free data. Training the low-level policy does require some data labeled with actions.
To summarize, main contribution in this paper (HIQL), a simple hierarchical method for offline goal-conditioned RL. HIQL extracts all the necessary components representation function, high-level policy, and low-level policy-from ffom a single goal-conditioned value function. Through our experiments on six types of state-based and pixel- based offline goal-conditioned RL benchmarks, previous offline goal-conditioned RL methods, especially in complex, long- hong-oorizon tasks, scales to high-dimensional observations, and is capable of incorporating action-free data.
propose Hierarchical Implicit Q-Learning
demonstrate that HIQL significantly outperforms
2 Related work
Our method draws on concepts from offline RL [54, 55]. goal-conditioned RL [4, 44, 79], hierarchical
RL [6. 62, 77, 85, 86, 96], and action-free RL [7, 12, 34, 80, 88, 105], providing way to effectively train general- ppreral-purpose purpose goal-conditioned policies from previously collected offline data. Prior work on goal-conditioned RI. has introduced algorithms based on variety of techniques, such as hindsight relabeling [4, 13, 27, 56, 57, 75, 100], contrastive learning [23, 24, 102), and state-occupancy matching [20, 60].

---

However, directly solving goal-reaching tasks is often challenging in complex. long-horizon environ- ments [35, 56, 65]. To address this issue, several goal-conditioned RL methods have been proposed based on hierarchical RL [11, 17, 51, 56, 65, 66, 81, 91, 103] or graph-based subgoal planning [22, 38, 40, 45, 46, 69, 78, 101]. Like these prior methods, our algorithm will use higher-level subgoals in a hierarchical policy structure, but we will focus on solving goal-reaching tasks from offline data. We use an offline RL algorithm [49] to train a goal-conditioned value function from the dataset, which allows us to simply extract the hierarchical policies in a decoupled manner with no need for potentially complex graph-based planning procedures. Another important difference from prior work is that we only train single conditioni value function, unlike previous hierarchical methods that train multiple hierarchical value functions [56, 65]. Perhaps surprisingly, we show that this can still significantly improve the performance of the hierarchical policcies.piiiiies. due to improved "signal-to-noiss" ratio (Section 4).
Our method is most closely related to previous works on hierarchical offline skill extraction and hierarchical offline igoal-conditionen RL. Offline skill extraction methods [2, 43, 50, 72, 76, 84] encode trajectory segments into a latent skill space, and learn to combine these skills to solve downstream tasks. The primary challenge in this setting is deciding how trajectories should be decomposed hierarchically, which can be sidestepped in our goal-conditioned setting since subgoals provide natural decomposition. Among goal-conditione approaches, hierarchical imitation learning [35, 59] jointly learns subgoals and low-level controllers from optimal demonstrations. These methods have two drawbacks: they predict subgoals in the raw observation space, and they require expert trajectories; our observation is that value function can alleviate both challenges, as it provides way to use suboptimal data and stitch across trajectories, as well as providing a latent goal representation in which subgoals may be predicted. Another class of methods plans through graph model to generate subgoals [25, 26, 58, 83]; our method simply extracts all levels of the hierarchy from a single unified value function, avoiding the high computational overhead of planning. Finally, our method is closely related to POR [97], which predicts the immediate next state as a subgoal; this can be seen as one extreme of our method without representations, although we show that more long-horizon subgoal prediction can be advantageous both in theory and practice.
3 Preliminaries
Problem setting. We consider the problem of offline goal-conditioned RL, defined by a Markov decision process MM- = (S, μl. p. and dataset D, where S denotes the state space, AA denotes the action space, P E P(S) denotes an initial state distribution, p E S x A + P(S) denotes transition dynamics distribution, and r(s.g) denotes goal-conditioned reward function. The dataset D consists of trajectories т = (so. op. $1, 01 ST.)).......). In some experiments, we assume that we have additional action-free dataset Ds that consists of state-only trajectories т, (80,81.- $1....,8T).. Unlike some prior work [4, 40, 46, 65, 101], we assume that the goal space is the same the state space (ie., G S). Our goal is to learn from D U DD optimal goal-conditioned policy [[ass,)) g) that maximizes J(x) Eg-r(())r=ppp()) T ( 9)] with pT(r)) 1u(s ((00)) II. 1T-1 x(a St, g)p(((+1 |
8t, az), where y is a discount factor and p(g) is a goal
distribution.
Implicit Q-learning (IQL). One of the main challenges with offline RL is that a policy can exploit overestimated values for out-of-distribun actions [55], as we cannot correct erroneous policies and values via environment interactions, unlike in online RI. To tackle this issue, Kostrikov et al. [49] proposed implicit Q-learning (IQL., which avoids querying out-of-sample actions by converting the max operator in the Bellman optimal equation into expectile regression. Specifically, IQL. trains an action-valie value function Qay (s, a) and state-value function Va, (s) with the following loss:
Lv [v( = B(s,a)=(| (Qr (s, (s)],,
(1)
(0() B(s,a,/)~ (Trask (s, a) + /Vo (s,
(2)
where Fiusk (s, a) denotes the task reward function, 0g denotes the parameters of the target Q network
[64], and LJ is the expectile loss with parameter E [0.5, Lz(x) |r 1(x 0)]2².
Intuitively, expectile regression can be interpreted as an asymmetric square loss that penalizes positive values more than negative ones. As result, when 7 tends to 1, Vo) gets closer to max, Q8g ()) (Equation (1)). Thus, we can use the value function to estimate the TD target (resk (s, a) + + max.' Q8a (s', a')) as (Fusk(s, a) + Vov, (s')) without having to sample actions a'.
3

---

After training the value function with Equations (1) and (2), IQL extracts the policy with ad advantage- weighted regression (AWR) [67, 70, 71, 73, 74, 94]:
Ja (O.) = "=..as/ (Qaa (5.a) Voy (s))) log ne, (a s)],
where / E "R."" denotes an inverse temperature parameter. Intuitively, Equation (3) encourages the policy to select actions that lead to large Q values while not deviating far from the data collection policy [71].
Action-free goal-conditioned IQL. The original IQL method described above requires both reward and action labels the offline data to train the value functions via Equations (1) and (2). However, in real-world scenarios, offline data might not contain task information or action labels, as in the case of task-agnostic demonstrations or videos. As such, we focus on the setting of offline goal-conditioned RL, which does not require task rewards, and provides us with way to incorporate state-only trajectories into value learning. We can use the following action-free variant [34, 97] of IQL to learn an offline con-icondi value function Vov (),g):
(0v) = "(eee") (r(s,, + +VVa, ()),g) - Ve, (s,
Unlike Equations (1) and (2). this objective does not require actions when fitting the value function. as it directly takes backups from the values of the next states.
Action-labeled data is only needed when extracting the policy. With the goal-conditioned value function learned by Equation (4), we can extract the policy with the following variant of AWR:
Jz =
A(s, a. g)) log Te, (a s,g)).
(5)
where we approximate A(s, a, g) as VVe + r(s,g) g) Voy (s, Intuitively, Equation (5) encourages the policy to select the actions that lead to the states having high values. With this action- free variant of IQL, we can train an optimal goal-conditionas data and extract the policy from action-labeled data that may be different from the passive dataset.
value function only using action-free
We note that this action-free variant of IQL is unbiased when the environment dynamics are determin- istic [34], but it may overestimate values in stochastic environments. This deterministic environment assumption is inevitable for learning unbiased value function solely from state trajectories. The reason is subtle but important: in stochastic environments, it is impossible to tell whether a good outcome was caused by taking good action or because of noise in the environment, As a result, applying action-free free IQL to stochastic environments will typically result in overestimating the value function, implicitly assuming that all noise is controllable. While will build our method upon Equation (4) in this work for simplicity, in line with many prior works on offline RL that employ sim- ilar assumptions [14, 33, 34, 41, 42, 93, 97], we believe correctly handling stochastic environments with advanced techniques (e.g.. by identifying controllable parts of the environment [92, 99]) is an interesting direction for future work.
4 Hierarchical policy structure for offline goal-conditioned RL
Coallcoonitionee offline RL provides a general framework for learning flexible policies from data, but the goal conl-coondtioned setting also presents an especially difficult multi-task learning problem for RL algorithms, particularly for long-horizen tasks where the goal is far away. In Section 4.1, discuss some possible reasons for this difficulty, from the perspective of the "signal-to-noise" ratio in the learned goal-conditioned value function. We then propose hierarchical policy extraction solution (Section 4.2) and compare the performances of hierarchical and flat policies in didactic environment, based on our theoretical analysis (Section 4.3).
4.1 Motivation: why non-hierarchical policies might struggle
One common strategy in offline RL is to first fit value function and then extract a policy that takes
actions leading to high values [3, 8, 29-31, 31, 49, 52, 67, 71, 95, 97, 98, 100]. This strategy can be directly applied to offline coal-conditioned RL by learning a goal-conditioned policy T[a st. g) that aims to maximize the learned coal-conditioned value function V( g), as in Equation (5). However, when the goal y is far from the current state 8, the learned goal-conditioned value function may not provide a clear learning signal for a flat, non-hierarchical policy. There are two reasons for this. First, the differences between the values of different next states (*++1,, (+++19)1 g)) may be small, as bad outcomes by taking suboptimal actions may be simply corrected in the next few steps, causing

---

()
V(s.9) g)
m(al/sr. 14. g]
x (e, 910s) of ( (144) 9
Goal
(a) Optimal value function
(h) Noisy value function
(c) Flat policy
(d) Hierarchical policy
Figure 2: Hierarchies allow us to better make use of noisy value estimates. (a) In this gridworld environment. the optimal value function predicts higher values for states that are closer the goal g (*) (b, However, a noisy value function results in selecting incorrect actions (-+) (d) Our method uses this same noisy value function to first predict an intermediate subgoal, and then select an action for reaching this subgoal. Actions selected in this way correctly lead to the goal.
only relatively minor costs. Second, these small differences can be overshadowed by the noise present in the learned value function (due to, for example, sampling error or approximation error), especially when the goal is distant from the current state, in which case the magnitude of the goal-conditioned value (and thus the magnitude of its noise or errors) is large. In other words, the Ssignal-to-noise ratio in the next time step values (($++1.9) can be small, not providing sufficiently clear learning signals for the flat policy. Figure illustrates this problem. Figure 2a shows the ground-truth optimal value function (* g) for given goal at each state, which can guide the agent to reach the goal. However, when noise is present in the learned value function (s, g) (Figure 2b), the flat policy (a 8,g) becomes erroneous, especially at states far from the goal (Figure 2c).
4.2 Our hierarchical policy structure
To address this issue, our main idea in this work, which we present fully in Section 5, is to separate policy extraction into two levels. Instead of directly learning single, flat, goal-conditioned policy (a st,9) that aims to maximize V(8+11,9). we extract both high-level policy (st+.. 8t.9) and a low-level policy ra(a St, S1+x), which aims to maximize ($ g) and V ((8++1, $1+k). respectively. Here, St+k can be viewed as waypoint subgoal. The high-level policy outputs intermediate subgoal states that are l steps away from s, while the low-level policy produces primitive actions to reach these subgoals. Although function in this way. this hierarchical scheme provides clearer learning signals for both policies. Intuitively, the high-level policy receives more reliable learning signal because different subgoals lead to more dissimilar values than primitive actions. The low-level policy also gets a clear signal (from the same value function) since it queries the value function with only nearby states, for which the value function is relatively more accurate (Figure 1b). As a result, the overall hierarchical policy can be more robust to noise and errors in the value function (Figure 2d).
extract both policies from the same learned value
4.3 Didactic example: hierarchical policies mitigate the signal-to-noise ratio challenge
To further understand the benefits of hierarchical policies, we study a toy example with one-dimensional state space (Figure 3). In this environment, the agent can move one unit to the left or right each time step. The agent gets a reward of 0 when it reaches the
goal; otherwise, always gets 11. The optimal goal-conditioned Figure 3: 1-D toy environment.
value function is hence given as V*(sgg) = - -ss- gi (assuming 7 = 1). We assume that the noise in the learned value function ((s.g) g) is proportional to the optimal value: i.e., (s V* 9) + + V* gs, where 22.g is sampled independently from the standard normal distribution and o is its standard deviation. This indicates that as the goal becomes more distant, the noise generally increases, a trend we observed in our experiments (see Figure 8).
In this scenario, we compare the probabilities of choosing incorrect actions under the flat and hierarchical policies. We assume that the distance between s and g is T (i.e., g = + and T > 1). Both the flat policy and the low-level policy of the hierarchical approach consider the goal-conditioned values at 8 + 1. The high-level policy evaluates the values at 8 + k; using k-step away subgoals. For the hierarchical approach, we query both the high- and low-level policies at every step. Given these settings, we can bound the error probabilities of both approaches as follows:
5

---

Puliey ervor boundl
50,
0.03
T' 50,
1000, 0.01
T 1000, 0.03
0.10-
0.5
0.4
0.00-
0.2
0.2
0.1
012
40
20
5.0
500
10.0
Subgoal steps &
Subgool steps
Subgoal steps
Sulgoal steps
Figure 4: Comparison of policy errors in flat vs. hierarchical policies in didactic environments. The hicrarchical policy, with an appropriate subgoal step, often yields significantly lower errors than the flat policy.
Proposition 4.1. In the environment described in Figure 3. the probability of the flat policy æ select- ing an incorrect action is given as E(x) = (-o/T=4)) and the probability of the hierarchical
policy ne Tth selecting an incorrect action is bounded as E(x² a T²) Σ
v2
(T////
ov where 4 denotes the cumulative distribution function of the standard normal distri-
22
bution, (xx) P(z < xi .2 _ e1
The proof can be found in Appendix E. We first note that each of the error terms in the hierarchical policy bound is always no larger than the error in the flat policy, implying that both the high- and low-level policies are more accurate than the flat policy. To compare the total errors, E(T) and E(r' n²), we perform a numerical analysis. Figure shows the hierarchical policy's error bound for varying subgoal steps in five different (T, a) settings. The results indicate that the flat policy's error can be significantly reduced by employing hicrarchical policy with an appropriate choice of k, suggesting that splitting policy extraction into two levels can be beneficial.
Hierarchical Implicit Q-Learning (HIQL)
Based on the hierarchical policy structure in Section 4, we now present a practical algorithm, which we call Hierarchical Implicit Q-Learning (HIQL), to extract hierarchical policies that are robust to the noise present in the learned goal-conditioned value function. We first explain how to train subgoal policy (Section 5.1) and then extend this policy to predict representations (learned via the value function), which will enable HIQI. to scale to image-based environments (Section 5.2).
5.1 Hierarchical policy extraction
As motivated in Section 4.2, split policy learning into two levels, with a high-level policy generating intermediate subgoals and a low-level policy producing primitive actions to reach the subgoals. In this way. the learned condiionit value function can provide clearer signals for both policies, effectively reducing the total policy eтror. Our method, HIQL, extracts the hicrarchical policies from the same value function learned by action-free IQL (Fquation (4)) using AWR-style objectives. While we choose to use action-free IQI. in this work, we note that our hierarchical policy extraction scheme is orthogonal the choice of the underlying offline RL algorithm used to train goal-conditioned value function.
HIQL trains both high-level policy Thi (5t4k St. g), which produces optimal k-step subgoals $14ks and low-level policy =5, (a se, $144), which outputs primitive actions, with the following objectives:
Jnhh (0%) Es (e A si+ks 9)) log thi (se4k st, st,
(6)
(0t) = E (ns [exp(3 A (s1,, a, sc+x)) log nô, (at sts st+x)],
(7)
where denotes the inverse temperature hyperparameter and we approximate A" St-ki g) as Ve, (S++&: g) Vo, (st.g) and A'(st, at, St+k) as Ve, (S1+1, (1441,8444)) Voy (St, St+k). We do not include rewards and discount factors in these advantage estimates for simplicity, as they are (mostly) constants or can be subsumed into the temperature 3 (see Appendix for further discussion). Similarly to vanilla AWR (Equation (5)), our high-level objective (Equation (6)) performs weighted regression over subgoals to reach the goal, and the low-level objective (Equation (7)) carries out weighted regression over primitive actions to reach the subgoals.

---

Algorithm 1 Hierarchical Implicit Q-Learning (HIQL)
1: Input: offline dataset D. action-free dataset Ds (optional, Ds D otherwise)
2: Initialize value function Vay (s, ((g)) with built-in representation @gg). high-level policy mb, (ze+4 (s,
low-level policy nē, [a ze4k), learning rates Ab, while not converged do
By + f-Bv Xv Voy Lv (0v) with (8t, 8s+1. g) ~ Ds
Train value function, Equation (4)
5: end while
6: while not converged do
7: Oi. <-d,. 0b + Aa Jn with (L4,,)) Ds
Extract high-level policy, Equation (6)
end while
9: while not converged do
10: 8g - 02 + de Vo, with (8t, de, 8541, Sten) ~ D Extract low-level policy, Equation (7)
11: end while
We note that Equation (6) and Equation (7) are completely separated from one another, and only the low-level objective requires action labels. As result, the value function and high-level policy of HIQL,, by further training them with potentially large amount of additional passive data. Moreover, the low-level policy is relatively easy to learn compared to the other components, as it only needs to reach local subgoals without the need for learning the complete global structure. This enables HIQL work well even with limited amount of action information, as we will demonstrate in Section 6.4.
leverage action-free data for both
5.2 Representations for subgoals
In nigh-dimensional domains, such txxel-based environments, directly predicting subgoals can be prohibitive or infeasible for the high-level policy. To resolve this issue, we incorporate representation learning into HIQL, letting the high-level policy produce more compact representations of subgoals. While one can employ existing action-free representation learning methods [34, 61, 68, 82] to learn state representations, HIQL simply uses intermediate layer of the value function representation, which can be proven to be sufficient for control. Specifically, we parameterize the goal- conditioned value function V(s, 9) with ((s, o(g)), and use ((g) as the representation of the goal. Using this representation, the high-level policy T² (2++k st.9) produces Z++A 6($++)) instead of Se+k, which the low-level policy ra 8,, 2(+R) takes as input to output actions (Figure la). In this way, we can simply learn compact goal representations that are sufficient for control with no separate training objectives or components. Formally, we prove that the representations from the value function are sufficient for action selection:
goal
Proposition 5.1 (Goal representations from the value function sufficient for action selection). Let V* g) be the value function for the optimal peward-maximice policy l' s.g) in a deterministic MDP. Let representation function o(g) be given. If this same value function can be represented in terms of goal representations ((g), then the reward-masn moxrrd-mximiizing policy can also be represented in terms of goal representations (()):
3 Vo(s, ((g)) s.1. Vo(s, ((9)) V (s.g) for all 8,
3 (a 8, ((g)) s.f. (a 8, p(g)) (a s. (. for all 8,g.
While Proposition 5.1 shows that the parameterized value function V(s, p())) provides a sufficient goal representation d, found that additionally concatenating the input á (i.e., using & @(gg. s)) instead of Φ(g)) [39] leads to better empirical performance (see Appendix A for details), and thus we use the concatenated variant of value function parameterization in our experiments. We provide pseudocode for HIQL in Algorithm I and the full training details in Appendices A and D.
6 Experiments
Our experiments will use six offline goal-conditioned tasks, aiming to answer the following questions:
1. How well does HIQL perform on a variety of goal-conditioned tasks, compared to prior methods?
2. Can HIQL solve image-based tasks, and are goal representations important for good performance?
Can HIQL utilize action-free data to accelerate learning?
4. Does HIQL mitigate policy errors caused by noisy and imperfect value functions in practice?
7

---

JE
L
(a) AntMaze
(b) Kitehen
(c) CALVIN
(n) Proegen Maze
(b) Visual AntMaze
(o) Roboverse
Figure 5: State-based benchmark environments.
Figure 6: Pixel-hased benchmark environments.
6.1 Experimental setup
We first describe our evaluation environments, shown in Figure 5 state- and Figure 6 (pixel- based). AntMaze [9, 87] is a class of challenging long-horizon navigation tasks, where the goal is to control an 8-DoF Ant robot to reach a given goal location from the initial position. We use the four medium and large maze datasets from the original D4RL benchmark [28]. While the large mazes already present a significant challenge for long-horizon reasoning, we also include two even larger mazes (AntMaze-Ultra) proposed by Jiang et al. [43]. Kitchen [35] is long-horizon manipulation domain, in which the goal is to complete four subtasks (e.g.. open the microwave or move the kettle) with a 9-DoF Franka robot. We employ two datasets consisting of diverse behaviors ('-partial' and '-mixed)) ) from the D4RL benchmark [28]. CALVIN [63], another long-horizz manipulation environment, also features four target subtasks similar to Kitchen. However, the dataset accompanying CALVIN [84] consists of much larger number of task-agnostic trajectories from 34 different subtasks, which makes it challenging for the agent to learn relevant behaviors for the goal. Procgen Maze [16] is a pixel-based maze navigation environment. We train agents on an offline dataset consisting of 500 or 1000 different maze levels with variety of sizes, colors, and difficulties, and test them on both the same and different sets of levels to evaluate their generalization capabilities. Visual AntMaze is vision-based variant of the AntMaze-Large environment [28]. We provide only a 64 x 64 x 3 camera image (as shown in the bottom row of Figure 6b) and the agent's s proprioceptive states, excluding the global coordinates. As such, the agent must learn to navigate the maze based on the wall structure and floor color from the image. Roboverse [25, 104] is a pixel-based, goal-conditioned robotic manipulation environment. The dataset consists of 48 x 48 x images of diverse sequential manipulation behaviors, starting from randomized initial object poses. We evaluate the agent's ' performance across five unseen goal-reaching tasks that require multi-stage reasoning and generalization. To train goal-conditioned policies in these benchmark environments, during training. we replace the original rewards with a sparse goal-conditioned reward function, r(8,9) g) = 0 (if 8 g), -1 (otherwise).
We compare the performance of HIQL with six previous behavioral cloning and offline RL methods. For behavioral cloning methods, we consider flat goal-conditioned behavioral cloning (GCBC) [19, 33] and hierarchical goal-conditioned behavioral cloning (HGCBC) with two-level policies For offline goal-conditioned RL methods, we evaluate a
variant of IQL [49] ("GC-IQL") (Section 3), which does not use hierarchy, and POR [97] ("GC-POR"), which uses hierarchy but does not use temporal abstraction (i.e., similar to k = 1 in HIQL) nor representation learning. In AntMaze, we additionally compare HIQL with two model-based approaches that studied this domain in prior work: Trajectory Transformer (TT) [41], which models entire trajectories with a Transformer [90], and TAP [43], which encodes trajectory segments with VQ-VAE [89] and performs model-based planning over latent vectors in hierarchical manner. We use the performance reported by Jiang et al. [43] for comparisons with TT and TAP. In our experiments, we use random sceds and represent 95% confidence intervals with shaded regions (in figures) or standard deviations (in tables), unless otherwise stated. We provide full details of environments and basclines in Appendix D.
6.2 Results on baste environments
We first evaluate HIQL in the five state-based environments LantMaze[MMediun Large, Ultra), Kitchen, and CALVIN) using nine offline datasets. We evaluate the performance of the learned policies by commanding them with the evaluation goal state g (i.e., the benchmark task target position in AntMaze, or the state that corresponds to completing all four subtasks in Kitchen and CALVIN), and measuring the average return with respect to the original benchmark task reward function. We test state-based environments. Table and
two versions of HIQL (without and with representations) Figure 7a show the results on the nine offline datasets, indicating that HIQL mostly achieves the best performance our experiments. Notably, HIQL. attains an 88% success rate on AntMaze-Lagge and 53% on AntMaze- Ultra, which is, to the best of our knowledge, better than any previously reported
8

---

Table 1: Evaluating HIQL on state-based offiin goal-conditioned RI. HIQL mostly outperforms six baselines on variety of benchmark tasks, including on different types of data. We show the standard deviations across 8 random seeds and refer to Appendix B for the full training curves. Baselines: GCBC [33]. HGCBC [35].
GC-IQL [49], GC-POR [97], TAP [43], TT [41].
Dataset
GCBC
HGCBC
GC-IQL
GC-POR
TAP
TT
HIQL (ours)
HIQL (wlo repr.
antimize reddii-ddivenss
67.3 $10.3
71.6 ss.
63.5 si4.
74.8 11.
85.0
100.0
56,8 144
89.9 6.35
antmaae medium play
71.9
66.3
70.9
71.4 210s
78.0
93.3
84.1
87.0
antmmiz-Hiee laee--largeeeeeee diverse
20.2 19.1
63.9 1ise
30.7 1:sa
49.0 m17.
82.0
60.0
88.2 a5.3
87.3 6:37
antmaze-large pargeplaay
23.1 #15.
61.7 #145
56.5 +144
63.2 =16.
71.0
06.7
86. 27.
81.2 =5.6
crimaze-ulraadiveee
14.4 19.7
30.4 a250
21,821122
29.8 =18
26.0
33.3
52.9 2:T4
52.6 2:17
antmaze ultra play-play
20.7 29.7
38.2 gr61
29.5
2915
g1.4
22.0
20.0
39.2
56.0 2:2.4
kitchen partial
38.5 aiLs
32.0 2167
39.2 233.5
18.4 214
65.0 300
46.3 29.6
kitchen-mixed
46.7
46.8
51.3
27.9
67.7
36.8 sa0.1
calvin
17.3 211.
3.1 25.8
7.8 2:10
12.4 1184
43.8 g300
23.4 a07.1
Table Evaluating HIQL on pixel-based offline goal-conditioned RL. HIQL scales to high-dimensional pixel-based enviromments with latent subgoal representations, achieving the best performance across the environ- ments. We refer to Appendix B for the full training curves.
Dataset
GCBC HGCBC (ppp)
GC-IQL
GC-POR (+
HIQL (ours)
pmegen-maze waaegenn-mze---00 -aa----a---------iiiii 500-rrain
16.8 2o1
14.3 2:1
22.5
75.8 1:8.1
82.5 10.0
proggen maze
14.5 45a
11.2 2a7
19.
53.8 1145
procgen-maze- 10000-r-n
27.2 sa.
15.0 e5.7
78.2 a7.2
82.0 e5.5
87.0 11..5
proegen maze test
12.0 as.
14.5 a5.
110.6
69.8 474
78.2 g:7.
Siual-aotmaazzdivens
7144
36.1
72.6 as.
47.4 217e
80.5 19.4
visual antmaaze play
64.4 a
23.8 28.5
70.4 200.6
57.0 1
1:4
visual antmuze navigate
33.2 an.
21.4 844
22.1 amaa
16.1 aina
45.7 a13.3
roboverse
26.2 84.5
26.4 864
31.2 19.7
46.6 67.4
61.5 15.3
IQM
IQM
HIQL GC-POR- GC-IQL- HGCBC- GCBC-
HIQL
GC-POR (+ rent
GC-IQL HGCBC (+ penor J
GCBC
0,30
0.45
0.60
0.75
0.2
0.4
0.6
Normalized Score
Normalized Score
(a) State-based environments
(b) Pixel-based environments
Figure 7: Performance comparison. Following the protocol proposed by Agarwal [1], we report the interquartile mean (IQM) metrics to evaluate the statistical significance of HIQL's performance on offline goal-conditioned RL benchmarks (Tables I and 2). We refer to Appendix B for the full Rliable plots.
result on these datasets. In manipulation domains, we find that having latent subgoal representations in HIQL. is important for enabling good performance. In CALVIN, while other methods often fail to achieve any of the subtasks due to the high diversity in the data, HIQL completes approximately two subtasks on average.
6.3 Results on bixel- environments
Next, to verify whether HIQL can scale to gigh-dimensional environments using goal representations, we evaluate our method on three pixel-based domains (Procgen Maze, Visual AntMaze, and Robo- verse) with image observations. For the prior hierarchical approaches that generate (HGCBC and GC-POR), we apply HIQL's 's balue- representation learning scheme to enable them handle the high-dimensional observation space. Table and Figure 7b present the results, showing that our hierarchical policy extraction scheme, combined with representation learning, improves performance in these image- based environments as well. Notably, in Procgen Maze, HIQL exhibits larger gaps compared to the previous methods on the test sets. This is likely because the high-level policy can generalize better than the flat policy, as it can focus on the long-term direction toward the goal rather than the maze's 's detailed layout. In Roboverse, HIQL is capable of generalizing to solve unseen robotic manipulation tasks purely from images, achieving an average success rate of 62%.
subgoals
6.4 Results with action-free data
As mentioned in Section 5. one of the advantages of HIQL its ability to leverage a potentially large amount of passive (action-free) data. To empirically verify this capability, we train HIQL on action- limited datasets, where we provide action labels for just 25% of the trajectories and use state-only trajectories for the remaining 75%. Table 3 shows the results from six different tasks, demonstrating
9

---

Table 3: HIQL can leverage passive, action-free data, Since method requires action information only for the low-level policy, which is relatively easier to learn, HIQL mostly achieves comparable performance with just 25% of action-labeled data, outperforming even baselines trained on full datasets.
Datuset
GC-IQL (full)
GC-POR (full)
HIQL (full)
HIQL (action-limite) vs. HIQL (full)
vs. Prev. best (full)
animaze- large diverse
50.7
19.0
88.2
88.9
+0.7
+-38.2
aolmmas-ultta-divnss
21.6 115.2
29.8 1114
4279
38,2 115.
-11.7
+8.4
kitchen mixed
51.3 4:za
27.9 4174
67.7 sea
59.1 194
-8.6
+7.8
calvin
12.4
35.8
-8.0
4-23.4
proegen-mann muze 500 train
72.5 sino
75.8 si2..
K2.5 s00
77.0 a:a.
5.55
11.2
procgen-maze- maze-500-rest
40.5 4
63.8
64.5 2107
65.5 21.4
41.0
+-11.7
Policy necuracy metric
GC-IQL.
GC-POR
(empe)
HIQL (ours)
All goals (train)
61.9 =13
64.3 #2.7
66.6 =21 (+2.3)
All goals (test)
60.3
63.6
68.0 2s1 (+1.4)
Distant goals (train)
49.3 ssa
48.1 14.
56.8
ax0 (47.5)
Distant goals (test)
47.5 ss.
47.2 a9.9 (+120)
Dotea JdNOE CEC
Figure 8: Value and policy errors in Proegen Maze: (left) As the distance between the state and the goal increases, the learned value function becomes noisier. (middle) We measure the accuracies of learned policies. (right) Thanks our hierarchical policy extraction scheme (Section 4.2), HIQL exhibits the best policy accuracy, especially when the goal is far away from the state. The blue numbers denote the accuracy differences between HIQL and the second-best methods.
that HIQL, even with limited amount of action information, can mostly maintain its original performance. Notably, action-limite HIQL still outperforms previous offline RL methods (GC-IQL and GC-POR) trained with the full action-labeled data. We believe this is because HIQL learns majority of the knowledge through hierarchical subgoal prediction from state-only trajectories.
6.5 Does HIQL mitigate policy errors caused by noisy value functions?
To empirically verify whether our two-level policy architecture is more robust to errors in the learned value function (i.e., the "signal-to-noise" ratio argument in Section 4), we compare the policy accuracies of GC-IQL (flat policy), GC-POR (hierarchy without temporal abstraction), and HIQL (ours) in Procgen Maze, by evaluating the ratio at which the ground-truth actions match the learned actions. We also measure the noisiness (i.e., standard deviation) of the learned value function with respect to the ground-truth distance between the state and the goal. Figure 8 shows the results. We first observe that the noise in the value function generally becomes larger as the state- goal distance increases. Consequently, HIQL achieves the best policy accuracy, especially for distant goals (s.st(ss,g g) 2 50), as its hierarchical policy extraction scheme provides the policies with clearer learning signals (Section 4.2).
We refer to Appendix C for further analyses, including subgoal visualizations and an ablation study on subgoal steps and design choices for representations.
Conclusion
We proposed HIQL as a simple yet effective hierarchical algorithm for offline goal-conditioned RL. While hierarchical RL methods tend to be complex, involving many different components and objec- tives, HIQL shows that it is possible to build a method where a single value function simultancously drives the learning of the low-level policy, the high-level policy, and the representations a relatively simple and casy-to-train framework. We showed that HIQL not only exhibits strong performance in various challenging coal-conditioned tasks, but also can leverage cetion-free data and enjoy the benefits of built-in representation learning for image- based tasks.
Limitations. One limitation of HIQL is that the objective for its action-free value function (Bqua- tion (4)) is unbiased only when the environment dynamics are deterministic. As discussed in Section 3, HIQL (and other prior methods that use action-free videos) may overestimate the value function in partially observed or stochastic settings. To mitigate the optimism bias of HIQL in stochastic environ- ments, we believe disentangling controllable parts from uncontrollable parts of the environment can be one possible solution [92, 99], which we leave for future work. Another limitation of our work is that we assume the noise for each V(s, g) is independent in our theoretical analysis (Proposition 4.1). While Figure 8 shows that the "signal-to-noie the independence assumption in our theorem might not hold in environments with continuous state spaces, especially when the value function is modeled by smooth function approximator.
argument empirically holds in our experiments,
10

---

Acknowledgments
We would like thank Aviral Kumar for an informative discussion about the initial theoretical results, Chongyi Zheng, Kuan Fang, and Fangchen Liu for helping set up the Roboverse environment, and RAIL members and anonymous reviewers for their helpful comments. This work was supported by the Korea Foundation for Advanced Studies (KFAS). the Fannie and John Hertz Foundation, the This
NSF GRFP (DGE2140739) AFOSR (FA9550-22-) 1 1-0273), and ONR research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley.
References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Mare G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Neural Information Processing Systems (NeurIPS), 2021.
[2] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofrr Nachum. Opal: Offline primitive discovery for accelerating offiie reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
[3] Gaon An, Scungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline rein- forcement learning with diversified q q-ensemble. In Neural Information Processing Systems (NeurIPS), 2021.
[4] Marcin Andrychowiez, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abheel, and Wojeiech Zaremba. Hindsight experience replay. In Neural Information Processing Systems (NeurIPS), 2017.
[5] Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/1607.06455 2016.
[6] Pue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI Conference on Artificial Intelligence (AAAI), 2017.
[7] Bowen Baker, lige Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. In Neural Information Processing Systems (NeurIPS), 2022.
[8] David Brandfonbrener. William Whitney, Rajesh Ranganath, and Joan Bruna. Offline without off policy evaluation. In Neural Information Processing Systems (NeurIPS), 2021.
[9] G. Brockman, Vicki Cheung, Ludwig Pettersson, J. Schneider, John Schulman, Jie Tang, and W. Zaremba. OpenAl Gym. ArXiv, abs/1606.01 01500000014..... 2016.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Neural Information Processing Systems (NeurIPS), 2020.
[11] Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning (ICML), 2021.
[12] Matthew Chang, Arjun Gupta, and Saurabh Gupta. Learning value functions from undirected state-only experience. In International Conference on Learning Representations (ICLR), 2022.
[13] Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan C. Julian, Chelsea Finn, and Sergey Levine. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In International Conference Machine Learning (ICML) 2021.
[14] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems (NeurIPS), 2021.

---

[15] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning (ICML). 2020.
[16] Karl Cobbe, Christopher Hesse, Jacoh Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforeement learning. In International Conference on Machine Learning (ICMI.), 2020.
[17] Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In Neural Information Processing Systems (NeurIPS), 1992.
[18] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HILT), 2019.
[19] Yiming Ding, Carlos Florensa, Mariano Phiclipp, and P. Abbeel. Goal-conditioned imitation lcarning. In Neural Information Processing Systems (NeurIPS), 2019.
[20] Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2021.
[21] Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018.
[22] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2019.
[23] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. In International Conference on Learning Representations (ICLR), 2021.
[24] Benjamin Bysenhach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal-conditioned reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2022.
[25] Kuan Fang, Patrick Yin, Ashvin Nair, and Sergey Levine. Planning to practice: Efficient online fine-tuning by composing goals in latent space. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
[26] Kuan Fang, Patrick Yin, Ashvin Nair, Homer Walke, Gengchen Yan, and Sergey Levine. Generalization with lossy affordances: Leveraging broad offline data for learning visuomotor tasks. In Conference on Robot Learning (CoRL), 2022.
[27] Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang. Dher: Hindsight experience replay for dynamic goals. In International Conference on Learning Representations (ICLR), 2019.
[28] Justin Fu, Aviral Kumar, Ofir Nachum, Tucker, and Sergey Levine. D4rl: Datasets for deep reinforcement learning. ArXiv, abs/2004.072119 2020.
[29] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning (ICML), 2019.
[30] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learnnn:: Maxent rl without entropy. In International Conference on Learning Representations (ICLR), 2023.
[31] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learming operator for simple yet effective offline and online rl. In International Conference on Machine Learning (ICML), 2021.
[32] Dibya Ghosh. dibyaghosh/faxrl_ng m. 2023. URL htttps://giiaie com/ http://///..baaahss/sssrr...
[33] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eyseubach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations (ICLR), 2021.
[34] Dibya Ghosh, Chethan Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. In International Conference on Machine Learning (ICML), 2023.
[35] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long- horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning (CoRL), 2019.
12

---

[36] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR), 2022.
[37] Dan Hendrycks and Kevin Cimpel. Gaussian error linear units (gelus). ArXiv, abs/1606.0844 2016.
[38] Christopher Hoang, Sungryull Sohn, Jongwook Choi, Wilka Carvalho, and Honglak Lec. Successor feature landmarks for long-horizon goal goal-conditioned reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2021.
[39] Zhang- Chang-Wei Hong, Ge Yang, and Pulkit Agrawal. Bilinear value networks. In International Conference on Learning Representations (ICLR), 2022.
[40] Zhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching. In Neural Information Processing Systems (NeurIPS), 2019.
[41] Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning big sequence modeling problem. In Neural Information Processing Systems (NeurIPS), 2021.
[42] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning (ICML), 2022.
[43] Zhengyao Jiang. Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktaschel, Edward Grefenstette, and Yuandong Tian. Efficient planning in a compact latent action space. In International Conference on Learning Representations (ICLR), 2023.
[44] Leslie Pack Kaclbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence (ICCAI), 1993.
[45] Junsu Kim, Younggyo Seo, and Jinwoo Shin. Landmark-guided subgoal generation in hierarchical reinforcement leaming. In Neural Information Processing Systems (NeurIPS), 2021.
[46] Junsu Kim, Younggyo Seo, Sungsoo Ahn, Kyunghwan Son, and Jinwoo Shin. Imitating graph-based planning with goal-conditioned policies. In International Conference on Learning Representations (ICLR), 2023.
[47] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
[48] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations (ICLR), 2021.
[49] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. International Conference on Learning Representations (ICLR), 2022.
[50] Sanjay Krishnan, Roy Fox, lon Stoica, and Ken Goldberg. Ddeo: Discovery deep continuous options for robot learning from demonstrations. In Conference on Robot Learning (CoRL), 2017.
[51] Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Sacedi, and Joshua B. Tenenbaum. Hicrarchical deep reinforcement lcarning: Integrating temporal abstraction and intrinsic motivation. In Neural Information Processing Systems (NeurlPS), 2016.
[52] Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. In Neural Information Processing Systems (NeurIPS), 2020.
[53] Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron C. Courville, G. Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement leamingg requires explicit regularization. In International Conference on Learning Representations (ICLR), 2022.
[54] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Reinforcement learning: State-of-the-ort, pages 45-73. Springer, 2012.
[55] Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu, Offline reinforcement learning: Tutorial, review, and perspectives open problems. ArXiv, abs/2005.01643 2020.
[56] Andrew Levy, George Dimitri Konidaris, Robert W. Platt. and Kate Saenko. Learning multi-level hierarchies with hindsight. In International Conference Learning Representations (ICLR), 2019.
13

---

[57] Alexander C. Li, Lerrel Pinto, and P. Abbeel. Generalized hindsight for reinforcement learning. In Neural Information Processing Systems (NeurlPS), 2020.
[58] Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Hierarchical planning through goal- conditioned offline reinforcement learning. IEEE Robotics and Automation Letters (RA-L). 7(4): 10223, 2022.
[59] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning (CoRL), 2019.
[60] Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goal- conditioned reinforcement learning via f-advantage regression. In Nearal Information Processing Systems (NeurIPS), 2022.
[61] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Oshert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training In International Conference on Learning Representations (ICLR), 2023.
[62] Marlos C. Machado, Marc G. Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. In International Conference on Machine Learning (ICML), 2017.
[63] Oier Mees, Lukas Hermann, Erick Rosete-Beas, Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditione policy learning for oag-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3):7327-7334, 2022.
[64] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
[65] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. In Neural Information Processing Systems (NeurIPS), 2018.
[66] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learning Representations (ICLR),
2019.
[67] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. ArXiv, abs/2006.099 09359, 2020.
[68] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhi Gupta. R3m: A universal visual representation for robot manipulation. In Conference on Robot Learning (CoRL), 2022.
[69] Soroush Nasiriany, Vitchyr H. Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. Neurol Information Processing Systems (NeurIPS), 2019.
[70] Gerhard Neumann and Jan Peters. Fitted q-iteration by advantage weighted regression. In Neural Information Processing Systems (NeurIPS), 2008
[71] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. weightege-weighted regression: Simple and scalable off-policy reinforcement learning. ArXiv, abs/1910.00177, 2019.
[72] Karl Pertsch, Youngwoon Lee, and Joseph Lim, Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL.) 2020.
[73] Jan Peters and Stefan Schaal. Reinforcement learning by rewird-weighted regression for operational space control. In International Conference on Machine Learning (ICML), 2007.
[74] Jan Peters, Katharina Muelling, and Yasemin Altun. Relative entropy policy search. AAAI Conference on Artificial Intelligence (AAAI). 2010.
[75] Vitchyr H. Pong. Shixiang Shane Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free free deep rl for model based control. In International Conference on Learning Representations (ICLR), 2018
[76] Erick Rosete- Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task-agnostic offline reinforcement learning. In Conference on Robot Learning (CoRL), 2022.
[77] Sasha Salter. Markus Wulfmeier, Dhruva Tirumala, Nicolas Manfred Otto Heess, Martin A. Riedmiller, Raia Hadsell, and Dushyant Rao. Mo2: Model-based offline options. In Conference on Lifelong Learning Agents (CoLLAs), 2022.
14

---

[78] Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi- Cemi-parametric topological memory for navigation. In International Conference Learning Reprexentations (ICLR), 2018.
[79] Tom Schaul, Dan Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning (ICML), 2015.
[80] Karl Schmeckpeper, Oleh Rybkin, Kostas Danillidis, Sergey Levine, and Chelsea Finn. Reinforcement learning with videos: Combining offline observations with interaction. In Conference on Robot Learning (CoRL), 2020.
[81] Jürgen Schmidhuber. Learning to generate sub sub-gooals for action sequences. In Artificial neural networks, 1991.
[82] Younggyo Seo, Kimin Lee, Stephen James, and Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning (CCML),202)
[83] Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinchart, and Sergey Levine. Recon: Rapid exploration for open-world navigation with latent goal models. In Conference on Robot Learning (CoRL), 2021.
[84] Lu Shi, Joseph Lim, and Youngwoon Lee. Skill-based model- reinforcement learning. In Conference on Robot Learning (CoRI.), 2022.
[85] Martin Stolle and Doina Precup. Learning options in reinforcement learning. In Symposium on Abstruction, Reformulation and Approximation, 2002.
[86] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):81-211 1999.
[87] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference on Inteligent Robots and Systems (IROS), 2012.
[88] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In International Joint Conference on Artificial Intelligence (IJCAI), 2018.
[89] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Neural Information Processing Systems (NeurIPS), 2017.
[90] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Neural Information Processing Systems (NeurIPS), 2017.
[91] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Manfred Otto Heess, Max Jaderherg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning (ICMI),, 2017.
[92] Adam Villaflor, Zheng Huang, Swapnil Pande, John M. Dolan, and Jeff G. Schneider. Addressing optimism bias in sequence modeling for reinforcement learning. In International Conference on Machine Learning (ICML), 2022.
[93] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang, Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference Machine Learning (ICML), 2023.
[94] Ziyun Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott E. Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Manfred Otto Heess, and Nando de Freitas. Critic regularized regression. In Neural Information Processing Systems (NewrIPS), 2020.
[95] Yifan Wu, G. Tucker, and Ofrr Nachum. Behavior regularized offline reinforcement learning. ArXiv, abs/1911. 1. 1361, 2019.
[96] Markus Wulfineirr, Dushyant Rao, Roland Hafner, Thomas Lampe, Abbas Abdolmaleki, Tim Hertweck, Michael Neunert, Dhruva Tirumala, Noah Siegel, Nicolas Manfred Otto Heess, and Martin A. Riedmiller. Data-efficient hindsight off off-policy option learning. In International Conference on Machine Learning (ICML), 2021.
[97] Haoran Xu, Li Jiang, Jianxiong Li, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning, In Neural Information Processing Systems (NeurIPS), 2022.
15

---

[98] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang. Victor Chan, and Xianyuan Zhan. Offline with ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations (ICLR), 2023.
[99] Mengjiao Yang, Dale Schuurmans, P. Abbeel. and Ofrr Nachum. Dichotomy of control: Separating what you can control from what you cannot. International Conference Learning Representations (ICLR). 2023.
[100] Rui Yang. Yiming Wenzhe Li, Hao Sun, Meng Fang. Yali Du Xiu Han, and Chongjie Zhang. Rethinking goal-conditiones supervised learning and its connection to offline rl. In International Conference on Learning Representations (ICLR), 2022.
[101] Lunjun Zhang, Ge Yang. and Bradly C. Stadie. World model as a graph: Learning latent landmarks for planning. In International Conference on Machine Learning (ICML), 2021.
[102] Tianjun Zhang, Benjamin Eysenbach, Ruslan Salakhutdinov, Sergey Levine, and Joseph Gonzalez. C- planning: An automatic curriculum for learning goal-reaching tasks. In International Conference on Learning Representations (ICLR), 2022.
[103] Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency-constraiin subgoals in hierarchical reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.
[104] Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdinov, and Sergey Levine. Stabilizing contrastive rl: Techniques for offline goal reaching. ArXiv, abs/2306.033466, 2023.
[105] Qinqing Zheng, Mikael Henaff, Brandon Amos, and Aditya Grover. Semi- supervise offline reinforcement learning with action-free free trajectories. International Conference on Machine Learning (ICML), 2023.
16

---

A Training details
Goal distributions. We train our goal-conditioned value function, high-level policy, and low-Ievel policy respectively with Equations (4), (6) and (7), using different goal-sampling distributions. For the value function (Equation (4)), we sample the goals from either random states, futures states, or the current state with probabilities of 0.3, 0.5, and 0.2, respectively, following Ghosh et al. [34]. We use Geom(1 - 7) for the future state distribution and the uniform distribution over the offline dataset for sampling random states. For the hierarchical policies, we mostly follow the sampling strategy of Gupta et al. [35]. We first sample a trajectory (80, 81,.... 8t, ....... 8T) from the dataset Dss and state from the trajectory. For the high-level policy (Fquation (6)), we either (i) sample g uniformly from the future states 8t, (Lg > 1) in the trajectory and set the target subgoal to Smin(t+k.t,) or (ii) sample uniformly from the dataset and set the target subgoal to $min(t+k.T). For the low-level policy (Equation (7)), we first sample a state 9 from D, and set the input subgoal to Smin(t+k,,)) in the same trajectory.
Ad dvvatagee estimates. principle, the advantage estimates for Equations (6) and (7) are respec- tively given
(sr; 9) = VMv ($1+E: 9) Σ r(sr,g) Vor (vc,
(8)
Asss, ae, $1tx) = VVo (SL+1, (1t1) + r(s,, $(tb) - Vor (s2, B(+b))
(9)
where we use the notations ke and bitk to incorporate the edge cases discussed in the previous paragraph (i.e., k k= min(k, 1 1) when we sample y from future states, k = min(k, T () when we sample g from random states, and 5e+s = #min(t+*,TT)) Here, we note that 8 * 9 and 8 * 8 always hold except for those edge cases. Thus, the reward terms in Equations (8) and (9) are mostly constants (under our reward function r(s, g) = (if s = g), -1 (otherwise)). )) as are the third terms (with respect to the policy inputs). As such, we practically ignore these terms for simplicity, and this simplification further enables us to subsume the discount factors in the first terms into the temperature hyperparameter B. We hence use the following simplified advantage estimates, which we empirically found to lead to almost identical performances in our experiments:
(se, S14k. 9) = Voy ())+k:9) (s 0z, 814k) = Ver ($t+1, Sctk) - Ver (sc, Sc+k)-
9) - Vov (8),9),
(10)
(11)
State representations. We model the out-
put of the representation function <(g) in V(s,((g)) with 10 10-dimensional latent vector and normalize the outputs of Φ(g) [53]. Empiri- cally, we found that concatenating to the input (i.e., using of ([[g,>) s)) instead of Φ(g). Figure 9), similarly to Hong et al. [39], improves perfor- mance in our experiments. While this might lose the sufficiency property of the representa- tions (i.e., Proposition 5.1), we found that the representations obtained in this way generally lead to better performance in practice, indicating that they still mostly preserve the goal informa- tion for control. We believe this is due to the imposed bottleneck on & by constraining its effective dimensionality to 9 (by using normalized 10-dimensional vectors), which enforces to retain bits regarding and to reference 8 only when necessary. Additionally, in pixel-based environments, we found that allowing gradient flows from the low-level policy loss (Equation (7)) to @ further improves performance. We ablate these choices and report the results in Appendix C.
A
([s se])
g
V
V(s.g)
6( kr 8c))
x x((a lis #e44)
Figure 9: Full architecture of HIQL. In practice, we (s,, o[19, s])) instead of (ss, ((g)) as we found that the former leads to better empirical performance.
Policy execution. At test time, we query both the high-level and low-level policies at every step, without temporal abstraction. We found that fixing subgoal states for more than one step does not significantly affect performance, we do not use temporal abstraction for simplicity.
17

---

mndmaaz--ediim-ddiee diverse
antiaaze meniiiz--mdii---eee play
antmaze- large-ivveee
HIQL
HIQL (w/l repc.)
6.5
GU-POR
GO IQL
RGCDC
x
Steps
GCDO
azttmaaz- orttinsz-llagg- large-alaall play
antmaae- antmaaz--ulra-dnne ultra-divers
antmaze ultra play
'T'
0.5
0:22
0.25
0L5
Stepa
Stops calvin
kitchen-partial
kitchen-mixed
Steps
X10
X12
Steps
Figure 10: Training curves for the results with state-based environments (Table 1), Shaded regions denote the 95% confidence intervals across random seeds.
pro preegre-maann-----os
HIQL
GCPOR (- repr.)
GCIQL
0,25
IGCDCC (1 rept.)
600
xX0
GCnC
0,30
Stope
Stape
Strpe
Stope
Figure 11: Training curves for the results with pixel-based environments (Table 2). Shaded regions denote the 95% confidence intervals across random seeds.
dnown
pil2 and place Jabao
nidk plan (llswe)
prsh Reck drows
Mock, done dlenee
HIQU
GC-POR repe
GC-RQU
(+ ngr)
OUBBC
Figure 12: Training curves for the five tasks [104] in Roboverse. Shaded regions denote the 95%
confidence intervals across random seeds.
We provide pseudocode for HIQL in Algorithm 1. We note that the high- and low-level policies can be jointly trained with the value function as well, in Kostrikov et al. [49].
B Additional Plots
include the training curves for Tables to Figures 10, and 13, respectively. We also provide the training curves for each of the five tasks [104] in Roboverse in Figure 12. We include the Rliable [1] plots in Figures and 15. We note that the numbers in Tables are normalized scores (see Appendix D), while the returns in the figures unnormalized ones.
18

---

antmazee lagg-iii-e-ee--ieeeeeee
antmaze-ullre ullaiivveeetra--iveeeee
kitchen-mixed
IIQL (action-limited)
Retur
HIQL (full)
GC-POR (full)
GC-IQL (till)
Steps
x
x
calvin
procgen- -aa-e-------------- -aa-ggu---mazz-00-ttrri 300-train
procgen maze- www.---z--------- 500-1est
0.75
0.0
Return
0.93
9.05-
0.5
0.00
Stepa
Stege
x1F
Steps
3010
Figure 13: Training curves for the results with action-free free data (Table 3). Shaded regions denote the 95% confidence intervals across random seeds.
Median
IQM
Mean
Optimality Gap
HIQL-
HIQL (w/o repr.)
GC-POIt GC-QLL
HGCBC
GCBC
0.30
0.45
0.80
0.150.30
0.45
0.60
0.75
0.45
0.6D
0.30
0.45
0.60
Normalized Soore
Figure 14: Rliable plots for state-based environments.
Median
IQM
Mean
Optimality Gap
HIQL- GC-POR (+ repr.) GC-IQL
HGCBC (+ mpr.) GCBC-
0.4
0.6
0.8
0.22
0.4
0.6
0.3
0.30
0.45
0.00
0.73
0.30
0.45
0.60
0.75
Normalized Soore
Figure 15: Rliable plots for pixel-based environments.
large-diverse (w/o repr.)
antrasse-lacge-play (wyo repe.)
antmmase wasge-di-geerrrr (w/ repe.)
autenase- pedemase-larggrr play zepr.)
Beturn
0.5
0.5
0.04
0L04 0.0
0.04
0.0
0.0
0.5
0.5 Stops
0.0
0is Steps
6.0
815 Stepo
Shepe
x1e
XII
x1I
x13
kiteber- partial (w/ repr.)
kischen mixsed repr.)
saivin (w/
Subgool stops
25
Steps
Steps
Steps
Figure 16: Ablation study of the subgoal steps HIQL generally achieves the best performances when is between 25 and 50. Even when & is not within this range, HIQL mostly maintains reasonably good performance unless is is too small (i.e., S 5). Shaded regions denote the 95% confidence intervals across 8 random seeds.
19

---

antmaze-laa antmazz-large-diveee.
kitchen-partial
calvin
bgg. 8))
0.70
Φ
0.30
1-
((g)
w/o repr.
0/01
0.0
11.5
Steps
Stape
Stepe
Figure 17: Ablation study of different parameterizations of the representation function. Passing and together to & improves performance in general. Shaded regions denote the 95% confidence intervals across 8 random seeds.
pregen--raze Ali-train
proggnn-maee mazze-500-tsst
w/ policy grad. w/o policy grad.
Neture
0.5-
00
000
000
Stage
Strge
Figure 18: Ablation study of the auxiliary gradient flow from the low-level policy loss é on pixel-based ProcGen Maze. This auxiliary gradient flow helps maintain goal information in the representations. Shaded regions denote the 95% confidence intervals across 8 random seeds.
C Ablation Study
Subgoal steps. To understand how the subgoal steps affect performance, we evaluate HIQL with six different & E (1, 15, 25, 50, 100) AntMaze, Kitchen, and CALVIN. On AntMaze, test both HIQL with and without representations (Section 5.2). Figure 16 shows the results, suggesting that HIQL generally achieves the best performance with le between 25 and 50. Also, HIQL still maintains reasonable performance even when k is not within this optimal range, unless k is too small.
Representation parameterizations. We evaluate four different choices of the representation function 6 in HIQL: (([9, (0--(10) s). ((g), and without á. Figure 17 shows the results, indicating that passing y and together to Φ generally improves performance. We hypothesize that this is because é,, when given both g and s, can capture contextualized information about the goals (or subgoals) with respect to the current state, which is often easier to deal with for the low-level policy, For example, in AntMaze, the agent only needs know the relative position of the subgoal with respect to the current position.
Auxiliary gradient flows for representations. We found that in pixel-based environments (e.g., Procgen Maze), allowing gradient flows from the low-level policy loss the representation function improves performance (Figure 18). We believe this is because the additional gradients from the policy loss further help maintain the information necessary for control. We also (informally) found that this additional gradient flow occasionally slightly improves performances in the other environments as well, but we not enable this feature state-based environments keep our method as simple as possible.
D Implementation details
We implement HIQL based on JaxRL Minimal [32]. Our implementation is available at the following run our experiments on an internal GPU cluster composed of TITAN RTX and A5000 GPUs. Each experiment state-based environ- ments takes no more than 8 hours and each experiment on pixel-based environments takes no more than 16 hours.
repository: https://githut comm/seohoggarr//HIC
20

---

D.1 Environments
AntMaze [9, 87] We use the "antmaze- aantmaze-medium-dee -evdiuum-ivvere-v2.,, "antmaze-meddee aredium-play-v22 "antmaze- large-diverse-v2', and "antmaze-large-llayyv UntMa we use the Satmazz-ulrraddivere--0 and "antmaze-ultra-play posed by Jiang et al. [43]. The maze in the AntMaze-Ulrra task is twice the size of the largest maze in the original D4RL datasct. Each dataset consists of 999 length-1000 trajectories, in which the Ant agent navigates from an arbitrary start location to another goal location, which does not necessarily correspond to the target evaluation goal. At test time, to specify a goal g for the policy, we set the first two state dimensions (which correspond to the r-y coordinates) to the target goal given by the environment and the remaining proprioceptive state dimensions to those of the first observation in the dataset. At evaluation, the agent gets reward of 1 when it reaches the goal.
datasets from the D4RL benchmark [28]. For
datasets pro-
Kitchen [35]. We use the 'kitchen- paatheen-prtial-OO and "kitchen-mixed-ee datasets from the D4RL benchmark [28]. Each dataset consists of 136950 transitions with varying trajectory lengths (approxi- task, the goal is to achieve the four subtasks of opening the microwave, moving the kettle, turning on the light switch, and sliding the cabinet door. The dataset contains a small number of successful trajectories that achieve the four subtasks. In the 'kitchen-mixed-vo task, the goal is to achieve the four subtasks of opening the microwave, moving the kettle, turning on the light switch, and turning on the bottom left burner. The dataset does not contain any successful demonstrations, only providing trajectories that achieve some subset of the four subtasks. At test time, to specify a goal g for the policy, we set the proprioceptive state dimensions to those of the first observation in the dataset and the other dimensions to the target kitchen configuration given by the environment. At evaluation, the agent gets reward of 1 whenever it achieves a subtask.
mately 227 steps per trajectory on average). In the Kitchen-partial-v
CALVIN [63]. We the offline dataset provided by Shi et al. [84], which is based the teleoperated demonstrations from Mees et al. [63]. The task is to achieve the four subtasks of opening the drawer, turning on the lightbulb, sliding the door to the left, and turning on the LED. The dataset consists of 1204 length-499 trajectories. In each trajectory, the agent achieves some of the 34 subtasks in an arbitrary order, which makes the dataset highly task-agnostic [84]. At test time, specify goal for the policy, we set the proprioceptive state dimensions to those of the first observation in the dataset and the other dimensions to the target configuration. At evaluation, the agent gets a reward of 1 whenever it achieves a subtask.
Procgen Maze [16]. We collect an offline dataset of goal-reaching behavior on the Procgen Maze suite. For each maze level, pre-compute the optimal goal-reaching policy using oracle, and collect a trajectory of 1000 transitions by commanding a goal, using the geal-neg policy to reach this goal, then commanding new goal and repeating henceforth. The 'procgen- maze-500' dataset consists of 500000 transitions collected over the first 500 levels and 'procgen-maze 1000' consists of 1000000 transitions over the first 1000 levels. At test time, we evaluate the agent on "challenging" levels that contain at least 20 leaf goal states (i.e., states that have only one adjacent state in the maze). We use 50 such levels and goals for each evaluation, where they are randomly sampled either between Level 0 and Level 499 for the "-train" tasks or between Level 5000 and Level 5499 for the "-test" tasks. The agent gets a reward of when reaches the goal.
Visual AntMaze. We convert the original state-based AntMaze environment into a pixel-based environment by providing both a 64 x 64 x 3-dimensional camera image (as shown in the bottom row of Figure 6b) and 27-dimensional proprioceptive states without global coordinates. For the datasets, we use the converted versions of the 'antmaze-largee laraaaz----areeddivee---2222 and 'antmaze-larre-playy from the D4RL benchmark [28] well as newly collected dataset, 'antmaze-large-naviiee navigate-v2, which consists of diverse navigation behaviors that visit multiple goal locations within an episode. The task and the evaluation scheme are the same as the original state-based AntMaze environment.
datasets
Roboverse [25, 104]. We use the same dataset and tasks used in Zheng et al. [104]. The dataset consists of 3750 length-300 trajectories,' out of which we use the first 3334 trajectories for training
'Wiie Zheng et al. [104] separate each length- 300 trajectory into four length-75 trajectories, we found that using the original length-300 trajectories improves performance in general.
21

---

(which correspond to approximately 1000000 transitions), while the remaining trajectories are used as validation set. Each trajectory in the dataset features four random primitive behaviors, such as pushing an object or opening a drawer, starting from randomized initial object poses. At test time, we employ the same five goal-reaching tasks used in Zheng et al. [104]. We provide a precomputed goal image, and the agent gets a reward of 1 upon successfully completing the task by achieving the desired object poses.
In Tables 1 to 3, we report the normalized scores with a multiplier of 100 (AntMaze, Procgen Maze, Visual AntMaze, and Roboverse) or 25 (Kitchen and CALVIN).
D.2 Hyperparameters
We present the hyperparameters used in our experiments in Table 4, where we mostly follow the network architectures and hyperparameters used by Ghosh et al. [34]. We use layer normalization [5] for all MI. layers. For pixel-based environments, we use the Impala CNN architecture [21] to handle image inputs, mostly with 512-dimensional output features, but we use normalized 10 10-dimensional output features for the goal encoder of HIQL's 's value function to make them easily predictable by the high-level policy, as discussed in Appendix A. We do not share encoders between states and goals, or between different components. As a result, in pixel-based environments, we use a total of five separate CNN encoders (two for the value function, two for the high-level policy, and two for the low-level policy, but the goal encoder for the value function is the same as the goal encoder for the low-level policy (Figure Ia)). In Visual AntMaze and Roboverse, we apply a random crop [48] (with probability 0.5) to prevent overfitting, following Zheng et al. [104].
During training, we periodically evaluate the performance of the learned policy at every 100K (state- based) or 50K (pixel-based) steps, using 52 (AntMaze, Kitchen, CALVIN, and Visual AntMaze), 50 (Procgen Maze), or 110 (Roboverse, 22 per each task) rollouts At evaluation, we use arg max actions for environments with continuous action spaces and e-greedy actions with e = 0.05 for environments with discrete action spaces (i.e., Procgen Maze). Following Zheng et al. [104], in Roboverse, we add Gaussian noise with a standard deviation 0.15 to the arg max actions.
To ensure fair comparisons, we use the same architecture for both HIQL and four baselines (GCBC, HGCBC, GC-IQL, and GC-POR). The discount factor y is chosen from {0.99, 0.995), the AWR temperature 3 from 3, 10}. the IQL expectile T from {0.7, 0.9} for each method.
For HIQL., we set (7,8,7] (. (0.99, 0.7) across all environments. For GC-IQL and GC-POR, use (7,8,77 r) = (0.99, 3, 0.9) (AntMaze-Medium, AntMaze- Large, and Visual AntMaze), (7.3, T) (0.995, 0.7) (AntMaze-Ultee Ultra), or (7,3,7 r) (0.99, 1, 0.7) (others). For the subgoal steps i in HIQL, we use k = 50 (AntMaze-Ultra) k = 3 (Procgen Maze and Roboverse), or k = 25 (others). HGCBC uses the same subgoal steps as HIQL for each environment, with the exception of AntMaze- Ultra, where we find it performs slightly better with k 25. For HIQL, GC-IQL, and GC-POR, in state-based environments and Roboverse, we sample goals for high-level or flat policies from either the future states in the same trajectory (with probability 0.7) or the random states in the dataset (with probability 0.3). We sample high-level goals only from the future states in the other environments (Procgen Maze and Visual AntMaze).
E Proofs
E.1 Proof of Proposition 4.1
For simplicity, we assume that T/k is an integer and k S T.
"These numbers include two additional rollouts for video logging (except for Procgen Maze).
22

---

Table 4: Hyperparameters.
Hyperparameter
Value
gradient steps
1000000 (AntMaze), 500000 (others)
Batch size
1024 (state-based), 256 (pixel-based)
Policy MLP dimensions
(256, 256)
Value MLP dimensions
(512, 512, 512)
Representation MLP dimensions "state-basede
(512, 512, 512)
Representation architecture (pixel-based)
Impala CNN [21]
Nonlinearity
GELU [37]
Optimizer
Adam [47]
Learning rate
0.0003
Target network smoothing coefficient
0.005
Proof. Defining 21 := 21,T and 22 := 2-1,7, the probability of the flat policy и selecting an incorrect
action can be computed as follows:
{TT) P[V((s + 1, S V(s 1,9)
(12)
(I) s <V(-1,T)
(13)
0z1) S -(T+ = =Pizoo(T - -) (T +1) -2]
(14)
(15)
PIzzo +15 2
(16)
V2
Φ
(17)
where is a standard Gaussian random variable, and we use the fact that the of two independent Gaussian random variables with standard deviations of 01 and 02 follows a normal distribution with a standard deviation of / / + oz.
Similarly, the probability of the hierarchical policy mf n² selecting an incorrect action is bounded using a union bound as
E(T ~T*)= - E(T²) + E(x²)
(18)
= =PIL(s + k. (k - V(s
1,8+
(19)
= (kV T) $ Vi-k, +) $ v(-1,a)
(20)
① (- (-o0/ (T/K)/ +1
V2
V2
+
(21)
0V +1
E.2 Proof of Proposition 5.1
We first formally define some notations. For E S,a A,g S, and representation function S + Z, we denote the goal-conditioned state-value function as V(s.g) g). the action-value function as Q(s, a, 9), the parameterized state-value function Vo(s, 2) with z d(g). and the parameterized action-value function as Qs a, 2). We assume that the environment dynamics are deterministic, and denote the deterministic transition kernel as p(s, a) s'. Accordingly, we have Q(s, (. = Vpp(s, a), 9) = V(s', g) and Qo(s, 2) = Vo (p(s. a), 2) = Vols', 2). We denote the optimal value functions with the superscript "*" egg., V* g). We assume that there exists parameterized value function, which we denote (s, Φ(g)), that is the same as the true optimal
value function, Le., (* (J, (())) for all S and g E S.
23

---

Proof. For x², we have
"t" (a s.9) = arg max Q" aEA arg max (*('' s'EN,
(s, (,,0,))
(22)
(23)
arg max (b). s'EN, where N₂ denotes the neighborhood sets of 8, N, {8' p(s, s'). For T.. have
(24)
(a 22 = arg max 2 (s, 2) aEA
(25)
arg max V (s' z). S'N
(26)
By comparing Equation (24) and Equation (26), we can see that they have the same arg max action sets for all and g.
Subgoal Visualizations
We visualize learned subgoals in Figures 19 and 20 (videos are available at https: : //t:://seeopn me/ projects/hiql// For AntMaze-Large, we train HIQL without representations and plot the x-y coordinates of subgoals. For Proegen Maze, train HIQL with 10-dimensional representations and find the maze positions that have the closest representations (with respect to the Euclidean distance) to the subgoals produced by the high-level policy. The results show that HIQL learns appropriate k-step subgoals that lead to the target goal.
24

---

SHEEEEE SASEER CALTSTS CALCEAO BESEER SARCECC PALG
Figure 19: Subgoal visualization in AntMaze-Large. The red circles denote the target goal and the
blue circles denote the learned subgoals. Videos are available at https: : / /tt/////oongo/ee me/ j / hiql/.
25

---

Figure 20: Subgoal visualization in Procgen Maze. The red circles denote the target goal, the blue circles denote the learned subgoals, and the white blobs denote the agent. Videos are available at wt//m///ffj. htt//poeecsshhq/////
26


Reviews:
[{"review_id": "1xnfCLxhd7", "reviewer": "Reviewer_tvKa", "summary": "The paper proposes HIQL a hierarchical algorithm for offline goal-conditioned RL. The approach consists in utilizing a single action-free value function to acquire knowledge about the structure and employ two policies: a high-level policy that predicts or represents a waypoint, and a low-level policy that predicts the action required to reach that waypoint. The approach is well explained and motivated. The experiments comparing most state of the art approaches of offline-RL on the main benchmarks seem convincing.", "questions": "Why not using the TT in the kitchen scenario for comparison ?", "limitations": "Nothing noticeable IMO.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Reasonably novel approach, well explained and illustrated with convincing results on main offline RL benchmark of locomotion and manipulation against reasonable baselines.", "weaknesses": "Maybe performing TT on kitchen as baseline would have been useful to improve the comparison part.\nOtherwhile, no major weakness in my opinion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and constructive feedback about this work.\n\n* **Why not use TT in Kitchen for comparison?**\n\nAs mentioned in L303, we directly took the performances of the Trajectory Transformer (TT) and Trajectory Autoencoding Planner (TAP) from Jiang et al. [1], where these methods were not evaluated on the Kitchen benchmark. As such, we only made a comparison with these methods on AntMaze-{Medium, Large, Ultra}, which Jiang et al. [1] also used as a benchmark. We chose to use the numbers from prior work directly, since setting up and tuning these Transformer-based methods often requires a substantial amount of time and extensive computing resources.\n\nHowever, to address this comment, we additionally compare HIQL with GCPC [2], a recently proposed offline goal-conditioned RL method that also uses a Transformer to model trajectories similarly to TT. GCPC shows a better performance than TT in AntMaze-Large and reports its performance on goal-conditioned Kitchen (but not on AntMaze-Ultra). We present the results below, where we took the GCPC results from the original paper [2]:\n\n\n| Task                   | GCBC            | GC-IQL          | TAP    | TT               | GCPC [2]        | HIQL (ours)              |\n|------------------------|-----------------|-----------------|--------|------------------|-----------------|--------------------------|\n| antmaze-medium-diverse | $67.3 \\pm 10.1$ | $63.5 \\pm 14.6$ | $85.0$ | $\\mathbf{100.0}$ | $70.8$          | $86.8 \\pm 4.6$           |\n| antmaze-medium-play    | $71.9 \\pm 16.2$ | $70.9 \\pm 11.2$ | $78.0$ | $\\mathbf{93.3}$  | $70.4$          | $84.1 \\pm 10.8$          |\n| antmaze-large-diverse  | $20.2 \\pm 9.1$  | $50.7 \\pm 18.8$ | $82.0$ | $60.0$           | $77.2$          | $\\mathbf{88.2} \\pm 5.3$  |\n| antmaze-large-play     | $23.1 \\pm 15.6$ | $56.5 \\pm 14.4$ | $74.0$ | $66.7$           | $79.2$          | $\\mathbf{86.1} \\pm 7.5$  |\n| antmaze-ultra-diverse  | $14.4 \\pm 9.7$  | $21.6 \\pm 15.2$ | $26.0$ | $33.3$           | -               | $\\mathbf{52.9} \\pm 17.4$ |\n| antmaze-ultra-play     | $20.7 \\pm 9.7$  | $29.8 \\pm 12.4$ | $22.0$ | $20.0$           | -               | $\\mathbf{39.2} \\pm 14.8$ |\n| kitchen-partial        | $38.5 \\pm 11.8$ | $39.2 \\pm 13.5$ | -      | -                | $\\mathbf{65.0}$ | $\\mathbf{65.0} \\pm 9.2$  |\n| kitchen-mixed          | $46.7 \\pm 20.1$ | $51.3 \\pm 12.8$ | -      | -                | $61.0$          | $\\mathbf{67.7} \\pm 6.8$  |\n\nThe table above shows that, despite its simplicity, HIQL achieves the best (including ties) performance in both antmaze-large-{diverse, play} and kitchen-{partial, mixed}, generally outperforming the more computationally intensive Transformer-based methods. Putting it all together, we hope that the omission of TT/TAP results on Kitchen (which was not evaluated with these methods in prior work) is reasonable.\n\nPlease let us know if there are any additional concerns or questions.\n\n**References**: Please see our global response."}, {"review_id": "l6EgIXiwBG", "reviewer": "Reviewer_xrL8", "summary": "This paper introduces hierarchical IQL (HIQL) for offline goal-conditioned reinforcement learning. HIQL uses the IQL algorithm to learn both a high-level waypoint policy as well as a low-level action policy; in both case, the goal-conditioned value is provided by the same value function learned using goal-conditioned IQL procedure. HIQL is evaluated on both state-based as well as image-based environments that require hierarchical planning. In all settings, HIQL outperforms prior methods", "questions": "I have listed several suggestions in the section above. Overall, I think this is a solid paper, and I recommend acceptance.", "limitations": "N/A. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "Originality:\nThis paper builds on IQL, which has already been extended to goal-conditioned settings, though not in the hierarchical manner this paper proposes. Furthermore, the use of single value function is a nice idea. The didactic examples and theoretical results are also interesting and contribute to the overall story. Therefore, I believe this paper contains enough original components for NeurIPS.\n\nQuality and Clarity:\nThis paper is very well written and presented. The experiments are rigorously executed and explained.\n\nSignificance:\nOffline GCRL is an emerging and important topic to study, and this paper proposes a conceptually simple and effective algorithm for the setting.", "weaknesses": "This is a strong paper without major weaknesses. There are several places to further improve the paper:\n\n1. An ablation that demonstrates that AWR is needed for both policies. I suspect that it could be possible that it is not super important for the low-level controller if the horizon is short enough. \n\n2. Connection with existing approaches that learns a high-level planner and uses an inverse dynamics model to back out low-level actions can be better drawn in related work.\n\n3. I believe more substantive error bound in the form of policy performance can be derived in addition to Proposition 4.1; [1] makes one such attempt.\n\n[1] Ajay, Anurag, et al. \"Opal: Offline primitive discovery for accelerating offline reinforcement learning.\" arXiv preprint arXiv:2010.13611 (2020).\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and constructive feedback about this work. As suggested by the reviewer, we ran an additional ablation experiment to study the use of AWR for both policies.\n\n* **An ablation that demonstrates that AWR is needed for both policies**\n\nWe ablated both low-level AWR and high-level AWR by replacing either of them with BC, and evaluated them on both AntMaze, which has a relatively large waypoint step ($k=25$ for ‘-large’ and $k=50$ for ‘-ultra’), and Procgen Maze, which has a relatively small waypoint step ($k=3$). We report the performance below (the results are averaged over 4 seeds and $\\pm$ denotes standard deviations).\n\n| Task                   | HIQL (ours) (High AWR, Low AWR) | Ablation (High AWR, Low BC) | Ablation (High BC, Low AWR) |\n|------------------------|--------------------------|-----------------------------|-----------------------------|\n| antmaze-large-diverse  | $88.2 \\pm 5.3$           | $60.1 \\pm 13.3$             | $\\mathbf{91.3} \\pm 5.1$     |\n| antmaze-ultra-diverse  | $\\mathbf{52.9} \\pm 17.4$ | $23.6 \\pm 9.2$              | $47.6 \\pm 13.7$             |\n| procgen-maze-500-train | $\\mathbf{82.5} \\pm 6.0$  | $74.0 \\pm 12.6$             | $17.5 \\pm 5.7$              |\n| procgen-maze-500-test  | $\\mathbf{64.5} \\pm 13.2$ | $52.0 \\pm 11.7$             | $19.5 \\pm 9.6$              |\n\nThe table above shows that both high-level AWR and low-level AWR are important for performance. However, their relative importance may depend on the dataset. In AntMaze, low-level AWR is more important than high-level AWR (due to the data collection strategy; the AntMaze datasets consist of single-goal-reaching trajectories with noisy actions, in which case BC is a reasonable objective for the high-level policy but not for the low-level policy). On the contrary, in Procgen Maze, high-level AWR is more important than low-level AWR (due to both the diversity of the dataset and the relatively small waypoint steps).\n\n* **Connection with existing approaches that learn a high-level planner and use an inverse dynamics model to back out low-level actions**\n\nThank you for the suggestion. We will discuss these approaches [18, 19, 20, 21, 22] in the related work section of the final version.\n\n* **An additional error bound based on suboptimality**\n\nThank you for the suggestion. In an earlier version of the draft, we made initial attempts at deriving a bound similar to that of Ajay et al., but found removing the assumption of two value functions (we only use one) to be difficult. If we are able to successfully derive a hierarchical performance bound based on a single value function, we will include it in the final paper.\n\nPlease let us know if there are any additional concerns or questions.\n\n**References**: Please see our global response.\n"}, {"review_id": "JsaCSIJsxD", "reviewer": "Reviewer_cj3e", "summary": "The paper identifies an issue with offline goal-conditioned reinforcement learning: namely that goal-conditioned value estimation can be noisy, so long-horizon tasks  can be difficult to accomplish due to accumulating errors in value estimation. In addition, when states are so close together, there is very little signal for learning as any mistakes can be corrected in future states. Thus, HIQL is proposed, which learns a high-level goal-conditioned policy to predict subgoals (such that there is sufficient learning signal), and a low-level goal conditioned policy to predict actions, which benefits from only considering nearby goals. Experiments show the advantages of this approach with respect to baseline methods in hierarchical and flat offline RL, as well as the ability to use unlabeled (without action) trajectory data to improve learning, and data efficiency.", "questions": "Besides the points in the weakness section, it's a bit unclear to me the novelty of this submission, so I would appreciate some further contextualization to other hierarchical methods that take advantage of data.", "limitations": "Limitations are addressed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The argument is extremely clear and well exposed\n- I appreciate the demonstration of issues with current methods (Fig. 2, 3, 4) before introducing the solution, which helps my understanding of the field in general. Such a simple demonstration, though perhaps not exactly well validated, may even be worth more as expository writing than the final proposed method.\n- The method is simple, in particular reuse of internal representations instead of learning some VAE or something more complex is nice.\n- The ability to have different data requirements for different parts of the policy is quite nice, given that action labels are harder to come by (Table 3).", "weaknesses": "- Though the exposition is very clear, sometimes it is a bit repetitive. In particular, the same justification for hierarchy is pointed out in line 36 and 160, as well as the repetition between sections 4.2 and section 5, though I can see how this is a matter of personal taste.\n- The difference in domain between Figure 2 and Figure 3 is a little bit odd, though this isn't a huge issue. The domain in Figure 3 is maybe a bit _too_ toy, which leads me to discount Proposition 4.1, the major point was already in Figure 2.\n- I'm a bit suspect of the claim that Procgen Maze is a good demonstration for a visual environment, given that the visual observation is already so structured. It would be nice to work on a more complex task within Procgen, like Coinrun, though I'm unaware of the data availability for these tasks.\n- Proposition 5.1 is unnecessary in my opinion, that or just a simple proof sketch could be given inline.\n- I think a more in-depth discussion of particularly related work is merited. Novelty is not very clear to me from the given presentation. In particular the proposed method is so simple that it's surprising to me that it does not already exist.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and constructive feedback about this work.\n\n* **“Novelty is not very clear to me from the given presentation.”**\n\nOne major difference between our approach and prior hierarchical methods is that we extract both policies (and even representations) from a **single** (non-hierarchical) value function. This eliminates the need for additional components, unlike previous hierarchical methods that train separate value functions [4, 5] or use potentially complex high-level subgoal planning procedures [6, 7, 8, 9]. Despite its simplicity, perhaps surprisingly, we show that this simple technique can significantly improve performance both in theory and in practice, due to an improved “signal-to-noise” ratio (Section 4). We have clarified this point in both the Introduction and Related Work sections in a revised version of the draft.\n\n* **Additional pixel-based environments**\n\nTo verify the effectiveness of HIQL in more diverse visual environments beyond Procgen Maze, we evaluated HIQL and prior methods on **two** additional pixel-based benchmarks: Roboverse and Visual AntMaze. **Roboverse** [3] is a pixel-based, goal-conditioned robotic manipulation task that requires multi-stage reasoning and generalization, where the agent must learn to control a robot arm to manipulate objects purely from pixels. We use the same dataset and tasks used by Zheng et al. [3]. **Visual AntMaze** is a vision-based variant of the AntMaze environment, where we provide the agent with a $64 \\times 64 \\times 3$ camera image and its proprioceptive states, excluding the global coordinates. Hence, the agent must learn to navigate the maze based on the wall structure and floor color from the image. Please find illustrations of these environments in **Figure 1** of the supplementary PDF. For the datasets, we render the ‘antmaze-large-diverse-v2’ and ‘antmaze-large-play-v2’ datasets from the D4RL benchmark. We additionally employ a more challenging dataset, ‘antmaze-large-navigate-v2’, which consists of diverse navigation behaviors that visit multiple goal locations within an episode.\n\nIn these two additional pixel-based environments, we compare the performances of HIQL (ours), goal-conditioned IQL (“GC-IQL”), goal-conditioned POR (“GC-POR”), HGCBC, and GCBC (with individually tuned hyperparameters). We report the results (averaged over 8 seeds, $\\pm$ denotes standard deviations) below:\n\n| Task | GCBC | HGCBC (+ repr.) | GC-IQL | GC-POR (+ repr.) | **HIQL (ours)** |\n|---|---:|---:|---:|---:|---:|\n| visual-antmaze-diverse | $71.4 \\pm 6.0$ | $35.1 \\pm 12.0$ | $72.6 \\pm 5.9$ | $47.4 \\pm 17.6$ | $\\mathbf{80.5} \\pm 9.4$ |\n| visual-antmaze-play | $64.4 \\pm 6.3$ | $23.8 \\pm 8.5$ | $70.4 \\pm 26.6$ | $57.0 \\pm 8.1$ | $\\mathbf{78.4} \\pm 4.6$ |\n| visual-antmaze-navigate | $33.2 \\pm 7.9$ | $21.4 \\pm 4.6$ | $22.1 \\pm 14.1$ | $16.1 \\pm 15.2$ | $\\mathbf{45.7} \\pm 18.1$ |\n| roboverse | $26.2 \\pm 4.5$ | $26.4 \\pm 6.4$ | $31.2 \\pm 8.7$ | $46.6 \\pm 7.4$ | $\\mathbf{61.5} \\pm 5.3$ |\n\nThe table above shows that HIQL outperforms previous methods in these vision-based tasks as well. Notably, in Roboverse, HIQL is capable of generalizing to solve unseen robotic manipulation tasks purely from images, achieving an average success rate of 62%. We have included these results and experimental details in a revised version of the paper.\n\n* **Writing suggestions**\n\nThank you for the suggestions! We will incorporate them into the final version of the paper.\n\nPlease let us know if there are any additional concerns or questions.\n\n**References**: Please see our global response.\n"}, {"review_id": "rdgYtpR3zE", "reviewer": "Reviewer_ZY68", "summary": "This paper introduces a hierarchical approach to address the issue of offline goal-conditioned problems, specifically when certain trajectories in the dataset have missing actions. The proposed method tackles this challenge by simultaneously learning a value function with a modified version of IQL and extracting the high-level and low-level policies of the learned value function. The high-level policy predicts the representation of waypoints, while the low-level policy, conditioned on the waypoint, determines the primitive actions. Extensive experimental results on offline goal-conditioned tasks demonstrate the effectiveness of the proposed approach.\n", "questions": "1. (**Clarification Question**) I am confused by the notation $V(s, \\phi(g))$, $\\pi(a|s, \\phi(g))$. I understand that for the high-level policy, the waypoint it predicted is the latent representation, not the pixel values of the image. But is $s$ also represented by the same state representation/encoder $\\phi$? So is it $V(\\phi(s), \\phi(g))$ (or with some stop gradient on either $\\phi(s)$ or $\\phi(g)) $?\n2. (**How to learn the value function $V$**) In this paper, the author adapts the approach of the IQL algorithm to learn the value function V (with some modification to account for missing actions). But this is not the only approach to learn the value function. [1] proposes to learn the value-function V by dualizing the objective, which also does not require action labels in the dataset. Maybe it would be nice to compare this approach and see which approach gives better quality of the value function, since it is crucial for the extraction of high-quality policies.\n\n[1] Ma et al. VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The writing is very easy to follow.\n2. The paper proposes an elegant approach to goal-conditioned offline RL problem by decomposing the high-level and low-level policy while sharing the same value functions V.\n3. The approach taken in this paper allows the adoption of the existing large amount of action-free data.\n4. Good empirical performance on a comprehensive set of evaluation environments with both state and pixel observations. \n\n", "weaknesses": "For some of the design choices adopted in this paper, I think there are some alternatives in the literature that are good to compare with. See my comment in the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and constructive feedback about this work.\n\n* **How to encode $s$ in $V(s, \\phi(g))$?**\n\nFor pixel-based environments, we encode $s$ and $g$ separately into $\\psi(s)$ and $\\phi(g)$ using two *different* CNNs, $\\psi$ and $\\phi$. We then concatenate $\\psi(s)$ to $\\phi(g)$ and model $V(\\psi(s), \\phi(g))$ using an MLP. (For state-based environments, we directly concatenate $s$ to $\\phi(g)$, without having a separate encoder for $s$.) Since only the $\\phi(g)$ component is used outside as a goal representation, we simply denoted $V(\\psi(s), \\phi(g))$ as $V(s, \\phi(g))$ in the paper (treating $\\psi(s)$ as a “black box” within $V$). However, we recognize that this could have led to some confusion, and we will clarify this point in the final version of the paper.\n\n* **Alternatives to IQL for learning a goal-conditioned value function**\n\nAs the reviewer pointed out, our hierarchical policy extraction scheme is *orthogonal* to the choice of the underlying offline RL algorithm used to learn a goal-conditioned value function $V(s, g)$. In this work, we chose to use IQL for its effectiveness and simplicity, as it does not require any other additional components and is easy to use. However, it is indeed possible to combine HIQL with other value-based offline RL algorithms, such as VIP [10], Contrastive RL [11], or Quasimetric RL [12] (or even CQL [13] or GoFAR [14] if we have action labels), and we believe our hierarchical policy structure can still be beneficial in these cases, as our theoretical results do not depend on the underlying value learning algorithm. We will clarify this point in the final paper. We believe studying different underlying offline RL algorithms that can further enhance this idea is an interesting direction for future research.\n\nPlease let us know if there are any additional concerns or questions.\n\n**References**: Please see our global response.\n"}, {"review_id": "EKPP2l17aK", "reviewer": "Reviewer_kqhw", "summary": "The paper introduces HIQL, a hierarchical algorithm for offline goal-conditioned RL. HIQL utilizes an action-free version of IQL to learn the value function and subsequently derives both high-level and low-level policies from this shared value function using AWR. The paper asserts that the hierarchical structure offers increased robustness to value function noise, leading to enhanced performance in achieving long-horizon goals. The effectiveness of the proposed method is validated through experiments conducted in both state-based and pixel-based tasks.", "questions": "* Additional explanation is needed to clarify how the hierarchical structure can be applicable in continuous situations, as well as a comparison with smoothed Q methods\n* More information should be provided to enable the reproduction of Figure 2.\n* Additional baselines for temporal difference learning and goal relabeling should be included in order to facilitate a more comprehensive and fair comparison.\n* Authors are encouraged to report the performance of the low-level policy in order to emphasize the necessity of a hierarchical structure.\n* Is it possible to expand the HIQL framework to include more than two levels? Considering that the parameter k is manually determined, the top level could be assigned a large k value, while a medium level would utilize a moderate k value.\n* A fix in Figure 1 and Eq (7) may be needed.\n\n\n## post rebuttal\nI am delighted to see that the authors have addressed all of my concerns, and I am pleased to raise my score for this paper.", "limitations": "One limitation mentioned in the paper is the necessity for deterministic dynamics. It would be beneficial for the authors to further elaborate on why disentangling the controllable aspects from the uncontrollable elements in the environment is advantageous.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Overall, the paper is well written and well organized.\n* The proposed method HIQL incorporates IQL into a hierarchical framework, similar to prior offline RL work POR. However, HIQL incorporates additional designs from hierarchical RL, including predicting latent waypoints and k-step waypoints. Consequently, it is a new approach from both directions, offering valuable insights for tackling offline GCRL with hierarchical RL.\n* The examples in Section 4.1 and Section 4.3 are helpful for readers to understand.\n* The experiments are thorough, showing improved performance on both state-based and image-based tasks.\n", "weaknesses": "* The given examples show that the hierarchical structure is more robust to value noises in discrete tasks. Generally, in continuous tasks, the value function tends to be smoother. This raises the question of whether the performance improvement observed in HIQL is solely attributed to the hierarchical structure. If the claim in the paper is right, a method using smoothed value function (e.g., [1]) without hierarchical structure may achieve comparable performance with HIQL. Additional explanation and comparison are encouraged.\n* More information to reproduce the Figure 2 need to be provided.  \n* In the appendix, the authors concat [s,g] before feeding into $\\phi$. Therefore, it is recommended that Figure 1 be revised to address this matter.\n* In the experiments, all the baselines considered are related to (weighted) imitation learning, making the inclusion of additional baselines utilizing temporal difference learning [2][3] necessary for a more comprehensive evaluation. In addition, it is notable that HIQL leverages the advantage of goal relabeling, while baselines such as IQL do not utilize this technique. Despite this, IQL has already demonstrated strong performance on the pixel-based Procgen Maze benchmark. Therefore, it remains unknown if HIQL can achieve comparable performance when compared to IQL+HER.\n* For the experiments, I encourage the authors to report the performance of the low-level policy without the high-level policy. This can effectively highlight the true usefulness of the hierarchical structure.\n* There is an error in Eq (7) in which the variable g is mistakenly used to train the low-level policy, while $s_{t+1}$ in the expectation is not used.\n\n[1] Hong Z W, Yang G, Agrawal P. Bilinear value networks[J]. arXiv preprint arXiv:2204.13695, 2022.\n\n[2] Chebotar Y, Hausman K, Lu Y, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills[J]. arXiv preprint arXiv:2104.07749, 2021. \n\n[3] Li J, Tang C, Tomizuka M, et al. Hierarchical planning through goal-conditioned offline reinforcement learning[J]. IEEE Robotics and Automation Letters, 2022, 7(4): 10216-10223.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and suggestions for improving the work. Below, we describe how we have revised the paper to clarify parts of the paper to address the questions raised by the reviewer. We also clarify how we have already compared to baselines that use HER, and present new results from three additional baselines. We believe that these changes strengthen the paper, and welcome additional suggestions for further improving the work.\n\n- **“It remains unknown if HIQL can achieve comparable performance when compared to IQL+HER”**\n\nWe would like to clarify that “IQL” in this paper always refers to “**IQL+HER**” (i.e., goal-conditioned IQL) (L297) (and likewise “POR” in this paper always denotes “POR + HER”), and they use the exactly **same** goal relabeling strategy as HIQL. This relabeling strategy (described in Appendix A) is directly taken from a prior work without modification [15]; thus it is not tuned for our method, which ensures a fair comparison. To further clarify that the baselines are also goal conditioned and employ hindsight relabeling, we will refer to them as “GC-IQL” and “GC-POR” in the final version of the paper.\n\n* **Additional baselines that do not use (weighted) imitation learning**\n\nThank you for the suggestion. We have now compared HIQL with **three** additional baselines that do not use (weighted) imitation learning.\n\nAs we were unable to find publicly available official implementations of Actionable Models (AM) [17] or HiGOC [9], we tried to evaluate AM using both our own re-implementation and another re-implementation by Ma et al. [14]. However, despite our extensive efforts to tune its hyperparameters on two different codebases, we were unable to achieve performance above 1% even on the simplest AntMaze-Medium tasks. Hence, we instead evaluated the performance of **goal-conditioned CQL** (“GC-CQL”) [13], another temporal difference learning-based method that uses a similar [17] conservative objective to AM. For a fair comparison, we used the same hindsight relabeling strategy for GC-CQL and tuned its hyperparameters individually for each D4RL task.\n\nMoreover, we made additional comparisons with two recent offline goal-conditioned RL methods (Contrastive RL [11] and GCPC [2]) that report their performances on the goal-conditioned variants of D4RL tasks (AntMaze and/or Kitchen). **Contrastive RL** is a goal-conditioned value learning method based on contrastive learning, and **GCPC** is an offline goal-conditioned RL method that models trajectories using a BERT-style Transformer with masking techniques. For these methods, we directly used the reported results from the respective papers [2, 11].\n\nWe present the full comparison results (averaged over 4 seeds) on D4RL environments in **Table 1** in the supplementary PDF. The results show that HIQL mostly achieves the best performance in these tasks, outperforming (or at least being as good as) the new baselines.\n\n* **Flat IQL with a smoothed value function**\n\nAs the reviewer pointed out, a regularized (smoothed) value function may also alleviate the signal-to-noise challenge in a continuous state space. To empirically verify this hypothesis, we evaluated the performance of flat goal-conditioned IQL (GC-IQL) with a bilinear value function $V(s, g) = f(s)^\\top \\psi(g)$ with 512-dimensional $f$ and $\\psi$.\n\nWe present the results across four different tasks in **Table 2** in the supplementary PDF. Despite the improved smoothness of the value function, Bilinear GC-IQL shows worse performance than the original GC-IQL. We believe this is due to the limited expressivity of bilinear value functions compared to the original monolithic value functions. We will include a discussion about continuous state spaces in the final version of the paper.\n\n* **Performance of the low-level policy without a high-level policy**\n\nAs per the reviewer’s suggestion, we evaluated the performance of HIQL’s low-level policy (without a high-level policy) on D4RL tasks. **Table 3** in the supplementary PDF presents the results across four different tasks. The results show that HIQL without a high-level policy almost completely fails to solve these tasks. This is because the low-level policy is trained to reach only nearby goals (Eq. (7)). If we train the low-level policy with the full set of goals, it becomes equivalent to GC-IQL, whose performance still falls behind that of HIQL. This highlights the necessity of our hierarchical structure.\n\n* **Is it possible to expand the HIQL framework to include more than two levels?**\n\nThank you for raising this point. As the reviewer mentioned, it is indeed possible to have a recursive structure, in which higher-level policies produce subgoals for the policies at the next level down, and only the lowest-level policy produces actions. At an earlier stage of this research, we tested 3- or 4-level hierarchies on AntMaze-Large with waypoint steps of (25, 5) or (100, 25, 5). However, we found that they do not significantly improve (or sometimes hurt) performance compared to the two-level policy structure. This is likely because (1) a two-level policy is sufficient for AntMaze-Large given its episode length, and (2) policy errors can accumulate as the hierarchy grows. However, we believe such a recursive structure has the potential to solve much longer-horizon problems (where an improved signal-to-noise ratio outweighs accumulated policy errors), and further extending HIQL in this way with a highly scalable architecture is an exciting future research direction.\n\n* **Code to reproduce Figure 2**\n\nWe have uploaded the ipynb file that we used to produce Figure 2 to our anonymized repository.\n\n* **Improving Figure 1, Typo in Equation (7)**\n\nThank you for the suggestions! We have addressed the issue with Figure 1 and fixed the typo in Equation (7) in a revised version of the paper.\n\nPlease let us know if there are any additional concerns or questions.\n\n**References**: Please see our global response."}]

Author rebuttal:
We appreciate all five reviewers’ constructive feedback and suggestions for improving the work. We would like to highlight the updates we made in our responses below.
- We evaluated HIQL and baselines on $\mathbf{2}$ **additional pixel-based environments**, Roboverse and Visual AntMaze, which demonstrate the effectiveness of HIQL in more diverse visual domains (Reviewer cj3e).
- We compared HIQL with $\mathbf{3}$ **additional baselines** — goal-conditioned CQL, contrastive RL, and GCPC — on D4RL tasks (Reviewers tvKa and kqhw).
- We ablated various aspects of HIQL, which shows the necessity of both high-level AWR and low-level AWR and the necessity of having a high-level policy (Reviewers xrL8 and kqhw).

Below are the references we use in our responses:

[1] Jiang et al., Efficient Planning in a Compact Latent Action Space. ICLR 2023.

[2] Zeng et al., Goal-Conditioned Predictive Coding as an Implicit Planner for Offline Reinforcement Learning. arXiv 2023.

[3] Zheng et al., Stabilizing Contrastive RL: Techniques for Offline Goal Reaching. arXiv 2023.

[4] Nachum et al., Data-Efficient Hierarchical Reinforcement Learning. NeurIPS 2018.

[5] Levy et al., Learning Multi-Level Hierarchies with Hindsight. ICLR 2019.

[6] Shah et al., Rapid Exploration for Open-World Navigation with Latent Goal Models. CoRL 2021.

[7] Fang et al., Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space. IROS 2022.

[8] Fang et al., Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks. CoRL 2022.

[9] Li et al., Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning. RA-L 2022.

[10] Ma et al., VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training. ICLR 2023.

[11] Eysenbach et al., Contrastive Learning as Goal-Conditioned Reinforcement Learning. NeurIPS 2022.

[12] Wang et al., Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning. ICML 2023.

[13] Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning. NeurIPS 2020.

[14] Ma et al., How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression. NeurIPS 2022.

[15] Ghosh et al., Reinforcement Learning from Passive Data via Latent Intentions. ICML 2023.

[16] Hong et al., Bilinear value networks. ICLR 2022.

[17] Chebotar et al., Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills. ICML 2021.

[18] Torabi et al., Behavioral Cloning from Observation. IJCAI 2018.

[19] Schmeckpeper et al., Reinforcement Learning with Videos: Combining Offline Observations with Interaction. CoRL 2020.

[20] Baker et al., Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos. NeurIPS 2022.

[21] Chang et al., Learning Value Functions from Undirected State-only Experience. ICLR 2022.

[22] Zheng et al., Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories. ICML 2023.

Meta-review: The paper proposes goal-conditioned off-line RL as general (pre-) training strategy and introduces a hiearchical variant. The reviewers appreciated novelty, theoretical results, descriptions of the shortcomings of the SOTA, clarity and presentation, execution and experiments.

Only minor weaknesses had been identified, mostly linked to some additional ablations and additional connections to related work. Overally, a strong and well-rounded paper has been identified.

The AC concurs and recommends acceptance.
Decision: Accept (spotlight)