DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Authors: Seungjae Lee
Jigang Kim
Inkyu Jang
H. Jin Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Hierarchical RL
Robotics

Abstract: Hierarchical Reinforcement Learning (HRL) has made notable progress in complex control tasks by leveraging temporal abstraction. However, previous HRL algorithms often suffer from serious data inefficiency as environments get large. The extended components, $i.e.$, goal space and length of episodes, impose a burden on either one or both high-level and low-level policies since both levels share the total horizon of the episode. In this paper, we present a method of Decoupling Horizons Using a Graph in Hierarchical Reinforcement Learning (DHRL) which can alleviate this problem by decoupling the horizons of high-level and low-level policies and bridging the gap between the length of both horizons using a graph. DHRL provides a freely stretchable high-level action interval, which facilitates longer temporal abstraction and faster training in complex tasks. Our method outperforms state-of-the-art HRL algorithms in typical HRL environments. Moreover, DHRL achieves long and complex locomotion and manipulation tasks.

DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Seungjae Leel. Jigang Kim 1.2, Inkyu Jang!:3 1.3 H. Jin Kim 1.3
Seoul National University
Artificial Intelligence Institute of Seoul National University (AIIS) 3 "Automation and Systems Research Institute (ASRI)
fysz0301, jgkim2020, leplusbon, hjinkim} @snu. ac kr
Abstract
Hierarchical Reinforcement Learning (HRL. has made notable progress in complex control tasks by leveraging temporal abstraction. However, previous HRL algo- rithms often suffer from serious data inefficiency as environments get large. The extended components, i.c., goal space and length of episodes, impose burden on either one or both high-level and low-level policies since both levels share the total horizon of the episode. In this paper, we present method of Decoupling Horizons Using a Graph Hierarchical Reinforcement Learning (DHRL) which can allevi- ate this problem by decoupling the horizons of high-level and low-level policies and bridging the gap between the length of both horizons using a graph. DHRL provides a freely stretchable high-level action interval, which facilitates longer temporal abstraction and faster training in complex tasks. Our method outperforms sate-offtthe-rr HRL algorithms in typical HRL environments. Moreover, DHRL achieves long and complex locomotion and manipulation tasks.
Introduction
Reinforcement Learning (RL) has been successfully applied to range of robot systems, such as locomo-
tion tasks 24 [8, learning to control aerial robots
I0 13, and robot manipulation
[44 22 Goal-
conditioned RL, which augments state with the goal
to train an agent for various goals [23] further raised the applicability of RI. in robot systems allow- ing the agent to achieve diverse tasks.
AntMaxeCompiox
High Horizon Lowelevel Horizon
Hierarchical Reinforcement Learning (HRL), which trains multiple levels of goal-conditioned RL, has im- proved the performance of RL in complex and sparse tasks with long horizons using temporally extended policy [25] 26 16). On the back of these strengths, HRL was adopted solve various complex robotics 18).
Figure 1: DHRL: By decoupling the hori- zons of both levels of the hierarchical net- work, DHRL not only solves long and sparse tasks but also significantly outperforms previ- ous state-of-the tfe-the-rt art algorithms.
tasks 20
However, HRI often has diffieulty in complex large environments because of training inefficiency. Previous studies speculated that the cause of this problem is the large goal space, and restricted the high-level action space alleviate this phenomenon [28 12]. Nevertheless, this approach performs well only in limited length and complexity and still suffers from the same trouble in larger environments.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We show that this practical limitation of HRL can be mitigated by breaking down the coupled horizons of HRI.. In previous HRI. frameworks, the horizons of the low level and high level are related to each other structurally because they
(Previous HRL (HRAC)) High-Level Interval
[Ours (DHRL))
High-Level Interval 10
30
50
30
12X12
0.821
0.028
0.894
0.951
0.941
share the total length of the episode. This rela- tion causes tradeoff between the training bur- den of both levels; if the intervals between high-
Map Size
20/24
Map 24X24
0.687
12X12
56X56
level actions increase (x-axis in Figure 2), the
56X38
0.951
0.911
0.401
low-level policy has to cover a wider range, and in the opposite case, the high-level policy takes charge of the extended burden alone in large environments (y-axis in Figure 2). This is the reason why the previous HRI. algorithms cannot cope with extended components of large envi- ronments (see Table [ for the performance of the previous HRL method at various intervals).
0,000
High
Low
3.0
0.8
0.6 0.4
0.2
0,0
Success Rate
Figure 2: Our method is scalable in large environ- ments by breaking down the relations between the two levels and allowing both levels to operate at their suitable horizons.
We break down the coupled horizons of HRL:
To break the relation between the horizons of both levels, adopt graph structure. In our method, the high-level policy can use a longer temporal abstraction while the lower one only takes charge of smaller coverage by decomposing the subgoal into several waypoints with graph. In this way, the HRL algorithm obtains the capability to stretch the interval of high-level action freely and achieves complex and large tasks, thanks to the enlarged strength of the HRL.
In summary, our main contributions are:
We show that the previous HRL structures are not scalable in large environments, and that this limitation can be mitigated by removing coupled traits of high level and low level.
To break down the coupled traits of HRL, we propose DHRL which decouples the horizons of high-level and low-level policies and bridges the gap using a graph.
Our algorithm outperforms state-of ofe---ftthhhee the-arr algorithms in typical HRL environments and achieves complex and long tasks.
2 Preliminaries
We consider finite-horizon Universal Markov Decision Process (UMDP) which can be represented as a tuple G.A, T, R. 7) where S, A and G are state space, goal space and action space respectively. The environment is defined by the transition distribution (( az) and reward function R. S x A x G + IR, where E S and a, E A are the state and action at timestep t respectively. Also, total return of trajectory r(8+11.9) (or r(8t, at, g)) is goal conditioned reward function and y is discount factor. Subgoal sg, waypoint wp and goal g are defined in goal space G and we consider G that is subspace of S with mapping v) S G.
(s0, 4o. ...., 81, ",,)HH is R(T,g) = > 7'r($111,9) where
HRL framework typically has high-level policy hi and low-level policy n each maintaining a separated replay buffer Bhi and Blo, Our method also follows the general HRL framework and employs both buffers that store high-level (st. 9t. sgt, Tt, St+e,) ) E hh and low-level transition data (st, wpt, at, Ft, 8)+1) E Blo, where C₁ is the interval between high-level action.
One of the key problems in learning HRL is that the low-level policy T/o is non-stationary and thus old data from past policies may contain different next-states St+e,, even though the high-level policy provides the same subgoal in the identical state. To bypass this pollcy discrepancy, HRL models use off-policy correction which relabels the old subgoal of high-level policy to be the most "plausible subgoal' that will result in a similar transition in old data with the current low-level agent [1] 28 12 (HIRO-style off-policy correction). Other approaches propose to relabel high-level action sgt to be achieved state sttr, (HAC-style hindsight action relabelling), reducing the computation cost to find plausible subgoal' [15).

---

traph-guider RL methods, which combine the strength of RL and planning by decomposing long-horizon task into multi-step sub- problems,, estimate the temporal distance between states and goals to construct a graph G = (V, E) on goal space G without additional prior knowledge about environments. Previous studies proposed various methods to recover distance from Q-value [4 27 If the agent gets 11 reward at every step except when it is in goal area where the agent gets 0 reward, then, (s, a alg) can reveal the temporal distance between to g as: (Refer to the Appendix for the detailed derivation.)
Dist(s g) = log, (1
However, it is known that recovering temporal distance correctly from the vanilla Q-network in this setting is challenging. For that reason, the previous methods use an additional Value function approximator (27 or distributional Q-networks
3
Related work
Graph-guided RI. Graphs have recently been used as con-aaraettric model in rein- forcement learning (RI.) to combine the advan- tages of RL and planning [4 9 27 6]. By de- composing long-horizon task into multi-step planning problems, these studies have shown better performance and data efficiency. Search on the Replay Buffer (SORB) [4] constructs a directed graph based on the states randomly extracted from replay buffer and Q-function- based edge cost estimation. The follow-up stud- ies further improved the performance of earlier graph-guided RLs 4 9 by combining addi- tional methods such as graph search on latent space 27 or model predictive control
Ous (DERL)
10GL
GradeGGaiee Guided RL
Train
Conl (ss
Suhgrel loge)
Stortee
Covaaie
Legg
Poley
Graph Edge
State lod
Figure 3: The differences between DHRL and the previous graph- based HRL methods. Our algo- rithm includes the graph structure between both levels explicitly while the previous methods use the graph only for training high-level policy or get- ting waypoints.
However, previous papers sidestepped the exploration problems in complex tasks through 'uniform 6), or work only on the dense reward settings [3]. We emphasize
initial state distribution' that the 'uniform initial state distribution' accesses privileged information about the environment during training by generating the agent uniformly within the feasible area of the map. This greatly reduces the scope of application of these algorithms. Unlike prior methods, reward settings and a 'fixed initial state distribution' without knowledge of the agent's 's surroundings, which makes practical for physical settings. For detailed examples and comparison of various initial state distributions, see Table 2 and Figure 11 in Appendix C.
can train from sparse
Constrained-subgoal HRL. To mitigate the training inefficiency issue of HRL, several researchers proposed methods that restrict the action of high-level agent be placed adjacent areas. Hier- archical Reinforcement Learning with k-step Adjacency Constraint (HRAC) 28 limits the sub- goal to be the adjacency space of the current state. Hierarchical reinforcement learning Guided by Landmarks (HIGL) [12] improved the data efficiency by adding novelty-based landmarks to ediaccncy--constramee HRL. However, these improvements only work on the limited length and complexity of the environment.
The previous work closest to our method HIGL. However, there are three key differences between the previous work and our approach. First, our model explicitly includes the whole graph structure while HIGL needs to train high-level action imitate the graph by using an additional loss term corresponding to the Euclidean distance from the nodes of the graph. Second, we use graph to decouple the horizons of high level and low level, unlike the previous method which uses the graph only for guidance. Most importantly, ours can achieve goals in long and complex environments. To the best of our knowledge, there is no prior HRL research train model that has decoupled the horizons of the two levels.

---

GGraph Construction]
[Planning Over the Graph)
GOG.F)
high
G(V.E)
low
"" MPrc)
"0 MPpc)
Figure 4: An overview of DHRL which includes the mid-level non- an-paaametric policy between the high and low levels. The high-level (orange box) policy delivers subgoal sg E 9 to the graph level (green box) and the graph instructs the low-level policy (blue box) to reach the waypoint up E G. o represents how long each high-level action operates for. The low level given C₁ steps to achieve the goal where ch # C1,--
4 Methods
We introduce Decoupling Horizons Using Graph in Hierarchical Reinforcement Learning (DHRL), which can separate the time horizons of high-level and low-level policies and bridge the gap between both horizons using graph. Our framework consists of high-level policy π (sgls, g). low-level policy z/ (a|s,,, wp). and a graph G. Given a goal 9t in the environment, the high-level policy outputs subgoal sg, (see Figure 4). Then, the shortest path from the current state to sgt is found on the graph. To do so, St and sg. are added to the existing graph structure, then a sequence of waypoints (8t, upPt,1, WP...,.... 8gt) is returned using a graph search algorithm. Finally, the low-level policy tries to achieve wPt,, during C₁2, steps.
The key point of our method is that the low-level horizon klo is unrelated to the high-level horizon high Since the high level generates one subgoal every cs steps, the H-step task is a H //--- step task for a high-level agent high = H/c,) where Oj is the interval between high-level action (ch, q). In other words, unlike the previous HRL methods which have the relationship of
h high x h Jow H,
(2)
our algorithm does not have such relations, removing an obstacle toward a scalable-RL. algorithm. Since the low-level horizon e is determined by the edge cost between waypoints, can also express the Cj between the i-1th waypoint and 5th waypoint as Cr but we omit the letter i in the later statements that do not need specify the waypoint.
In section 4.1, we explain how to construct a graph over states and find a path on the graph level. In section 4.2, present lowelevel policy which can recover temporal distance between states without overestimation. In section 4.3, We introduce a strategy to train our method through graph-agnostic off- policy learning. Finally, in section 4.4, propose additional techniques for better data efficiency in large environments.
4.1 Graph level: planning over the graph
This section details the graph search part in DHRL planning. We emphasize that every distance in the DHRL model based on temporal distance Disl(- - -), obtained through Eq. (1). Thus, our algorithm requires no further information about the environment (e.g. Euclidean distance between states) than general HRL settings.
To find the shortest path, we adopt Dijkstra's Algorithm, as in the previous study [4]. The differences from the previous methods 4 27 are the existence of the high-level policy, and whether the secondary path is considered. Let s S + G be the projection of states on the goal space. In the graph initialization phase, we samples n nodes (also called landmarks) using FPS algorithm 2 (Algorithm
2 in Appendix A) from (5)) where s E S is state sampled from T Then, we connect a directed edge 81 - 82 if the temporal distance from 81 to 82 is less than the cutoff-threshold. In the planning phase, the graph G(V, E) gets the subgoal sgt from the high-level policy and adds the projection of current state (*))) and sg; to V, so that the number of nodes in G becomes n +
Then, connect the edges with costs less than the
between the newly added nodes

---

and existing nodes. Next, we find the sequence of waypoints W (wpr.o = (/(s), WPt.1. 2 1p wpt., sgr) that connects from (b(ss) to sg, using a graph search algorithm. At this time, if there is no path from (s()s) to sgt, we adopt a secondary path from ((s)) to 591, where sgt is the closest node to 8yt among the nodes connected from (s,)). After finding the waypoint sequence, the graph level provides wpt.1 to the low-level policy and instructs it to reach wpt,1. If it has been Dist wpt,i-1 then we update the current tracking waypoint to the next waypoint upt, In order to reflect the newly discovered state, we update graph nodes at every N episodes.
uptt,) steps since the agent started tracking wpt, it achieves upt,..
However, it is challenging to train Q-network which recovers reliable temporal distance. We explain the reason and solution in the next section.
4.2 Low level: Separate Q-networks for Graph and Critic (SQGC)
Unlike previous graph-guided RL methods that leverage graph only after the actor and critic networks have been learned, we construct and utilize a graph during training. To do so, the low-level policy Q is evaluated during training assign edge costs. However, when T is not yet competent in achieving some goals or encounters difficult goals, the accumulation of failure experiences in the replay buffer causes underestimation of Q-values. This leads to overestimation of temporal distance reconstructed from Eq. (T) and spoils the graph near the overestimated region by making the node-selection algorithm (Algorithm 2 in Appendix A) select more and more graph nodes around the overestimated area. Therefore, temporal-distance distance reconstruction method is needed even when the policy is not sufficiently trained.
For this reason, we propose Separate Q-networks for Graph and Critic (SQGC) to prevent temporal distance overestimation. SQGC is composed of two identical Q-networks using different proportions of hindsight goal relabeling (HER) M.. The SQGC includes Q and @geaph where Q is for training n/o just like a typical application, and Qgaaph is for recovering temporal distance between nodes. We substitute wpp with up, == agt+t; in the sequential transition of single episode (st, ups, a,, (5t-1, upp); 814 It=1:H11 where ag means the achieved goal and fnr is random integer drawn from the uniform distribution between 0 and H t. To train Q Qgeaph we relabel 100% of upp in (8t, wpt, as, r($+41, wpe), s1+1) as up,, while we replace only 80% of wp, for Q
Our method can prevent overestimation in the distance recovery by relabelling the goals in all transitions for training Qgeaph because the experiences of failure replaced with the successful trajectory (see section 5.3 for the ablation study). Also, by maintaining the original Q to train Tio there is no degradation in the performance of the agent who might otherwise not be able to get negative feedback from failure since the failure will be relabeled as a desired goal.
However, it is still challenging to train HRL including graph level using off- off-po RL algorithms. We describe our approach to train DHRL using an off-policy algorithm in the next section.
4.3 High level: hindsight transitions for graph-agnostic off off-policy learning
Thanks to decoupling the time horizons of both levels in HRL, the high-level policy in our method can look further without any additional burden the low-level policy. In other words, We can stretch <n which represents how long each high-level action operates for. However, because of the extended interval and non-stationarity of the high-level MDP, it is challenging to train DHRL with an off-policy algorithm, which is important for data efficiency in that the off off-policy algorithm can use the previous data from the replay buffer.
This non- stattiottonntty of the high-level MDP is caused by the presence of a graph and low-level policy, Since the graph is gradually updated, it is challenging to train model using off-policy method from the data given by the previous graph. Furthermore, as ch gets longer, predicting the similarity of the trajectory of с₁ steps with only the first step of (a,aa (sa g:) gets more difficult when replace subgoal using HIRO-style off- off-policy correction [17 which is a popular method adopted by SOTA
HRL algorithms [28 [2]
To facilitate the off-policy learning for longer e₁ and the changing graph DHRL,, we adopt well-known hindsight action relabelling method proposed in Hierarchical Actor-Critic (HAC)
5

---

15 in the high-level replay buffer data (st. St, s9t. St+c.) E Bh Alongside the original tran- sitions data, copy the transition data and replace the subgoal (sg: "g (sga (s, 9c)) with the achieved goal 09t+es, after Op steps. Thus, we use both transitions (st: It, $9t, Te, 8t-ex) ) and
high
(st. 9t, agt++x> St+co) to train the high-level policy. Our method utilizes the optimality of the graph and low-level policy include graph in the learning process, while the HAC-style hindsight action relabelling method assumes the optimal low-level policy n' only.
By replacing the previous subgoal with the achieved goal, we can assume that this transition was obtained from a stationary graph and zlo* with an error below a bound which is set to be a function of the density of the graph. We provide theoretical analysis of the possibility of replacing the old off- off-policy graph with virtual stationary graph in this section and Appendix B.
Definition 4.1. Given compact state space S, G(V, E) an e resolutiion graph Va E S, 3e E V s.t. max( (ist((((s) + v), Dist(v + (s))) < c, where V C (b(S) and E {(0), u5)lus, Us V, ViDist(r; 0 - 11) < c..
Theorem 4.2. Ler G be an arbitrary e-resolution graph (e < c/2). Also, let x and Gg be the low-level policy and graph at the time of collecting the data. Off-policy error rate p(G) is the normalized distance error with respect to the total traversal distance according to the change of a and Gg to g'00 and G. If there is a path from s to g, the upper bound of off-policy error rate p(G) using a path obtained from graph search over G is 2e/c.
Theorem 4.2 shows that if we replace the previous subgoal with the achieved goal, and assume that these transitions are obtained from stationary c- c-resolution graph G, then the off-policy error is less than 2c/q. Thus, we can train DHRL through graph-agnostic off off-policy RL algorithm using the substituted transition.
4.4 Optional techniques: gradual penalty and frontier-based goal-shifting
In this section, we propose two additional tech- niques, gradual penalty and contier-based goal- shifting (FGS). These optional techniques can boost the performance of DHRL on some long tasks as shown in Figure aa) and (c). We note that these are not essential to train DHRL and ours outperforms previous HRI frameworks without these techniques (see section 5.3 for the ablation study).
Gradual Penally
Frontier- Goal-Shilling
Figure 5: Optional techniques for better data effi- ciency: gradual penalty encourages the high-level policy to output subgoal that the low-level agent can achieve. Frontier-based goal-shifting replaces goals in the explored area with new goals posi- tioned in the rim of the graph.
Gradual penalty. Similar to the subgoal test- ing in the previous method [15]. we propose gradual penalty (algorithm 4 in Appendix A), which can impose the penalty more delicately when the action space of the high-level agent gets large. We evaluate *9; of the original transition data (st, It, 89t, Ft, 8t-c,) ) by categorizing the following three cases; (a) close the graph and low level actually achieved the subgoal L1, (b) close to the graph but the low-level policy could not achieve the subgoal E L2 and (c) far from the graph E L3. In this way, we can impose the more detailed penalties to respond to the expansion of the high-level action space as the high-level interval C₁ stretches.
Frontier-based goal-shifting. This optional technique has been devised to accelerate learning in a complex environment (algorithm 5l Appendix A). FGS moves the final goal to the frontier area when g comes into the place where the graph is already laid out during training. To check whether the goal is in the graph area, we examine whether min.e (Dist(r (v - g)) is smaller than cut-off threshold, where V the set of the graph nodes. Alternative goals are the addition of random noise to nodes sampled from V proportional to - -((so, π TSSu, v)) FGS is similar to the previous goal-directed exploration research [2T] in that both use weighted samples, but ours does not maintain a generative model and samples the goals from the graph level in DHRL. Note that we do not use FGS when we compare DHRL with the previous ofate-of-the-art, since this FGS is beyond the main contribution, which is about HRL structure.

---

PointMaaze
AntMareSmall
Anthazee omblex
UR 50bstacle
Teat sit surcess rdle
AritMaze
"AntMazeBotlennck
Reacher3D
Teresx5sk
DHRLIDURS)
HiGLiDense!
(RAn(Dense)
HiGL/Sparse)
HRAC(Sparse)
Tiresaoo
Figure 6: DHRL significantly outperforms prior state-of offttfe-aarrrrrr art algorithms (success rate averaged over 4 random seeds and smoothed equally, and only the sparse settings for Reacher3D are tested as in the previous papers). Note that in AntMazeComplex, AntMazeBotlleneck and UR3Obstacle, the curves are not visible as they overlap at zero success rate.
5 Experiments
5.1 Environment description
We evaluate DHRL on robot environments based on the MuJoCo simulator including some sparse and long-horizon tasks. Firstly, various locomotion environments with 'fixed initial state distributions' are used to validate the temporal abstraction capability of DHRL long-horizon and cluttered environments. In test episodes of the locomotion environments, the agent gets one of the most challenging goals (i.e., the end of the maze). We also evaluate our algorithm robot arm environments in which the agent aims to make the end-effector touch the goal, to evaluate our method in more complex dynamics.
PointMaze / AntMazeSmall The point ant achieves the goal if it comes within 2.5 distance (success threshold) from the target point in 12 x 12 maze.
AntMaze 24 x 24 maze with success threshold 5
AntMazeBottleneck Bottleneck exists at the middle of the maze. The ant can barely pass through bottleneck.
AntMazeComplex : 56 56 maze with success threshold
Reacher3D 7-dof robot arm aims reach goal.
UR3Obstacle 6-dof robot arm aims to reach a goal in environment with several board- shaped obstacles.
We adopt TD3 algorithm (5] for high-level and low-level networks and use Dijkstra algorithm to find the shortest path in the graph level. Also, note that only few hyperparameters have been changed across various locomotion experiments: the number of nodes, penalty, and Ch. The results of DHRL in this paper are obtained using only sparse reward.
5.2 Experiment result
success rate
HRAC, HRAC,AntMazes AntMaze5mall
Ours,AntMan mntMaze5mm
Baselines. We compare our method with state- of-the-art algorithms with and without graph respectively; HIGL [12] and HRAC [28). For
E
h--20
08
ch-20
Chh330
Cheao
ch-
h--50
hh
more comparison with shallow RL (SAC)
h-
h-44
and vanilla HRL (HIRO) [17]. see Table in Appendix C.
0.
Timesteps(k)
-10
Timesepsii
Figure 7: Robustness to Cy: HRAC vs Ours
Comparison to state-of-the-art algorithms. Results are shown Figure [6] Thanks to decoupling the horizons of the levels, DHRI. could stretch the high-level interval and shows high data efficiency and success rate in various locomotion
7

---

(alAntMazeBotteneck
AntMAnaaaes mall
(bJAntMaaeeComllex
Obslacle
Obstacle
botacce
(e) Reacher3D
Figure 8: Locomotion and Reacher tasks in simulation: Additional to general tasks for HRL, evaluated our method in long and sparse tasks (see (a), (b), and (c) in the figure). The maps of the environments are not given the agent.
Table 1: The tradeoff in performance between the high level and the low level.
SUCCESS RATE
HRAC Cb. =5
HRAC C₁ = 10
HRAC Ch.
30
HRAC Ci.
50
DHRL
12 12 MAP
43.0%
88.4%
78.3%
4.5%
95.1 %
24 x 24 MAP
18.0%
48.9%
57.4%
16.4%
91.1
56 56 MAP
0.0%
0.0%
0.0%
0.0%
40.1 %
and goal-reaching tasks. Moreover, DHRL is the only algorithm that can succeed in complex envi- ronments (AntMazeComplex and AmtMazeBottleneck) When measuring the performance of the baselines, we selected Os with the best performance among the values c =10, 20, and 80.
As shown in Figure 6 previous HRL methods cannot solve long-horizon tasks. This is likely due to the coupling (c), = c;) of the high-level horizon high H/ch) and the low-level horizon low ct) resulting in an increased burden on either the high-level or the low-level policy. For long-horizon tasks (large H). if c₁ (= ex) fixed, then high increases and the high-level performance plunges (Table On the other hand, if ea (= ci) is increased, the low level has to manage wider area and the performance plunges as shown in Figure and Table
5.3 Ablation study
Decoupling horizons. In this section, we examine how decoupling horizons affects the performance of long-horizon HRL and evaluate whether DHRL can stretch the interval of high-level policy. As shown in Figure 9b), we tested various values of spacing of the high-level action, C₁- The result shows that even in long intervals of the high-level policy, DHRL shows consistent improved performance (Figure 9b) without major degradation unlike previous HRL methods (Figure 7 In particular, DHRL agent in long- onn-horizoon environments such as AntMazeComplex shows that extended high-level interval is crucial for exploration and performance (see the rightmost figure in Figure 9 b)). Considering that the baselines could not solve long-horizon tasks even with various values of cr. and the performance degrades with increasing Ca. we conclude that our method successfully decoupled the time steps both levels and stretched the interval between high-level actions. This trait allows the high-level policy to look further and take advantage of the extended temporal abstraction.

---

(a)
AntMaaeSmaal
AntMaze Complex
(c)
mat
AntMazeCompar
Test TesTSUECO rate
Tost SucCOsSS rate
Timesteps(k)
Emmeslpsslk
Timesleps(k)
Times
DHRL
+GG
DHRL-SQGC
Gradual Preaity
Without Penalty
[b]
AntMaze
AntMazeCompper
(d)
AntMareS mall
AntM
Teist SUcCesS rall
Test succeess rote
Timesteps(k) of DHAL
Timesteps(k)
Timestepsiki
Timesteps(k)
Ca of DHRL
DHAL
L3P (Random)
L3P (Fixed)
Figure 9: Ablation study. (a): The separate Q-network (SQGC) effectively alleviates the overestima- tion problem of temporal distance and FGS improves data efficiency. (b): Extending the high-level horizon is crucial in long-horizon tasks and DHRL successfully separates the horizons. (c): Gradual penalty improves the data efficiency in long tasks. (d): Without additional high-level policy above the graph level, the performance plunges without the 'uniform initial state distribution'.
Separate Q-network for Graph and Critic (SQGC). In this section, evaluate the effects of the Separate Q-network for Graph and Critic (SQGC) in our method. The red line in Figure which is variant of DHRL without an SQGC, clearly shows that the SQGC is critical to the performance. We empirically found that the overestimation of the temporal distance between states occurs especially near the obstacles (e.g. corners at the maze), and it is difficult for agent pass by without separate Q network for graph construction. This is consistent with our analysis that the experiences of failure spoil the ability to recover the temporal distance from Q-network.
Ablate high-level policy. Graph-guided RI., which maintains graph level and low-level policy, is also a variant of HRL in that the graph level is a non-parametric version of the high-level policy (see Figure B) From this point of view, the main difference between graph-guided RL and our method is the existence of the high-level policy. Then, why do we need additional high-level policy above the graph level?
Most graph-guided RL algorithms use 'uniform initial state distribution' train the agent in complex environments and such assumption could be expensive, especially in the physical world where it is challenging to start from different positions each time. Figure d()) shows the result of the graph- guided RI. algorithm, L.P 277. with uniform and fixed initial state distributions. This indicates that without "uniform initial state distribution', the performance and data efficiency the previous graph-guided RL method drops seriously even in the smallest environment we experimented with. In contrast, DHRL can explore without 'uniform initial state distribution' even in long environments thanks to the enlarged temporal abstraction and exploration performance. Considering that the high- level policy suggests a subgoal to graph level, we conclude that the high level facilitates better exploration of the graph-guided RL. Thus, DHRL that has exploration capability also can be seen as the improved version of graph-guided RL.
Optional techniques. The result in Figure (() shows that the gradual penalty can improve data efficiency in long tasks. By imposing penalty on subgoals that are far from the current graph, the gradual penalty encourages the high-level agent to output a subgoal near the graph. Since the low-level agent can hardly achieve subgoals far from the explored area, the gradual penalty accelerates the training by providing more valid goals. FGS also can improve training efficiency in complex environments (Figure D a)). This means that it is important to provide goals near the unexplored areas which help the agent explore the unseen states in complex environments. With FGS, DHRI. efficiently guides the agent by sampling the goals located on the rim of the graph.

---

Conclusion
We proposed a Decoupling Horizons Using a Graph in Hierarchical Reinforcement Learning (DHRL), which is a data-efficient HRL algorithm leveraging graph to expand the range of problems that HRL can solve, by decoupling horizons and allowing both levels to operate at their suitable horizons. Our experimental results show that our method successfully separates the horizons of the levels and outperforms prior sfate-of-the-ase methods. We believe that our method can present a direction towards scalable HRL allowing the hierarchical policy take advantage of the extended temporal abstraction and have lower dependency on the horizon of the environment. In this paper, we adopted vanilla graph construction algorithm and sparse reward settings. We expect that if a novelty-based node selection is added to the graph construction, the performance could be further improved.
Limitation. While our algorithm shows successful results in complex and long-horizee tasks, might be difficult to construct graph in some tasks that require complex interactions with the environment because of the higher-dimensionll state space. We expect that this limitation can be solved by extending our algorithms into latent state space, or by incorporating node sparsification algorithm, which we leave for future works.
7 Acknowledgement
This work was supported by Al based Flight Control Research Laboratory funded by Defense Acquisition Program Administration under Grant UD200045CD, Also, Seungjae Lee is grateful for financial support from Hyundai Motor Chung Mong-Kor Foundation.
References
[1] Marcin Andrychowicz. Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 5055-5065 2017.
[2] David Arthur and Sergei Vassilvitskii. K-means++: The advantages of careful seeding. In Proceedings of the Eighteenth Anmual ACM-SIAM Symposium on Discrete Algorithms, SODA '07, page 1027-1035, USA, 2007. Society for Industrial and Applied Mathematics. ISBN 9780898716245
[3] Akhil Bagaria, Jason K Senthil, and George Konidaris. Skill discovery for exploration and planning using deep skill graphs. International Conference on Machine Learning, pages 521-531. PMLR, 2021.
[4] Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[5] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.
[6] Robert Gieselmann Florian T Pokorny. Planning-augmented hierarchical reinforcement learning. JEEE Robotics and Automation Letters, 6(3):5097-511) 2021.
[7] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1 812.05905 2018.
[8] Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. In Robotics: Science and Systems, 2019.
[9] Zhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching. Advances in Neural Information Processing Systems, 32: 1942- 2019.
[10] Jemin Hwangbo, Inkyu Sa, Roland Siegwart, and Marco Hutter. Control of quadrotor with reinforcement learning. IEEE Robotics and Automation Letters, 2(4):209--2033, 2017.
[11] Deepali Jain, Atil Iscen, and Ken Caluwaerts. Hierarchical reinforcement learning for quadruped loco- motion. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7551-7557. IEEE, 2019.
10

---

[12] Junsu Kim, Younggyo Seo, and Jinwoo Shin. Landmark-guun subgoal generation in hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[13] Nathan O Lambert, Daniel S Drew, Joseph Yaconelli, Sergey Levine, Roberto Calandra, and Kristofer SJ Pister. Low-level control of quadrotor with deep model-based reinforcement learning. IEEE Robotics and Automation Letters, 4(4):4224 4230, 2019.
[14] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334 1373, 2016.
[15] Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In Proceedings of International Conference on Learning Representations, 2019.
[16] Ofrr Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learning Representations, 2018.
[17] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. Advances in Neural Information Processing Systems, 31:3303-3313, 2018.
[18] Ofrr Nachum, Michael Ahn, Hugo Ponte, Shixiang Shane Gu, and Vikash Kumar. Multi-agent manipulation locomotion using hierarchical sim2real. Conference on Robot Learning, pages 110-121. PMLR. 2020.
[19] Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. Advances in Neural Information Processing Systems, 32: 14843-14854, 2019.
[20] Xue Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Gruphics (TOG), 36(4):1-13, 2017.
[21] Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering supervud reinforcement learning. In International Conference on Machine Learning, pages 7783-7792. PMLR, 2020.
[22] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: (tfl::170..0000000 10087, 2017.
[23] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pages 1312-1320. PMLR, 2015.
[24] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1889-1897, Lille, France, 07-09 Jul 2015. PMLR. https:tts:////tt btttpp://proceeednggg mlr. pwwppreeevevv v37 schul html
[25] Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, et al. Strategic attentive writer for learning macro-actions. Advances in Neural Information Processing Systems, 1113448634944 2016.
[26] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hicrarchical reinforcement learning. In International Conference on Machine Learning, pages 3540-35 3549. PMLR, 2017.
[27] Lunjun Zhang, Ge Yang, and Bradly C Stadie. World model graph: Learning latent landmarks for plamning. In International Conference on Machine Learning, pages 12611- PMLR, 2021.
[28] Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating aelaccency-constraine subgoals in hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
11


Reviews:
[{"review_id": "faKTubLPK9m", "reviewer": "Reviewer_dQCH", "summary": "The method proposed in the paper suggests that supporting a decoupling between the high and low-level policy control frequency can result in better performance. Often in HRL, there are limitations in the communication between the high and low-level policy, and this communication is the source of poor performance for these models, keeping them from outperforming other methods on more complex tasks. This paper proposes the reduce over-estimation and increase generalization by introducing a method that combines graph-based methods (Like SORB) and HRL methods like HIRO to create a more versatile HRL agent with fewer learning limitations. Methods from HIRO are used to reliable data for better learning the low-level policy, and methods from SORB-like methods are used to help propose better subgoals.\n\n--------\n\nThe score has been updated after the first responses.\n\n----- \n\nThe score has been updated again after further discussions.", "questions": "Questions/Comments:\n\n- section 4.1 discusses the particular Graph Search algorithm that is used by the method. it would be extremely helpful to the reader if the description in this area can also make more connections to Prior work to allow the reader to open understand the novelty and in particular what is different between these graph search algorithms in this section and prior work like sore\n- the paragraph break online 180 is very jarring. and the next paragraph it's not clear which particular property is being discussed and mitigated.\n- There does not appear to be any information related to the statistical validity of the findings in the experiments. how are the number of random seeds used in the experiments determined? what is the confidence that we have in the overall statistical tests that we are using to compare different methods?\n- Figure 10 appears to only show comparisons to HIRO across a few of the scenarios. it's important to have some additional comparisons to understand how this method compares to popular prior methods.\n- The claim that the paper points of a fundamental reason for HRL structures being not scalable is unclear. It will help if there is some theoretical grounding to place this contribution. Figure 3 appears to be more of a concept diagram than a fundamental reason in the optimization that outlines the limitation.", "limitations": "The authors note some limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The method proposed in the paper has some potential nice connection to shy the results are strong.\n- HRL is challenging, and this method may be a nice step toward eliminating some of the issues with HRL methods. \n- Blending high-level policies with graph-based methods provides a nice trade-off between the weakness of both.\n\nCons:\n- The authors note that the reason their method is an improvement over prior graph-based methods is that it doesn't rely on or it can support sparse reward methods and doesn't rely on random resets. In some cases, random resets can be even more plausible and being able to reset an agent back to the same exact initial conditions given certain constraints inside of the environment. It would be very helpful if the authors can expand on this discussion in this section to help us better understand the reasoning behind the method and how it is different and an improvement over prior methods.\n- One of the points argued in the paper is that being able to decouple the horizons of the mobile high-level policy should make training better. It is discussed in the paper in the introduction but it's not a fairly clear argument as to why having this disconnect is going to result in better performance. In particular, some understanding or framework that places this inside of having better generalization and/or control across the policies would likely assist the readers with appreciating the insights in this method. This would also help readers better understand when this method will be an improvement over other methods.\n- To help the organization of the paper, if the earlier sections that outline additions to the method reference later parts of the experiment section, the reader can reference and understand the importance of these additions. Otherwise, as the paper is now, it's rather difficult to infer the particular contribution importance of changes over prior methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_Doc1esccxn", "reviewer": "Reviewer_ctmE", "summary": "The paper proposes to use a graph construction and search method in Hierarchical RL to break down subgoals from the high-level policy into fine-grained subgoals for the low-level policy. Essentially, the paper proposes to use 3 levels: the highest and lowest levels use RL methods and the middle level is using a graph, which is built using learned temporal distances.\nIn this way, the horizons of the levels are more decoupled and long-horizon problems can be solved. Its benefits are shown in several simulated robotic environments such as maze tasks and manipulator reaching tasks.", "questions": "- Can you report the numbers of just the graph+lowlevel (so essentially L3P) AFTER you learned the graph with your method. I expect the numbers to be as high as with your method, which would be valuable information for further research.\n\n- Please clarify Eq 1.\n\n- What is the size of the graphs that are constructed? (is it the number of landmarks?)\n\n- There are threshold parameters for reaching a waypoint/subgoal by the low-level. I did not see the parameter used or this. Also how sensitive is the method to this choice?", "limitations": "Limitations have been addressed", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the method is sound\n- the performance is much better than the baselines\n- ablations were conducted on most of the components\n\n\nWeaknesses:\n- the language of the paper needs improvements\n- the clarify can be improved\n- I suspect, the high level is only relevant for exploration and is potentially not required for solving the task.\n\nDetails:\n- You are missing a lot of articles in your text, so many occurrences, that I will not mention them here. For instance: ... Using Graph in HRL -> ... Using a Graph \n- you should replace: \"extremely long\" with \"very long\" or just \"long\"\n- L28: using a temporally...\n- L80: I think it is not a great practice to use two-letter variable names in mathematics (sg)\n- Eq 1: I am wondering whether there is maybe a sign error in EQ 1. Maybe, I am mistaken, but I thought it should be $\\log_\\gamma (1 - (\\gamma-1)Q)$. I think the derivation of that formula might be nice in the Appendix. \n- L94: Graph as recently.... ->  Graphs have recently been used ...\n- L106ff: citations missing\n- Sec 3: I think you should compare your method to multilevel hierarchical methods as you have 3 levels\n   A method that also has not a fixed time horizon for the low level is HiTS [1] but it does not solve the long-horizon problem. \n- L139: why is c_l also depending on \"i\"? Later you only have c_l without i. c_h is also not defined unless I have overlooked it.\n- L183: We -> we\n- L188: formulation with \"whole\" and \"part\" is not ideal. You mean all occurrences vs a subset (here 80%)...\n- L213: how is that different from HAC? You write it is inspired, but to me, it looks just like HAC.\n- L220: why are \"forall\" and \"exists\" in superscript?\n- L231: I thought the problem is not off-policy correction, but a non-stationarity of the semi-MDP.\n- L241ff: I am missing the details here. I could not implement it from what is written here. Also for FGS, more details should be provided (in Appendix and references, for instance).\n- L268: A threshold of 2.5 in a 12x12 maze is really a huge goal area. Is that standard? Similar in the AntMaze. What is the size of the Ant? A threshold of 5 is really big. Please specify what is a standard-settings and what you changed. If you can solve it with a smaller threshold, that might also be a strong additional experiment.\n\nL299: horizon^{high} maybe h^{high}. Also, such words should not be italic. What is the difference between c and horizon?\nL360: grammar   \n   \n[1] Hierarchical Reinforcement Learning with Timed Subgoals, Gürtler et. al, NeurIPS 2021\n\n-- post rebuttal --\nmost of my concerns are addressed and I am raising my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "JwU1G2QCFJq", "reviewer": "Reviewer_B21k", "summary": "Paper proposes a new method for hierarchical RL which decouples the horizon of the high level policy (outputs subgoals) from the horizon of the low level policy (outputs actions given target waypoint), by adding an additional planning phase between the high level and low level policies which searches over a graph for a sequence of waypoints which reaches the subgoal. The experiments show that this decoupling of horizons allows RL algorithms to better solve long-horizon tasks, and that the proposed algorithm outperforms prior HRL algorithms.  ", "questions": "N/A", "limitations": "Authors noted that constructing the graph used for planning may be difficult for environments with high-dimensional state spaces, and proposes to instead learn the graph over a latent space. \n\nNo negative societal impact that I am aware of.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Nicely written paper which clearly explains the difficulties of using prior HRL algorithms (horizon of low-level and high-level policies are coupled, making it difficult to scale to very long horizon tasks), and proposes method of address this weakness by decoupling the horizon of the high-level and low-level policies\n- Experiments clearly show improvements over previous HRL algorithms on a variety of tasks\n- Thorough ablation studies that illustrate the importance of different components in the algorithm \n\nWeaknesses\n- Description of method is a bit hard to digest because the algorithm involves many parts, and each part involves many procedures. Maybe adding some algorithm boxes which summarize each section, and also an overarching algorithms box which connects all the subparts will aid in understanding.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GPsRC0LLRW9", "reviewer": "Reviewer_sUno", "summary": "In this work, the authors tackle long horizon reinforcement learning tasks by introducing a hierarchical reinforcement learning (HRL) method called DHRL. DHRL consists of 3 layers - a high-level policy that outputs a subgoal, a mid-level graph that outputs waypoints towards the subgoal, and a low-level policy that outputs actions to navigate towards the waypoints. Doing so decouples the planning horizons between the high-level and low-level policies allowing for more efficient planning of longer horizon tasks. Because the mid-level graph is not differentiable, experience relabeling is adopted to train the high- and low-level policies separately. The method is evaluated on a maze navigation and manipulator goal tracking tasks in simulation and results show significant improvement over comparison RL and HRL methods in terms of sample efficiency and final performance.", "questions": "- If I understand correctly, the high- and mid-level planning is similar to a probabilistic road map only instead of randomly sampling, the high-level policy provides the samples towards the goal. If this is the case, wouldn't the method be stuck at the beginning when the high-level policy is untrained? If not, then how is the graph constructed?\n\n- what is most time consuming component of DHRL in terms of inference and how does this component scale with the dimensionality of its input?\n\n- I would suggest avoiding overly strong but vague claims such as \"fundamental reason\", \"extremely complex and long tasks\"\n", "limitations": "Having a graph middle layer inherits some of the limitations of graph-based planning methods. This includes the inefficiency caused by (1) potentially high dimensional state space (mentioned by the authors in Section 6); (2) closely situated obstacles or corridors as in a cluttered environment. Predicates can also be somewhat limited in terms of representing safety in the observation space.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The method is conceptually sound and the improvements on the chosen tasks are significant. I also liked the details and practical considerations the authors included in their discussions.\n\nWeaknesses\n- Navigation task and goal tasks are conceptually simple. Long horizon tasks should be accompanied by increased complexity and practicality in the real world. The authors are encouraged to try DHRL on kitchen tasks (https://github.com/rmrafailov/kitchen ) and more complex manipulation tasks (GitHub - stepjam/RLBench: A large-scale benchmark and learning environment. )", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept