Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Paper proposes a new method for hierarchical RL which decouples the horizon of the high level policy (outputs subgoals) from the horizon of the low level policy (outputs actions given target waypoint), by adding an additional planning phase between the high level and low level policies which searches over a graph for a sequence of waypoints which reaches the subgoal. The experiments show that this decoupling of horizons allows RL algorithms to better solve long-horizon tasks, and that the proposed algorithm outperforms prior HRL algorithms.  
strengths_and_weaknesses: Strengths
- Nicely written paper which clearly explains the difficulties of using prior HRL algorithms (horizon of low-level and high-level policies are coupled, making it difficult to scale to very long horizon tasks), and proposes method of address this weakness by decoupling the horizon of the high-level and low-level policies
- Experiments clearly show improvements over previous HRL algorithms on a variety of tasks
- Thorough ablation studies that illustrate the importance of different components in the algorithm 

Weaknesses
- Description of method is a bit hard to digest because the algorithm involves many parts, and each part involves many procedures. Maybe adding some algorithm boxes which summarize each section, and also an overarching algorithms box which connects all the subparts will aid in understanding.
questions: N/A
limitations: Authors noted that constructing the graph used for planning may be difficult for environments with high-dimensional state spaces, and proposes to instead learn the graph over a latent space. 

No negative societal impact that I am aware of.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors tackle long horizon reinforcement learning tasks by introducing a hierarchical reinforcement learning (HRL) method called DHRL. DHRL consists of 3 layers - a high-level policy that outputs a subgoal, a mid-level graph that outputs waypoints towards the subgoal, and a low-level policy that outputs actions to navigate towards the waypoints. Doing so decouples the planning horizons between the high-level and low-level policies allowing for more efficient planning of longer horizon tasks. Because the mid-level graph is not differentiable, experience relabeling is adopted to train the high- and low-level policies separately. The method is evaluated on a maze navigation and manipulator goal tracking tasks in simulation and results show significant improvement over comparison RL and HRL methods in terms of sample efficiency and final performance.
strengths_and_weaknesses: Strengths
- The method is conceptually sound and the improvements on the chosen tasks are significant. I also liked the details and practical considerations the authors included in their discussions.

Weaknesses
- Navigation task and goal tasks are conceptually simple. Long horizon tasks should be accompanied by increased complexity and practicality in the real world. The authors are encouraged to try DHRL on kitchen tasks (https://github.com/rmrafailov/kitchen ) and more complex manipulation tasks (GitHub - stepjam/RLBench: A large-scale benchmark and learning environment. )
questions: - If I understand correctly, the high- and mid-level planning is similar to a probabilistic road map only instead of randomly sampling, the high-level policy provides the samples towards the goal. If this is the case, wouldn't the method be stuck at the beginning when the high-level policy is untrained? If not, then how is the graph constructed?

- what is most time consuming component of DHRL in terms of inference and how does this component scale with the dimensionality of its input?

- I would suggest avoiding overly strong but vague claims such as "fundamental reason", "extremely complex and long tasks"

limitations: Having a graph middle layer inherits some of the limitations of graph-based planning methods. This includes the inefficiency caused by (1) potentially high dimensional state space (mentioned by the authors in Section 6); (2) closely situated obstacles or corridors as in a cluttered environment. Predicates can also be somewhat limited in terms of representing safety in the observation space.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The method proposed in the paper suggests that supporting a decoupling between the high and low-level policy control frequency can result in better performance. Often in HRL, there are limitations in the communication between the high and low-level policy, and this communication is the source of poor performance for these models, keeping them from outperforming other methods on more complex tasks. This paper proposes the reduce over-estimation and increase generalization by introducing a method that combines graph-based methods (Like SORB) and HRL methods like HIRO to create a more versatile HRL agent with fewer learning limitations. Methods from HIRO are used to reliable data for better learning the low-level policy, and methods from SORB-like methods are used to help propose better subgoals.

--------

The score has been updated after the first responses.

----- 

The score has been updated again after further discussions.
strengths_and_weaknesses: Pros:
- The method proposed in the paper has some potential nice connection to shy the results are strong.
- HRL is challenging, and this method may be a nice step toward eliminating some of the issues with HRL methods. 
- Blending high-level policies with graph-based methods provides a nice trade-off between the weakness of both.

Cons:
- The authors note that the reason their method is an improvement over prior graph-based methods is that it doesn't rely on or it can support sparse reward methods and doesn't rely on random resets. In some cases, random resets can be even more plausible and being able to reset an agent back to the same exact initial conditions given certain constraints inside of the environment. It would be very helpful if the authors can expand on this discussion in this section to help us better understand the reasoning behind the method and how it is different and an improvement over prior methods.
- One of the points argued in the paper is that being able to decouple the horizons of the mobile high-level policy should make training better. It is discussed in the paper in the introduction but it's not a fairly clear argument as to why having this disconnect is going to result in better performance. In particular, some understanding or framework that places this inside of having better generalization and/or control across the policies would likely assist the readers with appreciating the insights in this method. This would also help readers better understand when this method will be an improvement over other methods.
- To help the organization of the paper, if the earlier sections that outline additions to the method reference later parts of the experiment section, the reader can reference and understand the importance of these additions. Otherwise, as the paper is now, it's rather difficult to infer the particular contribution importance of changes over prior methods.
questions: Questions/Comments:

- section 4.1 discusses the particular Graph Search algorithm that is used by the method. it would be extremely helpful to the reader if the description in this area can also make more connections to Prior work to allow the reader to open understand the novelty and in particular what is different between these graph search algorithms in this section and prior work like sore
- the paragraph break online 180 is very jarring. and the next paragraph it's not clear which particular property is being discussed and mitigated.
- There does not appear to be any information related to the statistical validity of the findings in the experiments. how are the number of random seeds used in the experiments determined? what is the confidence that we have in the overall statistical tests that we are using to compare different methods?
- Figure 10 appears to only show comparisons to HIRO across a few of the scenarios. it's important to have some additional comparisons to understand how this method compares to popular prior methods.
- The claim that the paper points of a fundamental reason for HRL structures being not scalable is unclear. It will help if there is some theoretical grounding to place this contribution. Figure 3 appears to be more of a concept diagram than a fundamental reason in the optimization that outlines the limitation.
limitations: The authors note some limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes to use a graph construction and search method in Hierarchical RL to break down subgoals from the high-level policy into fine-grained subgoals for the low-level policy. Essentially, the paper proposes to use 3 levels: the highest and lowest levels use RL methods and the middle level is using a graph, which is built using learned temporal distances.
In this way, the horizons of the levels are more decoupled and long-horizon problems can be solved. Its benefits are shown in several simulated robotic environments such as maze tasks and manipulator reaching tasks.
strengths_and_weaknesses: Strengths:
- the method is sound
- the performance is much better than the baselines
- ablations were conducted on most of the components


Weaknesses:
- the language of the paper needs improvements
- the clarify can be improved
- I suspect, the high level is only relevant for exploration and is potentially not required for solving the task.

Details:
- You are missing a lot of articles in your text, so many occurrences, that I will not mention them here. For instance: ... Using Graph in HRL -> ... Using a Graph 
- you should replace: "extremely long" with "very long" or just "long"
- L28: using a temporally...
- L80: I think it is not a great practice to use two-letter variable names in mathematics (sg)
- Eq 1: I am wondering whether there is maybe a sign error in EQ 1. Maybe, I am mistaken, but I thought it should be $\log_\gamma (1 - (\gamma-1)Q)$. I think the derivation of that formula might be nice in the Appendix. 
- L94: Graph as recently.... ->  Graphs have recently been used ...
- L106ff: citations missing
- Sec 3: I think you should compare your method to multilevel hierarchical methods as you have 3 levels
   A method that also has not a fixed time horizon for the low level is HiTS [1] but it does not solve the long-horizon problem. 
- L139: why is c_l also depending on "i"? Later you only have c_l without i. c_h is also not defined unless I have overlooked it.
- L183: We -> we
- L188: formulation with "whole" and "part" is not ideal. You mean all occurrences vs a subset (here 80%)...
- L213: how is that different from HAC? You write it is inspired, but to me, it looks just like HAC.
- L220: why are "forall" and "exists" in superscript?
- L231: I thought the problem is not off-policy correction, but a non-stationarity of the semi-MDP.
- L241ff: I am missing the details here. I could not implement it from what is written here. Also for FGS, more details should be provided (in Appendix and references, for instance).
- L268: A threshold of 2.5 in a 12x12 maze is really a huge goal area. Is that standard? Similar in the AntMaze. What is the size of the Ant? A threshold of 5 is really big. Please specify what is a standard-settings and what you changed. If you can solve it with a smaller threshold, that might also be a strong additional experiment.

L299: horizon^{high} maybe h^{high}. Also, such words should not be italic. What is the difference between c and horizon?
L360: grammar   
   
[1] Hierarchical Reinforcement Learning with Timed Subgoals, Gürtler et. al, NeurIPS 2021

-- post rebuttal --
most of my concerns are addressed and I am raising my score.
questions: - Can you report the numbers of just the graph+lowlevel (so essentially L3P) AFTER you learned the graph with your method. I expect the numbers to be as high as with your method, which would be valuable information for further research.

- Please clarify Eq 1.

- What is the size of the graphs that are constructed? (is it the number of landmarks?)

- There are threshold parameters for reaching a waypoint/subgoal by the low-level. I did not see the parameter used or this. Also how sensitive is the method to this choice?
limitations: Limitations have been addressed
review_title: null

Meta-review: metareview: The reviewers appreciated the authors' response with additional experiments and clarifications. Given the feedback from the reviewers and the discussion, I would like to recommend this paper for acceptance and congratulate the authors on a strong submission. I encourage the authors to address the reviewers' comments for the final version of the paper.


confidence: Certain
Decision: Accept