Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an agent that combines causal discovery and hierarchical reinforcement learning (HRL), combined into causality-driven HRL, or CDHRL. The environment variables, which form the basis for the nodes of the structural causal model that the agent uses, are assumed to be available through an oracle. Training happens in two stages: first, the causal structure is discovered by training the agent to reach an expanding set of subgoals, after which the agent is trained on the actual task. On the two environments that are evaluated, the CDHRL method discovers reasonable causal graphs and outperforms the two baselines. Several ablations show that those results are robust to noise in the provided environment variables and limited iteration depth.
strengths_and_weaknesses: - Strength: Combining causality with HRL makes lots of sense. Often, HRL approaches don’t explicitly address the causality, but assume it implicitly. Assuming the environment variables are available, providing the assumption of a causal structure as an inductive bias is as natural as transformers are for language or convnets for images.
- Strength: The interaction between the subgoal construction and the discovery process effectively induces an exploration reward, so the proposed agent not only learns about causal structure in the environment that it sees, but also ensures it sees many interesting aspects of the environment.
- Strength: the paper is well-structured and easy to read.
- Weakness: the availability of the environment variables is assumed. The authors mention a few methods aimed at discovering those, but no experiments are included where discovered environment variables are being used.
- Weakness: The results are a bit limited. Two tasks, two baselines. They make the point that in the right circumstances, this method is effective, but they don’t really address the question which circumstances are the right ones.
- Weakness: the paper is in need of a round of language editing.
questions: - Could the authors discuss the differences between their contribution and those of Corcoll et al.? Both are aiming to introduce causality into hierarchical reinforcement learning. Not citing Corcoll et al. seems like an oversight.
- Did the authors consider environments like the ant maze in the MEGA paper? While they may not be the most interesting from a causal perspective, they would be interesting regression tests.
- Could the authors comment on the causality graphs discovered for the Eden environment? The graph for minecraft is very helpful.
limitations: - The assumption that the EVs are available through an oracle is limiting. The results presented are for cases where they are available. Discovering them in an unsupervised way can be problematic - the signal for the usefulness of candidate EVs will have to come from a combination of task reward and causality discovery success. Hence, discovering EVs is likely to require interaction with the causality discovery process, which would create a cold-start problem. The question of EV discovery is not addressed in any of the experiments. That isn’t a strict necessity for this paper, but it does limit the applicability of the method, and addressing it would make the paper stronger.
- The environment variables are assumed to be discrete. This assumption makes the exploration easier by limiting the effective number of states that need to be considered in the causal graph. Again, addressing this aspect, ideally with experiments (which would presumably take place on other environments than the ones used) would clarify the applicability domain of the proposed method.
- The causal structure of the environment is assumed to be small enough in number to constitute a nicely explorable graph - the pre-training phase runs ‘until no new causality is discovered’. This assumption won’t generally hold, and addressing the question of how much to invest in causality discovery would be valuable for understanding the generalizability of the results.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a causality driven hierarchical reinforcement learning framework to build hierarchical structures in an iterative boosting way. The paper shows experimental results in 2D-Minecraft and Eden environments with sparse rewards. 
strengths_and_weaknesses: Strengths: 
The paper is well written and clear. The experiments are well done. 

Weakness: 
The weakest part of the paper is that it is not clear where the novelty of the paper lies. It would be greatly appreciated if the authors differentiate their work against prior work. 

Multiple previous work tries to find graph like structures in environments for finding skills/options etc
For example, Robot learning from demonstration by constructing skill trees.


Important Prior Work on HRL and Minecraft: 
1) Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations
2) Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution
questions: It is not clear to me the need of using ideas from causality? Given some environment variables, relationships between variables can be discovered without ideas from causality.

Also, intervention sampling seems very close to the notion of exploration. Just that it is a bit smarter to explore in a smaller space using environment variables.  
limitations: The assumptions that environment variables are given to us is the main limitation of the paper. Finding these environment variables along with the structure, is difficult.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new hierarchical reinforcement learning method that could automatically discover hierarchical structure with a learned causality graph. To build the causality graph, the proposed method introduces the “Environment Variables” as nodes in that graph, which requires some task-specific knowledge. The causality graph and subgoal-based hierarchical policy are simultaneously learned and facilitate each other.  
strengths_and_weaknesses: ## Significance
Automatic hierarchical structure discovery is a fundamental research problem in deep reinforcement learning. This paper provides a new way to solve this problem, which owes remarkable significance. 

## Originality
The idea of using a causality graph to discover hierarchical structure is novel. However, the proposed method relies on the “Environment Variables”. The way of obtaining the “Environment Variables” is non-trivial, so I doubt whether this method could be applied to more general settings, where the oracle “Environment Variables” are hardly accessible. 

## Clarity
The writing of this paper is mostly clear, but seems in a rush. For example, there is a latex compile error in Line 485. The authors are encouraged to polish up the writing in the future.

## Quality 
The idea of using a causality graph to facilitate the discovery of hierarchical structure makes sense, since there are causality relationships among subtasks in most hierarchical tasks. The proposed method is compared against previous goal-conditioned HRL works. However, claiming HAC as a state-of-the-art method is not precise. A lot of more advanced goal-conditioned HRL works have emerged in recent years [1, 2], where the World Model paper also proposed to build a graph to guide the subgoal selection. The authors are encouraged to compare the proposed method to more advanced goal-conditioned HRL methods.

[1] Zhang er al., World Model as a Graph: Learning Latent Landmarks for Planning, ICML 2021.
[2] Li et al., Learning Subgoal Representations with Slow Dynamics, ICLR 2021


questions: The assumption of oracle Environment Variables is a little strong. Could the proposed method generalize to more common settings where no oracle Environment Variables are available?

The blue learning curve in Figure 4(c) has not been plotted to 40000 training steps. Will it converge steadily at the end of the learning?

limitations: The authors have discussed the limited application to environments with discrete environment variables in Section 7, and have not touched on another limitation of requiring oracle environment variables. Those two limitations are related.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a formulation of hierarchical reinforcement learning that utilizes causal structure between environmental variables to generate subgoals. The discovered subgoals and lower-level policies generated to reach the subgoals are subsequently adapted using a learnt hierarchical agent that solves downstream tasks. The authors test their method in addition to several hierarchical RL baselines on the 2D-Minecraft and Eden environments and obtain promising results on the both. They show faster convergence in both environments in addition to interpretable causal graphs. 

strengths_and_weaknesses: Strengths:
1. The problem considered by the authors is one of strong relevance. 
2. The approach is principled with well-made figures. 
3. The results are repeated across seeds and are in challenging environments. 

Weaknesses
1. Directionality: It's unclear how the algorithm learns directionality for causes and effects. From my understanding, the interventional data is generated by lower-level policies in the action space that attempt to set the environmental variables to a particular value. Thus, it appears the order in which environment variables are selected matters. What if a variable is selected in the initial iterations for which the causal parents are discovered in a later iteration? Is the causal graph updated in such a setting? For example, if the agent decides to learn policies that intervene on Ironore before it intervenes on stonepickax, what happens then?
2. Commentary on Scalability of during large number of EVs: I believe that the authors need to add some information about the scalability of their approach with environmental variables. In the current draft, they say “ to verify whether newly associated variables are controllable as supposed, we compare the final training success rates of the new subgoals with a preset threshold phi before adding them to the subgoal hierarchy”. How are they selected? How many are selected? How are the lower-level policies learned? I would imagine through equation (4), but how long do you train, etc should be added. 
3. The algorithm should not be relegated to Supplementary work. This draft sorely misses the details afforded by it and would help ground a lot of the description. 
4. Some assumptions on the density of the underlying causal graph: I believe the authors need to state some assumptions about the density of the underlying causal graph, in particular, that currently, they assume that the underlying causal graph is dense and that each of the environment variables influences the others. I can imagine a situation where mechanisms are sparse and that learning policies which intervene on a causal factor only allows one to intervene on the immediate children of the same causal factor, but these children no longer have any other effect variables linked to them.   
5. Description of Subgoal Hierarchy needed: What is a subgoal hierarchy? From reading the paper several times, I was able to discern that is an ordering on the elements in the goal space that are linked due to their temporal dependencies, i.e., getting IO is a child of SP because SP is needed to mine IO. Some mathematical precision is necessary to describe this especially since it forms the core of your algorithm. Compressing section 5 by avoiding newlines will generate room enough for such an explanation. 
6. Assumptions on the utility of causal graphs for subgoal: A central assumption made by the authors is that in their setup, nodes on the causal graph are all useful in goal generation. This is in general not true and worth stating. For example, manipulation by a robotic hand of an object in its palm does not require knowledge of friction coefficients between the block and the ground.
7. Curious that the oracle is outperformed by your method. Is this a bug? Should it not be that the oracle with ground-truth causal graph inputs beat discovery?
8. Missing citing work: https://arxiv.org/abs/2010.03110 and https://proceedings.neurips.cc/paper/2021/file/c1722a7941d61aad6e651a35b65a9c3e-Paper.pdf and https://ieeexplore.ieee.org/document/9561439

questions: See the Weaknesses section for my questions. I believe adding incorporating your answers to the draft would be beneficial.  
limitations: See the Weaknesses section for my analysis of the assumptions/limitations of the work. I don't believe there is any cause for concern for negative societal impact.
review_title: null

Meta-review: metareview: After a strong rebuttal from the authors and an extensive discussion among the reviewers, I believe the paper's pros outweigh its cons and this paper will be a valuable contribution to NeurIPS. I recommend it for acceptance and encourage the authors to address the reviewers comments for the camera-ready version of the paper - specifically please add the posted clarifications and experiments from the rebuttal and justify the limitation of available environment variables.

confidence: Certain
Decision: Accept