Causality-driven Hierarchical Structure Discovery for Reinforcement Learning
Authors: Shaohui Peng
Xing Hu
Rui Zhang
Ke Tang
Jiaming Guo
Qi Yi
Ruizhi Chen
Xishan Zhang
Zidong Du
Ling Li
Qi Guo
Yunji Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: hierarchical reinforcement learning
causal discovery
causalty
subgoal

Abstract: Hierarchical reinforcement learning (HRL) has been proven to be effective for tasks with sparse rewards, for it can improve the agent's exploration efficiency by discovering high-quality hierarchical structures (e.g., subgoals or options). However, automatically discovering high-quality hierarchical structures is still a great challenge. Previous HRL methods can only find the hierarchical structures in simple environments, as they are mainly achieved through the randomness of agent's policies during exploration. In complicated environments, such a randomness-driven exploration paradigm can hardly discover high-quality hierarchical structures because of the low exploration efficiency. In this paper, we propose CDHRL, a causality-driven hierarchical reinforcement learning framework, to build high-quality hierarchical structures efficiently in complicated environments. The key insight is that the causalities among environment variables are naturally fit for modeling reachable subgoals and their dependencies; thus, the causality is suitable to be the guidance in building high-quality hierarchical structures. Roughly, we build the hierarchy of subgoals based on causality autonomously, and utilize the subgoal-based policies to unfold further causality efficiently. Therefore, CDHRL leverages a causality-driven discovery instead of a randomness-driven exploration for high-quality hierarchical structure construction. The results in two complex environments, 2D-Minecraft and Eden, show that CDHRL can discover high-quality hierarchical structures and significantly enhance exploration efficiency.

Causality-driven Hierarchical Structure Discovery for Reinforcement Learning
Shaohui Peng 1,2,3
Xing Hu'
Rui Zhang! Ke Tang*
Jiaming Guol22 Qi Yi1.66 Ruizhi Chen Xishan Zhang".
Zidong Du Ling Li25 Qi Guo Yunji Chen
SKL of Processors, Institute of Computing Technology, CAS
University of Chinese Academy of Sciences *Cambricon Technologies
*Department of Computer Science and Engineering, Southern University of Science and Technology
SKII. of Computer Science, Institute of Software, CAS University of Science and Technology of China
(pengshaohui 182, huxing, zhangrui, guoj 1am]ing188. zhangxishan, duzidong, guoqi, cyj) ac en, tangk30sustech. edu. cn, yiqi@mail ustc. edu, en, fruizhi, liling)0iscas.aet cn
Abstract
Hierarchical reinforcement learning (HRL) effectively improve agents' exploration efficiency on tasks with sparse reward, with the guide of high-quality hierarchical structures (e.g., subgoals or options). However, how automatically discover high- quality hierarchical structures is still great challenge. Previous HRL methods can hardly discover the hierarchical structures in complex environments due to low exploration efficiency by exploiting the randomness-drivee exploration paradigm. To address this issue, we propose CDHRL, a causality-driver hierarchical rein- forcement learning framework, leveraging a causality-driva discovery instead a randomness-ding driven exploration to effectively build high-quality hicrarchical structures in complicated environments. The key insight is that the causalities among environment variables are naturally fit for modeling reachable subgoals and their dependencies and perfectly guide to build high-quality hierarchical structures. The results in two complex environments, -D-Mieccraft and Eden, show that CDHRL significantly boosts exploration efficiency with the causality driven paradigm.
Introduction
Reinforcement learning (RL) enables the intelligence of the agents by learning to take actions in the environments for maximum rewards [32, 23, 21]. For complex tasks with large state spaces and sparse delayed rewards [14], hierarchical reinforcement learning (HRL) extends RL methods' successes by discovering beneficial hierarchical structures, which helps the agent train multiple levels of policies and explore efficiently using high-level actions [26].
The major challenge HRL methods is how to discover high-quality hierarchical structures (e.g., subgoals, skills, options) A straightforward method is to manually define the hierarchical structures based on task-specific prior knowledge. However, in observing that the hand-crafted
Corresponding author:
In this paper, the hierarchical structure refers to the subgoal hierarchy in the environment.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

hierarchical structures require human expertise and yield in low generality, some researchers propose methods to discover subgoals automatically [24, 19, 37, 25]. These methods focus on how to discover subgoals from experience but follow an inefficient randomness-driven exploration paradigm to obtain experience. Specifically, the agent explores the environment through the randomness of the policy, which is implemented by raising the entropy of the policy or adding random noise to actions, and then discovers subgoals from the collected experience. Randomness-drives exploration paradigm hardly discovers the high-quality hierarchical structure in complicated environments, such as Minecraft, because of low exploration efficiency, thus limiting the HRL method's performance.
To address the above issue, we propose a Causality- Driveen Hierarchical Reinforcement Learning (CDHRL) framework which leverages the causality in the environment as the guidance to discover high-quality hierarchical structures. The key insight is that causality in a system, which explains the dependencies among objects, states, events, or processes, can be naturally used to describe final goal with sub-goals structurally. Therefore, in CDHRL, we use the causalities among environment variables to guide the discovery of the subgoal hierarchy. Such causality-drive paradigm avoids the inefficient random exploration process, and significantly enhances the efficiency of discovering hierarchical structures in complicated environments. Moreover, the generated high- quality hierarchical structure, which consists of subgoals and their dependency, is more instructive and efficient during training than the traditional fully- fully-connected hierarchical structure.
In implementing CDHRL, we utilize an iterative boosting framework to progressively discover the causality and construct the hierarchical structure, since it is difficult to discover all causalities in complicated environments directly. The proposed CDHRL consists of two processes, causality discovery and subgoal hierarchy construction. Specifically, causality discovery refers discovery of causality between environment variables. In subgoal hierarchy construction, we find reachable subgoals, which are changes of controllable environment variables, and leverage causality to construct dependency of subgoals. The above two processes can iteratively boost each other: On one side, we can transform discovered causality between environment variables into the subgoal hierarchy. On the other side, we can utilize trained subgoal-based policies to enhance the efficiency of causality discovery. Taking a simplified example in Minecraft play, stonepickaxe (SP) and ironore(IO) are enviromment variables. After the agent learned subgoals of getting SP, CDHRL could control SP's distribution to discover causality from SP to IO, which indicates that the number of stonepickaxes affects the probability of acquiring ironore. Then, CDHRL adds subgoals about the effect variable 10 to the subgoal hierarchy and trains corresponding subgoal-based policies. After that, we in turn discover further causality with I0 as cause efficiently by utilizing subgoals about IO to change IO's distribution and collect intervention data. Such a causality- -nriaent--driven discovery paradigm makes the two processes, causality discovery and subgoal hierarchy construction, iteratively promote each other.
We verified our method in two typical complex tasks, including 2d-Minecraft [30] and simplified sandbox survival games Eden [4]. The results show that CDHRL discovers high-quality hierarchi- cal structures and significantly enhances exploration efficiency. Compared to the state-of oftte-of-thh-ar the HRL method HAC [19] and HRL enhanced by curriculum learning [27], our method significantly outperforms existing studies in terms of both final performance and learning speed.
2 Related Works
2.1 Hierarchical Reinforcement Learning
Discovery hierarchical structure plays an essential role the Hierarchical Reinforcement Learning (HRL) algorithm. Some methods reduce the difficulty of discovering the hierarchical structure by adding artificial prior, such as sub-tasks dependency graph [30] and manually defined subgoals [10, 33]. Others may manually restrict the form of hierarchical structure to reduce the search space [12, 11, 6]. By introducing prior information, these methods perform well on specific tasks. However, it not only requires solid expert knowledge but also sacrifices versatility. Our work does not introduce downstream task-specific information, and thus has a broader application.
Some researchers propose methods to discover subgoals automatically. Vezhnevets et al. [34] and Nachum et al. [24] restrict the goal space as state space and train goal-based hierarchical policy end-to-end. Levy et al. [19] enable the multi-level hierarchical structure with hindsight corrections. Some methods discover hierarchical structure by optimizing additional objectives. Zhang et al. [37] assume that the subgoals should be near to the current state. Nachum et al. [25] claim that the optimal

---

policy based on subgoals should have minimal loss than that based on primitive actions. Pitis et al. [27] consider the training progress for finding the next desired subgoal. Existing methods focus on discovering subgoals from data that is randomly collected. They are often applied in easy-to-explore environments like Maze [2] or Robot [9], but hard work in complicated environment due to the exploration paradigm. CDHRL follows the causality-drivee exploration paradigm, which is more suitable for complicated environments.
low exploration efficiency of their randomness-drivee
2.2 Causal Reinforcement Learning
Causal reinforcement learning is a research direction that combines the causal effect with reinforce- ment learning. Pitis et al. [28] leverages local causal structures to improve the sample efficiency of off-policy RI. Méndez-Molina et al. [22] makes a trade-off between exploration and exploitation based given causal graph. Corcoll and Vicente [7] distinguishes the controllable effect of the agent by conducting counterfactual detection. Sontakke et al. [31] and Seitzer et al. [29] discover simple causal influences to improving the efficiency of reinforcement learning. Guo al. [13] learns the causality between the hindsight effect variables and estimated values to decrease the variance in the policy training phase. Most of them utilize predefined causality graph as prior knowledge, or detect one-step causality to enhance RL method. However, none of them automatically discover complex causality graph in the environment to guide the exploration of hierarchical structure. Causal discovery methods [15, 16, 18] discover causality between variables through the data, which is an important issue in causal learning. Our method introduces a causal discovery-based exploration dramatically enhances the discovery efficiency of hierarchical structures in complex environments.
2.3 Relational Reinforcement Learning
Relational Reinforcement Learning (RRL) represents entities, actions, and policy by relational language to combine relation learning or reasoning with RL[8], which shares similar motivation with our method. Zambaldi et al. [36] adopt self-attention mechanism upon entities-centric representation to implements relation learning... Wang et al. [35] learns particular relations using a graph neural network to promote zero-shot performance on continuous control tasks. Chitnis et al. [5] and Kokel et al. [17] both learn relation model and planner in first-order language described environment. RRL aims combine relation induction and reasoning with RL to promote policy generalization, which often works in structural environments with clear definition of logical predicates. Our method adopts causality, which more simple and common than logical relation, to guide the direction of subgoals discovery and thus enhance its efficiency in complicated environments.
3 Preliminary
3.1 Subgoal- Markov Decision Process (MDP)
Hicrarchical reinforcement learning that adopts subgoals as hierarchical structure extends the MDP framework by introducing the subgoal space G. Subgoal-based MDP a six-tuple, (S, A, 7). including state set S, subgoal space G, and an action set A. The transition probability function T(s, a, s') represents the probability of transition from & to s' with executing action a. r(s, g. a, s') represents the goal-reaching reward function that indicates whether the agent achieves the subgoal g in transition (s, a, s'). And y is the discount rate. The target of reinforcement learning is to find subgoal-based policy π S,g + A that maximizes the accumulated rewards (x,(8))) g) E E| 7"h Retn+11 8, ft g). Our proposed framework aims to pre-train the subgoal- policy for downstream sparse reward tasks.
3.2 Causal Discovery
Causal discovery aims to infer causality through variables data under causal modeling and assump- tions. There are two key parts of it: how to represent causality and how to obtain the data for discovering causality.
Structural Causal Model (SCM) is a kind of model that describes the system's 's causality, SCM over a finite number M of random variables X; includes a directed acyclic graph (DAG) C' E (0, 1) M as the causality graph, and a set of generating functions fr of variables:
X, f((Xpa(t.C),.Ni N.), Vi {0..... M 1}.
(I)
3

---

A. Causality Discovery
C. Intervention Sampling
FV Causality
EV-based Causality Graph (EVCG)
Learn from Intervention Data
Zany
(l0)
Others
controllable variables using
Environment Variables
EV-based Subgoals
(EV)
7
71
B. Subgoals Hierarchy Construction
Reachablc Subgoals
2. Collect invention data further Causality Discovery
Causality Graph guides generate hierarchy
EV EV-based Subgoal Hierarchy
Subhoal Dopendoacy
Unreachable Subgoals
W-based Goal Space (EVGS)
Figure 1: Overview of CDHRL framework. A. Discover causality between Environment Vari- ables (EVs): Taking EV in Minecraft as a simplified example, we discover the causality between StonePickaxe (SP) and IronOre (IO) from SP's intervention data. B. Build EV-based subgoal hier- archy for HRL subgoal training. Guided by causality from SP to IO, we add reachable subgoals of increasing/decreasing IO values the subgoal hierarchy and train corresponding subgoal- based policies. C. Obtaining intervention data for causality discovery in the new round. After getting subgoal-base -bssed policies about IO, the agent can further change IO's distribution and collect inter- vention data required in causality discovery (Step A) of the next iteration. (The pscudo-code of the framework is in Appendix B.1)
Gj indicates that Xj is the parent of X,, which means that X, is the direct cause of X,. Mj the jointly-independen noise and Pa(i, C) is the collection of the parent nodes (direct cause variables) of X, in causality graph C.
Intervention sampling is typical operation in causal discovery. Unlike standard sampling, it sets the distribution of certain variables in the system as fix value uniform distribution, then samples to get intervention data. Intervention data has more causal information than observation data. In RL environments, we cannot do intervention directly because the variable generation mechanisms in the environment are unavailable. However, since the agent's 's behavior can affect the state of the environment, we train subgoal- based policies in CDHRL that can increase or decrease the values of variables to conduct the intervention.
4 Causality-Driven Hierarchical Reinforcement Learning
In this section, we present our Mousalliy-Driveen Hierarchical Reinforcement Learning (CDHRL) framework that leverages causality of Environment Variables for high-efficient hierarchical structure discovery. Figure 1 shows the architecture of CDHRL. To enable the causality discovery and the subgoal hierarchy construction to promote each other, we first adopt environment variables to build causality- -bsed hierarchical structure that bridges causality discovery and hierarchical reinforcement learning (Section 4. Then, we provide the causality discovery methodology in the context of MDP (Section 4.2 & Section 4.4). Additionally, propose the subgoal policy training methodology based on the causality-bart based hierarchical structure (Section 4.3).
4.1 Causality-based Hierarchical Structure
To bridge causality discovery and subgoal hierarchy construction, and to make them collaboratively promote each other, we introduce the main components of these two processes, causality graph and goal space, both based on environment variables. The Environment Variables (EV) are disentangled factors of the environment observation, which can include noise or lack some key information the true environment state. In other words, EV does not mean the perfect abstract representation of the environment state that the experts provide. Obtaining EV relatively convenient for environments providing state vector- based observation since information of different dimensions is disentangled.

---

For environments with purely image-based observation, finding EV has been largely studied, like CausalVAE [9] and DEAR [10]. Environment Eden and Mi-Minecraft both offer state vector-based observation. Considering that CDHRL focuses focuses on the hierarchical structure discovery guided by the
causality, we assume that EV can be obtained from observation by a function, O : S + X, which intercepts part of observation as EV vector. We also clarify EV in detail and conduct sensitivity analysis on EV in Appendix A to verify the reasonableness of the function O.
Environment Variable- Causality Graph (EVCG). The causality graph describes the sys- tem's 's causality through a directed acyelic graph (DAG), where the source nodes of directed edges represent cause variables and destination nodes represent effect variables. The causality graph is shown in Figure 1.A. The process learns the environment variable-based causality graph (EVCG) of the RL environment, where nodes consist of environment variables and edges represent causality. Environment Variables are key factors of environment state and the fundamental element of causality. Taking Minecraft as an example, environment variables consist of the number of tools (like StonePick- axe), materials (like IronOre), and surrounding objects (like Pigs). The causality represented by the arrow from StonePickaxe to IronOre means that the number of stone pickaxes the agent-owned affects the probability of acquiring iron ore.
Environment Variable-based Goal Space (EVGS). To leverage the discovered causality between the environment variables, we introduce environment variable- baasable-aaedddee goal space (EVGS) and construct the subgoal hierarchy upon EVGS, as shown in Figure I.B. Different from prior studies that adopt the state or latent spaces as subgoal space [19, 25]. EVGS consists of goals which are to change the value of environment variables. Thus the -oogoal--bsedd policies can be used to change the distribution of environment variables or complete downstream tasks. Formally, we introduce two fundamental changes, Increase (F;) and Decrease (Fa), for variable values:
F (( = 11,,0+12011
(2)
Fa(21,, = -1,,e+1<0,.."
where Xa. is the value of X; in state st. The EVGS is the Cartesian product of the environment variable set X' and the changes set Fehange = (Fi, Faj:
GEVaS =
E F, Fehange]:
(3)
The corresponding goal-reaching function r(se. 9. a, 8141) is: r(st, (X,, F), a, 8141) F(O(s)); ((141))),
(4)
where O is the mapping from state to variable values.
Collaboration between Causality Discovery and Subgoal Hierarchy Construction. With the EV-based causality graph and EV-based goal space, we can naturally transform the causality graph into a subgoal hierarchy. The subgoal hierarchy contains reachable subgoals selected from GEVGS through the causality graph and are organized by the causality between variables. Step A and B in Figure show an example. Specifically, we divide the environment variables into controllable and unknown ones for the agent. The controllable variables are variables whose related subgoals have been added into the subgoal hicrarchy (green ones, c.g., StonePickaxe) so that the agent can change their distribution to obtain their intervention data. Hence, in each iteration, the agent can discover causality from controllable variables (e.g.. StonePickaxe) to the unknown variables (e.g.. IronOre). Then, some unknown variables, which as the effect of the causality, are considered as new controllable variables for the agent, because the agent's 's behavior can affect them and thus recognize their cause variables. The subgoals related to new controllable environment variables are considered to be reachable. We then train reachable subgoal- subgoal hierarchy. In the next iteration, the agent can change the distribution of new controllable environment variables through subgoal-based policies and discover new causality which takes them as the cause variable. The iteration converges when there is no new causality, and the remained isolated variables are uncontrollable for the agent. Subgoals of uncontrollable variables will not be added to the subgoal hierarchy.
policies and add them to the
In this way, causality discovery and subgoal hierarchy construction can promote each other. On the one hand, the discovered causality between environment variables guides the agent to find available subgoals accurately and construct the subgoal hierarchy upon EVGS. On the other hand, the agent can utilize trained policies of subgoals to do intervention environment variables. More complex causality between variables can be discovered along with the sampling data from poor to rich. The alternating iterative optimization above can address the challenge of discovering hicrarchical structure.
5

---

4.2 Causality Learning
Causality in the Context of Markov Decision Process (MDP). We first introduce two key points of causality discovery in the context of MDP: what causality to discover and how to get learning data. 1) Causality is discovered within adjacent steps. According the properties of MDP, the state 81+1 is only determined by state 8, and action de- Hence, we redefine the causality equation I as X1,++1 == fe(Xpots (o(Xxo(s.)).. N:), which means the variable values at step + are only affected by that at step 1. In other words, we aim to learn the causality from transition data of adjacent steps in the agent's trajectory. 2) Intervention data is obtained through subgoal-based policy. As mentioned in the preliminary, it is hard to do intervention in the RL environment directly since the generation mechanisms are unavailable. Therefore, we control the variables' distribution through the agent's s behavior to sample intervention data for causal discovery. Specifically, with the application of subgoals in EVGS, the agent can change the distribution of controllable environment variables and collect the intervention data subsequently. More details about the intervention process based on subgoal-base policy are introduced in Section 4.4.
Structural Causal Model Learning. The structural Causal Model (SCM) is kind of model that describes the system's ss causality. We adopt a causal discovery methodology similarly to SDI [16] and intervention data collected from the agent's 's trajectories to learn the environment variables' SCM. As the equation shown, SCM over M variables can be described by two sets of parameters: structural parameters 7) E Rtdd x Ad model the causality graph C' and functional parameters Os model the generating functions f.
The parameters 17 and 9 are optimized through a two-phase iterative and continuous process. In the first function learning phase, we want to fix the input variables of generating function and optimize the functional parameters 0. So we sample hypothesis configuration C' of the causality graph from Bernoulli distribution (ar(()())) where represents the Sigmoid function. And then, we optimize 0 by maximizing the likelihood of the intervention data. In the second structure learning phase, we want to learn the causality graph under fixed generating functions f. We estimate the gradient of n through fixed 0 and REINFORCE-like predictor proposed by Bengio et al. [3]. For the causality from variable X, to X,, the gradient gij of miy is estimated from X3 's intervention data set {X} } by the following formula:
I (x) (x) (
Pij E(o(ns) c33
Vi E (O..... M 1}.
(5)
where k represents the k-th draw of hypothesis configuration C. (.) represents the X likeli- hood under the intervention data, The two optimization phases cycle until convergence. (Algorithm details are in Appendix B.2).
4.3 Subgoal-based Policy Training
As described in section 4.1, when new causalities are discovered, the subgoals related to new effect variables in the EVGS are added to the subgoal hierarehy by the guidance of new causalities. Figure 1.B shows the effect variable IO's 's subgoals are added to the subgoal hierarchy. Specifically, we first assumed the new effect variable X, is controllable, and subgoals of X, are reachable. We then train X₃ 's subgoal-based policies т r[a]s, ss (X2, F)). To induce an efficient training, the action space of π (al [s, (X,, F)) consists of subgoals related to X,'s cause variables and primitive action set A:
E F')X,, E Xmm(i.C). F' E Fehangel } UAA
(6)
Besides, to verify whether newly associated variables are controllable as supposed, we compare the final training success rates of the new subgoals with preset threshold deaueal before adding them to the subgoal hierarchy. When the corresponding subgoal-base based policies are successfully trained, variables will be confirmed controllable variables and can be intervened to obtain data in the next iteration. Along with the causality discovered from shallow to deep, the subgoal-based policies are trained layer by layer and finally consist of a multi-level hierarchy. After the above process converges to no new causality, subgoals related to remaining isolated variables are taken as unreachable ones. For details subgoal-based policy training, please refer the training algorithm in Appendix B.3.
Distinguishing reachable subgoals related to controllable environment variables and filtering un- reachable ones during policy training can reduce the goal space reasonably. Meanwhile, through

---

the dependency of subgoals suggested by causality between environment variables, we construct the action space for the subgoal-based policies, which more efficient than primitive action space. The above two improvements in policy training lead to more data-efficient learning process, verified in experiment results in Section 6.2).
4.4 Intervention Sampling
Considering that environment variables in RL environments cannot be directly intervened, we propose methodology that adopts subgoal-basee policies to obtain the intervention data. Benefit from variable-based goal space, the subgoal-bast -asse policy could effectively change the distribution of controllable variables.
The detailed intervention sampling includes the following steps: (1) For controllable variable X;, sample the desired value from the uniform distribution, and then use subgoals (X;, F) to drive the agent to change the value of X; to F;; (2) Collect the follow-up trajectory until X;'s value changes, and then obtain intervention data of X2 from all variables' values of adjacent steps. At the beginning of the optimization, there are no trained sabgoal-basee policies. Therefore, we set the agent's action as one of the variables first and do intervention on it to initiate the iterative process.
5 Experiment Setup
Environment To verify the effectiveness of CDHRL, we choose two environments with discrete action space, namely 2d-Minecraft [30] and Eden [4] (more details of environments are introduced in Appendix C.1. They have complex hierarchical structures and very sparse rewards. 2D-Minecraft [30] is a simplified 2D version of the famous Minecraft [14]. In one episode with limited steps, the agent must navigate the map, pick up various materials, and craft tools until obtaining the diamond. The agent only can get a positive reward after getting the diamond. Eden [4] is a survival game which is similar "Don't 't Starve." To make living in 40 40 grid world, the agent with 8 x 8 limited vision range must chase animals to obtain food for maintaining satiety, find rivers to get water for preventing being thirsty, and collect materials to craft tools for surviving in the night. The agent takes survival as the unique target and cannot get any positive rewards except a negative reward after it dies and ends the episode.
Implementation We implement the agent based on multi-level DQN with HER [1], considering the action space is discrete. The training is divided into two stages, pre-training and adaptation. In the pre-training stage, we train multi-level subgoal-based policies until no new causality is discovered. In the adaptation stage, we train an upper controller the pre-trained subgoals maximize the task reward. The SCM is implemented like SDI [16]. The environment supports the function O from environment state variable values.
We compare CDHRL with the following three methods for validation. For a fair compar- ison, we use the environment variable-based goal space GEVGS to replace the goal space in these algorithms to ensure that the performance promotion is because of the difference in hierarchical structure discovery methods. Oracle HRL (OHRL) is implemented and an oracle goal space. The goal space is subset of Ggv GS after artificial eliminating unreachable and useless subgoals. HAC [19] a powerful condiionnt HRL that discovers subgoals with a randomness- driven exploration paradigm. We implement a two-layer HAC with subgoal space GEvaS- LESSON [20] is modified coal-conditioned HRL method based on HAC that discovers subgoals from slowly changed features. MEGA [27] is a kind of goal-conditioned HRI enhanced by curriculum learning, which pre-train subgoals in the order of their training progress. We set the initial subgoal distribution as the uniform distribution on GEves and pre-trains subgoals with enough steps. After that, like the adaption stage in CDHRL, we train an upper controller on the trained subgoals.
Baselines
two-level DQN with HER
Results
We first carry out extensive experiments to compare the performance of CDHRL and the state- of-the-arts in complex environments with hard-to-explore hierarchical structure. Then, we dig deep ablation studies to show why CDHRL triumphs over the existing HRI. paradigm. At last, we showcase the discovered causality-based hierarchical structure, which can be reasonably interpreted.
7

---

1000
800
HAT
v/ood
MEGA
MEGA
600
LESSON
400
Shore
Iron piçazzo
200
0.0
5.
Training
0.0
Training steps
2.0
Damond
(a) Eden
(b) 2D-Minecraft
(c) Causality Graph in Minecraft
Figure 2: (a) Agent's 's survival time in Eden. (b) Success ratio in 2D-Minecraft. The vertical dotted lines indicate the end of pre-training of CDHRL and MEGA. Results are derived from average data in 8 trials. (c) The causal graph of 2D-Minecraft discovered by the agent. Some uncontrollable variables that unlinked are ignored here.
Harder
Get
MEGA
Get
IronPickaxe
Itere
Gettron
Itere3
COHRL
Iter=1
Get
Eacior
StorePickupe
Get
Stone
2000
4000
Occurrences
Figure 3: Exploration capability comparison. We select five explore milestones in 2D-Minecraft from easy-to-explore to hard-to-explore and record the occurrences 10K test episodes to compare the agent's 's exploration capability. More occurrences higher exploration capability. (a) CDHRL's exploration capability iteratively increases along with the construction of hierarchical structures. (b) CDHRL shows much better exploration capability than HAC and MEGA. All compared methods are tested after trained 800K steps.
the hard-to-ewt toxp-oo-e milestone represent
6.1 Comparative Analysis
We compare the performance on Eden and 2D-Minecraft of CDHRL with other methods. Figure 2 shows that CDHRL learns faster and achieves the best performance even though all the evaluated methods have the same subgoal space. Concretely, OHRL, HAC and LESSON do not have the pre-training stage and start learning directly. However, the success ratios of OHRL, HAC and LESSON in the first 1e6 steps of 2(b) keep nearly trajectory data of getting diamonds. CDHRI. learns much faster even if there is pre-training time overhead. MEGA has the pre-training stage for eurriculum learning to identify the near-term and long-term subgoals. However, it fails to identify the effective hierarchical structures in these two environments with relatively large sub-goal space. The hierarchical structure discovered by CDHRL not only significantly enhances the agent's exploration capability in sparse reward, but also identifies the reachable subgoals that promotes the learning speed of subgoal-base -aasedl-based policies. We will explain these phenomenons further in Section 6.2.
since they seldomly obtain the success
6.2 Insights
Further, we evaluate the detailed exploration capability and causal discovery efficiency and show how CDHRL outperforms existing HRL paradigms. We have two observations from the experiment results: 1) The causality-driven exploration paradigm enhances the exploration capability significantly. We experimentally evaluate the agent's exploration capability in 2D-Minerraff based on the achieved times of cxploration milestones in the environment. Figure 3(b) shows the exploration capability comparison of different methods. The CDHRL's achieved times of hard-to-cxplore milestones much higher than HAC and MEGA, indicating a much better exploration capability of CDHRL. Figure 3(a) shows the exploration capability increases along with iterations of causality discovery in CDHRL. The reason is that the agent's causality-drive driven exploration becomes more efficient as more subgoal-based policies and controllable variables, which other HRL methods cannot achieve. In conclusion, with the help of the causality-base based exploration paradigm, the agent can achieve better exploration efficiency.
2) The subgoal hierarchy construction and causality discovery iteratively promote each other.
On the one hand, the hierarchical structure makes the causality graph more accurate. We train the

---

LO
HANTROPIME Dia Mirreoo
0.0
0.8
a.G.
0.4
Remmisca peltyy randam 23003n
Headiaal
02
Weatiemene Seforofrrrese
Sarcor usion
Traningg Ltarations
Traniig taraaioos
Training Stops
Training Stops
(a) SHD
(b) SID
(c) CDHRL
(d) MEGA
Figure 4: (a) Structure Hamming Distance (SHD) and (b) Structural Interventional Distance (SID) between learned and ground-truth causality graph. The intervention data sampled by the assistance of hierarchical policy make the causality graph more accurate. The ground-trut truth causality graphs of Eden and 2D-Minecraft are in Appendix C.2.(c) and (d) are learning curve of two subgoals.
causal model in 2D-Mineccraft using intervention data collected by hierarchical policy and random action, respectively. We record the causality graph of each iteration and compute their Structure Hamming Distance (SHD) and Structural Interventional Distance (SID) relative to the ground truth, where SHD and SID are typical metrics for the accuracy of causality graphs. The lower distance means the more accurate causality graph. Figure 4(a) and 4(b) show the result. The accuracy of the causality graph learned using the hierarchical policy is much better than using random action, especially for SID. Such results indicate that utilizing the hierarchical policy to obtain intervention data is essential for causality discovery.
On the other hand, the construction of the subgoal hierarchy is more efficient and reasonable. We com- pare the convergence speed and stableness of CDHRL and MEGA to achieve two subgoals in Eden. For these former is the prerequisite goal of the latter one and should be learnt by agent first. Figure 4(c) and 4(d) show that CDHRL achieves faster and more stable learning process than MEGA. We attribute the poor performance of MEGA to the fact that it cannot identify the dependency of the subgoals and is unable to learn subgoals in order, resulting in much lower performance, especially when the environment is complex with complicated dependency of subgoals. In contrast, since X Meat is the cause variable of Saarttty. CDHRL infers the learning order of subgoals based on discovered causality and achieves stable and fast learning progress. Additionally, the reduced action space of multi-level policy transformed from causality also improves training efficiency.
subgoals, Matt ncrease = (X Meatt F:) and SatictyIncress
(X Satietys Fj), the
6.3 An Showcase of Causality- Hierarchical Structure
We showcase the established hierarchical structure based on discovered causality in 2D-MineCraft from the causality graph view. The learned causality graph can be reasonably interpreted. As shown in Figure 2(c), most of the causality the graph meets with human cognition. We also find an interesting phenomenon that CDHRL sometimes may discover long-term causality and ignore corresponding short-term causality, e.g. "Stick" to "Iron re" (green edge) instead of "Stick" to "Stone pickaxe" to "Iron ore" (red edges) in Figure 2(c). may benefit from such phenomenons because cutting unimportant causality details can improve the training speed and the generalization of hierarchical structure. Although there is some deviation between the discovered and the ground -outh causality graph, the causality- based hierarchical structure still improves the exploration efficiency.
7 Conclusion
Hierarchical reinforcement learning methods can hardly work well in complicated environments because of the low exploration efficiency of the randomness-driven exploration paradigm when discovering the hierarchical structure. We propose a causality-driven hierarchical reinforcement learning framework (CDHRL) to address the challenge, which autonomously builds high-quality hierarchical structures in an iterative boosting way. In two complicated environments with very sparse rewards, 2d-Minerraft and Eden, our method discovers a high-quality subgoal hierarchy and significantly enhances exploration efficiency. Limited in causal discovery algorithm capability and the complexity of variable changes set, our method can only be applied in environments with discrete environment variables. Moreover, CDHRL needs an extra disentangled encoder in purely image-basee -baee environments for finding environment variables. We leave broadening the application scope of CDHRL as our future work.

---

Acknowledgements
This work is partially supported by the National Key Research and Development Program of China(under Grant (013AA0003300),, the NSF of China(under Grants 61925208, 62002338, 62102399, U19B2019, 61732020), Beijing Academy of Artificial Intelligence (BAAI), CAS Project for Young Scientists in Basic Research(YSBRR Youth Innovation Promotion Association CAS and Xplore Prize.
References
[1] Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In NIPS, 2017. URL aSSfeddddaaaaaaaaaaaaaaaaafffff3
neurips...... wttps/cpppppp//////////// / Abstract. html.
[2] Akhil Bagaria and George Dimitri Konidaris. Option discovery using deep skill chaining. In ICLR, 2020.
[3] Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Nan Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher J. Pal. A meta-transfer objective for learning to disentangle causal mechanisms. In ICLR, 2020. URL https: httpa/:////p/ee///eeeeeeeee net/forum? CderyxWIgBFPS.
[4] Ruizhi Chen, Xiaoyu Wu, Yansong Pan, Kaizhao Yuan, Ling Li, Tian Yun Ma, Jiauan Liang, Rui Zhang, Kai Wang, Chen Zhang, Shaohui Peng, Xishan Zhang, Zidong Du, Qi Guo, and Yunji Chen. Eden: A unified environment framework for booming reinforcement learning algorithms, 2021.
[5] Rohan Chitnis, Tom Silver, Joshua B. Tenenbaum, Leslie Pack Kaelbling, and Tomas Lozano- Perez. Glib:
Efficient exploration for relational model-based reinforcement learning via goal- literal babbling. In AAAI, 2021.
[6] Caleb Chuck, Supawit Chockchowwat, and Scott Niekum. Hypothesis-driven skill discovery for hierarchical deep reinforcement learning. JEEE/RSJ International Conference on Intelligens IEEE, 2020. doi: 10. 0001109/RROS455444200009999009 .....9020220 9340891.
Robots and Systems, pages
URL https: //to:p:::///..oo org/10 10.
[7] Oriol Corcoll and Raul Vicente. Disentangling causal effects for hierarchical reinforcement learning. CoRR, abs/2010.0135/ 2020.
[8] Saso Dzeroski, Luc De Raedt, and Hendrik Blockeel. Relational reinforcement learning. In Jude W. Shavlik, editor, Proceedings of the Fifteenth International Conference on Machine Learning (ICML 1998), Madison, Wisconsin, USA, July 24-27, 1998, pages 136-143. Morgan Kaufmann, 1998.
[9] Benjamin Eysenbach. Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. ArXiv, abs/1802.06070, 2019.
[10] Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchi- cal reinforcement learning. In ICLR, 2017. URL httpps///qparreiiee.neett/ wwtp::wwwwwwwewtwmmn/e..m B1oK8aoxe.
[11] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. /CML, volume 80 of Proceedings of Machine Learning Re- search. PMLR, 2018. URL http://tt:/eeee httt:::/procceednn.. mlr.. wwt.:mml..pppaeee.e wwwppsee00/////ep weso//fforessse htmml........m.........mm.mmmm.m
[12) Sébastien Forestier, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. CoRR, /t////08.021900 2017. URL http::///axxvv.or/abbbo
[13] Jiaming Guo, Rui Zhang, Xishan Zhang. Shaohui Peng, Qi Yi, Zidong Du, Xing Hu, Qi Guo, and Yunji Chen. Hindsight value function variance reduction stochastic dynamic environment.
In IJCAI, pages 2476-2481 ijcai.org, 2021.
10

---

[14] William H. Guss, Brandon Houghton. Nicholay Topin, Phillip Wang, Cayden Codel. Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrationn. In IJCAI. ijeai.org, 2019. doi: (1196///jaai.2211/3333 URL. https: //op:::://d....// org/10. org/10.24963/ ijcai 2019/339.
[15] Amin Jaber, Murat Kocaoglu, Karthikeyan Shanmugam, and Elias Bareinboim. Causal discovery from soft interventions with unknown targets: Characterizzatio and learn- ing. In NIPS, 2020. URL. https://procesdingss neurips cc/paaper/202//aae (cc231aaseets00cc444333353222--2 wtt.://ccccssttttttt.o.m html.
[16] Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Chris Pal, and Yoshua Bengio. Learning neural causal models from unknown interventions. CoRR, abs/1910.01075, 2019. URL /tttt://axxi....o
abtp//1....111..0 01075.
[17] Harsha Kokel, Arjun Manoharan, Sriraam Natarajan, Balaraman Ravindran, and Prasad Tade- palli. Reprel: Integrating relational planning and reinforcement learning for effective abstraction. In ICAPS, 2021.
[18] Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien, Gradient- based neural DAG learning. In ICLR, 2020. URL rk1bKA4YDS.
[19] Andrew Levy, George Dimitri Konidaris, Robert Platt Jr., and Kate Saenko. Learning multi-level hierarchies with hindsight. In ICLR, 2019. URL ryzECoAcY7.
[20] Siyuan Li, Lulu Zheng, Jianhao Wang, and Chongjie Zhang. Learning subgoal representations with slow dynamics. In ICLR. OpenReview.net, 2021.
[21] Timothy P. Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016. URL /attt://arxxae /ttp////ww...r////////s./. 02971.
[22] Arquímides Méndez-Molina Ivan Feliciano-Avelino, Eduardo Morales, and Luis Enrique Sucar. Causal based q-learning Res. Comput. Sci., 149(3) 104, 2020. URL http://res. cic.ipa. i.c.ipa.mx/2000 149 /ttpa//49/.//aaaaaaaaaaeeeaeee
//uuaal222Baadtt22--aaaraagg pdf.
[23] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, and al. Human-level control through deep reinforcement learning. Nat., 518(7540):529-533 2015. doi: 10.1038/nature 14236. URL
https://aoi.org/// 1/////0..038//aa0 1038/nature14236.
[24] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In NIPS, 2018. URL http::////rroceedingg cwwuuiips.cppppppeee. 2018/haa
html.
[25] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. In ICLR, 2019. URL httpss///pperrviieee net/forum? id= emus0qF7.
[26] Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? CoRR, abs/1999.10061 2019. URL http:
org/ ttbb//9999 10618.
[27] Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In ICML. PMLR, 2020. URL http::/proceediing
mlr. pptsss/v19/pptsiii.a
html.
(28) Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. ArXiv, abs/2007. 000006............ 2020.
[29] Maximilian Seitzer, Bernhard Schölkopf, and Georg Martius. Causal influence detection for improving efficiency in reinforeement learning. In NeurlPS, pages 22905-22918 2021.

---

[30] Sungryull Sohn, Junhyuk Oh, and Honglak Lee. Hierarchical reinforcement learning for zero- shot generalization with subtask dependencies. In NIPS, 2018. URL httpatppp//proeeednngs neurips. html.
Absttrattte..raatt....... Abstract
[31] Sumedh A. Sontakke, Arash Mehrjou, Laurent Itti, and Bernhard Schölkopf. Causal curiosity: RL agents discovering self-supervised experiments for causal representation learning. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 9848-9855 PMLR, 2021.
[32] Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS. The MIT Press, 1999.
[33] Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J. Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. AAAI. AAAI Press, 2017. URL http://aaai... org/ocs////.o/ooooooooooooooooo oww//idee..ii//////ee//ee//eeeee
14630.
[34] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning.
In ICML. PMLR, 2017. URL http://prroceedings mlr prress/v70/vezznevetttt html.
[35] Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In ICLR, 2018.
[36] Vinícius Flores Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David P. Reichert, Timothy P. Lillicrap, Edward Lockhart, Mur- ray Shanahan, Victoria Langston, Razvan Pascanu, Matthew M. Botvinick, Oriol Vinyals, and Peter W. Battaglia. Deep reinforcement learning with relational inductive biases. In ICLR, 2019.
[37] Tianren Zhang, Shangqi Guo. Tian Tan, Xiaolin Hu, and Feng Chen. Gener- ating adjacency-constranned subgoals in hicrarchical reinforcement learning. In NIPS, 2020. f5
URL https: https://proceee h/topee//prooeedinggs neurips cc/ (ppper/2000 /hash/ Abstract. httml.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 7
(C) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No]
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
12

---

(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] We adopt open-source experimental
environments.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
We offer the code of method in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] There are no personally identifiable information offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "sVrWPTs9vGI", "reviewer": "Reviewer_vmvq", "summary": "The authors propose a formulation of hierarchical reinforcement learning that utilizes causal structure between environmental variables to generate subgoals. The discovered subgoals and lower-level policies generated to reach the subgoals are subsequently adapted using a learnt hierarchical agent that solves downstream tasks. The authors test their method in addition to several hierarchical RL baselines on the 2D-Minecraft and Eden environments and obtain promising results on the both. They show faster convergence in both environments in addition to interpretable causal graphs. \n", "questions": "See the Weaknesses section for my questions. I believe adding incorporating your answers to the draft would be beneficial.  ", "limitations": "See the Weaknesses section for my analysis of the assumptions/limitations of the work. I don't believe there is any cause for concern for negative societal impact. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The problem considered by the authors is one of strong relevance. \n2. The approach is principled with well-made figures. \n3. The results are repeated across seeds and are in challenging environments. \n\nWeaknesses\n1. Directionality: It's unclear how the algorithm learns directionality for causes and effects. From my understanding, the interventional data is generated by lower-level policies in the action space that attempt to set the environmental variables to a particular value. Thus, it appears the order in which environment variables are selected matters. What if a variable is selected in the initial iterations for which the causal parents are discovered in a later iteration? Is the causal graph updated in such a setting? For example, if the agent decides to learn policies that intervene on Ironore before it intervenes on stonepickax, what happens then?\n2. Commentary on Scalability of during large number of EVs: I believe that the authors need to add some information about the scalability of their approach with environmental variables. In the current draft, they say “ to verify whether newly associated variables are controllable as supposed, we compare the final training success rates of the new subgoals with a preset threshold phi before adding them to the subgoal hierarchy”. How are they selected? How many are selected? How are the lower-level policies learned? I would imagine through equation (4), but how long do you train, etc should be added. \n3. The algorithm should not be relegated to Supplementary work. This draft sorely misses the details afforded by it and would help ground a lot of the description. \n4. Some assumptions on the density of the underlying causal graph: I believe the authors need to state some assumptions about the density of the underlying causal graph, in particular, that currently, they assume that the underlying causal graph is dense and that each of the environment variables influences the others. I can imagine a situation where mechanisms are sparse and that learning policies which intervene on a causal factor only allows one to intervene on the immediate children of the same causal factor, but these children no longer have any other effect variables linked to them.   \n5. Description of Subgoal Hierarchy needed: What is a subgoal hierarchy? From reading the paper several times, I was able to discern that is an ordering on the elements in the goal space that are linked due to their temporal dependencies, i.e., getting IO is a child of SP because SP is needed to mine IO. Some mathematical precision is necessary to describe this especially since it forms the core of your algorithm. Compressing section 5 by avoiding newlines will generate room enough for such an explanation. \n6. Assumptions on the utility of causal graphs for subgoal: A central assumption made by the authors is that in their setup, nodes on the causal graph are all useful in goal generation. This is in general not true and worth stating. For example, manipulation by a robotic hand of an object in its palm does not require knowledge of friction coefficients between the block and the ground.\n7. Curious that the oracle is outperformed by your method. Is this a bug? Should it not be that the oracle with ground-truth causal graph inputs beat discovery?\n8. Missing citing work: https://arxiv.org/abs/2010.03110 and https://proceedings.neurips.cc/paper/2021/file/c1722a7941d61aad6e651a35b65a9c3e-Paper.pdf and https://ieeexplore.ieee.org/document/9561439\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aN4csjSAc7", "reviewer": "Reviewer_K7mE", "summary": "This paper proposes a new hierarchical reinforcement learning method that could automatically discover hierarchical structure with a learned causality graph. To build the causality graph, the proposed method introduces the “Environment Variables” as nodes in that graph, which requires some task-specific knowledge. The causality graph and subgoal-based hierarchical policy are simultaneously learned and facilitate each other.  ", "questions": "The assumption of oracle Environment Variables is a little strong. Could the proposed method generalize to more common settings where no oracle Environment Variables are available?\n\nThe blue learning curve in Figure 4(c) has not been plotted to 40000 training steps. Will it converge steadily at the end of the learning?\n", "limitations": "The authors have discussed the limited application to environments with discrete environment variables in Section 7, and have not touched on another limitation of requiring oracle environment variables. Those two limitations are related.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Significance\nAutomatic hierarchical structure discovery is a fundamental research problem in deep reinforcement learning. This paper provides a new way to solve this problem, which owes remarkable significance. \n\n## Originality\nThe idea of using a causality graph to discover hierarchical structure is novel. However, the proposed method relies on the “Environment Variables”. The way of obtaining the “Environment Variables” is non-trivial, so I doubt whether this method could be applied to more general settings, where the oracle “Environment Variables” are hardly accessible. \n\n## Clarity\nThe writing of this paper is mostly clear, but seems in a rush. For example, there is a latex compile error in Line 485. The authors are encouraged to polish up the writing in the future.\n\n## Quality \nThe idea of using a causality graph to facilitate the discovery of hierarchical structure makes sense, since there are causality relationships among subtasks in most hierarchical tasks. The proposed method is compared against previous goal-conditioned HRL works. However, claiming HAC as a state-of-the-art method is not precise. A lot of more advanced goal-conditioned HRL works have emerged in recent years [1, 2], where the World Model paper also proposed to build a graph to guide the subgoal selection. The authors are encouraged to compare the proposed method to more advanced goal-conditioned HRL methods.\n\n[1] Zhang er al., World Model as a Graph: Learning Latent Landmarks for Planning, ICML 2021.\n[2] Li et al., Learning Subgoal Representations with Slow Dynamics, ICLR 2021\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JNRGq5hU_YG", "reviewer": "Reviewer_6RRy", "summary": "The paper proposes a causality driven hierarchical reinforcement learning framework to build hierarchical structures in an iterative boosting way. The paper shows experimental results in 2D-Minecraft and Eden environments with sparse rewards. ", "questions": "It is not clear to me the need of using ideas from causality? Given some environment variables, relationships between variables can be discovered without ideas from causality.\n\nAlso, intervention sampling seems very close to the notion of exploration. Just that it is a bit smarter to explore in a smaller space using environment variables.  ", "limitations": "The assumptions that environment variables are given to us is the main limitation of the paper. Finding these environment variables along with the structure, is difficult. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe paper is well written and clear. The experiments are well done. \n\nWeakness: \nThe weakest part of the paper is that it is not clear where the novelty of the paper lies. It would be greatly appreciated if the authors differentiate their work against prior work. \n\nMultiple previous work tries to find graph like structures in environments for finding skills/options etc\nFor example, Robot learning from demonstration by constructing skill trees.\n\n\nImportant Prior Work on HRL and Minecraft: \n1) Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations\n2) Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6pQ38aKvwdW", "reviewer": "Reviewer_E3Uf", "summary": "The paper proposes an agent that combines causal discovery and hierarchical reinforcement learning (HRL), combined into causality-driven HRL, or CDHRL. The environment variables, which form the basis for the nodes of the structural causal model that the agent uses, are assumed to be available through an oracle. Training happens in two stages: first, the causal structure is discovered by training the agent to reach an expanding set of subgoals, after which the agent is trained on the actual task. On the two environments that are evaluated, the CDHRL method discovers reasonable causal graphs and outperforms the two baselines. Several ablations show that those results are robust to noise in the provided environment variables and limited iteration depth.", "questions": "- Could the authors discuss the differences between their contribution and those of Corcoll et al.? Both are aiming to introduce causality into hierarchical reinforcement learning. Not citing Corcoll et al. seems like an oversight.\n- Did the authors consider environments like the ant maze in the MEGA paper? While they may not be the most interesting from a causal perspective, they would be interesting regression tests.\n- Could the authors comment on the causality graphs discovered for the Eden environment? The graph for minecraft is very helpful.", "limitations": "- The assumption that the EVs are available through an oracle is limiting. The results presented are for cases where they are available. Discovering them in an unsupervised way can be problematic - the signal for the usefulness of candidate EVs will have to come from a combination of task reward and causality discovery success. Hence, discovering EVs is likely to require interaction with the causality discovery process, which would create a cold-start problem. The question of EV discovery is not addressed in any of the experiments. That isn’t a strict necessity for this paper, but it does limit the applicability of the method, and addressing it would make the paper stronger.\n- The environment variables are assumed to be discrete. This assumption makes the exploration easier by limiting the effective number of states that need to be considered in the causal graph. Again, addressing this aspect, ideally with experiments (which would presumably take place on other environments than the ones used) would clarify the applicability domain of the proposed method.\n- The causal structure of the environment is assumed to be small enough in number to constitute a nicely explorable graph - the pre-training phase runs ‘until no new causality is discovered’. This assumption won’t generally hold, and addressing the question of how much to invest in causality discovery would be valuable for understanding the generalizability of the results.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strength: Combining causality with HRL makes lots of sense. Often, HRL approaches don’t explicitly address the causality, but assume it implicitly. Assuming the environment variables are available, providing the assumption of a causal structure as an inductive bias is as natural as transformers are for language or convnets for images.\n- Strength: The interaction between the subgoal construction and the discovery process effectively induces an exploration reward, so the proposed agent not only learns about causal structure in the environment that it sees, but also ensures it sees many interesting aspects of the environment.\n- Strength: the paper is well-structured and easy to read.\n- Weakness: the availability of the environment variables is assumed. The authors mention a few methods aimed at discovering those, but no experiments are included where discovered environment variables are being used.\n- Weakness: The results are a bit limited. Two tasks, two baselines. They make the point that in the right circumstances, this method is effective, but they don’t really address the question which circumstances are the right ones.\n- Weakness: the paper is in need of a round of language editing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept