Deep Reinforcement Learning with Causality-based Intrinsic Reward
Authors: Peng Zhang
Furui Liu
Zhitang Chen
Jianye HAO
Jun Wang
Venue:  (2021)
Keywords: Reinforcement Learning
Causal Relation

Abstract: Reinforcement Learning (RL) has shown great potential to deal with sequential decision-making problems. However, most RL algorithms do not explicitly consider the relations between entities in the environment. This makes the policy learning suffer from the problems of efficiency, effectivity and interpretability.  In this paper, we propose a novel deep reinforcement learning algorithm, which firstly learns the causal structure of the environment and then leverages the learned causal information to assist policy learning. The proposed algorithm learns a graph to encode the environmental structure by calculating Average Causal Effect (ACE) between different categories of entities, and an intrinsic reward is given to encourage the agent to interact more with entities belonging to top-ranked categories, which significantly boosts policy learning.  Several experiments are conducted on a number of simulation environments to demonstrate the effectiveness and better interpretability of our proposed method.

Under review as a conference paper at ICLR 2021
DEEP REINFORCEMENT LEARNING WITH CAUSALITY- BASED INTRINSIC REWARD
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement Learning (RL) has shown great potential to deal with sequential decision- maaking problems. However, most RL algorithms do not explicitly con- sider the relations between entities in the environment. This makes the policy learning suffer from the problems of efficiency, effectivity and interpretability. In this paper, we propose a novel deep reinforcement learning algorithm, which firstly learns the causal structure of the environment and then leverages the learned causal information to assist policy learning. The proposed algorithm learns a graph to encode the environmental structure by calculating Average Causal Effect (ACE) between different categories of entities, and an intrinsic reward is given encourage the agent to interact more with entities belonging to top-ranked cate- gories, which significantly boosts policy learning. Several experiments are con- ducted on a number of simulation environments to demonstrate the effectiveness and better interpretability of our proposed method.
INTRODUCTION
Reinforcement learning (RL) is a powerful approach towards dealing with sequential decision- making problems. Combined with deep neural networks, deep reinforcement learning (DRL) has been applied in a variety of fields such as playing video games (Mnih et al., [015)] Vinyals et al.] 2019 [erner al. 2019). mastering the game of Go (Silver et al. (2016) and robotic control (Ried- miller et al. 2018). However, current DRL algorithms usually learn a black box policy approximated by deep neural network directly using the state transitions and reward signals, without explicitly understanding the structure information of the environment.
Compared with DRL. agents, an important reason why humans are believed to be better at learning the ability to build model on the relations between entities in the environment and then reason based on it. This ability is an important component of human cognition (Spelke & Kinzler 2007). As the learning process continues, through interactions with the environment and observations of it, human can gradually understand its actions' causal effects on the entities as well as the relations between entities and then reason based on them to figure it out the most important actions to take in order to improve the efficiency. In scenarios that contain multiple entities with complicated rela- tions, optimal policy may be obtained only when the structured relation information is captured and exploited. However, most current DRL algorithms do not consider structured relation information explicitly. The knowledge learned by agent is implicitly entailed in the policy function, which are usually unexplainable neural networks. Therefore, whether the relations are well understood and exploited by the agent is unknown. When the environment is with high com- plexity, blackbox learning of policies suffers from low efficiency, while policy learning over explicit representation of entity relations can significantly boost the learning efficiency. Based on the fact that entities in an environment are often not independent but causally related, we argue that disen- tangling the learning task into two sequential tasks, namely relational structure learning and policy learning, and leveraging an explicit environmental structure model facilitate the policy learning process of DRL, agents are expected to boost the performance. With the learned relational structure information, the agent performs exploration with a tendency of prioritizing interaction with critical entities, which is encouraged by intrinsic rewards, to learn optimal policy effectively.
action-value
Taking this inspiration, we propose deep reinforcement learning algorithm which firstly learns the relations between entities and then recognize critical entity categories and develop an intrinsic reward based approach to improve policy learning efficiency and explainability. The proposed algo-

---

Under review as a conference paper at ICLR 2021
rithm learns graph to encode the relation information between categories of entities, by evaluating causal effect of one category entities to another. Thereafter, intrinsic reward based on the learned graph is given to agent to encourage it to prioritize interaction with entities belonging to impor- tant categories (the categories that are root causes in the graph). Previous works also use graphs to provide additional structured information for the agent to assist policy learning (Wang et al. 2018 Vijay et al. 2019). However, graphs leveraged by these works are provided by human and thus rely heavily on prior knowledge. Compared with their methods, our algorithm overcomes the deficiency that the graph can not be generated automatically. Our approach requires no prior knowledge and can be combined with existing policy-basee or value-based DRL algorithms to boost their learning performance. The key contributions of this work are summarized as follows:
We propose a novel causal RL framework that decomposes the whole task into the structure learning and causal structure aware policy learning.
The learned causal information is leveraged by giving causality based intrinsic reward to agent, to encourage it to interact with entities belonging to critical categories for accom- plishing the task.
We design two new game tasks which contain multiple entities with causal relations as benchmarks to be released to the community. The new benchmarks are designed in such ways that categories of objects are causally related. Experiments are conducted designed simulation environments, which show that our algorithm achieves sfate-of-the-art performance and can facilitate the learning process of DRL. agents under other algorithmic frameworks.
The paper is organized as follows. In Section 2 we introduce deep reinforcement learning and Average Causal Effect (ACE), which are key components of this work. Then illustrate our we show the experimental results on the designed environment demonstrate the effectiveness of our framework. In Section 5 we introduce previous works that relate to our method. Finally, conclusions and future work are provided in Section 6
algorithm in Section 3 in details. In Section [
BACKGROUND
2.1 DEEP REINFORCEMENT LEARNING
An MDP can be defined by a 5-tuple A,P, 7). where is the state space, AA is the action space. P is the transition function, R is the reward function and is the discount factor (Sutton & Barto| 2018). A RL agent observes a state st E at time step t. Then it selects an action a from the action space A following policy π (a, (ar)st), which is a mapping of state space action space. After taking the action, the agent receives a scalar reward r, according to R(st ac). Then the agent transits to the next state 9:+1 according to the state transition probability P (3e4 |sc, at). A RL agent aims to learn a policy that maximizes the cumulative discount reward, which can be formulated as
R₁ = Σi 7 rrrt where T is the length of the whole episode. In the process of learning an optimal policy, a RL agent generally approximates the state-value function V (s) or the action value function Qu (s, a). The state value function is the expected cumulative future discounted reward from state with actions sampled from policy π:
T
(s) E
Σ
Deep Reinforcement Learning (DRL) which combines Deep Neural Networks (DNNs) with RL can be an effective way to deal with high- high-dimensional state space. It benefits from the representa- tion ability of DNNs, which enable automatic feature engineering and end-to-end learning through gradient descent.
Several effective algorithms have been proposed in the literature and we use A2C in this paper as our basic algorithm, which is a synchronous version of A3C (Mnih et al., 2016). A2C consists of variety of actor-critic algorithms (Sutton et al. 2000). It directly optimizes the policy no parameter- ized by 0 to maximize the objective J(Ø) = R. EEk **re+*) by taking steps in the direction of V (().. The gradient of the policy can be written as:
[() [o log mo A" as)
2

---

Under review as a conference paper at ICLR 2021
where as) (= (T()) is the advantage function. The advantage function can be estimated by one-step TD-error A 01) = + (V (se),, where Vo(s) is the approx- imation of the state-value function V (8) parameterized by Φb
2.2 CAUSAL NEURAL NETWORK ATTRIBUTIONS
Attributions are defined the effect of input feature the prediction function's output (Sun- dararajan et al. 2017) and Chattopadhyay et al. (2019) propose a neural network attribution method- ology built from first principles of causality. Chattopadhyay as a Structural Causal Model (SCM), and proposes a new method to compute the Average Causal Effect of an input neuron on an output neuron based on the do(-) calculus (Pearl 2009).
al. (2019) views the neural network
Definition 1. (Average Causal Effect). The Average Causal Effect (ACE) of binary random vari- able x on another random variable y is commonly defined Ely [lyx = 1)] Ely]do(z = 0)].
While the above definition is for binary-valued random variables, the domain of the function learnt by neural networks is usually continuous. Given neural network with input layer I and output layer las we hence measure the ACE of an input feature E l with value a on an output feature y E ln as:
do(z.=a)) AC AC
Eyy do(x; = a)] - baselinez.
(3)
Definition 2. (Causal Attribution). The causal attribution of inpul neuron x; for an ouiput neuron y is defined as ACE do(
In Equation 3 an ideal baseline would be any point along the decision boundary of the neural network, where predictions are neutral. However, Kindermans et al. (2019) showed that when a reference baseline is fixed to specific value (such as zero vector), attribution methods are not affine invariant. Therefore, Chhatoppadyyy al. 2019) proposed the average ACE of z; on y as the baseline value for 2;:
baseline,, Ez, Eyly Meylyldoff [d(x; a)ll
In this paper, we use causal attribution method to infer the relations between entities.
3 METHOD
In this section, we present a novel DRL framework named CARE (CAusal RElation) that enables the agent to infer the causal relationship between entities. Figure iillsttrate Digustaiilsstrtttss the overall framework of CARE. CARE adopts a two-stage training paradigm. In the first stage, the agent learns a model of the environment by minimizing the prediction error of the state transition. Then, calculate ACE values between categories of entities, which are used for constructing a causal graph G. When G is at hand, nodes ranked lower cannot cause nodes ranked higher. This order is used to measure the importance of the categories, and intrinsic reward is given based on the causal ordering. Specifically, the agent receive a one-step "enhanced" reward Ft where:
able to obtain the causal ordering of all categories, that is, permutation such that
1 + +r
(5)
where re is the original extrinsic reward given by the environment, and r is the intrinsic reward, designed by the learning algorithm to encourage the agent to maximize the effect of its behaviour on the change states of critical entities. We describe the details later sections.
This section is organized as follow. In section 3.1 we first introduce eategory-orientee state factor- ization. In section? 3.2, we describe how get the relation graph & and in section 3.3 we show how to calculate the intrinsic reward v.C
3.1 CATEGORY-ORIENTED STATE FACTORIZATION
We focus on environments which contain multiple categories of entities. The category of entity is the acting rules that govern the actions of the entity. Each entity is within one category, and the entities within one category share the same acting rules. An example of the category of entity appears in the experimental section. Consider environment consisting of two kinds of sheep:
3

---

Under review as a conference paper at ICLR 2021
Action
Luv
Causal Relation
Intrinsic
Extrinsie
Enviroumental Model
Analysis
Reward
Reward
Action
Causal Ordering
Agent
Figure 1: Overall framework CARE, CARE firstly learns a model environment which takes states of different categories of entities as input and predicts the next step states of all categories. Then we perform causal relation analysis on the model to get rolation graph among all categories. Finally, intrinsic reward based the graph and the effect of agent behavior entities is given the agent to assist policy learning.
one is the ewe that takes random walk. The other is the lamb that always follow the ewe. Then there are two categories of entities in the environment. In this paper, infer the causal relations among categories. Another view is to infer the causal graph among entities the environment. However, certain abstraction of entities is beneficial and simplifies the learning because quite often dynamic and interactive environment, entities could pop up or disappear as the result of actions taken by the agent or the environmental evolution. Therefore, maintaining graph with changing nodes could be quite challenging, rendering the learning algorithm unnecessarily complicated. We thus choose category-level casuality inference for scalability. The category of each entity is given as a prior, or generated by applying computer machine vision technology such as unsupervised or pretrained object segmentation models (Agnew & Domingos 2220)), shape and color analysis (Ren al. 2015 He et al. 2017). We introduce factored space to represent the states. The factored state space consisting of entities space of the ith category. At time f, the state of the entities of ith category st E S', and the state S of all entities is composed of local states of all category factorization ensures that each category entities is independently represented. this paper, the state is represented using K-channel feature map, each corresponding to one category of entities. More details can be found in the Appendix.
categories is S S x... x SK where St the state
[s] s7 ..., sK,, This
3.2 CAUSAL RELATION ANALYSIS
In this section, we demonstrate how to obtain the casual graph. Firstly, we learn a model of the envi- ronment, which predicts the next step state of each category of entities. Thereafter, we perform av- erage causal effect analysis between each pair of categories. Namely, conditioning on all other cate- gories, compute measurement quantifying the influence of one category on the other. Based on this, are able to recover the whole causal graph, and the causal ordering of categorics. In hypoth- esis, vertices with higher ranking tant the environment, and we will give intrinsic rewards based on the influence of the agent's action on different categories of entities.
Dcvorr
Conv
Action
more impor-
Figure Architecture of the environmental model. The model contains one encoder and K de- coders each corresponding coder takes current state input and the decoders predict the step state of each category.
category. The en-
Environment Model Learning
To learn the environment model, we first ran- dom agent to interact with the environment to col- lect buffer of experience B {(St, 4z [
4

---

Under review as a conference paper at ICLR 2021
It contains T tuples of states 8, E S, actions de A, and follow-up states 8t+1 S, which is reached after taking action eg. Our goal is to predict next step state of each category of entities to understand the environment, without agent's s policy training. It should be noted that our model is only used to analyze the relations between category of entities, i.e. the causal ordering of entity categories, instead of using the model for planning like model-based RL. Therefore, our method does not require extremely high model accuracy.
Our model employs an encoder-decoder architecture (See Figure 2) The encoder CNN operating directly on observations from the environment, and the output of the encoder 2 Enc(ss, 0z) is flattened to a vector. The decoder is composed of several deconvolutional layers. There are K decoders Dec each corresponding a category of entities, which take the encoded vector as input and predict the next state of each category of entities.
Dec (2+)
(6)
The model is trained by minimizing the following loss:
L
Ed d( 88+1)
where d(ss-1 s44)) denotes the distance between the predicted next state 8/41 and the one.
Calculating ACE
After getting the environment model trained, we can calculate ACE values of each pair of categories following the method described in Section 2.2 which computes ACE by intervening the states of each category of entities. Specifically, the ACE of a category i on another category can be calcu- lated by:
ACE t+ do(a)==t) - 1104ss = st)] baseline
8
Here s E St is the interventional state. do(s) = st) means that we manually control sf to another state sy known do calculus in causal analysis. The baseline,; is calculated by:
baseline,t = EEa st,, 1
Ido(s' = =
(9)
By definition, the interventional expectation B[s/ |do( = =,,]] is written as E [S Ido(s) = xt)] / 8 1 Ido(s) s²t)) ds
(10)
Computing the integral intractable because the exact distribution of s is unknown. Thus, we approximate Equation by empirical historical distribution sampling:
E E|s/41 (do(s) = st)
Σ
(1I)
where 8y s1 Dec (Enc( am)) is the predicted next state of category j and $ [s) .... St. ...,.. K] is the interventional state which sets s s while leaves other categories unchanged. By C B is a batch of experience sampled from the buffer with sample size N. The maximal ACE value is used as the final effect of category on category j:
ACE = max (ACE
e do(e)=o!)
(12)
In practice, it is also computed by sampling from the historical states set of the category i.
After getting pairwise ACE values, we are able get the causal graph G (V,E) of all categories of entities. V is the set of all vertices and each vertex represents a category of entities. E is the set of all edges and eij E & represents that category i causes category j. Let H be the K K adjacency matrix of G. which is obtained by the edge directing rule:
{b. 0, otherwise
11 if ACE,,) > ACE
Hi =
(13)
5

---

Under review as a conference paper at ICLR 2021
Since is assumed to be directed acyclic graph (DAG), there feedback loops any path starting from category and then back to itself. Consequently, there exists causal ordering of all vertices. Causal ordering is permutation μ of all index of vertices {1....; Kj. where vertices ranked higher cannot be caused by ones ranked lower. The nodes ranked higher are hypothetically more critical entity categories for the task. This will be used for designing intrinsic reward. Based on this, we define the criticality of entity category.
Definition 3. (Criticality of Eniit Category). The criticality of entity calegory is defined as the ranking of the category the causal ordering (s.
3.3 INTRINSIC REWARD
We adopt the tendency to prioritize critical entities by giving intrinsic rewards to the agent in addition the original extrinsic reward for policy learning. The basic idea is that actions that have a relatively large effect entities whose category ranks higher in the causal ordering are rewarded. Based on the learned model described in Section 3.2 we define the effect [s(s,, az) the agent's behavior on the ith category of entities, similar to ACE
as:
1, az) = f((s 1 [AI
Σ fi(s;,a) aEAA
(14)
Here fi(st, a.) Dec' Enc(s;, at) denotes the learned model and |A]] is the size of the action space. This method calculates the effect of the agent's action the certain category of entities and the second term [AT Eoea fi(st, a) in equation 14 serves as the baseline when calculating ACE.
The intrinsic reward is defined as r EK rt""
G.A. For each category, the intrinsic reward is:
g Br, if if
(15)
0, otherwise
Here B; and are hyperparameters. is constrained that the rewards of categories along the causal ordering are non-increasing 8mg 2 8pp 2 causal ordering.
Pag. where μr corresponds to position in the
4 EXPERIMENTAL RESULTS
In this section, we evaluate CARE on two test domains (see Figure 3), Shepherd and Ant, where an agent is required to have the ability of relational reasoning learn an ideal policy. We com- pare CARE with flat DRL method (A2C) and Relational DRL (Zambaldi et al. 2018). which is an A2C based algorithm and uses multi-head attention to extract relation between entities. Experiment details can found in the Appendix.
Shepherd (agent)
Frod: Queens can obtain energy cating collected by workers.
Lamb: follows the
Agent can atteck ants. Worker ant: collect Food and deliver the queen.
Ewe: moves around.
Queen ant: generate new enough energy.
Sheepfold.
Figure 3: The Shepherd game (lef() and the Ant game (right).
4.1
SHEPHERD
In the Shepherd game (Figure 3 lef(), a shepherd is expected to drive all of the ewe and lambs, which are randomly distributed in the ranch the beginning, back into the sheepfold. The agent's

---

Under review as a conference paper at ICLR 2021
-5
Ours
Episode Reward (1e3)
Relational A2C
Episode Reward (1e2)
Relational
//1
-2
A2C
-4
-6
2.5
5.0
10.0 12.5
10
20
30
40
60
Update Number (1e3)
Updite Number (1e3)
(b)
Figure 4: Experimental results on the two test games.(a) Learning curves of the Shepherd game, (b) Learning curves of the Ant game. All experiments are averaged over 5 random seeds.
objective to finish the task using as little time as possible. Unless the shepherd gets close and drives the sheep, the ewe will walk around the ranch and the lambs follow the ewe. The shepherd viewed as the agent here and has five actions: up, down, left, right and drive. Each action makes the agent move to the corresponding direction with one step size except for drive. If the agent takes the drive action, sheep near the agent will driven to move towards the sheepfold. The game ends when all sheep are in the sheepfold or the time exceeds fixed period. At every step, the agent receives a reward equal the negative sum of distance from each sheep to the sheepfold. Here we have K = 2 categories of entities.
We firstly evaluate our algorithm this game. The experimental result is shown in Figure ((a) The result shows that our method converges to better policy with higher mean episode return than other methods. Flat A2C agent also learn relatively good policy with slightly worse performance compared with our method, but takes a longer time. This result shows that understanding the causal relations and leveraging the learned relational information process of DRL agents. The performance the Relational DRL algorithm is not to our expectation, possibly because the attention mechanism does not capture correct relations of the entities in the environment.
significantly boost the learning
The causal graph learned by CARE is in Figure 6 (t). Here E, L and A denote Ewe, Lamb and Agent respectively. The edge from Euse Lamb represents that lambs are attracted by the ewe. The two edges from the Agent to Ewe and Lamb means that the ewe and lambs are driven by the agent. Although the agent is also considered node, it is ranked in the causal ordering. The ACE values for getting this graph are listed in Table
We also evaluate the effect [i(s, a) cal- culated by Equation [4] which is the cornersoone of the intrinsic reward. We firstly sampled a state from the historical trajectories, and then manually set the agent to every grid in the field. At each grid, calculate the agent's effect the category of ewe using Equation [4 Finally we visualize the calculated ef- fect in Figure(3) The value of coordinate (I, y) in the heatmap corresponds to ef- fect of the agent on the ewe when is on position (x, y) and taking action drive. As shown
Figure 5: Visualization the lewe(s, drive) in the Shepherd game. The heatmap is generated by intervening the position of the agent to every grid in the field to get Tewe(s, drive) using Equation 14
the Figure [5] the caleu- lated Tewe(' (8, drive) is high only when the agent is near the ewe. This be- cause the ewee will be driven to move towards the sheepfold only when the shepherd and sheep are close enough. This result shows that the effect of the agent the target category of entities is well

---

Under review as a conference paper at ICLR 2021
modeled. Moreover, this result also shows that choose the hyperparameter å easily, because there is large gap between the calculated [((8,)) a) when the agent's behavior affects ith category of entities or not.
Table ACE values in the Shepherd game.
Table 2: ACE values in the Ant game.
ACE
exce
lamb
ageev
ACE
queen
worker
Food
agent
ease
1.52e-03
1.03c-06
queen worker
2.97e-03
7.71e-04
1.45c-06
lamb
5.14c-05 3.65e-05
1.06e-06
4.33e-05
2.42c-03
4.13e-06
agent
6.44e-06
Food
1.22e-06
3.72e-03
2.72e-07
agent
1.76e-06
2.12e-04
3.86c-05
4.2 ANT
In the Ant game (Figure right), the agent is expected to kill all ants in the field. There are two queen ants and four worker ants at the beginning. Queen ants move around the field. Worker ants will firstly go the nearest food and then bring to a queen ant. The queen ant obtains some energy by eating the food. If the queen ant's energy exceeds a threshold, it will generate a new worker ant. Food will continue to be produced in the fixed positions. For this environment, there are K = 3 categories of entities. The agent has five actions: up, down, left, right, each of which makes the agent move one step towards the corresponding direction and attack, which kill the ant around the agent if there exists. The game ends when all ants are killed or the time span exceeds fixed period. The agent will receive reward of +1 if it kill one ant, whether it is a worker ant or queen ant. At the end of the episode, the agent will receive reward of -110 x 100 m.) where n and m denote the number of left worker ants and queen ants respectively.
An optimal policy should prioritize the task of killing the queen ant. Otherwise, worker ants will be continuously pro- duced by the queen ants and the number of ants grows up this game, compar- ing flat A2C and Relational DRL. The experimental results are given in the Figure 4(b) this game, we observe that our method learns policy that kills the queen ants first and then other ants. However, flat A2C and Relational DRL both learn a policy that keeps the queen ants alive but stays at certain position around the food wait for and kill worker ants coming for food.
E
very fast. We evaluate our algorithm
W
Figure 6: Learned graphs of the Shepherd game (left) and the Ant game (right).
We show the learned causal graph in Figure 6 (right). Q, W, F and A denote Queen, W Worker, Food and Agent respec-
tively. The causal ordering is (Q, F, W] and the calculated ACE values be found in the Table 2
Since our algorithm has graph learning procedure, we also record the rumning time the algo- rithms in Table 3. Our algorithm takes longer time than A2C when the training step is the same. However, given the performance gain, we think the cost in time reasonable: our algorithm learns better policy, which is not obtainable by other algorithms. The performance gap is especially obvious Ant game.
Table 3: Running time (seconds) the Shepherd and Ant games.
Shepherd
Ant
Method
Graph learning
RL policy learning
Graph learning
RL policy learning
CARE
1002
1149
3201
5136
A2C
779
4516
Relational
2251
9919
5 RELATED WORK
Our framework learns a graph and the graph entails the relationship between entities in the environ- ment. Compared with model-based reinforcement learning algorithms (Sutton) 1991 Silver al.
8

---

Under review as a conference paper at ICLR 2021
Heess et al. 2015 Nagabandi et al. 2018 which usually learn environment dynamics and plan or learn on the learned dynamics to reduce the interaction with the real environment, our method focuses on learning relations of entities but not environment dynamics. The learned causal graph used to order the category of entity. Mahadevan & Maggioni] (2007); Mabu et al. (2007); Met- zen| (2013); (Shoeleh & Asadpour (2017); Gonchez-Gonzalez et al. (20T8) also use graphs to learn a representation of the environment. However, these methods still focus on learning environment dynamics and thus these problems are usually solved via model-based RI..
2008
The learned graph could be viewed as a structure description of the environment. Applying structure knowledge of environments in RL has been studied in previous works, WWan et al. (2018) explicitly models the structure of an agent as a graph and uses a GNN to approximate a policy. Vijay et al. (2099) builds a knowledge graph as prior for the agent, which illustrates different relations between entities in the environment. However, graphs leveraged by these two works priors provided by human. Compared with these works, our algorithm supports automatic graph learning and requires no human prior knowledge. (Ammanabrolu & Riedl (2019) proposed KG-DQN, which constructs knowledge graph to represent the environment and uses Graph Neural Networks to extract features of the graph. This work nevertheless only adapts to Adxe-udee Game, because their knowledge graph can be only generated from natural language. /Zambaldi et al. (2018) use multi-head attention to extract relation between entities. However, their method solves problem from the aspect of entity instead of category. Notice that our model deploys a mncoder-decoder structure for processing the input signals. This is used by Shi et al. (2020) known as self-superviser interpretable network for extracting task-relevant attention masks which are interpretable features for agent's decisions. (Ag- new & Domingos (2020) uses object-centric state representations and exploits the object interactions and dynamics for identifying task-relevant object representations.
CONCLUSIONS
In this paper, we propose a novel deep reinforcement learning algorithm, which firstly learns en- vironmental causal structure and then leverages the learned relational information to assist policy learning. Experimental results show that our algorithm has good performance, indicating that incor- porating the environmental structure for reasoning is a promising research direction. Future work includes studying environments with dynamical graphs, and improving the training efficiency of the framework.
REFERENCES
William Agnew and Pedro Domingos. Relevance-guided modeling of object dynamics for rein- forcement learning. htpp://ariiv.oo//b//220:..::3333 3884, 2020.
Prithviraj Ammanabrolu and Mark Riedl. Playing text-adventure games with graph-based deep reinforcement learning. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3557-3565,
2019.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 191200668, 2019.
Aditya Chattopadhyay, Piyushi Manupriya, Anirban Sarkar, and Vineeth N Balasubramanian. Neu- ral network attributions: A causal perspective. In International Conference on Machine Learning (ICML), pp. 981-990, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In IEEE International Conference on Computer Vision (ICCV), pp. 2961-2969, 2017.
Nicolas Heess, Gregory Wayne, David Silver, Timothy P. Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Infor- mation Processing Systems (NIPS), pp. 2944-2952, 2015.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo. Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. Springer, 2019.

---

Under review as a conference paper at ICLR 2021
Shingo Mabu, Kotaro Hirasawa, and Jinglu Hu. A graph-based evolutionary algorithm: Genetic network programming (GNP) and its extension using reinforcement learning. Evolutionary Com- putation, 15(3)3669-39, 2007.
Sridhar Mahadevan and Mauro Maggioni. Proto-value functions: A laplacian framework for learn- ing representation and control in markov decision processes. Journal of Machine Learning Re-
search, 8:2169-2231, 2007.
Jan Hendrik Metzen. Learning graph-based representations for continuous reinforcement learning domains. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part I, pp- 81-96, 2013.
Mnih, K Kavukcuuglu. D Silver, A. A. Rusu, J Veness, M. G. Bellemare, A Graves, M Riedmiller, A. K. Fidjeland, and G Ostrovski. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference Machine Learning (ICML), pp. 1928-1937, 2016.
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dynam- ics for model-based deep reinforcement learning with model-free fine-tuning, In International Conference Robotics and Automation (ICRA), pp. 7559 7566, 2018.
Judea Pearl. Causality. Cambridge university press, 2009.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS), pp. 91-99, 2015.
Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International Conference Machine Learning (ICML), pp- 4341-4350, 2018
Alvaro Sanchez-Gonzales Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin A. Ried- miller, Raia Hadsell, and Peter W. Battaglia. Graph networks as learnable physics engines for inference and control. International Conference on Machine Learning (ICML), pp. 4467-4476, 2018.
Wenjie Shi, Zhuoyuan Wang, Shiji Song, and Gao Huang. Self-supervised discovering of causal features: Towards interpretable reinforcement learning. arXiv preprint arXiv:2003.07063 2020.
Farzaneh Shoeleh and Masoud Asadpour. Graph based skill acquisition and transfer learning for continuous reinforcement learning domains. Pattern Recognition Letters, 87:104-116, 2017.
D Silver, A. Huang, C. J. Maddison, A Guez, L Sifre, den Driessche G Van, J Schrittwieser, Antonoglou, V Panneershelvam, and M Lanctot. Mastering the game of go with deep neural networks and tree search. Nature, 429(7587):484444 489, 2016.
David Silver, Richard S. Sutton, and Martin Müller. Sample-based learning and search with per- manent and transient memories. In International Conference Machine Learning (ICML), pp. 968-975, 2008.
Elizabeth S. Spelke and Katherine D. Kinzler, Core knowledge. Developmental Science, 10(1): 89-96, 2007.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic autrbbttoo for deep networks. In International Conference on Machine Learning (ICML), pp. 3319-3328, 2017.
Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bulletin, 264) 1991.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
10

---

Under review as a conference paper at ICLR 2021
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Infor- mation Processing Systems (NIPS), pp. 1057-1063, 2000.
Varun Kumar Vijay, Abhinav Ganesh, Hanlin Tang, and Arjun Bansal. Generalization to novel objects using prior relational knowledge. arXiv preprint arXiv: 1906. 11315, 2019.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czameeki, Michaël Mathicu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 375(7782)350-334, 2019.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International Conference on Learning Represenlations (ICLR), 2018.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International Conference Learning Representations (ICLR), 2018.
A APPENDIX
A.1 EXPERIMENTAL SETTINGS
State Representation
In both the Shepherd game and the Ant game, the whole game field is grid world with size 18 x 18. The game state is represented by a tensor of size K 18 18. Each channel represents the state of one category of entities.
Environmental model
The encoder is composed of three convolutional layers. The first, second and third layer have 64, 128 and 256 3 filters with stride respectively, each followed by Batch Normalization layer. The output of the encoder is flatten to a vector and then enter two fully connected layers both with 512 hidden units. The action is encoded into vector of size 32 by three FC layers with 100 hidden units and then is concatenated with the output of the encoder. Each decoder takes the output of encoder as input and pass it into four fully connected layers, with 900, 300, 512, 512 hidden units respectively. Then the output of the FC layers taken as input of three deconvolutional layers. The first two layers have 128 and 64 3 x 3 filters with stride 2 respectively. The last layer has num. categories 4 4 filters also with stride 2. Each deconvolutional layer is also followed by a Batch Normalization layer.
When training the model, we collect a buffer with 5000 episodes. We use the Adam optimizer to train the model with a learning rate of le - 4 and batch size of 32.
Parameter Setting of RL
CARE and flat A2C use the same network architecture. The actor and critic share the same first two convolutional layers and FC layer. The two convolutional layers have 64 and 32 3 x 3 filters with stride 2 respectively. The FC layer has 128 hidden units. The critic and the actor both take the output of the share part and pass it into two FC layers with 512 hidden units. Finally they output the value and the action distribution. For the Relational DRL. algorithm, we use an open source implementation Other parameters are listed as follow:
11

---

Under review as a conference paper at ICLR 2021
Table 4: Shepherd
CARE 16
A2C
Relational
Number of process
16
16
Discount factor
0.99
0.99
0.99
Optimizer
RMSProp
RMSProp
RMSProp
Learning rate
Te
Te
Entropy term coefficient
0.01
0.01
0.01
8
le
5s
(10, 0) (decays to 0 at 3000th update)
Table 5: Ant
CARE
A2C
Relational
Number of process
16
16
Discount factor
0.99
0.99
0.99
Optimizer
RMSProp
RMSPrrp
RMSProp
Learning rate
7e
7e
Entropy term coefficient
0.5
0.1
0.1
Se
3s
(100, 0,
12


Reviews:
[{"review_id": "ssATOlkLmg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new deep reinforcement learning algorithm that learns the causal graph representation of the environment and leverages it to assist policy learning.\n\nThe authors should define a critical entity since they mentioned it many times in the article. Since the learned graph considers categories as nodes, I did not understand\nthe reference of critical entities as top-ranked nodes in the graph (Sec. Calculating ACE).\n\nI would like also to see a proper definition of the entities and how the learned causal graph of entities impacts the performance of the proposed method.\n\nThere is confusion about the nodes in the causal graph, sometimes the authors mention them as entities:\n\nAbstract: “critical entities of the causal graph”\nIntroduction:: ”The proposed algorithm learns a graph to encode the relation information between entities”.\n\nAnd they change to consider nodes as categories of entities:\n\nMethod: “we calculate ACE values between categories of entities, which are used for constructing a causal graph G”\nSec.3.1: “we focus on the causal relation among categories of entities.”\n\nFor my review, I am considering nodes as categories of entities, given that the authors discuss a restriction on learning the category causal graph\nrather than the entity causal graph to make their approach scalable. But this inconsistency and lack of definition lead to confusion.\n\nMy main concern about this paper is about the experimentation, specifically: baselines. \n\nI consider that the requirement of no prior knowledge is an interesting and valuable contribution of this paper. However, in scenarios in which we have such information, I do not consider that we should ignore it. Thus, I consider it is an open research question the application of the proposed method in these scenarios.\nIn this sense, I consider that the approaches proposed in [1,2] should be considered as baselines in an experiment where CARE receives the causal graph and does not\n“update\" it during the sequential decision-making process. The categories of the given entity causal graph could be inferred to build the causal relation among categories\nof entities as assumed as input to CARE. \n\nI did not understand why CARE is not appropriate to Text-Adventure Game [3].\n\n[1] Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.\n[2] Varun Kumar Vijay, Abhinav Ganesh, Hanlin Tang, and Arjun Bansal. Generalization to novel objects using prior relational knowledge. arXiv preprint arXiv:1906.11315, 2019.\n[3] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2018.\n\nReferences should point to the published work rather than the arxiv entries, when the former is available (e..g. Neural Network Attributions: A Causal Perspective, ICML’19).\n\nTypo:\n\n - Sec.3.2: “of several deconvlutional …” -> “of several deconvolutional …”", "extra_scores": {}, "extra_text": {}}, {"review_id": "LvNGA5yMjwd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe paper proposes a deep reinforcement learning algorithm of advantage actor-critic (A2C), which ﬁrstly learns the causal structure of the environment and then leverages the learned causal information to assist policy learning. The causal structure is computed by calculating Average Causal Effect (ACE) between different categories of entities, and the authors assume that an intrinsic reward is given to encourage the agent to interact more with critical entities of the causal graph. Two experiments were conducted on simulation environments (Shepherd and Ant tasks) and demonstrated the effectiveness in obtaining the rewards and interpretable and accurate detection of the true graph.\n\nReasons for score: \nThe motivation and solutions were simple and clear, but the related work about deep reinforcement learning leveraging causal relation was not properly introduced. Furthermore,  there were unclear points in the implementation (without sharing codes) and the results. I think the idea is interesting and contributed to this community, but for the above reasons, it is difficult to provide a higher rating. \n\n\nPros:\n1. The authors designed causality-based intrinsic reward to an agent, to encourage it to interact with critical entities for accomplishing the task.\n2. Two experiments were conducted on simulation environments (Shepherd and Ant tasks with heterogeneous agents) and demonstrated the effectiveness in obtaining the rewards and interpretable and accurate detection of the true graph.\n\nCons:\n1. There were also other researches in deep reinforcement learning leveraging causal relation (e.g., [i] and [ii] below). The title and the first contribution of this paper should be more specific and the related work should be included in the main text.\n2. There were unclear points in the implementation (without sharing codes) and the results (below). \n\n[i] W. Shi et al.  Self-Supervised Discovering of Causal Features: Towards Interpretable Reinforcement Learning, arXiv, 2020 \n[ii] W. Agnew and P. Domingos. Relevance-Guided Modeling of Object Dynamics for Reinforcement Learning, arXiv, 2020 \n\nOther comments:\n\nThere is little information about the algorithm of Relational DRL (Zambaldi et al., 2018) as one of the baseline. Relational DRL seems to be an attention-based A2C algorithm, but is the difference only related to the attention-based approach? At least, whether it is an A2C algorithm or not should be clarified. \n\nIn both experiments, is the number of categories K the number of agents? This was not specified.\n\nI did not understand Figure 5 right and the sentence (”The value of coordinate (x, y) in the heatmap corresponds to the effect of the agent on the ewe when it is on position (x, y) and taking action drive”). The correspondence of xy coordinates with Figure 5 left might help my understanding. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "DHaTE2Q-F_c", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed to learn causal relations to accelerate the\nlearning process in deep reinforcement learning. The proposed approach\nfirst learns a graph to represent the causal structure of the\nenvironment calculating the Average Causal Effect (ACE) between different\ncategories. It uses an intrinsic reward to encourage interacting with\nthe most relevant entities of the causal graph which accelerates the\nlearning process.\n\nThe proposed approach first need to create a factored state space,\ngiven by the user. It is not clear how intuitive is this process and\nif it can be performed automatically.\n\nThey first learn a model of the environment, predicting the next state of\neach category of entities. For that they obtain data of the agent\ninteracting with the environment and use an encoder-decoder approach\nto obtain k categories. Then predict the next state based on the\ncurrent state and an action, where the decoder predicts the next state\nof each category. \n\nIt is not clear how to determine the number of categories? (how many\ndecoders?) If there is a single agent interacting with different\nobjects, a new category needs to be defined for each object?\n\nIt then evaluates the ACE between every pair of categories. Following\nChattopadhyay et al., they used the causal attribution method to infer\nthe causal relations between entities. The proposed approach then\nconstructs a DAG, where an edge between category \"i\" and \"j\" is placed\nif ACE i->j > ACE j->i. The DAG represents a causal ordering where\ncategories with higher ACE value are placed before. Each category is\ngiven an intrinsic reward, in addition to the extrinsic reward, based on\nits order.\n\nThe approach was tested on two domains and compared against a flat DRL\n(A2C) and Relational DRL with clearly better results.\n\nThe results should also include the time used to obtain the causal graphs.\n\nTypos:\n... analysis et, al (Ren et al. ...\n... deconvlutional ...\n... cause that the ewe ... => ... cause the ewe ...\n... the the ...\n\nPros:\n- The use of causal relations to accelerate the learning process\nCons:\n- The approach requires manual segmentation of the state space into\ncategories \n- Tested on two new domains and compared against a baseline an another\nalgorithm, so it is not completely clear the generality of the approach.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work proposed to learn causal structure of the environment and use the average causal effect of different categories of the environment, between the current and next state after performing an action as intrinsic reward to assist policy learning. While the reviewers find the ideas presented in the paper interesting and of potential, there are some concerns regarding proper introduction and comparison to related works, and clarity of the algorithm itself. While the two experimental results presented in the paper do show the potential of the work, it is missing an important baseline to disentangle the effectiveness of introducing the causal structure alone vs intrinsic reward. For example, how would A2C with curiosity or surprised based intrinsic reward, which also introduce the surprisingness of the next state as a result of performing an action as additional reward perform on these tasks? 

Decision: Reject