Deep Reinforcement Learning with Causality-based Intrinsic Reward

1 INTRODUCTION . Reinforcement learning ( RL ) is a powerful approach towards dealing with sequential decisionmaking problems . Combined with deep neural networks , deep reinforcement learning ( DRL ) has been applied in a variety of fields such as playing video games ( Mnih et al. , 2015 ; Vinyals et al. , 2019 ; Berner et al. , 2019 ) , mastering the game of Go ( Silver et al. , 2016 ) and robotic control ( Riedmiller et al. , 2018 ) . However , current DRL algorithms usually learn a black box policy approximated by a deep neural network directly using the state transitions and reward signals , without explicitly understanding the structure information of the environment . Compared with DRL agents , an important reason why humans are believed to be better at learning is the ability to build model on the relations between entities in the environment and then reason based on it . This ability is an important component of human cognition ( Spelke & Kinzler , 2007 ) . As the learning process continues , through interactions with the environment and observations of it , human can gradually understand its actions ’ causal effects on the entities as well as the relations between entities and then reason based on them to figure it out the most important actions to take in order to improve the efficiency . In scenarios that contain multiple entities with complicated relations , optimal policy may be obtained only when the structured relation information is captured and exploited . However , most current DRL algorithms do not consider structured relation information explicitly . The knowledge learned by an agent is implicitly entailed in the policy or action-value function , which are usually unexplainable neural networks . Therefore , whether the relations are well understood and exploited by the agent is unknown . When the environment is with high complexity , blackbox learning of policies suffers from low efficiency , while policy learning over explicit representation of entity relations can significantly boost the learning efficiency . Based on the fact that entities in an environment are often not independent but causally related , we argue that disentangling the learning task into two sequential tasks , namely relational structure learning and policy learning , and leveraging an explicit environmental structure model to facilitate the policy learning process of DRL agents are expected to boost the performance . With the learned relational structure information , the agent performs exploration with a tendency of prioritizing interaction with critical entities , which is encouraged by intrinsic rewards , to learn optimal policy effectively . Taking this inspiration , we propose a deep reinforcement learning algorithm which firstly learns the relations between entities and then recognize critical entity categories and develop an intrinsic reward based approach to improve policy learning efficiency and explainability . The proposed algo- rithm learns a graph to encode the relation information between categories of entities , by evaluating causal effect of one category of entities to another . Thereafter , intrinsic reward based on the learned graph is given to an agent to encourage it to prioritize interaction with entities belonging to important categories ( the categories that are root causes in the graph ) . Previous works also use graphs to provide additional structured information for the agent to assist policy learning ( Wang et al. , 2018 ; Vijay et al. , 2019 ) . However , graphs leveraged by these works are provided by human and thus rely heavily on prior knowledge . Compared with their methods , our algorithm overcomes the deficiency that the graph can not be generated automatically . Our approach requires no prior knowledge and can be combined with existing policy-based or value-based DRL algorithms to boost their learning performance . The key contributions of this work are summarized as follows : • We propose a novel causal RL framework that decomposes the whole task into the structure learning and causal structure aware policy learning . • The learned causal information is leveraged by giving causality based intrinsic reward to an agent , to encourage it to interact with entities belonging to critical categories for accomplishing the task . • We design two new game tasks which contain multiple entities with causal relations as benchmarks to be released to the community . The new benchmarks are designed in such ways that categories of objects are causally related . Experiments are conducted on our designed simulation environments , which show that our algorithm achieves state-of-the-art performance and can facilitate the learning process of DRL agents under other algorithmic frameworks . The paper is organized as follows . In Section 2 , we introduce deep reinforcement learning and Average Causal Effect ( ACE ) , which are key components of this work . Then we illustrate our algorithm in Section 3 in details . In Section 4 , we show the experimental results on the designed environment to demonstrate the effectiveness of our framework . In Section 5 , we introduce previous works that relate to our method . Finally , conclusions and future work are provided in Section 6 . 2 BACKGROUND . 2.1 DEEP REINFORCEMENT LEARNING . An MDP can be defined by a 5-tuple ( S , A , P , R , γ ) , where S is the state space , A is the action space , P is the transition function , R is the reward function and γ is the discount factor ( Sutton & Barto , 2018 ) . A RL agent observes a state st ∈ S at time step t. Then it selects an action at from the action space A following a policy π ( at|st ) , which is a mapping of state space to action space . After taking the action , the agent receives a scalar reward rt according toR ( st , at ) . Then the agent transits to the next state st+1 according to the state transition probabilityP ( st+1|st , at ) . A RL agent aims to learn a policy that maximizes the cumulative discount reward , which can be formulated as Rt = ∑T k=0 γ krt+k where T is the length of the whole episode . In the process of learning an optimal policy , a RL agent generally approximates the state-value function Vπ ( s ) or the action value function Qπ ( s , a ) . The state value function is the expected cumulative future discounted reward from a state with actions sampled from a policy π : Vπ ( s ) = Eπ [ T∑ k=0 γkrt+k|St = s ] . ( 1 ) Deep Reinforcement Learning ( DRL ) which combines Deep Neural Networks ( DNNs ) with RL can be an effective way to deal with high-dimensional state space . It benefits from the representation ability of DNNs , which enable automatic feature engineering and end-to-end learning through gradient descent . Several effective algorithms have been proposed in the literature and we use A2C in this paper as our basic algorithm , which is a synchronous version of A3C ( Mnih et al. , 2016 ) . A2C consists of a variety of actor-critic algorithms ( Sutton et al. , 2000 ) . It directly optimizes the policy πθ parameterized by θ to maximize the objective J ( θ ) = Eπ [ ∑T k=0 γ krt+k ] by taking steps in the direction of ∇θJ ( θ ) . The gradient of the policy can be written as : ∇θJ ( θ ) = Eπ [ ∇θ log πθ ( a|s ) Aπ ( s , a ) ] , ( 2 ) where Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) is the advantage function . The advantage function can be estimated by one-step TD-error Âπ ( st , at ) = rt + γVφ ( st+1 ) − Vφ ( st ) , where Vφ ( s ) is the approximation of the state-value function V π ( s ) parameterized by φ . 2.2 CAUSAL NEURAL NETWORK ATTRIBUTIONS . Attributions are defined as the effect of an input feature on the prediction function ’ s output ( Sundararajan et al. , 2017 ) and Chattopadhyay et al . ( 2019 ) propose a neural network attribution methodology built from first principles of causality . Chattopadhyay et al . ( 2019 ) views the neural network as a Structural Causal Model ( SCM ) , and proposes a new method to compute the Average Causal Effect of an input neuron on an output neuron based on the do ( . ) calculus ( Pearl , 2009 ) . Definition 1 . ( Average Causal Effect ) . The Average Causal Effect ( ACE ) of a binary random variable x on another random variable y is commonly defined as E [ y|do ( x = 1 ) ] − E [ y|do ( x = 0 ) ] . While the above definition is for binary-valued random variables , the domain of the function learnt by neural networks is usually continuous . Given a neural network with input layer l1 and output layer ln , we hence measure the ACE of an input feature xi ∈ l1 with value α on an output feature y ∈ ln as : ACEydo ( xi=α ) = E [ y|do ( xi = α ) ] − baselinexi . ( 3 ) Definition 2 . ( Causal Attribution ) . The causal attribution of input neuron xi for an output neuron y is defined as ACEydo ( xi=α ) . In Equation 3 , an ideal baseline would be any point along the decision boundary of the neural network , where predictions are neutral . However , Kindermans et al . ( 2019 ) showed that when a reference baseline is fixed to a specific value ( such as a zero vector ) , attribution methods are not affine invariant . Therefore , Chattopadhyay et al . ( 2019 ) proposed the average ACE of xi on y as the baseline value for xi : baselinexi = Exi [ Ey [ y|do ( xi = α ) ] ] ( 4 ) In this paper , we use causal attribution method to infer the relations between entities . 3 METHOD . In this section , we present a novel DRL framework named CARE ( CAusal RElation ) that enables the agent to infer the causal relationship between entities . Figure 1 illustrates the overall framework of CARE . CARE adopts a two-stage training paradigm . In the first stage , the agent learns a model of the environment by minimizing the prediction error of the state transition . Then , we calculate ACE values between categories of entities , which are used for constructing a causal graph G. When G is at hand , we are able to obtain the causal ordering of all categories , that is , a permutation such that nodes ranked lower can not cause nodes ranked higher . This order is used to measure the importance of the categories , and intrinsic reward is given based on the causal ordering . Specifically , the agent receive a one-step “ enhanced ” reward rt where : rt = r G t + r ext t , ( 5 ) where rextt is the original extrinsic reward given by the environment , and r G t is the intrinsic reward , designed by the learning algorithm to encourage the agent to maximize the effect of its behaviour on the change of states of critical entities . We describe the details in later sections . This section is organized as follow . In section 3.1 , we first introduce category-oriented state factorization . In section 3.2 , we describe how to get the relation graph G and in section 3.3 , we show how to calculate the intrinsic reward rGt .

Summary: The paper proposes a deep reinforcement learning algorithm of advantage actor-critic (A2C), which ﬁrstly learns the causal structure of the environment and then leverages the learned causal information to assist policy learning. The causal structure is computed by calculating Average Causal Effect (ACE) between different categories of entities, and the authors assume that an intrinsic reward is given to encourage the agent to interact more with critical entities of the causal graph. Two experiments were conducted on simulation environments (Shepherd and Ant tasks) and demonstrated the effectiveness in obtaining the rewards and interpretable and accurate detection of the true graph.