Causal Reasoning from Meta-reinforcement learning

Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents . Here we explore whether modern deep reinforcement learning can be used to train agents to perform causal reasoning . We adopt a meta-learning approach , where the agent learns a policy for conducting experiments via causal interventions , in order to support a subsequent task which rewards making accurate causal inferences . We also found the agent could make sophisticated counterfactual predictions , as well as learn to draw causal inferences from purely observational data . Though powerful formalisms for causal reasoning have been developed , applying them in real-world domains can be difficult because fitting to large amounts of high dimensional data often requires making idealized assumptions . Our results suggest that causal reasoning in complex settings may benefit from powerful learning-based approaches . More generally , this work may offer new strategies for structured exploration in reinforcement learning , by providing agents with the ability to perform—and interpret—experiments . 1 INTRODUCTION . Many machine learning algorithms are rooted in discovering patterns of correlation in data . While this has been sufficient to excel in several areas ( Krizhevsky et al. , 2012 ; Cho et al. , 2014 ) , sometimes the problems we are interested in are fundamentally causal . Answering questions such as “ Does smoking cause cancer ? ” or “ Was this person denied a job due to racial discrimination ? ” or “ Did this marketing campaign cause sales to go up ? ” all require an ability to reason about causes and effects and can not be achieved by purely associative inference . Even for problems that are not obviously causal , like image classification , it has been suggested that some failure modes emerge from lack of causal understanding . Causal reasoning may be an essential component of natural intelligence and is present in human babies , rats and even birds ( Leslie , 1982 ; Gopnik et al. , 2001 ; 2004 ; Blaisdell et al. , 2006 ; Lagnado et al. , 2013 ) . There is a rich literature on formal approaches for defining and performing causal reasoning ( Pearl , 2000 ; Spirtes et al. , 2000 ; Dawid , 2007 ; Pearl et al. , 2016 ) . Here we investigate whether procedures for learning and using causal structure can be produced by meta-learning . The approach of meta-learning is to learn the learning ( or inference ) procedure itself , directly from data . We adopt the specific method of Duan et al . ( 2016 ) and Wang et al . ( 2016 ) , training a recurrent neural network ( RNN ) through model-free reinforcement learning . We train on a large family of tasks , each underpinned by a different causal structure . The use of meta-learning avoids the need to manually implement explicit causal reasoning methods in an algorithm , offers advantages of scalability by amortizing computations , and allows automatic incorporation of complex prior knowledge ( Andrychowicz et al. , 2016 ; Wang et al. , 2016 ; Finn et al. , 2017 ) . Additionally , by learning end-to-end , the algorithm has the potential to find the internal representations of causal structure best suited for the types of causal inference required . 2 PROBLEM SPECIFICATION AND APPROACH . This work probed how an agent could learn to perform causal reasoning in three distinct settings – observational , interventional , and counterfactual – corresponding to different types of data available to the agent during the first phase of an episode . In the observational setting ( Experiment 1 ) , the agent could only obtain passive observations from the environment . This type of data allows an agent to infer associations ( associative reasoning ) and , when the structure of the underlying causal model permits it , to estimate the effect that changing a variable in the environment has on another variable , namely to estimate causal effects ( cause-effect reasoning ) . In the interventional setting ( Experiment 2 ) , the agent could directly set the values of some variables in the environment . This type of data in principle allows an agent to estimate causal effects for any underlying causal model . In the counterfactual setting ( Experiment 3 ) , the agent first had an opportunity to learn about the causal graph through interventions . At the last step of the episode , it was asked a counterfactual question of the form “ What would have happened if a different intervention had been made in the previous time-step ? ” . Next we will formalize these three settings and patterns of reasoning possible in each , using the graphical model framework ( Pearl , 2000 ; Spirtes et al. , 2000 ; Dawid , 2007 ) 1 , and introduce the meta-learning methods that we will use to train agents that are capable of such reasoning . 2.1 CAUSALITY . Causal relationships among random variables can be expressed using causal directed acyclic graphs ( DAGs ) ( see Appendix ) . A causal DAG is a graphical model that captures both independence and causal relations . Each node Xi corresponds to a random variable , and the joint distribution p ( X1 , ... , XN ) is given by the product of conditional distributions of each node Xi given its parent nodes pa ( Xi ) , i.e . p ( X1 : N≡X1 , ... , XN ) = ∏N i=1p ( Xi|pa ( Xi ) ) . Edges carry causal semantics : if there exists a directed path fromXi toXj , thenXi is a potential cause of Xj . Directed paths are also called causal paths . The causal effect of Xi on Xj is the conditional distribution ofXj givenXi restricted to only causal paths . G A E H p ( A ) p ( E|A ) p ( H|A , E ) G→E=e A E H p ( A ) δE=e p ( H|A , E ) An example causal DAG G is given in the figure on the left , where E represents hours of exercise in a week , H cardiac health , andA age . The causal effect ofE onH is the conditional distribution restricted to the path E→H , i.e . excluding the path E←A→H . The variable A is called a confounder , as it confounds the causal effect with non-causal statistical influence . Simply observing cardiac health conditioning on exercise level from p ( H|E ) ( associative reasoning ) can not answer if change in exercise levels cause changes in cardiac health ( cause-effect reasoning ) , since there is always the possibility that correlation between the two is because of the common confounder of age . Cause-effect Reasoning . The causal effect can be seen as the conditional distribution p→E=e ( H|E = e ) 2 on the graph G→E=e above ( right ) , resulting from intervening on E by replacing p ( E|A ) with a delta distribution δE=e ( thereby removing the link from A to E ) and leaving the remaining conditional distributions p ( H|E , A ) and p ( A ) unaltered . The rules of do-calculus ( Pearl , 2000 ; Pearl et al. , 2016 ) tell us how to compute p→E=e ( H|E=e ) using observations from G. In this case p→E=e ( H|E=e ) =∑ Ap ( H|E=e , A ) p ( A ) 3 . Therefore , do-calculus enables us to reason in the intervened graph G→E=e even if our observations are from G. This is the scenario captured by our observational setting outlined above . Such inferences are always possible if the confounders are observed , but in the presence of unobserved confounders , for many DAG structures the only way to compute causal effects is by collecting observations directly from G→E , i.e . by actively intervening on the world to fix the value of the variable E= e and observing the remaining variables . In our interventional setting , outlined above , the agent has access to such interventions . 1This approach typically decouples the challenges of causal induction , i.e . of inferring the structure of the causal graph from data , and that of causal reasoning on the induced graph . The formalism we describe here assumes that the structure of the causal graph is known . In our experiments however , our agents concurrently carry out causal induction . 2In the causality literature , this distribution would most often be indicated with p ( H|do ( E=e ) ) . We prefer to use p→E=e ( H|E=e ) to highlight that intervening on E results in changing the original distribution p , by structurally altering the causal DAG . 3Notice that conditioning on E=e would instead give p ( H|E=e ) = ∑ Ap ( H|E=e , A ) p ( A|E=e ) . Counterfactual Reasoning . Cause-effect reasoning can be used to correctly answer predictive questions of the type “ Does exercising improve cardiac health ? ” by accounting for causal structure and confounding . However , it can not answer retrospective questions about what would have happened . For example , given an individual iwho has died of a heart attack , this method would not be able to answer questions of the type “ What would the cardiac health of this individual have been had they done more exercise ? ” . This type of question requires estimating unobserved sources of noise and then reasoning about the effects of this noise under a graph conditioned on a different intervention . 2.2 META-LEARNING . Meta-learning refers to a broad range of approaches in which aspects of the learning algorithm itself are learned from the data . Many individual components of deep learning algorithms have been successfully meta-learned , including the optimizer ( Andrychowicz et al. , 2016 ) , initial parameter settings ( Finn et al. , 2017 ) , a metric space ( Vinyals et al. , 2016 ) , and use of external memory ( Santoro et al. , 2016 ) . Following the approach of ( Duan et al. , 2016 ; Wang et al. , 2016 ) , we parameterize the entire learning algorithm as a recurrent neural network ( RNN ) , and we train the weights of the RNN with model-free reinforcement learning ( RL ) . The RNN is trained on a broad distribution of problems which each require learning . When trained in this way , the RNN is able to implement a learning algorithm capable of efficiently solving novel learning problems in or near the training distribution . Learning the weights of the RNN by model-free RL can be thought of as the “ outer loop ” of learning . The outer loop shapes the weights of the RNN into an “ inner loop ” learning algorithm . This inner loop algorithm plays out in the activation dynamics of the RNN and can continue learning even when the weights of the network are frozen . The inner loop algorithm can also have very different properties from the outer loop algorithm used to train it . For example , in previous work this approach was used to negotiate the exploration-exploitation tradeoff in multi-armed bandits ( Duan et al. , 2016 ) and learn algorithms which dynamically adjust their own learning rates ( Wang et al. , 2016 ; 2018 ) . In the present work we explore the possibility of obtaining a causally-aware inner-loop learning algorithm . See the Appendix for a more formal approach to meta-learning . 3 TASK SETUP AND AGENT ARCHITECTURE . In the experiments , in each episode the agent interacted with a different causal DAG G. G was drawn randomly from the space of possible DAGs under the constraints given in the next paragraph . Each episode consisted of T steps , and was divided into two phases : information and quiz . The information phase , corresponding to the first T−1 steps , allowed the agent to collect information by interacting with or passively observing samples from G. The agent could potentially use this information to infer the connectivity and weights of G. The quiz phase , corresponding to the final step T , required the agent to exploit the causal knowledge it collected in the information phase , to select the node with the highest value under a random external intervention . Causal graphs , observations , and actions . We generated all graphs onN=5 nodes , with edges only in the upper triangular of the adjacency matrix ( this guarantees that all the graphs obtained are DAGs ) , with edge weights , wji∈ { −1,0,1 } ( uniformly sampled ) , and removed 300 for held-out testing . The remaining 58749 ( or 3N ( N−1 ) /2−300 ) were used as the training set . Each node ’ s value , Xi ∈R , was Gaussiandistributed . The values of parentless nodes were drawn from N ( µ = 0.0 , σ = 0.1 ) . The conditional probability of a node with parents was p ( Xi|pa ( Xi ) ) = N ( µ = ∑ jwjiXj , σ = 0.1 ) , where pa ( Xi ) represents the parents of nodeXi in G. The values of the 4 observable nodes ( the root node , was always hidden ) , were concatenated to create vt and provided to the agent in its observation vector , ot= [ vt , mt ] , wheremt is a one-hot vector indicating external intervention during the quiz phase ( explained below ) .4 In both phases , on each step , t , the agent ’ s action , at , was a discrete choice from the range { 1 ... 2 ( N−1 ) } . Action choices in { 1 ... N − 1 } corresponded to information actions , and choices in { N ... 2 ( N − 1 ) } corresponded to quiz actions . 4While a simple domain provides the most unencumbered test for causal reasoning , we also carried out simulations with more complex causal graphs ( graphs with non-linear connections , and larger graphs of size N = 6 ) and stronger requirements for generalization ( holding-out entire equivalence classes of causal graphs from training ) to demonstrate the robustness of our approach ( see Appendix ) . Information phase . In the information phase , an information action , at , caused an intervention on the at-th node , setting its value to Xat = 5 . We choose an intervention value outside the likely range of sampled observations , to facilitate learning of the causal graph . The observation from the intervened graph , G→Xat=5 , was sampled similarly to G , except the incoming edges toXat were severed , and its intervened value was used for conditioning its children ’ s values . The node values in G→Xat=5 were distributed as p→Xi=5 ( X1 : N\i|Xi=5 ) . If a quiz action was chosen during the information phase , it was ignored , the G values were sampled as if no intervention had been made , and the agent was given a penalty of rt=−5 in order to encourage it to take quiz actions at only during quiz phase . After the action was selected , an observation was provided to the agent . The default length of this phase was fixed to T =N =5 since in the noise-free limit , a minimum of T−1=4 interventions are required in general to resolve the causal structure , and score perfectly on the test phase . Quiz phase . In the quiz phase , one non-hidden node was selected at random to be intervened on externally , Xj , and its value was set to −5 . We chose an intervention value of −5 never previously observed by the agent in that episode , thus disallowing the agent from memorizing the results of interventions in the information phase to perform well on the quiz phase . The agent was informed of this by the observed mT−1 ( a one-hot vector which indicated which node would be intervened on ) , from the final pre-quiz phase time-step , T−1 . Note , mt was set to a zero-vector for steps t < T−1 . A quiz action , aT , chosen by the agent indicated the node whose value would be given to the agent as a reward . In other words , the agent would receive reward , rT =XaT− ( N−1 ) . Again , if a quiz action was chosen during the information phase , the node values were not sampled and the agent was simply given a penalty of rT =−5 . Active vs passive agents . Our agents had to perform two distinct tasks during the information phase : a ) actively choose which nodes to set values on , and b ) infer the causal DAG from its observations . We refer to this setup as the “ active ” condition . To control for ( a ) , we created the “ passive ” condition , where the agent ’ s information phase actions are not learned . To provide a benchmark for how well the active agent can perform task ( a ) , we fixed the passive agent ’ s intervention policy to be an exhaustive sweep through all observable nodes . This is close to optimal for this domain – in fact it is the optimal policy for noise-free conditional node values . We also compared the active agent ’ s performance to a baseline agent whose policy is to intervene randomly on the observable nodes in the information phase , in the Appendix . Two kinds of learning The “ inner loop ” of learning ( see Section 2.2 ) occurs within each episode where the agent is learning from the evidence it gathers during the information phase in order to perform well in the quiz phase . The same agent then enters a new episode , where it has to repeat the task on a different DAG . Test performance is reported on DAGs that the agent has never previously seen , after all the weights of the RNN have been fixed . Hence , the only transfer from training to test ( or the “ outer loop ” of learning ) is the ability to discover causal dependencies based on observations in the information phase , and to perform causal inference in the quiz phase . Agent Architecture and Training We used a long short-term memory ( LSTM ) network ( Hochreiter & Schmidhuber , 1997 ) ( with 96 hidden units ) that , at each time-step t , receives a concatenated vector containing [ ot , at−1 , rt−1 ] as input , where ot is the observation5 , at−1 is the previous action ( as a one-hot vector ) and rt−1 the reward ( as a single real-value ) 6 . The outputs , calculated as linear projections of the LSTM ’ s hidden state , are a set of policy logits ( with dimensionality equal to the number of available actions ) , plus a scalar baseline . The policy logits are transformed by a softmax function , and then sampled to give a selected action . Learning was by asynchronous advantage actor-critic ( Mnih et al. , 2016 ) . In this framework , the loss function consists of three terms – the policy gradient , the baseline cost and an entropy cost . The baseline cost was weighted by 0.05 relative to the policy gradient cost . The weighting of the entropy cost was annealed over the course of training from 0.05 to 0 . Optimization was done by RMSProp with =10−5 , momentum = 0.9 and decay = 0.95 . Learning rate was annealed from 3×10−6 to 0 . For all experiments , after training , the agent was tested with the learning rate set to zero , on a held-out test set . 5 ’ Observation ’ ot refers to the reinforcement learning term , i.e . the input from the environment to the agent . This is distinct from observations in the causal sense ( referred to as observational data ) i.e . samples from a casual structure where there is no information about interventions that have been carried out . 6These are both set to zero for the first step in an episode .

Summary: This paper aims at training agents to perform causal reasoning with RL in three settings: observational (the agent can only obtain one observational sample at a time), interventional (the agent can obtain an interventional sample at a time for a given perfect intervention on a given variable) and a counterfactual setting (the agent can obtains interventional samples, but the prediction is about the case in which the same noise variables were sampled, but a different intervention was performed) . In each of these settings, after T-1 steps of information gathering, the algorithm is supposed to select the node with the highest value in the last step. Different types of agents are evaluated on a limited simulated dataset, with weak and not completely interpretable results.