Causal Induction from Visual Observations for Goal Directed Tasks

Causal reasoning has been an indispensable capability for humans and other intelligent animals to interact with the physical world . In this work , we propose to endow an artificial agent with the capability of causal reasoning for completing goal-directed tasks . We develop learning-based approaches to inducing causal knowledge in the form of directed acyclic graphs , which can be used to contextualize a learned goal-conditional policy to perform tasks in novel environments with latent causal structures . We leverage attention mechanisms in our causal induction model and goal-conditional policy , enabling us to incrementally generate the causal graph from the agent ’ s visual observations and to selectively use the induced graph for determining actions . Our experiments show that our method effectively generalizes towards completing new tasks in novel environments with previously unseen causal structures . 1 INTRODUCTION . Causal reasoning is an integral part of natural intelligence . The capacity to reason about cause and effect has been observed in humans and other intelligent animals as a means of survival ( Blaisdell et al. , 2006 ; Taylor et al. , 2008 ) . Such capacity plays a crucial role for young children in their interaction with the physical world . As behavioral psychology studies have shown , young children discover the underlying causal mechanisms from their play with the world ( Schulz & Bonawitz , 2007 ) , and their knowledge of causality in turn facilitates their subsequent learning of objects , concepts , languages , and physics ( Rehder , 2003 ; Corrigan & Denton , 1996 ) . Nowadays , data-centric methods in artificial intelligence , such as deep networks , have achieved tremendous success in learning associations between inputs and outputs from large amounts of data , such as images to class labels ( He et al. , 2016 ) . However , empirical evidence indicates that the absence of correct causal modeling in these methods is one potential reason for a lack of generalization , causing image captioning models to generate unrealistic captions ( Lake et al. , 2017 ) , deep reinforcement learning policies to fail in novel problem instances ( Edmonds et al. , 2018 ) , and transfer learning models to adapt slower to new distributions ( Bengio et al. , 2019 ) . In this work , we propose to endow a learning-based interactive agent with the capacity of causal reasoning for completing goal-directed tasks in visual environments . Imagine that a household robot enters a new home for the first time . Without prior knowledge of the wiring configuration , it has to toggle the switches and sort out the correspondences between lights and switches , before it can be commanded to turn on the kitchen light or the bathroom light . We refer to the first phase of toggling switches as causal induction , where the agent discovers the latent cause and effect relations via performing actions and observing their outcomes ; and we refer to the second phase of turning on specific lights as causal inference , where the agent uses the acquired causal relations to guide its actions for the completion of a task . To build an effective computational model for causal induction and inference , we have to address generalization towards novel causal relations and new task goals at the test time , both of which can be unseen during training . We cast this as a meta-learning problem of two phases following Dasgupta et al . ( 2019 ) . In the first stage , we use a causal induction model to construct a causal structure , i.e. , a directed acyclic graph of random variables , from observational data from an agent ’ s interventions . In the second stage , we use the causal structure to contextualize a goal-conditional policy to perform the task given a goal . However , in contrast to Dasgupta et al . ( 2019 ) we explicitly construct the causal structure instead of a latent feature encoding , leading to substantially better generalization towards new problem instances in multi-step tasks as opposed to simplistic one-step querying . To this end , we propose two technical contributions : 1 ) an iterative causal induction model with attention , which learns to incrementally update the predicted causal graph for each observed interaction in the environment , and 2 ) a goal-conditioned policy with an attention-based graph encoding , forcing it to focus on the relevant components of the causal graph at each step . We find that by factorizing the induction and inference processes through the use of causal graphs , it generalizes well to unseen causal structures given as few as 50 training causal structures . We compare our approach to using the ground-truth causal structure ( which provides oracle performance ) , a non-iterative architecture which directly predicts the causal structure , and encoding the observation data into the LSTM ( Hochreiter & Schmidhuber , 1997 ) memory of the policy similar to prior work ( Dasgupta et al. , 2019 ) . We demonstrate that our method outperforms the baselines and achieves close to oracle performance in terms of both recovering the causal graph and success rate of completing goal-conditioned tasks , across several task sizes , types , and number of training causal structures . 2 PROBLEM STATEMENT . We formulate the agent ’ s interaction in a Goal-Conditioned Markov Decision Process ( MDP ) defined by a six tuple ( S , A , p , G , r , γ ) , where S is the state space , A is the action space , p : S × A → S defines the transition dynamics , G is the goal space , r : S ×A× G → R is a reward function where r ( s , a , g ) gives the one-step immediate reward conditioned on the goal g ∈ G , and γ is the discount factor . Our goal is to learn a goal-conditioned policy πG : S × G → A that maximizes the expected sum of rewards Rt = ∑∞ k=t γ k−tr ( sk , ak , g ) . In this work , we not only want πG to generalize well across goals in G , but consider a more ambitious aim of making πG generalize across a set of MDPs . We considerM = { M ( 1 ) , M ( 2 ) , . . . , M ( K ) } as the entire set of K MDPs with the same state space and action space but different transition dynamics , where M ( k ) is defined by ( S , A , p ( k ) , G , r , γ ) . The dynamics p ( k ) determines underlying causal relations between states and actions . Taking the same action at the same state could lead to a different next state under a different dynamics . We expect our agent to operate on its first-person vision and has no access to the latent causal relations . It receives high-dimensional RGB observations and has to induce a causal model from observational data . As illustrated in Figure 1 , the overall procedure has two stages : 1 ) we execute a heuristic interaction policy πI to collect a sequence of transitions τ = { ( s1 , a1 ) , ( s2 , a2 ) , . . . } , which is consumed by an induction model F to construct a latent causal model Ĉ = F ( τ ) ; and 2 ) we use the causal model Ĉ to contextualize a goal-conditioned policy πG such that it can perform tasks in the new MDP with novel causal relations . In this work we focus on causal graphs Ĉ which are bipartite with a fixed set of “ cause ” variables mapping to a fixed set of “ effect ” variables . We formulate this as a meta-learning problem ( Dasgupta et al. , 2019 ; Finn et al. , 2017 ) . We partition the set of all MDPsM into two disjoint setsMtrain andMtest . During training , we learn our induction model F and goal-conditioned policy πG withMtrain . During testing , we evaluate whether F is able to learn from the observational data from πI in a novel MDP fromMtest to construct a causal model that can be used by πG to perform tasks in this new MDP . Direct modeling of causal relations in raw pixel space is intractable due to the large dimensionality . Following Chalupka et al . ( 2014 ) , we assume that cause and effect in our problems can be defined on a handful of causal macro-variables . For example , the illuminance of the kitchen ( the agent ’ s visual observation ) is determined by the on and off of the kitchen light ( macro-variable ) , which is caused by toggling the state of the switch that controls the light ( another macro-variable ) . This assumption enables us to construct a directed acyclic causal model Ĉ to represent the causal effects of actions on these macro-variables . Given the set of macro-variables , the induction model F predicts directed edges between them from visual observations . A primary challenge here is the confounders raised from partial observability and spurious correlations in the agent ’ s visual perception . For example , illuminance changes in the kitchen might be due to turning on/off the kitchen light or the living room light . Hence , it requires the agent to disentangle the correct causal relations from visual inputs . 3 METHOD . The goal of our method is to enable a policy to complete goal-conditioned vision-based control tasks in environments with unseen causal structures , given only a short trajectory of observational data in the environment . Prior work ( Dasgupta et al. , 2019 ) has shown promising results on simplistic one-step querying problems using an LSTM-based policy which encodes the interaction into the policy ’ s memory . Our hypothesis is that to generalize in complex multi-step control problems , a more structured induction and policy scheme will be required . To address this , we propose iterative updates and attention bottlenecks in the induction model Ĉ = F ( { ( s1 , a1 ) , ( s2 , a2 ) , . . . } ) and in the policy a = πG ( s , g , Ĉ ) respectively , which we demonstrate significantly improves generalization to unseen causal structures . 3.1 ITERATIVE CAUSAL INDUCTION NETWORK . Inducing the causal structure from raw sensory observations requires accurately capturing the unique effect of each action on the environment , while accounting for confounding effects of other actions . We hypothesize that the causal induction network that best generalizes will be one which disentangles individual actions and their corresponding effect , and only updates the relevant components of the causal graph . We implement this idea in our iterative model , where we begin with an initial guess of the causal structure Ĉ which has all edge weights of 0 ( meaning we assume no causal relationships ) . We then use an Observation Encoder E to map each image of the observational data to an encoding s and compute the state residual Ri = E ( si+1 ) − E ( si ) ( 1 ) between subsequent time steps i and i + 1 . This Ri , which captures the change in state is then concatenated with the corresponding action ai , and then fed into the Edge Decoder module . The output of the Edge Decoder module is an update to the edge strengths of the causal graph ∆Ĉ . This update is applied to each observed transition , that is Ĉt+1 = ∆Ĉ + Ĉt , and at the final layer the whole graph is encoded to do a final edge update before the causal graph is predicted ( see Figure 2 ) . The Edge Decoder takes either the encoded R and a , or the encoded edge matrix of ĈH , and outputs a 1×N soft attention vector α and a 1×N change to the edge weights ∆e , where N is the number of actions in the environment . The attention vector α is used to weight which nodes in the causal graph the edge update ∆e should be applied to . Thus at each iteration the update amounts to : Ĉt+1 = ( α T∆e ) + Ĉt , α = φ ( Rt+1 , at+1 ) ( 2 ) where φ is the Transition Encoder , a fully connected module ( see Appendix A for details ) . Using this attention mechanism further encourages the module to make independent updates , which we observe enables better generalization .

Summary: This paper proposes an interactive agent that tries to infer the underlying causal structure by interacting with the environment; the authors called it "causal induction."  The inferred graph will later help the agent complete goal-directed tasks referred to as a "causal inference" stage. Notably, the agent directly learns from visual inputs.