Causal Induction from Visual Observations for Goal Directed Tasks
Authors: Suraj Nair
Yuke Zhu
Silvio Savarese
Li Fei-Fei
Venue:  (2020)
Keywords: meta-learning
causal reasoning
policy learning

Abstract: Causal reasoning has been an indispensable capability for humans and other intelligent animals to interact with the physical world. In this work, we propose to endow an artificial agent with the capability of causal reasoning for completing goal-directed tasks. We develop learning-based approaches to inducing causal knowledge in the form of directed acyclic graphs, which can be used to contextualize a learned goal-conditional policy to perform tasks in novel environments with latent causal structures. We leverage attention mechanisms in our causal induction model and goal-conditional policy, enabling us to incrementally generate the causal graph from the agent's visual observations and to selectively use the induced graph for determining actions. Our experiments show that our method effectively generalizes towards completing new tasks in novel environments with previously unseen causal structures.

Under review as a conference paper at ICLR 2020
CAUSAL INDUCTION FROM VISUAL OBSERVATIONS FOR GOAL DIRECTED TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
Causal reasoning has been an indispensable capability for humans and other in- telligent animals to interact with the physical world. In this work, we propose to endow an artificial agent with the capability of causal reasoning for completing goal-directed tasks. We develop learning-basse based approaches to inducing causal knowledge the form of directed acyclic graphs, which can be used to contex- tualize learned goal-conditional policy to perform tasks in novel environments with latent causal structures. We leverage attention mechanisms in our causal induction model and goal-conditional policy, enabling to incrementally generate the causal graph from the agent's visual observations and to selectively use the induced graph for determining actions. Our experiments show that our method effectively generalizes towards completing new tasks in novel environments with previously unseen causal structures.
INTRODUCTION
Causal reasoning is an integral part of natural intelligence. The capacity to reason about cause and effect has been observed in humans and other intelligent animals as means of survival (Blaisdell ctt al., 2006; Taylor et al. 2008). Such capacity plays crucial role for young children in their interaction with the physical world. As behavioral psychology studies have shown, young children discover the underlying causal mechanisms from their play with the world (Schulz & Bonawitz [2007). and their knowledge of causality in turn facilitates their subsequent learning of objects, concepts, languages, and physics (Rehder 2003 Corrigan & Denton! 1996).
Nowadays, data-centric methods in artificial intelligence, such as deep networks, have achieved tremendous success in learning associations between inputs and outputs from large amounts of data, such as images to class labels (He et al. 2016). However, empirical evidence indicates that the absence of correct causal modeling in these methods is one potential reason for lack of generalization, causing image captioning models to generate unrealistic captions (Lake et al. 2017) deep reinforcement learning policies to fail in novel problem instances (Edmonds et al. 2018). and transfer learning models to adapt slower to new distributions (Bengio et al. 2019).
In this work, propose endow learning- interactive agent with the capacity of causal reasoning for completing goal-directed tasks in visual environments. Imagine that a household robot enters new home for the first time. Without prior knowledge of the wiring configuration, it has to toggle the switches and sort out the correspondences between lights and switches, before it can be commanded to turn on the kitchen light or the bathroom light. We refer the first phase of toggling switches as causal induction, where the agent discovers the latent cause and effect relations via performing actions and observing their outcomes; and we refer to the second phase of turning on specific lights as causal inference, where the agent uses the acquired causal relations to guide its actions for the completion of a task. To build an effective computational model for causal induction and inference, we have address generalization towards novel causal relations and new task goals at the test time, both of which can be unseen during training.
We cast this as a meta-learning problem of two phases following Dasgupta et al. (2019). In the first stage, we use a causal induction model to construct a causal structure, i.e., a directed acyclic graph of random variables, from observational data from agent's 's interventions. In the second stage, use the causal structure to contextualize goal-conditional policy to perform the task given a goal. However, in contrast Dasgupta al. (2019) we explicitly construct the causal structure instead of

---

Under review as a conference paper at ICLR 2020
Training Step #1
Training Step #2
Testing
Train causal induction model F in set of training environments
Fix train goal-conditioned policy Ty in the same environments
Fix F, Fix n.. evaluate on a test environment
with unseen causal structure
interaction policy
state
state
test eriiironment
1
train
Exy
TT
Ex Eees
Fest
action
action
training environments
action
action
state
T
goal- conditioned policy
k train
k observational data
observational
causal induction model
predicted
causal induction
predicted causal structure
causal
structure
model
Figure I: Overview Causal Induction and Inference Procedure. During training each episode
samples one of K training environments and uses the interaction policy æΙ probe the enviromment and collect trajectory of visual observations. Using supervised learning we train the causal induction model F, which takes as input the trajectory of observational data and constructs Ć, the estimate of Cl which captures the underlying causal structure. Then, the predicted structure Ô' is provided input the policy मद conditioned goal g. which learns to the causal model efficiently complete specified goal in the training environments. At test time, F and TG are fixed and the agent evaluated on new environments with unseen causal structures.
latent feature encoding, leading to substantially better generalization towards new problem instances multi-step tasks as opposed to simplistic one-step querying.
To this end, we propose two technical contributions: I) an iterative causal induction model with atten- tion, which learns to incrementally update the predicted causal graph for each observed interaction in the environment, and 2) a goal-conditioned policy with an attention basdd graph encoding, forcing focus on the relevant components of the causal graph each step. We find that by factorizing the induction and inference processes through the use of causal graphs, it generalizes well to unseen causal structures given as few as 50 training causal structures. We compare our approach to using the ground-truth causal structure (which provides oracle performance), non-iterative architecture which directly predicts the causal structure, and encoding the observation data into the LSTM (Hochreiter Schmidhuber 1997) memory of the policy similar to prior work (Dasgupta et al.) 2019). We demonstrate that our method outperforms the baselines and achieves close to oracle performance in terms of both recovering the causal graph and success rate of completing goal-conditioned tasks, across several task sizes, types, and number of training causal structures.
PROBLEM STATEMENT
We formulate the agent's interaction in a Goal- Cold-Conitionedd Markov Decision Process (MDP) defined by six tuple A, p. G, 7),, where S is the state space, A is the action space, p S x A - S defines the transition dynamics, G is the goal space, r S A x G IR is reward function where r(s, g) gives the one-step immediate reward conditioned on the goal g E G, and 7 is the discount factor. Our goal is to learn goal-conditioned policy मद
+ A that maximizes the expected
sum of rewards R2 Σ k k ( (k.g).
In this work, we only want मद generalize well across goals in G. but consider more ambitious aim of making Th generalize across set of MDPs. We consider M (1(1) .M(2) ..... M(K) as the entire set of K" MDPs with the same state space and action space but different transition dynamics, where (A) is defined by (S,A, (*), G.r, r)) The dynamics p determines underlying causal relations between states and actions. Taking the same action at the same state could lead to different next state under different dynamics. We expect our agent to operate on its test-person vision and has access to the latent causal relations. It receives high-dimensional RGB observations and has
2

---

Under review as a conference paper at ICLR 2020
Edge Decoder
1
Input
weighting
Obsarvation Encedeer
Observation Encodar
Observation Encodder
Attertion
Edge Update
a
state residual
Flattened
Transition Encecder
Transition Encoder
Encodar
Predicted
Edge Decador
Edge Decoder
Edge Docader
Graph
Update
Induced
Causal Graph
Go
Gn
Figure 2: Iterative Causal Induction Network. Our iterative network architecture for inducing the causal structure from a visual trajectory of observational data with horizon H. First each frame is encoded into latent state embedding s. Then the difference between state embeddings across time steps (state residual) is computed, and concatenated with the corresponding action. This fed into the Edge Decoder module, which predicts edge update, well as attention vector which is used to weight how the edge update is applied to nodes. This results an estimate of the causal graph O at the jth iteration. On the last step one more edge update based on the current graph is applied, and final predicted graph Ó is outputted.
to induce a causal model from observational data. As illustrated in Figure [ the overall procedure has two stages: 1) we execute a heuristic interaction policy ςy to collect a sequence of transitions {(s1, a1), (82, -.). which is consumed by an induction model F to construct latent causal model Ö = F(r); and 2) we use the causal model Ć contextualize goal-conditioned policy mo such that it can perform tasks in the new MDP with novel causal relations. In this work we focus on causal graphs C' which are bipartite with fixed set of "cause" variables mapping "effect" variables. We formulate this as a meta- leat-llnnning problem (Dasgupta et al. 2019 Finn et al., 2017). We partition the set of all MDPs M into two disjoint sets Merain and Mtest. During training, we learn our induction model F and goal-conditioned policy TG with Mirain During testing, we evaluate whether F is able to learn from the observational data from πy in a novel MDP from Meest to construct a causal model that can be used by म. to perform tasks in this new MDP.
fixed set of
Direct modeling of causal relations in pixel space is intractable due to the large dimensionality. Following Chalupka et al. (2014), we assume that cause and effect in our problems can be defined on handful of causal macro- variablee.............. For example, the illuminance of the kitchen (the agent's 's visual observation) is determined by the on and off of the kitchen light (macro-variable). which is caused by toggling the state of the switch that controls the light (another macro-variable)) This assumption enables us to construct a directed acyclic causal model Ò to represent the causal effects of actions on these ancro-variables. Given the set of macro-variables, the induction model F predicts directed edges between them from visual observations. A primary challenge here is the confounders raised from partial observability and spurious correlations in the agent's visual perception. For example, illuminance changes in the kitchen might be due to turning on/off the kitchen light or the living room light. Hence, it requires the agent disentangle the correct causal relations from visual inputs.
METHOD
The goal of our method is to enable policy to complete goal-conditioned vision-based control tasks in enviromments with unseen causal structures, given only a short trajectory of observational data in the environment. Prior work (Dasgupta et al. 2019) has shown promising results on simplistic one-step querying problems using an LSTM-based policy which encodes the interaction into the policy's memory. Our hypothesis is that to generalize in complex multi-step control problems, more
3

---

Under review as a conference paper at ICLR 2020
structured induction and policy scheme will be required. To address this, we propose iterative updates and attention bottlenecks in the induction model C ((1 a1), (2)..)).....) and in the policy TT(s, 9, C') respectively, which we demonstrate significantly improves generalization unseen causal structures.
3.1 ITERATIVE CAUSAL INDUCTION NETWORK
Inducing the causal structure from raw sensory observations requires accurately capturing the unique effect of each action on the environment, while accounting for confounding effects of other actions. We hypothesize that the causal induction network that best generalizes will be one which disentangles individual actions and their corresponding effect, and only updates the relevant components of the causal graph.
We implement this idea in our iterative model, where we begin with an initial guess of the causal structure Ö' which has all edge weights of 0 (meaning we assume no causal relationships). We then use an Observation Encoder E to map each image of the observational data to an encoding s and compute the state residual
(I) between subsequent time steps i and i This Rg which captures the change in state is then concatenated with the corresponding action ag and then fed into the Edge Decoder module. The output of the Edge Decoder module is an update to the edge strengths of the causal graph AC. This update is applied to each observed transition, that is C.+1 ÁČ + Ct, and at the final layer the whole graph is encoded to do final edge update before the causal graph is predicted (see Figure 2 The Edge Decoder takes either the encoded R and a, or the encoded edge matrix of ČH. and outputs a x N soft attention vector a and a x N change to the edge weights Ae, where N is the number of actions in the environment. The attention vector graph the edge update Ae should be applied to. Thus at each iteration the update amounts to:
R4 = E(s++1) - E(ss)
used weight which nodes in the causal
Ć++1 = (a Ae) + Ce = o( Re+1, a++1)
(2)
where ø is the Transition Encoder, fully connected module (see Appendix A for details). Using this attention mechanism further encourages the module make independent updates, which we observe enables better generalization.
3.2 LEARNING GOAL- COND-CONDI POLICIES
The objective of the policy is given an initial image so, goal image g, and the predicted causal structure Č, reach the goal within T time steps. Additionally, the policy To (T, g. C') is reactive one, and thus can only solve the goal- coal-conditioned task if it can learn to use the predicted causal structure Ć. That is - since the policy has no memory, it cannot learn to induce the graph internally during inference time, and thus must use the causal graph Ć.
We hypothesize that like the causal induction model, the policy which best generalizes is one which learns to focus exclusively on the edges the causal graph which are relevant to the current step of the task. To that end we propose an attention bottleneck in the graph encoding, which encourages the policy select edges pertaining to "effect" each step, which enables better generalization.
Specifically, the policy encodes the current image s and goal image g. Based on this encoding outputs an attention vector of size 1 N over the "effects" in the causal graph. This vector is used to perform a weighted sum over the outputs of the N x N causal graph (N causes, N effects, and edges between them), resulting in size N vector of the selected edges e. The selected edges and visual encodings are used to output the final action:
a = 22 Eg (s. g). (2(e)) e = Č. a (E,
where Ey, has identical architecture to the image encoder in F, but which encodes both current and goal image and o; are all fully connected layers (see Figure
3.3 MODEL TRAINING
The induction network F is trained using supervised learning in the limited set of training environ- ments, specifically to minimize the & reconstruction loss between the ground-truth causal graph C and the predicted causal graph č = F([(s1, 01), a2)...)....
4

---

Under review as a conference paper at ICLR 2020
Action
One-to-One
Ome-to-Many
fully connected
goal image
fully connected
Obsernation Encoder
Attention i
Many-to-One
Masterswitch
current
connected
image
Predicted
masterswitch
light
Graph C
Figure 3: Types of Causal Structures (Left) We explore four types of causal structures, ONE-TO- ONE, ONE-TO-MANY, MANY-TO-ONE and MASTERSWITCH. These cover a standard causal mapping, common cause causal patterns, common effect causal patterns, and causal chains. Goal- Conditioned Policy (Right). The policy takes as input the current image, goal image, and predicted causal graph C. The current image and goal image concatenated channel wise and encoded. This encoding is used to predict attention vector over the "effects" in C which extract the relevant edges, which is then concatenated with the image encoding to predict the action.
The policy Fद is trained using the DAgger (Ross et al. 2010) algorithm by imitating a planner using the ground-truth causal graph the training environments. Then nद tested in unseen environments with only visual inputs and goal images. Specifically, in the training environments, the planner uses the ground-truth graph and privileged low dimensional state/goal information to compute an optimal plan to the goal, yielding deterministic and uni-modal online expert. At each time step, the expert's action is added to the replay buffer of the policy, which is then trained using a standard cross entropy loss to imitate the expert given the current image and goal image. The policy is also injected with e-greedy noise during training, with e 0.3. Network architectures and additional training details can be found in the appendix.
EXPERIMENTS
Through our experiments, we investigate three complementary questions: 1) Does our iterative induction network enable better causal graph induction?, 2) Does our attention bottleneck in the graph encoding in enable the policy to generalize better to unseen causal structures?, and 3) By combining our proposed F and TG, are we able to outperform the current state-of-the-ae Dasgupta et al. (019) visual goal-directed tasks?
4.1 EXPERIMENTAL SETUP
Task Definition: We examine the multi-step task of light switch control. In particular an agent has control of N switches (macro-variables)) which have some underlying causal structure of how they control N lights (macro-variables). However, the macro-variables of the lights manifest themselves in noisy visual observations, whose partial observability and overlap result in confounding factors. The objective of the agent is starting from an initial state to control the switches to reach a specified lighting goal g, where both the state and goal are x 32 3 images. However specified in the problem setup, the underlying causal structure is unknown to the agent, all that is provided is limited observational data from the environment. Thus the agent must (1) induce the causal structure between the switches and lights from the observational data, then (2) use it to reach the goal.
We explore different types of causal patterns between the switches and lights (See Figure 3 The first type of causal structure is ONE-TO- in which each switch maps to one light. The second type of causal structure is MANY-TO-ONE (Common Effect (Keil (0066), where each switch controls one light, but multiple switches may control the same light. The third type is ONE-TO-MANY (Common Cause (Kill 2006)), where all lights are controlled by at most one switch, but a single switch may control more than one light. Lastly,
also explore the MASTERSWITCH domain in which there a Causal Chain (Keil (006) where only once one master switch is activated can the other switches causal effects be observed, applied on top of ONR-TO-ONE causal structure.
5

---

Under review as a conference paper at ICLR 2020
One-to-One
One-to-Meny
Many-to-One
Masterswitch
5 Switches
0.8-
0.8
0.8
se
6.6
0.6
0.6
0.6
2.44
0.4
0.4
0.4
0.2
0.2
0.0
0.0
1.0
1.0
1.0
1.0
7 Switches
0.8-
0.8-
0.8
0.9
Stote
0.6
0.6
0.6
0.4
50
100 Structures
500
10
100
500
10
100 Structures
300
50
100
500
Number
Structures
Structures
TCIN
ICIN (No Attn)
ICIN (Ours)
Figure 4: Fl Score on Unseen Causal Structures, The Fl Scores of edges on the predicted causal structure compared to the ground-truth on unseen causal structures. We compare across variable numbers of seen structures (10, 50, 100, 500) and problem size (5, 7). Our iterative approach with attention outperforms the comparisons across almost all settings.
Causal Graph:
We represent the causal structure C as a bipartite graph with directed edges between N switch variables and M light variables in the environment, with edge strength corresponding to the likelihood that that switch controls the indicated light, with additional N edges in the MASTERSWITCH setting. In practice, we use M N, so the space of possible edges is x N, but proposed approach can be applied for arbitrary number of "cause" or "effect" variables.
The trajectory of observational data from which we induce the causal structure in visual space, consisting of 32 x 32 x 3 image well as size A = N action vector for each timestep of the H timestep trajectory. The visual scene consists of the lights mounted in a MuloCo (Todorov et al. 2012) simulated house with 3 rooms, where the lights mounted in the rooms and hallway, and the effect of each light is rendered onto the floor when they are turned on, which is captured by a top down camera. The illumination from the lights overlap, resulting in confounding factors, which must be disentangled by the model in order to correctly predict the causal structure. The state space (and goal space) of the policy are also 32 32 images of the same environment and camera. The action space A consists of size N, one discrete action for each switch. During policy learning, the goals are sampled uniformly from the space possible lighting configurations under the environments causal structure, i.c., the set of all reachable states in the environment.
Environment Setup:
The observational data is collected using heuristic interaction agent Ty, which executes a simple policy. In the all but the MASTERSWITCH case, the policy simply takes each action once (horizon H = N). In the MASTERSWITCH setting the exploratory agent presses each switch until one has effect on the environment, and then proceeds press each of the other switches (horizon up to = 2
Collecting Observational Data:
4.2 EVALUATION METHODS
We evaluate the following methods and baselines examine the effectiveness of our causal induction model and goal-conditioned policy.
First examine how much our iterative causal induction network (ICIN) improves performance on inducing the causal graph we compare against non-iterative induction model which uses temporal convolutions (TCIN), as well as an ablation of our method which uses an iterative model without the attention mechanism (ICIN (No Attn)). We compare these approaches based on Fl score of recovering the ground-truth causal graph.
Next, we compare the performance of the goal conditioned policy using all variants (ICIN, ICIN (No Attn), TCIN), compared to the previous work of Dasgupta et al. (2019) which induces the graph using the latent memory of the policy (Memory). Specifically, memory, and before running DAgger, feed the interaction trajectory one step at a time through the policy. This end-to-end approach allows the causal graph be encoded into the latent memory of the policy, to be used when doing a goal-din goal-directed task. We also compare to Memory (RL/Low Dim) version of Dasgupta et al. 2019) which has access privileged state information (ground-truth
provide the policy गद with LSTM

---

Under review as a conference paper at ICLR 2020
One to One
One-to Many
Mony-to-One
Masterswichh
1.0
1.0
1.0
1.0
5 Switches
0.8-
0.8
0.6
0.4
0.4
0.2
0.2
0.0
10
50
100
10
50
100
0.0
10
50
160
0.0
10
50
160
Number of Seen Causal Structures
Number of Sean Caural Structures
Number of Seen Cousal Strutures
Number of Causal Structures
One One
One Kany
Many-to-One
Switches
1.9
1.0
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
10
50
100
500
10
50
150
500
0.0
10
50
100
500
Number of Seon Causal Structures Memory (RL/Low Dim)
Number of Seen Causal Structures
Number of Seen Cousal Structures
Memory
+ TCIN
+
ICIN (No Attn)
+ ICIN (Ours)
Oracle
Figure 5: Policy Success Rates (Unseen Causal Structures). The final success rates of the goal- conditioned policy for each method on unseen causal structures for either 10, 50, 100, or 500 seen causal structures for 5 or 7 switches. Our iterative approach achieves the best generalization on unseen tasks across almost all settings.
states), and is trained using model-free reinforcement learning with dense reward. In this setting the same visual interaction trajectory is encoded into the policy's 's LSTM memory, but the actual policy receives binary vector for state and goal and is trained using the PPO algorithm (Schulman et al.. 2017). We also add a comparison to using the ground-truth causal graph (even at test time) as an Oracle (Oracle), which provides an upper bound on performance. All methods are compared based on success rates in unseen environments. Implementation details can be found in Appendix [
Lastly, understand how critical the attention bottleneck in the goal-conditioned policy is for generalization, we compare the success rates in unseen causal structures of the goal-come coal-condition policy using a graph induced by ICIN, with and without the attention bottleneck.
4.3 CAUSAL INDUCTION EVALUATION
First we examine our approach's ability induce the causal model from the trajectory of observational data. We report the Fl score (threshold=0.5) between edges of the predicted causal graph and ground- truth causal graph. We compute the Fl scores across up to 100 unseen tasks given 10, 50, 100, or 500 seen structures in the 5 and switch problems (see Figure 4 We observe that across almost all settings, our iterative approach with attention (ICIN) dominates. While our method without attention generally outperforms the non-iterative bascline, both fall significantly behind our final approach, suggesting that the modularity that attention provides plays large role in enabling generalization. Furthermore, observe that iterative attention approach especially outperforms the others when there are less training causal structures, which would suggest that by forcing the network to make independent updates, it able learn general method for induction with limited training examples. This is likely because the attention forces the model to learn to update a single edge given a single observation, while being agnostic to the total graph, which is far less likely to overfit the training structures. Qualitative examples of ICIN can be found in Appendix
4.4 GOAL-CONDD POLICY EVALUATION
We examine the success rate of the converged policy म on 500 trials in unseen causal structures with randomized goals in F We observe most settings that the Memory based approach of Dasgupta et al. (2019) provides a strong baseline. outperforming the TCIN and ICIN (No Attn) baselines. We suspect that it learns that best imitate the expert, has to encode relevant information from the interaction trajectory into its latent memory, implicitly performing induction. While this works, it generalizes to unseen structures much worse than our proposed method, likely due to the compositional structure of our approach. The memory baseline which uses low dimensional states (Memory (RL/Low Dim)), and is trained via model-free RL also performs well, in fact beating our approach
few cases in the 7 switch, MANY-TO-ON setting. However in general the performance of this approach is much lower than ours, and likely the cases in which does succeed can be attributed to its use of privileged information (ground-truth states) instead of visual
7

---

Under review as a conference paper at ICLR 2020
observations. In almost all cases ICIN significantly outperforms all baselines. In fact, in the 5 switch case our ICIN method nearly matches the Oracle, suggesting that it almost perfectly induces C.
Finally, study the importance of our proposed attention bottleneck in the graph encoding in TG(s, C)+ which forces the policy to focus only relevant edges at each timestep. We examine the success rate of using T with the attention bottleneck compared to just flattening O given the current image, goal image, and predicted causal graph from ICIN.
We find that using the attention bottleneck in the graph encoder of TG yields roughly 10% increase in success rate in the ONE-TO- ONE (1:1) and MASTERSWITCH (MS) cases, and roughly 40% increase in success rate the ONE-TO-MANY (1:K) and MANY-TO-OW (K:1) cases. This is because by encouraging the policy to pick relevant edges, it has led to a modular policy which can 1) identify the changes it wants to make in the environ- ment and 2) predict the necessary action based on the causal graph C; enabling better generalization.
TG
1:1
1:K
K:1
MS
No Attn
0.84
0.51
0.59
0.90
Ours
1.0
0.95
0.94
0.98
Table 1: Policy Attention Bottleneck.
We observe in that across all settings using the attention bottleneck in the graph en- coder of the policy significantly improves performance. Tested on 5 switches and 50 seen structures.
5 RELATED WORK
Causal reasoning has been extensively studied by a broad range of scientific disciplines, such as social sciences (Yee) 1996), medical sciences (Kuipers & Kassirer 1984), and econometrics (Zellner) 1979). In causality literature, structural causal model (SCM) (Pearl 2009) has offered formal framework of modeling causation from statistical data and counterfactual reasoning. SCMs are directed graph that represents causal relationships between random variables. Both causal induction (constructing SCMs from observational data) (Shimizu et al. 2006 Hoyer et al., 2009 Peters et al. 2014 Ortega & Braun 2014) and causal inference (using SCMs to estimate causal effects) (Bareinboim et al. 2015 Bareinboim & Pearl 2016) algorithms have been developed. Conventional methods have limited applicability in complex domains where observational data is high-dimensional and partially observable. Recent work has shown that causal learning can take advantage of the representational
power of deep learning methods for inducing causal relationships from interventions (Dasgupta et al., 2019) and for improving policy learning via counterfactual reasoning (Buesing et al.] (2018). However, they have focused on toy-sized problems with low-dimensional states. In contrast, our model induces and makes use of the causal structure for complex interactive tasks from raw image observations.
Lake et al. (2017) are among the first to discuss the limitations of state-of-the deep learning models in causal reasoning. There has been growing amount of efforts in marrying the complementary strengths of deep learning and causal reasoning. Causal modeling has been explored in several contexts, including image classification (Chalupka et al.) 2014), generative models (Kocaoglu lu et al.) 2017), robot planning (Kurutach et al. 2018), policy learning (Buesing al., 2018), and transfer learning (Bengio et al.) 2019). Pioneer work on causal discovery with deep networks has applied to time series data in healthcare domains (Kale et al. 2015 Nauta et al. 2019). In addition, adversarial learning (Kalainathan et al. 2018), graph neural networks (Yu et al.) 2019), and gradient-based DAG learning (Lachapelle et 2019)) have been recently introduced to causal discovery, but largely focusing on small synthetic datasets, Most relevant to ours are Dasgupta et al. (2019) and Edmonds] al. 2018), which investigated causal reasoning in deep reinforcement learning agents. In contrast to them, our method directly learns an explicit causal structure from raw observations to solve multi-step. goal-conditioned tasks.
Generalization to new environments and new goals has been a central challenge for lcarning- based interactive agents. This problem has been previously studied in in the context of domain adapta- tion (Tzeng et al. 2015| [Png et al. (017). system identification (Yu et al., 2017] Zhou et al. (201)) meta-learning (Finn et al. 2017 Sæmundsson et al., 2018), and multi-task learning (Andrychowicz
al. 2017 Schaul et al. 2015). These works have addressed variations in dynamics, visual appear- ances, and task rewards, while assuming fixed causal structures. Instead, we focus on changes in the latent causal relationships that determine the preconditions and effects of actions. We do so by leveraging attention mechanisms to focus only on relevant parts of the data, and idea also explored in prior work (Bengio 2017).
8

---

Under review as a conference paper at ICLR 2020
6 CONCLUSIONS
We have proposed novel techniques for 1) causal induction from raw visual observations and 2) causal graph encoding for goal-conditioned policies, both of which lead better generalization unseen causal structures. Our key insight is that by leveraging iterative predictions and attention bottlenecks, facilitates our causal induction model and goal-conditioned policy focus on the relevant part of the causal graph. Using this approach we show better generalization towards novel problem instances than previous works with limited training causal structures.
In this work, we induce the causal structure from observational data collected by a heuristic policy. We plan explore more complex tasks where probing the environment to discover the causal structure requires more sophisticated strategies, and develop algorithms that jointly learn the interaction policy.
REFERENCES
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. CoRR, abs/1707.01495, 2017.
Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 113(27):335--352, 2016.
Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: causal approach. In Advances in Neural Information Processing Systems, pp. 1342-1350, 2015.
Yoshua Bengio. The consciousness prior. CoRR, abs/1709 2017. URL http:://axxi
orq/abs/ 1709. 08568
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A veta-transsfer objective for learning to disentangle causal mechanisms. arXiv preprint 1rXiv:1901.109) 2019.
Aaron P Blaisdell, Kosuke Sawa, Kenneth Leising, and Michael R Waldmann. Causal reasoning in rats. Science, 31 1016663) 2006.
Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean-Baptiste Lespiau, and Nicolas Heess. Woulda, coulda, shoulda: Counterfactuallyy-nn -uidedd policy search. arXiv preprini arXiv: 1811 1..6722, 2018
Krzysztof Chalupka, Pietro Perona, and Frederick Eberhardt. Visual causal feature learning, arXiv preprint arXiv: arXiv: 1412.2309, 2014.
Roberta Corrigan and Peggy Denton. Causal understanding as a developmental primitive. Develop- mental review, (6(2):162-200 1996.
Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson. Causal reasoning from retafreeinorremment learning. arXiv preprint arXiv:1901.08162 2019.
M. Edmonds, J. Kubricht, C. Summers, Y. Zhu, B. Rothrock, S.C. Zhu, and H. Lu. Human causal transfer: Challenges for deep reinforcement learning. Annual Meeting of the Cognitive Science Society, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. CoRR, abs/ 1e//10033034000 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Conference Computer Vision and Pattern Recognition, 2016.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alee Radford, John Schul- man, Szymon Sidor, and Yuhuai Wu. Stable baselines. https: /tt::://ittii higgth.......... com/hill-al stable birble-base. nes 2018.

---

Under review as a conference paper at ICLR 2020
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):1335 1780, November 1997. ISSN 7667. doi: doi. 10. 1162/ neco 1997. .99999.99 .9. 8.8.1773)
19t.://cc..1999..88. URL httt///:
Patrik O Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. In Advances in neural information processing systems, pp. 289-696, 2009.
Diviyan Kalainathan, Olivier Goudet, Isabelle Guyon, David Lopez-Paz, and Michèle Sebag. Sam: Structural agnostic model, causal discovery and penalized adversarial learning. arXiv preprint arXiv:1803.04923. 2018.
David C Kale, Zhengping Che, Mohammad Taha Bahadori, Wenzhe Li, Yan Liu, and Randall Wetzel. Causal phenotype discovery via deep networks. In AMIA Annual Symposium Proceedings, volume 2015, pp. 677. American Medical Informatics Association, 2015.
Frank C. Keil. Explanation and understanding. Annual Review of Psychology, 57(1)2227224,, 2006. doi: 10.1 40.1144///a 17.290044.... 102.1.02.9.000
annurev. pswch. 102904 1 901 Co.
PMID: 16318595.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causal- GAN: Learning causal implicit generative models with adversarial training. arXiv preprint 1709v 2017.
Benjamin Kuipers and Jerome P Kassirer. Causal reasoning in medicine: analysis of protocol. Cognitive Science, 8 8(4):363-385, 1984.
Thanard Kurutach, Aviv Tamar, Ge Yang. Stuart J Russell, and Pieter Abbeel. Learning plannable representations with Causal InfoGAN. In Advances in Neural Information Processing Systems, pp. 8733-8744, 2018.
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-based neural dag learning. arXiv preprint arXiv: (9t:::990.0000 02226, 2019.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Meike Nauta, Doina Bucur, and Christin Seifert. Causal discovery with attention-based convolutional neural networks. Machine Learning and Knowledge Extraction, 1(1):312-340, 2019.
Pedro A Ortega and Daniel Braun. Generalized thompson sampling for sequential decision-making and causal inference. Complex Adaptive Systems Modeling, 2(1):2, 2014.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVitt,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.
Judea Pearl. Causality. Cambridge university press, 2009.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. CoRR, abs/1710.06537, 2017.
Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. The Journal of Machine Learning Research, 15(1):2009-205: 2014.
Bob Rehder. Categorization causal reasoning. Cognitive Science, 27 2003.
Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. No-regret reductions for imitation learning and structured prediction. CoRR, abs/1011.0686, 2010. URL http::t///// http///axxvvoooo//o abs/ 1011 0686
10

---

Under review as a conference paper at ICLR 2020
Steindór Sæmmundsson, Katja Hofmann, and Mare Peter Deisenroth. Meta reinforcement learning with latent variable gaussian processes. arXiv preprint arXiv:1803.07551, 2018.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1312-1320, Lille, France, 07-09 Jul 2015. PMLR. URL http::/ppooceedngs.. wwtp://www...p........ea.m.
schaul15.html
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/ 00007706677 2017.
Laura Schulz and Elizabeth Baraff Bonawitz. Serious fun: preschoolers engage in more exploratory play when evidence is confounded. Developmental psychology, 43(4):1045, 2007.
Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, and Antti Kerminen. linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7(0ct):2033-2030 2006.
Alex H Taylor, Gavin Hunt, Felipe S Medina, and Russell D Gray. Do new caledonian crows solve physical problems through causal reasoning? Proceedings of the Royal Society B: Biological Sciences, 276(1655) 2008.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, pp. 5026-5033. IEBE, 2012.
Eric Tzeng, Coline Devin, Judy Hoffman, Chelsea Finn, Xingehao Peng, Sergey Levine, Kate Saenko, and Trevor Darrell. Towards adapting deep visuomotor representations from simulated to real environments. arXiv preprint arXiv: 151.0711 2(3), 2015.
Albert Yee. The causal effects of ideas on policies. International Organization, 50(1):69-108 1996.
Wenhao Yu, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning universal policy with online system identification. CoRR, abs/ 7ls.02223,,,, 2017.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. arXiv preprint arXiv: /904. /0098, 2019.
Arnold Zellner. Causality and econometrics. Camnegie-Rochester conference series on public policy, volume 10, pp. 9-54. Elsevier, 1979.
Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment probing interaction policies. ICLR, 2019.

---

Under review as a conference paper at ICLR 2020
A ARCHITECTURE DETAILS
A.1 INDUCTION MODELS
A.1.1 OBSERVATION ENCODER
The image encoder used in all models takes as input 32 32 x image of the scene, feeds through 3 convolutional layers with each followed by ReLU activation and 2 2 Max Pooling. The output filters of the convolutions are 8, 16, and 32 respectively, and the resulting x 4 x 32 tensor is mapped to latent vector of size N using single fully connected layer, where N is the number of switches/lights.
A.1.2 ICIN TRANSITION ENCODER
The transition encoder used in our iterative model takes as input state residual of dimension N and an action of size N + concatenated together, and feeds it through fully connected layers of size 1024 and 512 each with ReLU activation, followed by another layer which outputs attention vector of size N (or N + 1 in the MASTERSWITCH case) with SoftMax activation and an edge update of size N with Sigmoid activation. The first two layers trained with dropout of 30%.
A.1.3 ICIN Ć ENCODER
The causal graph encoder used in the last step our iterative model takes as input the N N (or N + N x N in the MASTERSWITCII Case) flattened edge weights of the current graph Čи and feeds them though single fully connected layer, which outputs an attention vector of size N (or N + 1 in the MASTERSWITCH Case) with SoftMax activation and an edge update of size N with Sigmoid activation.
A.1.4 ICIN (No ATTN)
The non-iterative ablation of our method has an identical architecture, but instead of the third fully connected layer outputting an attention vector of size N with SoftMax activation and an edge update of size N with Sigmoid activation, it instead updates full set of N x N edge weights with Sigmoid activation.
A.1.5 TCIN
The temporal convolution induction network uses the same image encoder our approach. However the size N state encodings are then concatenated with the size + action labels, and then are passed through three layers of temporal convolutions, with filter size [256, 128, 128] and size kernel with stride The horizon H by 128 dimensional output is then flattened and fed through fully connected layers of size 1024 and 512, each with 30% dropout. Finally, the size N x N causal graph outputted with Sigmoid activation.
A.2 POLICY ARCHITECTURE
A.2.1 ATTENTION BASED TG
The same image encoder as the induction models used for the policy, except as input takes a 32 x 32 x 6 image which contains the current image and goal image, concatenated channel wise. The encoded image is then flattened and fed through a fully connected layer of size 128, which then outputs attention vector of size N, which is used to do weighted
the edges of the N x N causal graph, producing a vector of size N. This is then encoded to size 128, concatenated with the 128 dim image encoding, and def through 2 more fully connected layers of size 64 and ultimately outputting the final action prediction of size N.
In the non attention version the architecture is identical except the full graph flattened, then encoded and concatenated directly with the image encoding.
A.3 MEMORY BASELINE
In this baseline There also an LSTM Cell of hidden dimension 256 which the image encoding and action encoding are fed into, which is then fed through fully connected layers of size 256, 64 which output the action.
image encoder above, except there is an additional input for action.
12

---

Under review as a conference paper at ICLR 2020
A.4 MEMORY (RL/Low DIM) BASELINE
The policy is a MLP-LSTM policy implemented in Hill et al. (2018), with fully connected layers of size 64, and an LSTM layer with 256 hidden units. It is augmented with additional input heads for each step of the observational data, namely 32 32 images and size N actions.
B TRAINING DETAILS
B. CAUSAL INDUCTION MODEL (F)) TRAINING
Each causal induction model is trained for each split of seen/unseen causal structures as described in the experiments. The F is trained offline on all T Train and corresponding C train for 60000 training iterations using Adam optimizer (Kingma & Ba 2015) with learning rate 0.0001 and batch size 512. They are implemented using PyTorch (Paszke et al. 2017 and trained on NVIDIA Titan X GPU.
B.2 DAGGER POLICY TRAINING
The policies trained with DAgger (Ross al., 2010) are trained in the training environments with episode horizon T = 2N. The policy takes actions in the environment, and at each step an expert action is appended to the policy's 's memory buffer. The policy then trained imitate the experience in the memory. The expert uses the ground-truth causal graph and ground-truth low dimensional states to compute the difference between the goal and current state, and based on the graph what action needs to be taken. Each policy is trained for 100000 episodes, with learning rate 0.0001 and batch size from the memory of 32.
B.3 RL POLICY TRAINING
The Memory (RL/Low Dim) baseline is trained using the Proximal Policy Optimization algo- rithm (Schulman et al. 2017) as implemented Stable-Baselines (Hill al.] 2018 They use hyper-parameters y 0.99, 128 steps per update, entropy coefficient 0.01, learning rate 0.00025, value function coefficient 0.0001 and 0.95. The policy itself consists of two fully connected layers of size 64, well as an LSTM layer consisting of 256 size hidden state. The policy is trained until the policy performance converges on unseen causal structures, and capped at max of million episodes. In this experiment, set the horizon T of each episode equal to the number of switches/lights N.
C CAUSAL INDUCTION QUALITATIVE EXAMPLES
Here we demonstrate example trajectory and how the causal induction model iteratively builds the causal graph.
Observation
Action
Switch
Switch
Switch
Switch
Switch
Q 2 0 o
Predicted
oooO
Graph
Update
Figure 6: Sample of Causal Induction. Here we show example of our Iterative Causal Induction
Model for switches, in the Mone-y case. Given the trajectory of actions and images of the scene, the model needs reason about which lights were turned on, and how what update this implies in the graph. In this example, the first observed action turns on one of the switches, and the model makes the corresponding update the graph. The next switch does so the model outputs no update to the graph. The next action sees one light go on, and updates the corresponding switch. The next action turns on two lights, and the graph is updated Lastly, since one light remains unaccounted for, the model knows to add that edge to the graph. Note: The edges and updates are soft updates, but the model learns to predict close to exactly for edges and exactly 0 for non-edges.
change the lighting
reflect this.
13


Reviews:
[{"review_id": "ryg9pL2m9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\n\nThis paper proposes an interactive agent that tries to infer the underlying causal structure by interacting with the environment; the authors called it \"causal induction.\"  The inferred graph will later help the agent complete goal-directed tasks referred to as a \"causal inference\" stage. Notably, the agent directly learns from visual inputs.\n\nBoth the induction and inference phases heavily rely upon the attention mechanism, which ensures that the agent only focuses on the relevant components of the causal graph. During the induction phase, the agent incrementally updates the predicted causal graph through each interaction using an attention-based edge decoder. During the inference phase, the attention bottleneck also showed to improve the agent's generalization ability.\n \nThey have shown that the proposed model outperforms several baselines in a synthetic environment that uses switches to control lights. They have also demonstrated the model's generalization ability by operating on unseen causal graphs and new task goals.\n\n\n[Major Comments]\n\nMy primary concern about this work is the scope of its applicability.\n\nFor the causal induction phase, the proposed method makes a strong assumption that it can access the ground truth causal relationship during training. The authors can directly read this information from the synthetic environments used in this paper, yet, in more complex real-world situations, we might not know the underlying causal structure for supervised training the induction model.\n\nFor learning the goal-conditioned policies, the authors also assume that they have access to the ground truth causal graph. They use this information to generate the expert demonstrations, which, I presume, are deterministic and unimodal (correct me if I'm wrong). In the real world, a human may be able to infer the underlying causal structure from the observations by interacting with the environment and provide the demonstration data. However, the demonstration may be noisy or form a multi-modal distribution. While I agree that learning from demonstration is an effective way of guiding the learning of the policy, I'm not sure if the method can generalize to more realistic scenarios.\n\nFor inferring the causal graph, the authors also assume that they know the \"cause\" set and the \"effect\" set, which is already a DAG by construction. Instead of inferring the direction of the edge, they are solving an easier problem of deciding whether a directed edge between a \"cause\" node and an \"effect\" node exists or not. The assumption on the graph structure also limits the method's applicability, as, in the real world, the direction of the edge is not always known in advance. Smoking may cause lung cancer, but it is possible that lung cancer may make people smoke more.\n\nI feel this paper makes strong assumptions on both the induction and inference stages, as well as the structure of the causal graph, which greatly limits the applicability of the approach.\n\n[Detailed Comments]\n\nI also have a few questions regarding the details of this paper.\n\nIn Section 3.1, the authors said that \"N is the number of actions in the environments,\" which is a bit confusing. Before this point, the authors did not discuss the relationship between the size of the graph and the size of the action set. Only until Section 3.2 did I realize that N is the number of both \"cause\" set and \"effect\" set. It would be better to discuss the size of the graph and the action set at an earlier position.\n\nHow is the expert planner implemented? Is the expert's policy deterministic and unimodal? I feel this is an important detail to include.\n\nIt is related to the previous question. What will happen if we learn the goal-conditional policy from scratch? How much does imitation learning help with policy learning? Again, the assumption that we know the ground truth causal graph may not be feasible in the real world.\n\nIn Section 3.3, the authors said that \"the expert's action is added to the memory of the policy.\" However, the authors also noted that \"the policy has no memory\" in Section 3.2, which seems to contradict each other. Does the \"memory\" mean replay buffer in Section 3.3?\n\nDoes the number of switches fixed across all environments and always the same as the number of lights? Also, are all the lights mounted in the same location? I'm wondering if the model is invariant to the order of the cause nodes and effects nodes in the graph, and can it generalize to larger environments, more lights, and different room configurations.\n\nIn Figure 4, TCIN seems to have the best performance in the \"Masterswitch\" environment when there are 500 seen causal structures. What might be the reason?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rkx7SCb-Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a paper about a very interesting topic, involving both learning (in a supervised way) to induce a causal graph and taking advantage of it in a goal-conditioned policy.\n\nThis is clearly a timely topic and I loved the motivations of the paper. My main difficulty was with understanding the actual architecture and its motivation, but I believe this is fixable but is a serious impediment to being able to evaluate the paper, as it stands. I was a bit disappointed to see that training is mostly supervised (both providing the ground truth causal graph and an oracle policy as target) but on the other hand it is impressive to obtain these results with raw images as input and the comparative results are good.\n\nFirst, I would like to better understand the insight behind the architecture of F and several things would need to be clarified to enable reproducibility and making sense of the equations. I would start by suggesting to add an example illustrating why simply seeing a (state,next-state,action) triplet is sufficient to obtain a bit of evidence in favour of a particular edge of the graph. Since this is supervised learning of the causal graph, I imagine that the semantics of the node is predetermined, which is a bit disappointing (but doing otherwise would be understandably much more challenging). Second, I don't understand the structure of the causal graph C. What are the input nodes? action values? action x state cross-product? What are the output nodes? Effect variables? Why would C be NxN and not have different input and output dimensions? All this really needs to be clarified. Based on the 1st eqn of page 4 (PLEASE NUMBER YOUR EQUATIONS!!!) it looks like C is number of actions by number of actions, which does not seem consistent with any reasonable interpretation. The authors should also clarify how R is computed (if it is a straight difference of the encoder output, put up an equation for example) and how delta e is computed.\n\nThen in sec 3.2 the authors talk about a weighted sumn involving selected edges. I imagine this is soft-attenetion but it needs to be clarified with equations and explanations. What is the \"content\" associated with each edge e which gets averaged in the soft attention?\n\nI found a possibly interesting parallel between the focus of attention on one edge of the graph  at a time (figure 3) and the ideas of the \"Consciousness Prior\" (Bengio 2017, on arXiv) bottleneck (where only a small tuple of variables, corresponding to an edge here, is considered at each time step in order to reason, plan, decide etc). The fact that in the experiments this attention mechanism helps seems to support the sparsity of dependencies hypothesis underlying the consciousness prior.\n\nMy rating is weak reject but I am ready to upgrade with appropriate explanations answering the above questions.\n\n--- post-rebuttal  addition ---\n\nThe authors have satisfied most of my concerns and I have upgraded my rating to weak-accept.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1x6aci4qB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors describe a method for endowing an artificial agent with causal reasoning  when completing goal-directed tasks. Causal reasoning knowledge is encoded in a directed acyclic bipartite graph or slightly more complicated \"master switch\" variation.\n\nEssentially the method is:\n  - train a model F to predict causal graphs (for which we have ground truth data) from trajectories generated with a heuristic,\n  - train an policy \\pi_G attending over the causal graph to solving tasks.\n\nAt inference time F an \\pi_G are frozen and are evaluated on similar tasks to the ones in training, where however the (unseen) causal graph is new. Thus the evaluation is in a meta-learning scenario, where the model has to learn how to learn to solve tasks.\n\nThe experimental setting is an agent controlling 5 or 7 switches in a simulated environment and observing 32x32x3 images of the environment.\n\nThe authors report significant improvements over an existing baseline (Dasgupta et al. (2019)). The key to the improvements is in the iterative prediction of the causal graph (for F) and in having attention over the causal graph (in \\pi_G).\n\nThe paper is describes a very nice new method and makes interesting points about the role that causal reasoning can play in modeling agents in goal-directed tasks. The paper is strengthened by improvements on existing baselines based on their intuitions.\n\nQuestions:\n\n* The authors claim that constructing an explicit causal structure, instead of a latent feature encoding, leads to better generalization in “long-horizon” tasks -- but they seem to only test on one task, fairly limited in complexity.\n\n* On page 1, the authors claim that empirical evidence suggests the lack of correct causal modeling is an important factor for lack of generalization, generation of unrealistic captions, and difficulties in transfer learning. This seems to be a bit of an overreach. It’s possible that many empirical problems could be solved to a large degree by advances in machine learning without having to resort to explicit causal modeling.\n\n* On page 2, is the reward function r the same for all MDPs?\n\n* Could it be said from the start that pi_I is heuristic and described in section 4.1?\n\n* There seem to be strong assumptions on the structure of C that are only stated late in the paper. C is initially presented as an arbitrary acyclic graph, but from Figure 2 it appears to instead be a bipartite graph with N source nodes and N target nodes, and potential connections from any source node to any target node, where N is the number of actions. This should be explained early on. The “master switch” variation does not seem to exactly fit the mathematical descriptions from the “Methods” section.\n\n* \\hat{C}_H does not seem to be defined except inside Figure 2. It should be defined in the text.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission presents an approach to uncovering causal relations in an environment via interaction. The topic is interesting and the work is timely. However, the experimental setting is quite simplistic and the approach makes strong assumptions that limit its applicability. The reviewers are split. R2 raised their rating from 3 to 6 following the authors' responses and revision, but R1 maintained their rating of 3 and posted a response that justifies this position. In light of the limitations of the work, the AC recommends against accepting the submission.
Decision: Reject