Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
Authors: Nan Rosemary Ke
Aniket Rajiv Didolkar
Sarthak Mittal
Anirudh Goyal
Guillaume Lajoie
Stefan Bauer
Danilo Jimenez Rezende
Michael Curtis Mozer
Yoshua Bengio
Christopher Pal
Venue:  (2021)
Keywords: causal induction
model based RL

Abstract: Inducing causal relationships from observations is a classic problem in machine learning. Most work in causality starts from the premise that the causal variables themselves have known semantics or are observed. However, for AI agents such as robots trying to make sense of their environment, the only observables are low-level variables like pixels in images. To generalize well, an agent must induce high-level variables, particularly those which are causal or are affected by causal variables. A central goal for AI and causality is thus the joint discovery of abstract representations and causal structure. In this work, we systematically evaluate the agent's ability to learn underlying causal structure. We note that existing environments for studying causal induction are poorly suited for this  objective because they have complicated task-specific causal graphs with many confounding factors. Hence, to facilitate research in learning the representation of high-level variables as well as causal structure among these  variables, we present a suite of RL environments created to systematically probe the ability of methods to identify variables as well as causal structure among those variables. We evaluate various representation learning algorithms from literature and found that  explicitly incorporating structure and modularity in the model can help causal induction in model-based reinforcement learning.

Under review as a conference paper at ICLR 2021
SYSTEMATIC EVALUATION OF CAUSAL DISCOVERY IN VISUAL MODEL BASED REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Inducing causal relationships from observations is classic problem in machine learning. Most work causality starts from the premise that the causal variables themselves are observed. However, for Al agents such as robots trying to make sense of their environment, the only observables are low-level variables like pixels in images. To generalize well, an agent must induce high-level variables, par- ticularly those which are causal or are affected by causal variables. central goal for AI and causality is thus the joint discovery of abstract representations and causal structure. However, we note that existing environments for studying causal induction are poorly suited for this objective because they have complicated task-specific causal graphs which are impossible to manipulate parametrically (e.g., number of nodes, sparsity, causal chain length, etc.). In this work, our goal is to fa- cilitate research in learning representations of high-level variables as well as causal structures among them. In order to systematically probe the ability of methods to identify these variables and structures, we design suite of benchmarking RI. environments. We evaluate various representation learning algorithms from the literature and find that explicitly incorporating structure and modularity in models can help causal induction in model-based reinforcement learning.
INTRODUCTION
Deep learning methods have made immense progress on many reinforcement learning (RL) tasks in recent years. However, the performance of these methods still pales in comparison to human abilities in many cases. Contemporary deep reinforcement learning models have a ways to go to achieve robust generalization (Nichol et al., 2018), efficient planning over flexible timescales (Silver Ciosek, 2012), and long-term credit assignment (Osband et al., 2019). Model-based methods in RL (MBRL) can potentially mitigate this issue (Schrittwieser et al., 2019). These methods observe sequences of state-action pairs, and from these observations are able to learn a self-supervised model of the environment. With well-trained world model, these algorithms environment and look ahead to future events to establish better value estimates, without requiring expensive interactions with the environment (Sutton, 1991). Model-based methods can thus be far more sample-efficient than their model-free counterparts when multiple objectives are to be achieved in the same environment. However, for model-based approaches to be successful, the learned models must capture relevant mechanisms that guide the world, i.c., they must discover the right causal variables and structure. Indeed, models sensitive to causality have been shown be robust and easily transferable (Bengio et al., 2019; Ke et al., 2019). As result, there has been recent surge of interest in learning causal models for deep reinforcement learning (de Haan et al., 2019; Dasgupta al., 2019; Nair et al., 2019; Goyal et al., 2019; Rezende et al., 2020). Yet, many challenges remain, and systematic framework to modulate environment causality structure and evaluate models' capacity to capture it is currently lacking, which motivates this paper.
then simulate the
What limits the use of causal modeling approaches in many AI tasks and realistic RL settings is that most of the current causal learning literature presumes abstract domain representations in which the cause and effect variables are explicit and given (Pearl, 2009). Methods are needed to automate the inference and identification of such causal variables (i.e. causal induction) from low-level state

---

Under review as a conference paper at ICLR 2021
Put
Coll der
Rul: Sxarablss
(9 Types of Caussi dosphs
PJ Sice of Grypt
19 Sparsty
(1 Cause Gausr-Etrct Effett Deary
B Cessrvatore
( intevensors
Complesities Causal Graphs
Observational and Interventional Distributions
Figure I: (d)-(d); Different aspects contrituting to the complexity of causal graphs. ()) (i)) Difference between observational and interven- tional data. RL setting, actions are interventions enviromment. The hammer denotes an interveation. Interveation on variable not only affects direct children, but also all reachuble variables. Variables impacted by the intervention have darker shade.
representations (like images). Although one solution is manual labeling, it is often impractical and in some cases impossible to manually label all the causal variables. In some domains, the causal structure may not be known. Further, critical causal variables may change from one task to another, or from one environment to another. And in unknown environments, one ideally aims for an RL agent that could induce the causal structure of the environment from observations and interventions.
In this work, we seek to evaluate various model-based approaches parameterized to exploit structure of environments purposfully designed to modulate causal relations. We find that modular network architectures appear particularly well suited for causal learning. Our conjecture is that causality can provide a useful source of inductive bias improve the learning of world models.
Shortcomings of current RL development environments, and path forward. Most existing RL environments are not good fit for investigating causal induction in MBRL, they have a single fixed causal graph, lack proper evaluation and have entangled aspects of causal learning. For instance, many tasks have complicated causal structures as well as unobserved confounders. These issues make it difficult to measure progress for causal learning. As we look towards the next great challenges for RI. and AI, there a need to better understand the implications of varying different aspects of the underlying causal graph for various learning procedures.
Hence, to systematically study various aspects of causal induction (i.e., learning the right causal graph from pixel data), we propose new suite of environments biases, causal representations, and learning algorithms. The goal is to disentangle distinct aspects of causal learning by allowing the user choose and modulate various properties of the ground truth causal graph, such as the structure and size of the graph, the sparsity of the graph and whether variables observed not (see Figure (a)-(d)). We also provide evaluation criteria for measuring causal induction in MBRL that we argue help measure progress and facilitate further research in these directions. We believe that the availability of standard experiments and a platform that can easily be extended to test different aspects of causal modeling will play significant role in speeding up progress in MBRL.
platform for investigating inductive
Insights and causally sufficient inductive biases. Using our platform, we investigate the impact of explicit structure and modularity for causal induction MBRL. We evaluated two typical of monolithic models (autoencoders and variational autoencoders) and two typical models with explicit structure: graph neural networks (GNNs) and modular models (shown in Figure 5). Graph neural networks (GNNs) have a factorized representation of variables and can model undirected relationships between variables. Modular models also have a factorized representation of variables, along with directed edges between variables which can model directed relationship such A causing B, but not the other way around. We investigated the performance of such structured approaches on learning from causal graphs with varying complexity, such as the size of the graph, the sparsity of the graph and the length of cause-effect chains (Figure (a) - (d)).
The proposed environment gives novel insights in number of settings. Especially, we found that even our naive implementation of modular networks can scale significantly better compared to other models (including graph neural networks). This suggests that explicit structure and modularity such as factorized representations and directed edges between variables help with causal induction in MBRL. We also found that graph neural networks, such as the ones from Kipf et al. (2019) are good at modeling pairwise interactions and significantly outperform monolithic models under this setting. However, they have difficulty modeling complex causal graphs with long cause-efiie effeeetefttttt chains, such as the chain graph (demonstration of chain graphs are found in Figure I (i)). Another finding is that evaluation metries such as likelihood and ranking loss do not always correspond to the performance of these models in downstream RI. tasks.
2

---

Under review as a conference paper at ICLR 2021
Low Level Ohservations
Modular Object Representations
Causal Variables
Purple
Purple
Purple
Figure Illustration of key features can be based on subset of objects' properties. An efficient model should be able infer the high level cousal variables from raw pixel data and learn the underlying causel graph through interactions between these high level causal variables.
Enviromments objects interact according underlying causal graph
2 ENVIRONMENTS FOR CAUSAL INDUCTION IN MODEL-BASED RL
Causal models are frequently described using graphs in which the edges represent causal relationships. In these structural causal models, the existence a directed edge from to B indicates that intervening on A directly impacts B, and the absence of an edge indicates direct interventional impact (see Appendix for formal definitions). In parallel, world models in MBRL describe the underlying data generating process of the environment by modeling the next state given the current state-action pair, where the actions are interventions in the environment. Hence, learning world models in MBRL be seen a causal induction problem. Below, first outline how collection of simple causal structures can capture real-world MBRL cases, and we propose a set of elemental environments to express them for training. Second, describe precise ways evaluate models in these environments.
2.1 MINI-EN MIN-ENNVIONMMENTS, NIIOONNTRO EXPLICIT CASES FOR CAUSAL MODULATION IN RL
The ease with which an agent learns a task greatly depends on the structure of the environment's underlying causal graph. For example, it might be easier learn causal relationships in a collider graph see Figure I(a)) where all interactions variable X; impacts no more than one other variable Xj. hence the cause-effect chain has a length of at most 1. However, causal graphs such full graphs (see Figure (a)) can have more complex causal interactions, where intervening on one variable impacts can impact up to n 1 variables for graphs of size (see Figure 1). Therefore, performance on causal induction in MBRL to analyze how well the model performs on causal graphs of varying complexity.
pairwise, meaning that an intervention on one
important aspect of understanding model's
Impotant factors that contribute to the complexity of discovering the causal graph are the structure, size, sparsity of edges and length of cause-effect chains of the causal graph (Figure 1). Presence of unobserved variables also adds to the complexity. The size of the graph increases complexity because the number of possible graphs grows super-exponentially with the size of the graph (Eaton Murphy, 2007; Peters et al., 2016; Ke et al., 2019). The sparsity of graphs also impacts the difficulty of learning. as observed in (Ke et al., 2019). Given graphs the same size, denser graphs are often more challenging to learn. Futhermore, the length of the cause-effect chains can also impact learning. We have observed our experiments, that graphs with shorter cause-effect lengths such as colliders (Figure 1 (a)) can be easier to model compared to chain graphs with longer cause-effect chains. Finally, unobserved variables which commonly exist in real-world can greatly impact learning, especially if they are confounding causes (shared causes of observed variables).
Taking these factors into account, we designed two suites of (toy) environments: the physics environment and the chemistry environment, which discuss in more detail in the fol- lowing section. They are designed with a focus on the underlying causal graph and thus have a minimalist design that is easy visualize.
2.1.1 PHYSICS ENVIRONMENT: WEIGHTED FEIGHTED-BLOCK PUSHING
The physics environment simulates very simple physics in the world. It consists of blocks of different, unique weights. The rule for interaction between blocks is that heavier objects can push lighter ones. Interventions ammount move particular block, and the consequence depends whether the block next to it (if present) is heavier or lighter. For an accurate world model, inferring the weights becomes essential. Additionally,
allow the weight of the objects to be either observed through
3

---

Under review as a conference paper at ICLR 2021
Underlying
Underlying
Where should this new objeet placed the causal graph?
hefore Should Insert the object in the above cousil grophh
anditr insert the
The mosel heent seen the color sefere, So shoulint be ablle insert object in the acove cauaai graph witheut partorming
Figure 3: Demonstration of the weighted-hlooe block pushing envirommnnt (left: observed, right: unobserved) along with the feasihle generalizations that sctup provides.
the intensity of the color, unobserved, leading to environment settings described below. The underlying causal graph is an acyclic tournament, shown in Figure 3. For more details about the setup, please refer to Appendix E.
Fully observed setting. In the fully observed setting, all objects are given a particular color and the weight of each block is represented by the intensity of the color. Once the agent learns this underlying causal structure, it does not have to perform interventions on new objects in order to infer they will interact with the others.
Unobserved setting. In this setting, the weight of each object is not directly observable by its color. The agent thus needs interact with the object in order understand the order weights associated with the blocks. In this case, the weight of objects needs to be inferred through interventions. We consider two sub-divisions of this setting FixedUnobserved where there fixed assignment between the shapes of the objects and their weights and Unobserved where there is no fixed assignment between the shape and the weight, hence making a more challenging environment. We refer the reader to Appendix E.2 for details.
2.1.2 CHEMISTRY ENVIRONMENT
Colidee
The chemistry environment enables more complexity in the causal structure of the world by allowing arbitrary causal graphs. This is depicted by simple chemical reac- tions, where the state of an element can cause changes another variable's state. The environment consists objects whose positions are kept fixed and thus, uniquely identifiable.
Chain
Causal Graph
Intervention: Set square to Purple
number
Figure Demonstration truth causal graph and sample from same sample shown to demonstrate affect of interventions, right: the affect of interventions and how far they affect based underlying causal graph)
vanilla chemistry enviromment (left: ground
The interactions between different objects
take place according to the underlying causal graph which can either be randomly generated DAG, or specified by the user. An interaction consists of changing the color (state) of variable. At this point, the color of all variables affected by this variable (according to the causal graph) can change. Interventions change a block's color unconditionally. thus cutting the graph edge linking it with its parents in the graph. All transitions are probabilistic and defined by conditional probability tables (CPTs). visualization of the environment can be found Figure 4.
This environment allows for complete and thorough testing of causal models there are various degrees of complexities which can be easily tuned such as: (1) Complexity of the graph: We can test any model many different graphs thus ensuring that models performance is not only limited to few select graphs. (2) Stochasticity: By tuning the skewness of the probability distribution of each object
test how good is given model in modelling data uncertainty. In addition this also tune the number of object or the number colors to test whether the model generalizes to larger graphs and more colors. causally correct model should be able infer the causal relationships between observed objects, as well as their respective color distribution and its dependence on a causal parent's s distribution.
4

---

Under review as a conference paper at ICLR 2021
2.2 EVALUATING CAUSAL MODELS
In much of the existing literature, evaluation of learned causal models is based on the structural difference between the learned graph and the ground-truth graph (Peters et al., 2016; Zheng et al., 2018). However, this may not be applicable for most deep RL algorithms, as they do not necessarily learn an explicit causal structure (Dasgupta al., 2019; Ke al., 2020). Even if structure is learned. it may not be unique as several variable permutations can be equivalent, introducing an additional evaluation burden. Another possibility is to exhaustively evaluate models on all possible intervention predictions and all environment states, a process that quickly becomes intractable even for small environments, We therefore propose a few evaluation methods that can be used as a surrogate metrics to measure the model's performance recovering the correct causal structure.
Predicting Intervention Outcomes. While it may not be feasible to predict all intervention outcomes in an RL environment, propose that evaluating predictions an informative evaluation. Here, the test data is collected from the same environment used in training, ensuring a single underlying causal graph. Test data is generated from new episodes that are unseen during training. All interventions (actions) in the test episodes are randomly sampled and we evaluate the model's performance on this test set.
subset of interventions provides
Zero Shot Transfer. Here, we test the model's 's ability to generalize to unseen test environments, where the environment does not have exactly the same causal graph as training, but training and test causal graphs share some similarity. For example, in the observed Physics environment, a model that has learned the underlying causal relationship between color intensity and weight would be able to generalize to new variables with novel color intensity.
Downstream RL Tasks. Downstream RL tasks that require good understanding of the underlying causal graph of the environment are also good metrics for measuring the model's 's performance. For example, the physics environment, we can provide the model with target configuration in the form of some specific arrangement of blocks on a grid and the model needs to perform actions in the environment to reach the target configuration. Models that capture causal relationships between objects should achieve the target configuration more easily (as it is can predict intervention outcomes). For more details about this setup, please refer to Appendix C.
Metrics. We also evaluate the learned models on ranking metrics in the latent space well as reconstruction-basee metries in the observation space (Kipf et al., 2019). particular we measure and report Hits at Rank 1 (H@) evaluation in standard as well as transfer testing settings. We report these metrics for 1, 5 and 10 steps of prediction in the latent space (refer Appendix B).
@ 1), Mean Reciprocal Rank (MRR) and Reconstruction loss for
MODELS
MLP
A large variety of neural network models have been proposed as world models MBRL. These models can roughly be di- vided into two categories: monolithic mod- els and models that have structure and mod- ularity. Monolithic models typically have no explicit structure (other than layers). Some typical monolithic models are Au- toencoders and Variational Autoencoders (Kingma& & Welling, 2013; Rezende et al., 2014). Conversely, structured models have explicit architecture built into (or learned by) the model. Examples of such models are ones based on graph neural networks (Battaglia et al., 2016; Van Steenkiste et al., 2018; Kipf et al., 2019; Veerapaneni et al.,
Encoder
Decoddr
intractions undrected Modular
directed adper, nigher order drevassios
Figure 5: All models have components: encoder, decoder and transition model. transition models can either monolishic, modular models graph neural networks (GNNs). Monothlic models don't have explicit struc- ture. GNNs have factorized representation of variables. Modular models have both factorized representation of variables and directed edges to polentially model causal relationships such as A cansing B.
2020) and modular models (Ke et al., 2020;
Goyal et al., 2019; Mittal et al., 2020;
Goyal et al., 2020). We picked some commonly used models from these categories and evalu- ated their performance understand their ability for causal induction in MBRI.
5

---

Under review as a conference paper at ICLR 2021
To disentangle the architectural biases and effects of different training methodologies, trained all the models on both likelihood based and contrastive losses, respectively. All models share three common components: encoder, decoder and transition model. We follow similar training procedure as in Ha & Schmidhuber (2018); Kipf et al. (2019). Details of the architectures as well as the training protocols and losses be found in Appendix D.
3.1 MONOLITHIC MODELS
We evaluate causal induction on two commonly used monolithie models: multilayered autoencoders and variational autoencoders. We follow a similar setup as in Ha & Schmidhuber (2018). These models do not have strong inductive biases other than the number of layers used.
3.2 MODULAR AND STRUCTURED MODELS
Several forms of structure can be included in neural networks, including nadttarrtyy.uccorizzed variables, and directed rules. Taking the three factors into account, we consider two types of structured models in our paper, graph neural networks (GNN) and so called modular networks. Graph neural networks (GNN) (Gilmer al., 2017: Tacchetti et al., 2018; Battaglia al., 2018; Kipf et al., 2019) is a widely adopted relational model that have a factorized representation of variables and models pairwise interactions between objects while being permutation invariant. In particular, consider the C-SWM model (Kipf et al.. 2019), which is a state-of-art GNN used for modeling object interactions. Similar most GNNs, the C-SWM model learns factorized representations different objects but for modelling dynamics it considers all possible pairwise interactions, and hence the transition model is monolithic (i.e., not a modular transition model).
Modular networks on the other hand are composed of an initial encoder that factorizes inputs (images), and then a modular transition model (MTM) M. This internal model tasked to create separate factored representations for each objects in the environment, while taking into account all other objects' representations. This model also learns interactions between objects. The rules learned here are directed rules.
EXPERIMENTS
Our experiments seak to answer the following questions: (a) Does explicit structure and modularity help for causal induction MBRL? If so. then what type of structures provide good inductive bias for causal induction in MBRL? (b) How do different objective functions (likelihood or contrastive) impact learning? (c) How do different models scale to complex causal graphs? (d) Do prediction metrics (likelihood and ranking metrics) correspond to better downstream RL performance? (e) What are good evaluation criteria for causal induction in MBRL.
We report the performance of our models on both the Physics and the Chemistry environments, and refer the readers to Appendix D for implementation details.. All models are trained using the procedure described in Section D.2 and are evaluated based on ranking and likelihood metrics 1, 1,5 and 10 step predictions. For the Chemistry environment, we evaluate the models on causal graphs with varying complexity, namely- chain, collider and full graphs. These graphs vary in the sparsity of edges and the length of cause-effect chains. For the Physics environment, we evaluate the model in the fully observed setting as well as the unobserved setting.
4.1 EXPLICIT STRUCTURE AND CAUSAL INDUCTION
We found that for both the Physics and the Chemistry environments, models with explicit structure outperform monolithic models on both prediction metrics and downstream RL performances. In particular, models with explicit structure (GNNs and modular models) scale better to graphs of larger size and longer cause-effect chains.
The Physics environment has a complex underlying causal graph (full graph: refer Figure (a)). We found that GNNs performed well in this environment with 3 variables. They achieved good prediction metrics (Figure 7) and high RL performance (Figure 13) even at longer timescales. However, their performance drops significantly on environments with 5 objects both in terms of prediction metrics (Figure 8) and RL performance (Figure 14). We also see in Figure 8 and 14 that modular models scale much better compared to all other models, suggesting that they hold advantage for larger

---

Under review as a conference paper at ICLR 2021
Ste5s
Stepo
06
Banden Gready Contraativ
A2 VA2
Meeuir 1300
A2
VSC
07M
42
VA2
Stedvior
12.91
Unohserved Physies enviromment setting with objects. Here, (a) Random stands random policy. (b) greedy the policy with best greedy actions, (e) I. are models trained in 2 stages: pretraining the encoder! decader, following by only training the transition model, (d) NL.L with finetune models stages: pretraining the encoder! decoder, following by only training the transition model and then finetuning the encoder, decoder and transition models together. (e) Contrastive are models trained using a contrastive loss. The GNN and Modular models trained on constrastive signilicantly outperform monolithic models (autoencuders
Figure 6: Rate (higher bettor) different models training
step prediction
VAE). margin signilicantly increaes steps to reach the goal increase, suggesting that models with explicit structure and modularity have much better understanding of the world
causal graphs. Further, modular models and GNNs when evaluated on zero shot settings outperform monolithic models by a significant margin (Figures 19, 20 and Tables 15, 16).
For the chemistry environment, we find that modular models outperform all other models for almost all causal graphs terms of both prediction metrics (Fig 23) and RL performance (Fig 25). This is especially true on more complex causal graphs, such chain and full graphs which have long cause-effect chains. This suggests that modular models scales better to more complex causal graphs.
Overall, these results suggest that structure, and in particular modularity, help causal induction in MBRL when scaling up to larger and more complex causal graphs. The performance comparisons on modular networks and C-SWM (Kipf et al., 2019) suggest that both factorized representation of variables and directed edges between variables can help for causal induction in MBRL.
4.2 COMPLEXITY OF THE UNDERLYING CAUSAL GRAPH
There are several ways to vary complexity in a causal graph: size of the graph, sparsity of edges and length of cause-effect chain (Figure 1). Increasing the size of the graph significantly impacts all models' performances. We evaluate models on the Physics environments with 3 objects (Figure 7) and objects (Figure 8) and find that increasing the number of objects from 3 to has significant impact performance. Modular models achieve over 90 on ranking metrics over 10-step prediction for 3 objects while for objects, they achieve only 50 (almost half the performance on 3 objects). A similar pattern is found in almost all models. Another factor impacting complexity of the graph is the length of cause-effect chain. We see that collider graphs are the easiest to learn, with modular models and autoencoders significantly outpeforming all other models (Figure 23). This is because the collider graph has short pair-wisc wise interactions, i.e, intervention on any node in a collider graph can impact at most longer cause-effect chains. For a chain or a full graph of n nodes, an intervention on the htth node can impact all the subsequent (n k) nodes. Modeling interventions on chain and full graphs require modeling more than pairwise relationships, hence, making it much more challenging. We find that modular models slightly outperform all other models on these graphs.
other node. Chain and full graphs are significantly more challenging because of
4.3 PREDICTION METRICS AND RI. PERFORMANCE
As discussed in Section 2.2, there are multiple evaluation metrics based on either prediction metrics or RL performance. The performance of the model on one metric may not necessarily transfer to another. We would like to analyze this the case for the models trained under various environments. We first note that while the ranking metrics were relatively good for most models physics environments, most of them only did slightly better than a random policy on downstream RL, especially on larger graphs (Figures 7 12 and Tables 3 8 for ranking metrics; Figures 13 18 and Tables 9 14 for downstream RL). Figures 21, 22 and 27 show scatter plots for each pair of losses, with one loss on each axis. While there is some correlation between ranking metric and RI. performance (Modular
7

---

Under review as a conference paper at ICLR 2021
and GNN; Figure 21). we did not find this trend to be consistent across models and environment settings. We feel that these results give further evidence of need to evaluate on RL performance.
4.4 TRAINING OBJECTIVES AND LEARNING
Likelihood loss and contrastive loss (Oord al., 2018; Kipf et al., 2019) are two frequently used objectives for training world models in MBRL. We trained the models under each of these objective functions to understand how they impact learning. In almost all cases, models with explicit structure (modular models and GNNs) trained on contrastive loss perform better in terms of ranking loss compared to those trained on likelihood loss (refer to Figures 7 12). We don't 't see very clear trend between training objective and downstream RL performance but we do see a few cases where contrastively trained models performed much better than others (refer to Figures 6, 13, 17, 18 and Tables 9, 13, 14).
For other key insights and experimental conclusions on different environments, we refer the readers to Appendix E,6 for the physics environment and Appendix F.3 for the chemistry environment.
RELATED WORK
Video Prediction and Visual Question Answering. There exist a number of video prediction (Yi et al., 2019; Baradel et al., 2019) and visual question answering (Johnson et al., 2017) datasets that also make use of blocks world for visual representation. Though these datasets can appear visually similar to ours at first glance, they lack two essential ingredients for systematically evaluating models for causal induction MBRL. The first that they do not allow active interventions and hence make it challenging for evaluating model-based reinforcement learning algorithms. Another key point is that these environments do not allow one to systematically perturb different aspects of causal graphs, hence, preventing to systematically study the performances of models for causal induction.
RL Environments. There exist several benchmarks for multi-task learning for robotics (Meta- (Meta-World (Yu et al., 2019) and RLBench (James et al., 2020)) and for video gaming domain (Arcade Learning Environment, CoinRun (Cobbe et al., 2018), Sonic Benchmark (Machado et al., 2018), MazeBase (Nichol al., 2018) and Baby AI (Chevalier-Boisse these benchmarks do not allow one to systematically controll different aspects of causal models (such as the structure, the sparsity of edges and the size of the graph), hence making it difficult to systematically study causal induction in MBRL.
al., 2018)). However, as mentioned earlier,
Block World. The AI community has been using the "blocks world" for decades a testbed for various Al problems, including learning theory (Winston, 1970), natural language (Winograd, 1972), and planning (Fahlman, 1974). Block world allows to easily vary different aspects of the underlying causal structure, and also allow interventions to be performed on many high level variables of the environment giving rise to a large space of tasks which have well-defined relations between them.
DISCUSSIONS AND CONCLUSIONS
In our work, we focus on studying various model-based approaches for causal induction in model- based RI. We highlighted the limitations of existing benchmarks and introduced a novel suite of environments that can help measure progress and facilitate research in this direction. We evaluated various models under many different settings and discuss the essential problems and challenges in combining both fields i.e ingredients, that we believe are common in the real world, such as modular factorization of the objects and interactions of objects governed by some unknown rules. Using proposed evaluation framework, we demonstrate that structural inductive biases are beneficial to learning causal relationships and yield significantly improved performances in learning world models.
There are several interesting future directions that can be taken from here. One direction is extending the environments to settings such as metannea where different causal graphs are set for each episode of training. Another interesting direction is extending this to an environment where the cause and effect does not happen at fixed timescale. For example, if person smokes, it can take variable amount of time until they get cancer. This is very relevant for reinforcement learning, as this is tightly related to credit assignment in RL.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. Cophy: Counterfac- tual learning of physical dynamies. arXiv preprint arXiv: /2000, 2019.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances neural information processing systems, pp. 4502-4510, 2016.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1806.0/261, 2018.
Yoshua Bengio, Tristan Delcu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv: 1901. 10912, 2019.
Maxime Chevaller-Boisver., Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: First steps towards grounded language learning with a human in the loop. arXiv preprint arXiv:1810.08271 2018.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. arXiv preprint arXiv: /8/2003/// 2018.
Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson. Causal reasoning from meta-reinforrcement learning. arXiv preprint arXiv: 1911009900.012.. 2019.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, pp. 1169--11709, 2019.
Daniel Eaton and Kevin Murphy. Exact bayesian structure learning from uncertain interventions, In Artificial Intelligence and Statistics, pp. 107, 2007.
Scott Elliott Fahlman. A planning system for robot construction tasks. Artificial intelligence, 5(1): 1-49, 1974.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 1263-1272. JMLR. org, 2017.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:1909. 100,093,99 2019.
Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, and Michael Mozer. Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. arXiv preprint arXiv:2006.16220 2020.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint urXiv: 1803. 10122, 2018.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference Computer Vision and Patlern Recognition, pp. 2901-2910, 2017.
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Chris Pal, and Yoshua Bengio. Learning neural causal models from unknown interventions. arXiv preprint arXiv: 1rXiv:1910.0107. 2019.

---

Under review as a conference paper at ICLR 2021
Nan Rosemary Ke, Jane Wang. Jovana Mitrovic, Martin Szummer, Danilo J Rezende, et al. Amortized learning neural causal representations. arXiv preprint (XXi::2008.993000 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 412.0980, 2014.
Diederik P Kingma and Max Welling. arXiv:1312.61 2013.
Auto-encoding variational bayes.
arXiv preprint
Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. arXiv preprini arXiv: 1XXx::911..12477 2019.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 01:523-562, 2018.
Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie,
Michael Mozer, and Yoshua Bengio. Learning to combine top-down and bottom-up signals in recurrent neural networks with attention over modules. arXiv preprint arXiv:2006. /6981, 2020.
Suraj Nair, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Causal induction from visual observations for goal directed tasks. arXiv preprint arXiv: 2019.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: new benchmark for generalization in rl. arXiv preprint arXiv:/804.03720 2018.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1
2018.
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepezvari. Satinder Singh, et al. Behaviour suite for reinforce- ment learning. arXiv preprint arXiv: / 0908 03566, 2019.
Judea Pearl. Causality. Cambridge university press, 2009.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 78(5):947-1012, 2016.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. MIT press, 2017.
Danilo Rezende, Ivo Danihelka, George Papamakarios, Nan Rosemary Ke, Ray Jiang, Theophane Weber, Karol Gregor, Hamza Merzic, Fabio Viola, Jane Wang, et al. Causally correct partial models for reinforcement learning. arXiv preprint 20V:22002.028366 2020.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of The 31st International Conference on Machine Learning, pp. 1278-1286, 2014.
Julian Schrittwisser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et Mastering atari, go, chess and shogi by planning with learned model. arXiv preprint arXiv:1911.08265, 2019.
David Silver and Kamil Ciosek. Compositional planning using optimal option models. arXiv preprint arXiv: 120,.643,, 2012.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 100):160 1991.
Andrea Tacchetti, H Francis Song, Pedro AM Mediano, Vinicius Zambaldi, Neil C Rabinowitz, Thore Graepel, Matthew Botvinick, and Peter W Battaglia. Relational forward models for multi-agent learning. arXiv preprint 180iv: 1044, 2018.
10

---

Under review as a conference paper at ICLR 2021
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint TrXX::1802.11333,, 2018.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. Conference on Robot Learning, pp. 1439-1456. PMLR, 2020.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration. arXiv preprint NDV::995.0090 2019.
Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1-191, 1972.
Patrick H Winston. Learning structural descriptions from examples. 1970.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprini arXiv: 191000144, 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. arXiv preprint arXiv:1910.1050: 2019.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. DAGS with NO TEARS: Continuous optimization for structure learning. In Advances in Neural Information Processing Systems, pp- 9472-9442 29229943,, 2018.


Reviews:
[{"review_id": "tpAjAjAxgVv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the importance of incorporating structure and modularity in MBRL. Specifically, it compares monolithic models with models with explicit structure: GNNs and modular models. It also investigates the influences of varying complexity of graphs. \n\nPros:\n1. The authors state a general goal in the abstract: \"the joint discovery of abstract representations and causal structure\", which is promising and needs more investigation for sure.\n\nCons:\n1. The writing is a bit confusing. From the title and the abstract, the readers may expect that this paper is about causal discovery, i.e., learning casual relationships from data, while the casual structure actually is given in the main text.  \n2. In addition, this paper only compares existing models with or without structures, but I fail to see any novel or interesting ideas that this paper tries to deliver. \n3. Furthermore, from the experiments, the authors found that increasing the size of the graph impacts the performance significantly, e.g., increasing 3 nodes to 5 nodes. This result surprises me. I think if the structure is handled properly, the increase from 3 nodes to 5 nodes should not have such a big effect. The authors may investigate or propose other methods that make proper use of the structures. \n\nBased on the above reasons, I do not think this paper is ready to publish.\n\n\nPost-rebuttal:\nThanks for the feedback. The goal of learning causal representation is ambiguous, and it is absolutely a good research topic. However, I fail to see an obvious contribution of the current version. Researchers in this field are usually clearly aware of the limitations for existing methods in causal learning. The problem is how to handle it, e.g., how to give a appropriate definition of the causal variables, how to theoretically show the identifiability and consistency, and then propose a practical solution. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "gFeXm2wjuy", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== Summary\n\nThis paper proposes a benchmark that aims to systematically evaluate models' ability in learning representations of high-level variables as well as causal structures among them. The authors introduce two benchmarking RL environments:\n- One is in a physical domain where an agent is pushing blocks of different weights.\n- Another one is to simulate a chemistry environment, where the state of an element can cause changes to another variable's state according to the underlying causal graph.\n\nThe authors evaluate several representation learning algorithms from the literature and find that explicitly incorporating structure and modularity in models can help causal induction in model-based reinforcement learning.\n\n\n=== Strengths\n\nThis paper targets an important problem of assessing models' ability to automate the inference and identification of the causal variables from high-dimensional inputs like images.\n\nThe construction of the benchmark allows building causal graphs with varying complexity, such as the size of the graph, the sparsity of the graph, and the length of cause-effect chains.\n\nThe authors have evaluated several baseline models on the benchmark, including two typical monolithic models (autoencoders and variational autoencoders) and two models with explicit structure: graph neural networks (GNNs) and modular networks.\n\nThey have made several interesting observations, e.g., modular networks hold better scalability than other baselines, suggesting the benefits that explicit structure and modularity bring for causal induction in MBRL.\n\n\n=== Weaknesses\n\nMy primary concern of this paper is that the dataset is a bit too contrived, which makes it hard to know whether the observations from this benchmark can generalize to more complicated real-world scenarios.\n\nFor example, in the Physics Environment proposed in this paper, only heavier objects can push lighter ones, not the other way around. I understand the authors' desire to make the underlying graph a directed acyclic graph (DAG), but it does not reflect what will happen in the real world. One could imagine sliding a lighter object to collide with a heavier one; the motions of both objects are likely to change, where the interaction between them is bi-directional.\n\nAlso, in the Chemistry Environment, a few objects are connected by a randomly generated DAG, where interventions can change the color of the target and the subsequent blocks. In chemistry, a molecule is a group of atoms held together by chemical bonds, which are also bi-directional relationships. Are there any specific examples in chemistry where the graph is a DAG? I would appreciate it if the authors can elaborate on the connection between the design of the environment and \"chemistry.\"\n\nIn terms of the difficulty of the tasks, the results shown in Figure 6 suggest that the Greedy algorithm can achieve a near-perfect performance on the tasks, which consistently outperforms all other baselines. If a simple greedy algorithm can solve the tasks, does it mean that the benchmark may be a bit too simple, where a good understanding of the underlying causal structure may not be necessary? It would be better if the authors can discuss the necessity of causal induction in these tasks, and how is the ability to perform causal inference correlate with the metrics used in the benchmark.\n\nOverall, I feel the environments proposed in the paper are a bit too artificial, which does not reflect what's likely to happen in the real world. While I like the goal of this paper, I think a set of more realistic environments could greatly improve the significance and potential impact of this paper.\n\n\n\n=== Other comments\n\n\nThe font size of the image caption may be a bit too small.\n\nTypo: Section 2.1, \"Impotant\" --> \"Important\"\n\n\n=== Post rebuttal\n\nThe authors' rebuttal addressed some of my concerns, but my primary concern still remains that that benchmark may be a bit too contrived, where the observations made in this paper may not generalize to more complicated real-world situations. The authors also made some far-fetched arguments in the rebuttal by claiming some concurrent works [1, 2] as \"the 'real-world' version of the environments used in the paper,\" which, to be honest, further lowers the rating of the paper on my side: why is this paper worthy of acceptance if there exist more realistic benchmarks?\n\nI also agree with R1 and R3 that there are no new methods proposed in the paper, and the insights derived from benchmarking a set of existing methods may not be considered novel from the point of view of the ICLR audience. As a result, I keep my rating the same.\n\n[1] Physically Embedded Planning Problems: New Challenges for Reinforcement Learning, https://arxiv.org/abs/2009.05524\n\n[2] CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning, https://arxiv.org/abs/2010.04296", "extra_scores": {}, "extra_text": {}}, {"review_id": "STIcPRr_Q0s", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a suite of RL benchmarks to facilitate inducing causal relationships from visual observations. They show that structural inductive biases are beneficial for causal relationship learning and model-based RL by testing a variety of representation learning algorithms on this benchmark.\n\n##########################################################################\n\npros: \n\n+ The motivation is clear and interesting. Inducing causal relationships from pixel-level observations is an important topic to integrate causality into RL.\n\n+ Overall, the paper is well written.\n\n+ A comprehensive evaluation is conducted to highlight the usage of the proposed benchmark.\n\n##########################################################################\n \ncons: \n\n- Would you explain more about the physics environment about why the underlying graph is a causal graph, not just a statistical graphical model?\n\n- Why is PHYRE[1] benchmark not mentioned in the related work? You seem to work in the same direction and they have proposed more complicated and interesting tasks.\n\n[1] Bakhtin, A., van der Maaten, L., Johnson, J., Gustafson, L., & Girshick, R. (2019). Phyre: A new benchmark for physical reasoning. In Advances in Neural Information Processing Systems (pp. 5082-5093).\n\n\n##########################################################################\n\nPost rebuttal\n\nI'm happy with the author's response and would like to keep my original score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "EOeFsKSF70T", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is a review of model-based approaches of integrating causal inference to reinforcement learning (RL) in different environments (application areas). The authors provide software to analyse how three types of models (“monolithic”, i.e. latent space models without a graph-like structure of the latent space, graph neural networks (GNN) and “modular”, i.e. the C-SWM model (Kipf et al., 2020)) perform in two artificial “environments” devised by the authors (physics and chemistry) based on a number of metrics, some of them also proposed by the authors. The main contributions are the platform for evaluating models in the environments and the insights from the experiments performed on the selected models (taken from existing literature).\n\n\n*****Strengths:*****\n\nThe paper is very well-written, clear and easy to follow.\n\nThe paper provides a novel perspective on applications of causality modelling and the presented environments could be useful to practitioners in the respective fields of physics and chemistry.\n\n\n*****Weaknesses:*****\n\nAs the paper has the form of a review and a collection of insights, there seems to be no novelty from the point of view of either machine learning, causal inference or reinforcement learning. I am therefore not sure whether ICLR is the right venue for this paper.\n\nWhile the motivation for the causal problem tackled in the paper (inferring model-driven causal insights usable for RL agents from data available to agents/robots such as images) is well-defined, interesting and relevant, there seems to be little actual follow-up on it in the paper. The presented models do not seem to adhere to any causal formalism (Pearl’s graphs, Rubin’s potential outcomes) and their only “causal” interpretation is a naïve use of structures (graphs and DAGs) as descriptors of causal relationships without any formal justification. There appears to be no relation to causal structure learning. Neither GNN nor C-SWM were conceived as causal models, and if one wishes to use them as causal models, more justification for such use is needed (why should the discovered graphs correspond to some underlying causal relationships?). Moreover, there have been approaches to achieve what this paper seems to promise (discover causal relationships from images), e.g. (Discovering causal signals in images, Lopez-Paz et al., 2017).\n\n\n*****Post Rebuttal*****\nI would like to thank the authors for the detailed rebuttal.\n\nThe authors state: \"The main goal of our paper is NOT to introduce novel models, but rather to introduce a NOVEL benchmark and insights/ingredients to study causal induction in model-based RL\" and \"It is true that the models we use do not learn an explicit structure for causal learning\" which corresponds to my original reservations to the novelty of this paper. The authors introduce a benchmark / software for evaluating causal induction in RL models, where the user can specify a causal model and its influence on the environment can be examined. I remain unconvinced that the introduction of a RL evaluation benchmark (even one allowing for the presence of arbitrary causal networks) counts as novel at ICLR.\n\nFurther, the statement \"we used some of the typically common models for this purpose, such as GNNs and modular networks. Though these models do not learn an explicit causal graph, they do learn structure that could allow them to discover causal relationships under certain assumptions\" confirms my point that no causal formalism (e.g. connection to the data generating process) is accounted for. It merely means that directed relationships can be modelled.\n\nI agree with the authors that (Lopez-Paz et al., 2017) only uses observational data and does not have any connection to RL, but it is an example of an approach to extracting causal relationships from images in a sound way when it comes to causal inference. This is a side comment and does not influence my assessment of the paper.\n\nTo sum up, my reservations towards the degree of novelty in this paper have not changed (I agree with the authors' summary of their contributions, but disagree as to whether proposing a new benchmark constitutes novelty at ICLR) and I recommend a rejection of the paper in its current form.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a suite of benchmark visual model-based RL tasks to evaluate causal discovery approaches under systematically varying causal graphs. Despite some disagreement on this point among reviewers, I would come down on the side of saying that a better-executed version of this paper would have been a good fit at ICLR. However, its current drawbacks make this a borderline reject. The most important of these drawbacks is: it is unclear to what extent results on these simple environments translate to more realistic complex ones.

Reviewers have also pointed to omitted relevant work that could be discussed in future versions, such as PHYRE. Another relevant benchmark in this vein: https://arxiv.org/abs/1907.09620


Decision: Reject