Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning

1 INTRODUCTION . Deep learning methods have made immense progress on many reinforcement learning ( RL ) tasks in recent years . However , the performance of these methods still pales in comparison to human abilities in many cases . Contemporary deep reinforcement learning models have a ways to go to achieve robust generalization ( Nichol et al. , 2018 ) , efficient planning over flexible timescales ( Silver & Ciosek , 2012 ) , and long-term credit assignment ( Osband et al. , 2019 ) . Model-based methods in RL ( MBRL ) can potentially mitigate this issue ( Schrittwieser et al. , 2019 ) . These methods observe sequences of state-action pairs , and from these observations are able to learn a self-supervised model of the environment . With a well-trained world model , these algorithms can then simulate the environment and look ahead to future events to establish better value estimates , without requiring expensive interactions with the environment ( Sutton , 1991 ) . Model-based methods can thus be far more sample-efficient than their model-free counterparts when multiple objectives are to be achieved in the same environment . However , for model-based approaches to be successful , the learned models must capture relevant mechanisms that guide the world , i.e. , they must discover the right causal variables and structure . Indeed , models sensitive to causality have been shown to be robust and easily transferable ( Bengio et al. , 2019 ; Ke et al. , 2019 ) . As a result , there has been a recent surge of interest in learning causal models for deep reinforcement learning ( de Haan et al. , 2019 ; Dasgupta et al. , 2019 ; Nair et al. , 2019 ; Goyal et al. , 2019 ; Rezende et al. , 2020 ) . Yet , many challenges remain , and a systematic framework to modulate environment causality structure and evaluate models ’ capacity to capture it is currently lacking , which motivates this paper . What limits the use of causal modeling approaches in many AI tasks and realistic RL settings is that most of the current causal learning literature presumes abstract domain representations in which the cause and effect variables are explicit and given ( Pearl , 2009 ) . Methods are needed to automate the inference and identification of such causal variables ( i.e . causal induction ) from low-level state representations ( like images ) . Although one solution is manual labeling , it is often impractical and in some cases impossible to manually label all the causal variables . In some domains , the causal structure may not be known . Further , critical causal variables may change from one task to another , or from one environment to another . And in unknown environments , one ideally aims for an RL agent that could induce the causal structure of the environment from observations and interventions . In this work , we seek to evaluate various model-based approaches parameterized to exploit structure of environments purposfully designed to modulate causal relations . We find that modular network architectures appear particularly well suited for causal learning . Our conjecture is that causality can provide a useful source of inductive bias to improve the learning of world models . Shortcomings of current RL development environments , and a path forward . Most existing RL environments are not a good fit for investigating causal induction in MBRL , as they have a single fixed causal graph , lack proper evaluation and have entangled aspects of causal learning . For instance , many tasks have complicated causal structures as well as unobserved confounders . These issues make it difficult to measure progress for causal learning . As we look towards the next great challenges for RL and AI , there is a need to better understand the implications of varying different aspects of the underlying causal graph for various learning procedures . Hence , to systematically study various aspects of causal induction ( i.e. , learning the right causal graph from pixel data ) , we propose a new suite of environments as a platform for investigating inductive biases , causal representations , and learning algorithms . The goal is to disentangle distinct aspects of causal learning by allowing the user to choose and modulate various properties of the ground truth causal graph , such as the structure and size of the graph , the sparsity of the graph and whether variables are observed or not ( see Figure 1 ( a ) - ( d ) ) . We also provide evaluation criteria for measuring causal induction in MBRL that we argue help measure progress and facilitate further research in these directions . We believe that the availability of standard experiments and a platform that can easily be extended to test different aspects of causal modeling will play a significant role in speeding up progress in MBRL . Insights and causally sufficient inductive biases . Using our platform , we investigate the impact of explicit structure and modularity for causal induction in MBRL . We evaluated two typical of monolithic models ( autoencoders and variational autoencoders ) and two typical models with explicit structure : graph neural networks ( GNNs ) and modular models ( shown in Figure 5 ) . Graph neural networks ( GNNs ) have a factorized representation of variables and can model undirected relationships between variables . Modular models also have a factorized representation of variables , along with directed edges between variables which can model directed relationship such as A causing B , but not the other way around . We investigated the performance of such structured approaches on learning from causal graphs with varying complexity , such as the size of the graph , the sparsity of the graph and the length of cause-effect chains ( Figure 1 ( a ) - ( d ) ) . The proposed environment gives novel insights in a number of settings . Especially , we found that even our naive implementation of modular networks can scale significantly better compared to other models ( including graph neural networks ) . This suggests that explicit structure and modularity such as factorized representations and directed edges between variables help with causal induction in MBRL . We also found that graph neural networks , such as the ones from Kipf et al . ( 2019 ) are good at modeling pairwise interactions and significantly outperform monolithic models under this setting . However , they have difficulty modeling complex causal graphs with long cause-effect chains , such as the chain graph ( demonstration of chain graphs are found in Figure 1 ( i ) ) . Another finding is that evaluation metrics such as likelihood and ranking loss do not always correspond to the performance of these models in downstream RL tasks . 2 ENVIRONMENTS FOR CAUSAL INDUCTION IN MODEL-BASED RL . Causal models are frequently described using graphs in which the edges represent causal relationships . In these structural causal models , the existence of a directed edge from A to B indicates that intervening on A directly impacts B , and the absence of an edge indicates no direct interventional impact ( see Appendix A for formal definitions ) . In parallel , world models in MBRL describe the underlying data generating process of the environment by modeling the next state given the current state-action pair , where the actions are interventions in the environment . Hence , learning world models in MBRL can be seen as a causal induction problem . Below , we first outline how a collection of simple causal structures can capture real-world MBRL cases , and we propose a set of elemental environments to express them for training . Second , we describe precise ways to evaluate models in these environments . 2.1 MINI-ENVIRONMENTS : EXPLICIT CASES FOR CAUSAL MODULATION IN RL . The ease with which an agent learns a task greatly depends on the structure of the environment ’ s underlying causal graph . For example , it might be easier to learn causal relationships in a collider graph ( see Figure 1 ( a ) ) where all interactions are pairwise , meaning that an intervention on one variable Xi impacts no more than one other variable Xj , hence the cause-effect chain has a length of at most 1 . However , causal graphs such as full graphs ( see Figure 1 ( a ) ) can have more complex causal interactions , where intervening on one variable impacts can impact up to n − 1 variables for graphs of size n ( see Figure 1 ) . Therefore , one important aspect of understanding a model ’ s performance on causal induction in MBRL is to analyze how well the model performs on causal graphs of varying complexity . Impotant factors that contribute to the complexity of discovering the causal graph are the structure , size , sparsity of edges and length of cause-effect chains of the causal graph ( Figure 1 ) . Presence of unobserved variables also adds to the complexity . The size of the graph increases complexity because the number of possible graphs grows super-exponentially with the size of the graph ( Eaton & Murphy , 2007 ; Peters et al. , 2016 ; Ke et al. , 2019 ) . The sparsity of graphs also impacts the difficulty of learning , as observed in ( Ke et al. , 2019 ) . Given graphs of the same size , denser graphs are often more challenging to learn . Futhermore , the length of the cause-effect chains can also impact learning . We have observed in our experiments , that graphs with shorter cause-effect lengths such as colliders ( Figure 1 ( a ) ) can be easier to model as compared to chain graphs with longer cause-effect chains . Finally , unobserved variables which commonly exist in the real-world can greatly impact learning , especially if they are confounding causes ( shared causes of observed variables ) . Taking these factors into account , we designed two suites of ( toy ) environments : the physics environment and the chemistry environment , which we discuss in more detail in the following section . They are designed with a focus on the underlying causal graph and thus have a minimalist design that is easy to visualize . 2.1.1 PHYSICS ENVIRONMENT : WEIGHTED-BLOCK PUSHING . The physics environment simulates very simple physics in the world . It consists of blocks of different , unique weights . The rule for interaction between blocks is that heavier objects can push lighter ones . Interventions ammount to move a particular block , and the consequence depends on whether the block next to it ( if present ) is heavier or lighter . For an accurate world model , inferring the weights becomes essential . Additionally , one can allow the weight of the objects to be either observed through the intensity of the color , or unobserved , leading to two environment settings described below . The underlying causal graph is an acyclic tournament , shown in Figure 3 . For more details about the setup , please refer to Appendix E. Fully observed setting . In the fully observed setting , all objects are given a particular color and the weight of each block is represented by the intensity of the color . Once the agent learns this underlying causal structure , it does not have to perform interventions on new objects in order to infer they will interact with the others . Unobserved setting . In this setting , the weight of each object is not directly observable by its color . The agent thus needs to interact with the object in order to understand the order of weights associated with the blocks . In this case , the weight of objects needs to be inferred through interventions . We consider two sub-divisions of this setting - FixedUnobserved where there is a fixed assignment between the shapes of the objects and their weights and Unobserved where there is no fixed assignment between the shape and the weight , hence making it a more challenging environment . We refer the reader to Appendix E.2 for details .

Summary: This paper is a review of model-based approaches of integrating causal inference to reinforcement learning (RL) in different environments (application areas). The authors provide software to analyse how three types of models (“monolithic”, i.e. latent space models without a graph-like structure of the latent space, graph neural networks (GNN) and “modular”, i.e. the C-SWM model (Kipf et al., 2020)) perform in two artificial “environments” devised by the authors (physics and chemistry) based on a number of metrics, some of them also proposed by the authors. The main contributions are the platform for evaluating models in the environments and the insights from the experiments performed on the selected models (taken from existing literature).