DO-AutoEncoder: Learning and Intervening Bivariate Causal Mechanisms in Images

1 INTRODUCTION . Recent breakthrough of deep learning ( LeCun et al. , 2015 ; Goodfellow et al. , 2016 ) has significantly advanced the development of Artificial Intelligence ( AI ) . However , statistical prediction and inference models , no matter traditional machine learning models such as random forest , support vector machines or deep models have their limitations such as lacking generalizability , being vulnerable to adversarial attacks as well as lacking interpretability ( Zhang et al. , 2016 ; Papernot et al. , 2016 ) . All these limitations prevent AI from wide applications to high stake scenarios including healthcare , autonomous driving ( Chen et al. , 2015 ) etc . Tremendous amount of efforts from the AI community has been devoted to understanding and improving the robustness of deep models and researchers come to consensus that after training with huge amount of data , deep models exploit “ superficial ” association in data to make prediction . However , such kind of subtle association relationship is unstable across domains . For example , the association relationship between pixels related to “ grassland ” and the label “ elephant ” is real but it is rather weak . Deep models , trained with hundreds of thousands of natural images where elephants reside on grassland , exploit too much pixel level association of the image to the label . Unfortunately , such models fail to detect an elephant if the background of the image is changed to a living room . More and more evidences show that deep models exploit association instead of causation when making the prediction , rendering it weak in term of domain generalization and robustness against adversarial attacks . As a consequence , researchers are advocating bringing causality into the life-cycle of deep neural networks . In Judea Pearl book “ The book of why ” ( Pearl & Mackenzie , 2018 ) , he gives ladder of causation : The lowest level is concerned with patterns of association in observed data . The next level focuses on intervention and the highest one is counterfactual inference . In order to realize real Machine Intelligence , we must replace reasoning by association with reasoning by causality . However , how to incorporate causality into current machine learning framework is very challenging and little work has been found yet until the most recently , researchers start to look at disentangled representation learning . In the deep learning community , researchers realized that a good representation enables better generalizability and might possibly enjoy better interpretability . Disentangled representation is believed to be a good representation and become a research focus in a past few years ( Hsu et al. , 2017 ; Narayanaswamy et al. , 2017 ; Locatello et al. , 2018 ) . The idea behind disentangled representation learning is that we prefer the learned representations as independent as possible such that they correspond to the independent generative factors of the image , e.g . the illumination , azimuth , elevation etc for 3D graphics . The advantage of disentangled representation is that it offers better explainablity and it is claimed that deep models with disentangled representations enjoy better generalization performance for downstream tasks . Although significant progress has been achieved in the area of deep representation learning in the past few years , most of existing work assume that the representations or the latent codes are statistically independent and they are trained by adding a regularization term of mutual information between latent codes to the loss function ( Chen et al. , 2016 ) . It is of great interest and has shown advantage over entangled representation , but forcing the latent codes to be independent might limit itself both in term of theoretical understanding and practical applications . In various scenarios , we are rather interested in learning latent representations that correspond to the causally related concepts . For example , given an image consisting concepts like tree , sunshine and shadow , an idea representation learning algorithm is expected to yield latent codes that could each corresponds to the above three concepts . Obviously , these three concepts are by no means independent , i.e . the concept of tree and that of sunshine together causes the concept of shadow . If we are able to extract latent codes with their causal structure , we are able to intervene the whole system , e.g . intervene the latent code of shadow such that we can generate images with the shadow removed . This could be potentially of great value in many applications such as autonomous driving . In this paper , we propose a framework called DO-AutoEncoder ( DO-AE ) to learn causally related concepts in images , which is achieved by firstly discovering the causal relationship of concepts in the images by comparing complexities of generative models ( G ( cause ) , G ( effect|cause ) ) in either hypothetic directions and the generative models are realized by AutoEncoder structures . The latent codes are then used to reconstruct the original input images and the whole AutoEncoder structure is trained by minimizing the reconstruction error . The framework mimics the whole data generating mechanism , i.e . the joint distribution P ( cause , effect ) is factorized as P ( cause ) P ( effect|cause ) according to the causal direction . As Figure 10 shows , the light is projected from the vertical direction upside down , yielding the time-varying shadow of a swinging pendulum . So the causal direction is from angle ( of pendulum ) to length ( of shadow ) . We aim at developing deep representation learning model and algorithm to discover such causally related concepts from a series of images describing the law of physics . In addition to the pendulum data , in this paper we create a series of dataset named PHY2D , the collection of images of abstract graphic description of physics laws . 2 RELATED WORK . People have studied causality for a long time and related work can be found in philosophical publications , religious scriptures and science publications . Ancient Greek famous thinker Aristotle proposed “ Four causes ” which includes the final cause , the material cause , the formal cause and the efficient cause . He believes that all natural phenomena follow the causal law . Scottish philosopher David Hume recognizes cause-effect relation as both a philosophical relation and a natural relation , and he argued that this relation is pivotal in reasoning ( Hume , 2003 ) . ( Harari , 2014 ) provide that human ancestor ’ s causal imagination is the basis for mutual communication . The instinct of human beings to explore the cause and effect of things makes human beings more intelligent , and causal thinking is the foundation of the development of science and technology . Causality is a bridge between science and philosophy . In science , causality can be found in several fields such as economics ( Moneta et al. , 2013 ) , neuroscience ( Ramsey et al. , 2014 ) , environment science ( Sugihara et al. , 2012 ) and especially machine learning . Pearl shows that machine learning without causal inference will encounter Yule-Simpsons Paradox , which means the model can get two completely contradicting conclusions with the same dataset . Recent efforts from machine learning community try to link causality and ML methods . To name a few , ( Schölkopf et al. , 2012 ) argues that semi-supervised learning only works in anti-casual direction . ( Bengio et al. , 2019 ) propose a meta-learning framework to determine the causal relationship between two observed variables . The correct modeling according to the causal direction usually leads to faster adaptation to shifted data distribution due to interventions to the system . Using machine learning for analyzing physics is a hot topic ( Watters et al. , 2017 ; Fragkiadaki et al. , 2015 ; Janner et al. , 2018 ) . ( Bakhtin et al. , 2019 ) provide a 2D physical classical mechanical puzzle dataset to encourage more useful models for physics . In interdisciplinary field of computer vision and causality , ( Lebeda et al. , 2015 ) provide statistical tools based on transfer entropy to explore the causality relationship between the camera motion and the object motion to predict object location . The prediction accuracy is increased over non-causal predictor . ( Lopez-Paz et al. , 2017 ) focus on establishing the existence of causality based on observable statistical signal carried in image datasets , by using causal discovery technique Neural Causation Coefficient ( NCC ) . These recent advances show that causal inference in images is gaining increasing amount of attention from the AI community . 3 METHOD . As mentioned in Sec 1 , we aim at representation learning that correspond to the concepts as well as their causal structure in images . To achieve this target , the first step is to discover the causal relationship between the concepts in images , i.e . we need to identify which one is the cause or effect . In this paper , we follow the Independent Mechanism principle proposed by ( Janzing & Scholkopf , 2010 ) . Secondly , we need to learn the generating mechanisms of the cause and the generating mechanism mapping the cause to the effect . We propose to use AutoEncoder as the generative model . Given observations of two variables X and Y , distinguishing the cause from effect is a challenging problem . There is a postulate named Independent Mechanism principle ( Janzing & Scholkopf , 2010 ) stating that the joint distribution factorization has asymmetric statistical behaviors in causal versus anticausal directions . Specifically , decomposition of the joint distribution according to the causal direction is natural and simpler than the anti-causal direction . This principle can also be explained by the famous Occam ’ s Razor principle . If there are multiple models that can explain the data equally well , we prefer the simplest one . The definition of simplicity varies in different methods . For example , ( Janzing & Scholkopf , 2010 ) argues that the Kolmogorov complexity of conditional and marginal distributions is smaller in causal direction than that in anticausal direction . It can be written as the following inequation : K ( PX ) +K ( PY |X ) ≤ K ( PY ) +K ( PX|Y ) , ( 1 ) where K ( · ) is the Kolmogorov complexity . However , the Kolmogorov complexity is not computable , and thus researchers proposed other statistical methods including Mimimal Discription Length ( MDL ) or Reproducing Kernel Hilbert Space ( RKHS ) norms ( Sun et al. , 2007 ) as approximation . Recently , ( Bengio et al. , 2019 ) showed that the joint distribution decomposed according to the causal direction is much easier to learn and is able to adapt much faster to the distributional shift in data than the anti-causal direction . Inspired by this work , in this paper we propose to learn the causal direction of concepts in images according to model complexities . Firstly , we introduce how to identify the causal direction according to the loss after model training . Denote by A , B the ground truth hypothetic cause and effect concepts in the images , e.g . in the pendulum case , if we assume that the pendulum causes the shadow , then A corresponds to the concept of pendulum and B corresponds to the concept of shadow . We further denote by Â and B̂ the corresponding generated concepts by two generative models f ( enc ; θ ) and g ( Â ; η ) , where enc is the original input image and θ and η are model parameters . In this paper , we use AutoEncoder the generative functions . The likelihood of the generative concepts is defined as follows : PG ( Â , B̂ ) : = exp ( −‖A− Â‖2 ) exp ( −‖B − B̂‖2 ) ( 2 ) Two generative models , i.e . the model generating the cause and the model mapping the cause to the effect are then trained by minimizing the negative log-likelihood as follows : θ∗ , η∗ = arg minE [ ‖A− f ( enc , θ ) ‖2 ] + E [ ‖B − g ( Â ; η ) ‖2 ] Note that E [ ‖B − g ( Â ; η ) ‖2 ] = E [ ‖B − g ( A ; η ) + g ( A ; η ) − g ( Â ; η ) ‖2 ] ≤ E [ ‖B − g ( A ; η ) ‖2 ] + E [ ‖g ( A ; η ) − g ( Â ; η ) ‖2 ] ≤ E [ ‖B − g ( A ; η ) ‖2 ] + LE [ ‖A− Â‖2 ] , if we assume that g is Lipschitz continuous and we denote by L the Lipschitz constant . We then can get the lower bound of the log-likelihood , i.e . Lb = − ( 1 + L ) E [ ‖A− f ( enc , θ ) ‖2 ] − E [ ‖B − g ( A ; η ) ‖2 ] ( 3 ) However , this still requires choosing L which might be unknown . We further relax the loss function to be LA→B = − ( 1 + L ) E [ ‖A− f ( enc , θ ) ‖2 ] − ( 1 + L ) E [ ‖B − g ( A ; η ) ‖2 ] ( 4 ) To identify the causal direction , we only need to compare which direction gives lower loss , given the same model complexity , i.e . { A→ B , if LA→B > LB→A ; B → A , if LB→A < LA→B ; NO conclusion , otherwise . ( 5 ) Our whole pipeline is in Figure 2 . The whole DO-AE inputs an image and then tries to reconstruct it at the output . Instead of using a single AutoEncoder , we propose a modularized one by decomposing the whole system into two sub-modules , and each is an AutoEncoder . The first module aims at generating the hypothetic cause , followed by a second one to generate the hypothetic effect from the hypothetic cause . By controlling the model capacity , the likelihood are then used to decide which direction is more plausible to be the causal direction . We prefer the direction with higher likelihood according to the Independent Mechanism principle . One should be noted that we can not use a model with too rich flexibility otherwise the aforementioned method might not work as it could be the case that with very expressive models , the likelihoods of both hypothetic direction are almost equally good and thus not distinguishable . We verify this conjecture in the experiment section . Note that in experiments we simply set L = 0 since its value would not have an impact on the inference rule .

Summary: This paper presented an image data that are generated from two variables using some physics law. It also proposed a model to identify the causal relationship between the two variables using the image dataset. The method, in general, utilize the general idea that the causal direction is easier for the model to describe than the anti-causal direction. So the image is fad into a VAE based model in two different ways. The one with lower loses represents the correct causal direction.