Neural Causal Discovery with Learnable Input Noise

Learning causal relations from observational time series with nonlinear interactions and complex causal structures is a key component of human intelligence , and has a wide range of applications . Although neural nets have demonstrated their effectiveness in a variety of fields , their application in learning causal relations has been scarce . This is due to both a lack of theoretical results connecting risk minimization and causality ( enabling function approximators like neural nets to apply ) , and a lack of scalability in prior causal measures to allow for expressive function approximators like neural nets to apply . In this work , we propose a novel causal measure and algorithm using risk minimization to infer causal relations from time series . We demonstrate the effectiveness and scalability of our algorithms to learn nonlinear causal models in synthetic datasets as comparing to other methods , and its effectiveness in inferring causal relations in a video game environment and real-world heart-rate vs. breath-rate and rat brain EEG datasets . 1 INTRODUCTION . From an early age , humans have a remarkable ability to infer causal relations from pure observations ( White & Milne ( 1997 ) ; Scholl & Tremoulet ( 2000 ) ; Buchsbaum et al . ( 2012 ) ) . By observing that a left dot moving towards a right dot and the right dot moves correspondingly ( the launching effect , see Michotte ( 1963 ) ) , a human can quickly infer that the left dot causes the right dot to move ( Scholl & Tremoulet ( 2000 ) ) . In fact , much of our way of thinking is via cause and effect . Causal analysis also permits counterfactual reasoning , answering what would have happened if the cause had happened differently . The learning of causality also constitutes much of the scientific endeavor , for example finding the cause of a certain cancer ( Bosch et al . ( 2002 ) ) , or discovering gene regulatory networks ( Lozano et al . ( 2009 ) ) . In addition , causality plays a key role in neuroscience ( Neves et al . ( 2008 ) ; Seth et al . ( 2015 ) ) , economics ( e.g . Granger ( 1969 ) ; Stock & Watson ( 1989 ) ) and finance ( Hiemstra & Jones ( 1994 ) ; Granger et al . ( 2000 ) ) . The study of causality has a long history , yet the application of neural nets to learning causal relations has been scarce . There have been various works that propose methods to infer causal structures with limited model space ( e.g . Granger ( 1969 ) ) which may not be able to model complex nonlinear causal relations , or propose measures to quantify causal strength ( e.g . Schreiber ( 2000 ) ; Janzing et al . ( 2013 ) ) that may not scale to high-dimensional data . For these , the use of universal function approximators like neural nets may be beneficial in inferring causality , which motivate us to propose causal measures that are not only theoretically founded , but also amenable to the learning of function approximators . On the other hand , in the deep learning community , the learning of causal models has not been prevalent , in part due to the lack of theoretical understanding between learning a prediction model and learning a causal model . This also motivates us to propose causal measures , obtained via learning a prediction model , that can deduce causality . The contributions of this work are as follows : • We propose a novel measure to quantify causality from observational data , and an effective algorithm , Causal Inference with Learnable Noise ( CILN ) , to estimate it . It is based on minimizing a learnable noise risk of a prediction model , allowing function approximators such as neural nets to learn complex causal relationships . • We demonstrate on nonlinear synthetic datasets that our method outperforms other causal measures by a large margin , and can scale to a larger number of time series . We also demonstrate that our models are effective on real-world datasets . 2 RELATED WORK . The study of causality has a long history , and has been approached from different perspectives . Pearl ( e.g . Pearl ( 2002 ; 2009 ) ; Pearl et al . ( 2009 ) ) defines causality in terms of intervention and structural dependence , under the structural equation models ( SEM ) . Granger ( Granger ( 1969 ) ; Granger & Newbold ( 1986 ) ) defines causality via prediction : if the prediction of Y via a linear model can be improved by including the information of X , then X causes Y in the Granger sense . Since its proposal , Granger causality has been widely applied in economics ( e.g . Joerding ( 1986 ) ) and neuroscience ( e.g . Deshpande et al . ( 2009 ) ; Seth et al . ( 2015 ) ) . To learn nonlinear causal relations , later works also extend Granger causality to kernel methods ( Ancona et al . ( 2004 ) ; Marinazzo et al . ( 2008a ; b ) ; Sindhwani et al . ( 2012 ) ) . To clear up the relations between these two notions of causality , White et al . ( 2011 ) provide conditions under which Granger causality can deduce direct structural causality in a general settable system framework ( White & Chalak ( 2009 ) ) , with direct structural causality as a natural extension of Pearl causality in settable systems . Our method also utilizes the high-level idea of inferring causal relations via prediction , and building on the work of White et al . ( 2011 ) we propose a novel measure that can likely uncover direct structural causality under certain conditions . Numerous methods have been proposed to discover causal structures from data . One important class is constraint-based methods , for example PC ( Spirtes et al . ( 2000 ) ) , rankPC ( Harris & Drton ( 2013 ) ) , IC ( Pearl ( 2002 ) ) , and FCI ( Spirtes et al . ( 2000 ) ) , which require repetitive conditional independence tests . Hybrid methods , e.g . MMHC ( Tsamardinos et al . ( 2006 ) ) , require repetitive estimation of conditional association score ( e.g . using conditional mutual information ) . In comparison , under the scope of time series , our method can simultaneously discover multiple variables that directly cause the variable of interest . Score-based methods search for the structure that yields the optimal score w.r.t . the data , generally using greedy search methods , for example GES ( Chickering ( 2002 ) ) , rankGES ( Nandy et al . ( 2018 ) ) and GIES ( Hauser & Bühlmann ( 2012 ) ) . This in general requires Θ ( N2 ) steps ( N denoting the number of nodes in the graph ) , and the number of neighboring states may grow very large at each step . In comparison , our method only requires training N models for causal discovery with time series . In addition , various measures have been proposed to quantify causality . Schreiber ( 2000 ) proposes transfer entropy as a measure for causality . It measures the mutual information between the current Y and the past of X , conditioned on the past of Y , to quantify the directional information transfer . As noted in Marinazzo et al . ( 2008a ) , Granger causality as defined in Granger & Newbold ( 1986 ) implies nonzero transfer entropy . Janzing et al . ( 2013 ) analyze several causal measures , and conclude that they are unsatisfactory measures of causal strength . They then propose causal influence , defined via the KL-divergence between the original joint distribution and the distribution with a set of causal arrows broken . We note that both the calculation of transfer entropy and causal influence requires density estimation of the full joint distributions of the input and output , which could easily become difficult in high dimensions . This motivates us to propose new causal measures based on risk minimization , which is much easier with high input dimensions . Besides , various other works have also proposed methods to infer causal structure under some specific conditions , for example , causal additive noise models ( Hoyer et al . ( 2009 ) ) , information-geometric causal inference ( Daniusis et al . ( 2012 ) ; Janzing et al . ( 2012 ) ) , dynamic causal modeling ( Friston et al . ( 2003 ) ) , etc . There has also been works that approach causality from a machine learning perspective , or have architectures that put causality to mind . Lopez-Paz et al . ( 2015 ) study causal inference as a supervised learning problem , where the pairwise causal directions are given as labels for training . Louizos et al . ( 2017 ) utilize a variational autoencoder ( VAE ) structure to estimate the unknown latent space summarizing the confounders and the causal effect . Kipf et al . ( 2018 ) propose a neural relational inference architecture , which simultaneously infers the interactions and learning the dynamics from observational data . Designing causal inference into the model architecture can also be beneficial in certain applications . For example , Kansky et al . ( 2017 ) propose a Schema Network that allows re- gression planning from a goal through causal chains , and demonstrates zero-shot learning in a suite of variations of Atari Breakout games . Our method also relates to sparse learning/feature selection methods , for example L1 regularization and group L1 regularization ( e.g . Meier et al . ( 2008 ) ; Scardapane et al . ( 2017 ) ) . Although L1 or group L1 regularization encourage sparsity in the weights of neural nets , the regularization is model and input dependent , in contrast to our method ’ s invariance to model structure change and inputs rescaling . For example , a three-layer linear network and a collapsed one-layer linear network representing the same function can induce different L1 regularizations . For another example , if one of the input time series is scaled by a factor of 0.1 while other variables remain unchanged , the L1 regularization will be different due to a differently learned model . In comparison , our learnable noise risk when minimized is invariant to the above two kinds of changes , making it especially suitable to discover causality where the scale of different time series may span orders of magnitude , and the model structure may vary , as demonstrated in Section 4.1 . 3 METHOD . 3.1 PROBLEM DEFINITION . Although causal data can be inferred from observations at individual time points , we are primarily concerned with time series . Time series have extra structure that is particularly useful for causal inference : causes must precede their effects . Therefore , we consider N time series x ( 1 ) , x ( 2 ) , ... x ( N ) , where each time series x ( i ) = ( x ( i ) 1 , x ( i ) 2 , ... x ( i ) t , ... ) and each x ( i ) t ∈ RM is an M -dimensional vector . Denote X ( i ) t−1 = ( x ( i ) t−K , x ( i ) t−K+1 , ... x ( i ) t−1 ) with maximum time horizon of K , and Xt−1 = { X ( i ) t−1 } , i = 1 , 2 , ... N . We also denote X ( ĵ ) t−1 = Xt−1\X ( j ) t−1 , i.e . Xt−1 excluding X ( j ) t−1 , to notationally differentiate with the variable of interest X ( j ) t−1 . We assume that the time series is generated by a canonical settable system ( White et al . ( 2011 ) ) . We adopt the settable system ( White & Chalak ( 2009 ) ) paradigm due to the following reasons : ( 1 ) as a natural extension to SEMs , it facilitates optimization , equilibrium , and learning ; ( 2 ) it can formally link Granger causality with Pearl causality ; ( 3 ) it is general enough to encompass a large number of practical scenarios , including time series . Intuitively , each variable in a settable system can either be determined by direct setting ( which formalizes interventions ) , or by a response function on the settings of other variables . A canonical settable system is a settable system where each variable ’ s setting equals its response , allowing the system to evolve naturally without intervention , formalizing time series . In this paper , we also assume causal sufficiency ( Peters et al . ( 2017 ) ) , i.e. , each time series x ( i ) can only be structurally caused by the time series from x ( 1 ) , x ( 2 ) , ... x ( N ) , and generated by stationary response functions hi that are unknown to the learner : x ( 1 ) t : = h1 ( Xt−1 , u1 ) x ( 2 ) t : = h2 ( Xt−1 , u2 ) ... x ( N ) t : = hN ( Xt−1 , uN ) ( 1 ) for t = K + 1 , K + 2 , ... . Here ui ∈ RM , i = 1 , 2 , ... N are noise variables that are mutually independent , are independent of any X ( i ) t−1 , x ( i ) t , i ∈ { 1 , 2 , ... N } , and are effective arguments of the response functions hi . Also in this paper , we only consider “ causality in mean '' , i.e . the causal relations , if exists , influence the mean value of other variables . For any i , j ∈ { 1 , 2 , ... N } , we assume that the variables ( X ( ĵ ) t−1 , X ( j ) t−1 , x ( i ) t ) have probability density function P ( X ( ĵ ) t−1 , X ( j ) t−1 , x ( i ) t ) . We will leave time series with hidden variables ( therefore confounding can occur ) for future work . We note that even without considering hidden variables , Eq . 1 is very general and already encompasses a wide range of scenarios . In order to state the goal of the learner who wants to learn causality from the observations x ( 1 ) , x ( 2 ) , ... x ( N ) , we need to rigorously define causality . Here , we restate the definitions of direct structural causality ( White et al . ( 2011 ) ) and Granger causality ( Granger & Newbold ( 1986 ) ) using our notations of the system Eq . 1 , the former being a natural extension to Pearl causality ( Pearl ( 2009 ) ) in the settable systems . Direct structural causality ( White et al . ( 2011 ) ) We say X ( j ) t−1 , j 6= i does not directly structurally cause x ( i ) t , if for all possible values of X ( ĵ ) t−1 and ul , l ∈ 1 , 2 , ... N , the function X ( j ) t−1 → hi ( Xt−1 , ui ) is constant in X ( j ) t−1 . Otherwise , we say X ( j ) t−1 directly structurally causes x ( i ) t . Granger ( 1969 ) defines causality in terms of conditional expectations . Later , Granger & Newbold ( 1986 ) define it using conditional distributions . We use the latter definition , since it is more general , and also facilitates connection with direct structural causality , and the algorithms proposed in this paper . In this work , we only consider causality between different time series , i.e . requiring j 6= i as default . Granger causality ( Granger & Newbold ( 1986 ) ) We say X ( j ) t−1 , j 6= i does not Granger-cause x ( i ) t , if P ( x ( i ) t |X ( j ) t−1 , X ( ĵ ) t−1 ) = P ( x ( i ) t |X ( ĵ ) t−1 ) , i.e . the conditional distribution function of x ( i ) t given X ( j ) t−1 , X ( ĵ ) t−1 is identical to the conditional distribution function of x ( i ) t given X ( ĵ ) t−1 . Otherwise , we say X ( j ) t−1 Granger-causes x ( i ) t . The goal of the learner is , given only x ( 1 ) , x ( 2 ) , ... x ( N ) , determine for any x ( i ) t , whether X ( j ) t−1 directly structurally causes x ( i ) t for each j = 1 , 2 , ... N .

Summary: In the manuscript entitled "Neural Causal Discovery with Learnable Input Noise" the authors describe a method for automated causal inference under the scenario of a stream of temporally structured random variables (with no missingness and a look-back window of given size).  The proposed approach combines a novel measure of the importance of fidelty in each variable to predictive accuracy of the future system state ("learnable noise risk") with a flexible functional approximation (neural network).  Although the setting (informative temporal data) is relatively restricted with respect to the general problem of causal inference, this is not unreasonable given the proposed direction of application to automated reasoning in machine learning.  The simulation and real data experiments are interesting and seem well applied.