Optimal Transport for Causal Discovery

1 INTRODUCTION . Determining causal direction between two variables is a fundamental and challenging causal discovery task ( Janzing et al. , 2012 ) . Conventional constraint-based and score-based causal discovery methods identify causal structures only up to Markov equivalent classes ( Spirtes et al. , 2001 ) , in which some causal directions are undetermined . To address this challenge , properly constrained functional causal models ( FCMs ) have been proposed . FCMs represent the effect as a function of its cause and independent noise , and can help identify the causal direction between two variables by imposing substantial structural constraints on model classes , such as additive noise models ( ANMs ) ( Shimizu et al. , 2006 ; Hoyer et al. , 2008 ) and post-nonlinear models ( PNLs ) ( Zhang and Hyvärinen , 2009b ) . While some of the models , such as PNLs , are highly flexible , the constraints are still restrictive and difficult to interpret and relax . Inevitably , the performance of these methods is sensitive to model assumptions and optimization algorithms , especially in real-world applications . To handle the mentioned issues , we consider FCMs from a dynamical-system view . By augmenting a time dimension for the static causal discovery task , we interpret FCMs with dynamical causal processes under the least action principle ( Arnol ’ d , 2013 ) . The new interpretation connects FCMs with a large class of models in dynamical systems . It then provides more freedom to model causal influences , possibilities to derive new causal discovery criteria , and a potential direction to generalize causal models with identifiable causal direction . In particular , we exploit the above idea by leveraging the intrinsic connection between FCMs and optimal transport . Optimal transport is originally introduced by Monge ( 1781 ) , which has been ap- FIX ( GA77 , pBTi ) plied in a large range of applications , not only because it is a natural way to describe moving particles ( Ambrosio et al. , 2012 ) but also because of its recent improvement of the computational methods ( Cuturi , 2013 ; Kolouri et al. , 2019 ) . Recently , it has also been largely applied to generative models for measuring the distance of probability distributions ( Arjovsky et al. , 2017 ; Kolouri et al. , 2018 ; Genevay et al. , 2018 ) . Among different optimal transport definitions , the L2 Wasserstein distance got extensive applications in statistics ( Rachev and Rüschendorf , 1998 ) , functional analysis ( Barthe , 1998 ) , and multiple disciplines of science ( McCann , 1997 ; Otto , 1997 ) . The dynamical formulation of the L2 Wasserstein distance is introduced by Benamou and Brenier ( 2000 ) for relaxing the computational costs . We find that in the context of the dynamical formulation , FCMs can be connected with optimal transport . Furthermore , with the dynamical interpretation of optimal transport , one can naturally understand FCMs from a dynamical-system view , which makes it possible to derive new criteria to identify causal direction . Moreover , it also enables us to develop practical algorithms with optimal transport for static causal discovery tasks in the bivariate case without learning a regression model . Accordingly , our main contributions are : 1 . Dynamical interpretation of FCMs in the bivariate case . We provide dynamical interpretations of optimal transport under the constraints of FCMs . Furthermore , we introduce a time variable , determine the underlying dynamical process under the least action principle ( Arnol ’ d , 2013 ) for the static bivariate causal discovery task , and characterize properties of the corresponding dynamical systems ( Sec . 3.1 and Sec . 3.2 ) . 2 . A criterion of determining causal direction between two variables for ANMs . We study the corresponding dynamical systems of FCMs , and prove that ANMs correspond to volume-preserving pressureless flows . Moreover , based on the divergence of their velocity fields , we propose a criterion to determine causal direction , and show that under the identifiability conditions of ANMs , it is a valid criterion for ANMs , which can be extended to PNLs directly ( Sec . 3.2 ) . 3 . An optimal transport based approach ( DIVOT ) for distinguishing cause from effect between two . variables . DIVOT inherits the advantages of one-dimensional optimal transport . It can be computed efficiently , and does not require independence tests , learning a regression model , or deriving likelihood functions for complicated distributions . Experimental results show that our method is robust to the choice of models , and has a promising performance compared with the state-of-the-art methods on both synthetic and real cause-effect pair datasets ( Sec . 4 and Sec . 6 ) . 2 PRELIMINARIES . 2.1 OPTIMAL TRANSPORT . In this work we mainly use the L2 Wasserstein distance and we introduce its framework following the notations and definitions of ( Benamou and Brenier , 2000 ) in this section . FIX ( pBTi ) The underdetermined Jacobian problem . Suppose that two density functions , ρ0 ( x ) and ρT ( x ) where x ∈ Rd , are non-negative and bounded with total mass one . The transfer of ρ0 ( x ) to ρT ( x ) is realized with a smooth one-to-one map M : Rd → Rd . The Jacobian problem is to find M that satisfies the Jacobian equation : ρ0 ( x0 ) = ρT ( M ( x0 ) ) |det ( ∇M ( x0 ) ) | , xT =M ( x0 ) , ( 1 ) where det ( · ) denotes determinant . This is an underdetermined problem as many maps can be the solutions . Thus , a natural way is to choose the optimal one in the sense of lowest cost depending on applications . A common cost function is the Lp Wasserstein distance ( Kantorovich , 1948 ) . Lp Wasserstein distance and its one-dimensional closed-form solution . The Lp Wasserstein distance between p0 and pT , denoted by Wp ( p0 , pT ) , is defined by Wp ( p0 , pT ) p = infM ∫ |M ( x0 ) − x0|pp0 ( x0 ) dx0 , where p ≥ 1 . We say that the optimal transport M∗ achieving the infimum solves the optimal transport ( or Lp Monge-Kantorovich ) problem . In this work , we mainly use the square of the L2 Wasserstein distance , denoted by W 22 . Moreover , following the approach of Kantorovich ( 1948 ) , the one-dimensional ( 1D ) Lp Wasserstein distance has a closed-form solution , e.g. , the 1D optimal solution of W 22 is M∗ = P−1T ◦ P0 , ( 2 ) where P0 and PT are the cumulative distribution functions for p0 and pT , and “ ◦ ” represents the function composition . In practice , the 1D optimal solution in Eqn . ( 2 ) can be computed with the average square distance between the sorted samples from p0 and pT ( Kolouri et al. , 2019 ) . 2.2 FUNCTIONAL CAUSAL MODELS . A FCM represents the effect Y as a function f ( · ) of the direct cause X and independent noise Ey , where function f describes the causal influence of X on Y , and Ey is the exogenous vari- able/noise . Without any additional assumption on the functional classes , the causal direction is not identifiable ( Hyvärinen and Pajunen , 1999 ; Zhang et al. , 2015a ) . Roughly speaking , this is because given variable pair ( X , Y ) , one can always construct Y = f ( X , Ey ) and another different FCM , X = f̃ ( Y , Ex ) , such that both of them are consistent with the data distribution and have independent “ noise ” ( Hyvärinen and Pajunen , 1999 ; Zhang et al. , 2015a ) . Several works further introduce proper assumptions on the functional form of the FCMs and the distributions of the involved variables , and then identify causal direction in the bivariate case , i.e. , the independence of cause and noise only holds in the causal direction . Examples include the ANM and PNL models : ANM : Y = g ( X ) + Ey ; ( 3 ) PNL : Y = h ( g ( X ) + Ey ) , ( 4 ) where g and h are nonlinear functions and h is invertible . 3 DYNAMICAL INTERPRETATION OF FUNCTIONAL CAUSAL MODELS . We first show the connection between FCMs and optimal transport in Sec . 3.1 . In Sec . 3.2 , we further elaborate the analogy between the optimal transport problem and the causal direction determination problem . We then study the optimal transport under the constraints of FCMs , show the corresponding dynamical systems of FCMs , and characterize the properties of such systems . 3.1 CAUSALITY AND OPTIMALITY OF THE MAPS IN THE JACOBIAN PROBLEM . As introduced in Sec . 2.2 , FCMs are used to approximate the true data generation process . Given the FCM , Y = f ( X , Ey ) , we rewrite it in a Jacobian equation way , xT = [ X Y ] = [ Ex f ( X , Ey ) ] =M ( [ Ex Ey ] ) =M ( x0 ) , ( 5 ) where x0 , xT ∈ R2 , their probability densities p0 , pT ≥ 0 , and M : R2 → R2 . As an analogy to the mass transfer scenario ( Monge , 1781 ) , we consider the samples of independent noise Ex and Ey as the particles of materials in the mass transfer problem and regard the map M in Eqn . ( 5 ) as a special transformation of the independent noise samples . As shown in Fig . 1 , one can consider the data points are transferred from the original positions ( which are unmeasured ) in the plane Ex–Ey at time 0 to the observed positions in the plane X–Y at time T . Such transformation considers the transfer as a causal process which moves the unmeasured independent noise x0 = [ Ex , Ey ] ′ 1 and consequently leads to the observations xT = [ X , Y ] ′ . From the perspective of FCM-based causal FIX ( pBTi ) discovery approaches , causal influences are represented by FCMs which represent the effect as a function of its direct cause and an unmeasured noise satisfying the FCM constraints : 1 “ ′ ” denotes the transpose of vectors or matrices . ( i ) The map constraint : the values of X are determined by the values of its corresponding noise , i.e. , X = Ex , while the values of the effect depend on cause X and noise Ey ; ( ii ) The independence constraint : the noise terms are independent , i.e , Ex is independent of Ey . How is causality defined with the FCM constraints connected with the optimality of maps in optimal transport ? The optimal transport M∗ with the minimal Lp Wasserstein distance is not necessary to be the causal one given by Eqn . ( 5 ) , because it has no information about the FCM constraints or the true data generation process . In other words , given two sample sets of x0 and xT , the couplings FIX ( pBTi ) given by optimal transport are not necessary to be the ones generated from the ground-truth FCM . 3.2 DYNAMICAL INTERPRETATION OF FCMS : OPTIMAL TRANSPORT UNDER THE FCM CONSTRAINTS . In this section we jointly consider causality and optimality of the maps in the Jacobian problem . It provides both a causal sense of the transformation and a dynamical view of FCMs . We first recap the dynamical formulation of theL2 Wasserstein distance , and then study such dynamical systems under the FCM constraints . Furthermore , we show their properties under the FCM and ANM constraints . Dynamical interpretation of the L2 Wasserstein distance . Benamou and Brenier ( 2000 ) formulate the L2 Monge-Kantorovich problem as a convex space-time minimization problem in a continuum mechanics framework . Fixing a time interval [ 0 , T ] , they introduce the concepts of smooth time-dependent density ρ ( t , xt ) ≥ 0 and velocity fields v ( t , xt ) . When they are clear from context , we denote them by ρ and v. Because we are considering the bivariate case , xt ∈ R2 and v ∈ R2 . Then , they give the dynamical formulation of W 22 : W 22 ( p0 , pT ) = inf ρ , v T ∫ R2 ∫ T 0 ρ ( t , xt ) |v ( t , xt ) |2dxtdt , ( 6 ) s.t . { initial and final conditions : ρ ( 0 , · ) = p0 , ρ ( T , · ) = pT the continuity equation : ∂tρ+∇ · ( ρv ) = 0 . They prove that the solutions of ( 6 ) are pressureless potential flows , of which the fluid particles are FIX ( pBTi ) not subject to any pressure or force and their trajectories are determined given their initial positions and velocities or given their initial and final positions . Suppose that M∗ is the solution given by W 22 . The corresponding flows follow time evolution equation , xt = x0 + t T v ( t , xt ) , where v ( t , xt ) = v ( 0 , x0 ) =M∗ ( x0 ) − x0 and t ∈ [ 0 , T ] . ( 7 ) The time evolution equation shows that xt is just a convex combination of x0 and M∗ ( x0 ) . As an analogy between the optimal transport problem and causal direction determination , the density ρ and the velocity v of moving particles can be considered as the probability density and the velocity of changing values of data points . Moreover , the dynamical interpretation of the L2 Wasserstein distance introduces a time variable and provides a natural time interpolation ρ ( t , xt ) of ρ0 and ρT together with the velocity field v ( t , xt ) under the least action principle . Similarly , we can also have the natural time interpolation p ( t , xt ) between p0 and pT as well as the velocity field v ( t , xt ) under the least action principle , which is the dynamical interpretation of FCMs . Dynamical L2 Wasserstein distance under the FCMs constraints . First , we introduce FCM constraints in the context of the dynamical L2 Wasserstein distance . According to time evolution equation ( 7 ) , we know that the velocity is fully determined by the initial and final values of xt . We first consider FCM constraint ( i ) . For the initial and final values of the cause , the value of its observation X is equal to its noise value Ex in Eqn . ( 5 ) . Consequently , denoting v = [ vx , vy ] ′ , where vx and vy represent the velocities along the Ex/X-axis and Ey/Y -axis respectively as shown in Fi.g 1 , we know that ∀t ∈ [ 0 , T ] , xt ∈ R , vx ( t , xt ) = 0 . As for FCM constraint ( ii ) , it implies that the initial noise of cause and effect , corresponding to Ex and Ey at the time 0 , are independent . Therefore , we have the FCM constraints for the dynamical L2 Wasserstein distance , ( I ) The map constraints : v ( t , xt ) = [ vx ( t , xt ) , vy ( t , yt ) ] ′ , ∀xt , yt ∈ R , t ∈ [ 0 , T ] , vx ( t , xt ) = 0 ; ( II ) The independence constraint : Two random variables ( cause and effect ) at the initial time have the joint probability density function p0 ( x0 ) and they are independent . Second , we characterize the properties of the dynamical optimal transport ( L2 Wasserstein distance ) under constraints ( I ) and ( II ) . According to Eqn . ( 5 ) , the form of M∗ is determined as M∗ ( x0 ) = [ Ex f ( Ex , Ey ) ] ; ( 8 ) otherwise , the FCM constraints will be violated . Moreover , the two-dimensionalW 22 under the FCM constraints can be computed with the one-dimensional W 22 as shown in Prop . 1 ( the derivation is in App . D ) , Proposition 1 . Under constraints ( I ) and ( II ) the square of the L2 Wasserstein distance between p0 and pT is W 22 ( p0 , pT ) = EEx [ W 22 ( p ( Ey ) , p ( Y |Ex ) ) ] . ( 9 ) Furthermore , we consider the constraints of ANMs and characterize the corresponding dynamical systems for now , which can be directly extended to PNLs as mentioned in Sec . 4.2 . Based on constraints ( I ) and ( II ) , we further introduce the ANM constraint , ( III ) the effect is the sum of noise and a nonlinear function of cause as defined in Eqn . ( 3 ) . Theorem 1 ( Zero divergence of the velocity field ) . Under constraints ( I ) and ( II ) , the dynamical systems given by the L2 Wasserstein distance are pressureless flows . Further under ANM constraint ( III ) , they become volume-preserving pressureless flows , of which the divergence of the velocity field , v ( t , xt ) = [ vx ( t , xt ) , vy ( t , yt ) ] ′ , satisfies div v ( t , xt ) = ∂vx ( t , xt ) ∂xt + ∂vy ( t , yt ) ∂yt = 0 , ∀t ∈ [ 0 , T ] , xt , yt ∈ R , where div is the divergence operator in vector calculus . Thm . 1 determines the corresponding dynamical systems of FCMs and ANMs by analyzing their densities and velocity fields ( the details of the proof are in App . D ) , and shows a property of the corresponding dynamical systems of ANMs . Importantly , the property indicates a potential criterion for causal direction determination , that in the causal direction the divergence of the velocity field is zero everywhere , while in the reverse direction it may not always hold . Next , we will verify the criterion rigorously , propose an algorithm based on it , and show the extension for the PNL cases .

Summary: The authors frame the bivariate causal discovery problem in terms of the analysis of a dynamical-system. They use results from the field of optimal transport to interpret additive noise models from this framework. They also develop a novel criterion and a causal discovery algorithm based thereupon, and compare their results with previous methods used in bivariate causal discovery.