A Distance Correlation-based Kernel for Nonlinear Causal Clustering in Heterogeneous Populations

We consider the problem of causal structure learning in the setting of heterogeneous1 populations , i.e. , populations in which a single causal structure does not adequately2 represent all population members , as is common in biological and social sciences.3 To this end , we introduce a distance covariance-based kernel designed specifically4 to measure the similarity between the underlying nonlinear causal structures of5 different samples . This kernel enables us to perform clustering to identify the6 homogeneous subpopulations . Indeed , we prove the corresponding feature map is7 a statistically consistent estimator of nonlinear independence structure , rendering8 the kernel itself a statistical test for the hypothesis that sets of samples come from9 different generating causal structures . We can then use existing methods to learn10 a causal structure for each of these subpopulations . We demonstrate using our11 kernel for causal clustering with an application in genetics , allowing us to reason12 about the latent transcription factor networks regulating measured gene expression13 levels.14 1 Introduction15 Learning causal relationships from observational and experimental data is one of the fundamental16 goals of scientific research , and causal inference methods are thus used in a wide variety of fields . The17 resulting variety of applications nevertheless share some common difficulties , such as causal inference18 from complex time-series data ( Eichler , 2012 ) or the underlying causal structure being obscured19 by unmeasured confounders ( Greenland et al. , 1999 ) . Another common difficulty , especially for20 applications in the biological and social sciences , is causal inference from heterogeneous populations21 ( Xie , 2013 ; Brand and Thomas , 2013 ) —addressing this difficulty is our main motivation.22 In general terms , we understand a heterogeneous population to be one whose members are not23 adequately described by a single model but rather better described by a collection of models . Within24 our context of causal structure learning , this means a population is heterogeneous if some samples25 are generated by different causal structures—we call this structural heterogeneity . We note that there26 are other kinds of heterogeneity , such as that in samples generated by different joint distributions27 over the same causal structure , which are not the scope of this work.28 A specific example of structural heterogeneity can be found in genetics : causal methods are used to29 learn the structure of gene regulatory networks ( Emmert-Streib et al. , 2012 ) , and gene expression data30 from a single recording or experiment may include thousands of genes , many of which are involved31 in entirely different networks ( Liu , 2015 ) ; thus , attempting to learn a single causal structure for all of32 the genes will obscure the fact that different sets of them have different structures.33 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . The bulk of our work in this paper , and our main contribution , is to introduce the dependence34 contribution kernel , which facilitates a flexible and easily extensible approach to causal clustering:35 first perform clustering to identify structurally homogeneous subsets of samples , and then proceed36 with the actual learning task on each cluster . We prove that our kernel is a statistically consistent37 estimator of the similarity of the causal structures underlying different samples and can thus be used38 to find clusters that minimize structural heterogeneity for causal structure learning tasks . Furthermore,39 the kernel is derived from the distance covariance ( Székely et al. , 2007 ) , imbuing it with the ability40 to detect nonlinear dependence . It can easily be used in a wide array of clustering algorithms , such41 as k-means , DBSCAN , spectral clustering , or any other method that analogously makes use of a42 similarity ( or distance ) measure between samples ( Filippone et al. , 2008 ) .43 The rest of the paper is organized as follows : We finish this section by discussing some of the most44 relevant related work from the causal inference and statistics literature . All of Section 2 is devoted to45 the theory underlying our dependence contribution kernel , including a comparison of the familiar46 product-moment covariance with the distance covariance ( Section 2.1 ) , defining an equivalence47 class of causal models with a convenient representation in the kernel space ( Section 2.2 ) , and the48 actual definition of our kernel and proofs of its relevant properties ( Section 2.3 ) . Next , in Section49 3 , we demonstrate causal clustering with the kernel on a heterogeneous gene expression data set,50 finding structurally homogeneous clusters for which we then learn latent causal measurement models,51 allowing us to reason about the different transcription factor networks responsible for regulating the52 measured gene expression levels . Finally , we conclude in Section 4 mentioning possible future work.53 1.1 Related Work54 Causal inference in heterogeneous populations sometimes refers to data-fusion ( Bareinboim and55 Pearl , 2016 ) , i.e. , combining known homogeneous subpopulations and performing causal inference56 on the resulting heterogeneous population , or similarly , it can refer to meta-learning using known57 subpopulations ( Sharma et al. , 2019 ) . Other times , it refers to estimating heterogeneous treatment58 effects ( Xie et al. , 2012 ; Athey and Imbens , 2015 ) . However , in our case , the subpopulations are not59 known and we rather consider the problem of learning which samples come from which subpopulation,60 and these are differentiated according to structure instead of treatment effect.61 Previous work on causal clustering has focused more on the causal modeling aspect , using stronger62 assumptions about the underlying structures to learn more detailed models . For example , Kummerfeld63 et al . ( 2014 ) ; Kummerfeld and Ramsey ( 2016 ) focus on causal clustering in measurement models,64 with the goal of clustering different features together to study their latent causal structure , based on65 tetrad constraints within the linear product-moment covariance matrix . Huang and Zhang ( 2019 ) 66 define a class of causal models facilitating mechanism-based clustering , learning causal models both67 for clusters of samples as well as a shared one for all samples , assuming the underlying structures68 are linear non-Gaussian . Saeed et al . ( 2020 ) characterize distributions arising from mixtures of69 directed acyclic graph ( DAG ) causal models ( i.e. , causal models without latent or selection variables ) ,70 trying to learn both the component DAGs and a representation of how they are mixed . All of these71 approaches , like most causal inference methods , make specific ( and for some applications , restrictive ) 72 assumptions about the underlying distributions or causal structures.73 In contrast , our method is not tied to specific distributional assumptions such as linearity or74 ( non ) Gaussianity—we assume there are enough samples for statistical inference , as well as the75 usual causal Markov and faithfulness assumptions . For the first step , we cluster samples together if76 they ( implicitly , in the kernel space ) have similar nonlinear independence structures . For the second77 step , causal structure learning , any existing method ( along with its corresponding assumptions ) can in78 principle be used . In our gene expression data application ( Section 3 ) , the measurement dependence79 inducing latent ( MeDIL ) causal model framework ( Markham and Grosse-Wentrup , 2020 ) , which80 assumes the data consists of measurement variables that are causally connected only through latent81 variables , seems appropriate , however other applications can easily use other methods . For example,82 component and mixture DAGs ( Saeed et al. , 2020 ) can be better learned when one first knows which83 samples come from which component—clustering with our kernel ensures samples in different84 clusters come from different DAGs , and so using their method instead of the MeDIL framework85 would be a natural choice for applications in which a DAG ( without any latents ) is more appropriate.86 2 Theory87 2.1 Product-moment Covariance , Distance Covariance , and Dependence Contribution88 Though there is more to causal relationships than probabilistic dependence , causal inference methods89 based on graphical models ultimately rely on at least implicitly learning conditional independence90 ( CI ) relations . CI relations can be estimated in many ways , with different dependence measures91 and tests each having their own theoretical guarantees and being better suited for distributions of92 various different kinds of data ( e.g. , categorical , discrete , or continuous ) and with various kinds93 of relationships ( e.g. , linear , monotonic nonlinear , arbitrary nonlinear ) and with different testing94 assumptions ( see Tjøstheim et al. , 2018 , for a comprehensive overview ) .95 A widely used measure of dependence is the product-moment covariance , often just called covariance,96 which is defined for two zero-mean random variables X1 and X2 as the scalar value cov ( X1 , X2 ) =97 E [ X1X2 ] . This can be extended from a pair of random variables to every pair of variables in a98 random vector , thus returning a matrix instead of a scalar . The covariance matrix for a vector of99 zero-mean random variables X = ( X1 , . . . , Xm ) can be estimated from a set S ∈ Rn , m of n samples100 as Σ̂X = 1nS > S , and the j , j′-th value of Σ̂X is thus the estimate ˆcov ( Xj , X ′j ) .101 Two random variables being probabilistically independent ( denoted ⊥ ) implies that their product-102 moment covariance is zero , i.e. , Xj ⊥ Xj′ =⇒ cov ( Xj , Xj′ ) = 0 ( importantly , the inverse of this103 does not hold ) . Thus , the estimated product-moment covariance can be used in statistical hypothesis104 testing for probabilistic independence ( Wasserman , 2013 , Ch . 10 ) : Xj and Xj′ are assumed to105 be independent if and only if ˆcov ( Xj , Xj′ ) is sufficiently close to 0 . However , this method has106 an important problem : the product-moment covariance is only a valid test statistic against linear107 dependence.108 Székely et al . ( 2007 ) introduce the distance covariance to remedy this problem : random variables are109 probabilistically independent if and only if their distance covariance is zero , i.e. , Xj ⊥ Xj′ ⇐⇒110 dCov ( Xj , Xj′ ) = 0 , resulting in the estimated distance covariance being a valid test statistic against111 all types of dependence . The distance covariance is related to the product-moment covariance by112 dCov2 ( Xj , Xj′ ) = cov ( |Xj −X ′j | , |Xj′ −X ′j′ | ) − 2cov ( |Xj −X ′j | , |Xj′ −X ′′j′ | ) , where ( X ′j , X ′j′ ) 113 and ( X ′′j , X ′′ j′ ) are independent and identically distributed ( iid ) copies of ( Xj , Xj′ ) ( Székely and114 Rizzo , 2014 ) . The key intuition here is that the distances ( e.g. , |Xj −X ′j | ) constitute a nonlinear115 projection , so that using the linear product-moment covariance in this projected space allows for the116 detection of nonlinear dependence in the original space.117 Note that dCov is typically defined to be a scalar value when taken between two arbitrary-dimensional118 random vectors , but our restricted presentation of it above in terms of random variables is to make119 it more obviously analogous to the product-moment covariance between random variables . Thus,120 corresponding to Σ̂X for random vectors , we define the following:121 Definition 1 Let S ∈ Rn , m be a set of n samples from the vector of random variables X =122 ( X1 , . . . , Xm ) . For each j ∈ { 1 , . . . , m } and i , i′ ∈ { 1 , . . . , n } , define the pairwise distance matrix123 Dj , with values given by Dji , i′ : = |Si , j − Si′ , j | . Now define the corresponding doubly-centered124 matrices Cji , i′ : = D j i , i′ − D̄ji , · − D̄j · , i′ + D̄j · , · , where putting a bar over the matrix and replacing125 an index i or i′ with · denotes taking the mean over that index . Define the matrix L ∈ Rn2 , m so126 that each column is a flattened doubly-centered distance matrix , L : = ( vec ( C1 ) , . . . , vec ( Cm ) ) ,127 where vec ( Cj ) denotes “ flattening ” matrix Cj into a column vector . Finally , the estimated distance128 covariance matrix over sample S is defined as ∆̂X : = 1n2L > L.129 Analogous to Σ̂X , the j , j′-th entry of ∆̂X corresponds to ˆdCov 2 ( Xj , Xj′ ) —indeed it is mathemati-130 cally equivalent to computing each pairwise distance covariance value and then manually filling in131 the matrix . The novelty of our Definition 1 is in finding a matrix of pairwise values instead of a single132 value for the distance covariance between random vectors , which helps provide an intuition for our133 next definition:134 Definition 2 Let S ∈ Rn , m be a set of n samples from the vector of random variables X = ( X1 , . . . , Xm ) ; note that we consistently use indices i , i′ ∈ { 1 , . . . , n } and j , j′ ∈ { 1 , . . . , m } . Let D ∈ Rn , n , m denote the 3-dimensional array of stacked pairwise distance matrices defined by Di , i′ , j : = |Si , j − Si′ , j | , and use C ∈ Rn , n , m to denote these same distance matrices after being doubly-centered , i.e. , Ci , i′ , j : = Di , i′ , j−D̄i , · , j−D̄· , i′ , j+D̄· , · , j , where replacing an index i or i′ with · denotes the entire ( lower-dimensional ) subarray over that index , and writing a bar , D̄ , denotes taking the mean over that subarray . Then standardize the doubly-centered distances to get Zi , i′ , j : = Ci , i′ , j D̄· , · , j . Finally , the dependence contribution map , ϕ : Rm → Rm , m , is defined as ϕ ( Si , · ) : = Z > i , · , ·Zi , · , · − T ( α ) , where T ( α ) ∈ Rm , m is a matrix of scaled critical values corresponding to a given significance level135 α with zeros along the diagonal , i.e. , T ( α ) j , j′ = { 0 , if j = j′ 1 nχ 2 1−α ( 1 ) , otherwise , with χ21−α ( 1 ) being the136 1− α quantile of the chi-square distribution with 1 degree of freedom.137 Notice the similarity between Definitions 2 and 1 : if we set T ( α ) to be a matrix of 0s and forgo138 standardization ( i.e. , use C instead of Z ) , then 1n2 ∑n i=1 ϕ ( Si , · ) = ∆̂X . Now , the differences : ∆̂X is139 a single matrix computed over an entire set of samples , whereas ϕ is a map that projects each given140 sample to a new feature space ; each entry of ∆̂X is simply a distance covariance value , whereas each141 entry of the sum of ϕ ( Si , · ) over i , by using standardization ( using Z instead of C ) and subtracting a142 critical value , corresponds to the result of using a distance covariance value in a statistical hypothesis143 test for independence—indeed:144 Lemma 3 Let S ∈ Rn , m be a set of n iid samples from random variables X1 , . . . , Xm with finite first moments . For a given significance level α , under the null hypothesis of Xj ⊥ Xj′ , the test reject h∅ if ( n∑ i=1 ϕ ( Si , · ) ) j , j′ > 0 is statistically consistent against all types of dependence.145 Proof . This follows from ( Székely and Rizzo , 2009 , Theorem 5 and Corollary 2 ) and how ϕ is146 defined to correspond to the difference between distance covariance and critical values . 147 These differences between ∆̂X and ϕ serve two important purposes : first , they ensure ϕ maps to a148 Hilbert space so that our Definition 9 is a corresponding kernel function ( Schölkopf et al. , 2001 ) ; and149 second , as the name “ dependence contribution map ” suggests , they ensure ϕ ( Si , · ) is informative not150 just about distance covariance but about nonlinear dependence and about how the inclusion of sample151 Si , · in a set of samples S contributes to the dependence patterns estimated from S— this is the key152 intuition behind how our kernel function is used to learn structurally homogeneous sample subsets,153 as explicated in the following sections.154 2.2 Causal Graphs in Kernel Space155 In general , a full causal structure can only be learned with sufficient data about the effects of156 interventions , and thus causal structure learning from purely observational data is usually possible157 only up to an equivalence class of causal graphs ( Spirtes et al. , 2000 ; Pearl , 2009 ) . For example , the158 classic PC and IC algorithms , under the assumptions of no selection bias and no confounding by159 latent variables , do not necessarily return a fully-specified DAG but instead return a mixed graph,160 containing possibly directed and undirected edges , representing the Markov equivalence class ( Spirtes161 and Glymour , 1991 ; Pearl and Verma , 1995 ) .162 We now define a set of equivalence classes for ancestral graphs ( AGs ) , which—unlike causal DAGs—163 do not assume the absence of selection bias and latent confounders ( Richardson et al. , 2002 ) :164 Definition 4 Consider an arbitrary ancestral graph A with the set of vertices V A and edge function EA , and denote the set of unconditional m-connection statements entailed by their corresponding unique maximal ancestral graph as MA = { ( j , j′ ) : j /⊥m j′ | ∅ } ⊆ V A × V A . For any ancestral graph A′ such that V A′ = V A , define the unconditional equivalence relation denoted by ‘ ∼U ’ as A ∼U A′ if and only if MA = MA ′ . Lemma 5 This lemma has two parts : ( i ) the relation ∼U is an equivalence relation over the set of165 ancestral graphs A ; ( ii ) for an arbitrary ancestral graphA ∈ A , the bidirected graph UA = ( V A , EU ) ,166 where EU maps all pairs ( j , j′ ) ∈MA to the bidirected edge symbol ‘ ↔ ’ , is a unique representative167 of the equivalence class [ A ] .168 Proof . For ( i ) , recall that an equivalence relation is any relation satisfying reflexivity , symmetry,169 and transitivity ( Devlin , 2003 ) , all of which are satisfied by ∼U because of its correspondence to170 the relation ‘ = ’ between sets . Thus , to prove ( ii ) , it suffices to show that the map s : A/∼U →171 A , [ A ] 7→ UA is injective ( i.e , that it is a section ) and that [ s ( [ A ] ) ] = [ A ] ( Mac Lane , 2013 ) . The172 key to the proof is the observation that UA , because it contains only bidirected edges , is maximal173 and therefore entails exactly the unconditional m-separation statements MA , thus by ( i ) we have174 UA ∼U A or equivalently UA ∈ [ A ] or equivalently [ UA ] = [ A ] . Let A , A′ be arbitrary AGs , and175 assume s ( [ A ] ) = s ( [ A′ ] ) . Then by definition of s we have UA = UA′ , and by the observation above,176 UA ∈ [ A′ ] and thus [ A ] = [ A′ ] , making s injective . And finally , by the definition of s and also by177 the observation above , [ s ( [ A ] ) ] = [ UA ] = [ A ] , completing the proof . 178 This equivalence relation and its representatives has some important but perhaps subtle properties.179 First , it is different from Markov equivalence over AGs ( which is characterized by partial ancestral180 graphs , PAGs ) ( Zhang , 2007 ) —it uses only unconditional m-separation while PAGs are learned from181 conditional m-separation statements . Second , because all DAGs are AGs , ∼U is also an equivalence182 relation over DAGs . Third , being a representative means that every equivalence class includes exactly183 one fully bidirected graph ( along with other equivalent AGs ) . Fourth , because each representative is184 formed by considering m-connected paths , UA is not equivalent to what would be generated by some185 “ edge-wise ” procedure , such as simply replacing every edge in a PAG/AG/DAG/Markov random186 field/moralized DAG with bidirected edges.Finally , its most important property is that it facilitates187 Theorem 8 , for which we first need a few more definitions.188 Definition 6 Given arbitrary ancestral graphs A , A′ ∈ A over the same set of vertices , define the Hamming similarity product , denoted ‘ • ’ as • : A× A→ A and A • A′ 7→ H , whereH = ( V A , EH ) and the function EH ( j , j′ ) = ‘ ↔ ’ if and only if EA ( j , j′ ) = EA′ ( j , j′ ) .189 In words , the Hamming similarity product between two ancestral graphs returns a fully bidirected190 graph , with edges only where the two graphs have the same edge type . Now , shifting from ancestral191 graphs to real-valued square matrices:192 Definition 7 Let ‘ ∼O ’ denote the orthant equivalence relation ( ‘ orthant ’ is the generalization of ‘ quadrant ’ from R2 to arbitrarily higher dimensions ) in square real matrices , i.e. , for matrices Y , Y ′ ∈ Rm , m and with the element-wise function sign ( Y ) j , j′ = { 1 , if Yj , j′ > 0 or j = j′ −1 , otherwise , Y ∼O Y ′ if and only if sign ( Y ) j , j′ = sign ( Y ′ ) j , j′ for all j , j′ . Theorem 8 Let a be the map from the set of unconditional equivalence classes over ancestral graphs193 with m vertices , Am/∼U = Um , to the set of orthant equivalence classes over the image of ϕ,194 i.e. , m×m symmetric real matrices with positive diagonal entries , ϕ ( Rm ) /∼O = Om , defined by195 a : U 7→ O , where Oj , j′ = { 1 , if EU ( j , j′ ) = ‘ ↔ ’ or j = j′ −1 , otherwise . Then a is a group isomorphism196 between ( Um , • ) and ( Om , ) , where ‘ ’ denotes the element-wise product.197 Proof . First , note that ( Um , • ) is indeed a group , satisfying the three group axioms ( Artin , 2011 ) :198 the representative of its identity element is the fully connected bidirected graph over m vertices , U1 ; 199 each element is its own inverse ; and • is associative . Likewise , ( Om , ) is a group with identity200 element [ 1m , m ] , each element its own inverse , and the associative element-wise product operator.201 Now , to show the two groups are isomorphic , it suffices to show ( i ) that a is bijective and ( ii ) that202 for arbitrary U , U ′ ∈ Um , a ( U ) a ( U ′ ) = a ( U • U ′ ) . For ( i ) notice that if U 6= U ′ , then there must203 be at least one pair of vertices j , j′ such that EU ( j , j′ ) 6= EU ′ ( j , j′ ) and thus clearly Oj , j′ 6= O′j , j′ ,204 so a in injective . Furthermore , notice that every distinct O ∈ Om is the image of some graph U ,205 so a is also surjective . For ( ii ) , for every j , j′ ∈ { 1 , . . . , m } , the definitions of a , , and • ensure206 a ( U ) j , j′ a ( U ′ ) j , j′ = 1 ⇐⇒ EU ( j , j′ ) = EU ′ ( j , j′ ) ⇐⇒ 1 = a ( U • U ′ ) , completing the207 proof . 208 For causal inference , which ( often , but not necessarily ) amounts to taking several samples in real209 space and inferring a single corresponding member in the space of ancestral graphs ( or , more often,210 its quotient set by some equivalence relation ) , Theorem 8 means we can compare the different graphs211 of different sample sets without having to first move to the ancestral graph space.212 Finally , notice the space of real square matrices is not a typical sample space but rather precisely ( a213 superspace of ) the space that our dependence contribution map ϕ ( Definition 2 ) maps samples to—this214 means that mapping samples with ϕ allows us to make use of the group isomorphism . Though this215 already provides an intuition for why using ϕ would help with causal clustering , explicitly mapping216 each sample with it would be unnecessarily computationally expensive , and we are ultimately217 interested in morphisms between metric spaces ( not just groups ) of samples and graphs . To address218 this , we thus now move on to defining a kernel for ϕ.219 2.3 The Dependence Contribution Kernel220 Definition 9 Let S , Z , T , and ϕ be as in Definition 2 . We define the dependence contribution kernel using the Frobenius ( denoted by the subscript F ) inner product and norm : κ ( Si , · , Si′ , · ) = 〈ϕ ( Si , · ) , ϕ ( Si′ , · ) 〉F ‖ϕ ( Si , · ) ‖F ‖ϕ ( Si′ , · ) ‖F A more convenient expression for applying the kernel to a data set is obtained by first defining a221 helper kernel , γ along with vec from Definition 1:222 γ ( Si , · , Si′ , · ) = 〈ϕ ( Si , · ) , ϕ ( Si′ , · ) 〉F = ( ( vec ( Zi , · ) > vec ( Zi′ , · ) ) 2 − Zi , ·T Z > i , · − Zi′ , ·T Z > i′ , · + ‖T ‖22 This allows us to write κ ( s , s′ ) = γ ( Si , · , Si′ , · ) γ ( Si , · , Si , · ) 1 2 γ ( Si′ , · , Si′ , · ) 1 2 Finally , note that κ can be readily implemented on an entire set of samples , returning an entire223 Gram ( kernel ) matrix instead of a scalar value , by replacing the matrix operations above with tensor224 operations and specifying the correct axes along which summation occurs—an implementation can225 be found in our open source Python package at https : //non-anonymous-link.after-review.226 A proper distance metric can also be obtained from this kernel through function composition:227 arccos ◦κ . The key idea behind the kernel is that it is the cosine similarity in the space that ϕ maps228 to , meaning for arbitrary sample points x , x′ it evaluates to cos ( θ ) , where θ is the angle between229 ϕ ( x ) and ϕ ( x′ ) . In this space , θ represents the dissimilarity of the dependence patterns underlying230 x and x′ , without being biased by the possibly different magnitudes of ϕ ( x ) and ϕ ( x′ ) due to231 differing variances . Indeed , it can be used as a statistical test of whether samples come from different232 dependence structures and therefore causal models:233 Theorem 10 Let S ∈ Rn , m , S′ ∈ Rn′ , m be sets of n , n′ iid samples drawn respectively from the random variables X = ( X1 , . . . , Xm ) and X ′ = ( X ′1 , . . . , X ′ m ) with finite first moments . Then , n∑ i=1 n′∑ i′=1 κ ( Si , · , S ′ i′ , · ) < 0 =⇒ ∃j , j′ ∈ { 1 , . . . , m } such that I ( Xj , Xj′ , ∅ ) 6= I ( X ′j , X ′j′ , ∅ ) . Proof . Through Slutsky ’ s Theorem ( see Takeshi , 1985 , Theorem 3.2.7 ) and the continuous mapping theorem ( see Van der Vaart , 2000 , Theorem 2.3 ) , the consistency of ϕ ( Lemma 3 ) guarantees the consistency of κ . Because the numerator of κ is a Frobenius inner product of ϕ , n∑ i=1 n′∑ i′=1 κ ( Si , · , S ′ i′ , · ) ∝ n∑ i=1 n′∑ i′=1 m∑ j=1 m∑ j′=1 ϕ ( Si , · ) j , j′ϕ ( S ′ i′ , · ) j , j′ . Thus , in order for ∑ i , i′ κ ( Si , · , S ′ i′ , · ) < 0 , there must be a j and j ′ for which ϕ ( Si , · ) j , j′ > 0 but234 ϕ ( S′i′ , · ) j , j′ < 0 ( or vice versa ) , and thus the hypothesis test in Lemma 3 would reject the null235 hypothesis that Xj ⊥ Xj′ but fail to reject that X ′j ⊥ X ′j′ . 236 Corollary 11 Due to the relationship between independence structure and causal structure , an237 immediate of result of Theorem 10 is that ∑ i , i κ ( Si , · , S ′ i′ , · ) < 0 implies X and X ′ have different238 causal structures.239 Theorem 12 Let d be the distance measure between unconditional equivalence classes of ancestral240 graphs overm vertices , d ( U , U ′ ) = m2−| { ( j , j′ ) : EU •U ′ ( j , j′ ) = ‘ ↔ ’ } |−m . For given sample sets241 S , S′ ( i.e. , real n×mmatrices ) , use ϕ̄ ( S ) to denote the mean of the sample in kernel space , ∑ i ϕ ( Si , . ) ,242 and say S ∼K S′ if and only if ϕ̄ ( S ) ∼O ϕ̄ ( S′ ) ; denote the corresponding quotient set by this243 equivalence class as Rn , m/ ∼K= Kn , m and a representative from each equivalence class as Q ∈ [ S ] .244 Let δ be the distance between sets of samples in K defined as δ ( Q , Q′ ) = m2− 12n2 ∑ i , i′ γ ( Qi , · , Q ′ i , · ) .245 Let b : Um → Kn , m , b : U 7→ Ω , where Ω is the unique element in K such that sign ( ϕ̄ ( Ω ) ) = a ( U ) .246 Then b is a distance-preserving map ( i.e. , an isometry ) from the metric space ( Um , d ) to ( Kn , m , δ ) .247 Proof . Notice that ( Um , d ) is indeed a metric space ( Choudhary , 1993 , Ch . 2 ) : d ( U , U ′ ) = 0 iff248 U−1 •U ′ is the empty graph , which happens iff U = U ′ ; the symmetry of d follows from the symmetry249 • ; and for subadditivity of d , observe that for vertices j , j′ in arbitrary 2-vertex graphs U , U ′ , U ” we250 have either d ( U , U ” ) = 2 , in which case d ( U , U ′ ) + d ( U ′ , U ” ) = 4 , or we have d ( U , U ” ) = 0 , in251 which case d ( U , U ′ ) + d ( U ′ , U ” ) is either 0 or 4—in both cases d ( U , U ” ) ≤ d ( U , U ′ ) + d ( U ′ , U ” ) ; 252 this easily extends to graphs of arbitrary numbers of vertices . Likewise , ( Kn , m , δ ) is a metric space:253 δ ( Q , Q′ ) = 0 ⇐⇒ 12n2 ∑ i , i′ γ ( Qi , · , Q ′ i , · ) = m 2 ⇐⇒ ϕ̄ ( Q ) j , j′ = ϕ̄ ( Q ) j , j′ , for all j , j′ , so iff254 Q = Q′ ; symmetry and subadditivity of δ follow from the symmetry and subadditivity of γ.255 Finally , to show b is an isometry , we must show ( i ) that it is bijective and ( ii ) that for all U , U ′ ∈ Um,256 d ( U , U ′ ) = δ ( b ( U ) , b ( U ′ ) ) . For ( i ) , observe that by the group isomorphism a and definition of b , we257 have U 6= U ′ =⇒ a ( U ) 6= a ( U ′ ) =⇒ Q 6= Q′ =⇒ b ( U ) 6= b ( U ′ ) and so b is injective . Also258 observe that because K is exactly the set of representatives of orthant equivalence classes of sample259 sets in kernel space , then for every Q ∈ K , there exists a U such that b ( U ) = Q , and so b is surjective.260 For ( ii ) , isomorphism a and the relation between element-wise product and Frobenius inner product261 allow us to write d ( U , U ′ ) = m2 − ∑ j , j′ ( O O′ ) j , j′ = m2 − 〈O , O′〉F . Substituting O , O′ with262 their corresponding Ω , Ω′ , and because the Frobenius inner product is a sesquilinear form , we can263 write d ( U , U ′ ) = m2 − 1n2 ∑ i , i′〈ϕ ( Ωi , · ) , ϕ ( Ω′i , · ) 〉F , which by Definition 10 finally gives us that264 d ( U , U ′ ) = δ ( Ω , Ω′ ) , completing the proof . 265 In less formal terms , Theorem 12 shows how the space of unconditional equivalence classes of266 ancestral graph corresponds to the space of real matrices , which is a common space for samples to lie267 in . More specifically , it shows how the structure defined by distances between graphs is the same as268 the structure defined by distances between sets of samples and how this sample distance is related to269 our kernel κ . Note that this is much stronger than Theorem 10 : not only can κ tell us that two sets of270 samples come from different causal models , it gives a measure of just how different the causal models271 are , in terms of their differing unconditional nonlinear independencies/m-separation statements.272 To summarize , we began by defining ϕ ( Definition 2 ) , which maps a given data set into a new273 higher-dimensional feature space . This feature space corresponds to a space of causal graphical274 models , such that samples which are similar in the new feature space must come from similar causal275 models ( Theorem 8 ) . Our main contribution then is to propose the dependence contribution kernel276 κ ( Definition 9 ) .This kernel κ is guaranteed not only to tell us that two sets of samples come from277 different causal models ( Theorem 10 and Corollary 11 ) but furthermore exactly how different the278 causal models are ( Theorem 12 ) , all without the computational expense of explicitly projecting279 samples or learning causal models . Thus , κ is well-suited for addressing the causal clustering280 problem and ensures that resulting clusters will be structurally homogeneous so that subsequent281 causal structure learning will be more informative.282 3 Application283 We use kernel k-means with our dependence contribution kernel to cluster a gene expression data284 set and then use the measurement dependence inducing latent ( MeDIL ) causal model framework285 for structure learning within each cluster ( Markham and Grosse-Wentrup , 2020 ) . The goal of286 causal clustering here is to reason about the different latent transcription factor ( TF ) networks287 governing gene expression ( see Verny et al. , 2017 ; Hackett et al. , 2020 , for other latent causal model288 approaches to learning TF networks ) . The original data set comes from Iyer ( 1999 ) and can be found289 at genome-www.stanford.edu/serum/data/fig2clusterdata.txt , with subsequent analysis290 by Dhillon et al . ( 2003 , 2004 ) . All of the code for our analysis is open source and available at291 https : //non-anonymous-link.after-review.292 The data consists of the measured gene expression levels of 517 different genes from human fibroblast293 cells in response to serum exposure , measured at 11 different time points , i.e. , there are 517 samples294 and 11 different features . In genetics applications , it is not unusual to consider genes to be samples295 and expression ( over time ) to be features—indeed the three previous analyses of this data all have296 this approach—and the intuition is simply that we wish to cluster genes based on patterns in their297 expression levels over time , in order to identify subsets of genes that are controlled by the same gene298 regulatory network . Also notice that such data exemplifies the structurally heterogeneous populations299 discussed in Section 1 : different genes can of course be regulated by different TFs , and so we can300 better represent the data by first clustering it into subpopulations that are more homogeneous and301 then performing causal structure learning on each subpopulation.302 For clustering , we used k = 6 , which we found by looking at both the Variance Ratio Criterion303 ( Caliński and Harabasz , 1974 ) and the Silhouette Coefficients ( Rousseeuw , 1987 ) , computed with the304 scikit-learn machine learning toolbox ( Pedregosa et al. , 2011 ) . We implemented ( unweighted ) kernel305 k-means ourselves , using the pseudocode given by Dhillon et al . ( 2004 ) , with initial mean points306 drawn uniformly at random from the sample set , and with significance level α = 0.1 for the kernel307 parameter T ( α ) . We then used the MeDIL ( Markham et al. , 2020 ) package to learn the dependence308 structure and latent causal models for each cluster.309 Figure 1 shows an example of our results for three of the six gene clusters : Figure 1a shows their310 distance covariance heatmaps and estimated nonlinear dependence structure with significance level311 α = 0.1 ( so the axes are the 11 different features , i.e . the time , in hours , at which gene expression312 level was measured ) , while Figure 1b shows their corresponding causal structures , with measurement313 variables M0–M10 for each of the features and learned latent variables L for different posited TFs.314 The results show a clear difference in causal structure for the different clusters and allow us to reason315 about the latent TFs regulating genes in different clusters : notice that the latents in cluster K1 each316 cause only two or three measurement variables that tend to be close together—e.g. , L1 causes M1317 and M2 , indicating the TF corresponding to L1 is “ short-acting ” , only affecting gene expression318 from 30 minutes ( M1 ) to 1 hour ( M2 ) after serum exposure ; in contrast , the latents in cluster K3319 each cause between two and seven measurement variables that tend to be more spread out—e.g.,320 L1 causes M1 and M7 , indicating the corresponding TF is more complicated , “ long-acting ” but not321 continuously so , affecting gene expression 30 minutes ( M1 ) and 12 hours ( M7 ) after serum exposure,322 but independently of gene expression in the time between.323 Our results are especially noteworthy compared what happens if one ignores the heterogeneity of the324 data and learns a causal structure for the entire data set without first clustering with our kernel : in325 that case , all of the measurement variables are dependent , with a single latent causing all of them,326 and no meaningful conclusions can be drawn about how unmeasured transcription factors regulate327 measured gene expression , i.e. , the heterogeneity obscures the underlying causal structures.328 4 Discussion329 We address the problem of causal clustering—that is , finding the different causal structures underlying330 a structurally heterogeneous data set . Our main contribution is to develop the dependence contribution331 kernel and prove its suitability for the causal clustering task . This allows us to first use the kernel332 with existing clustering methods , such as kernel k-means or DBSCAN , to identify homogeneous333 subpopulations . Then we use existing causal structure learning methods on each subpopulation.334 The kernel guarantees that each subpopulation is more structurally homogeneous and therefore the335 resulting causal structures better capture the causal structures within the data than if a single model336 were learned for the entire heterogeneous population.337 Furthermore , we prove several interesting theoretical properties of our kernel , including ( i ) that338 it can be used as a statistical test for the hypothesis that two sets of samples come from different339 causal structures , as well as ( ii ) how it induces a metric space that is isometric to the one defined340 by Hamming distance between ancestral graphs , i.e. , comparing sets of samples with our kernel is341 equivalent to first estimating the causal graphs of the different sets and then comparing those graphs.342 Beyond the practical applications of our kernel , as shown by our application in reasoning about latent343 transcription factor networks that regulate gene expression , this work also draws from and suggests344 further fruitful connections between a variety of fields , including causal inference , kernel methods,345 and algebraic statistics.346 References347 Artin , M. ( 2011 ) . Algebra . Pearson Prentice Hall.348 Athey , S. and Imbens , G. W. ( 2015 ) . Machine learning methods for estimating heterogeneous causal349 effects . Stat , 1050 ( 5 ) :1–26.350 Bareinboim , E. and Pearl , J . ( 2016 ) . Causal inference and the data-fusion problem . Proceedings of351 the National Academy of Sciences , 113 ( 27 ) :7345–7352.352 Brand , J. E. and Thomas , J. S. ( 2013 ) . Causal effect heterogeneity . In Handbook of Causal Analysis353 for Social Research , pages 189–213 . Springer.354 Cai , H. , Zheng , V. W. , and Chang , K. C.-C. ( 2018 ) . A comprehensive survey of graph embedding:355 Problems , techniques , and applications . IEEE Transactions on Knowledge and Data Engineering,356 30 ( 9 ) :1616–1637.357 Caliński , T. and Harabasz , J . ( 1974 ) . A dendrite method for cluster analysis . Communications in358 Statistics , 3 ( 1 ) :1–27.359 Choudhary , B . ( 1993 ) . The Elements of Complex Analysis . New Age International.360 Devlin , K. ( 2003 ) . Sets , functions , and logic : An introduction to abstract mathematics . CRC Press.361 Dhillon , I. S. , Guan , Y. , and Kulis , B . ( 2004 ) . Kernel k-means , spectral clustering and normalized362 cuts . Proceedings of the 2004 ACM SIGKDD International Conference on Knowledge Discovery363 and Data Mining - KDD ’ 04.364 Dhillon , I. S. , Marcotte , E. M. , and Roshan , U . ( 2003 ) . Diametrical clustering for identifying365 anti-correlated gene clusters . Bioinformatics , 19 ( 13 ) :1612–1619.366 Eichler , M. ( 2012 ) . Causal inference in time series analysis . Wiley Series in Probability and Statistics,367 page 327–354.368 Emmert-Streib , F. , Glazko , G. , Gökmen , A. , and De Matos Simoes , R. ( 2012 ) . Statistical inference369 and reverse engineering of gene regulatory networks from observational expression data . Frontiers370 in Genetics , 3:8.371 Filippone , M. , Camastra , F. , Masulli , F. , and Rovetta , S. ( 2008 ) . A survey of kernel and spectral372 methods for clustering . Pattern recognition , 41 ( 1 ) :176–190.373 Greenland , S. , Pearl , J. , and Robins , J. M. ( 1999 ) . Confounding and collapsibility in causal inference.374 Statistical Science , 14 ( 1 ) .375 Gretton , A. , Bousquet , O. , Smola , A. , and Schölkopf , B . ( 2005 ) . Measuring statistical dependence376 with hilbert-schmidt norms . Algorithmic Learning Theory , pages 63–77.377 Gretton , A. , Fukumizu , K. , Teo , C. , Song , L. , Schölkopf , B. , and Smola , A . ( 2008 ) . A kernel statistical378 test for independence . In Platt , J. , Koller , D. , Singer , Y. , and Roweis , S. , editors , Advances in379 Neural Information Processing Systems 20 , pages 585–592 . MIT Press.380 Hackett , S. R. , Baltz , E. A. , Coram , M. , Wranik , B. J. , Kim , G. , Baker , A. , Fan , M. , Hendrick-381 son , D. G. , Berndl , M. , and McIsaac , R. S. ( 2020 ) . Learning causal networks using inducible382 transcription factors and transcriptome-wide time series . Molecular Systems Biology , 16 ( 3 ) : e9174.383 Huang , B. and Zhang , K. ( 2019 ) . Specific and shared causal relation modeling and mechanism-based384 clustering . Advances in Neural Information Processing Systems ( NeurIPS ) .385 Iyer , V. R. ( 1999 ) . The transcriptional program in the response of human fibroblasts to serum . Science,386 283 ( 5398 ) :83–87.387 Kummerfeld , E. and Ramsey , J . ( 2016 ) . Causal clustering for 1-factor measurement models . In388 Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and389 Data Mining , pages 1655–1664 . ACM.390 Kummerfeld , E. , Ramsey , J. , Yang , R. , Spirtes , P. , and Scheines , R. ( 2014 ) . Causal clustering for391 2-factor measurement models . In Joint European Conference on Machine Learning and Knowledge392 Discovery in Databases , pages 34–49 . Springer.393 Liu , Z.-P. ( 2015 ) . Reverse engineering of genome-wide gene regulatory networks from gene expres-394 sion data . Current Genomics , 16 ( 1 ) :3–22.395 Mac Lane , S. ( 2013 ) . Categories for the working mathematician , volume 5 . Springer Science & 396 Business Media.397 Markham , A. , Chivukula , A. , and Grosse-Wentrup , M. ( 2020 ) . MeDIL : A Python package for causal398 modelling . In Proceedings of the 10th International Conference on Probabilistic Graphical Models399 ( PGM ) . PMLR.400 Markham , A. and Grosse-Wentrup , M. ( 2020 ) . Measurement dependence inducing latent causal401 models . In Conference on Uncertainty in Artificial Intelligence ( UAI ) , pages 590–599 . PMLR.402 Pearl , J . ( 2009 ) . Causality . Cambridge University Press.403 Pearl , J. and Verma , T. ( 1995 ) . A theory of inferred causation . In Studies in Logic and the Foundations404 of Mathematics , volume 134 , pages 789–811 . Elsevier.405 Pedregosa , F. , Varoquaux , G. , Gramfort , A. , Michel , V. , Thirion , B. , Grisel , O. , Blondel , M.,406 Prettenhofer , P. , Weiss , R. , Dubourg , V. , Vanderplas , J. , Passos , A. , Cournapeau , D. , Brucher,407 M. , Perrot , M. , and Duchesnay , E. ( 2011 ) . Scikit-learn : Machine learning in Python . Journal of408 Machine Learning Research , 12:2825–2830.409 Richardson , T. , Spirtes , P. , et al . ( 2002 ) . Ancestral graph markov models . The Annals of Statistics,410 30 ( 4 ) :962–1030.411 Rousseeuw , P. J . ( 1987 ) . Silhouettes : A graphical aid to the interpretation and validation of cluster412 analysis . Journal of Computational and Applied Mathematics , 20:53–65.413 Saeed , B. , Panigrahi , S. , and Uhler , C. ( 2020 ) . Causal structure discovery from distributions arising414 from mixtures of dags . In International Conference on Machine Learning , pages 8336–8345.415 PMLR.416 Schölkopf , B. , Herbrich , R. , and Smola , A. J . ( 2001 ) . A generalized representer theorem . In417 International Conference on Computational Learning Theory , pages 416–426 . Springer.418 Sejdinovic , D. , Sriperumbudur , B. , Gretton , A. , and Fukumizu , K. ( 2013 ) . Equivalence of distance-419 based and RKHS-based statistics in hypothesis testing . The Annals of Statistics , pages 2263–2291.420 Sharma , A. , Gupta , G. , Prasad , R. , Chatterjee , A. , Vig , L. , and Shroff , G. ( 2019 ) . MetaCI : Meta-421 learning for causal inference in a heterogeneous population . CoRR , abs/1912.03960.422 Spirtes , P. and Glymour , C. ( 1991 ) . An algorithm for fast recovery of sparse causal graphs . Social423 Science Computer Review , 9 ( 1 ) :62–72.424 Spirtes , P. , Glymour , C. , and Scheines , R. ( 2000 ) . Causation , Prediction , and Search . MIT Press.425 Székely , G. J. , Rizzo , M. L. , and Bakirov , N. K. ( 2007 ) . Measuring and testing dependence by426 correlation of distances . The Annals of Statistics , 35 ( 6 ) :2769–2794.427 Székely , G. J. and Rizzo , M. L. ( 2009 ) . Brownian distance covariance . The Annals of Applied428 Statistics , 3 ( 4 ) :1236–1265.429 Székely , G. J. and Rizzo , M. L. ( 2014 ) . Partial distance correlation with methods for dissimilarities.430 The Annals of Statistics , 42 ( 6 ) :2382–2412.431 Takeshi , A . ( 1985 ) . Advanced econometrics , volume 1 . Harvard university press.432 Tjøstheim , D. , Otneim , H. , and Støve , B . ( 2018 ) . Statistical dependence : Beyond pearson ’ s ρ. arXiv433 preprint arXiv:1809.10455.434 Van der Vaart , A. W. ( 2000 ) . Asymptotic statistics , volume 3 . Cambridge university press.435 Verny , L. , Sella , N. , Affeldt , S. , Singh , P. P. , and Isambert , H. ( 2017 ) . Learning causal networks436 with latent variables from multivariate information in genomic data . PLoS computational biology,437 13 ( 10 ) : e1005662.438 Wasserman , L. ( 2013 ) . All of statistics : a concise course in statistical inference . Springer Science & 439 Business Media.440 Xie , Y . ( 2013 ) . Population heterogeneity and causal inference . Proceedings of the National Academy441 of Sciences , 110 ( 16 ) :6262–6268.442 Xie , Y. , Brand , J. E. , and Jann , B . ( 2012 ) . Estimating heterogeneous treatment effects with observa-443 tional data . Sociological Methodology , 42 ( 1 ) :314–347.444 Zhang , J . ( 2007 ) . A characterization of markov equivalence classes for directed acyclic graphs with445 latent variables . In Conference on Uncertainty in Artificial Intelligence ( UAI ) .446 Checklist447 1 . For all authors ... 448 ( a ) Do the main claims made in the abstract and introduction accurately reflect the pa-449 per ’ s contributions and scope ? [ Yes ] The first , fifth , and sixth sentences are covered450 thoroughly in Sections 1 , 3 , and 4 , while the rest are covered thoroughly in Section 2.451 ( b ) Did you describe the limitations of your work ? [ Yes ] In Section 1.1 and throughout452 Section 2453 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] Our work454 has no direct potential negative societal impact—just the same indirect potential most455 theoretical work has456 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to457 them ? [ Yes ] 458 2 . If you are including theoretical results ... 459 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] Yes , general460 assumptions in Section 1.1 as well as more specific assumptions within the statement461 of each relevant theorem/lemma/etc.462 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] Proofs follow each463 Theorem and Lemma in the text464 3 . If you ran experiments ... 465 ( a ) Did you include the code , data , and instructions needed to reproduce the main ex-466 perimental results ( either in the supplemental material or as a URL ) ? [ Yes ] Though467 the linked pages contain identifying information , so we ’ ve included only placeholder468 “ https : //non-anonymous-link.after-review ” links469 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they470 were chosen ) ? [ Yes ] 471 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-472 ments multiple times ) ? [ N/A ] We didn ’ t run an experiment multiple times but rather473 analyzed a real data set474 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type475 of GPUs , internal cluster , or cloud provider ) ? [ No ] It runs in just a few seconds , even476 on an old , underpowered laptop477 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 478 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] 479 ( b ) Did you mention the license of the assets ? [ Yes ] We mentioned that it ’ s all open source ; 480 details can be found in their respective repos/documentation481 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 482 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re483 using/curating ? [ Yes ] The data is publicly available484 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable485 information or offensive content ? [ N/A ] It ’ s gene expression data , so neither of these486 are an issue487 5 . If you used crowdsourcing or conducted research with human subjects ... 488 ( a ) Did you include the full text of instructions given to participants and screenshots , if489 applicable ? [ N/A ] 490 ( b ) Did you describe any potential participant risks , with links to Institutional Review491 Board ( IRB ) approvals , if applicable ? [ N/A ] 492 ( c ) Did you include the estimated hourly wage paid to participants and the total amount493 spent on participant compensation ? [ N/A ] 494

Summary: This paper focuses on the problem of causal structural learning from heterogeneous populations, where the label or index of the population is unknown. The main contribution of this paper is on developing a distance covariance-based kernel to measure the similarity between the underlying nonlinear causal structures of different samples, so that we can first perform clustering to identify the homogeneous subpopulations and then use existing methods to learn a causal structure for each of these subpopulations. The proposed method is tested on genetic data.