Learning latent causal graphs via mixture oracles

1 Introduction . Understanding causal relationships between objects and/or concepts is a core component of human reasoning , and by extension , a core component of artificial intelligence [ 40 , 53 ] . Causal relationships are robust to perturbations , encode invariances in a system , and enable agents to reason effectively about the effects of their actions in an environment . Broadly speaking , the problem of inferring causal relationships can be broken down into two main steps : 1 ) The extraction of high-level causal features from raw data , and 2 ) The inference of causal relationships between these high-level features . From here , one may consider estimating the magnitude of causal effects , the effect of interventions , reasoning about counterfactuals , etc . Our focus in this paper will be the problem of learning causal relationships between latent variables , which is closely related to the problem of learning causal representations [ 63 ] . This problem should be contrasted with the equally important problem of causal inference in the presence of latent confounders [ e.g . 3 , 18 , 35 , 68 , 72 ] ; see also Remark 2.1 . Causal graphical models [ 53 , 54 ] provide a natural framework for this problem , and have long been used to model causal systems with hidden variables [ 23–26 , 58 , 59 ] . It is well-known that in general , without additional assumptions , a causal graphical model given by a directed acyclic graph ( DAG ) is not identifiable in the presence of latent variables [ e.g. , 53 , 71 ] . In fact , this is a generic property of nonparametric structural models : Without assumptions , identifiability is impossible , however , given enough structure , identifiability can be rescued . Examples of this phenomenon include linearity [ 3 , 6 , 15 , 28 , 78 ] , independence [ 1 , 10 , 78 ] , rank [ 15 , 28 ] , sparsity [ 6 ] , and graphical constraints [ 3 , 4 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we consider a general setting for this problem with discrete latent variables , while allowing otherwise arbitrary ( possibly nonlinear ) dependencies . The latent causal graph between the latent variables is also allowed to be arbitrary : No assumptions are placed on the structure of this DAG . We do not assume that the number of hidden variables , their state spaces , or their relationships are known ; in fact , we provide explicit conditions under which all of this can be recovered uniquely . To accomplish this , we highlight a crucial reduction between the problem of learning a DAG model over these variables—given access only to the observed data—and learning the parameters of a finite mixture model . This observation leads to new identifiability conditions and algorithms for learning causal models with latent structure . Overview Our starting point is a simple reduction of the graphical model recovery problem to three modular subproblems : 1 . The bipartite graph Γ between hidden and observed nodes , 2 . The latent distribution P ( 𝐻 ) over the hidden variables 𝐻 , and 3 . A directed acyclic graph ( DAG ) Λ over the latent distribution . From here , the crucial observation is to reduce the recovery problems for Γ and P ( 𝐻 ) to the problem of learning a finite mixture over the observed data . The latter is a well-studied problem with many practical algorithms and theoretical guarantees . We do not require parametric assumptions on this mixture , which allows for very general dependencies between the observed and hidden variables . From this mixture model , we extract what is needed to learn the full graph structure . This perspective leads to a systematic , modular approach for learning the latent causal graph via mixture oracles ( see Section 2 for definitions ) . Ultimately , the application of these ideas requires a practical implementation of this mixture oracle , which is discussed in Section 6 . Contributions More precisely , we make the following contributions : 1 . ( Section 3 ) We provide general conditions under which the latent causal model 𝐺 is identifiable ( Theorem 3.2 ) . Surprisingly , these conditions mostly amount to nondegeneracy conditions on the joint distribution . As we show , without these assumptions identifiability breaks down and reconstruction becomes impossible . 2 . ( Section 4 ) We carefully analyze the problem of reconstructing Γ under progressively weaker assumptions : First , we derive a brute-force algorithm that identifies Γ in a general setting ( Theorem 4.2 ) , and then under a linear independence condition we derive a polynomial-time algorithm based on tensor decomposition and Jennrich ’ s algorithm ( Theorem 4.8 ) . 3 . ( Section 5 ) Building on top of the previous step , where we learn the bipartite graph and sizes of the domains of latent variables , we develop an efficient algorithm for learning the latent distribution P ( 𝐻 ) from observed data ( Theorem 5.4 ) . 4 . ( Section 6-7 ) We implement these algorithms as part of an end-to-end pipeline for learning the full causal graph and illustrate its performance on simulated data . A prevailing theme throughout is the fact that the hidden variables leave a recognizable “ signature ” in the observed data through the marginal mixture models induced over subsets of observed variables . By cleverly exploiting these signatures , the number of hidden variables , their states , and their relationships can be recovered exactly . Previous work Latent variable graphical models have been extensively studied in the literature ; as such we focus only on the most closely related work on causal graphical models here . Early work on this problem includes seminal work by Elidan et al . [ 22 ] , Friedman et al . [ 27 ] , Martin and VanLehn [ 47 ] . More recent work has focused on linear models [ 3 , 28 , 68 , 78 ] or known structure [ 21 , 38 , 65 ] . When the structure is not known a priori , we find ourselves in the realm of structure learning , which is our focus . Less is known regarding structure learning between latent variables for nonlinear models , although there has been recent progress based on nonlinear ICA [ 36 , 50 ] . For example , [ 80 ] proposed CausalVAE , which assumes a linear structural equation model and knowledge of the concept labels for the latent variables , in order to leverage the iVAE model from [ 36 ] . By contrast , our results make no linearity assumptions and do not require these additional labels . While this paper was under review , we were made aware of the recent work [ 46 ] that studies a similar problem to ours in a general , nonlinear setting under faithfulness assumptions . It is also worth noting recent progress on learning discrete Boltzmann machines [ 11 , 12 ] , which can be interpreted as an Ising model with a bipartite structure and a single hidden layer—in particular , there is no hidden causal structure . Nevertheless , this line of work shows that learning Boltzmann machines is computationally hard in a precise sense . More broadly , the problem of learning latent structure has been studied in a variety of other applications including latent Dirichlet allocation [ 8 , 9 ] , phylogenetics [ 51 , 64 ] , and hidden Markov models [ 2 , 30 ] . A prevailing theme in the causal inference literature has been negative results asserting that in the presence of latent variables , causal inference is impossible [ 20 , 32 , 61 , 62 ] . Our results do not contradict this important line of work , and instead adopts a more optimistic tone : We show that under reasonable assumptions—essentially that the latent variables are discrete and well-separated— identifiability and exact recovery of latent causal relationships is indeed possible . This optimistic approach is implicit in recent progress on visual relationship detection [ 52 ] , causal feature learning [ 14 , 43 ] , and interaction modeling [ 37 , 41 ] . In this spirit , our work provides theoretical grounding for some of these ideas . Mixture models and clustering While our theoretical results in Sections 3-5 assume access to a mixture oracle ( see Definition 2.5 ) , in Section 6 we discuss how this oracle can be implemented in practice . To provide context for these results , we briefly mention related work on learning mixture models from data . Mixture models can be learned under a variety of parametric and nonparametric assumptions . Although much is known about parametric models [ e.g . 42 ] , of more interest to us are nonparametric models in which the mixture components are allowed to be flexible , such as mixtures of product distributions [ 31 , 33 ] , grouped observations [ 60 , 75 ] and general nonparametric mixtures [ 7 , 66 ] . In each of these cases , a mixture oracle can be implemented without parametric assumptions . In practice , we use clustering algorithms such as 𝐾-means or hierarchical clustering to implement this oracle . We note also that the specific problem of consistently estimating the order of a mixture model , which will be of particular importance in the sequel , has been the subject of intense scrutiny in the statistics literature [ e.g . 16 , 19 , 39 , 45 ] . Broader impacts and societal impact Latent variable models have numerous practical applications . Many of these applications positively address important social problems , however , these models can certainly be applied nefariously . For example , if the latent variables represent private , protected information , our results imply that this hidden private data can be leaked into publicly released data , which is obviously undesirable . Understanding how to infer unprotected data while safeguarding protected data is an important problem , and our results shed light on when this is and isn ’ t possible . 2 Background . Let 𝐺 = ( 𝑉 , 𝐸 ) be a DAG with 𝑉 = ( 𝑋 , 𝐻 ) , where 𝑋 ∈ R𝑛 denotes the observed part and 𝐻 ∈ Ω : = Ω1 × · · · × Ω𝑚 denotes the hidden , or latent , part . Throughout this paper , we assume that each Ω𝑖 is a discrete space with |Ω𝑖| ≥ 2 . We assume further that there are no edges between observed variables and no edges from observed to hidden variables , and that the distribution of 𝑉 satisfies the Markov property with respect to 𝐺 ( see the supplement for definitions ) . Under these assumptions , 𝐺 decomposes as the union of two subgraphs 𝐺 = Γ ∪ Λ , where Γ is a directed , bipartite graph of edges pointing from 𝐻 to 𝑋 , and Λ is a DAG over the latent variables 𝐻 . Similar assumptions have appeared in previous work [ 3 , 46 , 78 ] , and although nontrivial , they encapsulate our keen interest in reconstructing the structure Λ amongst the latent variables , and captures relevant applications where the relationships between raw observations is less relevant than so-called “ causal features ” [ 13 , 14 ] . See Figure 1 for examples . Throughout this paper , we use standard notation such as pa ( 𝑗 ) for parents , ch ( 𝑗 ) for children , and ne ( 𝑗 ) for neighbors . Given a subset 𝑉 ′ ⊂ 𝑉 , pa ( 𝑉 ′ ) : = ∪𝑗∈𝑉 ′ pa ( 𝑗 ) and given a subgraph 𝐺′ ⊂ 𝐺 , pa𝐺′ ( 𝑉 ′ ) : = pa ( 𝑉 ′ ) ∩𝐺′ , with similar notation for children and neighbors . We let 𝐴 ∈ { 0 , 1 } |𝑋|×|𝐻| denote the adjacency matrix of Γ and denote its columns by 𝑎𝑗 ∈ { 0 , 1 } |𝑋| . Remark 2.1 . Our goal is to learn the hidden variables 𝐻 and the causal graph between them , defined above by Λ . To accomplish this , our main result ( Theorem 3.2 ) shows how to identify ( Γ , P ( 𝐻 ) ) , from which Λ can be recovered ( see Section 3 for details ) . It is important to contrast this problem with problems involving latent confounders [ e.g . 3 , 18 , 35 , 68 , 72 ] , where the goal is to learn the causal graph between the observed variables 𝑋 . In our setting , there are no edges between the observed variables .

Summary: This paper considers learning discrete latent variables (the causes) and their dependencies, given observed variables. It is assumed that there is a DAG between the hidden nodes and a bipartite graph with edges from the hidden nodes to the obseved nodes, and no edges between the observed nodes. The idea is that a distribution of each subset of observed variables is described by a mixture model where each component corresponds to an assignment of values to  hidden variables which are the causes of these observed variables. The paper then presents proofs that from such mixture models (leaned in practice by k-means in the article), the distribution and the graphs related to the hidden nodes can be recovered.