Learning latent causal graphs via mixture oracles

1 Introduction . Understanding causal relationships between objects and/or concepts is a core component of human reasoning , and by extension , a core component of artificial intelligence [ 40 , 53 ] . Causal relationships are robust to perturbations , encode invariances in a system , and enable agents to reason effectively about the effects of their actions in an environment . Broadly speaking , the problem of inferring causal relationships can be broken down into two main steps : 1 ) The extraction of high-level causal features from raw data , and 2 ) The inference of causal relationships between these high-level features . From here , one may consider estimating the magnitude of causal effects , the effect of interventions , reasoning about counterfactuals , etc . Our focus in this paper will be the problem of learning causal relationships between latent variables , which is closely related to the problem of learning causal representations [ 63 ] . This problem should be contrasted with the equally important problem of causal inference in the presence of latent confounders [ e.g . 3 , 18 , 35 , 68 , 72 ] ; see also Remark 2.1 . Causal graphical models [ 53 , 54 ] provide a natural framework for this problem , and have long been used to model causal systems with hidden variables [ 23–26 , 58 , 59 ] . It is well-known that in general , without additional assumptions , a causal graphical model given by a directed acyclic graph ( DAG ) is not identifiable in the presence of latent variables [ e.g. , 53 , 71 ] . In fact , this is a generic property of nonparametric structural models : Without assumptions , identifiability is impossible , however , given enough structure , identifiability can be rescued . Examples of this phenomenon include linearity [ 3 , 6 , 15 , 28 , 78 ] , independence [ 1 , 10 , 78 ] , rank [ 15 , 28 ] , sparsity [ 6 ] , and graphical constraints [ 3 , 4 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we consider a general setting for this problem with discrete latent variables , while allowing otherwise arbitrary ( possibly nonlinear ) dependencies . The latent causal graph between the latent variables is also allowed to be arbitrary : No assumptions are placed on the structure of this DAG . We do not assume that the number of hidden variables , their state spaces , or their relationships are known ; in fact , we provide explicit conditions under which all of this can be recovered uniquely . To accomplish this , we highlight a crucial reduction between the problem of learning a DAG model over these variables—given access only to the observed data—and learning the parameters of a finite mixture model . This observation leads to new identifiability conditions and algorithms for learning causal models with latent structure . Overview Our starting point is a simple reduction of the graphical model recovery problem to three modular subproblems : 1 . The bipartite graph Γ between hidden and observed nodes , 2 . The latent distribution P ( 𝐻 ) over the hidden variables 𝐻 , and 3 . A directed acyclic graph ( DAG ) Λ over the latent distribution . From here , the crucial observation is to reduce the recovery problems for Γ and P ( 𝐻 ) to the problem of learning a finite mixture over the observed data . The latter is a well-studied problem with many practical algorithms and theoretical guarantees . We do not require parametric assumptions on this mixture , which allows for very general dependencies between the observed and hidden variables . From this mixture model , we extract what is needed to learn the full graph structure . This perspective leads to a systematic , modular approach for learning the latent causal graph via mixture oracles ( see Section 2 for definitions ) . Ultimately , the application of these ideas requires a practical implementation of this mixture oracle , which is discussed in Section 6 . Contributions More precisely , we make the following contributions : 1 . ( Section 3 ) We provide general conditions under which the latent causal model 𝐺 is identifiable ( Theorem 3.2 ) . Surprisingly , these conditions mostly amount to nondegeneracy conditions on the joint distribution . As we show , without these assumptions identifiability breaks down and reconstruction becomes impossible . 2 . ( Section 4 ) We carefully analyze the problem of reconstructing Γ under progressively weaker assumptions : First , we derive a brute-force algorithm that identifies Γ in a general setting ( Theorem 4.2 ) , and then under a linear independence condition we derive a polynomial-time algorithm based on tensor decomposition and Jennrich ’ s algorithm ( Theorem 4.8 ) . 3 . ( Section 5 ) Building on top of the previous step , where we learn the bipartite graph and sizes of the domains of latent variables , we develop an efficient algorithm for learning the latent distribution P ( 𝐻 ) from observed data ( Theorem 5.4 ) . 4 . ( Section 6-7 ) We implement these algorithms as part of an end-to-end pipeline for learning the full causal graph and illustrate its performance on simulated data . A prevailing theme throughout is the fact that the hidden variables leave a recognizable “ signature ” in the observed data through the marginal mixture models induced over subsets of observed variables . By cleverly exploiting these signatures , the number of hidden variables , their states , and their relationships can be recovered exactly . Previous work Latent variable graphical models have been extensively studied in the literature ; as such we focus only on the most closely related work on causal graphical models here . Early work on this problem includes seminal work by Elidan et al . [ 22 ] , Friedman et al . [ 27 ] , Martin and VanLehn [ 47 ] . More recent work has focused on linear models [ 3 , 28 , 68 , 78 ] or known structure [ 21 , 38 , 65 ] . When the structure is not known a priori , we find ourselves in the realm of structure learning , which is our focus . Less is known regarding structure learning between latent variables for nonlinear models , although there has been recent progress based on nonlinear ICA [ 36 , 50 ] . For example , [ 80 ] proposed CausalVAE , which assumes a linear structural equation model and knowledge of the concept labels for the latent variables , in order to leverage the iVAE model from [ 36 ] . By contrast , our results make no linearity assumptions and do not require these additional labels . While this paper was under review , we were made aware of the recent work [ 46 ] that studies a similar problem to ours in a general , nonlinear setting under faithfulness assumptions . It is also worth noting recent progress on learning discrete Boltzmann machines [ 11 , 12 ] , which can be interpreted as an Ising model with a bipartite structure and a single hidden layer—in particular , there is no hidden causal structure . Nevertheless , this line of work shows that learning Boltzmann machines is computationally hard in a precise sense . More broadly , the problem of learning latent structure has been studied in a variety of other applications including latent Dirichlet allocation [ 8 , 9 ] , phylogenetics [ 51 , 64 ] , and hidden Markov models [ 2 , 30 ] . A prevailing theme in the causal inference literature has been negative results asserting that in the presence of latent variables , causal inference is impossible [ 20 , 32 , 61 , 62 ] . Our results do not contradict this important line of work , and instead adopts a more optimistic tone : We show that under reasonable assumptions—essentially that the latent variables are discrete and well-separated— identifiability and exact recovery of latent causal relationships is indeed possible . This optimistic approach is implicit in recent progress on visual relationship detection [ 52 ] , causal feature learning [ 14 , 43 ] , and interaction modeling [ 37 , 41 ] . In this spirit , our work provides theoretical grounding for some of these ideas . Mixture models and clustering While our theoretical results in Sections 3-5 assume access to a mixture oracle ( see Definition 2.5 ) , in Section 6 we discuss how this oracle can be implemented in practice . To provide context for these results , we briefly mention related work on learning mixture models from data . Mixture models can be learned under a variety of parametric and nonparametric assumptions . Although much is known about parametric models [ e.g . 42 ] , of more interest to us are nonparametric models in which the mixture components are allowed to be flexible , such as mixtures of product distributions [ 31 , 33 ] , grouped observations [ 60 , 75 ] and general nonparametric mixtures [ 7 , 66 ] . In each of these cases , a mixture oracle can be implemented without parametric assumptions . In practice , we use clustering algorithms such as 𝐾-means or hierarchical clustering to implement this oracle . We note also that the specific problem of consistently estimating the order of a mixture model , which will be of particular importance in the sequel , has been the subject of intense scrutiny in the statistics literature [ e.g . 16 , 19 , 39 , 45 ] . Broader impacts and societal impact Latent variable models have numerous practical applications . Many of these applications positively address important social problems , however , these models can certainly be applied nefariously . For example , if the latent variables represent private , protected information , our results imply that this hidden private data can be leaked into publicly released data , which is obviously undesirable . Understanding how to infer unprotected data while safeguarding protected data is an important problem , and our results shed light on when this is and isn ’ t possible . 2 Background . Let 𝐺 = ( 𝑉 , 𝐸 ) be a DAG with 𝑉 = ( 𝑋 , 𝐻 ) , where 𝑋 ∈ R𝑛 denotes the observed part and 𝐻 ∈ Ω : = Ω1 × · · · × Ω𝑚 denotes the hidden , or latent , part . Throughout this paper , we assume that each Ω𝑖 is a discrete space with |Ω𝑖| ≥ 2 . We assume further that there are no edges between observed variables and no edges from observed to hidden variables , and that the distribution of 𝑉 satisfies the Markov property with respect to 𝐺 ( see the supplement for definitions ) . Under these assumptions , 𝐺 decomposes as the union of two subgraphs 𝐺 = Γ ∪ Λ , where Γ is a directed , bipartite graph of edges pointing from 𝐻 to 𝑋 , and Λ is a DAG over the latent variables 𝐻 . Similar assumptions have appeared in previous work [ 3 , 46 , 78 ] , and although nontrivial , they encapsulate our keen interest in reconstructing the structure Λ amongst the latent variables , and captures relevant applications where the relationships between raw observations is less relevant than so-called “ causal features ” [ 13 , 14 ] . See Figure 1 for examples . Throughout this paper , we use standard notation such as pa ( 𝑗 ) for parents , ch ( 𝑗 ) for children , and ne ( 𝑗 ) for neighbors . Given a subset 𝑉 ′ ⊂ 𝑉 , pa ( 𝑉 ′ ) : = ∪𝑗∈𝑉 ′ pa ( 𝑗 ) and given a subgraph 𝐺′ ⊂ 𝐺 , pa𝐺′ ( 𝑉 ′ ) : = pa ( 𝑉 ′ ) ∩𝐺′ , with similar notation for children and neighbors . We let 𝐴 ∈ { 0 , 1 } |𝑋|×|𝐻| denote the adjacency matrix of Γ and denote its columns by 𝑎𝑗 ∈ { 0 , 1 } |𝑋| . Remark 2.1 . Our goal is to learn the hidden variables 𝐻 and the causal graph between them , defined above by Λ . To accomplish this , our main result ( Theorem 3.2 ) shows how to identify ( Γ , P ( 𝐻 ) ) , from which Λ can be recovered ( see Section 3 for details ) . It is important to contrast this problem with problems involving latent confounders [ e.g . 3 , 18 , 35 , 68 , 72 ] , where the goal is to learn the causal graph between the observed variables 𝑋 . In our setting , there are no edges between the observed variables .

Summary: This paper proposes a method for recovering a distribution over a set of discrete latent random variables, given observations from a set of observed children of these variables. The method additionally recovers the set of edges between latent and observed variables. The method is written in terms of a “mixture oracle”, i.e., a subroutine which returns mixture weights and components for the distribution over any subset of observed variables. Empirical results show that the method is effective at recovering the underlying causal graph.