Question: The example of In Section 4: The Phase I of the algorithm firstly selects the cluster { X_1, X_2, X_3 }. What is the reason? Why not is the { X_4, X_5, X_6 }?
Evidence: Our algorithm, presented in Algorithm 1, consists of three stages. As is common in causal structure learning, we present our algorithm with implementation details of hypothesis testing abstracted away. In particular, we will assume access to two subroutines, whose implementation details will be given in Section 4.1.
The first subroutine tests H ci (X j , X A | X B ), which denotes the null hypothesis that X j and X A are conditionally independent given X B . The second subroutine tests H vt (X A , X B ), which denotes the null hypothesis that all tetrads of Σ A,B vanish.
In the first stage (Algorithm 2, see also Fig.), we identify clusters of observed variables with the same latent parent. However, note that since this stage only identifies leaves with the same latent parent, it is not guaranteed to identify all nodes with the same latent parent. This stage simultaneously recovers an ordering over these clusters. Thus, in the second stage (Algorithm 3, see also Fig.), we iterate over pairs of clusters output from the first stage, identify pairs of clusters
Remaining nodes
Figure: Phase 1: Let the true LFCM be the graph in Fig.). In our algorithm's first phase (see Algorithm 2) we perform the following steps: (a) Compute tetrad scores between all pairs of nodes. (b) For each pair of nodes, test the null hypothesis that all tetrads are zero. Construct a graph with the edge i − j for any pair of nodes where we do not reject the null hypothesis. (c) Extract a clique from this graph to be a cluster (e.g., by picking the largest clique with arbitrary tie breaking), remove these nodes and repeat with remaining nodes.
with the same latent parent, and merge them, while leaving the ordering of the clusters intact. In the third stage (Algorithm 4, see also Fig.), we use the clustering and ordering information discovered in the previous two stages to learn a DAG over both latent and observed variables. In particular, given a node X j in cluster C j which comes before the cluster C i in our ordering, we wish to determine whether X j has an edge to the associated latent variable L i . By Assumption 2, this can be accomplished by checking partial correlations between X j and the nodes in C i . These stages compose a consistent algorithm, as established in the following theorem and proven in Appendix B.
In Section 3, we have already discussed the importance of the first three structural assumptions from Definition 1. In Appendix C, we show how our algorithm fails under a violation of the double-parent assumption.
Theorem 2 Let G be a linear LFCM and let X ∈ R n×p be a matrix of samples of the observed variables X 1 , . . . , X p . Then Algorithm 1 is consistent under Assumptions 1, 2, and 3, i.e., as n → ∞, we have P( Ĝ = G) → 1.
Next, we outline the complexity of our algorithm, using placeholders for the complexities of hypothesis tests in order to keep our results general. Let f (d, n) denote the cost of performing the hypothesis test H vt on d statistics from n samples, when all sufficient statistics are pre-computed. Let g(p) denote the complexity of an algorithm used to find a clique in a graph on p nodes. While finding the largest clique in a graph is in general NP-hard, we can avoid this complexity, since the consistency of our algorithm does not rely on picking the largest clique at each step, only a clique of 3 or more nodes (picking larger cliques is simply a tool for improving statistical accuracy).
In the following, let M be the maximum size of any returned cluster, and let K be the number of clusters discovered by the algorithm. By definition, M ≤ p and K ≤ p, so replacing these quantities by p gives complexities that are only in terms of the known problem parameters. However, such upper bounds can be highly pessimistic. If the true graph has few nodes per cluster, or a small number of latent nodes, then with enough samples, M and K will also be small, respectively.  Proof (a) Computing all tetrads and their associated p-values is O(p 4 ). In each round, we perform O(p 2 ) hypothesis tests, each on O(p 2 ) statistics, so that the complexity at each round is O(p 2 f (p 2 , n)). After performing these tests, we identify the largest clique in a graph of O(p) nodes, so that the total run time per round is O(p 2 f (p 2 , n) + g(p)). At most p rounds are required, resulting in the stated complexity.
(b) We must check O(p 2 ) pairs of clusters for whether or not they should be merged, and the maximum size of the union of any such pair is O(M ). To check whether O(M ) nodes belong to the same cluster, we require a hypothesis test on O(M 4 ) statistics, so that this step takes O(p 2 f (M 4 , n)).
(c) We perform O(pK) hypothesis tests, each based on O(M ) partial correlations.
Assume that we use the Sidak adjustment procedure explained in the next section (so that f (d, n) = O(d)), and a greedy algorithm for picking cliques (so that g(p) = p 3 ). Then, replacing K and M by p, we have that Algorithm 2 takes O(p 5 ), Algorithm 3 takes O(p 6 ), and Algorithm 4 takes O(p 3 ), so that the overall complexity of our algorithm is at most O(p 6 ). Even in this pessimistic analysis, this complexity is relatively low for causal structure learning, which is known to be NPhard in general, and for which variants of the best-known algorithms, such as PC and GES, typically have complexity O(p d+2 ), where d is the maximum in-degree of the graph.
We evaluate our algorithm in two settings. First, we evaluate in a purely synthetic setting, which allows us to generate SCMs which exactly match our proposed model. Then, we evaluate in a semisynthetic setting, modifying real data to more closely match our proposed model while demonstrating that our approach has promise in real-world biological settings.
Figure 2: Phase 1: Let the true LFCM be the graph in Fig. 3(c). In our algorithm’s first phase (see Algorithm 2) we perform the following steps: (a) Compute tetrad scores between all pairs of nodes. (b) For each pair of nodes, test the null hypothesis that all tetrads are zero. Construct a graph with the edge i − j for any pair of nodes where we do not reject the null hypothesis. (c) Extract a clique from this graph to be a cluster (e.g., by picking the largest clique with arbitrary tie breaking),
remove these nodes and repeat with remaining nodes.****Figure 3: Phases 2 and 3. (a) Merge pairs of clusters based on vanishing tetrad tests. (b) Introduce latent nodes, and add edges from latent nodes to children. (c) Add parents of latent nodes based on conditional independence testing
Answer: We assume that the two clusters you refer to with the same tetrad constraints are \{ X_1, X_2, X_3 \} and \{ X_7, X_8, X_9 \} - as seen in Fig. 2b, the cluster \{ X_4, X_5, X_6 \} does not have the same tetrad constraints.
If this interpretation is correct, then you are right, the algorithm can pick either cluster (e.g., selecting between the two at random), or we could have opted to select all clusters at each step. If we instead picked \{ X_7, X_8, X_9 \}, the initial ordering between clusters would be different, but the ordered clustering after Phase 2 - the “merging” step in Algorithm 3 - would be the same, leading to the same final result.
We will clarify that this “tie-breaking” does not affect the final result. The true graph used in our examples is given in Fig 3c as the final output of the algorithm; we will move this to come before Fig 2.