SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Interpretation. Theorem 3.4 shows that invoking Theorem 3.2 with the assignment  $\mathbf{v}_1 = \mathbf{x}$  and  $\mathbf{v}_2 = \mathbf{X} \backslash \{\mathbf{x}\}$  can identify the parents of  $\mathbf{x}$  in the hierarchical model. Intuitively, we can identify latent variables one "level" above the current level  $\mathbf{X}$ . Moreover, as the estimated variables are equivalent to true variables up to one-to-one mappings, we take a step further in Section 3.3 to show that this procedure can be applied to the newly estimated variables iteratively to traverse the hierarchical model "level" by "level". In the degenerate case where  $\mathbf{v}_1$  does not have any parents, i.e., the root of the hierarchical model, the identified variable  $\mathbf{z}$  in the basis model corresponds to the  $\mathbf{v}_1$  itself - we can regard  $\mathbf{s}_1$  as a constant and  $g_1(\cdot)$  as an identity function in Theorem 3.2.
3To avoid cluttering, we slightly abuse the bold lowercase font to represent either an individual vector or a vector set (which can be viewed as a concatenation).
(b)
Figure 3: Evolution of active set  $\mathbf{A}$  in Algorithm 1. We mark the active set  $\mathbf{A}$  with shaded gray circles before each iterations of Algorithm 1, with Figure 3a, Figure 3b, and Figure 3c corresponding to iteration 1, 2, and 3. Before iteration 1,  $\mathbf{A}$  is set to  $\mathbf{X}$  by default. At iteration 1,  $\mathbf{z}_2$ ,  $\mathbf{z}_3$ , and  $\mathbf{z}_4$  can be estimated by the basis model; however, only  $\mathbf{z}_4$  can be updated into  $\mathbf{A}$ . Otherwise, directed paths would be introduced by  $\mathbf{z}_2$  and  $\mathbf{z}_3$ .
Discussion on assumptions. Assumption 3.3-ii corresponds to Assumption 3.1-i for the basis model. Intuitively, it states that the information in the system is preserved without unnecessary duplication. As far as we know, the existing literature on causal variable identification for nonlinear models [Hyvarinen et al., 2019, Khemakhem et al., 2020, Kong et al., 2022, Lyu et al., 2022, Yao et al., 2022] universally assumes the invertibility of such a mapping. Without this assumption, we cannot guarantee the identification of latent variables from observed variables in the nonlinear case.
Assumption 3.3-iii is an extension of Assumption 3.1-ii to the hierarchical case, with  $\mathbf{A}$  and  $\mathbf{z}_{\mathbf{A}}$  playing the role of  $(\mathbf{v}_1,\mathbf{v}_2)$  and  $\mathbf{z}$  respectively.
Assumption 3.3-iv excludes degenerate cases where an edge connects components that do not influence each other. This is equivalent to the causal minimality and generally considered necessary for causal identification with observational data [Peters et al., 2017]. For instance, we cannot distinguish  $Y \coloneqq f(X) + N_{Y}$  and  $Y \coloneqq c + N_{Y}$  if  $f$  is allowed to differ from constant  $c$  only outside  $X$ 's support. Further, when  $\mathbf{v}_1 = \mathbf{x}$  is a pure child of a latent variable  $\mathbf{z}$ , Condition 2.4-i ensures that  $\mathbf{v}_2 \coloneqq \mathbf{A}_{\mathbf{z}} \setminus \{\mathbf{v}_1\}$  contains at least one pure child or descendant of  $\mathbf{z}$  to fulfill Assumption 3.1-iii.
Implications on measurement causal models. Despite being an intermediate step towards the global structure identification, Theorem 3.4 can be applied to a class of nonlinear measurement models with arbitrary latent structures and identify all the latent variables. Then, the latent structures in the measurement model can be identified by performing existing causal discovery algorithms, such as the PC algorithm [Spirtes et al., 2001], on the identified latent variables. We leave the detailed presentation of this application as future work.
3.3 Global Identifiability of Latent Hierarchical Causal Models
Equipped with the local identifiability (i.e., Theorem 3.4), the following theorem shows that all causal structures and latent variables are identifiable in the hierarchical model.
Theorem 3.5. Under assumptions in Theorem 3.4, all latent variables  $\mathbf{Z}$  in the hierarchical model can be identified up to one-to-one mappings, and the causal structure  $\mathbf{G}$  can also be identified.
Comparison to prior work. Theorem 3.5 handles general nonlinear cases, whereas prior work [Pearl, 1988, Choi et al., 2011, Huang et al., 2022, Xie et al., 2022] is limited to linear function or discrete latent variable conditions. Structure-wise, our identifiability results apply to latent structures with V-structures and certain triangle structures (see discussion in Appendix C.1) beyond generalized trees as studied in prior work [Choi et al., 2011, Pearl, 1988, Huang et al., 2022], require fewer pure children and no neighboring variables in comparison with [Xie et al., 2022, Huang et al., 2022], and can determine directions for each edge (c.f., Markov-equivalent classes in Huang et al. [2022]).
Proof sketch with search procedures. The proof can be found in Appendix B.3. In particular, we develop a concrete algorithm (i.e., Algorithm 1) and show that it can successfully identify the causal structure and latent variables asymptotically. We give a proof sketch of Theorem 3.4 below by navigating through Algorithm 1 and illustrate it with an example in Figure 3.
Stage 1: identifying parents with the basis model (line 4-line 5). As shown in Theorem 3.4, the basis model can identify the latent parents of leaf-level variables in the hierarchy. In Figure 3a, we can identify  $\mathbf{z}_2$  when applying the basis model with the assignment  $\mathbf{v}_1 \coloneqq \{\mathbf{x}_1\}$  and  $\mathbf{v}_2 \coloneqq \mathbf{X} \setminus \mathbf{v}_1$ . Naturally, the basic idea of Algorithm 1 is to iteratively apply the basis model to the most recently
<sup>4</sup>We refer to Silva et al. [2006b] for a general measurement model definition. Here, we consider a popular type of measurement models that has been widely used in the literature (see Xie et al. [2020]) in which observed variables do not cause other variables.
identified latent variables to further identify their parents, which is the purpose of Stage 1 (line 4-line 5). In Algorithm 1, we define as active set  $\mathbf{A}$  the set of variables to which we apply the basis model. For example,  $\mathbf{A}$  equals to  $\mathbf{X}$  in the first round (Figure 3a) and becomes  $\{\mathbf{x}_1,\mathbf{x}_2,\mathbf{x}_3,\mathbf{z}_4,\mathbf{x}_7,\mathbf{x}_8,\mathbf{x}_9\}$  in the second round (Figure 3b).
Stage 2: merging duplicate variables (line 6). As multiple variables in  $\mathbf{A}$  can share a parent, dictionary  $\mathbf{P}(\cdot)$  may contain multiple variables that are one-to-one mappings to each other, which would obscure the true causal structure and increase the algorithm's complexity. Stage 2 (Line 6) detects such duplicates and represents them with one variable. In Figure 3a, setting  $\mathbf{v}_1$  to any of  $\mathbf{x}_1$ ,  $\mathbf{x}_2$ , and  $\mathbf{x}_3$  would yield an equivalent variable of  $\mathbf{z}_2$ . We merge the three equivalents by randomly selecting one.
Stage 3: detecting and merging super-variables (line 7 - line 9). Due to the potential existence of V-structures, variables in  $\mathbf{A}$  may have multiple parents and produce super-variables in  $\mathbf{P}$ . For instance, at the second iteration of Algorithm 1 (i.e., Figure 3b), the basis model will be run on  $\mathbf{v}_1 = \mathbf{z}_4$  and  $\mathbf{v}_2 = \{\mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_3, \mathbf{x}_7, \mathbf{x}_8, \mathbf{x}_9\}$  and output a variable equivalent to the concatenation  $(\mathbf{z}_2, \mathbf{z}_3)$ . Leaving this super-variable untouched will be problematic, as we would generate a false causal structure  $\tilde{\mathbf{z}} \rightarrow \mathbf{z}_4$  where  $\tilde{\mathbf{z}}$  is the estimated super-variable  $(\mathbf{z}_2, \mathbf{z}_3)$ , rather than recognizing  $\mathbf{z}_4$  is the child of two already identified variables  $\mathbf{z}_2$  and  $\mathbf{z}_3$ . Line 7 to line 9 detect such super-variables by testing whether each variable  $\hat{\mathbf{z}}$  in  $\mathbf{P}$  is equivalent to a union of other variables in  $\mathbf{P}$ . If such a union  $\tilde{\mathbf{Z}} := \{\hat{\mathbf{z}}_1, \dots, \hat{\mathbf{z}}_m\}$  exists, we will replace  $\hat{\mathbf{z}}$  with  $\tilde{\mathbf{Z}}$  in all its occurrences. In Figure 3b, we would split the variable equivalent to  $[\mathbf{z}_2, \mathbf{z}_3]$  into variables of  $\mathbf{z}_2$  and  $\mathbf{z}_3$  individually. If  $\hat{\mathbf{z}}$  is tested to be a super-variable, i.e., it can perfectly predict another variable  $\hat{\mathbf{z}}'$  in  $\mathbf{P}$  and  $\hat{\mathbf{z}}'$  cannot predict  $\hat{\mathbf{z}}$  perfectly, and the equivalent union cannot be found, we will track  $\hat{\mathbf{z}}$  in line 9 to prevent it from being updated into  $\mathbf{A}$  at line 16.
Stage 4: detecting and avoiding directed paths in  $\mathbf{A}$  (line 12-line 14). Ideally, we would like to repeat line 4 to line 9 until reaching the root of the hierarchical model. Unfortunately, such an approach can be problematic, as this would cause variables in active set  $\mathbf{A}$  to have directed edges among them, whereas Theorem 3.4 applies to leaf variable set  $\mathbf{X}$  which contains no directed edges. In Figure 3a,  $\mathbf{P}$  would contain  $\{\mathbf{z}_2,\mathbf{z}_3,\mathbf{z}_4\}$ , as each of these latent variables has pure observed children in  $\mathbf{X}$ . However, due to direct paths within  $\mathbf{P}$ , i.e.,  $\mathbf{z}_2\rightarrow \mathbf{z}_4$  and  $\mathbf{z}_3\rightarrow \mathbf{z}_4$ , we cannot directly substitute  $\mathbf{X}$  with  $\{\mathbf{z}_2,\mathbf{z}_3,\mathbf{z}_4\}$  in  $\mathbf{A}$ . To resolve this dilemma, we proactively detect directed paths emerging with newly estimated variables and defer the local update of such estimated variables to eliminate direct paths. For directed path detection, we introduce a corollary of Theorem 3.4 as Corollary 3.6.
Corollary 3.6. Under assumptions in Theorem 3.4, for any  $\mathbf{z} \in Pa(\mathbf{A})$  where  $\mathbf{A}$  is the active set in Algorithm 1, we consider  $\mathbf{v}_1 \coloneqq \mathbf{z}$  and  $\mathbf{v}_2 \coloneqq \mathbf{A} \backslash (\tilde{C} h(\mathbf{z}) \cap \mathbf{A})$  where  $\tilde{C} h(\mathbf{z})$  is a strict subset of  $\mathbf{z}$ 's children (i.e., when the active set  $\mathbf{A}$  contains at least one child of  $\mathbf{z}$ 's), estimating the basis model yields  $\hat{\mathbf{z}}$  equivalent to  $\mathbf{z}$  up to a one-to-one mapping.
Corollary 3.6 can be obtained by setting  $\mathbf{v}_1$  as an ancestor of  $\mathbf{v}_2$  and  $\mathbf{s}_1$  as a degenerate variable (i.e., a deterministic quantity) in Theorem 3.4. Leveraging Corollary 3.6, we can detect whether each newly identified latent variable in  $\mathbf{P}$  would introduce directed paths into  $\mathbf{A}$  if they were substituted in. If directed paths exist, the variable  $\hat{\mathbf{z}}$  would contain the same information as  $\mathbf{v}_1\coloneqq \mathbf{z}$ , which prediction tests can evaluate. In this event, we will suppress the update of the  $\mathbf{z}$  at this iteration. That is, we still keep its children in  $\mathbf{A}$ . This directed-path detection is conducted in lines 12- 14, after properly grouping variables that share children in lines 10- 11. As shown in Figure 3b,  $\mathbf{z}_2$  and  $\mathbf{z}_3$  are not placed in  $\mathbf{A}$ , even if they are found in the first iteration. This update only happens when  $\mathbf{z}_4$  has been placed in  $\mathbf{A}$  at the second iteration.
Generally, a latent variable enters  $\mathbf{A}$  only if all its children reside in  $\mathbf{A}$ . We can show that  $\mathbf{A}$  contains all the information of unidentified latent variables -  $\mathbf{A}$  d-separates the latent variables that have not been placed in  $\mathbf{A}$  and those were in  $\mathbf{A}$  once. Equipped with such a procedure, we can identify the hierarchical model iteratively until completion. We discuss Algorithm 1's complexity in Appendix D.
In this section, we present experiments to corroborate our theoretical results in Section 3. We start with the problem of recovering the basis model in Section 4.2, which is the foundation of the overall identifiability. In Section 4.3, we present experiments for hierarchical models on a synthetic dataset and two real-world datasets.
Algorithm 1 Identification of Latent Hierarchical Models. A: the active set, X: the observed variable set, P/JointP: dictionaries that store the relations between variables in A and estimated variables.
1: Initialize the active set:  $\mathbf{A}\gets \mathbf{X}$    
2: while  $\mathbf{A}\neq \emptyset$  do   
3: initialize an empty set  $\mathbf{R}$  and empty dictionaries  $\mathbf{P}$  ,JointP;   
4: for each active variable  $\mathbf{a}\in \mathbf{A}$  do   
5: estimate the basis model with  $\mathbf{v}_1 = \mathbf{a}$  and  $\mathbf{v}_2 = \mathbf{A}\backslash \{\mathbf{a}\}$  to obtain  $\hat{\mathbf{z}}$  ,and  $\mathbf{P}\leftarrow \mathbf{P}\cup \{(a,\hat{\mathbf{z}})\}$    
6: merge equivalent variables in  $\mathbf{P}$  .   
7: for each variable  $\hat{\mathbf{z}}$  in  $\mathbf{P}$  do   
8: if  $\exists \hat{\mathbf{z}}^{\prime}\in \mathbf{P}$  that  $\hat{\mathbf{z}}$  can perfectly predict but  $\hat{\mathbf{z}}^{\prime}$  cannot predict  $\hat{\mathbf{z}}$  then if  $\exists \tilde{\mathbf{Z}}\subseteq \mathbb{P}\setminus \{\hat{\mathbf{z}}\}$  that perfectly predicts  $\hat{\mathbf{z}}$  ,then replace  $\hat{\mathbf{z}}$  with  $\tilde{\mathbf{Z}}$  ; else:  $\mathbf{R}\leftarrow \mathbf{R}\cup \{\hat{\mathbf{z}}\}$    
10: cluster  $\hat{\mathbf{z}}$  variables in  $\mathbf{P}$  into  $\{\mathbf{Z}_i\}_{i = 1}^m$  such that  $\mathbf{Z}_i$  is either a singleton or contains spouse variables; store each cluster  $Z_{i}$  and its children set  $\mathbf{H}_i$  as a pair  $(\mathbf{Z}_i,\mathbf{H}_i)$  into JointP;   
12: for each pair  $(\mathbf{Z}_i,\mathbf{H}_i)\in$  JointP do   
13: estimate the basis model with  ${\mathrm{v}}_1={\mathrm{Z}}_i$  and  ${\mathrm{v}}_2={\mathrm{A}}\backslash{\mathrm{H}}_i$  to obtain a variable  $\hat{\mathbf{z}}_{\mathrm{test}}$  . if  $\exists {\bf z}^{\prime}\in {\bf Z}_{i}$  such that  $\hat{\mathbf{z}}_{\mathrm{test}}$  can perfectly predict  ${\bf z}^{\prime}$  ,then  $\mathbf{R}\leftarrow \mathbf{R}\cup \mathbf{Z}_i$    
15: for each pair  $(\mathbf{Z}_i,\mathbf{H}_i)\in$  JointP do   
16: if no variables in  $Z_{i}$  has been tracked by R, i.e.,  $Z_{i}\cap R = \emptyset$  , then substitute.  $\mathbf{H}_i$  with  $Z_{i}$  in A. remove variable a from A if a I A \\ {a}.   
18: Return: all the past active sets A and parent sets P.
Figure 4: Evaluated hierarchical models. We denote the estimation  $R^2$  scores around corresponding latent variables. We can observe that all latent variables can be identified decently, justifying our theoretic results.
Synthetic data generation. For the basis model (Figure 2), we sample  $\mathbf{z} \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ ,  $\mathbf{s}_1 \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  and  $\mathbf{s}_2 \sim \mathcal{N}(\mathbf{A}\mathbf{z} + \mathbf{b},\mathbf{I})$ , where the dependence between  $\mathbf{z}$  and  $\mathbf{s}_2$  is implemented by randomly constructed matrix  $\mathbf{A}$  and bias  $\mathbf{b}$ . We choose the true mixing function  $\mathbf{g}$  as a multilayer perceptron (MLP) with Leaky-ReLU activations and well-conditioned weights to facilitate invertibility. We apply element-wise  $\max\{z,0\}$  to  $\mathbf{z}$  before inputting  $[\mathbf{z},\mathbf{s}_1]$  to  $g_1$  and element-wise  $\min\{z,0\}$  to  $\mathbf{z}$  before inputting  $[\mathbf{z},\mathbf{s}_2]$  to  $g_2$ , so that  $\mathbf{v}_1$  is generated by the positive elements of  $\mathbf{z}$  and  $\mathbf{v}_2$  is generated by the negative elements of  $\mathbf{z}$ . This way,  $g_1$  and  $g_2$  are jointly invertible but not individually invertible. For latent hierarchical models (Figure 4), we sample each exogenous variable  $\varepsilon$  as  $\varepsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  and each endogenous variable  $\mathbf{z}$  is endowed with a distinct generating function  $g_{\mathbf{z}}$  parameterized by an MLP, i.e.,  $\mathbf{z} = g_{\mathbf{z}}(\mathrm{Pa}(\mathbf{z}),\varepsilon_{\mathbf{z}})$ .
Real-world datasets. We adopt two real-world datasets with hierarchical generating processes, namely a personality dataset and a digit dataset. The personality dataset was curated through an interactive online personality test [Project, 2019]. Participants were requested to provide a rating for each question on a five-point scale. Each question was designed to be associated with one of the five personality attributes, i.e., agreeableness, openness, conscientiousness, extraversion, and neuroticism. The corresponding answer scores are denoted as  $a_{i}$ ,  $o_{j}$ , etc, as indicated in Figure 5. We use responses (around 19,500 for each question) to six questions for each of the five personality attributes. For the digit dataset, we construct a multi-view digit dataset from MNIST [Deng, 2012]. We first randomly crop each image to obtain two intermediate views and then randomly rotate each of the intermediate views independently to obtain four views. This procedure gives rise to a latent structure similar to that in Figure 4a. We feed images to a pretrained ResNet-44 [He et al., 2016] for dimensionality reduction  $(28 \times 28 \rightarrow 64)$  and run our algorithm on the produced features.
Estimation models. We implement the estimation model  $(\hat{g}_1, \hat{g}_2, \hat{f})$  following Equation 2, where  $\hat{f}$  can be seen as an encoder that transforms  $(\mathbf{v}_1, \mathbf{v}_2)$  to the latent space and  $(\hat{g}_1, \hat{g}_2)$  act as the decoders
dz=dS1=ds2=2	dz=dS1=2, ds2=3	dz=dS1=ds2=4	dz=dS1=4, ds2=6
Joint invertibility (Ours)	0.93±0.09	0.90±0.10	0.89±0.02	0.83±0.12
Individual invertibility	0.67±0.06	NA	0.67±0.09	NA
Table 1: The basis model identifiability. We show the identifiability for  $\mathbf{z}$  under various data dimensionalities  $d_{z}$ ,  $d_{s_1}$ , and  $d_{s_2}$  for  $\mathbf{z}$ ,  $\mathbf{s}_1$ ,  $\mathbf{s}_2$ . We compare our results with prior work that assumes both  $g_{1}$  and  $g_{2}$  are invertible individually. NA indicates that the model is not applicable when the dimensionalities of  $\mathbf{s}_1$  and  $\mathbf{s}_2$  differ. The results are averaged over 30 random seeds.
Figure 5: The causal structure of the personality dataset learned by our method. The Letter in each variable name indicates the personality attribute. Subscripts correspond to question indices. Some observed variables are not shown in the figure, as they are not clustered with other variables, suggesting their distant relation to the system.
generating  $\mathbf{v}_1$  and  $\mathbf{v}_2$  respectively. We parameterize each module with an MLP with Leaky-ReLU activation. Training configurations can be found in Appendix E.
Evaluation metrics. Due to the block-wise nature of our identifiability results, we adopt the coefficient of determination  $R^2$  between the estimated variables  $\hat{\mathbf{z}}$  and the true variables  $\mathbf{z}$ , where  $R^2 = 1$  suggests that the estimated variable  $\hat{\mathbf{z}}$  can perfectly capture the variation of the true variable  $\mathbf{z}$ . We apply kernel regression with Gaussian kernel to estimate the nonlinear mapping. Such a protocol is employed in related work von Kugelgen et al. [2021], Kong et al. [2022]. We repeat each experiment over at least 3 random seeds.
4.2 Basis Model Identification
The results for the basis model are presented in Table 1 and Figure 9. We vary the number of components of each latent partition  $(d_{\mathbf{z}}, d_{\mathbf{s}_1}, d_{\mathbf{s}_2})$ . We can observe that the model with individual invertibility (as assumed in prior work [von Kugelgen et al., 2021, Lyu et al., 2022]) can only capture around half of the information in  $\mathbf{z}$ , due to their assumption that both  $g_1$  and  $g_2$  are invertible, which is violated in this setup. In contrast, our estimation approach can largely recover the information of  $\mathbf{z}$  across a range of latent component dimensions, verifying our Theorem 3.2. Moreover, prior work [von Kugelgen et al., 2021] assumes  $g_1 = g_2$ , and therefore cannot be applicable when the dimensionalities of  $\mathbf{s}_1$  and  $\mathbf{s}_2$  differ (e.g.,  $d_{s_1} = 2$ ,  $d_{s_2} = 3$ ), hence the "NA" in the table. Figure 9 in Appendix E.2 shows scatter-plots of the true and the estimated components with  $d_z = d_{s_1} = d_{s_2} = 2$ . We can observe that components of  $\mathbf{z}$  and those of  $\hat{\mathbf{z}}$  are highly correlated, suggesting that the information of  $\mathbf{z}$  is indeed restored. In contrast,  $\hat{\mathbf{z}}$  contains very little information of  $\mathbf{s}_1$ , consistent with our theory that a desirable disentanglement is attainable. Additional experiments on varying sample sizes can be found in Appendix E.2.
4.3 Hierarchical Model Identification
Synthetic data. We present the evaluation of Algorithm 1 on latent hierarchical models in Figure 4, Table 2, and Table 4-5 in Appendix E.3. In Figure 4, we can observe that all variables can be estimated decently despite a slight recovery loss from a lower to a higher level. Table 2 presents the pair-prediction scores within pairs of estimations while learning the V-structure model in Figure 4b. We can observe that scores within the sibling pairs  $(\mathbf{x}_1,\mathbf{x}_2)$  and  $(\mathbf{x}_4,\mathbf{x}_5)$  are much higher than non-sibling pairs. Notably, the estimate from  $\mathbf{v}_1 = \mathbf{x}_3$  can perform accurate prediction over other estimates, whereas the other estimates fail to capture it faithfully. This is consistent with Theorem 3.4: the basis model with  $\mathbf{v}_1 = \mathbf{x}_3$  will output a variable equivalent to the concatenation of  $\mathbf{z}_2$  and  $\mathbf{z}_3$ , explaining the asymmetric prediction performances. These results empirically corroborate Theorem 3.5.
Personality dataset. From Figure 5, we can observe that our nonlinear model can correctly cluster the variables associated with the same attribute together in the first level, which is consistent with the intentions of these questions. It suggests that conscientiousness, agreeableness, and neuroticism are closely related at the intermediate level, whereas extraversion and neuroticism are remotely related. Some observed variables are not shown in the figure, as they are not clustered with other variables, indicating that they are not closely related to the system. This may provide insights into questionnaire design.
(a) The learned causal structure.
Figure 6: Multi-view digit dataset results. Figure 6a:  $(\mathbf{v}_1, \mathbf{v}_2)$  and  $(\mathbf{v}_3, \mathbf{v}_4)$  form two clusters, as they share the aspect-ratio factor within the cluster while distinct in the rotation-angle factor. Table 6b: each box  $(\mathbf{a}, \mathbf{b})$  shows the  $R^2$  score obtained by applying the estimated variable produced by treating one specific view  $\mathbf{a}$  as  $\mathbf{v}_1$  to predict the estimated variable produced by treating view  $\mathbf{b}$  as  $\mathbf{v}_1$ .
V1	V2	V3	V4
v1	×	0.69 ± 0.001	0.51 ± 0.01	0.51 ± 0.005
v2	0.65 ± 0.02	×	0.48 ± 0.01	0.47 ± 0.03
v3	0.48 ± 0.02	0.42 ± 0.03	×	0.78 ± 0.05
v4	0.53 ± 0.08	0.48 ± 0.02	0.75 ± 0.05	×
Digit dataset. Figure 6a and Table 6b present the causal structure learned from the multi-view digit dataset. We can observe that we can automatically cluster the two views sharing more latent factors. This showcases that our theory and approach can handle
(b) Pairwise predictions.
x1	x2	x3	x4	x5
x1	×	0.85 ± 0.000	0.53 ± 0.01	0.57 ± 0.002	0.55 ± 0.003
x2	0.83 ± 0.006	×	0.52 ± 0.002	0.54 ± 0.001	0.53 ± 0.000
x3	0.90 ± 0.002	0.88 ± 0.002	×	0.86 ± 0.001	0.90 ± 0.006
x4	0.57 ± 0.001	0.54 ± 0.002	0.55 ± 0.003	×	0.86 ± 0.003
x5	0.55 ± 0.006	0.56 ± 0.001	0.55 ± 0.002	0.83 ± 0.003	×