SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For Condition B.1-ii, we can observe that if  $\mathbf{z}$  is updated into  $\mathbf{A}$  at this iteration, all its children would be in  $\mathbf{A}$  at the beginning of this iteration and get removed at Step 16. Therefore, the update would not introduce directed paths to  $\mathbf{A}$ . Condition B.1-iii holds for the same reason: the newly introduced variables in the active set  $\mathbf{A}$  at the end of the iteration separate their children in the active set  $\mathbf{A}$  at the beginning of the iteration and all undiscovered latent variables.
So far, by induction, we have shown that Condition B.1-iii, Condition B.1-ii, and Condition B.1-iii hold at the end of each iteration. Given the analysis above, at each iteration, we update variables whenever all their children and their coparents' children are active, i.e., Case B.2-i and Case B.2-iii, and the causal relations between the parents and the children are encoded in JointP.
We further argue that at every iteration, there exists at least one undiscovered latent variable that has all its children in  $\mathbf{A}$ . We focus on the trimmed graph of  $\mathbf{Z}_{\mathbf{A}}$  and  $\mathbf{A}$ , where  $\mathbf{Z}_{\mathbf{A}}$  refers to all undiscovered variables. As the graph is of finite size and all variables in  $\mathbf{Z}_{\mathbf{A}}$  have directed paths ending at active variables in  $\mathbf{A}$ , the active variable a farthest from the root does not have siblings in  $\mathbf{Z}_{\mathbf{A}}$ ; otherwise, there would be a' (i.e., the child of a's sibling in  $\mathbf{Z}_{\mathbf{A}}$ ) in  $\mathbf{A}$  that was further from the root than a.
In sum, we have shown that there will exist at least one update at each iteration, and each update will lead to the discovery of new latent variables and correct causal structures. Therefore, Algorithm 1 can successfully identify the underlying hierarchical causal graph and each latent variable up to one-to-one mappings.
B.4 Illustration of a Specific Case
Figure 7 shows a specific case to demonstrate how we can avoid assuming specific functional classes. For active set  $\mathbf{A} = \{\mathbf{z}_2, \mathbf{z}_4, \mathbf{z}_5\}$ , if we treat  $\mathbf{z}_2$  as  $\mathbf{v}_1$ ,  $\mathbf{z}_1$  will become  $\mathbf{z}$  in the basis and  $[\mathbf{z}_4, \mathbf{z}_5]$  will become  $\mathbf{v}_2$ . This set of  $(\mathbf{z}, \mathbf{v}_1, \mathbf{v}_2)$  satisfies Assumption 2-i, as all information about  $\mathbf{z}_1$  is contained by  $\mathbf{z}_2$ ,  $\mathbf{z}_4$ , and  $\mathbf{z}_5$ . Assumption 2-iii and Assumption 2-ii are direct consequences of Assumption 3.3-iv and Assumption 3.3-iii. Also, we can always find an independent variable  $\mathbf{s}_1 := \varepsilon_{z_2}$  in the basis model. Therefore, Theorem 3.4 guarantees the estimated  $\hat{\mathbf{z}}$  to be a one-to-one mapping of  $\mathbf{z}_1$ .
C Relaxation of Structural Conditions
C.1 Relaxation of Condition 2.4-ii
We remark that Condition 2.4-ii is introduced to simplify the presentation of our main results - as the first step of nonlinear-latent-hierarchical causal discovery, we wish to present the basic techniques more clearly - handling triangles involves additional components of the algorithm and could potentially compromise the readability.
We give an example in Figure 8 with a triangle structure to illustrate how our theory and algorithm can handle triangles. We refer to variables in a triangle under the following convention: 1) root: the variable which is the cause of the other two variables (e.g.,  $\mathbf{z}_1$  in Figure 8), 2) sink: the variable which is the effect of the other two variables (e.g.,  $\mathbf{z}_3$  in Figure 8), and 3) bridge: the variable which is the effect of the root variable and the cause of the sink variables (e.g.,  $\mathbf{z}_2$  in Figure 8).
Figure 8: An example of triangle structures, i.e.,  $\mathbf{z}_1$ ,  $\mathbf{z}_2$ , and  $\mathbf{z}_3$ .
In the following, we illustrate the procedures to recognize and handle triangle structure by navigating through Figure 8. First, the directed path detection procedure portrayed in Section 3.3 can pinpoint the directed path between the bridge variable and the sink variable (e.g.,  $\mathbf{z}_2$  and  $\mathbf{z}_3$ ). We need to determine whether the two variables indeed belong to a triangle. To this end, we can lump the bridge variable and the sink variable together as  $\mathbf{v}_1$ . We perform the basis model estimation over  $\mathbf{v}_1$  and the active set  $\mathbf{A}$  excluding  $\mathbf{v}_1$ 's children. In Figure 8, we perform the basis model over  $\mathbf{v}_1 = [\mathbf{z}_2, \mathbf{z}_3]$  and  $\mathbf{A} = \{\mathbf{z}_2, \mathbf{z}_3, \mathbf{x}_7, \mathbf{x}_8, \mathbf{x}_9\}$ , which returns an estimate of  $\mathbf{z}_1$ . Further, we can determine that the bridge variable  $\mathbf{z}_2$  and the estimated variable from the lumped  $\mathbf{v}_1$  constitute the parents of the sink variable  $\mathbf{z}_3$  by mutual prediction, which ascertains the two variables are indeed the bridge variable and the sink variable of a triangle, respectively. In this example, we find that the estimated variables of  $\mathbf{z}_2$  and  $\mathbf{z}_1$  together contain identical information to the estimated variable of  $\mathbf{z}_3$ , which reveals that  $\mathbf{z}_2$  and  $\mathbf{z}_3$  are the bridge variable and the sink variable of a triangle. This concludes the triangle detection procedure. With this knowledge, we can handle the triangle structure by lumping the bridge variable  $\mathbf{z}_3$  and the sink variable  $\mathbf{z}_3$  into a super-variable and proceed with this modification.
C.2 Admitting Non-leaf variables into X
Identical to the directed path detection detailed in Section 3.3, Corollary 3.6 can detect non-leaf observed variables and exclude them from the initial active set  $\mathbf{X}$  in Algorithm 1. Thus, the problem can be reduced back to the case without observed non-leaf variables, which can be tackled by Algorithm 1.
D Algorithm 1 Complexity
The dominant complexity term in our proposed algorithm is deep learning training, which is of a complexity  $\mathcal{O}(R*n)$  where  $R$  is the number of levels in the hierarchical model, and  $n$  is the number of observed variables. In contrast, recent works on linear hierarchical models provide algorithms with complexity  $\mathcal{O}(R*(1+n)^{p+1})$  [Huang et al., 2022] and  $\mathcal{O}(R*n!)$  [Xie et al., 2022] where  $p$  is the largest number of variables that share children. Our algorithm achieves the lowest complexity in terms of the algorithmic complexity. As we allow for multi-dimensional variables, the dimensionality of each variable also contributes to the overall wall-clock time through the deep learning model dimension, which is a hyperparameter for the experimenter.
E.1 Additional Experimental Details
We provide additional details of experiments in Section 4. For the basis model evaluation in Section 4.2, then encoder  $\hat{f}$  and decoders  $(\hat{g}_1,\hat{g}_2)$  are 4-layer MLP's with a hidden dimension 30 times as large as the input data and Leaky-ReLU  $(\alpha = 0.2)$  activation functions. For the synthetic hierarchy experiments in Section 4.3, the model layers are 8 and 50 times as large as the input data. For the personality dataset in Section 4.3, the model layers are 4 and 8 times as large as the input data. For the multi-view digit dataset in Section 4.3, the model layers are 4, and 4 times (encoder) and 2 times (decoder) as large as the input data The model configurations are summarized in Table 3.
# encoder layers	# decoder layers	encoder widths	decoder width
Basis models (Section 4.2)	4	4	30×	30×
Synthetic hierarchy (Section 4.3)	8	8	50×	50×
Personality dataset (Section 4.3)	4	4	8×	8×
Digit dataset (Section 4.3)	4	4	4×	2×
Table 3: Estimation model configuration for each experiment.
We apply Adam to train each model for 20,000 steps with a learning rate of  $1e - 3$ . For hierarchical models, we consistently use 2-layer MLPs for generating functions and set the exogenous variable dimensionality as 2. We evaluate the  $R^2$  score with kernel regression over 8192 samples for each estimated variable pair. Figure 6.b, The matched pairs give significantly higher scores than the unmatched ones (e.g., Table 2 and Figure 6b). For instance, in the first row of Table 2, the estimated latent variable with This gap allows us to set a threshold for each experiment to distinguish the matched estimated variables (e.g., a threshold of 0.6 suffices for the experiment in Table 2).
E.2 Additional Results on the Basis Model
Figure 9 visualizes the dependence between the estimated and true components. We can observe that the  $\mathbf{z}$  is highly correlated with  $\hat{\mathbf{z}}$  whereas  $\mathbf{s}_1$  has little correlation with  $\hat{\mathbf{z}}$ , which verifies Theorem 3.2 that we can successfully identify  $\mathbf{z}$ .
True  $[Z]_1$
True  $[Z]_2$
True[S1]
True  $[S_1]_2$
True  $[Z]_{1}$
Figure 9: The scatter plot between the estimated  $\hat{\mathbf{z}}$  and the true  $(\mathbf{z},\mathbf{s}_1)$ . The scatter points are obtained in an estimation run with  $d_{z} = d_{s_{1}} = d_{s_{2}} = 2$ . We can observe that the estimated components of  $\hat{\mathbf{z}}$  are highly correlated with those of  $\mathbf{z}$  and have little correlation with those  $\mathbf{s}_1$ , empirically verifying Theorem 3.2.
True  $[S_1]$
Although our theory only concerns asymptotic properties, our approach can perform stably at relatively low-sample regimes. As shown in Figure 10, our algorithm can achieve decent identification scores when the sample number exceeds 5000. The performance peaks with little variance when the sample size is over 10000.
E.3 Additional Results on Synthetic Hierarchical Models.
Table 4 and Table 5 contain the mutual prediction scores for the hierarchical structures in Figure 4a and Figure 4c. We can observe that the estimated variables that yield high mutual prediction scores are indeed under the same causal cluster, which gives signals for structure learning.
Figure 10:  $R^2$  scores for basis model identification. The true basis model (Figure 2 in the manuscript) has latent dimensionality  $d_z = d_{s_1} = d_{s_2} = 2$ . The basis model training and data generation follow the training configuration in the main text. We vary the number of training samples and measure the  $R^2$  score to assess the identification result. Each result is averaged over 3 random seeds.
x1	x2	x3	x4
x1	×	0.83 ± 0.03	0.57 ± 0.04	0.50 ± 0.08
x2	0.86 ± 0.01	×	0.58 ± 0.03	0.52 ± 0.04
x3	0.6 ± 0.02	0.59 ± 0.05	×	0.77 ± 0.00
x4	0.59 ± 0.02	0.59 ± 0.05	0.81 ± 0.04	×
E.4 Additional Results on the Personality Dataset
We randomly and evenly partition each set of six questions into two variables. Following this procedure, we have observed ten variables  $\mathbf{c}_1, \mathbf{c}_2, \mathbf{a}_1, \mathbf{a}_2, \mathbf{n}_1, \mathbf{n}_2, \mathbf{e}_1, \mathbf{e}_2, \mathbf{o}_1,$  and  $\mathbf{o}_2$ , each of 3-dimension. The Letter in each variable name indicates the attribute to which the variable belongs, to facilitate readability. Figure 11 contains the results. Analogous to Figure 5, the learned structure faithfully reflects the questionnaire structure – questions designed for the same personality trait are clustered together.
Table 4: Pairwise predictions among estimates in Figure 4a. Each box  $(\mathbf{a},\mathbf{b})$  shows the  $R^2$  score obtained by applying the estimate produced by treating a as  $\mathbf{v}_1$  to predict the estimate produced by treating b as  $\mathbf{v}_1$ . We can observe that the prediction scores within sibling pairs  $(\mathbf{x}_1,\mathbf{x}_2)$  and  $(\mathbf{x}_3,\mathbf{x}_4)$  are noticeably higher than other pairs, showing a decent structure estimation.
z4	x3	x4	x5
z4	×	0.81 ± 0.011	0.48 ± 0.006	0.52 ± 0.000
x3	0.87 ± 0.002	×	0.47 ± 0.003	0.54 ± 0.002
x4	0.57 ± 0.000	0.52 ± 0.006	×	0.77 ± 0.005
x5	0.58 ± 0.001	0.57 ± 0.015	0.81 ± 0.000	×
Table 5: Pairwise predictions among estimated variables in Figure 4c. The pattern is consistent with Table 4 and Table 2.
Figure 11: The casual graph learned by our method, where each observed variable (c, a, n, e, o) is of three dimensions.