Identification of Nonlinear Latent Hierarchical Models
Authors: Lingjing Kong
Biwei Huang
Feng Xie
Eric Xing
Yuejie Chi
Kun Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Causal discovery
causal representation learning
latent variable models
causal structure learning
causal identifiability.
TL;DR: We provide identifiability theory for both causal structures and latent variables in latent hierarchical models beyond linearity.

Abstract: Identifying latent variables and causal structures from observational data is essential to many real-world applications involving biological data, medical data, and unstructured data such as images and languages. However, this task can be highly challenging, especially when observed variables are generated by causally related latent variables and the relationships are nonlinear. 

In this work, we investigate the identification problem for nonlinear latent hierarchical causal models in which observed variables are generated by a set of causally related latent variables, and some latent variables may not have observed children. We show that the identifiability of causal structures and latent variables (up to invertible transformations) can be achieved under mild assumptions: on causal structures, we allow for multiple paths between any pair of variables in the graph, which relaxes latent tree assumptions in prior work; on structural functions, we permit general nonlinearity and multi-dimensional continuous variables, alleviating existing work's parametric assumptions. Specifically, we first develop an identification criterion in the form of novel identifiability guarantees for an elementary latent variable model. Leveraging this criterion, we show that both causal structures and latent variables of the hierarchical model can be identified asymptotically by explicitly constructing an estimation procedure. To the best of our knowledge, our work is the first to establish identifiability guarantees for both causal structures and latent variables in nonlinear latent hierarchical models.

Identification of Nonlinear Latent Hierarchical Models
Lingjing Kong', Biwei Huang", Feng Xie Eric Xing Yuejie Chi', and Kun Zhang
Carnegie Mellon University
Uniieersity of California San Diego
Beijing Technology and Business University
*Mohamed bin Zayed University of Artificial Intelligence
Abstract
Identifying latent variables and causal structures from observational data essen- tial many real-world applications involving biological data, medical data, and unstructured data such as images and languages. However, this task can be highly challenging, especially when observed variables are generated by causally related latent variables and the relationships are nonlinear. In this work, we investigate the identification problem for nonlinear latent hierarchical causal models in which observed variables are generated by a set of causally related latent variables. and some latent variables may not have observed children. We show that the identifi- ability of causal structures and latent variables (up to invertible transformations) can be achieved under mild assumptions: on causal structures, we allow for mul- tiple paths between any pair of variables in the graph, which relaxes latent tree assumptions in prior work; on structural functions, we permit general nonlinearity and multi-dimensional continuous variables, alleviating existing work's 's parametric assumptions. Specifically, we first develop an identification criterion the form of novel identifiability guarantees for elementary latent variable model. Lever- aging this criterion, we show that both causal structures and latent variables the hierarchical model can be identified asymptotically by explicitly constructing an estimation procedure. To the best of our knowledge, our work is the first to establish identifiability guarantees for both causal structures and latent variables in nonlinear latent hierarchical models.
Introduction
Classical causal structure learning algorithms often assume no latent confounders. However, it usually impossible to enumerate and measure task-related variables in real-world scenarios. Neglecting latent confounders may lead to spurious correlations among observed variables. Hence, much effort has been devoted to handling the confounding problem. For instance, Fast Casual Inference and its variants [Spirtes et al., 2000 Pearl 2000 Colombo et al. 2012 Akbari et al. 2021 leverage conditional independence constraints to locate possible latent confounders and estimate causal structures among observed variables, assuming causal relationships among latent variables. This line of approaches ends up with an equivalence class, which usually consists of many directed acyclic graphs (DAGs).
Later, several approaches have been proposed tackle direct causal relations among latent variables with observational data. These approaches are built upon principles such as rank constraints (Silva
al. [2006a] Kummerfeld and Ramsey| [2016] Huang al.) 2002)). matrix decomposition CChan
drasekaran al. [201] [2012 Anandkumar et al. [013 high-order moments [Shimizu et al. 2009
Cai et al. 2019 Xie et al. 2020 Adams et al. 2021 Chen et al. 2022 copula models [Cui et al.
2018 and mixture oracles Kivva al. 2021
Pear 1988 Zhang (0044. Choi et al. [2011] Drton
et al. 2017]. Zhou et al. 2020 Huang et al. 2020 extend such approaches to handle tree structures
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

%
42
X8
23
Z2
Z3
74
X4
X12
Z/7
X1
X2
X7
X3
X15
X4
X5
X6
(b)
Examples of latent hierarchical graphs satisfying Conditions 2.4 x, denotes observed variables and 3, denotes latent variables. The node size represents the dimensionality of each variable. We note that our graph condition permits multiple paths between two latent variables (c.g., 21 and 24 in Figure Ib], thus more general than tree structures.
Figure I:
where only one path is permitted between any pair of variables. Recently, Huang et al. 2022 propose to use rank-deficiency constraints to identify more general latent hierarchical structures. common assumption behind these approaches is that either the causal relationships should be linear or the variables should be discrete. However, the linearity and rather restrictive in real-world scenarios. Moreover, these approaches only focus on structure learning without identifying the latent variables and corresponding causal models. We defer a detailed literature review Appendix
assumptions
In this work, we identify the hierarchical graph structure and latent variables simultaneously for general nonlinear latent hierarchical causal models. Specifically, we first develop novel identifiability guarantees on a specific latent variable model which we refer to as the basis model (Figure (2). We then draw connections between the basis-model identifiability and the identification of nonlinear latent hierarchical models. Leveraging this connection, we develop an algorithm to localize and estimate latent variables and simultaneously learn causal structures underlying the entire system. We show the correctness of the proposed algorithm and thus obtain identifiability of both latent hierarchical causal structures and latent variables for nonlinear latent hierarchical models. In sum, our main contributions are as follows.
Analogous to independence tests in PC algorithm [Spirtes et al. 2000 and rank rank-deficiency tests in Huang al. [022]. we develop a novel identifiability theory (Theoeem 3.2 as fundamental criterion for locating and identifying latent variables in general nonlinear causal models.
We show structure identification guarantees for latent hierarchical models admitting continuous (c.f., mie-dimensional and often discrete [Pearl] 1988 Choi al., 2011 Huang
ct al.
2022 Xie et al.) (2022] variables, general nonlinear (c.f., linear [Pearl] 1988 Huang
ct al.
2022
Xie al. [2022] structural functions, and general graph structures (c.f., generalized
trees
Pearl 1988 Choi et al.) 2011 Huang et al.) (022)).
Under the same conditions, we provide identification guarantees for latent variables up to invertible transformations, which, to the best of our knowledge, is the first attempt in cases of nonlinear
hierarchical models.
We accompany our theory with an estimation method that can asymptotically identify the causal structure and latent variables for nonlinear latent hicrarchical models and validate on multiple synthetic and real-world datasets.
2 Nonlinear Latent Hierarchical Causal Models
In this section, we introduce notations and formally define the latent hierarchical causal model. We focus on causal models with directed acyclic graph (DAG) structures and denote the graph with G, which consists of the latent variable set Z ;= {Zi....,Zml the observed variable set X = [x]...., x,), and the edge set E. Each variable is a random vector comprising potentially multiple dimensions, i.e., X; E Rdd, and z, Rd=,, where du; and de, stand for the dimensionalities of x, E X and z, E Z, respectively. Both latent variables and observed variables are generated recursively by their latent parents:
9r. eP.())),
2j 9z, (Pa(zz), E2y);
'We refer to leaf variables in the graph as observed variables for ease of exposition. The theory in this work is also applicable when some non-leat variables happen be observed, which discuss in Append C.2 C.2

---

where Pa(zy) represents all the parent variables of 2, and εx, represents the exogenous variable generating zj. Identical definitions apply to those notations involving x, All exogenous variables C.;, 22 are independent of each other and could also comprise multiple dimensions. We now define a general latent hierarchical causal model with a causal graph G == (Y, E) in Definition 2.1
Definition 2.1 (Latent Hierarchical Causal Models), The variable set Y consists of observed variable set X and latent variable set Z, and each variable is generated by Equation
In light of the given definitions, we formally state the objectives of this work:
1. Structure identification: given observed variables X, we would like to recover the edge set E.
2. Larenl-variable identification: given observed variables X, we would like to obtain a set of variables 2 == (21,.... 2m where for i E [m]] 2; and 2; are identical up to an invertible mapping, i.e., = hz().), where h2 is an invertible function.
Definition 2.1 gives a general class of latent hierarchical causal models. On the functional constraint, the general nonlinear function form (Equation[]) TD renders it highly challenging to identify either the graph structure or the latent variables. Prior work Pearl [1988]. Choi et al. [201T). Xie et al. [2022), Huang et al. [0022] relies either linear model conditions discrete variable assumptions. In this work, we remove these constraints to address the general nonlinear case with continuous variables. On the structure constraint, identifying arbitrary causal structures is generally impossible with only observational data. For instance, tree-like structures are assumed in prior work [Pearl] 1988] Choi present relaxed
jtt al. 2011 Huang et al. [2022] for structural identifiability. In the following, structural conditions under which we show structural identifiability.
Definition 2.2 (Pure Children). V; a pure child of another variable Vj. if v the only parent of V; in the graph, i.e., Pa(v.) = {v
Definition 2.3 (Siblings). Variables v; and v, are siblings if Pa(v:) n Pa(v,) # 0.
Condition 2.4 (Structural Conditions).
i Each latent variable has least 2 pure children.
ii There are directed paths between any two siblings.
Intuitively, Condition allowss each latent variable to leave a footprint sufficient for identification. This excludes some fundamentally unidentifiable structures. For instance. if latent variables z and Z2 share the same children i.e. 2 + X and Z2 + X, the two latent variables cannot be identified without further assumptions, while pure children would help in this case. The pure child assumption naturally holds in many applications with many directly measured variables, including psychometrics. image analysis, and natural languages. We require fewer pure children than prior work (Silva et al.) 2006a Kummerfeld and Ramsey] 2016) and place no constraints on the number of neighboring variables as prior work [Huang et al. 2022 Xie et al. 2022].
Condition 2.4 fi avoids potential triangle structures in the latent hierarchical model. Triangles present
significant obstacles for identification in triangle structure formed by 2 - Zg 23 and 2 - Zg, it is difficult to discern how 2 influences Z3 without functional constraints, as there are two directed paths between them. We defer the discussion of how to use our techniques handle more general graphs that include triangles to Appendix C.T Condition 2, 2. is widely adopted in prior work,
especially those on tree structures [Pearl] 1988 Choi al. 201 Drton et al., 2017] (which cannot
handle multiple undirected paths between variables as we do) and more recent work (Huang et al.
2022]. To the best of our knowledge, only Xie et al. [2022] manage handle triangles the latent hierarchical structure, which, however, heavily relies on strong assumptions on both the function class (linear functions), distribution (non- noon-Gaussian), and structures (the existence of neighboring variables).
3 Identifiability of Nonlinear Latent Hierarchical Causal Models
This section presents the identifiability and identification procedure of causal structures and latent variables in nonlinear latent hierarchical causal models, from only observed variables. First, in Sec- tion 3.1
we introduce a latent variable model (i.e., the basis model in Figure 2) whose identifiability
We are concerned about representation vectors corresponding to each individual variable. Thus, identifiability up to invertible transforms suffices. Generally, stronger identifiability (e.g.. component-wil wise) necessitates additional assumptions, e.g., multiple distributions Khemakhem et al. 2020). sparsity assumptions [heng et al.
2022
3

---

$2
1==1 and E T, such that Figure 2: The basis model. Vi and from nonlinear functions gi and g2 which can be distinct and non-invertible individually, We also admit depen- dence between z and 82, as indicated by the dashed edge.
generated
V2
underpins the identifiability of the hicrarchical model. In Section 3.2 we establish the connection between the basis model and the hicrarchical model. Leveraging this, in Section 3.3 we show by construction that both causal structures and latent variables are identifiable in the hierarchical model.
3.1 Basis Model Identifiability
We present the data-generating process of the basis model in Equation
2 and illustrate it in Figure 2
V1 = 91 (1.
V2 92 (2,
(2)
where V1 Vi C Rd.,, V2 1/2 variables consisting of potentially multiple components. The data-generating process admits potential dependence between and $2 and dependence across dimensions in each variable. We denote the entire latent variable == Iz. $1, s2] C and the mapping from с to (V1, V2) as g - V1 x V/.
2 2 Rdd Si Si C Rdd; and 82 S2 C Rd are
Below, we introduce notations and basic definitions that we use throughout this work. Indexing: For a matrix M. we denote its i-th row as M.,, k.., its j-th column as M.,. and its (i,j) entry as Ma Similarly, for an index set I (I,..., m} {L,.... n), we denote It, := {- (i. E I} and Z.j == {i: (i E Z}.
Subspaces: We denote a subspace R" defined by an index set as RR5. where R3 = {z E R"
vi $ S, = O}.
Support of matrix-valued functions: We define the support of a matrix-valued function M(x) X' - IR mxx Supp(M) == ((i.j) Fx X, s.t. M(x),,) * 0), i.e., the set of indices whose corresponding entries are nonzero for some x E X.
In Theorem 3.2 below, show that the latent variable is identifiable up to an invertible transforma- tion by estimating a generative model (f2.2. Psi Ps,, 9) according to Equation 2 We denote Jacobian matrices of and as Jg and Jg. and their supports G and 9. respectively. Further, we denote as T set of matrices with the same support as that of the matrix-valued function J. (c)
Assumption 3.1 (Basis model conditions).
i (Differentiability & Invertibility): The mapping g(c) = (v1, v2) is a differentiable invertible function with differentiable inverse.
(Subspace span]: For all E (1,....,d, + drz). there exists (c(0) ) Guil span((J, (J]]((((e))) (c(() (c:)).)))))) 11.1) = Rgg, and [J (T): S.. ii [Edge Connectivity]: For all j. E {l..... dz}. there exist in. (I....,dy) and ivs (duy....,dd, ddy + d such that (1o), EG and (1oz, E G.
Theorem 3.2. Under Assumption 3.1 if generative model (P Po 9) follows the data-generating process in Equation D and matches the true joint distribution:
(V1, v2) [a)= = Privz (V1, v2), V(v1, E V X,,
(3)
then the estimated variable 2 and the true variable are equivalent up invertible transformation.
The proof can be found in Appendix B.I
Interpretation. Intuitively, we can infer the latent variable z shared by the two observed variables vi and v2). in a sense that the estimated variable 2 contains all the information of z without mixing any
information from 81 and 8y. The identifiability up to an invertible mapping extensively employed
in identifiability theory [Casey and Westner 2000 Hyvärinen and Hoyer 2000 Le al. 2011 Theis
2006 von Kügelgen et al. 202 Kong et al. 2022].
Discussion on assumptions. Assumption 3.1 assumes the mapping y preserves latent variables' information, guaranteeing the possibility of identification. Such assumptions are universally adopted the existing literature on nonlinear causal model identification [Hyvarinen et al. 22099 Khemakhem
al. 2020 Kong et al. 2022 Lyu et al. 2022 von Kügelgen et al. 2021]]

---

guarantees that the influence of changes adequately across its domain, as discussed in prior work [Lachapelle al.) 2022 Zheng et al. 2022]. This eliminates situations where the Jacobian matrix is partially constant, causing it to insufficiently capture the connection between the observed and latent variables. This condition is specific for nonlinear functions, and counterpart can be derived for linear cases (Zheng et al.. 2022].
Assumption 3.1
Assumption 3.1 j is formal characterization of the common cause variable z in the basis model. This assumption excludes the scenario where some dimensions of z only influence one of V and V2, in which case these dimensions of z are not truly the common cause of V1 and V2 and should be modeled as separate variable rather than part of z. This is equivalent to causal minimality, which is necessary for casual structure identification for general function classes with observational data Peterrs et al. 2017].
Comparison with prior work. In contrast to similar data-generating processes in prior work (Lyu ett al., 2022 von Kügelgen et al.) 2021]. Theorem [3. allows the most general conditions. First, Theorem 3.2 does not necessitate the invertibility for g1 91(+) and g2(+) individually as (Lyu et al. 2022 von Kügelgen et al. 2021]. The invertibility of gi and g2 amounts to asking that z z's information duplicate over Vi and V2. In contrast, Assumption j only requires that z's information is stored in (V1, V2) jointly. Moreover, generating functions (1(-) and 92(-) are assumed identical in [von Kügelgen al. 202T J. as opposed being potentially distinct in Theorem 3.2 Additionally, Theorem 3.2 allows for dependence between z and S2 which is absent in (Lyu et al. 2022].
These relaxations expand the applicability of the basis model. For example, the distinction between Φ1 and 92 is indispensable in our application to the hierarchical model, as we cannot assume each latent variable generates its many children and descendants through the same function. Our proof technique is distinct from prior related work, which can be of independent interest to the community,
3.2 Local Identifiability of Latent Hierarchical Models
In this section, we build the connection between the basis and hierarchical models. Specifically, we show in Theorem 3.4 that a careful grouping of variables in the hierarchical model enables us to apply Theorem 3.2 to dentify latent variables and their causal relations a local scope. To this end, we modify Assumption 3.1 in basis models and obtain Assumption 3.3 for hierarchical models.
Assumption 3.3 (Hierarchical model conditions).
(Differentiability)) Structure equations in Equation | are differentiable.
ii cetemmaatioo-oooseevttttiiii Any Z and exogenous variable tiable functions of all observed variables, i.e., z = fx) and E = fX)..
be expressed as differen-
in [Subspace span]: For each set A that d-separates all its ancestors And(A) and observed variables X, i.e., X L Anc(A)|A, for any ZA E Pa(A), there exists an invertible mapping from a set of ancestors of A containing ZA to the separation set A. such that this mapping satisfies the subspace span condition (i.e., Assumption 3.1
iv [Edge connectivity]: The fanction between euch latent variable z and each of its children z' hus Jacobian J, such that for all j E (1....,dz}, there exists i E (1,...,dy, dz) ), such that (i,j) is in the support of J.
Theorem 3.4. In a latent hierarchical causal model that satisfies Condition 2.4 and Assumption 3.3 we consider X; E X as Vı and X\,V as V2 in the basis model (Figure 2). 3 With an estimation model (Pr.s..Ps) 9) that follows the data- (ata-agnn process in Equation 2 the estimated à is a one-to-one mapping of the parent(s) of V1, i.e., 2 = ((a((v))) where h(-) is an invertible function.
The proof can be found in Appendix B.2
Interpretation. Theorem 3.4 shows that invoking Theorem 3.2 with the assignment V x and V2 = =X (x(x)) (x) can identify the parents of x in the hierarchical model. Intuitively, we can identify latent variables one "level" above the current level X. Moreover, as the estimated variables are equivalent to true variables up to one-to-one mappings, we take a step further in Section 3.3 to show that this procedure can be applied to the newly estimated variables iteratively to traverse the hierarchical model "level" by "level", In the degenerate case where V1 does not have any parents, i.c., the root of the hierarchical model, the identified variable in the basis model corresponds to the v1 itself we can regard $ as a constant and g1 (-) as an identity function in Theorem 3.2
"To avoid cluttering, we slightly abuse the bold lowercase font to represent either an individual vector or a vector set (which can be viewed concatenation).
5

---

21
711
X1
X7
X8
X9
X1
X2
Xx X9
X1
X2 X3
X7 X8
Xa
X1
X5
X8
X1
X5
X6
(a)
(b)
(c)
Figure 3: Evolution of active set A in Algorithm 1 We mark the active set A with shaded gray circles before each iterations of Algorithm with Figure Figure 3b and So
to iteration 1, 2. and 3.
Before iteration 1, set to by default. At iteration I. 22, Z3. and 24 can be estimated by basis model; however, only 24 can be updated into A. Otherwise, directed paths would be introduced by 22 and 23.
Discussion assumptions. Assumption 3.3 corresponds to Assumption 3.J for the basis model. Intuitively, states that the information in the system is preserved without unnecessary duplication. As far as we know, the existing literature models [Hyvarinen et al. 2019 Khemakhem et al. 2020 Kong et al. [0022] Lyu et al.) 2022 Yao et al. 2022] universally assumes the invertibility of such mapping. Without this assumption, cannot guarantee the identification of latent variables from observed variables in the nonlinear case.
causal variable identification for nonlinear
Assumption 3.3 is an extension of Assumption 3.J to the hierarchical case, with A and Za
playing the role of (V1, V2) and z respectively.
Assumption 3.3 iv excludes degenerate cases where an edge connects components that do not influence each other. This equivalent to the causal minimality and generally considered necessary for causal identification with observational data [Peters et al., 2017]. For instance, we cannot distinguish Y := f(X) Ny and Y == c c+ Ny if f is allowed to differ from constant c only outside X's support. Further, when V1 x is pure child of latent variable 2, Condition ( ensures that V2 = Az } contains at least one pure child or descendant of z to fulfill Assumption 3.1
Implications on measurement causal models. Despite being an intermediate step towards the global structure identification, Theorem 3.4 can be applied g with arbitrary latent structures and identify all the latent variables. Then, the latent structures in the measurement model can be identified by performing existing causal discovery algorithms, such as the PC algorithm (Spirtes et al. [200]], on the identified latent variables. We leave the detailed presentation of this application as future work.
class of nonlinear measurement models
3.3 Global Identifiability of Latent Hierarchical Causal Models
Equipped with the local identifiability (i.e., Theorem 3.4,, the following theorem shows that all causal structures and latent variables are identifiable in the hierarchical model.
Theorem 3.5. Under assumptions in Theorem 3.4 all latent variables Z in the hierarchical model can be identified up to one-to-one mappings, and the causal structure G can also be identified.
Comparison to prior work. Theorem 3.5 handles general nonlinear cases, whereas prior work (Pearl 1988 Choi et al. 2011 Huang et al.) 2022, Xie et al. 2022] is limited to linear function or discrete latent variable conditions. Structure- wise, our identifibility results apply to latent structures with V-structures and certain triangle structures (see discussion in Appendix C.1] beyond generalized
trees as studied in prior work [Choi et al., 2011 Pearl| 1988 /Huang et al.) 2022 require fewer pure
children and no neighboring variables in comparison with [Xie et al. 2022 Huang et al. 2022 and
can determine directions for each edge (c.f., Markov-equivalent classes in Huang et al. 2022
Proof sketch with search procedures. The proof can be found in Appendix
B.3 In particular,
we develop a concrete algorithm (i.e., Algorithm [ and show that it can successfully identify the causal structure and latent variables asymptotically. We give proof sketch of Theorem 3.4 below by navigating through Algorithm I and illustrate with an example Figure
Stage 1: identifying parents with the basis model (line line 5). As shown in Theorem 3.4 the basis model can identify the latent parents of leaf-level variables in the hierarchy. In Figure can identify 2n when applying the basis model with the assignment v := {x,} and V2 == Vi- Naturally, the basic idea of Algorithm [ is iteratively apply the basis model to the most recently
"We refer to Silva et al. (2006)) for general measurement model definition. Here, we consider a popular type of measurement models that has been widely used in the literature (see Xie et al. 2020 in which observed variables do not cause other variables.

---

identified latent variables to further identify their parents, which is the purpose of Stage (line 4 line In Algorithm|| [ we define as active set A the set of variables to which we apply the basis model. For example, A equals to X in the first round (Figure 3ab and becomes (x1; X2, X3, 24, X7, X8, X9) in the second round (Figure(3b)
Stage 2: merging duplicate variables (line[). As multiple variables in A can share a parent, dictionary P(.) may contain multiple variables that are one-to-one mappings to each other, which would obscure the true causal structure and increase the algorithm'ss 's complexity. Stage 2 (Line 6 detects such duplicates and represents them with one variable. In Figure Ba setting V1 to any of X1, X2, and x3 would yield an equivalent variable of Zz. We merge the three equivalents by randomly selecting one.
Stage 3: detecting and merging super-variables (line 7 line 9 Due to the potential existence of V-structures, variables in A may have multiple parents and produce super-variables in P. For instance, the second iteration of Algorithm [ (i.e., Figure 3bb, the basis model will be run on Vi = 24 and V2 = {x1, X2, X3, X7, Xs, Xg) and output a variable equivalent to the concatenation (z2, Z3). Leaving this super-variable untouched will be problematic, as we would generate false causal structure 2 + 24 where 2 is the estimated super-variable (Z2, Za). rather than recognizing Z, the child of two already identified variables Z2 and za. Line too line line-ddeter detect such super-variables by lesting whether each variable 2 in Pi is equivalent to a union of other variables in P. If such a union Z ;= {ž1..... }m exists, we will replace 2 with Z in all its occurrences. In Figure 3b we would split the variable equivalent to [z2, z3] into variables of Za and Z3 individually. If 2 tested to be a super-variable, i.c., it can perfectly predict another variable 2' in P and 2' cannot predict 2 perfectly,
and the equivalent union cannot be found, we will track à in line
9 to prevent from being updated
into A at Iine
Stage 4: detecting and avoiding directed paths in A (line 12 line 14). Ideally, we would like repeat line 4 to line until reaching the root of the hierarchical model. Unfortunately, such an approach can be problematic, this would cause variables in active set A have directed edges among them, whereas Theorem 3.4aapplies to leaf variable set X which contains no directed edges. In Figure 3a would contain Zz, Z3, 2,}, as each of these latent variables has pure observed children in X. However, due to direct paths within P. i.c., Z2 + Zι and Z3 + Z1. we cannot directly substitute X with {z2, 23, 3 in A. To resolve this dilemma, we proactively detect directed paths emerging with newly estimated variables and defer the local update of such estimated variables to eliminate direct paths. For directed path detection, we introduce corollary of Theorem 3.4 as Corollary [3.6]
Corollary 3.6. Under assumptions in Theorem 3.4 for any z E Pa(A) where A is the active set in Algorithml we consider V1 == z and V2 == A (Ch(z)) n A) where Ch(z) is a strict subset of children (i.e., when the active set A contains at least one child of z Ss,, estimating the basis model yields equivalent to z up to a one-to-one map-inng mapping.
Corollary 3.6 can be obtained by setting V1 as an ancestor of V2 and s as a degenerate variable (i.e., a deterministic quantity) in Theorem 3.4 Leveraging Corollary 3.6 we can detect whether each newly identified latent variable in P would introduce directed paths into A if they were substituted in. If directed paths exist, the variable 2 would contain the same information as V, := z, which prediction tests can evaluate. In this event, we will suppress the update of the z at this iteration. That is, we still keep its children in A. This directed-patt -ath detection is conducted in lines 12 14] after properly grouping variables that share children in lines [ [T] As shown Figure 22 and 23 are not placed in A, even if they are found in the first iteration. This update only happens when 2, has been placed in A at the second iteration.
Generally, a latent variable enters A only if all its children reside in A. We can show that A contains all the information of unidentified latent variables A d-separates the latent variables that have not been placed in A and those were in A once. Equipped with such a procedure, we can identify the hierarchical model iteratively until completion. We discuss Algorithm [ complexity in Appendix
Experimental Results
In this section, we present experiments to corroborate our theoretical results in Section 3 We start with the problem of recovering the basis model in Section 4.2. which is the foundation of the overall identifiability. In Section 4.3 we present experiments for hierarchical models on synthetic dataset and two real-world datasets.
7

---

Algorithm I Identification of Latent Hierarchical Models. A: the active set, X: the observed variable set, P/JointP: dictionaries that store the relations between variables in A and estimated variables.
1: Initialize the active set: t X:
2: while A o 0 do
3: initialize an empty set R and empty dictionaries JointP::
4: for each active variable E A do
5: estimate the basis model with V1 =a and V2 AMa to obtain 2, and < PU {(a,2))
6: merge equivalent variables in P.
7:
for each variable in P do
8:
if 32' E P that 2 can perfectly predict but é' cannot predict 2 then
9:
32 CP {&} that perfectly predicts é, then replace with 2; else: R- U--UU(2 (2) into {Z:) such that Z, is either singleton or contains spouse variables; store each cluster Z: and its children set H; a pair (Z,, H.) into JointP;
10: cluster & variables
11:
12:
for cach pair (Z,, H.) E JointP do
13:
estimate the basis model with Vı ZN and === A, to obtain a variable 2neus
14:
3z' L. such that Zien can perfectly predict z', then R RUZ, for each pair (Z, H;) JointP do if no variables in Z. has been tracked by R. i.c., Z, n = 0, then substitute H. with Z, in A.
15:
16:
17:
remove variable a from A if a A \. {a].
18:
Return: all the past active sets A and parent sets P.
0.76 +
2² 0.78 + 0.03
R² m 0.7G + 0.02
n²a 0.86 + 0.09
R 0.88 0.07
R2
0.09
0.90 0.06
0.11p
X4
X1
X2
X3
X1
X2
(a) balanced tree.
(b) With V-structure.
(c) An unbalanced tree.
Figure 4: Evaluated hierarchical models. We denote the estimation 1t2 scores around corresponding latent variables, We can observe that all latent variables can be identified decently, justifying our theoretic results.
4.1 Experimental Setup
Synthetic data generation. For the basis model (Figure we sample (O(, ~ ~N. I), and $2 (AAz + b, I), where the dependence between and $2 is implemented by randomly constructed matrix A and bias b. We choose the true mixing function g as a multilayer perceptron (MLP) with Leaky-Rel. wlement-wise max(2 z, 0} to before inputting [z, $1] to 91 and element- min(z, 0} to z before inputting [z, S2] 92. that is generated by the positive elements of and va is generated by the negative elements of z. This way, @1 and 92 are jointly invertible but not individually invertible. For latent hicrarchical models (Figure 4), we sample each exogenous variable as ~ (O I) and each endogenous variable z is endowed with a distinct generating function Is parameterized by an MIP, i.e., == 9x (2),
activations and well-conditioned weights to facilitate invertibility. We apply
Real-world datasets. We adopt two real-world datasets with hierarchical generating processes, namely a personality dataset and digit dataset. The personality dataset was curated through an interactive online personality test [Project] 2019]. Participants were requested to provide a rating for each question on five-point scale. Each question was designed be associated with of the five personality attributes, i.c., agreeableness, openness, conscientiousness, extraversion, and neuroticism. The corresponding answer scores are denoted as ag Oj. etc, as indicated in Figure 5 We use responses (around 19,500 for each question) to six questions for each of the five personality attributes. For the digit dataset, we construct a multi-view digit dataset from MNIST [Deng 2012 We first randomly crop each image to obtain two intermediate views and then randomly rotate each of the intermediate views independently to obtain four views. This procedure gives rise to latent structure similar to that in Figure 4a We feed images to pretrained ResNet-44 [He et al. 2016) for dimensionality reduction (28 x 28 64) and run our algorithm on the produced features.
Estimation models. We implement the estimation model (ğ), 92, f) following Equation where can be seen as encoder that transforms (V1, V2) to the latent space and (91, 92) act the decoders
8

---

d2 d,, deg =2
d,
day = 2, d,y
=3
3
dg 6= d. deg =4
d, de = 4, dz 6
Joint invertibility (Ours)
0.93 + 0.00
0.90 + 0.10
0.89 = 0.02
0.83 = 0.12
Individual invertibility
0.67 + 0.06
NA
0.67 = 0.09
NA
Table 1: The basis model identifiability. We show the identifiability for z under various data dimensionalitics d2, day. and da, for z, S1, 52. We compare our results with prior work that assumes both 91 and 92 are invertible individually. NA indicates that the model is not applicable when the dimensionalities of i and $2 differ. The results are averaged over 30 random seeds.
Figure 5: The causal structure the personality dataset learned by our method. The Letter in each variable name indicates the personality attribute. Sub- scripts correspond to question indices. Some observed variables are shown in the figure, as they are not clustered with other variables, suggesting their distant relation to the system.
q da
4 of
d2
by
02
ef
02
do
b eg
di
of 0g bs
generating V1 and respectively. We parameterize each module with an MLP with Leaky-ReLU activation. Training configurations can found Appendix F
Evaluation metrics. Due the block-wise nature of our identifiability results, we adopt the coefficient of determination It2 between the estimated variables & and the true variables z, where R suggests that the estimated variable 2 can perfectly capture the variation of the true variable 2. We apply kernel regression with Gaussian kernel to estimate the nonlinear mapping. Such a protocol is employed in related work yoon Kügelgen et al. [2021]. Kong et al. [0022]. We repeat each experiment over at least 3 random seeds.
4.2 Basis Model Identification
The results for the basis model are presented Table 1 and Figure We vary the number of components of each latent partition (dz, dg, dgg). ). We can observe that the model with individual invertibility (as assumed in prior work [von Kügelgen et al. [021] Lyu et al. 2022) can only capture around half of the information in z, due to their assumption that both 91 and Io are invertible, which is violated in this setup. In contrast, our estimation approach can largely recover the information of z across range of latent component dimensions, verifying work [yon Kügelgenn et al., 2021] assumes = 92, and therefore cannot be applicable when the dimensionalities of s and S2 differ (c.g., d,, 2, dz, 3), hence the "NA" in the table. Figure 9 in Appendix E.2 shows scatter-plots of the true and the estimated components with dg = ds, = de, 2. We can observe that components of and those of & are highly correlated, suggesting that the information of z is indeed restored. In contrast, 2 contains very little information of s1, consistent with our theory that a desirable disentanglement is attainable. Additional experiments on varying sample sizes can be found in Appendix E.2
Theorem 3.2 Moreover, prior
4.3 Hierarchical Model Identification
Synthetic data. We present the evaluation of Algorithm Ton latent hierarchical models in Figure 4 Table and Table in Appendix E.3 In Figure we observe that all variables can be estimated presents the pair- prediction scores within pairs of estimations while learning the V-structure model in Figure 4b] We can observe that scores within the sibling pairs (x1, X2) and (x4, x5) are much higher than non-sibling pairs. Notably, the estimate from v X3 can perform accurate prediction over other estimates, whereas the other estimates fail to capture it faithfully. This is consistent with Theorem 3.4 the basis model with V1 X3 will output variable equivalent the concatenation of Z and Z3, explaining the asymmetric prediction performances. These results empirically corroborate Theorem 3.5
decently despite a slight recovery loss from a lower to a higher level. Table 2
Personality dataset. From Figure we can observe that nonlinear model correctly cluster the variables associated with the same attribute together in the first level, which is consistent with the intentions of these questions. It suggests that conscientiousness, agreeableness, and neuroticism closely related at the intermediate level, whereas extraversion and neuroticism are remotely related. Some observed variables are not shown in the figure, as they are not clustered with other variables, indicating that they are not closely related to the system. This may provide insights into questionnaire design.

---

Crapping
Rotation
V1
Vi
0.89 0.0.
0.51 1.00
0.51 0.005
0.48 0.01
0.47
0.48 0.02
0.42 = 0.03
0.78 0.05
0.53 0.08
0.48 + 0.02
0.75 0.05
(a) The learned causal structure.
(b) Pairwise predictions.
Figure 6: Multi-view digit dataset results. Figure 63 (V1, Vz) and (Va, v4) form clusters, they share the aspect-ratio factor within the cluster while distinct the rotation angle factor. Table each box (a, b) shows the R² score obtained by applying the estimated variable produced by treating one specific view a as Vi to predict the estimated variable produced by treating view b as V1-
Digit dataset. Fig- ure [6al and Table [6b present the causal structure learned from the multi-view digit dataset. We can observe that we can automatically cluster the two views sharing more latent factors. This showcases that our theory and ap- proach can handle high-dimensional variables, whereas prior causal structure learning work mostly assumes that all variables are one-dimensional.
X3
X4
X5
X1
X
0.85 = 0.000
0.53
= 0.01
0.57
= 0.002
0.55 + 0.003
X2
0.83
0.006
X
0.52 0.002
0.54
1 0.001
0.53 1: 0.000
Xs
0.90
0.002
0.88
0.002
0.86 0.001 X
0.90
0.006
0.57 + 0.001
0.54 + 0.002
0.55
0.003
0.003
0.55 1 0.006
0.56 1: 0.001
0.55 0.002
0.83
0.003
Table 2: Pairwise predictions among estimated variables in Figure 4b Fach box (a, b) shows the R² score obtained applying the estimated variable produced by treating a Vi predict that produced by treating the prediction scores within sibling pairs are noticeably higher than other pairs, suggesting decent structure estimation. In particular, the estimate from V1 = x can predict other estimates accurately, whercas not the other way round, confirming will recover the information of both Zı and zy. The results
V1. We observe that
theory that are averaged over 30 random seeds.
5 Conclusion
In this work, we investigate the identifiability of causal structures and latent variables in nonlinear latent hierarchical models. We provide identifiability theory for both the causal structures and latent variables without assuming linearity/discretneess as in prior work [Pearl. 1988 Choi al. 2011] general than (generalized) latent Choi et al. 201 I Huang et al. 2022). Together with the theory, we devise an
Huang et al. 2022
Xie et al. 20222 while admitting structures
trees Pearl 1988
identification algorithm and validate it across multiple synthetic and real-world datasets.
We have shown that our algorithm yields informative results across various datasets, However, is essential to acknowledge that its primary role is as a theoretical device for our identification proof. It may not be well-suited to large-scale datasets, e.g., ImageNet, due its nature as discrete search algorithm. In future research, we aim to integrate our theoretical insights into scalable continuous- optimization-bo
algorithms (Zheng et al., 2018) and deep learning. We believe that our work facilitates the understanding of the underlying structure of highly complex and high-dimensions datasets, which is the foundation for creating more interpretable, safer, and principled machine learning systems.
Acknowledgment. We thank anonymous reviewers for their constructive feedback. The work of LK and YC is supported in part by NSF under the grants CCF- 1901 199 and DMS-2134080. This project also partially supported by NSF Grant 2229881, the National Institutes of Health (NIH) under Contract RO1HL15980S grant from Apple Inc., grant from KDDI Research Inc., and generous gifts from Salesforce Inc., Microsoft Research, and Amazon Research.
10

---

References
J. Adams, N. Hansen, and K. Zhang. Identification of partially observed linear causal models: Graph- ical conditions for the non-gaussian and heterogeneous cases. Advances Neural Information Processing Systems, 34, 2021.
K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning, 2023.
S. Akbari, E. Mokhtarian, A. Ghassami, and N. Kiyavash. Recursive causal structure learning in the presence of latent variables and selection bias. In Advances in Neural Information Processing Systems, volume 34, pages 10119-10130, 2021.
A. Anandkumar, D. Hsu, A. Javanmard, and S. Kakade. Learning linear bayesian networks with latent variables. International Conference on Machine Learning, pages 249-257 2013.
S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Schölkopf, and P. Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing, 2023.
R. Cai, F. Xie, C. Glymour, Z. Hao, and K. Zhang. Triad constraints for learning causal structure of latent variables. In Advances in Neural Information Processing Systems, pages 12863-12872, 2019.
M. Casey and A. Westner. Separation of mixed audio sources by independent subspace analysis. In ICMC, pages 354-161,22 2000.
V. Chandrasckaran, S. Sanghavi, P. A. Parrilo, and A. S. Willsky, Rank-sparsity incoherence for matrix decomposition. SIAM Journal on Optimization, 21(2):572-555 2011.
V. Chandrasckaran, P. A. Parrilo, and S. Willsky. Latent variable graphical model selection via convex optimization. Annals of Statistics,
2012.
Z. Chen, F. Xie, J. Qiao, Z. Hao, K. Zhang, and R. Cai. Identification of linear latent variable model with arbitrary distribution. Proceedings 36th AAAI Conference on Artificial Intelligence (AAAI), 2022.
M. J. Choi, V. Y. Tan, A. Anandkumar, and A. S. Willsky, Learning latent tree graphical models. Journal of Machine Learning Research, 12:1771-1812, 2011.
D. Colombo, M. H. Maathuis, M. Kalisch, and S. Richardson. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, pages 294-321, 2012.
R. Cui, P. Groot, M. Schauer, and T. Heskes. Learning the causal structure of copula models with latent variables. In UAI, pages 188-197. AUAI Press, 2018.
L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, (9(6): 141-142, 2012.
M. Drton, S. Lin, L. Weihs, and P. Zwiemik. Marginal likelihood and model selection for gaussian latent tree and forest models. Bernoulli, 2322)11002 2017.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
B. Huang, C. J. H. Low, F. Xie, C. Glymour, and Zhang. Latent hierarchical causal structure discovery with rank constraints. arXiv preprint arXiv:2210.01799 2022.
F. Huang, N. U. Naresh, I. Perros, R. Chen, J. Sun, and A. Anandkumar. Guaranteed scalable learning of latent tree models. In Uncertainty in Artificial Intelligence, pages 883-8988 PMLR, 2020.
A. Hyvärinen and P. Hoyer. Emergence of phase-and shift-invariant features by decomposition of natural images into independent feature subspaces. Neural computation, 12(7):1005-177 2000.
A. Hyvarinen, H. Sasaki, and R. Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
11

---

Y. Jiang and B. Aragam. Learning nonparametric latent causal graphs with unknown interventions, 2023.
I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207 2207-2217 2217. PMLR, 2020.
B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Learning latent causal graphs via mixture oracles. Advances in Neural Information Processing Systems. 34, 2021.
L. Kong, S. Xie, W. Yao, Y. Zheng, G. Chen, P. Stojanov, V. Akinwande, and K. Zhang. Partial disentanglement for domain adaptation. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1455-114472. PMLR, 17-23 Jul 2022. URL httphhttts//pcceeeeeeeeee
mlr. wwwpp/eesvvv..... v 122 kong2aa html
E. Kummerfeld and J. Ramsey. Causal clustering for f-factor measurement models. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1655-1664. ACM, 2016.
S. Lachapelle, P. R. López, Y. Sharma, K. Everett, R. L. Priol, A. Lacoste, and S. Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica, 2022.
Q. V. Le, W. Y. Zou, S. Y. Yeung, and A. Y. Ng. Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In CVPR 20/ pages 3361- 3368. IEEE, 2011.
W. Liang, A. Kekić, J. von Kügelgenn S. Buchholz, M. Besserve, L. Gresele, and B. Schölkopf. Causal component analysis, 2023.
Q. Lyu, X. Fu, W. Wang, and S. Lu. Understanding latent correlation-based multíview learning and self-supervision: An identifiability perspective. In International Conference on Learning Representations, 2022. /ttp:/tttp.//ppeeeeeeeee...e www.eerrrwiee..eet/ouummmmm (
J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, 1988.
J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, NY, USA, 2000. ISBN 0-521-77362-8.
J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA, 2017.
O.-.. P. Project. Big five personality test, 2019. URL hwl.lagpe///pppppyooooerrrrrrree (PIP-BFFI Accessed: 2023-5-16.
S. Shimizu, P. O. Hoyer, and A. Hyvärinen. Estimation of linear non-gaussian acyclic models for latent factors. Neurocomputing, 72(7-9):202--2222 2009.
R. Silva, R. Scheine, C. Glymour, and P. Spirtes. Learning the structure of linear latent variable models. Journal of Machine Learning Research, 1(Feh):191-246, 2006a.
R. Silva, R. Scheines, C. Glymour, and P. Spirtes. Learning the structure of linear latent variable models. Journal of Machine Learning Research, 7:191-246, 2006b.
Spirtes, N. Glymour, and R. Scheines. Causation, Prediction, and Search. MIT press, 2000.
Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search. MIT Press, Cambridge, MA, 2nd edition, 2001.
C. Squires, A. Seigal, S. Bhate, and C. Uhler. Linear causal disentanglement via interventions, 2023.
Theis. Towards a general independent subspace analysis. Advances in Neural Information Processing Systems, 19: 19:1361-1366 2006.
12

---

B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer. Score-bases causal representation learning with interventions, 2023.
von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and Locatello. Self-supervised learning with data augmentations provably isolates content from style. arXiv preprint Ar0.v:2106. 040.1199100061, 2021.
Xie, R. Cai, B. Huang, C. Glymour, Z. Hao, and K. Zhang. Generalized independent noise condition for estimating latent variable causal graphs. In Advances in Neural Information Processing Systems, pages 14891-14902, 2020.
F. Xie, B. Huang, Z. Chen, Y. He, Z. Geng, and K. Zhang. Identification of linear non-gaussian latent hierarchical structure. In International Conference on Machine Learning, pages 24370-24387. PMLR, 2022.
W. Yao, Y. Sun, A. Ho, C. Sun, and K. Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint arXiv:21 2021.
W. Yao, G. Chen, and K. Zhang. Learning latent causal dynamics. arXiv preprint arXiv:2202.04828, 2022.
Zhang, C. Squires, K. Greenewald, A. Srivastava, K. Shanmugam, and C. Uhler. Identifiability guarantees for causal disentanglement from soft interventions, 2023.
N. L. Zhang. Hierarchical latent class models for cluster analysis. The Journal of Machine Learning Research, 5:697-723, 2004.
X. Zheng, Aragam, K. Ravikumar, and Xing. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems, 31, 2018.
Y. Zheng, I. Ng. and K. Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. arXiv preprint N.XKv:220060775:11 2022.
C. Zhou, X. Wang, and J. Guo. Learning mixed latent tree models. Journal of Machine Learning Research, 21(249):1-35, 2020.
13

---

Appendix for "Identification of Nonlinear Latent Hierarchical Models"
A Detailed Literature Review
Previous causal discovery approaches. which allow latent confounders and causal relationships among those latent variables, either assume linear causal relationships or assume discrete data. Representative approaches along this line include rank deficiency- methods, generalized independent noise condition-base methods, and mixture oracles-basee methods. (1) Rank deficiency-basse -based. By testing the rank deficiency over cross-covariang matrices over observed variables, one is able to locate latent variables and identify the causal relationships among them in
methods, matrix decomposition- based
models. [Silva et al. 2006a], Kummerfeld and Ramsey (2016) make use of the Tetrad condition, special case of the rank rank-deficiency constraints, to handle the case where each observed variable is influenced by only one latent parent, and each latent variable has at least three pure measured children. The Tetrad condition has also been used to identify latent tree structure [Pearl 1988.. Recently, the rank-deficiency constraints have been extended to identify more general hierarchical structures [Huang et al. 2022]. (2) Matrix decomposition-bant bbaeed. It has been shown that, under certain conditions, the precision matrix can be decomposed into a low-rank matrix and sparse matrix, where the low-rank matrix represents the causal structure from latent variables to observed variables and the sparse matrix gives the structural relationships over observed variables. To achieve such decomposition, certain assumptions are imposed on the structure [Chandrasckaran et al., 2011 2012. Anandkumar et al. 2013]. e.g., there should be three times more measured variables than latent variables. (3) Generalized independent noise (GIN) condition- based. The GIN condition is an extension of the independent noise condition in the existence of latent confounders, relying on higher-order statistics to identify latent structures. In particular, Xie et al. (2020) proposes
GIN-based approach that allows multiple latent parents behind every pair of observed variables and can identify causal directions among latent variables. Moreover, Adams al. (202T] gives necessary
and sufficient structural constraints in the linear, non-Gaussian or heterogeneous case, to identify the
latent structures. (4) Mixture gracles-based a mixture oracles-basee method to identify the latent variable graph that allows nonlinear causal relationships. However, it requires that the latent variables are discrete and each latent variable has measured variables as children. Thanks to the discreteness assumption, it can handle general DAGs over latent variables. On the other hand, regarding the scenario of latent hierarchical structures, most previous work along this line assumes a tree structure and requires that each variable has only one dimension and that the data is either Gaussian or discrete [Pearl] 1988 Zhang 2004 Choij et al.] 2011 Drton et al.) 2017 Huang et al., 2020). In contrast, we address the general nonlinear case with continuous variables. Moreover, our conditions allow for multiple undirected paths between two variables and thus more general than tree-based assumptions in prior work.
method-based. Recently, Kivva et al. [202]] proposed
Another related research line is latent-variable identifiability literature. [Hvarinenn et al. (2019) 9
Khemakhem et al. [2020). [Kong et al. [2022] have shown that with an additional observed variable to modulate latent independent variables, the latent independent variables are identifiable. Recently, Yao et al. [021 2022] allow time-delayed causal relationships among latent variables. However, for time-delayen causal relations, the causal direction is fixed and predefined, and moreover, they assume that all latent variables have measured variables as children, avoiding the hierarchical cases. Prior workfvon Kügelgen et al., 202T Lyu et al. 2022] studies latent-variable models related to our proposed basis model (which serves as tool and is defined in Section 2 in this work, but with more restrictive functional and statistical assumptions. To the best of our knowledge, no prior work has managed to identify latent variables or causal structures in nonlinear latent hierarchical models.
We note that our work focuses on identifying latent causal models from observational data and structure conditions. Another important line of work [Ahuja et al. 2023] Squires et al. 2023 Varici et 2023] 2iang and Aragam| 2023 Liang et al., 2023] Buchholz et al. [2023] Zhang et al. 2023] utilizes interventional data for this purpose. Specifically, these works leverage multiple data distributions generated by one causal model under distinct interventions, which are accessible in applications like biological experiments. The accessibility of interventional data can allow for relaxed structure conditions. Hence, one should consider this tradeoff when faced with causal identification problems.
14

---

B
Proofs
B.1
Proof for Theorem 3.2
The original assumptions and theorem replicated here for reference.
In Theorem 3.2, we show that the latent variable shared by V1 and V2 identifiable up to one-to-one mapping by estimating generative model (P2,a, 9) according to Equation 2 We denote the support Jacobian matrix Jg and Jg as and 9 respectively and denote by matrix with the same support as T(c) in J (c)c)T(c).
Assumption 3.1 (Basis model conditions).
(Differentiability & Invertibility): The mapping g(c) (V1, V2) is a differentiable invertible function with differentiable inverse.
Jesl and such that (Subspace span]: For all i E {l.....doy dd, + dez there exists (c(1)) Gua span([J, (cl))) (2r))) R and (J IR... G...
ii (Edge Connectivity]: For all j. E {I,..., dz}. there exist in (b...,,dr.. and ivz [dv....., dv, + de,), such that (te): E G and (te2) E G.
Theorem 3.2. Under Assumption 3.1 if generative model (Pz.32) PoL. () follows the data-generating process in Equation 2add matches the true joint distribution:
Priva (Vivs(Vi, v2) Prive (V1, v2), V(v1,v2) x V,
then the estimated variable 2 and the true variable equivalent up invertible transformation.
Proof. We first define the indeterminacy function:
= 9 og
which is a smooth and invertible function h + thanks to Assumption 3.1
According to the chain rule and inverse function theorem, we have the following relation among the Jacobian matrices:
J (C(C) = (c) (c).
For of exposition, denote M(c) = J (c) in the following.
We define the support notations as follows:
g == supp(J g)),
supp(J (J g))
T = supp(M).
for any i {l...., de, + dodp, there exists (c(0) 19c.1
Because of Assumption
3.1 ii
such that
( (Sa:ly G...
Since
GLal forms basis of Rd. for any Gi.s we can express canonical basis (c)(
vector ejo Rd as:
//
Σ ar Je-JJy(c )c(0
(5)
teG
where a E a coefficient.
Then, following Assumption 3.1 there exists deterministic matrix T such that
Jo. - T
Σ ar () Rd S.."
(6)
EEG.
where E is due the fact that each element the summation belongs Rd G..""
15

---

Therefore,
Vj Gi.3 T3. E Rd
D2.
Equivalently, we have:
v(G5) Eg, (1) x Tz: C...
We would like to show that 2 is not influenced by $1 and S2, which is equivalent T Ji.ja 0 for
je E {l....,d, d and ja E {dg + (,...,,d2.. de).
We first will prove this for ja {dz + 1,..., d2 + d, by contradiction. Suppose that {(2; E T with J= E (1...., } and js, E (d2 + -.....,., d
Thanks to Assumption 3.1 there must exist dog E {dd, +l,..., du, + dd such that, (dm2) E G.
It follows from Equation that:
{ing x Thsi C - (in2) E S.
(8)
However, due to the structure of 02. [ "eeleeg Js, = O. which results in a contradiction. Therefore, such (Fog ja,] does not exist. The same reasoning gives rise to that (ja.ja,) $ T with ja E {
and Jäz E {d= + da + 1...., de Therefore, we have shown that Ti..j.
for j= (1...., d.)
and ja E {d= + 1, ..... de). As T is invertible, it follows from the block-matrix inverse formulae
that T = 0 for js E {l..... d₂) and ja E {dz + 1...... .de. In conclusion, 2 is not influenced by (81,82). Thus, we have shown that there is a one-to-one mapping between and
B.2 Proof for Theorem 3.4
The original theorem is copied below.
Assumption 3.3 (Hierarchical model conditions). i [Diiferentiibilltyy] Structure equations in Equation 7 are differentiable.
Any z E and exogenous variable e can be expressed as differen- tiable functions of all observed variables, i.e., fx) and (X).
cenfommation-consevaattt
in (Subspace span]: For each set A that d-separates all its ancestors Anc(A) and observed variables X. i.e., X L. Anc(A) for any ZA E Pa(A), there exists an invertible mapping from a set of ancestors of A containing ZA to the separation set A, such that this mapping satisfies the subspace span condition (i.e., Assumption 3.1 i..
iv [Edge connectivity]: The function between each latent variable and each of its children z' has a Jucobian J, such that for all j E (l....,d3, dZ), there exists E {l,..., dz } such that (1,5) is in the support of J.
Theorem 3.4. In a latent hierarchical causal model that satisfies Condition 2.4 and Assumption 3.3 F
we consider X, E X as vı and \v1 as V2 the basis model (Figure 2). With an estimation model (fr Pe 9) that follows the Lata-generating process in Equation 2 the estimated à is a one-to-one mapping of the parent(s) of v1, i.e., h(Pa(vi)) where h(() is an invertible function.
Proof. We show that performing estimation following the generating process Equation 2 to any
Vi = X; E X and V2 = X is equivalent to the estimation of the model in Figure with Pa(v1). Therefore, the identifiability result ensues, thanks to Theorem 3.2
First, we show that for each selection of v1, we can locate (z, S2) such that the conditions for the basis model in Theorem 3.2 are satisfied. We choose (z, 52) as follows.
We choose the z to be all parents of v z = Pa(vi)
We choose 8j to be exogenous variables that cause vi together with z: 8) == Ev
'To avoid cluttering, we slightly abuse the bold lowercase font to represent either an individual vector or vector set (which can be viewed concatenation).
16

---

To obtain $2, we trace back (traversing backward every edge) from the va recursively and execute the following steps. At each step of backtracking, we include any cause (exogenous variables and endogenous variables) that is not situated on any directed path from 2 to V2- The backtracking for each path halts when the most recent step recovers an endogenous variable out of the directed paths, or it reaches z. We note that such a choice for $2 is not unique the above procedure is only one instance.
Invertibility. From (z, S1, S2) to (V1, V2). we can observe that (z, 52) by construction include all the information to generate (V1, V2). i.c., X, as (z, $ Sg) d-separate their parents/ancestors from (V1, V2) and contain all necessary exogenous variables. From (V1, V2) to (1, we can observe that as the tuple (V1, V2) comprises the entire observable set X, which contains the information of all latent exogenous and endogenous variables according to the general invertibility (Assumption 3.3 of the hierarchical model. Therefore, the mapping is invertible.
Conditional independence: Vi V2 As z is chosen to the parents of Vi and there is no edge among the observables X (i.e., leaf variables), the local Markov property implies conditional independence.
Subspace span. As (V1, v2) == X satisfies the tree conditions in Assumption 3.3 Pa(z), the subspace span condition follows from Assumption 3.3
W.r.t,, Z
Edge connectivity. We assume that in the hierarchical model, the function between each latent variable and each of its pure child z' has non non-degenerate Jacobian matrix in sense that for all j (1....,d, } there exists E {l....,dey }. such that (i.j) is included in the Jacobian matrix's s support (i.e., Assumption 3.3 liv. Therefore, since each latent variable has at least pure children (i.e., Condition 2.4 and Vi contains 1 variable, at least pure child of z or a descendant of a pure child of z will appear in V2. Thus, for each
{1,..- de), there exist iej and in such that both J) and (fvg:) are contained in the support of the Jacobian matrix, which fulfills the edge connectivity condition (i.e., Assumption 3.1 i.j.
Thus, it follows from Theorem 3.2 that the estimated variable is a one-to-one mapping to the true variable
B.3 Proof for Theorem 3.5
Theorem 3.5. Under assumptions in Theorem 3.4 all latent variables Z in the hierarchical model can be identified up one-to-one mappings, and the causal structure G can also be identified.
Proof. We will show by induction that each iteration of Algorithm I fuulills the following conditions: Condition B.1 (Active-set conditions).
i Each element the active set is a one-to-one mapping of a distinct variable Y.
There are no directed paths among variables in A.
ii The graph is d-separated by into latent variables ZA that have not been included in A and those ZA that were in A (but not now): ZA L ZZ A.
We will first verify the base case where active set A is assigned observable set X. Condition B.I is automatically satisfied due the initial assignment. As X are all leaf variables, there no directed paths within X, and therefore Condition B. is satisfied. Condition B.1 iillis also met trivially, as ZA 0 at the initial step. So far, we have verified the base case.
Now, we make the inductive hypothesis that all conditions hold before an iteration and show that these conditions are maintained after the iteration.
We first note that Condition enable us to apply Theorem 3.4 to the modified graph consisting of variables in A as the bottom layer
Condition B. fi and Condition B. ii
the inductive hypothesis
17

---

and all latent variables that have not been placed in A. This modified graph satisfies the structural properties required by Theorem 3.4
All paths end at active (observed) variables.
There are directed paths among the active variables.
We now analyze the updates to A made at Step 16 Algorithm - each iteration. For a specific variable E P(A), there are the following cases.
Case B.2
cases).
i has only pure children and all of them are in A. z has only pure children and not all of them are in A.
ii z has coparents and all its children and its co co-parents' children are in A.
iv has coparents and not all its children and its co-parents' children are in A.
For Case B.2 when each pure child a of is treated as V1, the basis model will yield z. So, there certainly estimates of in P(a). As possesses least 2 pure children, all of which belong to A. there will be duplicates, i.c., multiple equivalent one-to-one mappings of z, which we merge into one at Step Therefore, for each pure child a of the only element P(a) is identical one-to-one mapping of z. It follows that JointP contains a pair (Z, H) where Z consists z alone and H contains all its pure children. This fact, together with Condition B.1 iii implies that Zus would be the parent of z, which cannot perfectly predict z at Step [4] Therefore, Algorithm the pure children of with the estimate of z in A.
D will substitute all
We now switch to Case B.2 B
Analogous to the process in Case B.2 i after Step 6 we will have P(a) = {z} for each active pure child. However, not all pure children of are active, JointP((z)) H certainly contains descendants of z.
does not include all of the pure children of z and follows from Corollary that Eis will be one-to-one mapping of and can predict it perfectly at
Step 14 Therefore, z will not be updated to A at this iteration.
we will show that the update takes place for and its coparents. Without loss of generality, we assume that z only has one coparent z'. As both has multiple pure children (i.e.. Condition 2.4 and all of them are in A, the dictionary P will contain z and z' in their keys and P(z) and P(zz) contain their pure children respectively. Suppose would be a singleton set containing one-to-one mapping of x" : [z. z'). after Step 6 At Step 9 z" will be recognized as a combination of z and x' and replaced with them. follows that Joint? will contain pair IZ H) such that Z (z, z²) and H is composed of all the children of the two coparents. At Step 16 both z and z' will be updated into A. Cases with more than one coparents can be dealt with in the same manner.
For Case B.2 iii
a shared child of z and z', then P(a)
For Case B.2 iv we show that the update will not be carried out for z and its coparents. Following the notation in Case B.2 ten
we assume that z has a coparent z', and these two share at least one active child. If neither nor has any pure active children, they will be regarded as supermode z" |z, x'] and the corresponding H contains only the shared active children. Then, A \ H will certainly contain descendants of both z and z'. Consequently, 2is will be equivalent to the supernode z" and predicts it perfectly at Step 14 In this case, neither z and z' will be updated into A. If only one of z and z' has no pure children in A, then neither will be updated into A due to Step 9 Without loss of generality, we assume that z' does not have any active pure children, whereas Z does. Then, there will be no estimated z' in the values of P. but there will be estimated z and z') As a results, Step g will prevent z and z' from updated to A. If both z and z have active pure children, then there will be a pair (Z, H) in JointP such that Z = {z, z'} and H contains all their active children. As H does not contain all the children of and z'. the A H will contain descendants of at least one of and z'. Then 2es will contain all information of at least one of z and z' and predicts it perfectly at Step 16 which aborts the update of and z'. Therefore, no updates will happen in this case.
As each of the newly estimated variables in the values of JointP a one-to-one mapping to distinct variable in Z, the iteration preserves Condition B.
18

---

Z1
Z3
Z1
Zs
Figure 7: A specific case to demonstrate how we can avoid assuming specific functional classes.
can observe that if z updated into A this iteration, all its children would be in A at the beginning of this iteration and get removed at Step T6 Therefore, the update would introduce directed paths to A. Condition B. m1 holds for the same reason: the newly introduced variables in the active set A at the end of the iteration separate their children in the active set A at the beginning of the iteration and all undiscovered latent variables.
For Condition B.1
So far, by induction, We have shown that Condition B. ii Condition B. ii and Condition B. i11 hold the end of each iteration. Given the analysis above, each iteration, whenever all their children and their coparents' children are active, i.c., Case B.2 i and Case B.2 and the causal relations between the parents and the children are encoded in JointP.
update variables
We further argue that at every iteration, there exists at least one undiscovered latent variable that has all its children in A. We focus on the trimmed graph of ZA and A, where ZA refers to all undiscovered variables. As the graph is of finite size and all variables in ZA have directed paths ending at active variables in A, the active variable farthest from the root does not have siblings in ZA: otherwise, there would be a' (i.e., the child of a's sibling in AA) in that was further from the root than a.
In sum, we have shown that there will exist at least one update each iteration, and each update will lead to the discovery of new latent variables and correct causal structures. Therefore, Algorithm [ can successfully identify the underlying hierarchical causal graph and each latent variable up to one-to-one mappings.
B.4 Illustration of a Specific Case
Figure 7 shows a specific case to demonstrate how we can avoid assuming specific functional classes. For active set A {zg. 24, 25 }. if we treat 22 V1, 2, will become in the basis and [za, 25] will become V2. This set of (z, V1, V2) satisfies Assumption
as all information about Z₁ is contained by 22, 24, and 25. Assumption 2 and Assumption 2 ii direct consequences of Assumption 3.3 liv
and Assumption 3.3 Also, we can always find an independent variable S := E in the basis model. Therefore, Theorem guarantees the estimated å to be one-to-one mapping of Z1.
C Relaxation of Structural Conditions
C.1 Relaxation of Condition 2.4 ii
We remark that Condition 2.4 as the first step nonlinear- meentthirarcchcall causal discovery, we wish to present the basic techniques more clearly handling triangles involves additional components of the algorithm and could potentially compromise the readability.
4 is introduced to simplify the presentation of main results
We give an example in Figure with triangle structure to illustrate how our theory and algorithm can handle triangles. We refer to variables in triangle under the following convention: 1) root: the variable which is the cause of the other two variables (e.g., 2, in Figure 8). 2) sink: the variable which the effect of the other two variables (e.g., 23 Figure 8). and 3) bridge: the variable which the effect of the root variable and the cause of the sink variables (e.g., Z2 in Figure 8).
19

---

61
22
Z4
X1
X3
X8
X5
Figure 8: An example of triangle structures, i.e. Z1, Z2, and Z3.
In the following, we illustrate the procedures to recognize and handle triangle structure by navigating through Figure ( First, the directed path detection procedure portrayed in Section can pinpoint the directed path between the bridge variable and the sink variable (e.g., Z₂ and Z3). We need to determine whether the two variables indeed belong to a triangle. To this end, we can lump the bridge variable and the sink variable together as V1. We perform the basis model estimation over and the active set A excluding V1 's children. In Figure 8 we perform the basis model over V1 [zg. Z3] and A {Z2, 23, X7, X8, X9). which returns estimate of Zı. Further, bridge variable 22 and the estimated variable from the lumped V constitute the parents of the sink variable 23 by mutual prediction, which ascertains the two variables are indeed the bridge variable and the sink variable of a triangle, respectively. In this example, we find that the estimated variables of Z2 and 2 together contain identical information to the estimated variable of Z3. which reveals that 22 and 23 are the bridge variable and the sink variable of triangle. This concludes the triangle detection procedure. With this knowledge, can handle the triangle structure by lumping the bridge variable Z3 and the sink variable 23 into a super-variable variable and proceed with this modification.
can determine that the
C.2 Admitting Non-leaf variables into X
Corollary 3.6 can detect non-leaf observed variables and exclude them from the initial active set X in Algorithm Thus, the problem can be reduced back to the case without observed non-leaf variables, which can be tackled by Algorithm
Identical to the directed path detection detailed in Section 3.3
D Algorithm 1 Complexity
The dominant complexity term in our proposed algorithm is deep learning training, which is of complexity O(R + n) where R is the number of levels in the hierarchical model, and n is the number
of observed variables. In contrast, recent works linear hierarchical models provide algorithms with complexity ORR * (1++)) (n)++) [Huang et al. 2022] and O * n!) [Xie et al. 2022 where is the largest number of variables that share children. Our algorithm achieves the lowest complexity in terms of the algorithmic complexity. As we allow for multi-ddmmensooall variables, the dimensionality of each variable also contributes to the overall wall-clock time through the deep learning model dimension, which is a hyperparameter for the experimenter.
E Experiments
E.1
Additional Experimental Details
We provide additional details of experiments in Scction For the basis model evaluation in Section 4.2 then encoder f and decoders (û1. are 4-layer MLP's with a hidden dimension 30 times as large as the input data and Leaky-ReLU (a 0.2) activation functions. For the synthetic hierarchy experiments in Section 4.3 the model layers are and 50 times as large as the input data.
For the personality dataset in Section 4.3 the model layers are and 8 times large the input data. For the multi-view digit dataset in Section 4.3] the model layers are 4, and times (encoder)
and 2 times (decoder) as large the input data The model configurations summarized in Table 3
20

---

# encoder layers
# decoder layers
encoder widths
decoder width
Basis models (Section 4.2
30x
30x
Synthetic hierarchy (Section 4.3
50x
50x
Personality dataset (Section
8x
Digit dataset (Section 4.3
4
4x
2x
Table 3: Estimation model configuration for each experiment.
We apply Adam to train each model for 20, 000 steps with a learning rate of le - 3. For hicrarchical models, consistently dimensionality as 2. We evaluate the R² score with kernel regression over 8192 samples for each estimated variable pair. Figure 6.b, The matched pairs give significantly higher scores than the unmatched ones (e.g.. Table 2 and Figure 66b.. For instance, in the first row of Table 2, the estimated latent variable with This gap allows us set threshold for each experiment distinguish the matched estimated variables (e.g.. a threshold of 0.6 suffices for the experiment in Table 2
2-layer MI.Ps for generating functions and set the exogenous variable
E.2 Additional Results on the Basis Model
Figure 9 visualizes the dependence between the estimated and true components. We can observe that the is highly correlated with whereas has little correlation with 2, which verifies Theorem 3.2 that we can successfully identify z.
Esterna Lit
Trae 17
Figure 9: The scatter plot between the estimated á and the true (2, s1). The scatter points are obtained in an estimation run with d, = d,, = d,, = 2. We can observe that the estimated components of å are highly correlated with those of and have little correlation with those $1. empirically verifying Theorem 3.2
Although our theory only concerns asymptotic properties, our approach can perform stably at relatively low-sample regimes. As shown in Figure 10 algorithm can achieve decent identification scores when the sample number exceeds 5000. The performance peaks with little variance when the sample size is over 10000.
E.3 Additional Results on Synthetic Hierarchical Models.
Table 4 and Table contain the mutual prediction scores for the hierarchical structures in Figure 4a and Figure 4c We can observe that the estimated variables that yield high mutual prediction scores indeed under the same causal cluster, which gives signals for structure learning.
21

---

1.0
0.8
0.6
R² scooes
0.4
0.2
0.0
1000
10000 Number of training samples
2000
5000
100000
Figure 10: R2 scores for basis model identification. The basis model (Figure in the manuscript) has latent dimensionality d, d,, dæp = 2. The basis model training and data generation follow the training configuration the main text. We vary the number of training samples and measure the R² score to assess the identification result. Each result is averaged over 3 random seeds.
X1
X2
X3
X4 0.50 + 0.08
X1
X
0.83 + 0.03
0.57
+ 0.04
0.86 + 0.01
0.58 + 0.03
0.52 + 0.04
0.6 + 0.02
0.59
0.05
X
0.77 0.00
0.59 + 0.02
0.59
0.05
0.81
0.04
Table Pairwise predictions among estimates in Figure 4a Each box (a, b) shows the R score obtained by applying the estimate produced by treating a V1 to predict the estimate produced by treating b as v1. We can observe that the prediction scores within sibling pairs (x1, x2) and (xs, x4) are noticeably higher than other pairs, showing decent structure estimation.
E.4 Additional Results on the Personality Dataset
We randomly and evenly partition each set of six questions into two variables. Following this procedure, have observed variables C₁, C2, a₁ ag, 11, n2 e1, e2, O1, and O2, each of 3- dimension. The Letter in each variable name indicates the attribute to which the variable belongs, facilitate readability. Figure [ contains the results. Analogous Figure [5] the learned structure faithfully reflects the questionnaire structure questions designed for the same personality trait are clustered together.
2
X3
X
Z
0.81 0.011
0.43
0.006
0.52 0.000
X3
0.87 0.002
0.47 1 0.003
0.54
0.002
0.57 + 0.000
0.52 + 0.006
0.77 - 0.005
X5
0.58 + 0.001
0.57 + 0.015
0.81 + 0.000
X
Table 5: Pairwise predictions among estimated variables in Figure 4c with Tad Table
The pattern consistent
22

---

Figure 11: The casual graph learned by method, where each observed variable (c. n, 0) is of three dimensions.
23


Reviews:
[{"review_id": "VW5UokOIJi", "reviewer": "Reviewer_wEjX", "summary": "This paper presents an identification strategy that allows the unique reconstruction of a latent hierarchical model, including both the graphical structure and (up to invertible transformations) the values of the latent variables. Assumptions include faithfulness, some structural assumptions (weak compared to related work), and other more technical assumptions. The algorithm is positively evaluated on various datasets.", "questions": "* line 97-99, \"a latent variable with no pure children can be merged into its children without affecting the overall generating process\": this merging appears to not be possible for the middle z in $x \\leftarrow z \\rightarrow x \\leftarrow z \\rightarrow x \\leftarrow z \\rightarrow x$. Could you explain what you mean exactly?\n\n* Assumption 3.1(ii) and text before it: The definition of T depends on c, but for what value? Further, how is T quantified over in assumption 3.1(ii)?\n\n* If $g$ is linear, the subspace span condition (ii) will not hold (except in 1-dimensional cases, I suppose). Can this be addressed by something like \"restructure latent variables by merging redundant components\" (line 156)?\n\n  * Related: in line 200-202, the identity function doesn't satisfy assumption 3.1(ii) if $d_z > 1$\n\n* Assumption 3.3(ii): Is the independence in 1) automatically satisfied if  $A_z$ and $X$ have nonempty intersection? (Please make this explicit or rewrite to avoid this case.) I suppose condition 2) refers to the induced subgraph of $A_z$, as a set of node does not strictly speaking contain any paths. Is there a minimum length to these paths? A 0-length path always exists, and if you mean a 1-length path, you could simply ask if there is an arrow between two nodes in $A_z$.\n\n* Assumption 3.3(iii): The \"function between each latent variable $z$ and each of its children $z'$ (I assume separately for each $z'$?) also depends on other parents of $z'$. Should this assumption hold for some/most/all of the values of those parents?\n\n* Algorithm line 17: remove variables independent of what? In the example, the only role of this line appears to be to clear A if it contains just the root. What else should this line do?\n\nThings to clarify:\n\n* latent variable identification: I'd recommend mentioning early on that this is up to transformations. Also, when this is first mentioned on the top of page 3, the objective is impossible to obtain without the assumption 3.3(i), which appears quite a bit later.\n\n* Condition 2.3 (ii): I recommend to give a definition for \"siblings\" here\n\n* In the algorithm, how do you detect equivalence of two constructed latent variables? (Mutual perfect predictability I suppose?)\n\n* Algorithm line 9: There needs to be \"\\ {$\\hat{z}$} after $P$.\n\n* Line 352-353: the part after \"whereas\" doesn't match the results\n\nTextual comments:\n\n* line 47 \"variable variables\"\n\n* line 171: \"descents\" -> \"descendants\"\n\n* line 211-212: \"$v_2$ always contains\" -> \"there always exists a $v_2$ [$\\neq v_1$]\"\n\n* line 248 & 337: \"an variable\" -> \"a\"\n\n* line 266 & 524: \"an one-to-one mapping\" -> \"a\"\n\n* line 523: \"proof\" should instead refer to theorem / assumptions", "limitations": "The paper has many assumptions, which are hard to interpret. While some effort is made to discuss most of these (exceptions: see weaknesses), this discussion leaves some questions (see above).", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Identifying latent structure (and even the values of latent variables themselves) is very powerful and could be used in many applications.\n\n* There are only weak assumptions on the graphical structure.\n\n* The experimental results look very good.", "weaknesses": "* Differentiability is assumed, but not mentioned explicitly (another assumption refers to Jacobians). The strength of a continuity/differentiability assumption is compounded by the assumption of invertibility.\n\n* I find it hard to assess how strong the subspace span condition is, in part because it is not written up clearly; see questions below.\n\n* Footnote 1 / appendix C.2 address the case where some non-leaf variables are observed. For this case, I think the structural assumptions become quite strong, so I do not consider this a strong part of the paper's contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your thorough reading and valuable insights! Below, we provide individual responses to your comments.\n\n**Q1: Differentiability is not explicit.**\n\nThank you for pointing this out. This condition should definitely be given, and we have made this assumption explicit in our revision. We'd like to note that the differentiability assumption and invertibility are adopted in almost all continuous latent-variable identification papers [18,19,21,24,36,37]. \n\n**Q2: The observed non-leaf case is not a strong contribution.**\n\nWe agree with you – Footnote 1 is only meant to illustrate that our theorem applies naturally to models with observed non-leaf variables beyond our main discussion, where all non-leaf variables are latent.\n\n**Q3: ``Line 97-99,..., the merging seems not possible….’’**\n\nThank you for raising the point. We meant to describe scenarios where latent variables are unidentifiable for lack of unique footprints. For instance, if two latent variables $z_{1}$ and $z_{2}$ share the same set of children $X$, i.e., $ z_{1} \\to X$ and $z_{2} \\to X$, then the two latent variables cannot be identified without further assumptions, while pure children would help in this case. \n\nIn light of your remark, we will replace the sentence in lines 97-99 “for instance, …” with “in a toy graph $\\{z_{1}, z_{2}} \\to X$, two latent variables $z_{1}$ and $z_{2}$ cannot be distinguished without further assumptions.”\n\n**Q4: The definition of $T$.**\n\nThank you for the question. In lines 135-136, we define $T$ as a fixed matrix that shares the support (defined in lines 130-132) of the matrix-valued function $T(c)$. Therefore, $T$ and $T(c)$ refer to two different objects, and $T$ is not dependent on $c$. This assumption is adopted in prior work [A]. Sorry for the confusion – we will replace the fixed matrix $T$ with $Q$ in the revision.\n\n**Q5: ``If $g$ is linear, the subspace span condition (ii) will not hold….’’**\n\nGreat observation! The subspace span assumption (ii) stipulates that the generating function's Jacobian should vary sufficiently within its support and is specialized for nonlinear functions, as discussed in [A,B]. Prior work [A] derives counterpart assumptions for the linear case (see Proposition 1, Assumption ii in [A]). These conditions can be adapted to our setup for the linear case. In addition, Prior work [B] supplies an explanation of the subspace span assumption (ii) and an example of functions that satisfies such an assumption (see Section 2.4.1. in [B]). In light of your question, we will include this discussion and the example in our revision to make our work thorough.\n\nRegarding the identity function, we note that its Jacobian matrix $G$ is an identity matrix. The resultant subspace $R_{ G_{ i , :} }^{d_{c}}$ is only a 1-d subspace (see the subspace definition in line 128) in $R^{d_{c}}$, because row $i$ only has one nonzero element $G_{i,i} = 1$. Consequently, Assumption 3.1 (ii) is met even if $d_{c} > 1$.\n\n**Q6: Assumption 3.3 (ii) and path lengths.**\n\nGreat question! When $A_{z} $ and $ X$ have a nonempty intersection, the independence condition 1) does not necessarily hold. Take a toy graph $z \\to x_{1}$, $z \\to x_{2}$ and set $A_{z} = \\\\{ x_{1} \\\\}$. We have $ X \\cap A_{z} = \\\\{ x_{1}\\\\} \\neq \\emptyset$ and $ z \\not\\perp X | A_{z} $ because the path $z \\to x_{2}$ is unblocked.\n\nAs you correctly identified, we refer to the subgraph induced by variables in $A_{z}$, and the paths refer to paths with nonzero lengths. Thank you for making this distinction, and we will explicitly make this point in our manuscript.\n\n**Q7: Assumption 3.3 (iii).**\n\nGreat question! As this assumption is made on the support of the Jacobian matrix (see lines 130-132), it is satisfied as long as the component $z_{j}$ has a nonzero influence (i.e., partial derivative) on its child's component $z'_{i}$ at *some* values of parents of $z’$.\n\n**Q8: Algorithm line 17.**\n\nAlgorithm line 17 removes variable $a \\in A$ if $a$ is independent of all other variables in $A$, i.e., the complement set $ A  \\backslash \\\\{ a \\\\} $.\nSome root variables may enter $A$ before the other root variables for graphs with multiple root variables. Line 17 removes these identified root variables from the active set $A$ before the iteration. One instance would be $z_{1} \\to \\\\{ x_{1}, x_{2}, x_{3}\\\\}$, $z_{2} \\to \\\\{ x_{3}, x_{4}, x_{5}\\\\}$, $z_{3} \\to \\\\{ x_{6}, x_{7}\\\\}$, and $ z_{4} \\to \\\\{ z_{2}, z_{3} \\\\} $. In this case, one root variable $z_{1}$ would enter $A$ before the other root variable $z_{4}$ and would get cleared earlier. We will improve this in the manuscript, thanks to your feedback.\n\n**Q9: Mention “up to transformations” early on and the definition of siblings.**\n\nWonderful suggestions -- this will definitely help us improve the readability! We will mention \"up to invertible transformations\" in the introduction and abstract, and we will add a footnote for the objective to indicate that assumptions such as Assumption 3.3 (i) are necessary. We will include the definition of siblings right before introducing Condition 2.3.\n\n**Q10: Equivalence detection.**\n\nYou are totally right! We use mutual predictions to detect the equivalence between two estimated variables. In our implementation, this is done by kernel regression (line 314).\n\n**Q11: Algorithm line 9.**\n\nThank you for pointing this out! We will correct it as you suggested.\n\n**Q12: ``Line 352, …, does not match…’’.**\n\nThank you! We will replace \"closely\" with \"remotely\".\n\n**Q13: Typos.**\n\nWe are grateful for all the listed typos, and we will correct them in the paper. Thank you for your time and effort!\n\n**References:**\n\n[A] On the Identifiability of Nonlinear ICA: Sparsity and Beyond. Zheng et al.\n\n[B] Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA. Lachapelle et al.\n\nPlease let us know if you have further concerns, and please consider raising the score if we have cleared existing concerns  – thank you so much!\n"}, {"review_id": "SUzRzXEOFx", "reviewer": "Reviewer_gbV3", "summary": "This paper addresses the problem of identifying latent variables and causal structures from observational data in the context of nonlinear latent hierarchical causal models. Such models are common in real-world applications involving biological, medical, and unstructured data such as images and languages. The paper presents novel identifiability guarantees and an estimation procedure for both the causal structure and latent variables in these models, under mild assumptions on causal structures and structural functions.\n\nCompared to the previous methods, the main contributions are\n\n1. The author propose the basis model with theoretical guarantees (Theorem 3.2) as a foundation for constructing the identifiability for general nonlinear hierarchical causal model.\n\n2. They show structure identification guarantees for general latent hierarchical models admitting continuous multi-dimensional variables, general nonlinear structural functions, and general graph structures, which go beyond the limitations of previous works that assume linear functions or discrete variables.\n\n3. An estimation method is presented that can asymptotically identify the causal structure and latent variables for nonlinear latent hierarchical models. The authors validate their method on synthetic and real-world datasets.\n", "questions": "The questions are mentioned in the weakness section.", "limitations": "The author included a discussion on the limitations regarding the computational complexity. But I think more statics like wall clock time and detailed discussion on the complexity should be added. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Originality\nThe paper presents several original contributions in the context of causal discovery for nonlinear latent hierarchical models. The authors develop a novel identifiability theory (Theorem 3.2) for basis model, which serves as a fundamental criterion for locating and identifying latent variables in general nonlinear hierarchical causal models. This theory relax some of the limitations compared to previous works. Moreover, the paper provides the estimation procedure for general hierarchical structures as well as latent variables up to a invertible mapping. \n\n## Clarity\nThe paper is logically structured and well-written. I especially appreciated the intuitive explanation behind each assumptions and conditions, making it accessible to general audience. The intuitive explanation of the algorithm is particularly helpful and allowing the reader to capture the main ideas behind the estimation procedure.\n\n## Significance\nThe paper addresses the causal discovery problem in the context of nonlinear latent hierarchical models.\nThe proposed identifiability guarantees and estimation procedure relax some of the limitations compared to the previous work, which itself is a great contribution. This paper is likely to have a notable impact on the field of causal discovery and inspire further research in this area.\n\n\n", "weaknesses": "This paper is generally well-written, however, I still have some questions:\n\n1. Scalability: The author admits the computational limitations. I still think the discussion on the computational complexity should added. Also it is very helpful to report the wall clock comparison to the other baselines regarding the high dimensional problem. \n\n2. Joint invertible mapping: One of the basic assumptions is that the latent and observational variables are jointly invertible. I wonder if this is not satisfied, you mentioned that the identifiability cannot be established. Could you point out the relevant references? Also, in the experiments, why you use MLP as the function $g$? In general MLP is not invertible, for example, we can construct an MLP that map the $\\mathbb{R}^n$ to a fixed point. In this case, you you cannot recover the original latent information. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your encouraging comments and valuable suggestions! We will include your feedback and the discussions in our revision. Below are our responses to individual questions.\n\n**Q1: Computation complexity and wall-clock times.**\n\nThank you for the wonderful suggestion – this will help future readers better understand the complexity aspect! We will include the following discussion on computational complexity in our work: \n\n\"The dominant complexity term in our proposed algorithm is the deep learning training which is of a complexity $O( R * n )$ where $R$ is the number of levels in the hierarchical model, and n is the number of observed variables. In contrast, recent works on linear hierarchical models provide algorithms with complexity $O( R * (1+n)^{p+1} )$ [15] and $O(R * n!)$ [35] and $p$ is the largest number of variables that share children. Our algorithm achieves the lowest complexity in terms of the algorithmic complexity.  As we allow for multi-dimensional variables, the dimensionality of each variable also contributes to the overall wall-clock time through the deep learning model dimension, which is a hyperparameter for the experimenter. This cost can grow significantly for high-dimensional datasets like ImageNet.\"\n\nIn the attached PDF file, we report detailed wall-clock times for all experiments. We note that the wall-clock time can vary significantly depending on the CPU availability, as we run kernel regression on CPUs for $R^{2}$ score computation. Most of our experiments were launched on a group server and experienced contending CPU usage to certain extents. \n\nOur basis model estimation takes roughly the same time as the baseline method in Table 1.\nAs no existing causal discovery approaches are designed for nonlinear hierarchical models with multi-dimensional variables (to the best of our knowledge), we cannot find a baseline to compare with on this aspect.\n\n**Q2: Joint invertible mapping and MLPs.**\n\nGreat questions! We’d like to note that, to the best of our knowledge, existing latent variable identification literature universally adopts the invertibility (or, more broadly, injectiveness) assumption. Please see [18,19,21,24,36,37].\n\nThe fundamental issue is as follows. The observed variables are generated as a function of latent variables, i.e., $ x := g(z_{1}, ..., z_{i}, ..., z_{n})$ where $x$ is the observed variable, $z_{i}$ is the latent variable, and $g$ is the function. \nIdentification results typically entail that we recover/identify each latent variable $z_{i}$ from the observed variable $x$, i.e., $z_{i} = f_{ z_{i} } (x)$ via a learned function $ f_{z_{i}} $.\nTo express each $z_{i}$ as a function of $x$, it is necessary that $x$ preserves all original information of $z_{1}$, ..., $z_{n}$, which is equivalent to the mapping $g$ is invertible (injective).\n\nRegarding MLPs, we fully agree that general MLPs can be non-invertible. \nIn our experiments, we follow the implementation of prior work [18,19,21] to adopt leaky-ReLU and dimension-preserving MLP layer weights with large condition numbers to facilitate invertibility. We will expand on this in our paper to make it clearer -- thank you so much for the suggestion!\n\n\n\nPlease let us know if you have further concerns – thank you!\n"}, {"review_id": "rClE7ll6LT", "reviewer": "Reviewer_mbLW", "summary": "The goal of this paper is to identify the hierarchical graph structure and latent variables for general nonlinear latent hierarchical causal models. The paper reduce the problem to identification of the so-called basis model and proves the connection between latent hierarchical model and basis model. ", "questions": "1. In the definition of assumption 3.2 ii, is T just any matrix with the right support? \n2. Assumption 3.3 i, are the functions supposed to be invertible to match assumption 3.1?\n3. In general, it’s hard to figure out how necessary are the conditions in Assumption 3.3?", "limitations": "The limitations are addressed in section 5. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The theoretical results seems solid and is backed by experiment on both synthetic and real world datasets. ", "weaknesses": "My main concerns with the paper are that it relies on strong assumptions but identifiability results are relatively weak:\n\n    1. Faithfulness assumption is a rather strong assumption. \n    2. Learning latent variables up to invertible transformations is quite weak. A stronger identifiability result in the literature usually learns latent variables up to affine transformation or permutation. But I am not positive if that is possible with this problem setting. \n    3. The pure child assumption is also a strong assumption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your careful assessment and valuable feedback! Below, we respond to your concerns raised in Weaknesses and Questions.\n\n**Q1: Strengths of the assumptions.**\n\nThank you for the feedback! We admit that our approach relies on a number of assumptions; this is because our approach aims to learn representations from a single distribution. This problem is inherently challenging. Without assumptions that are generally sensible, it is generally impossible to achieve nontrivial results.\n\nAs one such essential assumption, faithfulness is adopted in all causal discovery literature (unless further assumptions are introduced) to eliminate cases where the data distribution does not faithfully describe the causal graph. It is a reasonable assumption for many real-world applications -- faithfulness attributes the statistical independence to the graph structure rather than unlikely coincidence, as articulated in Section 3.5.2 in [30] and [C].\nSimilarly, the pure child assumption eliminates certain unidentifiable cases where two latent variables share the same set of children. For instance, $z_{1} \\to X$ and $z_{2} \\to X$, then $z_{1}$ and $z_{2}$ cannot be identified without further assumptions. The pure children assumption has been widely adopted in causal discovery literature [4,9,15,25,35], often more stringent than ours (e.g., tree structures).\n\n**Q2: Strength of the identifiability.**\n\nRegarding identifiability, we are concerned bout representation vectors corresponding to each individual variable. As far as we know, one has to assume multiple distributions or other additional assumptions to obtain stronger identifiability, whereas we only assume a single distribution. From this perspective, our assumption is weaker since the result is applicable even with a single distribution.\n\n**Q3: Definition of $T$.**\n\nGreat question! $T$ is a fixed matrix sharing the support of the matrix-valued function $T(c)$ and satisfying Assumption 3.1 (ii). This assumption is also adopted in [B]. We will make this point explicit in the revision.\n\n**Q4: Assumption 3.3 (i) and invertibility.**\n\nGood question! Assumption 3.3 (i) requires that each latent variable $z$ and exogenous variable $\\epsilon$ can be expressed as functions of all observed variables $X$, which is essentially an invertible assumption on the generating process from $(z, \\epsilon)$ to $X$.\n\n\n**References:**\n\n[A] Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA. Lachapelle et al.\n\n[B] On the Identifiability of Nonlinear ICA: Sparsity and Beyond. Zheng et al.\n\n[C] Replacing Causal Faithfulness with Algorithmic Independence of Conditionals. Lemeire and Janzing.\n\n\nPlease let us know if you have any further concerns, and please consider raising the score if existing concerns are addressed -- thank you very much!"}, {"review_id": "KLO0KN3aZe", "reviewer": "Reviewer_jcJm", "summary": "The paper introduces a class of latent DAG models which allow for hierarchical structure between latent variables. The class of models allows for more general structure than previously considered classes (e.g., tree-structured latent models). The model also allows for general nonlinear relationships between variables. The authors prove that, under additional conditions on the Jacobians of the causal mechanisms, the latent variables in these models are identifiable up to a component-wise invertible transformation. They conclude with experiments showing that the latent variables are recovered to an acceptably high level of accuracy.  ", "questions": "Please provide detailed responses to the points raised in \"Weaknesses\". My initial opinion is that the paper is not ready for publication due to the combination of (1) the re-writing required for acceptable clarity and (2) the insufficiency of the experimental results.", "limitations": "Limitations are adequately addressed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**Significance**: The allowance for non-linear relationships and the generality of the latent structure are substantive improvements over previous works.", "weaknesses": "## Related work\nThe authors mostly do a solid job reviewing related works, with an additional section in the appendix dedicated to such a review. However, they conspicuously leave out a strongly related line of recent work which identifies latent causal graphs from interventional data, e.g. [1,2,3]. I think the comparison with this line of works is important context: there is a tradeoff between structural assumptions and the assumption that interventional data is available. The approach to identifiability which is used in this paper is not the only viable approach, and might be a worse fit for some applications than the intervention-based approach.\n\n[1] Ahuja, K., Wang, Y., Mahajan, D., & Bengio, Y. (2022). Interventional Causal Representation Learning.\n[2] Squires, C., Seigal, A., Bhate, S. S., & Uhler, C. (2023). Linear Causal Disentanglement via Interventions.\n[3] Varici, B., Acarturk, E., Shanmugam, K., Kumar, A., & Tajer, A. (2023). Score-based Causal Representation Learning with Interventions.\n\n## Clarity\nThe most significant weakness of this paper is in terms of clarity; in my opinion, the paper needs substantial re-writing efforts to improve clarity. Here are some of the points which need to be clarified, from most important to least:\n\n1. **Deterministic functions vs. conditional probabilities:** Is each variable a deterministic function of its parent, or is there randomness? It seems from Assumption 3.1(ii) and Assumption 3.3(i), that the variables need to be deterministically related. However, deterministic relations introduce problems for faithfulness, which is assumed in Definition 2.1(iii). Thus, it isn't clear if the assumptions for the theorems are even mutually consistent, which is a **major** problem for this work.\n2. **Algorithm 1:** Several new ideas are introduced all at once in the description of Algorithm 1. The paper would benefit from a more gradual development of these ideas. For example, Stage 3 of the algorithm, which discusses detecting and merging super-variables, is necessary due to issues which had not been introduced. For any of the steps which are essential to include in the main paper, there should be an accompanying example that demonstrates the necessity of the step. The authors should reflect on which steps are essential for intuitively understanding the algorithm, and should consider moving other steps to the appendix.\n3. **Other:** On line 215, the authors say that Theorem 3.4 can be applied to models with arbitrary latent structure, but this seems to contradict that the Theorem requires Condition 2.3 - what are the authors trying to say here?\n\n## Experimental Results\nSection 4 has a couple of major weaknesses. \n\nFirst, the description of the experiments is not sufficiently detailed (unless I have missed some descriptions, in which case the paper should be re-organized so that the relevant details are easier to find). For example, when performing hierarchical model identification via Algorithm 1, there are several steps where one must test whether one variable can \"perfectly predict\" another variable. How is this converted into a test when running the algorithm in a finite-sample regime? As another example, how many instance (random seeds) are the synthetic results averaged over? This detail should be in the figure caption. \n\nSecond, I don't find that the experiments are comprehensive enough. The experiments should, at the least, evaluate the performance of the method across a range of sample sizes. This is important for demonstrating that the algorithm is consistent: despite the identifiability guarantees, the algorithm may fail to be consistent since it relies on perfect optimization of an autoencoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We respond to your concerns as follows and will include your suggestions as indicated.\n\n**Q1: Related work on interventional-based causal identification.**\n\nThank you for the suggestion! We agree that there are two possible ways to learn causal representation: utilizing specific graph conditions as in our paper or assuming and leveraging interventional data for which you provide valuable references. We will include the following in our related work.\n``We note that our work focuses on identifying latent causal models from observational data and structure conditions. Another important line of work [A,B,C,D,E,F,G] utilizes interventional data for this purpose. Specifically, these works leverage multiple data distributions generated by one causal model under distinct interventions, which are accessible in applications like biological experimentations. The accessibility of interventional data can allow for relaxed structure conditions. Hence, one should consider this tradeoff when faced with causal identification problems.’’\n\n**Q2: Deterministic functions vs. conditional probabilities.**\n\nThank you for the question. We are wondering if there is a misunderstanding of the structure causal model (SCM) and our assumption, and let us now provide clarification.\nThe SCMs we use in Equation 1 are standard ones with exogenous noise, i.e.,  the mappings from parents $\\text{Pa}(x_{i})$ to the children $x_{i}$ are not deterministic due to exogenous variables $\\epsilon_{x_{i}}$, although $g_{x_{i}}$ itself is a deterministic mapping from $ (\\text{Pa}(x_{i}), \\epsilon_{x_{i}}) $ to $ x_{i} $.\nSimilarly, $g$ in Assumption 3.1 (ii) is the mapping from all the input variables $(z, s_{1}, s_{2})$ to $(v_{1}, v_{2})$, i.e., $(v_{1}, v_{2}) := g( z, s_{1}, s_{2} ) $, where exogenous variables $\\epsilon$ can be considered contained in $s_1$ and $s_2$. Thus, the mapping from $z$ to $(v_{1}, v_{2})$ is random, although $g$ itself is a deterministic function.\n\nRegarding Assumption 3.3 (i), consider a simple causal model $ x := g(z, \\epsilon) $ where $g$ is an invertible function and $x$ is the child of $z$. Although we can express $ z $ as a function of $ x $, i.e., the first $d_{z}$ dimensions of the inverse function \n $g^{-1} ( x ) $ as in Assumption 3.3 (i),  the causal module $ p ( x | z ) $ is non-degenerate due to the randomness in $\\epsilon$. The suggested work [A] assumes a similar generating process, where the latent variables can be recovered from the observed variables.\n\n**Q3: Algorithm 1 development.**\n\nThank you for the constructive suggestion. We will add more illustrative examples to demonstrate each step's necessity as you suggested. For instance, we will append the following sentence to \"...the concatenation $(z_{2}, z_{3})$\" in line 248. \n\"Leaving this super-variable untouched will be problematic, as we would generate a false causal structure $\\tilde{z} \\to z_{4}$ where $\\tilde{z}$ is the estimated super-variable $(z_{2}, z_{3})$, rather than recognizing $z_{4}$ is the child of two identified variables $z_{2}$ and $z_{3}$.\"\n\n**Q4: Line 215 contradicts Condition 2.3.**\n\nWe note that the remark in line 215 is made on measurement models, as specified in this line and the paragraph title in line 213. \nFor measurement models (which means that each latent variable has enough measured children, see [D]), our theory can handle arbitrary structures among latent variables as long as each latent variable has enough pure children, which are used to identify latent variables with our approach.\n\n**Q5: Experiment details.**\n\nThank you for detailed feedback. We evaluate the $R^{2}$ score with kernel regression (line 314) over 8192 samples within each estimated variable pair. As shown in Table 2 and Figure 6.b, the matched pairs give significantly higher scores than the unmatched ones. For instance, in the first row of Table 2, the estimated latent variable with $x_{1}$ as $v_{1}$ and that with $x_{2}$ as $v_{1}$ both correspond to $z_{2}$ in Figure 4 b and thus archives $0.85$, which is significantly higher than the other scores (around $0.55$) in that row. This gap allows us to set a threshold for each experiment to distinguish the matched estimated variables (e.g., a threshold of $0.6$ suffices for the experiment in Table 2).\nWe will include detailed steps in the revised version.\n\nWe repeated each experiment for at least 3 random seeds, as written in line 315 of the experiment setup. We will further include this in the caption to make it more visible.\n\n**Q6: Experiments not comprehensive enough.**\n\nThank you for the suggestion. We have extended our experiments to make the results more informative. For instance, as you suggested, we vary the sample size for the basis model experiments where $d_{z}=d_{s_{1}}=d_{s_{2}}=2$. The plot is included in the PDF file. Our algorithm can achieve decent identification scores when the sample number is larger than 5000. The performance peaks with little variance when the sample size is over 10000, which suggests that the approach can enjoy asymptotic performances.\n\n\n**References:**\n\n[A] Interventional Causal Representation Learning. Ahuja et al.\n\n[B] Linear Causal Disentanglement via Interventions. Squires et al.\n\n[C] Score-based Causal Representation Learning with Interventions. Varici et al.\n\n[D] Learning nonparametric latent causal graphs with unknown interventions. Jiang and Aragam.\n\n[E] Causal Component Analysis. Liang et al.\n\n[F] Learning linear causal representations from interventions under general nonlinear mixing. Buchholz et al.\n\n[G] Identifiability Guarantees for Causal Disentanglement from Soft Interventions. Zhang et al.\n\n[H] Causality 2n edition. Pearl.\n\nPlease let us know if you have further comments – thank you so much!\n"}]

Author rebuttal:
We thank all reviewers for their valuable feedback and dedicated time! We are encouraged that they find our theoretical contribution significant/substantive (wEjX, gbV3, mbLW, jcJm) and well-supported by experimental results (wEjX, mbLW). We address the individual comments in separate responses and will incorporate the reviewers’ suggestions in our revision.

Meta-review: All reviewers except one (jcJm) argued for accepting the paper. For this reviewer their main conerns were on the related work, paper clarity, and experiments. Their recommendations were to (a) include a discussion on identifying latent causal graphs from interventional data, (b) clarify whether deterministic or random functions are assumed in the graph, (c) more gradually introduce algorithm 1, (d) clarify if there is a contradiction between theorem 4.3 and condition 2.3, (e) improve description of experiments, (f) add additional experiments where sample size is varied. The authors responded convincingly to each of these points. If the reviewer had continued to engage in the discussion they would have likely voted to accept (as they mention in a response to the authors). Further, the authors did a stellar job responding to the concerns of other reviewers. For these reasons, I vote to accept. Authors: you’ve already made improvements to respond to reviewer changes, if you could double check their comments for any recommendation you may have missed on accident that would be great! The paper will make a nice contribution to the conference!
Decision: Accept (poster)