Title: On the Parameter Identifiability of Partially Observed Linear Causal Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Linear Causal Model

Parameter Identification
Abstract: Linear causal models are important tools for modeling causal dependencies and yet in practice, only a subset of the variables can be observed. In this paper,  we examine the parameter identifiability of these models by investigating whether the edge coefficients can be recovered given the causal structure and partially observed data. Our setting is more general than that of prior research—we allow all variables, including both observed and latent ones, to be flexibly related, and we consider the coefficients of all edges, whereas most existing works focus only on the edges between observed variables. Theoretically, we identify three types of indeterminacy for the parameters in partially observed linear causal models. We then provide graphical conditions that are sufficient for all parameters to be identifiable and show that some of them are provably necessary. Methodologically, we propose a novel likelihood-based parameter estimation method that addresses the variance indeterminacy of latent variables in a specific way and can asymptotically recover the underlying parameters up to trivial indeterminacy. Empirical studies on both synthetic and real-world datasets validate our identifiability theory and the effectiveness of the proposed method in the finite-sample regime.

On the Parameter Identifiability of Partially Observed Linear Causal Models

Abstract

Linear causal models are important tools for modeling causal dependencies and yet in practice, only a subset of the variables can be observed. In this paper, we examine the parameter identifiability of these models by investigating whether the edge coefficients can be recovered given the causal structure and partially observed data. Our setting is more general than that of prior research-we allow all variables, including both observed and latent ones, to be flexibly related, and we consider the coefficients of all edges, whereas most existing works focus only on the edges between observed variables. Theoretically, we identify three types of indeterminacy for the parameters in partially observed linear causal models. We then provide graphical conditions that are sufficient for all parameters to be identifiable and show that some of them are provably necessary. Methodologically, we propose a novel likelihoodbased parameter estimation method that addresses the variance indeterminacy in a specific way and can asymptotically recover the underlying parameters up to trivial indeterminacy. Empirical studies on both synthetic and real-world datasets validate our identifiability theory and the effectiveness of the proposed method in the finitesample regime. Code: https://github.com/dongxinshuai/scm-identify .

1 Introduction and Related Work

Causal models, which serve as a fundamental tool to capture causal relations among random variables, have achieved great success in many fields [49, 39, 40, 44]. A fundamental problem in the field is how and to what extent can we identify the underlying causal model given observational data. When all variables are observed, the problem has been well studied: the underlying structure can be identified up to the Markov equivalence class, e.g., by the PC [49] or GES [13] algorithm; when the structure is given, the causal coefficient (direct causal effect) between two variables can also be identified [8, 39].

However, in real-world systems, the variables of interest may only be partially observed. Thus, considerable efforts have been dedicated to identification of causal models in the presence of latent variables. One line of research focuses on structure learning given partially observed variables. Notable approaches include FCI and its variants [49, 38, 14, 2], as well as ICA-based [23, 43], tetrad-based [48, 28], high-order moments-based [46, 11, 58, 1, 12], and rank constraint-based [48, 24, 18] methods.

In this paper, we focus on the the identification of parameters of a partially observed model. Specifically, given the causal structure of and observational data from a partially observed causal model, we are interested in identifying all the parameters, and thus the underlying causal model can be fully specified. To identify the parameters, a classical way is to project the directed acyclic graph (DAG) with latent variables to an acyclic directed mixed graph (ADMG) or partially ancestral graph [42], without explicitly modeling the latent confounders. Based on ADMG, graphical criteria such as halftrek [20], G-criterion [9], and some further developments [51, 29] have been proposed to establish the parameter identifiability. Another way is to leverage do-calculus, proxy variables, and instrumental

3

variables [47, 39, 25] to identify the direct causal effect, which corresponds to the edge coefficient in linear causal models. For a more detailed discussion of related work, please refer to Appendix D.

Despite the effectiveness of current methods for parameter identification, however, they have two main drawbacks: they require all the variables to be connected in specific ways, and only focus on identifying the edge coefficients between observed variables. To this end, in this paper we propose a novel framework that considers a more general setting for parameter identification. To be specific, we allow all variables, including both observed and latent ones, to be flexibly related, and we aim to recover the edge coefficients among all variables, even including those from a latent variable to another latent variable or another observed variable, which previous methods cannot handle. We summarize our contributions as follows.

To the best of our knowledge, we are the first to consider parameter identifiability of partially observed causal model in the most general scenario-all variables, including both observed and latent ones, are allowed be flexibly related, and edge coefficients between any pair of variables are concerned. In contrast, most existing works consider only the edges between observed variables.

Theoretically, we identify three types of parameter indeterminacy in partially observed linear causal models. We then provide graphical conditions that are sufficient for all parameters to be identifiable and show that some of them are provably necessary. These necessary conditions also offer insights into scenarios where the parameters are guaranteed to be non-identifiable.

Methodologically, we propose a novel likelihood-based parameter estimation method, which parameterizes population covariance in specific ways to address variance indeterminacy. Our empirical studies on both synthetic and real-world data validate the effectiveness of our method in the finite-sample regime, even under certain misspecification of the underlying causal model.

2 Preliminaries

2.1 Problem Setting

In this work, we focus on partially observed linear causal models, defined as follows.

Definition 1 (Partially Observed Linear Causal Models) . Let G := ( V G , E G ) be a DAG. Each variable V i ∈ V G follows a linear structural equation model V i = ∑ V j ∈ Pa G ( V i ) f j,i V j + ϵ V i , where V G := L G ∪ X G = { L i } i m =1 ∪ { X i } m + n i = m +1 contains m latent variables and n observed variables. Pa G ( V i ) denotes the parent set of V i , f j,i denotes the edge coefficient from V j to V i , and ϵ V i represents the Gaussian noise term of V i .

̸

We drop the subscript G in L G and X G when the context is clear. We use V , V , and V to denote a random variable, a set of variables, and a set of sets of variables, respectively. In Definition 1, the relations between variables can also be written in the matrix form as V G = F T V G + ϵ V G , where F = ( f j,i ) i,j ∈ [ m + n ] is the weighted adjacency matrix. Here, f j,i = 0 if and only if V j is a parent of V i in G . We also write

where Ω is the diagonal covariance matrix of ϵ V G .

Our objective is to identify F , the causal edge coefficients of the model, given observational data and the causal structure G . Denote by Σ L and Σ X the population covariance matrix of latent variables L and observed variables X , respectively; their precise formulations are provided in Proposition 1. We also denote by σ i,j the ( i, j ) -th entry of Σ X . In this work, we assume that the noise terms of latent variables, ϵ L , have unit variance, i.e., Ω ϵ L = I , which will be justified later in Section 3.1. Note that variables are partially observed and thus we only have access to i.i.d. samples of observed variables X . As variables are jointly Gaussian, the observations can asymptotically be summarized as the population covariance matrix Σ X . In other words, we aim to identify F and Ω given Σ X and G . The identification of parameters is important in that, once we identify the parameters, the underlying causal model is fully specified, and thus we can flexibly calculate causal effects, infer interventional distributions, and finally answer counterfactual queries [39]. It is worth noting that, for parameter identification, the structure G is assumed to be known, which is different from the setting of causal discovery where the goal is to identify G from data.

L1

X2

X3

X4

X5

X6

L1

X2

X3

X4

X5

X6

X2

Bidirected edge

X3

X4

X5

X6

2.2 Framework Comparison

Without latent variables, it has been shown all parameters are identifiable [8]. However, the problem becomes very challenging when latent variables exist. There are two lines of research. One focuses on the use of do-calculus, proxy variables, and instrumental variables to identify direct causal effects among observed variables [47, 39, 25] (in linear models the direct causal effect is captured by the edge coefficient). Another line addresses latent confounders by projecting a DAG with latent variables into an ADMG, where the confounding effects of latent variables are simplified and represented by correlation among noise terms [20, 9, 51, 29]. An example is in Figure 1, where (a) is the original graph and (c) is the projected ADMG whose bidirected edges correspond to correlated noise terms.

Compared to the two previous lines of thought, our framework has two advantages. To begin with, we additionally considers the identifiability of coefficients of edges that involve latent variables. For example, in Figure 1, we aim to identify all the coefficients including the one from L 1 to X 3 , i.e., f 1 , 3 . In contrast, the proxy variable framework and the latent projection framework identify only the coefficients among observed variables: the proxy variable framework focuses only on the direct causal effect from one observed variable to another observed variable, while the latent projection framework transforms all latent variables into bidirected edges and thus can never identify the coefficient of the edge that has a latent variable as the head or tail.

Furthermore, the projection framework deals with latent variables in a rather brute-force way: dense latent confounding effects among observed variables may be caused by only a small number of latent variables, but that information is lost during projection. For example, in Figure 1, (a) and (b) share the same ADMG after projection, i.e., (c). However, as we will show later, parameters in (a) can be identified, while in (b) the parameters cannot. If we only consider the ADMG in (c), then we can never capture this nuance and thus cannot identify the coefficients that we might be able to.

3 Identifiability Theory

3.1 Definition of Parameter Identifiability and Indeterminacy

We follow the notion of generic identifiability and define parameter identifiability as follows. Definition 2 (Identifiability of Parameters of Partially Observed Linear Causal Models) . Let θ = ( F, Ω) ∈ Θ . We say that θ is generically identifiable, if the mapping ϕ ( θ ) = Σ X is injective, for almost all θ ∈ Θ with respect to the Lebesgue measure.

Definition 2 indicates if parameter θ is identifiable, then there does not exist θ ′ ∈ Θ that entails the same observations as those of θ . As in the typical literature of parameter identification, we consider generic identifiability to rule out some rare cases where the parameters for that structure is generally identifiable, but with some specific parameterization, the parameters cannot be identified. This is similar to faithfulness in causal discovery [49] and we will provide an example in Example 1. We next introduce three important indeterminacies about parameter identification when latent variables exist.

Theorem 1 (Indeterminacy of Scaling of Ω ϵ L ) . Consider a model that follows Def. 1 with number of latent variables m ≥ 1 and θ = ( F LL , F LX , F XL , F XX , Ω ϵ L , Ω ϵ X ) . Let Λ be any invertible diagonal matrix, and ˜ θ = ( ˜ F LL , ˜ F LX , ˜ F XL , ˜ F XX , ˜ Ω ϵ L , ˜ Ω ϵ X ) , where

Then, ˜ θ and θ entail the same observations, i.e., ˜ Σ X = Σ X . Furthermore, we have ˜ Σ L = ΛΣ L Λ .

(c) G 3 . G 2 's structure is not identifiable due to the existence of G 3 .

Figure 2: Illustrative examples to show that the graphical condition for structure-identifiability and parameter-identifiability could be very different.

A similar theoretical result is provided in [4], and yet our setting is much more general and takes that of [4] as a special case: in our setting, all variables including latent and observed ones can be arbitrarily related while in [4] latent variables cannot be the effect of observed variables.

̸

Remark 1 (Implication of Theorem 1) . A key implication of Theorem 1 is that, without further assumption, the edge coefficients involving latent variables, i.e., ( F LL , F LX , F XL ) , can never be identified, as there always exists a diagonal matrix Λ such that ˜ θ and θ entail the same observations but ( ˜ F LL , ˜ F LX , ˜ F XL ) = ( F LL , F LX , F XL ) . Thus, in the rest of this paper, we assume that the noise terms of latent variables, ϵ L , have unit variance, i.e., Ω ϵ L = I . Under this assumption, we have ( ˜ Ω ϵ L ) i,i = Λ 2 i,i (Ω ϵ L ) i,i = 1 , i ∈ [ m ] , which implies Λ i,i = ± 1 . As such, this assumption makes parameter identifiability possible. However, even though we fix the scaling of Ω ϵ L , there still exists indeterminacy about the sign of parameters, captured by Theorem 2.

Theorem 2 (Group Sign Indeterminacy) . Consider a model that follows Def. 1 with number of latent variables m ≥ 1 , θ = ( F LL , F LX , F XL , F XX , Ω ϵ L , Ω ϵ X ) , and Ω ϵ L = I . Let S be a diagonal sign matrix (entries are either 1 or -1 ), and ˜ θ = ( ˜ F LL , ˜ F LX , ˜ F XL , ˜ F XX , ˜ Ω ϵ L , ˜ Ω ϵ X ) , where

Then, ˜ θ and θ entail the same observations, i.e., ˜ Σ X = Σ X , and ( ˜ Σ L ) ii = (Σ L ) ii , ∀ i ∈ [ m ] .

Remark 2 (Remark on Theorem 2) . The indeterminacy described in Theorem 2 is referred to as group sign indeterminacy for the following reason: According to the theorem, flipping the sign of S i,i is equivalent to flipping the signs of all coefficients of edges involving the latent variable L i . This transformation preserves the resulting observations Σ X . In essence, each group consists of coefficients of edges involving a particular latent variable.

Example 1 (Example for Group Sign Indeterminacy and Generic Identifiability) . In Figure 2 (b), given the structure and Σ X , by assuming Ω ϵ L = I , the parameters are generally identifiable up to group sign indeterminacy. Specifically, there exist three equality constraints with three free parameters: f 1 , 2 f 1 , 3 = σ 2 , 3 , f 1 , 2 f 1 , 4 = σ 2 , 4 , and f 1 , 3 f 1 , 4 = σ 3 , 4 . The solutions are: (i) f 1 , 2 = √ σ 2 , 3 σ 2 , 4 σ 3 , 4 , f 1 , 3 = σ 2 , 3 /f 1 , 2 , f 1 , 4 = σ 2 , 4 /f 1 , 2 and (ii) f 1 , 2 = -√ σ 2 , 3 σ 2 , 4 σ 3 , 4 , f 1 , 3 = -σ 2 , 3 /f 1 , 2 , f 1 , 4 = -σ 2 , 4 /f 1 , 2 . The two solutions are different only in terms of group sign. However, if we set f 1 , 2 = 0 , then the parameters are not identifiable (as we will encounter division where the divisor is zero). These rare cases of parameters are of zero Lebesgue measure so we rule out these cases for the definition of identifiability, as in Definition 2.

Intuitively speaking, group sign indeterminacy arises because one may multiply the latent variable L i by -1 and accordingly flip the signs of all edge coefficients involving L i . Note that such an indeterminacy is rather minor for the following reason. (i) In practice, we can always anchor the sign of some edges according to our preference or prior knowledge in order to eliminate the group sign indeterminacy. For example, in Figure 4, if we expect that L2 should be understood as Extraversion instead of non-Exterversion, we can add one additional constraint during our parameter estimation such that the edge coefficient from L2 to E1 ("I am the life of party.") will be positive (as we believe E1 should be positively related to Extraversion). (ii) On the other hand, there are some application scenarios that are not influenced by the group sign indeterminacy, such as causal effect estimations between certain variables. We note that, as the indeterminacy of group sign is rather minor, in the following if the parameters are identifiable only up to group sign indeterminacy, we still say that the parameters are identifiable.

Definition 3 (Orthogonal Transformation Indeterminacy) . Consider a model that follows Def. 1 with number of latent variables m ≥ 1 , θ = ( F LL , F LX , F XL , F XX , Ω ϵ L , Ω ϵ X ) , and Ω ϵ L = I . We say that there exists an orthogonal transformation indeterminacy in the identification of parameters if there exists a non-diagonal orthogonal matrix Q such that ( F LL , F LX , F XL , F XX , Ω ϵ L , Ω ϵ X ) and ( ˜ F LL , ˜ F LX , ˜ F XL , ˜ F XX , ˜ Ω ϵ L , ˜ Ω ϵ X ) share the same support and entail the same observations, where

The orthogonal transformation indeterminacy is the major indeterminacy we consider in the presence of latent variables. Such an indeterminacy also arises in factor analysis [45, 7], which can be viewed as a special case of the data generating procedure considered in Definition 1. Here we only give the definition and will later provide Theorem 4 with an example that captures the scenarios where such indeterminacy exists.

It is worth noting that the graphical condition for structure identifiability and parameter identifiability could be very different. For example, G 1 in Figure 2 (a) is structure-identifiable, and yet the parameters are not identifiable even if the structure is given. In contrast G 2 in Figure 2 (b) is not structure-identifiable, as there exists another structure G 3 in Figure 2 (c) such that G 2 and G 3 can never be differentiated from observational distribution; and yet if G 2 is given, its parameters are identifiable (as in Example 1). Therefore, in this paper, we first consider the cases where the structure can be identified and then study which further conditions are needed for the identifiability of parameters. This will give rise to conditions under which the whole causal model can be fully specified.

3.2 Graphical Condition for Structure Identifiability

To explore the conditions for the whole causal model to be specified, we start with the structure identifiability of partially observed linear causal models. Recent advances have shown that if certain graphical conditions are satisfied [24, 18], even though all variables including latent ones are allowed to be very flexibly related, the causal structure can still be identified. Next, we focus on the conditions by [18], which takes that of [24] as special cases. Roughly speaking, the identifiability of the structure of a partially observed linear causal model is built upon the identifiability of atomic covers, defined as follows (with effective cardinality defined as ||V|| = | ( ∪ V ∈V V ) | and PCh G defined in Appendix B.2).

Definition 4 (Atomic Cover [18]) . Let V ∈ V G be a set of variables, where l out of | V | are latent, and the remaining | V | -l are observed. V is an atomic cover if V is a single observed variable, or if the following conditions hold:

(i) There exists a set of atomic covers C , with ||C|| ≥ l +1 , such that ∪ C ∈C C ⊆ PCh G ( V ) .

(ii) There exists a set of covers N with ||N|| ≥ l +1 , s.t. ( ∪ N ∈N N ) ∩ ( ∪ C ∈C C ) = ∅ , every element in ∪ N ∈N N is a neighbour of every element in V , and V d-separates N and C .

(iii) There does not exist a partition P of V , s.t., all elements in P are atomic covers.

The intuition that we build structure identifiability upon the notion of atomic covers is as follows. When a set of latent variables share the same set of children and neighbors, it is impossible to differentiate these latent variables from each other, and thus we need to consider them together as the minimal identifiable group to build up the identifiability of the whole structure. Such a minimal identifiable group of variables is defined as an atomic cover. Roughly, for a group of variables to be qualified as an atomic cover, it has to have enough children and neighbors. An example is as follows.

Example 2 (Example of Atomic Cover) . Consider the graph in Fig. 3. V = { L 1 , X 6 } is an atomic cover. This is because there exist C = {{ X 9 } , { X 10 }} with ||C|| ≥ l +1 = 2 such that (i) in Def. 4 is satisfied. And there exist N = {{ X 11 } , { X 12 }} (or, N = {{ X 4 } , { X 5 }} ) with ||N|| ≥ l +1 = 2 such that (ii) in Def. 4 is satisfied. We can also find that (iii) in Def. 4 is satisfied. Thus { L 1 , X 6 } is an atomic cover. Another example would be in Figure 8, where { L 1 , L 2 } is an atomic cover.

Condition 1 (Basic Conditions for Structure Identifiability [18]) . G satisfies the basic graphical condition for identifiability, if every latent variable belongs to at least one atomic cover in G and for each atomic cover with latent variables, any of its children is not adjacent to any of its neighbours.

Condition 2 (Condition on Colliders [18]) . In G , if (i) there exists sets of variables V , V 1 , V 2 , and T such that every variable in V is a collider of two atomic covers V 1 , V 2 , and T is a minimal set of variables that d-separates V 1 from V 2 , and (ii) there exists at least one latent variable in V ∪ V 1 ∪ V 2 ∪ T , then we must have | V | + | T | ≥ | V 1 | + | V 2 | .

Example 3 (Example that satisfies Conditions 1 and 2) . Consider Figure 3. All latent variables in the graph belong to at least one atomic cover and thus Condition 1 is satisfied. Plus, Condition 2 is also satisfied. This is because the sets of variables V , V 1 , V 2 , and T that satisfy (i) and (ii) in Condition 2 are V = { X 12 } , V 1 = { L 1 , X 6 } , V 2 = { L 2 } , and T = { X 4 , X 5 } , and we also have | V | + | T | ≥ | V 1 | + | V 2 | . Therefore, the graph in Figure 3 satisfies both Conditions 1 and 2.

The identifiability theory of structure is as follows. For a graph G , if Condition 1 and Condition 2 are satisfied, then asymptotically the structure is identifiable up to the Markov equivalence class (MEC) of O a ( O s ( G )) (definitions of O a ( · ) and O s ( · ) can be found in Appendix B.4). Roughly speaking, the underlying causal structure of G can be identified except that the directions of some edges cannot be determined. Next, we will show that, given any DAG in the identified equivalence class together with Σ X , the parameters of the model are also identifiable, if certain conditions are satisfied.

3.3 Identifiability of Parameters

In this section we show that, given graphical Conditions 1 and 2, the causal coefficients F in Definition 1 are also identifiable, if certain conditions are satisfied.

Theorem 3 (Sufficient Condition for Parameter Identifiability (up to group sign), Based on Structure Identifiability) . Assume that G satisfies Conditions 1 and 2 and thus the structure can be identified up to the MEC of O a ( O s ( G )) . For any DAG in the equivalence class, the parameters are identifiable, if both the following hold:

(i) For any atomic cover V = X ∪ L , | L | ≤ 1 .

(ii) If an atomic cover V = X ∪ L satisfies | L | = 0 and | X | ≥ 1 , then all simple treks (Def. 5) between L and X do not contain any latent variables that are not in L .

̸

Theorem 3 provides a sufficient condition such that the parameters are identifiable. Now, for a better understanding of Theorem 3, we provide an example of it as follows.

̸

Example 4 (Example for Theorem 3) . The graph G in Figure 3 satisfies the conditions for parameter identifiability in Theorem 3. Specifically, condition (i) in Theorem 3, is satisfied as all atomic covers contain no more than one latent variable. Plus, condition (ii) in Theorem 3 is also satisfied, as the atomic cover V = X ∪ L = { L 1 } ∪ { X 6 } satisfies | L | = 0 and | X | ≥ 1 and all simple treks between { L 1 } and { X 6 } contain only observed variables except { L 1 } . Therefore, the parameters are identifiable for the graph in Figure 3.

Next, we discuss under which conditions the parameters are guaranteed to be not identifiable. As discussed in Section 3.1, there are three kinds of indeterminacy. The first one can be solved by assuming unit variance of the noise terms of latent variables while the second one group sign indeterminacy is rather trivial such that we still consider parameters as identifiable even if group sign indeterminacy exists. Therefore, we will focus on the third one, orthogonal transformation indeterminacy, in what follows.

Theorem 4 (Condition for the Existence of Orthogonal Transformation Indeterminacy) . Consider the model in Definition 1. If a set of latent variables L with | L | ≥ 2 , have the same parents and children, then there must exist orthogonal transformation indeterminacy regarding the edge coefficients F . In other words, F can at most be identified up to orthogonal transformation indeterminacy.

Example 5 (Example for Thm. 4) . Consider Fig. 8. The graph satisfies the conditions in Thm. 4 as the parents and children of L 1 and L 2 are exactly the same. Therefore, there must exist orthogonal transformation indeterminacy for the edge coefficients F and thus the parameters are not identifiable.

The Theorem 4 above indicates that, if there exist two latent variables that share the same parents and children, then the edge parameters can at most be identified up to orthogonal transformation. This directly implies a necessary condition for parameter identifiability as follows.

Corollary 1 (General Necessary Condition for Parameter Identifiability) . For parameters to be identifiable, every pair of latent variables has to have at least one different parent or child.

Corollary 1 captures a necessary condition in the general cases such that parameters are identifiable. If we further consider the graphs that are also structure identifiable (as we need to identify the structure first to fully specified the causal model), we further have the following Corollary 2 by considering the notion of atomic covers (the proofs of both corollaries can be found in the Appendix).

Corollary 2 (Necessary Condition about Atomic Covers for Parameter Identifiability) . Assume G satisfies Conditions 1 and 2 and thus the structure can be identified up to the MEC of O a ( O s ( G )) . For any DAG G in the equivalence class, for G 's parameters to be identifiable, every atomic cover must contain no more than one latent variable.

Remark 3 (Necessity of Conditions in Theorem 3) . Condition (i) in Theorem 3 is provably necessary: by Corollary 2, for parameters to be identifiable, one has to assume (i) in Theorem 3.

Establishing a necessary and sufficient condition is always highly non-trivial in various tasks. For example, for the identification of linear non-Gaussian causal structure with latent variables, researchers initially developed sufficient conditions with three pure children in [46], later relaxed to two in [11, 58], before ultimately achieving both necessary and sufficient conditions in [1]. Similarly, for parameter identification, although the condition we proposed is not a necessary and sufficient one, it could serve as a stepping stone towards tighter and ultimately the necessary and sufficient condition for the field.

Below, we also provide a sufficient condition for parameter identifiability that does not rely on structure identifiability in Theorem 5. It is particularly useful when the structure is directly given by some domain experts.

Theorem 5 (Sufficient Condition for Parameter Identifiability (up to group sign) without Requiring Structure Identifiability) . In G , if for every latent variable L there always exist another three distinct variables (which can be latent or observed), such that two of the three are pure children of L and the rest one is a neighbor of L , then the parameters are identifiable.

Identifiability theory often focuses on the asymptotic case, i.e., we assume that we know the structure and the population covariance matrix Σ X . However, in practice, we only have access to i.i.d. data with finite sample size and thus only have the sample covariance matrix. Therefore, in the next section, we will propose a novel method to estimate the parameters in the finite sample cases.

4 Parameter Estimation Method

4.1 Objective

Our goal is to estimate F in Definition 1, given the causal structure G and observational data. The key is to parameterize the population covariance Σ X using θ = ( F, Ω) and then maximize the likelihood of observed sample covariance ˆ Σ X . To make this technically precise, we provide a closed-form expression of Σ X in terms of θ in the following proposition, with a proof given in Appendix A.7.

Proposition 1 (Parameterization of Population Covariance) . Consider the model defined in Def. 1. Let M := ( I -F LL -F LX ( I -F XX ) -1 F XL ) -1 and N := ( ( I -F LL ) F -1 XL ( I -F XX ) -F LX ) -1 . Then, the population covariance matrices of L and X can be formulated as

The formulations of Σ L and Σ X are rather complicated due to the general scenario we considered, i.e., latent variables can be the cause or the effect of latent and observed variables. That is, the submatrices F LL , F LX , F XL and F XX defined in the above proposition can all have nonzero entries. In most existing works, at least one of these submatrices are assumed to be zero. For instance, factor analysis assumes that F LL , F XL and F XX are zero, while [32] assumes that F LL and F XL are zero. Furthermore, Proposition 1 also provides insight into the indeterminacy involved when identifying the parameters, such as the indeterminacy of variance in Theorem 1 and the orthogonal transformation indeterminacy in Theorem 4.

Similar to factor analysis [45, 7, 21], we assume ϵ V are Gaussian and thus X are jointly Gaussian. Thus, the negative log-likelihood of observational data can be formulated as

where K is the number of i.i.d. observations. With the parameterized negative log-likelihood, we estimate the edge coefficients by minimizing the negative log-likelihood, as

where the entries of matrix F that do not correspond to an edge in G are constrained to be zero during the optimization.

Note that in Eq. (4) the constraint that the noise terms of latent variables have unit variance is crucial to deal with the variance indeterminacy defined in Theorem 1. In practice, it is also favorable to use another constraint to address the variance indeterminacy, i.e., the constraint that all the latent variables have unit variance. This leads to an alternative objective as

where the entries of F that do not correspond to an edge in G are also constrained to be zero.

Both objectives in Eqs. (4) and (5) can be employed, and yet using the second one gives rise to edge coefficients that are easier to understand. To be concerete, if we normalize all observed variables to have unit variance, then using Eq. (5) would give rise to ˆ F such that -1 ≤ ˆ F i,j ≤ 1 , ∀ i, j ∈ [ m ] . An example can be found in Figure 4. However, it may not be straightforward to realize the constraint in Eq. (5). To this end, in the next section we introduce a way to parameterize Σ X using F , such that the required constraint in Eq. (5) can be automatically satisfied. Later in Section 5.2, we also empirically compare the performance of using Eq. (4) with that of using Eq. (5).

4.2 Parameterization Trick of Covariance Matrix

In this section, we introduce how trek rules can be employed to parameterize Σ X while the unit variance constraint on latent variables in Eq. (5) can be elegantly satisfied. We start with the definition of trek. For readers who are less familiar with treks, please refer to Appendix B.1 for examples.

Definition 5 (Treks [50]) . In G , a trek from X to Y is an ordered pair of directed paths ( P 1 , P 2 ) where P 1 has a sink X , P 2 has a sink Y , and both P 1 and P 2 have the same source Z , i.e., top ( P 1 , P 2 ) = Z . A Trek is simple if P 1 and P 2 have no intersection except their common source Z .

At this point, we are able to parameterize each entry of Σ X using ( F, { σ ii } n + m i =1 ), instead of ( F, Ω ), by making use of the (simple) trek rule [50], as follows:

where S ( V i , V j ) is the set of all simple treks between V i and V j , and f P is the path monomial along P defined as f P := Π k → l ∈ P f kl .

By this form of parameterization, we can simply set all entries of { σ ii } n + m i =1 as 1 (which is equivalent to requiring all variables to have unit variance), such that the constraint in Eq. (5) can be automatically satisfied. For a better understanding of how to use the simple trek rule for parameterization, we provide an example as follows.

Example 6 (Example for Parameterization using Simple Trek) . In Figure 7 (a), there are four simple treks between X 4 and X 5 , as shown in (b). By the simple trek rule and further assuming that all variables have unit variance, the covariance between X 4 and X 5 , σ 4 , 5 , can be formulated as f 1 , 4 f 1 , 5 + f 3 , 4 f 3 , 5 + f 2 , 1 f 1 , 4 f 2 , 3 f 3 , 5 + f 2 , 3 f 3 , 4 f 2 , 1 f 1 , 5 .

5 Experiments

We validate our identifiability theory and parameter estimation method on synthetic and real-life data.

5.1 Setting and Evaluation Metric

Webegin with our experimental setting of synthetic data. The causal strength f ij is uniformly sampled from [ -2 , 2] and the noise terms are Gaussian with variance uniformly from [1 , 5] . We consider 20 graphs. 10 of them should be parameter-identifiable up to group sign indeterminacy according to our identifiability theory and we refer to them as GS Case (examples in Figure 10 in Appendix). Another 10 should be parameter-identifiable up to group sign and orthogonal transformation indeterminacy and we refer to them as OT Case (examples in Figure 11 in Appendix). On average each graph contains 15 variables, 3 out of them are latent. We consider three different sample sizes: 2k, 5k, and 10k. We use three random seeds to generate the causal model and report the mean performance as well as the std.

(a) MSE up to group sign indeterminacy.

(b) MSE up to orthogonal transformation.

[06]1donot have

a good imagination.

Openness

[E2]1 don't talk a lot.

[E10]1amquiet

aroundstrangers.

-0.67

[E9]1don'tmind being

the center of attention.

[E8]1don'tliketo draw

Extraversion

[03]1havea

+0.74

+

-0.73

[05]thave

excellent ideas.

[010]1am fullof

[E7]1talktoalot of

90+

0.67

+0.59

-0.46

vivid imagination.

+0.53

+0.73

+0.83

ideas.

[E3]1feelcomfortable

around people.

+0.73

+67[E1]amthelifeofthepaty.

[07]1amquickto

understand things.

-0.71

Agreeableness

[08]1usedifficult

interested in

[04]1amnot

+0.54

+0.62

[02]1havedificulty

understandingabstract

[01]ihavea

rich vocabulary.

+0.58

+0.51

-0.54

L6

+0.66

+0.27

+0.23

[E4]1keepin the

background.

[E5]Istart conversations.

people feel at ease.

[A10]1make

+0.73

+0.39

+0.45

[A1]fel litle

concern for others.

-0.42

+0.61

+0.72

[A2] 1 am interested in people.

+0.60

emotions.

[A9]1fee others

out for others.

[A8]1take time

abstract ideas.

ideas.

reflecting on things.

[09]1spend time

-0.09

[A3]tinsult people.

-0.34

L3

-0.69

+0.56

[A7]lamnot really

Interested in others.

-0.13

Conscientiousness

[C5]1get chores

+0.26

+0.21

[N5]1am

easily disturbed.

upset easily.

[N6]1get

+0.10/

0.12

[A4]1sympathize with

+0.77

-0.68

other people's problems.

[A5]1am not interested in

[A6] have a soft heart.

[C8]I shirk my duties.

+0.31

+0.43

others' feeings.

950-

mess of things.

[C7]like order.

[C1]1am always

prepared.

[C4]1makea

[C10]1am

+

+0.61

+0.45

-0.63

-0.52

+0

L5

-0.62

+0.38

→[C3]Ipay attention to details.

+0.59

[C9]Ifollow a schedule.

[C6]1ftenforgetoputthings

-0.27

[N3]1worry

about things.

+0.40

?

[N10]

[N9]1get

irritated easily.

oftenfeel blue

+0.42

+0.49

+0.42

N8]1haved

frequentmood

swings.

0.36[N2]1am relaxed

most of the time.

[N1]1get

stressed out easily.

Neuroticism

exacting in my work.

belongings around.

[C2]1leavemy

back intheirproper place.

+0.79

my mood a lot.

As the optimization in Eq (4) is nonconvex, we will rely on 30 random starts and choose the one with the best likelihood. We report the performance of the proposed method with two different objectives. (i) Parameter Estimator with objective defined in Eq. (4), referred to as Estimator, and (ii) Parameter Estimator with objective defined in Eq. (5) and Trek Rule parameterization trick in Eq (6), referred to as Estimator-TR.

It is worth noting that our setting is very general in that we allow latent variables and observed variables to be causally connected in a very flexible way, and we consider the identification of parameters of edges that can involve both observed and latent variables. Therefore, to the best of our knowledge, no current method can achieve the same goal to serve as the baseline (which also shows the novelty of the proposed method). As such, we mainly focus on comparing our estimation result with the ground truth parameters. We use two MSE-based metrics defined as follows.

MSE up to group sign: suppose the ground truth parameter is F and our estimation is ˆ F . The MSE up to group sign is defined as ∥| F |-| ˆ F |∥ 2 2 ∥ F ∥ 0 , where | · | takes element wise absolute value, ∥ · ∥ 2 denotes the Frobenius norm and ∥ · ∥ 0 denotes the number of nonzero entries of a matrix.

MSE up to orthogonal transformation: the MSE up to orthogonal transformation is defined as

where the optimization is solved by Adam [27] and the orthogonal matrix Q can be directly parameterized in PyTorch.

5.2 Synthetic Data Performance

We report the performance using synthetic data in Tables 1a and 1b, where both our Estimator and Estimator-TR achieve very good identification performance. For example, in the GS scenario with 10k samples, our Estimator achieves 0.0.0012 MSE up to group sign and our Estimator-TR achieves 0.0003 MSE up to group sign. The good performance by Estimator and Estimator-TR not only validates our estimation method, but also empirically verifies our identifiability theory.

5.3 Misspecification Behavior

In this section, we show that the proposed estimation method still performs well, even under model misspecification: violation of normality and violation of linearity.

As for violation of normality, we use uniform noise terms for the underlying model, and thus the distribution is not jointly Gaussian anymore. We aim to see to what extent can the proposed method still recover the correct parameters. The result is shown in Table 2 in the Appendix, which shows even when the normality is violated, we can still estimate the parameters pretty well. The reason lies in that our proposed asymptotic identifiability result holds true, even when we do not assume Gaussianity; as we only make use of the second-order statistics of the distribution, the additive noise in Definition 1 can follow any other continuous distribution.

To simulate the violation of linearity, we employ the leaky ReLU function during the generation process, as V i = LRELU ( ∑ Vj ∈ Pa ( Vi ) f ji V j + ϵ Vi ) , LRELU ( x ) = max( αx, x ) . When α is close to 1, the function is close to a linear one, and when α is close to 0, the model is very nonlinear. The result is shown in Table 3 and we found that our estimation method is quite robust to small violations of linearity. For example, for Estimator-TR in GS case with 10k sample size, if we set α = 0 . 8 , we still get a small MSE of 0.001. Even when α decreases to 0.6, the MSE is around 0.005, which is still small. However, when α is decreased to 0 . 3 , the underlying model is considerably nonlinear, and the MSE increases to 0.027.

5.4 Implementation Details, Runtime Analysis, and Scalability

Our code is based on Python3.7 and PyTorch [37]. Data is standardized and the optimization in Eqs. (4), (5), and (7) are solved by Adam [27], with a learning rate of 0 . 02 . We conduct all the experiments with single Intel(R) Xeon(R) CPU E5-2470. All experiments can be finished within 2 hours. We note that our method is very computationally efficient. First, the computational cost is almost irrelevant to sample size: we only need to calculate the sample covariance matrix once and cache it for further use during the optimization. Plus, our estimation method can handle a large number of variables. For example, the running time of our method are roughly 10 seconds, 2 minutes, and 10 minutes for 20 variables, 50 variables, and 100 variables respectively. For 300 variables, which is a considerably large number for typical experiments considered in causal discovery papers, the estimation can still be finished within around one hour.

It is also worth noting that model misspecifications do not influence the computation cost of our method. We briefly discuss the efficiency of checking whether conditions in Theorem 3 hold, together with what if conditions do not hold in solving real-life problems in Appendices A.8 and A.9.

5.5 Real-World Data Performance

In this section, we employ a famous psychometric dataset -Big Five dataset https: //openpsychometrics.org/ , to validate our method. It consists of 50 indicators and close to 20,000 data points. There are five dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism (O-C-E-A-N). Each is measured with 10 indicators. Data is standardized.We employ the RLCD method [18] to determine the MEC and GIN [58] to decide the remaining directions. Then we employ the proposed Estimator-TR to estimate all the edge coefficients. The structure satisfies the condition in Theorem 5 so we know that the parameters are identifiable.

The estimated edge coefficients are shown in Figure 4. We found that our estimated coefficients are well aligned with existing psychology studies. For example, according to [16, 17], being successful in exploratory endeavors depends on the stability to pursue them. This is illustrated in our result where L5 +0 . 26 - - - - → L6 and L3 +0 . 39 - - - - → L2 indicates that Conscientiousness positively influence openness and Agreeableness positively influences Extraversion. Moreover, it has been shown that people are likely to weigh the outcomes of their actions, thus, their level of Conscientiousness coupled with Neuroticism may prohibit them from engaging in risky behaviors (L5 -0 . 27 - - - - → N10 -0 . 12 - - - - → L3 +0 . 39 - - - - → L2) [54]. Such consistency with current psychometric studies again validates the effectiveness of the proposed method in parameter estimation of real-life systems.

6 Conclusion

In this paper, we characterize indeterminacy of parameter identification and provide conditions for identifiability. Finally, we propose a novel estimation method and validate it by empirical study.

References

Jeffrey Adams, Niels Hansen, and Kun Zhang. Identification of partially observed linear causal models: Graphical conditions for the non-gaussian and heterogeneous cases. Advances in Neural Information Processing Systems , 34:22822-22833, 2021.

Sina Akbari, Ehsan Mokhtarian, AmirEmad Ghassami, and Negar Kiyavash. Recursive causal structure learning in the presence of latent variables and selection bias. In Advances in Neural Information Processing Systems , volume 34, pages 10119-10130, 2021.

Animashree Anandkumar, Daniel Hsu, Adel Javanmard, and Sham Kakade. Learning linear bayesian networks with latent variables. In International Conference on Machine Learning , pages 249-257, 2013.

Ankur Ankan, Inge Wortel, Kenneth Bollen, and Johannes Textor. Combining graphical and algebraic approaches for parameter identification in latent variable structural equation models. In International Conference on Artificial Intelligence and Statistics , pages 7252-7264. PMLR, 2023.

Rina Foygel Barber, Mathias Drton, Nils Sturma, and Luca Weihs. Half-trek criterion for identifiability of latent variable models. The Annals of Statistics , 50(6):3174-3196, 2022.

Elias Bareinboim and Jin Tian. Recovering causal effects from selection bias. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, 2015.

Paul A Bekker and Jos MF ten Berge. Generic global indentification in factor analysis. Linear Algebra and its Applications , 264:255-263, 1997.

Carlos Brito and Judea Pearl. A new identification condition for recursive models with correlated errors. Structural Equation Modeling , 9(4):459-474, 2002.

Carlos Brito and Judea Pearl. Generalized instrumental variables. arXiv preprint arXiv:1301.0560 , 2012.

Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable causal discovery from interventional data. In Advances in Neural Information Processing Systems , 2020.

Ruichu Cai, Feng Xie, Clark Glymour, Zhifeng Hao, and Kun Zhang. Triad constraints for learning causal structure of latent variables. Advances in neural information processing systems , 32, 2019.

Zhengming Chen, Feng Xie, Jie Qiao, Zhifeng Hao, Kun Zhang, and Ruichu Cai. Identification of linear latent variable model with arbitrary distribution. In Proceedings 36th AAAI Conference on Artificial Intelligence (AAAI) , 2022.

David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research , 3(Nov):507-554, 2002.

Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics , pages 294-321, 2012.

Philipp Dettling, Mathias Drton, and Mladen Kolar. On the lasso for graphical continuous lyapunov models. In Causal Learning and Reasoning , pages 514-550. PMLR, 2024.

Colin G. DeYoung. Higher-order factors of the big five in a multi-informant sample. J Pers Soc Psychol , 91(6)(6):1138-1151, 2006.

Colin G. DeYoung. Cybernetic big five theory. J Res Pers , 56:33-56, 2015.

Xinshuai Dong, Biwei Huang, Ignavier Ng, Xiangchen Song, Yujia Zheng, Songyao Jin, Roberto Legaspi, Peter Spirtes, and Kun Zhang. A versatile causal discovery framework to allow causally-related hidden variables. In International Conference on Learning Representation , 2024.

Mathias Drton, Rina Foygel, and Seth Sullivant. Global identifiability of linear structural equation models. The Annals of Statistics , 39(2):865-886, 2011.

Rina Foygel, Jan Draisma, and Mathias Drton. Half-trek criterion for generic identifiability of linear structural equation models. The Annals of Statistics , pages 1682-1713, 2012.

Richard L Gorsuch. Factor analysis: Classic edition . Routledge, 2014.

Roger A. Horn and Charles R. Johnson. Matrix Analysis . Cambridge University Press, 2 edition, 2012.

Patrik O Hoyer, Shohei Shimizu, Antti J Kerminen, and Markus Palviainen. Estimation of causal effects using linear non-gaussian causal models with hidden variables. International Journal of Approximate Reasoning , 49(2):362-378, 2008.

Biwei Huang, Charles Jia Han Low, Feng Xie, Clark Glymour, and Kun Zhang. Latent hierarchical causal structure discovery with rank constraints. arXiv preprint arXiv:2210.01798 , 2022.

Yimin Huang and Marco Valtorta. Pearl's calculus of intervention is complete. arXiv preprint arXiv:1206.6831 , 2012.

Yonghan Jung, Jin Tian, and Elias Bareinboim. Learning causal effects via weighted empirical risk minimization. Advances in neural information processing systems , 33:12697-12709, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Erich Kummerfeld and Joseph Ramsey. Causal clustering for 1-factor measurement models. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1655-1664. ACM, 2016.

Daniel Kumor, Carlos Cinelli, and Elias Bareinboim. Efficient identification in linear structural causal models with auxiliary cutsets. In International Conference on Machine Learning , pages 5501-5510. PMLR, 2020.

Manabu Kuroki and Judea Pearl. Measurement bias and effect restoration in causal inference. Biometrika , 101(2):423-437, 2014.

Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradientbased neural DAG learning. In International Conference on Learning Representations , 2020.

Dennis Leung, Mathias Drton, and Hisayuki Hara. Identifiability of directed gaussian graphical models with one latent source. Electronic Journal of Statistics , 10, 05 2015.

Wang Miao, Zhi Geng, and Eric J Tchetgen Tchetgen. Identifying causal effects with proxy variables of an unmeasured confounder. Biometrika , 105(4):987-993, 2018.

Ignavier Ng, Xinshuai Dong, Haoyue Dai, Biwei Huang, Peter Spirtes, and Kun Zhang. Scorebased causal discovery of latent variable causal models. In Forty-first International Conference on Machine Learning , 2024.

Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. Advances in Neural Information Processing Systems , 33:17943-17954, 2020.

R. M. O'BRIEN. Identification of simple measurement models with multiple latent variables and correlated errors. Sociological Methodology , 24, 1994.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Judea Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, New York, NY, USA, 2000.

Judea Pearl. Causal inference in statistics: An overview. Statistics surveys , 3:96-146, 2009.

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017.

O. Reiersøl. On the identifiability of parameters in thurstone's multiple factor analysis. Psychometrika , 15(2), 1950.

Thomas Richardson and Peter Spirtes. Ancestral graph Markov models. The Annals of Statistics , 30(4):962-1030, 2002.

Saber Salehkaleybar, AmirEmad Ghassami, Negar Kiyavash, and Kun Zhang. Learning linear non-gaussian causal models in the presence of latent variables. Journal of Machine Learning Research , 21(39):1-24, 2020.

Bernhard Schölkopf. Causality for machine learning. arXiv preprint arXiv:1911.10500 , 2019.

Alexander Shapiro. Identifiability of factor analysis: Some results and open problems. Linear Algebra and its Applications , 70:1-7, 1985.

Shohei Shimizu, Patrik O Hoyer, and Aapo Hyvärinen. Estimation of linear non-gaussian acyclic models for latent factors. Neurocomputing , 72(7-9):2024-2027, 2009.

Ilya Shpitser and Judea Pearl. Identification of joint interventional distributions in recursive semimarkovian causal models. In Proceedings of the National Conference on Artificial Intelligence , volume 21, page 1219. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2006.

Ricardo Silva, Richard Scheines, Clark Glymour, Peter Spirtes, and David Maxwell Chickering. Learning the structure of linear latent variable models. Journal of Machine Learning Research , 7(2), 2006.

Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search . MIT press, 2000.

Seth Sullivant, Kelli Talaska, and Jan Draisma. Trek separation for gaussian graphical models. arXiv:0812.1938. , 2010.

Jin Tian. Parameter identification in a class of linear structural equation models. In Twenty-First International Joint Conference on Artificial Intelligence , 2009.

Jin Tian and Judea Pearl. A general identification condition for causal effects. In Aaai/iaai , pages 567-573, 2002.

Sivan Toledo. Locality of reference in lu decomposition with partial pivoting. SIAM Journal on Matrix Analysis and Applications , 18(4):1065-1081, 1997.

Nicholas A. Turiano, Daniel K. Mroczek, Jan Moynihan, and Benjamin P. Chapman. Big 5 personality traits and interleukin-6: evidence for 'healthy neuroticism' in a us population sample. Brain Behav Immun , pages 83-89, 2013.

Gherardo Varando and Niels Richard Hansen. Graphical continuous lyapunov models. In Conference on Uncertainty in Artificial Intelligence , pages 989-998. Pmlr, 2020.

L. L. WEGGE. Local identifiability of the factor analysis and measurement error model parameter. Journal of Econometrics , 70, 1996.

B. Williams. Identification of the linear factor model. Econometric Reviews , 39(1), 2020.

Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in neural information processing systems , 33:14891-14902, 2020.

Xun Zheng. Learning DAGs with Continuous Optimization . PhD thesis, PhD thesis, Carnegie Mellon University, 2020.

A Proofs

A.1 Proof of Theorem 1

Theorem 1 (Indeterminacy of Scaling of Ω ϵ L ) . Consider a model that follows Def. 1 with number of latent variables m ≥ 1 and θ = ( F LL , F LX , F XL , F XX , Ω ϵ L , Ω ϵ X ) . Let Λ be any invertible diagonal matrix, and ˜ θ = ( ˜ F LL , ˜ F LX , ˜ F XL , ˜ F XX , ˜ Ω ϵ L , ˜ Ω ϵ X ) , where

˜ F LL = Λ -1 F LL Λ , ˜ F LX = Λ -1 F LX , ˜ F XL = F XL Λ , ˜ F XX = F XX , ˜ Ω ϵ L = Λ 2 Ω ϵ L , ˜ Ω ϵ X = Ω ϵ X Then, ˜ θ and θ entail the same observations, i.e., ˜ Σ X = Σ X . Furthermore, we have ˜ Σ L = ΛΣ L Λ .

Proof of Theorem 1. Let ( F LL F LX F XL F XX ) = ( A B C D ) and ( ˜ F LL ˜ F LX ˜ F XL ˜ F XX ) = ( ˜ A ˜ B ˜ C ˜ D ) . Let M ˜ ˜

and N be matrices defined as in Proposition 1, and similarly for M and N . We then have

and

By Proposition 1, the latent covariance matrix ˜ Σ L after rescaling of the parameters is given by

This implies that the variance of each latent variable L i is scaled by Λ 2 ii . By Proposition 1, the observed covariance matrix ˜ Σ X after rescaling of the parameters is given by

A.2 Proof of Theorem 2

Theorem 2 (Group Sign Indeterminacy) . Consider a model that follows Def. 1 with number of latent variables m ≥ 1 , θ = ( F LL , F LX , F XL , F XX , Ω ϵ L , Ω ϵ X ) , and Ω ϵ L = I . Let S be a diagonal sign matrix (entries are either 1 or -1 ), and ˜ θ = ( ˜ F LL , ˜ F LX , ˜ F XL , ˜ F XX , ˜ Ω ϵ L , ˜ Ω ϵ X ) , where

L1

L2

X6

X7

L3

L5

8X

Xg

X10

X11

X12

X13

X14

X15

X16

X17

X3

X4

L1

X5

L2

X6

X7

X8

Xg

X10

X11

X12

X13

Proof of Theorem 2. Let ( F LL F LX F XL F XX ) = ( A B C D ) and ( ˜ F LL ˜ F LX ˜ F XL ˜ F XX ) = ( ˜ A ˜ B ˜ C ˜ D ) . Since S is a diagonal sign matrix, we have

Note that S is an invertible diagonal matrix. By Theorem 1, we have ˜ Σ X G = Σ X G and ˜ Σ L G = S Σ L G S , and thus ( ˜ Σ L G ) ii = (Σ L G ) ii , ∀ i ∈ [ m ] .

A.3 Proof of Theorem 3

The structure identifiability part is that if G satisfies Condition 1 and Condition 2, the structure of G can be identified up to the Markov equivalence class of O a ( O s ( G )) , which is by Theorem 12 in [18].

Next we will focus on the proof of parameter identifiability part, i.e., for any DAG in the equivalence class, if (i) and (ii) in Theorem 3 are satisfied, the parameters are identifiable (up to group sign). Without loss of generality, we assume that all variables have unit variance and zero mean. The reason is that if we can show that the parameters are identfiable (up to group sign) under this assumption, then it is straightforward to show that they are also identifiable under the original assumption where Ω ϵ L = I .

Lemma 1. Let X , Y be two set of variables, we have Σ Y | X = x = Σ Y -Σ YX Σ -1 X Σ XY .

̸

Lemma 2. Consider a graph G that satisfies (i) and (ii) in Theorem 3. For an atomic cover V in G with one latent variable, V = { L } ∪ { X i } k i =1 ( k could be zero), if it has an observed pure child C and the coefficients of the edges from V to C , i.e., f L → C , f X1 → C , . . . , f Xk → C , are known, then for any variable A , such that A = C and A is not a descendant of C , σ L , A can be calculated as ( σ A , C -Σ k i =1 σ Xi , A f Xi → C ) /f L → C .

Proof of Lemma 2. By the definition of atomic covers, all variables in V = { L } ∪ { X i } k i =1 are not adjacent. By trek rule and the fact that C is a pure child of V , all treks from C to A go through V , and thus by the trek rule we have σ L , A f L → C +Σ k i =1 σ Xi , A f Xi → C = σ A , C .

Remark: This lemma implies that we can find all edge coefficients of the graph in a bottom-up fashion. Roughly speaking, for a latent variable L that belongs to an atomic cover V , once we identify f L → C , f X1 → C , ..., f Xk → C where C is an observed pure child of V , we can take L as if it is observed. More specific explanations can be found in the following proof.

Theorem 3 (Sufficient Condition for Parameter Identifiability (up to group sign), Based on Structure Identifiability) . Assume that G satisfies Conditions 1 and 2 and thus the structure can be identified up to the MEC of O a ( O s ( G )) . For any DAG in the equivalence class, the parameters are identifiable, if both the following hold:

(i) For any atomic cover V = X ∪ L , | L | ≤ 1 .

(ii) If an atomic cover V = X ∪ L satisfies | L | = 0 and | X | ≥ 1 , then all simple treks (Def. 5) between L and X do not contain any latent variables that are not in L .

̸

Proof of Theorem 3. Consider a graph G that satisfies (i) and (ii) in Theorem 3. We first show that if all the pure children of an atomic cover V are observed, then all the edge coefficients from the atomic cover to its children are identifiable (up to group sign). To this end, we categorize the scenarios into four cases and prove them separately.

(a) V = { X } contains a single observed variable. The proof for this case is trivial as the edge coefficient from X to its pure child C is simply σ X , C .

(b) V = { L } contains a single latent variable. By Condition 1 there must exist C 1 , C 2 , and X N , such that C 1 , C 2 are pure children of V and X N is an observed variable that has a trek to V . Then we have

By these three equations, we can solve f L → C1 and f L → C2 . If V has more than two pure children, we can prove the identifiability similarly in a pairwise fashion.

(c) V = { L } ∪ { X i } k i =1 ( k ≥ 1 ) contains a single latent variable and k observed variables, where V has at least three pure children. We assume that there exist C 1 , C 2 , and C 3 , such that C 1 , C 2 , and C 3 are pure children of V . Let σ L |{ Xi } k i =1 = t . In this case, we have

By these three equations, we can solve f L → C1 , f L → C2 , and f L → C3 , with the only remaining free parameter t . In other words, we have f L → C1 ( t ) , f L → C2 ( t ) , and f L → C3 ( t ) .

Next, we show that ∀ i = 1 , ..., k, j = 1 , ..., 3 , f Xi → Cj can be identified. Specifically, as all simple treks between L and X contain only observed variables except L , and L and X are not directly adjacent, there must exist ˆ X such that ˆ X d-separates L and X . Thus, f Xi → Cj σ Xi | ˆ X ∪ X \ Xi = σ XiCj | ˆ X ∪ X \ Xi , by which f Xi → Cj can be solved.

Now we solve t . The key is that all the edge coefficients along all simple treks between X and L can be identified, by using Lemma 2, with only one free parameter t . Thus, we can make use of simple trek rule to parameterize Σ L , X as a function of t . By Lemma 1, σ L | X = t can also be formulated as a function of Σ L , X , and thus t can be solved. If V has more than three pure children, we just choose all the combinations of any three pure children.

(d) V = { L } ∪ { X i } k i =1 ( k ≥ 1 ) contains a single latent variable and k observed variables, where V has two pure children. By Condition 1, there must exist C 1 , C 2 as the pure children of V . If there exists one additional pure child, then it is the same as (c). Plus, if there only exist these two pure children, there must exist V N such that V N is a neighbor of V . If V N is observed, let X N = V N , otherwise we recursively take X N as the observed pure children of V N .

Let σ L |{ Xi } k i =1 = t . In this case, we have

Similar to (c), by solving the above, we have f L → C1 ( t ) and f L → C2 ( t ) .

Next, we show that ∀ i = 1 , ..., k, j = 1 , ..., 2 , f Xi → Cj can be identified. Specifically, as all simple treks between L and X contain only observed variables except L , and L and X are not directly adjacent, there must exist ˆ X such that ˆ X d-separates L and X . Thus, f Xi → Cj σ Xi | ˆ X ∪ X \ Xi = σ XiCj | ˆ X ∪ X \ Xi , by which f Xi → Cj can be solved.

Now we solve t . The key is that all the edge coefficients along all simple treks between X and L can be identified, by using Lemma 2, with only one free parameter t . Thus, we can make use of simple trek rule to parameterize Σ L , X as a function of t . By Lemma 1, σ L | X = t can also be formulated as a function of Σ L , X , and thus t can be solved.

Taking (a), (b), (c), (d) into consideration, for a graph that satisfies the conditions in Theorem 3, for an atomic cover V in the graph, if all pure children of it are observed, then all the edge coefficients from V to its pure children can be identified.

Now, we will prove by induction to show that, for a graph that satisfies the conditions in Theorem 3, for any atomic cover V in the graph, all the edge coefficients from V to its children can be identified, and thus all the edge coefficients of the graph can be identified (the set of all edge coefficients in the graph is the union of the set of edge coefficients from each V to each V 's children).

To this end, we first index all the atomic covers by the inverse causal ordering, such that leaf nodes have smaller indexes. Then we have a sequence of atomic covers V i , i = 1 , ..., C in the graph, where C is the number of atomic covers in the graph.

(i) We show for V i , i = 1 , all the edge coefficients from V 1 to its children can be identified. This is proved by considering (a) (b) (c) (d), as V 1 's children must be all observed; otherwise it cannot be indexed as 1.

(ii) We show that, for i > 1 , if for all V j , 1 ≤ j < i , all the edge coefficients from V j to V j 's children has been identified, then all the edge coefficients from V i to V i 's children can also be identified. This can be proved by combining (a) (b) (c) (d) with Lemma 2. If V i has children that are latent, then the latent children must belong to an atomic cover with a smaller index. Therefore, as all the edge coefficients from V j to V j 's children have been identified, by the use of Lemma 2, the latent children of V i can be taken as if they are observed. Therefore, all the edge coefficients from V i to V i 's children can also be identified.

Taking (i) and (ii) together, all the edge coefficients of the graph can be identified.

A.4 Proof of Theorem 5

Theorem 5 (Sufficient Condition for Parameter Identifiability (up to group sign) without Requiring Structure Identifiability) . In G , if for every latent variable L there always exist another three distinct variables (which can be latent or observed), such that two of the three are pure children of L and the rest one is a neighbor of L , then the parameters are identifiable.

Proof of Theorem 5. The proof is a special case of (b) in the proof of Theorem 3.

A.5 Proof of Theorem 4

Lemma 3. Let Σ X be the observed covariance matrix entailed by F LL , F LX , F XL , F XX , Ω ϵ L , Ω ϵ X . Let Q be an orthogonal matrix, and

˜ F LL = Q T F LL Q, ˜ F LX = Q T F LX , ˜ F XL = F XL Q, ˜ F XX = F XX , ˜ Ω ϵ L = Q T Ω ϵ L Q, and ˜ Ω ϵ X = Ω ϵ X .

Then, the matrices ˜ F LL , ˜ F LX , ˜ F XL , ˜ F XX , ˜ Ω ϵ L , ˜ Ω ϵ X can also entail the covariance matrix Σ X .

Proof of Lemma 3. Let ( F LL F LX F XL F XX ) = ( A B C D ) and ( ˜ F LL ˜ F LX ˜ F XL ˜ F XX ) = ( ˜ A ˜ B ˜ C ˜ D ) . Let M and N be matrices defined as in Proposition 1, and similarly for ˜ M and ˜ N . We then have

and

By Proposition 1, the latent covariance matrix ˜ Σ L is given by

By Proposition 1, the observed covariance matrix ˜ Σ X is given by

This indicates that the matrices ˜ A, ˜ B, ˜ C, ˜ D, ˜ Ω ϵ L and ˜ Ω ϵ X can also entail the covariance matrix Σ X .

Using Lemma 3, we can prove Theorem 4.

Theorem 4 (Condition for the Existence of Orthogonal Transformation Indeterminacy) . Consider the model in Definition 1. If a set of latent variables L with | L | ≥ 2 , have the same parents and children, then there must exist orthogonal transformation indeterminacy regarding the edge coefficients F . In other words, F can at most be identified up to orthogonal transformation indeterminacy.

Proof of Theorem 4. Let S 1 , S 2 , S 3 , S 4 , and S 5 be the indices of L , their latent parents, their latent children, their measured parents, and their measured children in G , respectively. Let Q be a | L | × | L | orthogonal matrix. Let ( F LL F LX F XL F XX ) = ( A B C D ) and ( ˜ F LL ˜ F LX ˜ F XL ˜ F XX ) = ( ˜ A ˜ B ˜ C ˜ D ) For matrices A,B,C , and D from matrix F , suppose that we replace A S 2 , S 1 , A S 1 , S 3 , C S 4 , S 1 , and B S 1 , S 5 with A S 2 , S 1 Q , Q T A S 1 , S 3 , C S 4 , S 1 Q , and Q T B S 1 , S 5 , respectively. Then, we will show that the entailed covariance matrix Σ X is unchanged.

Let U be an m × m orthogonal matrix such that: (i) U S 1 , S 1 = Q , (ii) the remaining diagonal entries are ones, and (iii) the remaining non-diagonal entries are zeros. Let

By Lemma 3, the matrices above can entail the same covariance matrix Σ X .

By construction of U , left multiplication of U T on B only affects B S 1 , ∗ ; specifically, it is equivalent to replacing B S 1 , ∗ with Q T B S 1 , ∗ . Furthermore, only the columns of S 5 in B S 1 , ∗ will be affected, because those columns correspond to the measured children of L . Therefore, all entries of ˜ B are the same as B , except that B S 1 , S 5 is replaced with Q T B S 1 , S 5 . Similar reasoning shows that all entries of ˜ C are the same as C , except that C S 4 , S 1 is replaced with C S 4 , S 1 Q .

Now consider U T AU . By the reasoning above, left multiplication of U T on A only is equivalent to replacing A S 1 , S 3 with Q T A S 1 , S 3 . Further right multiplication of U on U T A is equivalent to replacing ( U T A ) S 2 , S 1 with ( U T A ) S 2 , S 1 Q . Since S 1 , S 2 , and S 3 are mutually disjoint, all entries of ˜ A = U T AU are the same as A , except that A S 2 , S 1 and A S 1 , S 3 are replaced with A S 2 , S 1 Q and Q T A S 1 , S 3 , respectively.

Hence, for matrices A,B,C , and D , suppose we replace A S 2 , S 1 , A S 1 , S 3 , C S 4 , S 1 , and B S 1 , S 5 with A S 2 , S 1 Q , Q T A S 1 , S 3 , C S 4 , S 1 Q , and Q T B S 1 , S 5 , respectively. By the reasoning above, this is equivalent to replacing A , B , C , and D with ˜ A , ˜ B , ˜ C , and ˜ D , respectively, which (generically) share the same support and entail the same covariance matrix Σ X .

A.6 Proof of Corollary 1 and Corollary 2

Corollary 1 (General Necessary Condition for Parameter Identifiability) . For parameters to be identifiable, every pair of latent variables has to have at least one different parent or child.

Proof of Corollary 1. Proof by contradiction. If it is not the case that every pair of latent variables has to have at least one different parent or child, then there exist L such that | L | ≥ 2 and L share the same parents and children. Therefore by Theorem 4 there must exist orthogonal transformation indeterminacy regarding F , and thus the parameters are not identifiable.

Corollary 2 (Necessary Condition about Atomic Covers for Parameter Identifiability) . Assume G satisfies Conditions 1 and 2 and thus the structure can be identified up to the MEC of O a ( O s ( G )) . For any DAG G in the equivalence class, for G 's parameters to be identifiable, every atomic cover must contain no more than one latent variable.

Proof of Corollary 2. Proof by contradiction. If for a DAG in the equivalence class, there is an atomic cover that has more than one latent variable, then according to the definition of the concerned equivalence class, the latent variables in that atomic cover share the same parents and children. Then by Theorem 4 there must exist orthogonal transformation indeterminacy regarding F , and thus the parameters are not identifiable.

A.7 Proof of Proposition 1

Proposition 1 (Parameterization of Population Covariance) . Consider the model defined in Def. 1. Let M := ( I -F LL -F LX ( I -F XX ) -1 F XL ) -1 and N := ( ( I -F LL ) F -1 XL ( I -F XX ) -F LX ) -1 . Then, the population covariance matrices of L and X can be formulated as

Since matrices A and D are invertible, using the formula of 2 × 2 block matrix inversion [22, Chapter 0.7], we obtain

which implies

and

We then have and

̸

̸

We now discuss how M and N defined in Proposition 1 are invertible. Note that matrices I -D and I -F are invertible because structure G is acyclic. This implies det( I -F ) = 0 and det( I -D ) = 0 . Define

̸

Since det( U ) = 1 and det( I -F ) = 0 , we have

̸

̸

By the statement above and det( I -D ) = 0 , we have

̸

which implies that M is invertible. Similar reasoning can be used to show that N is invertible.

A.8 Computational Cost of Checking Whether the Conditions in Theorem 3 Hold

Here we want to investigate, given a structure, can we efficiently check whether the proposed sufficient conditions hold? To this end, we generate random graphs and each graph has 100 variables. According to our empirical result, such a check can be done very efficiently. Specifically, on average, given a structure with 100 variables, it only takes our Python code around 3 seconds to check whether the conditions hold.

which implies and thus

A.9 In Practice, What If the Conditions Do not Hold?

Our condition is useful in solving real-life problems. For example, in the psychometric study, we can properly design the questions with domain knowledge following the condition in Theorem 3 such that each single latent variable has enough observed variables as pure children and thus it can be ensured that all parameters are identifiable (as illustrated in our real-life data result in Figure 4).

On the other hand, even though sometimes the questionnaires and data were designed not so well such that the conditions are not satisfied for the identification of parameters, our Theorem 3 is still useful. In this case, we can still make use of our conditions to check the given structure, and find some local sub-structures where our conditions are satisfied. Consequently, it can be ensured that all the parameters of some sub-structures are identifiable, and we can employ our estimation method to find all the edge coefficients of these sub-structures.

B Additional Definitions, Graphs, Results, and Examples

B.1 Example of Treks

Example 7 (Example of Treks) . In Figure 7 (a), there are four treks between X 4 and X 5 : (i) X 4 ← L 1 → X 5 , (ii) X 4 ← X 3 → X 5 , (iii) X 4 ← L 1 ← X 2 → X 3 → X 5 , and (iv) X 4 ← X 3 ← X 2 → L 1 → X 5 , illustrated in Figure 7 (b).

B.2 Definition of Pure Children

Definition 6 (Parents, Children, and Descendants of a Set of Nodes [18]) . For a set of nodes X in G , we have Ch G ( X ) = ∪ X ∈ X Ch G ( X ) , Pa G ( X ) = ∪ X ∈ X Pa G ( X ) , and De G ( X ) = ∪ X ∈ X De G ( X ) .

Definition 7 (Pure Children of a Set of Nodes [18]) . Y are pure children of a set of nodes X in graph G , i.e., Y ∈ PCh G ( X ) , iff all of the following hold: (i) X ∩ Y = ∅ , (ii) Y ⊆ Ch G ( X ) , (iii) Pa G ( Y ) = X , and (iv) De G ( Y ) ∩ X = ∅ .

B.3 Definition of Neighbor and MEC

Definition 8 (Neighbor) . In G , nodes X and Y are neighbor of each other iff there exist an edge from X to Y or an edge from Y to X .

Definition 9 (Markov Equivalence Class (MEC)) . Two DAGS G 1 and G 2 belong to the same MEC, iif they share the same skeleton and v-structures.

B.4 Definition of Rank-invariant Graph Operator

The definitions are as follows with examples.

Definition 10 (Skeleton Operator [18]) . Given an atomic cover V in a graph G , and let S be the set of all atomic covers in G such that for all S ∈ S , S ⊂ V . For all V 1 ∈ V and all V 2 ∈ PCh G ( V ) \ ( ⋃ S ∈S PCh G ( S )) , if V 1 and V 2 are not adjacent, draw an edge from V 1 to V 2 . We denote such an operator as skeleton operator O s ( G ) .

Definition 11 (Intra atomic operator) . For every atomic cover V in structure G , if | V | ≥ 2 , we add edges between elements in V such that edges among V form a fully connected DAG. We denote such an operator as intra atomic operator O a ( G ) .

Example 8. Suppose the original graph is in Figure 9 (a). After the skeleton operator, we have O s ( G ) , which is shown in Figure 9 (b). After the intro atomic operator, we have O a ( O s ( G )) , which is shown in Figure 9 (c).

B.5 Graphs for Synthetic Data Experiments

Please refer to Figures 10 and 11.

B.6 Additional Result under Model Misspecification

Please refer to Tables 2and 3.

X2

L1

X3

X4

X5

X6

X2

X2

L1

X3

X3

X3

X4

X5

X4

X5

X4

X5

X4

X5

(a) Graph

G

to show treks.

(b) The four simple treks between

X

4

and

X

5

in (a).

X5

X6

L1

L2

L3

X

8x

Xg

X10

X11

X12

X13

X14

X15

X16

X17

X18

C Other Discussions

Our optimization problem in Eq. (4) is solved by gradient descent using PyTorch. Our current implementation is based on CPU but it can be further accelerated by using GPU. A very related discussion can also be found in [35].

The optimization problem in Eq. (5) is solved by gradient descent, which involves evaluating the LogDet and matrix inverse (for the gradient) terms (which is similar to continuous causal discovery methods based on Gaussian likelihood [35]). According to [53], the computational complexity is O ( td 3 ) , where d is the number of variables and t is the number of iterations of gradient descent respectively. Note that the computational cost is largely independent of the sample size as we only need to calculate the sample covariance once and save it for further use.

It is possible to perform inference on the learned parameters in our framework. To be specific, as we use maximum likelihood estimation for the parameters, some standard techniques can be readily used. For example, bootstrap can be employed to provide standard errors on linear coefficients and Chi-square test can also be done to examine the fitness of the model.

D Extended Related Work

One main line of research in latent variable estimation centers on factor-analysis-based methods. Representative studies include [41, 45, 36, 56, 7, 57]. Various other techniques have also been employed for latent structure and parameter identification, including over-complete ICA-based techniques [23, 43, 1] that leverage non-Gaussianity and matrix decomposition-based approaches [3]. However, these approaches typically consider latent variables with observed children, without considering parameter identification in latent hierarchical structures. A more related work is [4], but it considers a much simpler structure.

Another direction would be to project the graph to an ADMG and the latent confounding effects are encoded by correlated noise terms. Following this idea, graphical criteria such as half-trek [20, 5], G-criterion [9], and some further developments [51, 29, 42, 8, 19] has been proposed. Furthermore, another line of works involve studies on causal effect estimation in the presence of latent confounders [52, 6, 30, 33, 26], which often rely on instrumental variables or proxy variables for identification. Notice that in this task, the parameters may not be identified [30], although the causal effect from the treatment variable to the outcome variable can be identified.

(a) Graph

G

.

(b)

O

s

(

G

)

.

(c)

O

a

(

O

s

(

G

))

.

X3

X3

X3

L1

L2

X4

X5

L1

X4

X5

L1

X4

X5

X6

X7

8X

Xg

X6

X7

8X

Xg

X6

X7

X8

Xg

Furthermore, several existing works also solve an optimization problem that involves parameterization of maximum likelihood, such as those in continuous optimization for causal discovery [35, 59, 31, 10, 34] and parameter estimation of Lyapunov models [55, 15]. Differently, our formulation involving likelihood parameterization aims to estimate parameters of partially observed linear causal models.

E Limitations

One limitation of this work is that our theoretical results are based on the assumption of linear gaussian causal models. When data is not linear gaussian, we have also conducted experiments to see the performance of our method. It turns out that our method still performs well in the presence of certain extents of violation of normality and linearity. However, theoretical analysis under violation of linearity and normality would be interesting and the focus of future work.

F Broader Impacts

The goal of this paper is to advance the field of machine learning. We do not see any potential negative societal impacts of the work.

(a) Illustrative graph

G

1

.

(b) Illustrative graph

G

2

(c) Illustrative graph

G

3

.

(d) Illustrative graph

G

4

(e) Illustrative graph

G

5

X3

X4

L2

L2

L5

X5

X6

X7

X8

X9

X11

X6

XT

X8

X15

X16

X17

X3

X4

L1

L2

X4

X5

L3

L1

X5

X13

L2

X6

X7

X8

Xg

X10

X11

X12

X6

X7

8X

Xg

X10

X11)

X12

L1

L2

X4

X5

L3

X6

X7

X8

X9

X10

X11

X12

X4

X5

L1

X6

L2

L3

X

X8

Xg

X10

X11

X12

X15

X16

X17

(a) Illustrative graph

G

6

.

(b) Illustrative graph

G

7

(c) Illustrative graph

G

8

.

(d) Illustrative graph

G

9

(e) Illustrative graph

G

10

X3

L1

X4

X5

X6

X7

8X

L1

L2

L1

L2

X3

X4

X5

X6

X7

X8

X4

X5

X6

X7

X8

X4

X6

X7

L2

L3

L1

L2

L3

L5

X5

X6

X7

8x

Xg

X10

X11

X8

Xg

X10

X11

X12

X13

X14

X15

X16

X17

X5

X6

L1

L2

L3

X7

8X

X9

X10

X11

X12

X13

X14

X15

X16

X17

X18

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: the claims stated in the abstract and introduction are the same as what are stated in the theory and method part.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: A discussion about the limitations can be found in Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions together with necessary definitions are provided in the main text and all complete proofs are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The implementation details and the experimental settings are all provided in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The code and the data will be publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See the experiment section for the setting and details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The synthetic experiments were supported by error bars to show statistical significance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: It is not related to this work. No such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the assets are properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes novel methods for learning linear structural equation models from partially observed data (i.e., allowing for latent variables). The authors provide graphical identifiability conditions for such models and describe an algorithm for learning structural parameters from data via gradient descent. Results on synthetic and real-world examples suggest that the method works well in practice.\n\nSTRENGTHS:\nThe manuscript is clear and well-written. It makes a meaningful contribution to a well-studied topic, which I think will be of great interest to many NeurIPS readers and the causality community more generally. The theoretical sections on identifiability and indeterminacy are well reasoned, with helpful examples along the way. The implementation is elegant and efficient, with compelling results on simulated and real-world data.\n\nWEAKNESSES:\nI was a bit confused by the discussion of necessary graphical conditions following Thm. 1. First, I would recommend avoiding terms like \"pretty close to...necessary\", which doesn't mean much. We get a bit more insight in Remark 1, where it's revealed that condition (i) *is* in fact necessary, but condition (ii) is not. We learn that if (ii) does not hold, then \"there are...some rare cases where parameters can be identified.\" Oddly, we don't see any examples of such structures (either in the main text or the supplement), or learn what \"rare\" amounts to here (presumably *not* Lebesgue measure zero?) Specific counterexamples would help a great deal! Perhaps some sort of disjunctive graphical condition could do the trick, e.g. condition (i) + [(ii) OR (iii)], where (iii) covers those purportedly \"rare\" structures that violate (ii) but are still technically identifiable. Alternatively, I would cut all discussion of necessity from this section and move it instead to a discussion section later in the manuscript. (I appreciate that space is tight here, but with an extra page in a final version this could be a more satisfying solution).\n\n\nMinor errata: \n-All instances of \"upto\" should be \"up to\"\n-It appears that references 46 and 47 are the same\n-It appears that references 5 and 19 are the same\n-The word \"Gaussian\" is occasionally uncapitalized\n\nQUESTIONS:\n-I take it that the identiability conditions of Thm. 1 are untestable? Or perhaps they have some testable consequences in certain settings? If so, this would be very helpful for practitioners!\n\n-Would it be possible to perform inference on the learned parameters? For instance, could this method provide standard errors on linear coefficients? \n\n-Though I know the method is designed for the partially observable setting, I'm curious how it fares against competitors such as GES or PC when latent variables are absent?\n\nLIMITATIONS:\nYes, in Appx. D. May be worth moving this to the main text in the final submission, however.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of parameter identification in linear causal models, which is important and well-studied task in causality. The authors examine models that explicitly include both observed and latent variables. The identification of parameters in such models has not been studied so far and the authors are the first to formulate this problem and provide the first results in this regard. The main achievements of the paper are the sufficient and necessary conditions for parameter identifiability.  Moreover the authors conducted empirical studies on both synthetic and real-world data to validate the proposed methods.\n\nSTRENGTHS:\nThought causal structure learning in the presence of latent variables has been well studied in the literature, the parameter identification in models that explicitly include both observed and latent  variables -- as presented in the submission -- is new. The paper provides non-trivial sufficient and necessary graphical conditions for parameter identifiability and present them in the context of known graphical conditions for structure identifiability provided in [17].\n\nWEAKNESSES:\nThe sufficient condition for parameter identifiability assumes that G, in addition to conditions (i) and (ii) in Thm. 1, satisfies conditions 1 and 2 presented in section 3.2. I agree that conditions 1 and 2 imply that the structure G can be identified (as shown in [17]) however the  sufficient conditions formulated in this way are overall very restrictive. It would be interesting to have sufficient conditions also in the case of structures which do not satisfy conditions 1 and 2: It can happen that G can be identified even if it does not satisfy 1 and 2 or the structure is provided by a researcher / theory. The authors do not discuss if there exist cases which can be identified but which do not satisfy 1 or 2.\n\nAlso, it is not clear what is the gap between the sufficient and necessary conditions.\n\nThe next issue is that the authors do not discuss what is the computational complexity of parameter identification in linear causal models, that explicitly include both observed and latent variables.\n \nIt is not clear to what extent the sufficient and necessary conditions proposed in Section 3 are useful for (numerical) parameter estimation discussed in Section 4.\n\nQUESTIONS:\nPlease provide the motivation for considering edges / parameters from observed to latent variables and edges between latent variables.\n\nSee also my questions above.\n\nIn Line 69: explain that d=n+m\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the parameter identifiability of partially observed linear causal models, focusing on whether edge coefficients can be recovered given the causal structure and partially observed data. It extends previous research by considering relationships between all variables, both observed and latent, and the coefficients of all edges. The authors identify three types of parameter indeterminacy in these models and provide graphical conditions for the identifiability of all parameters, with some conditions being necessary. A novel likelihood-based parameter estimation method is proposed to address the variance indeterminacy of latent variables, validated through empirical studies on synthetic and real-world datasets, showing the effectiveness of the proposed method in finite samples.\n\nSTRENGTHS:\nThe authors consider the problem of parameter identifiability in partially observed causal models with linear structural equations and additive Gaussian noise. This has not been studied in a similar way before in the literature, especially when observed and latent variables are allowed to be flexibly related. Specifically, observed variables are allowed to be parents of unobserved variables. The authors also provide graphical conditions that are sufficient for all parameters to be identifiable and show that some of these conditions are provably necessary. To address the scenario where noise covariance matrix is unknown, they propose a novel likelihood-based parameter estimation method and validate it with empirical studies on synthetic and real-world data.\n\nWEAKNESSES:\nThe paper considers a restricted setting, allowing only linear relations between observed and unobserved variables and restricting the noise to be Gaussian. The theoretical guarantees hold under the scenario when the noise covariance matrix is unknown. Nonetheless, the authors mitigate this by providing an estimation scheme for the noise covariance matrix from limited data samples. The conditions of identifiability proposed are not fully necessary and sufficient; this is left for future work.\n\nQUESTIONS:\nI have  few questions and comments for the Authors:\n\n*  Line 139: The authors mention that the indeterminacy of group sign is rather minor. if the parameters are identifiable only up to group sign indeterminacy, we still say that the parameters are identifiable. It would be useful to explain why group sign indeterminacy is a minor issue. It might actually depend on the underlying task, and there might be application scenarios where it is not insignificant. Some details on this would be useful in the main paper.\n\n*  The additive noise is assumed to be Gaussian. Suppose the noise follows another continuous distribution such as Gamma or Student-t distribution. Can we use the identifiability results? If not, what is special about the Gaussian noise here compared to other continuous noise distributions?\n\n* Referring to Proposition 1, some matrix inverses need to be calculated to compute the noise covariance matrices. Will the inverses always exist? If yes, can you elaborate on how? If no, how does that impact the application of Proposition 1?\n\n* In the experiments section, the authors demonstrate the effect of sample size on the mean squared error (MSE) for the parameter matrix \\( F \\). It would be useful to also show how accurate the estimates of the noise covariance matrix depending on the sample size used.\n\nLIMITATIONS:\nThe authors clearly describe the problem setting and assumptions. I have mentioned the main limitation of the proposed method in the weaknesses section. I don't think there are any potential negative societal impacts of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces conditions under which DAGs can be recovered in the linear case where some nodes are observed and some are not. This DAG recovery involves computing edge weights between nodes in a causal graph. Nodes are allowed to be latent or observed, with varying types of edge weight indeterminacy depending on the structure of the causal system in question.\n\nSTRENGTHS:\nThere is certaintly interest in estimating DAG structure in practice. The work here presents useful results for how and when that estimation may or may not occur given the structure of the causal system. The extension to latent variables is a contribution, although in practice, latent nodes might increase the already difficult task of interpreting inferred DAG structure. The misspecification analysis is useful and the presence of examples in the text helpful (with a few caveats below). \n\nI like the title. The framing is at its strongest when the paper articulates the general conditions under which identifiability can and cannot be achieved. As for whether a given causal system in practice meets assumptions for strongest identifiability is in the end, to my eyes, a very difficult question. \n\nOverall, the paper is a solid contribution, although I believe it could be improved (see below).\n\nWEAKNESSES:\nThe text overall is generally well-written, with some caveats listed below. In my reading, the first half of the paper read more clearly than the second half. For example, I couldn't quite piece together from the discussion of estimation whether the estimated graph will be dense (all nodes connected to all other nodes, given the [seemingly?] continuous optimization being done, e.g., Eq 3. If all edges are connected to all other edges, then the relative usefulness seems to be weakened in that usually, investigators seek out a parsimonious representation of a causal system. \n\nI was also wondering what more established methods would yield as an empirical baseline (e.g., PC algorithm); currently, the Estimator-LM (estimator with Lagrange multiplers) is articulated as a baseline. This is one of the methods introduced in the paper. An external state-of-art baseline would be most informative. The authors state that \"no existing method...can achieve the same goal as ours.\" If this is because of the latent variable aspect, one could in principle restrict the MSE calculation to edges among observed nodes. In other words, perhaps there isn't a perfect analogue method but an imperfect comparison could be better than none. We could also get a visual comparison of the DAG among observed variables used in Figure 4 from some existing baseline methods for the Appendix. \n\nFinally, there is little discussion of uncertainty estimation. Uncertainty estimation in the observed DAG recovery case is hard, even more so here (presumably). \n\nThere are points where the text could, to me, use more clarity in the discussion: \n\n- Condition 1 seems relatively minimal and even intuitive. It seems very hard to know in practice if Condition 2 (line 193) holds or is even a minimal or very restrictive condition.\n\n- I appreciate the author(s)' inclusion of Example 2 and Example 3. I think the logic could be made clear, perhaps with additional shadings or labelings that would help us see which sets of nodes and edges are doing what work regarding Condition 1 and Condition 2. \n\n- I would make the \"pretty close\" language in 218 and 255 a bit more precise. Also, starting line 264. I would revise this from, \"it has considerable extents of necessity, and could be expected to serve as a stepping stone towards tighter and ultimately the necessary and sufficient condition for the field.\" to something that also is seomwhat more precise. Also, there is no guarantee that necessity+sufficiency will be found (or perhaps there is an impossibility), so would hedge this possibility somewhat. \n\n- Can you define what it means for \"QF and F\" to \"share the same support\" in this case? \"Support\" is often defined in causal inference settings as an event probability falling between 0 and 0; here, F is defined as the matrix embodying the causal edge coefficients, so I believe what is meant here is that QF and F do not share the same set of non-zero entries. Clarifying this would be helpful. I would also consider beginning with the example of indeterminacy before defining it to help the reader see your point intuitively before the formalization.\n\n- I would definition 7 into the main text. It is an important definition used multiple times and without it, it is hard to follow the atomic cover discussion. it's also a very short definition.\n\nMoreover, I would in general help the reader along by first explaining the concept before formally/technically defining it. Examples: \n\n- The term structure identifiability and parameter identifiability should be clearly defined before the terms are used. (I don't think I see a clear definition before use currently; I would move the paragraph beginning on line 171 up in the text, as it is a clear articulate of the point.) In a similar vein, I would explain what atomic covers are going to do before we jump into the definition on line 164. \n\nOther details would help this reader: \n\n- The MSE up to orthogonal transformation is an interesting metric. Some mention of how this optimization is done would be helpful.\n\n- I would add a sentence explaining whether GPU acceleration would or would not be helpful and why.\n\nI also noted several minor points listed here: \n\n- References used are inconsistent at times. Sometimes, we see reference to \"condition (i)\", others to Condition 1. \n\n- Line 212. Missing space \"identified upto the\" should read \"identified up to the\". This same occurs in line 140 (\"upto\" should read \"up to\") and in other parts of the text as well. \n\n- Line 147. \"indetermincay\" should read \"indeterminacy\". This typo occurs a few times in the text. \n\n- Line 129. Clarify what is meant by \"entails the same observation as that of...\" This also appears in line 142 (\"Entails the same observation\"). I assume this means something about implying the same probability distribution, but helping the uninitiated reader is usually appreciated. \n\n- Line 137. \"However, if we set f1,2 = 0, then the parameters are not identifiable. These rare cases of parameters are of zero Lebesgue measure so we rule out these cases for the definition of identifiability\" -> I would just say \"these presumably rare cases of parameters\". Probably some justification is needed to articulate why this should be rare in real causal systems. Does any prior literature speak to this?\n\n- Line 72. I believe \"the causal edge coefficient of the model\" should read \"the causal edge coefficients of the model\".\n\n- Capitalize \"Gaussian\" on line 286.\n\nQUESTIONS:\n- What are the implications of the diagonal covariance matrix $\\epsilon_{\\mathbf{V}_{\\mathcal{G}}}$? \n\n- Regarding, \"As variables are jointly Gaussian, asymptotically our observation can be summarized as population covariance over observed variable\". Wouldn't this statement also apply in finite samples under Gaussianity?\n\n- A major benefit seems to be identification of edge weights. If actual interpretation of the edge weights is going to be done in practice, group sign indeterminancy would limit applicability. Would it help to anchor the sign of one edge based on prior science? Guidance? \n\n- In Figure 4, are circulate nodes latent and nodes denoted by [LetterNumber] observed? If so, clearly articulate this in the figure label.\n\nLIMITATIONS:\nI see no major negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}