On the Parameter Identifiability of Partially Observed Linear Causal Models
Authors: Xinshuai Dong
Ignavier Ng
Biwei Huang
Yuewen Sun
Songyao Jin
Roberto Legaspi
Peter Spirtes
Kun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Linear Causal Model
Parameter Identification
Primary area: causal_inference
TL;DR: Parameter Identifiability of Partially Observed Linear Causal Model

Abstract: Linear causal models are important tools for modeling causal dependencies and yet in practice, only a subset of the variables can be observed. In this paper,  we examine the parameter identifiability of these models by investigating whether the edge coefficients can be recovered given the causal structure and partially observed data. Our setting is more general than that of prior research—we allow all variables, including both observed and latent ones, to be flexibly related, and we consider the coefficients of all edges, whereas most existing works focus only on the edges between observed variables. Theoretically, we identify three types of indeterminacy for the parameters in partially observed linear causal models. We then provide graphical conditions that are sufficient for all parameters to be identifiable and show that some of them are provably necessary. Methodologically, we propose a novel likelihood-based parameter estimation method that addresses the variance indeterminacy of latent variables in a specific way and can asymptotically recover the underlying parameters up to trivial indeterminacy. Empirical studies on both synthetic and real-world datasets validate our identifiability theory and the effectiveness of the proposed method in the finite-sample regime.

On the Parameter Identifiability of Partially Observed Linear Causal Models
Xinshuai Dong *
Ignavier Ng Biwei Huang?
Yuewen Sun Kun Zhang
Songyao Jin
Roberto Legaspi!
Peter Spirtes!
'Carnegie Mellon University University of California San Diego
Mohamed bin Zayed University of Artificial Intelligence 4KDDI Research
Abstract
Linear causal models are important tools for modeling causal dependencies and yet in practice, only subset of the variables can be observed. In this paper, we examine the parameter identifiability of these models by investigating whether the edge coef- ficients can recovered given the causal structure and partially observed data. Our setting is more general than that of prior research- we allow all variables, including both observed and latent ones, to be flexibly related, and we consider the coeffi- cients of all edges, whereas most existing works focus only the edges between observed variables. Theoretically, we identify three types of indeterminacy for the parameters in partially observed linear causal models. We then provide graphical conditions that are sufficient for all parameters to be identifiable and show that some of them are provably necessary. Methodologically, based parameter estimation method that addresses the variance indeterminacy in a specific way and can asymptotically recover the underlying parameters up trivial indeterminacy. Empirical studies on both synthetic and real-world datasets validate our identifiability theory and the effectiveness of the proposed method in the finite- sample regime. Code: bttpps//gitee tig...th..... cwm//ongxinshuuiii /scm cemm-identify
propose a novel likelihood-
Introduction and Related Work
Causal models, which serve as a fundamental tool to capture causal relations among random variables, have achieved great success in many fields 44 39 40
44 A fundamental problem in the field is how and to what extent can we identify the underlying causal model given observational data. When all variables are observed, the problem has been well studied: the underlying structure can be identified up to the Markov equivalence class, c.g., by the PC [49] or GES [13] algorithm; when the structure is given, the causal coefficient (direct causal effect) between two variables can also be identified [8] 39
However, in real-world systems, the variables interest may only be partially observed. Thus, consid- erable efforts have been dedicated to identification of causal models in the presence of latent variables. One line of research focuses on structure learning given partially observed variables. Notable ap- proaches include FCI and its variants 49
38
14
well as ICA-based 23 43
tetrad-based [48]
28 high-order moments-based
46
58
and rank constraint-based
48
24
18
methods.
In this paper, we focus on the the identification of parameters of partially observed model. Specifi- cally, given the causal structure of and observational data from a partially observed causal model, we interested in identifying all the parameters, and thus the underlying causal model can be fully specified. To identify the parameters, a classical way is to project the directed acyelic graph (DAG) with latent variables to acyclic directed mixed graph (ADMG) or partially ancestral graph [42). without explicitly modeling the latent confounders. Based on ADMG, graphical criteria such as half- trek (20), G-criterion [9, and some further developments [1]] 29 have been proposed to establish the parameter identifiability. Another way is to leverage do-calculus, proxy variables, and instrumental
*Equal contribution. 38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

variables 47 39 25 to identify the direct causal effect, which corresponds to the edge coefficient in linear causal models. For more detailed discussion of related work, please refer to Appendix [
Despite the effectiveness of current methods for parameter identification, however, they have two main drawbacks: they require all the variables to be connected in specific ways, and only focus identifying the edge coefficients between observed variables. To this end, in this paper we propose a novel framework that considers a more general setting for parameter identification. To be specific, we allow all variables, including both observed and latent ones, to be flexibly related, and we aim to recover the edge coefficients among all variables, even including those from a latent variable to another latent variable or another observed variable, which previous methods cannot handle. We summarize our contributions follows.
To the best of our knowledge, we are the first consider parameter identifiability of partially observed causal model in the most general scenario- all variables, including both observed and latent ones, are allowed flexibly related. and edge coefficients between any pair of variables are concerned. In contrast, most existing works consider only the edges between observed variables.
Theoretically, we identify three types of parameter indeterminacy in partially observed linear causal models. We then provide graphical conditions that are sufficient for all parameters to be identifiable and show that some them are provably necessary. These necessary conditions also offer insights into scenarios where the parameters are guaranteed to be non-identifiable.
Methodologically, we propose a novel likelihood basodd parameter estimation method, which parameterizes population covariance in specific ways to address variance indeterminacy. Our empirical studies on both synthetic and real-world data validate the effectiveness of our method in the finite-sample regime, even under certain misspecification of the underlying causal model.
Preliminaries
2.1 Problem Setting
In this work, focus partially observed linear causal models, defined follows.
Definition 1 (Partially Observed Linear Causal Models). Let G == (Vg, Eg) be DAG. Each variable Vi E Ve follows a linear structural equation model V, = (X))==++ contains m latent variables and n observed variables. Pag(V, (V,) denotes the parent set V, f3,3 denotes the edge coefficient from Vj to V,, and ov, represents the Gaussian noise term V,,
f4,V, +e,, where
Vg = Lg X = (L
We drop the subscript G in Lg and XG when the context is clear. We use V, V, and V to denote random variable, set of variables, and set of sets of variables, respectively. In Definition I the relations between variables can also be written in the matrix form as Vg = F F + eV.. where F (fys)).e(mmn)) is the weighted adjacency matrix. Here, fys & if and only if V, is a parent of V; in G. We also write
L
L F=
FLL
FLX
See 0 ) O
F
and Ω
FXL
Fxx
Dee
where Q is the diagonal covariance matrix of EV.-
Our objective is to identify F, the causal edge coefficients of the model, given observational data and the causal structure G. Denote by Σ, and Ex the population covariance matrix of latent variables L and observed variables X, respectively; their precise formulations are provided in Proposition T We also denote by o., the (i. 1-th entry of Σx. In this work, we assume that the noise terms of latent variables, éL, have unit variance, i.e., P.z I, which will be justified later in Section 3.1 Note that variables are partially observed and thus we only have access i.i.d. samples of observed variables X. As variables are jointly Gaussian, the observations can asymptotically be summarized as the population covariance matrix Σx. In other words, we aim to identify and S2 given Σx and G. The identification of parameters is important in that, once identify the parameters, the underlying causal model is fully specified, and thus we can flcxibly calculate causal effects, infer interventional distributions, and finally answer counterfactual queries [39]. It is worth noting that, for parameter identification, the structure G is assumed to be known, which is different from the setting of causal discovery where the goal is to identify from data.

---

X2
X
X;
%
(a) Graph Gr Its parameters can be fully identifiable.
(b) Graph Gz. parameters can- not be fully identified.
(c) G1 G2 have the same ADMG the latent projection framework.
Figure 1: Illustrations of the advantage of our framework. Within our framework, it can be shown that GI 's parameters can be identified (up to sign) while 92 's cannot. In contrast, the latent projection framework cannot even differentiate Gi from G2 as they share the same ADMG (c) after projection. Furthermore, with ADMG, any edge coefficient that involves a latent variable cannot be considered.
2.2 Framework Comparison
Without latent variables, it has been shown all parameters are identifiable [8] However, the problem becomes very challenging when latent variables exist. There are two lines of research. One focuses on the use of do-calculus, proxy variables, and instrumental variables to identify direct causal effects among observed variables [7 39 25 (in linear models the direct causal effect is captured by the edge coefficient). Another line addresses latent confounders by projecting DAG with latent variables into an ADMG, where the confounding effects of latent variables are simplified and represented by correlation among noise terms (20 9 ST 29]. An example is in Figure where (a) is the original graph and (c) is the projected ADMG whose bidirected edges correspond to correlated noise terms.
Compared to the two previous lines of thought, our framework has two advantages. To begin with, we additionally considers the identifiability of coefficients of edges that involve latent variables. For example, Figure [ we aim identify all the coefficients including the one from L1 to X3, i.e., fi.3. In contrast, the proxy variable framework and the latent projection framework identify only the coefficients among observed variables: the proxy variable framework focuses only on the direct causal effect from one observed variable to another observed variable, while the latent projection framework transforms all latent variables into bidirected edges and thus can never identify the coefficient of the edge that has a latent variable as the head or tail.
Furthermore, the projection framework deals with latent variables in a rather brute-force way: dense latent confounding effects among observed variables may be caused by only small number of latent variables, but that information lost during projection. For example, in Figure I (a) and (b) share the same ADMG after projection, i.e., (c). However, as we will show later, parameters in (a) can be identified, while in (b) the parameters cannot. If we only consider the ADMG in (c), then we can never capture this nuance and thus cannot identify the coefficients that we might be able to.
Identifiability Theory
3.1 Definition of Parameter Identifiability and Indeterminacy
We follow the notion of generie identifiability and define parameter identifiability as follows.
Definition 2 (Identifiability of Parameters of Partially Observed Linear Causal Models). Let 0 = (F,2) E θ. We say that 0 is generically identifiable, if the mapping ((0) Σx is injective, for almost all 0 E e with respect to the Lebesgue measure.
Definition 2 indicates if parameter is identifiable, then there does not exist 0' e that entails the same observations as those of 0. As in the typical literature of parameter identification, we consider generic identifiability to rule out some rare cases where the parameters for that structure is generally identifiable, but with some specific parameterization, the parameters cannot be identified. This is similar to faithfulness in causal discovery (49 and we will provide an example in Example We next introduce three important indeterminacies about parameter identification when latent variables exist. Theorem (Indeterminacy of Scaling of P.y)) Consider a model that follows Def. 7 with number of latent variables m. 2 1 and 0 = (FIL, Fix, FXI. Fxx. Po Dex) Let N be any invertible diagonal matrix, and θ (FLL, FLX. FXL, Fxx, Pey, Dex) where
FLL = A = FLX, PxL = PXLA, Fxx Fxx, Dey = = A2 Dex = Dex Then, i and 0 entail the same observations. i.e. Σx Ex. Furthermore, we have Et. AEIA
3

---

(a) G1. Its structure identifiable but its parameters (b) Gz. structure are not identifiable even if the structure is given (due is not identifiable but to orthogonal indeterminacy).
(c) Gs. G2's structure is not identifiable due to the existence of Gs.
params are identifiable.
Figure 2: Illustrative examples to show that the graphical condition for structure-identffabbiiyt and parameter-identifabliiy could be very different.
similar theoretical result is provided in [4], and yet our setting is much more general and takes that of 4 as a special case: in our setting, all variables including latent and observed ones can be arbitrarily related while in 4 latent variables cannot be the effect of observed variables.
Remark 1 (Implication of Theorem D. A key implication of Theorem 2 is that, without further
assumption, the edge coefficients involving latent variables, i.e., (FL.. Fix,, Fxt.), can never be identified, as there always exists a diagonal matrix A such that 0 and entail the same observations but (FLL, Fix, FxI.) + (FLL, Fix, Fxt.). Thus, in the rest of this paper, we assume that the noise terms of latent variables, éL, huve unit variance, i.e., DeL I. Under this assumption, we have (De, )1. A2, (De, )i.i 1, [m], which implies As,5 +1. As such, this assumption makes parameter identifiability possible. However, even though we fax the scaling of Dey, there still exists indeterminacy about the sign of parameters, captured by Theorem
Theorem 2 (Group Sign Indeterminacy). Consider a model that follows Def. 7 with number of lutent variables m 2 1. = (FLL. FLX, FXL, Fxx; Dea Dex): and De, = 1. Let S a diagonal sign matrix (entries are either or - 1), and f (FLL, FLX, FXL, Fxx, Sen. flex), where
FL. = SF FL. = SFLx. Fxt. FXLS, Fxx = Fxx, Drg = = I, Ôix Pxx
Then, Ô and 0 entail the same observations, i.e., Σx = Ex, and (Ét)a = -EI))i Vi [m]]
Remark (Remark on Theorem Z}. The indeterminacy described in Theorem i referred to as group sign indeterminacy for the following reason: According the theorem, flipping the sign of S is equivalent to flipping the signs of all coefficients of edges involving the latent variable Li- This transformation preserves the resulting observations Σx. In essence, each group consists of coefficients of edges involving a particular latent variable.
Example (Example for Group Sign Indeterminacy and Generic Identifiability). In Figure 2 given the structure and Σx, by assuming Dr, = I, the parameters are generally identifiable up to group sign indeterminacy. Specifically, there exist three equality constraints with three free purameters: S1,2J1,3 02.3, 11,2/1,4 02,4, and f1, 03.4. The solutions are: (i) S1.2
22.303.
f1.3 02.3/115 fIA =2,//11, and (ii) 1122-11
= f1,3=
-023/
03.1
05.4
fI.1 -02.//1. f1.2. The two solutions are different only in terms of group sign. However, if we set
f1.2 O, then the parameters are not identifiable (as we will encounter division where the divisor zero). These rare cases of parameters are of zero Lebesgue measure so we rule out these cases for the definition of identifiability as in Definition) 2
Intuitively speaking, group sign indeterminacy arises because one may multiply the latent variable L, by -1 and accordingly flip the signs of all edge coefficients involving L,. Note that such an indeterminacy is rather minor for the following reason. In practice, we can always anchor the sign of some edges according to our preference or prior knowledge in order to eliminate the group sign indeterminacy. For example, in Figure if expect that L2 should be understood as Extraversion instead of non-Exte Non-Exterversion we can add one additional constraint during our parameter estimation such that the edge coefficient from L2 to E1 ("I am the life of party.") will be positive (as we believe E1 should be positively related to Extraversion). (ii) On the other hand, there are some application scenarios that are not influenced by the group sign indeterminacy, such as causal effect estimations between certain variables. We note that, as the indeterminacy of group sign is rather minor, in the following the parameters are identifiable only up to group sign indeterminacy, we still say that the parameters are identifiable.

---

Definition 3 (Orthogonal Transformation Indeterminacy), Consider model that follows Def. 7 with number of latent variables m 2 1, 9 (FiL, Fix. FxI,Fxx, that there exists orthogonal transformation indeterminacy in the identification of parameters if there exists a non-diagonal orthogonal matrix Q such that (FIL, FLX, FXI, Fxx: Qee Sex)) and (FLL, Fix,,Fxx, Fix,FxI, Fxx, Dez, flex) share the same support and entail the same observations, where
and Dt, I. We say
Fu. - -Q"FF.O. Fixx = QTx,, FXt.- - FxI.Q, Fxx - Fxx, dig = QZ = Sixx - Pex
The orthogonal transformation indeterminacy is the major indeterminacy we consider in the presence of latent variables. Such an indeterminacy also arises in factor analysis [45]]] 77, which can be viewed as special case of the data generating procedure considered in Definition Here we only give the definition and will later provide Theorem 4 with an example that captures the scenarios where such
indeterminacy exists.
is worth noting that the graphical condition for structure identifiability and parameter identifiability could be very different. For example, 91 in Figure 2 (a) is structure-identifiable and yet the parameters are not identifiable even if the structure is given. In contrast 92 in Figure S (b) is not structure-identifiable,
there exists another structure G3 in Figure (c() such that G2 and G3 can never be differentiated from observational distribution; and yet if Go is given, its parameters are identifiable (as in Example [T] Therefore, in this paper, we first consider the cases where the structure can be identified and then study which further conditions are needed for the identifiability of parameters. This will give rise to conditions under which the whole causal model can be fully specified.
3.2 Graphical Condition for Structure Identifiability
To explore the conditions for the whole causal model to be specified, we start with the structure identifiability of partially observed linear causal models. Recent advances have shown that if certain graphical conditions are satisfied [24] [8]. even though all variables including latent ones are allowed to be very flexibly related, the causal structure can still be identified. Next, focus the conditions by [18], which takes that of (24) as special cases. Roughly speaking, the identifiability of the structure of partially observed linear causal model is built upon the identifiability of atomic covers, defined as follows (with effective cardinality defined as || = I(Uvev V)l and PChg defined in Appendix (B.2).
Definition 4 (Atomic Cover 18).. Let V E Vg be set of variables, where I out of [V] are latent, and the remaining [V| are observed. V is an atomic cover if V is a single observed variable, or if the following conditions hold:
(i) There exists set of atomic covers C, with ||C|| 2 I 1, such that Ucec C C PV))(V).
(ii) There exists a set of covers N with [[]]| 2 1+ 1, s.I. (UNeN (UNexN)n(UCEd C) = 0, every element in UNENN is neighbour of every element in V, and V d-separates N and C.
(iii) There does not exist a partition P of V, s.t., all elements in P are atomic covers.
The intuition that we build structure identi- fiability upon the notion of atomic covers is as follows. When a set of latent variables share the same set of children and neigh- bors, it is impossible to differentiate these latent variables from each other, and thus we need to consider them together as the minimal identifiable group to build up the identifiability of the whole structure. Such a minimal identifiable group of variables is defined as an atomie cover. Roughly, for group of variables to be qualified as atomic cover, it has to have enough children and neighbors. An example is as follows.
X
Xe
X=
X
X
X
X
X
Figure 3: An illustrative graph that satisfies the conditions for structure- identifiability. At the same time, it also satisfies the condition for parameter identifiability given the struc- ture and Vxx all the parameters are identifiable only up to group sign indeterminacy.
structure-identiiaailit
Example 2 (Example of Atomic Cover). Consider the graph in Fig.
V (6, is an atomic
cover. This is because there exist C = {(Xa). {X1o7} with ||C]| 2
= 2 such that (i) in Def. 4
is satisfied. And there exist N == {(X11,, (X12} (or, N = ([X+] (Xs with [[|]| 2 1 1 = 2
such that (ii) in Def. Dis is satisfied. We can also fin that (iii) in Def. 4 is satisfied. Thus {Lا, Xo} is an
atomic cover, Another example would be in Figure 8 where (L1, L2 } is an alomic cover.
5

---

(Basic Conditions for Structure Identifiability (8)). G satisfies the basic graphical condition for identifiability, if every latent variable belongs to at least one atomic cover in G and for each atomic cover with latent variables, any of its children is not adjacent to any of its neighbours.
Condition 1
Condition 2 (Condition on Colliders [88.. ). In G. (i) there exists sets of variables V, V1, 22, and T such that every variable in V is collider of two atomic covers V1, 22, and T is a minimal set of variables that d-separates V1 from 22, and (ü) there exists at least one latent variable in VUV, U V2 T, then we must have [V] + [T| 2 \V2l+
Example 3 (Example that satisfies Conditions] [). Consider Figure 3 All latent variables in the graph belong to at least one atomic cover and thus Condition is satisfied. Plus, Condition 2 is also satisfied. This is because the sets of variables V1. 22, and T that satisfy (i) and (ü) in
Condition 2 are = (X12), V1 = X6), V2 = {L2], and T = (Xa, Xs), and we also have VV + /V2 Therefore, the graph in Figure satisfies both Conditions [ [and? 2
[V] +
The identifiability theory of structure is as follows. For graph G. if Condition [ and Condition 2 are
satisfied, then asymptotically the structure is identifiable up to the Markov equivalence class (MEC) of (G)) (definitions of (-) and (-))) can be found in Appendix B.4]. Roughly speaking, the underlying causal structure of G can be identified except that the directions of some edges cannot be
determined. Next, we will show that, given any DAG in the identified equivalence class together with Ex. the parameters of the model are also identifiable, if certain conditions are satisfied.
3.3 Identifiability of Parameters
In this section we show that, given graphical Conditions I anddd 2 the causal coefficients F in Definition D are also identifiable, if certain conditions are satisfied.
Theorem 3 (Sufficient Condition for Parameter Identifiability (up to group sign), Based on Structure Identifiability). Assume that G satisfies Conditions | [ and thus the structure can be identified up the MEC of Oa(O, (G)). For any DAG in the equivalence class, the parameters are identifiable, if both the following hold:
(i) For any atomic cover == XUUL
(ii) If an atomic cover V = X U satisfies [L] + 0 and |X| 2
1, then all simple treks (Def. 5 between L and X do not contain any latent variables that are not in L.
Theorem 3 provides sufficient condition such that the parameters are identifiable. Now, for a better understanding of Theorem 3 we provide an example of it as follows.
Example 4 (Example for Theorem 3). The graph 9 in Figure 3 Satisfies the conditions for parameter identifiability in Theorem 3 Specifically, condition (i) in Theorem 3 is satisfied as all atomic covers contain no more than one latent variable. Plus, condition (ii) Theorem 3 is also satisfied, as {L1} U {Xs) satisfies [L] + 0 and X 2 I and all simple treks between }L1} and (X6) contain only observed variables except {.1.. Therefore, the parameters are identifiable for the graph in Figure
the atomic cover V = X KUL=
Next, we discuss under which conditions the parameters are guaranteed to be not identifiable. As discussed in Section 3.1] there are three kinds of indeterminacy. The first one can be solved by assuming unit variance of the noise terms of latent variables while the second one group sign indeterminacy is rather trivial such that we still consider parameters identifiable even if group sign indeterminacy exists. Therefore, we will focus on the third one, orthogonal transformation indeterminacy, in what follows.
Theorem 4 (Condition for the Existence of Orthogonal Transformation Indeterminacy). Consider the model in Definition [ If set of latent variables L with [L] 2 2, have the same parents and children, then there must exist orthogonal transformation indeterminacy regarding the edge coefficients F. In other words, F can at most be identified up to orthogonal transformation indeterminacy.
Example 5 (Example for Thm. 4. Consider Fig. g The graph satisfies the conditions in Thm. f as the parents and children of L1 and L2 are exactly the same. Therefore, there must exist orthogonal transformation indeterminacy for the edge coefficients F and thus the parameters are not identifiable.
The Theorem abbove indicates that, if there exist two latent variables that share the same parents and children, then the edge parameters at most be identified up orthogonal transformation. This directly implies a necessary condition for parameter identifiability as follows.
Corollary 1 (General Necessary Condition for Parameter Identifiability). For parameters to be identifiable, every pair of latent variables has to have at least one different parent or child.

---

Corollary captures necessary condition in the general cases such that parameters are identifiable. If we further consider the graphs that are also structure identifiable (as we need to identify the structure first to fully specified the causal model), we further have the following Corollary by considering the notion of atomic covers (the proofs of both corollaries can be found the Appendix).
Corollary 2 (Necessary Condition about Atomic Covers for Parameter Identifiability). Assume G satisfies Conditions 7 and and thus the structure can be identified up to the MEC of (O)). For any DAG in the equivalence class, for g's parameters to be identifiable, every atomic cover must contain no more than one latent variable.
Remark 3 (Necessity of Conditions in Theorem Condition (i) in Theorem is provably necessary: by Corollary 2 for parameters to be identifiable, one has to assume (i) in Theorem 3
Establishing a necessary and sufficient condition is always highly non-trivial in various tasks. For ex- ample, for the identification of linear non- non- Gaussian causal structure with latent variables, researchers initially developed sufficient conditions with three pure children in [46], later relaxed to two in 58), before ultimately achieving both necessary and sufficient conditions in M. Similarly, for parameter identification, although the condition we proposed is not a necessary and sufficient one, it could serve as stepping stone towards tighter and ultimately the necessary and sufficient condition for the field.
Below, we also provide a sufficient condition for parameter identifiability that does not rely on structure identifiability in Theorem 5 It is particularly useful when the structure is directly given by some domain experts.
Theorem 5 (Sufficient Condition for Parameter Identifiability (up to group sign) without Requiring Structure Identifiability). In 9, if for every latent variable L there always exist another three distinct variables (which can be latent or observed), such that two of the three are pure children of L and the rest one is a neighbor of L, then the parameters are identifiable.
Identifiabiliy theory often focuses on the asymptotic case, i.e., we assume that know the structure and the population covariance matrix Σx. However, in practice, we only have access to i...d. data with finite sample size and thus only have the sample covariance matrix. Therefore, in the next section, we will propose a novel method to estimate the parameters in the finite sample cases.
Parameter Estimation Method
4.1 Objective
Our goal is estimate F in Definition | given the causal structure and observational data. The key is to parameterize the population covariance Ex using 0 = (F, [2) and then maximize the likelihood of observed sample covariance Σx. To make this technically precise, we provide a closed-fore form expression of Σx in terms of in the following proposition, with a proof given in Appendix A.7
Proposition (Parameterization of Population Covariance). Consider the model defined in Def. 7 Let FLL - Fix - Fxx) -xx)) and N == ((I FLL) Fxx) - Fix)
M == (I Then, the population covariance matrices of and X can be formulated as
Σr TM P. M N No
Ex (x-
Lx Lig + + FFxx +
(2)
The formulations of Dt and Vxx are rather complicated due to the general scenario we considered, i.e. latent variables can be the cause or the effect of latent and observed variables. That is, the submatrices FLL. FLx, FxL and Fxx defined in the above proposition can all have nonzero entries. In most existing works, at least one of these submatrices are assumed to be zero. For instance, factor analysis assumes that FLL. FxL and Fxx are zero, while [32) assumes that FLL and FxL are zero. Furthermore, Proposition [ also provides insight into the indeterminacy involved when identifying the parameters, such as the indeterminacy of variance in Theorem [ and the orthogonal transformation indeterminacy in Theorem 4
Similar to factor analysis 45 7 221, we assume ev are Gaussian and thus X are jointly Gaussian. Thus, the negative likelioodo of observational data can be formulated as
L (t/2)
+ log det Σx),
(3)
where K is the number of i.i.d. observations. With the parameterized negative log-likelihood, we estimate the edge coefficients by minimizing the negative log-likelihood, as
7

---

F. - arg
arg min L, subject to Qee = I, F.
(4)
where the entries of matrix F that do not correspond an edge in 9 are constrained to be zero during the optimization.
Note that in Eq. (4) the constraint that the noise terms of latent variables have unit variance is crucial to deal with the variance indeterminacy defined in Theorem [ In practice, is also favorable to another constraint address the variance indeterminacy. i.e., the constraint that all the latent variables have unit variance. This leads to an alternative objective as
F, F2 = arg min C, subject to (EL)a: = 1, E [m]. F.
(5)
where the entries of F that do not correspond to an edge in G are also constrained be zero.
Both objectives in Eqs. 44 and (5) can be employed, and yet using the second one gives rise to edge coefficients that easier to understand. To be concerete, if we normalize all observed variables to have unit variance, then using Eq. (5) would give rise to such that -1 I Pi. S 1i,i,j E [m]. An realize the constraint in Eq. ( To this end, in the next section we introduce a way to parameterize Ex using F' such that the required constraint in Eq. (5) can be automatically satisfied. Later in Section 5.2 also empirically compare the performance of using Eq. (4) with that of using Eq. (5).
example can be found in Figure 4 However, it may not be straightforward
4.2 Parameterization Trick of Covariance Matrix
In this section, we introduce how trek rules can be employed to parameterize Ex while the unit variance constraint on latent variables in Eq. (5) can be elegantly satisfied. We start with the definition of trek. For readers who are less familiar with treks, please refer to Appendix [. for examples.
Definition 5 (Treks (50). In G. a trek from X to Y is an ordered pair of directed paths (P1, P2) where Pt has a sink X, P2 has a sink Y, and both Pi and P2 have the same source Z, i.e., top (P1, ; P2) Z. A Trek is simple if Pi and P2 have no intersection except their common source Z.
At this point, we are able to parameterize each entry of Ex using (F, (o instead of (F, S2). by making use of the (simple) trek rule [50], follows:
Jij
(6)
where SV,, V,) ) is the set of all simple treks between V; and Vj. and f is the path monomial along P defined as f == I
By this form of parameterization, to requiring all variables to have unit variance), such that the constraint in Eq. (5) can be automatically satisfied. For better understanding of how to use the simple trek rule for parameterization, provide an example as follows.
can simply set all entries of {ar, )
(which is equivalent
Example 6 (Example for Parameterization using Simple Trek). In Figure (a). there are four simple treks between Xa and Xs, as shown in (b). By the simple trek rule and further assuming that all variables have unit variance, the covariance between X4 and Xs, 0a,, can be formulated as f1, 5 Ja,4 J2Ilia22aaas + f2.
5
Experiments
We validate our identifiability theory and parameter estimation method on synthetic and real-life data.
5.1 Setting and Evaluation Metric
We begin with our experimental setting of synthetic data. The causal strength fyy is uniformly sampled from 2- 2] and the noise terms are Gaussian with variance uniformly from [1,5]. 5). We consider 20 graphs. 10 of them should be parameter-ide iddettiiabee up to group sign indeterminacy according to our identifiability theory and we refer to them GS Case (examples in Figure 10 in Appendix). Another 10 should be parameter-identifial we refer to them as OT Case (examples in Figure 11 in Appendix). On average each graph contains 15 variables, 3 out of them are latent. We consider three different sample sizes: 2k, 5k, and 10k. We use three random seeds to generate the causal model and report the mean performance as well the std.
up to group sign and orthogonal transformation indeterninacy and
8

---

Table 1: Experimental result on synthetic data using MSE (mean (std)).
MSE up group sign
MSE up to orthogonal.
Method
Estimator
Estimator-TR
Method
Estimator
Estimator- TR
2k
0.0023 (0.002)
0.0012 (0.0005)
2k
0.0278 (0,008)
0.0355 (0.015)
GS Case
5k
0.0014 (0.002)
0.0005 (0.0005)
OT Case
5k
0.0194 (0.002)
0.0352 (0.012)
10k
0,0012 (0.001)
0,0003 (0.0004)
10k
0.0182 (0.003)
0.0351 (0.015)
(a) MSE up group sign indeterminacy.
(b) MSE up orthogonal transformation.
[ELO smaude
(ES) Lole rad boake
Extraversion
[od]
Openness
[E2] Eaehtak
[eat] saba'' tiee
[oi] trown
[0a] Houre
[010] can
[ES]
[5) condenablee
[C]] teanth
Agreeableness
[oa] othan
(1) troer
L6
[ES]
[AZ]
(A10]
[04]
[AR] Inder
[00] Loundtred
[A3]
[A7] cam
Conscientiousness
HE.E
[N5] Ipp
[A5]
[A6]
[CS Eiendeee
Feth
[CB] ralokinn detes
[C4]
6.2
[N10 butt
[C7] orber
(C3)
[V33
[C]]
[C9] (iabes
[NS] warey
[C10]
(C) Joben Seger niep
Neuroticism
Figure 4: Estimated edge coefficients by the proposed method for Big Five human personality dataset. Variables whose name starts with "I." are latent variables while the others observed variables.
As the optimization in Eq (4 is nonconvex, we will rely 30 random starts and choose the one with the best likelihood. We report the performance of the proposed method with two different objectives. (i) Parameter Estimator with objective defined in Eq. (4). referred to Estimator, and (ii) Parameter Estimator with objective defined in Eq. (5) and Trek Rule parameterization trick in Eq (6) referred as Estimator-TR.
is worth noting that our setting is very general in that we allow latent variables and observed variables to be causally connected in a very flexible way, and consider the identification of parameters of edges that can involve both observed and latent variables. Therefore, to the best of our knowledge, current method achieve the same goal to serve the baseline (which also shows the novelty of the proposed method). As such, we mainly focus on comparing our estimation result with the ground truth parameters. We two MSE-based metrics defined as follows.
MSE up to group sign: suppose the ground truth parameter is F and our estimation is F. The MSE up to group sign is defined as Fl
[F where - takes element wise absolute value, || ||2 denotes
the Frobenius norm and || llo denotes the number of nonzero entries of a matrix.
MSE up to orthogonal transformation: the MSE up to orthogonal transformation is defined as
Fuu - 1 Fix - 1Q FLX I1 + Fxx - fxxQ2113 + Fx | II3 (7)
min Q.Q
||
where the optimization is solved by Adam 27 and the orthogonal matrix Q can be directly parameterized in Py Torch.
5.2 Synthetic Data Performance
We report the performance using synthetic data in Tables la and lb where both our Estimator and Estimator-TR achieve very good identification performance. For example, in the GS scenario with 10k samples, our Estimator achieves 0.0.0012 MSE up to group sign and our Estimator- TR achieves 0.0003 MSE up to group sign. The good performance by Estimator and Estimator- TR not only validates our estimation method, but also empirically verifies our identifiability theory.

---

5.3 Misspecification Behavior
In this section, we show that the proposed estimation method still performs well, even under model misspecification: violation of normality and violation of linearity.
As for violation of normality, we use uniform noise terms for the underlying model, and thus the what extent the proposed method
distribution is not jointly Gaussian anymore. We aim to still recover the correct parameters. The result is shown in Table in the Appendix, which shows even when the normality is violated, lies in that our proposed asymptotic identifiability result holds true, even when we do not assume Gaussianity;
still estimate the parameters pretty well. The reason
only make use of the second-order statistics of the distribution, the additive noise in Definition can follow any other continuous distribution.
To simulate the violation of linearity, we employ the leaky ReLU function during the generation process, as V, = L.RELU(EV
fJx + ev,), (RELU(x)) = max(oz, æ). When a is close to I, the function is close to a linear one, and when a is close to 0, the model is very nonlinear. The result is shown in Table lable[]aadd and we found that our estimation method is quite robust to small violations of linearity. For example, for Estimator- TR in GS case with 10k sample size, we set a = 0.8, we still get a small MSE of 0.001. Even when a decreases to 0.6, the MSE is around 0.005, which is still small. However, when a is decreased to 0.3, the underlying model is considerably nonlinear, and the MSE increases to 0.027.
5.4 Implementation Details, Runtime Analysis, and Scalability
Our code is based on Python3.7 and PyTorch [77. Data is standardized and the optimization in Eqs. 44. (5), and ( are solved by Adam 27 with a learning rate of 0.02. We conduct all the experiments with single Intel(R) Xeon(R) CPU E5-2470. All experiments can be finished within 2 hours. We note that our method is very computationally efficient. First, the computational cost almost irrelevant to sample size: we only need calculate the sample covariance matrix once and cache it for further use during the optimization. Plus, our estimation method can handle large number of variables. For example, the running time of our method are roughly 10 seconds, minutes, and 10 minutes for 20 variables, 50 variables, and 100 variables respectively. For 300 variables, which is a considerably large number for typical experiments considered causal discovery papers, the estimation can still be finished within around one hour.
also worth noting that model misspecifications do not influence the computation cost of our method. We briefly discuss the efficiency of checking whether conditions in Theorem 3 hold, together with what if conditions do not hold in solving real-life problems in Appendices A.8 and A.9
5.5 Real-World Data Performance
In this section, we employ a famous psychometric dataset Big Five dataset https l/openpsyycometrriccc org// to validate our method. It consists of 50 indicators and close to 20,000 data points. There are five dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism (O-C-E-A-N), Each is measured with 10 indicators. Data is standardized. We employ the RLCD method [88] to determine the MEC and GIN (58) to decide the remaining directions. Then we employ the proposed Estimator TR to estimate all the edge coefficients. The structure satisfies the condition in Theorem 5 so we know that the parameters are identifiable.
The estimated edge coefficients are shown in Figure 4 We found that our estimated coefficients are well aligned with existing psychology studies. For example, according to [6 17], being successful in exploratory endeavors depends on the stability pursue them. This is illustrated
in our result where L.5 L + and L3 + 0.39 + L2 indicates that Conscientiousness positively influence openness and Agreeableness positively influences Extraversion. Moreover, has been shown that people are likely to weigh the outcomes of their actions, thus, their level of Conscientiousness coupled with Neuroticism may prohibit them from engaging in risky behaviors (LS (L5-0027 0.27 NN0 100=00 0.12 - L3 0.39 L2) (54]. Such consistency with current psychometric studies again validates the effectiveness of the proposed method in parameter estimation of real-life systems.
Conclusion
In this paper, we characterize indeterminacy of parameter identification and provide conditions for identifiability. Finally, we propose novel estimation method and validate by empirical study.
10

---

7 Acknowledgement
This material based upon work supported by NSF Award No. 2229881, AI Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract R01HL1 and grants from Salesforce, Apple Inc., Quris AI, Florin Court Capital, and the MBZUAI-WIS grant.
References
[1] Jeffrey Adams, Niels Hansen, and Kun Zhang. Identification of partially observed linear causal models: Graphical conditions for the non-gaussian and heterogeneous cases. Advances in Neural Information Processing Systems, 34:22822-22883 2021.
[2] Sina Akbari, Ehsan Mokhtarian, AmirEmad Ghassami, and Negar Kiyavash. Recursive causal structure learning in the presence of latent variables and selection bias. In Advances Neural Information Processing Systems, volume 34, pages 10119-10130, 2021.
[3] Animashree Anandkumar, Daniel Hsu, Adel Javanmard, and Sham Kakade. Learning linear bayesian networks with latent variables. In International Conference on Machine Learning, pages 249-257 2013.
[4] Ankur Ankan, Inge Wortel, Kenneth Bollen, and Johannes Textor. Combining graphical and algebraic approaches for parameter identification latent variable structural equation models. In International Conference on Artificial Intelligence and Statistics, pages 7252-7264. PMLR,
2023.
[5] Rina Foygel Barber, Mathias Drton, Nils Sturma, and Luca Weihs. Half-trek criterion for identifiability of latent variable models. The Annals of Statistics, 5066)331443316 2022.
[6] Elias Bareinboim and Jin Tian. Recovering causal effects from selection bias. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[7] Paul A Bekker and Jos MF ten Berge. Generic global indentification in factor analysis. Linear Algebra and its Applications, 264:255-28 263, 1997.
[8] Carlos Brito and Judea Pearl. A new identification condition for recursive models with correlated errors. Structural Equation Modeling, 9(4):499-474, 2002.
[9] Carlos Brito and Judea Pearl. Generalized instrumental variables. arXiv preprint arXiv:1301.0560, 2012.
[10] Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable causal discovery from interventional data. In Advances in Neural Information Processing Systems, 2020.
[11] Ruichu Cai, Feng Xie, Clark Glymour, Zhifeng Hao, and Kun Zhang. Triad constraints for learning causal structure of latent variables. Advances in neural information processing systems, 32, 2019.
[12] Zhengming Chen, Feng Xie, Jie Qiao, Zhifeng Hao, Kun Zhang, and Ruichu Cai. Identification of linear latent variable model with arbitrary distribution. In Proceedings 36th AAAI Conference on Artificial Intelligence (AAAI), 2022.
[13] David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3(Nov):507-554, 2002.
[14] Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, pages 294 321, 2012.
[15] Philipp Dettling, Mathias Drton, and Mladen Kolar. On the lasso for graphical continuous Зyаpunov models. In Causal Learning and Reasoning, pages 514-550. PMLR, 2024.
[16] Colin G. DeYoung. Higher-order factors of the big five in a multi-informant sample. J Pers Soc Psychol, 11881)))138-11555 2006.

---

[17] Colin G. DeYoung. Cybernetic big five theory. J Res Pers, 56:33-56, 2015.
[18] Xinshuai Dong, Biwei Huang, Ignavier Ng. Xiangchen Song, Yujia Zheng, Songyao Jin, Roberto Legaspi, Peter Spirtes, and Kun Zhang. versatile causal discovery framework allow causally-related hidden variables. In International Conference Learning Representation, 2024.
[19] Mathias Drton, Rina Foygel, and Seth Sullivant. Global identifiabilty of linear structural equation models. The Annals of Statistics, 39(2):665-886, 2011.
[20] Rina Foygel, Jan Draisma, and Mathias Drton. Half-trek criterion for generic identifiability of linear structural equation models. The Annals of Staristics, pages 1682-1713, 2012.
[21] Richard Gorsuch. Factor analysis: Classic edition. Routledge, 2014.
[22] Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 2 edition, 2012.
[23] Patrik O Hoyer, Shohei Shimizu, Antti J Kerminen, and Markus Palviainen. Estimation of causal effects using linear non-gaussian causal models with hidden variables. International Journal of Approximate Reasoning. 49(2):362-378, 2008.
[24] Biwei Huang, Charles Jia Han Low, Feng Xie, Clark Glymour, and Kun Zhang. Latent hierarchical causal structure discovery with rank constraints, arXiv preprint arXiv:2210.01798, 2022.
[25] Yimin Huang and Marco Valtorta. Pearl's calculus of intervention is complete. arXiv preprint arXiv: 6831, 2012.
[26] Yonghan Jung, Jin Tian, and Elias Bareinboim. Learning causal effects via weighted empirical risk minimization. Advances in neural information processing systems, 13:12697-, 2020.
[27] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. JCLR, 2015.
(28) Erich Kummerfeld and Joseph Ramsey. Causal clustering for I-factor measurement models. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1655-1664. ACM, 2016.
[29] Daniel Kumor, Carlos Cinelli, and Elias Bareinboim. Efficient identification in linear structural causal models with auxiliary cutsets. In International Conference on Machine Learning, pages 5501-5510. PMLR, 2020.
[30] Manabu Kuroki and Judea Pearl. Measurement bias and effect restoration in causal inference. Biometrika, 101(2):423-433 437, 2014.
[31] Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient- based neural DAG learning. In International Conference on Learning Representations, 2020.
[32] Dennis Leung, Mathias Drton, and Hisayuki Hara. Identifiability of directed gaussian graphical models with one latent source. Electronic Journal of Statisiccs, 10, 05 2015.
[33] Wang Miao, Zhi Geng, and Eric J Tchetgen Tchetgen. Identifying causal effects with proxy variables of an unmeasured confounder. Biometrika, 105(4):987-993, 2018
[34] Ignavier Ng, Xinshuai Dong, Haoyue Dai, Biwei Huang, Peter Spirtes, and Kun Zhang. Score- based causal discovery of latent variable causal models. In Forty-first International Conference on Machine Learning, 2024.
[35] Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. Advances in Neural Information Processing Systems, 33:1 13:17943-17954 2020.
[36] R. M. O'BRIEN. Identification of simple measurement models with multiple latent variables and correlated errors. Sociological Methodology, 24, 1994.
12

---

[37] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Viiioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[38] Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, NY, USA, 2000.
[39] Judea Pearl. Causal inference in statistics: An overview. Statistics surveys, 3:96-146, 2009.
[40] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: founda- fions and learning algorithms. The MIT Press, 2017.
[41] O. Reiersøl. On the identifiability of parameters in thurstone's multiple factor analysis. Psy- chometrika, 15(2). 1950.
[42] Thomas Richardson and Peter Spirtes. Ancestral graph Markov models. The Annals of Statistics, 30(4):922-1000 2002.
[43] Saber Salchkaleybar, AmirEmad Ghassami, Negar Kiyavash, and Kun Zhang. Learning linear non-gaussian causal models in the presence of latent variables. Journal of Machine Learning Research, 21(9))11224, 2020.
[44] Bernhard Schölkopf. Causality for machine learning. arXiv preprint arXiv:1911.10500,, 2019.
[45] Alexander Shapiro. Identifiability of factor analysis: Some results and open problems. Linear Algebra and its Applications, 70:1-7, 1985.
[46] Shohei Shimizu, Patrik O Hoyer, and Aapo Hyvärinen. Estimation of linear non-gaussian acyclic models for latent factors. Neurocomputing, 72(7-9):204-20077 2009.
[47] Ilya Shpitser and Judea Pearl. Identification of joint interventional distributions in recursive semi- markovian causal models. In Proceedings of the National Conference on Artificial Intelligence, volume 21, page 1219. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2006.
[48] Ricardo Silva, Richard Scheines, Clark Glymour, Peter Spirtes, and David Maxwell Chickering. Learning the structure of linear latent variable models. Journal of Machine Learning Research, 7(2), 2006.
[49] Peter Spirtes, Clark Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.
[50] Seth Sullivant, Kelli Talaska, and Jan Draisma. Trek separation for gaussian graphical models. 1933, 2010.
[51] Jin Tian. Parameter identification in a class of linear structural equation models. In Twenty-Firs First International Joint Conference on Artificial Intelligence, 2009.
[52] Jin Tian and Judea Pearl. A general identification condition for causal effects. Aaaifiaai, pages 567-573, 2002.
[53] Sivan Toledo. Locality of reference in lu decomposition with partial pivoting. SIAM Journal on Matrix Analysis and Applications. 18(4):1065-1081) 1997.
[54] Nicholas A. Turiano, Daniel K. Mroczek, Jan Moynihan, and Benjamin P. Chapman. Big
5 personality traits and interleukin-6: evidence for "healthy neuroticism" in us population sample. Brain Behav Immun, pages 83-89, 2013.
[55] Gherardo Varando and Niels Richard Hansen. Graphical continuous Yyapunov models. In Conference on Uncertainty in Artificial Intelligence, pages 989 998. Pmlr, 2020.
[56] L. L. WEGGE. Local identifiability of the factor analysis and measurement error model parameter. Journal of Econometrics, 70, 1996.
[57] B. Williams. Identification of the linear factor model. Econometric Reviews, 39(1), 2020.
13

---

[58] Feng Xie. Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in neural information processing systems, 13:14891-14990, 2020.
[59] Xun Zheng. Learning DAGs with Continuous Optimization. PhD thesis, PhD thesis, Carnegie Mellon University, 2020.
14

---

A Proofs
A.1 Proof of Theorem 1
Theorem (Indeterminacy of Scaling of ). Consider a model that that follows Def. Twith number of latent variables diagonal matrix, and 0 (F'i,, Fix, FXL, Fxx, Sey Dex). where
1 and = (FLL; FLx, FXL: Fxx, Iey ((x)- Let A be any invertible
FLL A FLLA, FLx A FLX. FXL = FXLA, Fxx- = Fxx, Des = A Dex = Dex
Then, 0 and 0 entail the same observations, i.e,, Exx Ex. Furthermore, we have Et, = A.
FLL FXL
Fix
Ca D and FxL
FLL
Fix Fxx -xx)=
-
B 5
Proof of Theorem 7
Let
Let M
Fxx
and N be matrices defined as in Proposition] | and similarly for M and Ñ. We then have
M
I À B(1 -IIA-B(I-D--- D)-1
=A (1 A
(1-
B(1 D)
MA
and
N ( (I- B)
=
D) -)
D) (
NA.
By Proposition the latent covariance matrix Σi, after rescaling of the parameters is given by
Êτ. -M² M + N T Ñ
-A ()(A
- A NA NA
-A(M" M + N N)A
AA
This implies that the variance of each latent variable L; is scaled by A By Proposition the observed covariance matrix Σx after rescaling of the parameters given by
Σx = (I - D) Bd B B + Dex + Hey ÑB + D" N Dex ) -I-
= =I1 (1 -3 - B) (AELA)(AA
+ Sex + +ex B) (A (A Rix (1- DI- D)-1
= =(I (1- D)- (BTT + Sex + + NB + N (T-))
= Ex.
A.2 Proof of Theorem 2
Theorem 2 (Group Sign Indeterminacy). Consider a model that follows Def. 7 with number of latent variables m 1, 0 = (FLL. FLx, FXL. Fxx, Dcy Dex), and Dt. = T. Let S be diagonal sign matrix (entries are either 11), and f (FLL, FLX, FXI, St,. fex). where
Fr. = SF Fix SFix, Fxt. FxIS, Fxx = Fxx, P.c = Ua. = I, dix = Pix- Then, θ and 0 entail the same observations. i.e. Σx Ex. and (EL) (EL)s, Vi [m]]
15

---

4
42
L
X
X
X
Figure 5: simple graph that satisfies conditions Theorem latent variable, it has no observed variable.
as for each atomic cover with one condition (ii) in Theorem is satisfied in that L1, X5 can be d-separated by X3.
3
X5
X
L
X
X
X=I
X
Figure 6: more complicated graph that satisfies conditions in Theorem and there an atomic cover that has one latent variable and nonzero observed variables, e.g., {L1 Xs} } in the graph. The
FLL Fxt.
Fix Fxx
(
B D
Pi. Fxi.
Fix
Ā C
B D
Proof of Theorem Let
and
Since S
Pxx
is a diagonal sign matrix, we have
A= B=s-'B, Ō= CS, D= D, du = S
and dex = Pex
Note that S' is invertible diagonal matrix. By Theorem S, and thus (Et.)s (Etg),s )i, [m].
1 have Exo Exo and EL
A.3 Proof of Theorem
The structure identifiability part is that if satisfies Condition T and Condition 2 the structure of can be identified up to the Markov equivalence class of O,(O,,G)), which is by Theorem 12 in [18]
Next will focus the proof of parameter identifiability part, i.e., for any DAG in the equivalence class, if (i) and (ii) in Theorem 3 are satisfied, the parameters are identifiable (up to group sign). Without loss of generality, we assume that all variables have unit variance and zero mean. The reason is that if we can show that the parameters are identfiable (up to group sign) under this assumption, then straightforward to show that they also identifiable under the original assumption where Oz. = I.
Lemma 1. Let X, Y be two set of variables, we have ΣXIX== =
Exx - ExY.
Lemma 2. Consider a graph that satisfies (i) and (ii) Theorem 3 For alomic cover V in G with one latent variable, V (L)U k (k could be zero), if il has an observed pure child and the coefficients of the edges from V C, i.e. fL-C. fx- .... fx.-c. are known, then for any variable A. such that A + C and A is not a descendant of C, OLAA can be calculated as (0A.c - -
Proof of Lemma 2 adjacent. By trek rule and the fact that C pure child V, all treks from to A go through V, and thus by the trek rule we have OLAAfi-C
By the definition of atomic covers, all variables in V (X not
dAc--
16

---

Remark: This lemma implies that we can find all edge coefficients of the graph in a bottom-up fashion. Roughly speaking, for a latent variable L that belongs an atomic cover V, once we identify fi-sc, fx,->, ... /x---c - where C is an observed pure child of V, we can take L as if it is observed. More specific explanations can be found the following proof.
Theorem 3 (Sufficient Condition for Parameter Identifiability (up to group sign), Based Structure Identifiabilityy. Assume that G satisfies Conditions 7 ZandZan and thus the structure can be identified up to the MEC of (a(O, (G)). For any DAG in the equivalence class, the parameters are identifiabe. if both the following hold:
(i) For any atomic cover V= =
(ii) If atomic cover V X SUL satisfies [L] F 0 and |X| 2 1, then all simple treks (Def. 5 between L and X do not contain ant latent variables that are not in L.
Proof of Theorrm 3 Consider a graph that satisfies (i) and (ii) in Theorem We first show that if
all the pure children of an atomic cover V are observed, then all the edge coeficiints from the atomic cover its children are identifiable (up to group sign). To this end, we categorize the scenarios into four cases and prove them separately.
(a) V { contains single observed variable. The proof for this case is trivial as the edge coefficient from X to its pure child C is simply ox,c.
(b) V = } contains a single latent variable. By Condition I there must exist C1, C2, and XN. such that C₁, C2 are pure children of V and XN is an observed variable that has trek to V. Then we have
02 = fL 02 OL and C22xx fL
By these three equations, we can solve fi--, and fL--C.- If V has more than two pure children, we can prove the identifiability similarly in a pairwise fashion.
(c) V = (L} {X) (k: 2 1) contains a single latent variable and & observed variables, where V has at least three pure children. We assume that there exist C1, C2, and C3. such that C1, C2, and C3 are pure children V. Let LI(D) = t. this case, we have
(8)
(9)
&GGGIHHH = =tfL-c.fLc.
(10)
By these three equations, we can solve fL-CC,LLCCC.. and fL-C,, with the only remaining free parameter f. other words, we have fL-c, (t), (t), and (t)
Next, we show that Vi = 1, jjj= = .... 3. fx-tc can be identified. Specifically, as all simple treks between L and X contain only observed variables except L, and L and X are not directly adjacent, there must exist X such that X d-separates L and X. Thus, fX-G9% which fx-+, can be solved.
xx//XXxXx>> by
Now we solve I. The key is that all the edge coefficients along all simple treks between X and L can be identified, by using Lemma 2 with only one free parameter f. Thus, can make use of simple trek rule to parameterize Xx,x as a function of f. By Lemma I øL|x = can also be formulated as a function of Lx and thus can be solved. If V has more than three pure children, we just choose all the combinations of any three pure children.
(d) V = (X) (k > 1) contains a single latent variable and k observed variables, where V has two pure children. By Condition I there must exist C1, C2 as the pure children of V. there exists one additional pure child, then it is the same (c). Plus, if there only exist these two pure children, there must exist VN such that VN is a neighbor of V. If VN is observed, let XN = VN. otherwise we recursively take XN the observed pure children of VN.
Let Li In this case, we have
(G = -LL--GLL-
(11)
(c.x/ 11---C,
(12)
(.XX)) =1LL-G,0
(13)
17

---

Similar to (c), by solving the above, we have fL-c, (t) and (t),,
Next, show that Vi = , ..... K,j 1,.,.,,, Jx- can be identified. Specifically, all simple treks between L and X contain only observed variables except L, and L and are not directly adjacent, there must exist X such that X d-separates L and X. Thus, xxx-,%xXXXXXX which /x, can be solved.
xxc|&xxix; by
Now solve f. The key is that all the edge coefficients along all simple treks between X and L can be identified, by using l with only one free parameter t. Thus, we can make use of simple trek rule to parameterize X as a function of By Lemma L can also be formulated as function of Xt,x, and thus & can be solved.
Taking (a), (b), (c), (d) into consideration, for graph that satisfies the conditions in Theorem [ for an atomic cover V the graph, all pure children of it are observed, then all the edge coefficients from V to its pure children can be identified.
Now, we will prove by induction show that, for graph that satisfies the conditions in Theorem 3 for any atomic cover V in the graph, all the edge coefficients from V to its children can be identified, and thus all the edge coefficients of the graph can be identified (the set of edge coefficients the graph is the union of the set of edge coefficients from each V to each V"s children).
To this end, we first index all the atomic covers by the inverse causal ordering, such that leaf nodes have smaller indexes. Then we have sequence of atomic covers V1, i 1, .... C' in the graph, where C' is the number of atomic covers the graph.
(i) We show for Vi. i = 1, all the edge coefficients from V1 to its children can be identified. This is proved by considering (a) (b) (c) (d), V1 's children must be all observed; otherwise it cannot be indexed as 1.
(ii) We show that, for >> 1, if for all Vj, ك < i, all the edge coefficients from Vj to Vj's children has been identified, then all the edge coefficients from Vi to Vi's children can also be identified. This can be proved by combining (a) (b) (c) (d) with Lemma 2 If Vi has children that are latent, then the latent children must belong to an atomic cover with a smaller index. Therefore, as all the edge coefficients from Vj Vj's children have been identified, by the use of Lemma 2 the latent children of Vi can be taken as if they are observed. Therefore, all the edge coefficients from V1 to Vi's children can also be identified.
Taking (i) and (ii) together, all the edge coefficients of the graph can identified.
A.4 Proof of Theorem
Theorem 5 (Sufficient Condition for Parameter Identifiability (up to group sign) without Requiring Structure Identifiability). In G, for every latent variable L there always exist another three distinct variables (which can be latent or observed), such that two of the three are pure children of L and the rest one a neighbor of then the parameters are identifiable.
Proof of Theorem
The proof is special case of (b) in the proof of Theorem
A.5 Proof of Theorem
Lemma 3. Let Ex be the observed covariance matrix entailed by FLL, FLX, FxL, Fxx, Ren,, Let Q be an orthogonal matrix, and
Fr.. FtLQ, Fix = QTT Ftx, PXL. PxIQ, fxx Fxx, d. Q Q, and Dex Pix:
Then, the matrices FiI, Fix, FXL, Fxx, Dey dex also entail the covariance matrix Σx.
18

---

FLL FxL.
Fix Fxx
A
D) and
FLa Fx
Ftx Fxx =
A Ò
B D
Proof of Lemma
Let
Let
M
and N be matrices defined as in Proposition
[ and similarly for M and N. We then have
M = (T (T- Ā -
= -Q- - Q" AQ - (Q" B) (CQ))
- AI B
= QQ
and
Ñ = AIC D) -
= =(Q=Q ((1- -L1-AA (Ll--ACC-HE -1- D) -TT
(1 Q² B)
NQ.
By Proposition the latent covariance matrix Σn is given by
Σ rr M V Ñ
+ NT
= D M + INy
= (MT IT
By Proposition the observed covariance matrix Σx is given by
Σx = (1 D) BI + + NB + B S ( (I-
= DI- (Q"" B) Ω.x + Sex
D D)-1
= -I- D)- B 0xx + Ω.x NB + Bdd N' NT S D)-
=Ex
This indicates that the matrices A, B,C Ō, D, Ūr, and 9ix can also entail covariance matrix Σx.
Using Lemma we can prove Theorem
Theorem 4 (Condition for the Existence of Orthogonal Transformation Indeterminacy). Consider the model in Definition [ If set of latent variables with [L] 2 2, have the same parents and children. then there must exist orthogonal transformation indeterminacy regarding the edge coeffiiienss F. In other words, F can most be identified up to orthogonal transformation indeterninacy.
Proof of Theorem Let St, S2, S3. S4, and S5 be the indices of L, their latent parents, their latent children, their measured parents, and their measured children in G, respectively. Let Q be
) ( For matrices A, B, C, and D from matrix F, suppose that we replace As,,S, As, Cs, and Bs, with As,.s, Q, Q" As, Sy. Cs,,s, Q, and Q² Bs..s, respectively. Then, we will show that the entailed covariance matrix Σx is unchanged.
FiL
FLx
(A
and FiL
Fix
[L| x LL orthogonal matrix. Let
FxL
Fxx
C
FxL
Fxx
C D
Let U be m orthogonal matrix such that: (i) Us St Q. (ii) the remaining diagonal entries are ones, and (iii) the remaining non-diagonal entries are zeros. Let
Á = AU, B== BT= Č= CU, D D. Ω., UTS U =1, and dex = Dex
19

---

By Lemma 3 the matrices above can entail the same covariance matrix Σx.
By construction of U, left multiplication of U B only affects es specifically, it is equivalent to replacing , with Q" .s Furthermore, only the columns of Sş in Bs,,, will be affected, because those columns correspond to the measured children of L. Therefore, all entries of B are the same as B, except that Ss, is replaced with Q² S, Similar reasoning shows that all entries of C' are the same C, except that Cs,,S, is replaced with Cs
Now consider U' AU. By the reasoning above, left multiplication of U on A only is equivalent to replacing As, S, with Q" Ss, Further right multiplication of U on U A is equivalent to replacing (U" A)s,,s, Sn with (U" A) Since S₁, S2, and S3 are mutually disjoint, all entries of = U2 AU are same as A, except that As,.s, and As are replaced with As, Q and Q" Ss, respectively.
Hence, for matrices B,C, and D, suppose we replace As. As, Sa. Ss. CS.Ss,, and Bs with ss Q. Q" As, Cs,,s, s, Q, and Q" Bs, respectively. By the reasoning above, this is equivalent to replacing A, B, C, and D with A, B, C, and D, respectively, which (generically) share the same support and entail the same covariance matrix Σx.
A.6 Proof of Corollary and Corollary 2
Corollary 1 (General Necessary Condition for Parameter Identifiability). For parameters to be identifiable, every pair of latent variables has to have at least one different parent or child.
Proof of Corollary 7 Proof by contradiction. If it is not the case that every pair latent variables has to have at least one different parent or child, then there exist L such that [L] 2 2 and L share the same parents and children. Therefore by Theorem there must exist orthogonal transformation indeterminacy regarding F, and thus the parameters are not identifiable.
Corollary 2 (Necessary Condition about Atomic Covers for Parameter Identifiability). Assume G satisfies Conditions 7 and and thus the structure can be identified up to the MEC of (G)). For any DAG in the equivalence class, for G parameters to be identifiable, every atomic cover must contain no more than one latent variable.
Proof of Corollary 2 Proof by contradiction. If for a DAG in the equivalence class, there is an atomic cover that has more than one latent variable, then according the definition of the concerned equivalence class, the latent variables in that atomic cover share the same parents and children. Then by Theorem there must exist orthogonal transformation indeterminacy regarding F, and thus the parameters are not identifiable.
A.7 Proof of Proposition 1
Proposition 1 (Parameterizaation of Population Covariance). Consider the model defined in Def. 7 Let M == (I
FiL - Fix Fxx) FxL) and N == ((I FLL) Fxx) - Fix -1
Then, the population covariance matrices of L and X can be formulated as
Σg =M² Dey MM MNN N
(1)
Σx = - -x-x)x
F Ing FLX + Dox + +5xx
NFxx + F. ST
(T- Fxx) (2)
FiL
Fix Fxx
A B C
Proof of Proposition
Let F
D)
FxL
Since matrices and D are invertible, using the formula of 2 x 2 block matrix inversion 22 Chapter 0.7], we obtain
M
(1 (-F)-1- (-(I-D)-'CM (1-
20

---

which implies
MT (1-))- (-1)) (1- 1-D===((-DD-TMMMCC11--
(I
and
M (I-F) (-1- D)- (11- MT (I- + (1- (I- (1- -T-xxx (1-F)
-1 -TΩ
We then have
Στ = 2 M + A CT - (1
= - + N
and
Σx =
(1 (1- --0---000000-------0000000------ Six - D)
+(I- +I- 11
(1 -1- 11x +B" M M² MB MB+
+ 02x -) CMMB+ B C*
+ MBTMTC -ID - DD (1- (CMB)1
(1- (I- D) Dex + B Pex B² NTDOX )1 D)-1
We now discuss how M and N defined in Proposition I are invertible. Note that matrices D and T- F are invertible because structure is acyclic. This implies det(T F) * 0 and det(T D) + 0. Define
9) .
U
-11 (1DD---0 DC-
which implies
M B
- =
0 I-D)
and thus
det((T- - -F)U)= = det(M)) - -D)-
Since det(U') = 1 and det(I F) * 0, have
det((I-F)) - F) det([ - F) det(U)) 40.
By the statement above and det(1 - D) * O. we have
det(M) det((I - FJU) dct([I D) #0,
which implies that M is invertible. Similar reasoning can be used to show that N is invertible.
A.8
Computational Cost Checking Whether the Conditions Theorem Hold
Here we want to investigate, given structure, can we efficiently check whether the proposed sufficient conditions hold? this end, we generate random graphs and each graph has 100 variables. According to our empirical result, such check can be done very efficiently. Specifically, on average, given structure with 100 variables, only takes our Python code around seconds to check whether the conditions hold.
21

---

A.9 In Practice, What If the Conditions Do not Hold?
Our condition is useful solving real-life problems. For example, in the psychometric study, we can properly design the questions with domain knowledge following the condition in Theorem [ such that each single latent variable has enough observed variables as pure children and thus it can be ensured that all parameters are identifiable (as illustrated in our real-life data result in Figure 4
On the other hand, even though sometimes the questionnaires and data such that the conditions are not satisfied for the identification of parameters, our Theorem i still useful. In this case, we can still make use of our conditions to check the given structure, and find some local sub-structures where our conditions are satisfied. Consequently, it can be ensured that all the parameters of some sub-structures are identifiable, and we can employ our estimation method to find all the edge coefficients of these sub-structures.
designed not well
B Additional Definitions, Graphs, Results, and Examples
B.1 Example of Treks
Example 7 (Example of Treks). Figure (a), there are four treks between X4 and Xs: (i)
X4 4 Xs, (ii) X4 X3 Xs, (iii) X4 L X2 - X3 Xs, and (iv) X4 X3 X2
L1 + Xs. illustrated in Figure (b).
B.2 Definition of Pure Children
Definition 6 (Parents, Children, and Descendants of Set of Nodes [18] For set of nodes X in G, = Paxexas and Deg (X) = UXexDee(X))
we have Chs(X = Uxex (X) Pag(X) Definition 7 (Pure Children of Set of Nodes (18). Y pure children of set of nodes X in graph G, i.e., Y & PChg (X), isf all of the following hold: (i) XnXY Y = 0. (ii) Y C Cha (ii)) Pag(Y) (Y) = X, and (iv) Deg Y)nX G.
B.3 Definition of Neighbor and MEC
Definition (Neighbor). G, nodes X and Y are neighbor of each other iff there exist an edge from X to Y or an edge from Y to X.
Definition (Markov Equivalence Class (MEC)). Two DAGS 91 and G2 belong to the same MEC, iif they share the same skeleton and v-structures.
B.4 Definition of Rank-invariant Graph Operator
The definitions Definition 10 (Skeleton Operator [18). Given an atomic cover V in a graph G, and let S be the set of all atomic covers in G such that for all S E S, S V. For all V1 E V and all V2 E PChs (V))(uess PS), V1 and V2 are not adjacent, draw an edge from V1 to V2. We denote such an operator as skeleton operator Og()).
follows with examples.
Definition 11 (Intra atomic operator). For every atomic cover V in structure G, i 2 2, we add edges between elements V such that edges among form a fully connected DAG. We denote such an operator as intra atomic operator Og(G).
Example 8. Suppose the original graph Figure 9 (a). After the skeleton operator, we have O,(9), which is shown in Figure] (b). After the intro atomic operator, we have (O, which shown (
B.5 Graphs for Synthetic Data Experiments
Please refer to Figures 10 and
B.6 Additional Result under Model Misspecification
Please refer to Tables and
22

---

X
(a) Graph 9 to show treks.
(b) The four simple treks between X4 and in (a).
Figure Illustrative figure to show how to parameterize Ex, by the use of simple trek rule.
L
4
L
Xn
X
X
X
X
X15
X16
X
X
Figure 8: An illustrative graph to show orthogonal transformation indeterminacy. An atomic cover of it, {L,, L2), has more than one latent variable, and thus there exists orthogonal transformation indeterminacy regarding coefficients of edges that involve {L1, L2).
C Other Discussions
Our optimization problem Eq. 4 is solved by gradient descent using PyTorch. Our current implementation is based CPU but it can be further accelerated by using GPU. very related discussion can also found 335
The optimization problem in Eq. 5 is solved by gradient descent, which involves evaluating the LogDet and matrix inverse (for the gradient) terms (which similar to continuous causal discovery methods based Gaussian likelihood [35)). According to [53] the computational complexity is O(td), where the number of variables and is the number of iterations of gradient descent respectively. Note that the computational cost is largely independent of the sample size as we only need to calculate the sample covariance once and save it for further use.
It is possible to perform inference on the learned parameters in our framework. To be specific, as we use maximum likelihood estimation for the parameters, some standard techniques can be readily used. For example, bootstrap can be employed to provide standard errors on linear coefficients and Chi-square can also be done to examine the fitness of the model.
D Extended Related Work
One main line of research in latent variable estimation centers on sictor-analysis-oan based methods. Representative studies include [41 45
7 [7]. Various other techniques have also been employed latent structure and parameter identification, including over-complete ICA-based techniques [23 43 I that leverage non-Gaussianity and matrix decomposition--asee approaches 3 However, these approaches typically consider latent variables with observed children, without considering parameter identification in latent hierarchical structures. A more related work is (4, but it considers a much simpler structure.
36
56
Another direction would be to project the graph to an ADMG and the latent confounding effects are encoded by correlated noise terms. Following this idea, graphical criteria such as half trek (20, [5]. G-criterion [9]. and some further developments [51] 29 42 19 has been proposed. Furthermore, another line works involve studies on causal effect estimation in the presence of latent confounders [52]
30 33 26 which often rely on instrumental variables or proxy variables for identification. Notice that this task, the parameters may not identified [30]. although the causal effect from the
treatment variable to the outcome variable can be identified.
23

---

X3
X
h1
L2
X
X5
L
L2
X4
X5
L1
L2
X
X5
X
X7
Xg
X
X
X7
X
X
X
X7
Xg
X₃
(a) Graph
Og))).
(c) (O)
Figure 9: Example to illustrate graph operators O and O..
Table Performance under violation of normality using uniform noise terms in MSE (mean (std)).
Metric
MSE up group sign
Method
Estimator
Estimator- TR
2k
0.0017
0.0005
GS Case
5k
0.0018
0.0004
10K
0,0018
0.0003
Furthermore, several existing works also solve an optimization problem that involves parameterization of maximum likelihood, such as those in continuous optimization for causal discovery [35] 59] 31 10
34 and parameter estimation of Lyapunov models [55] Differently, our formulation involving likelihood parameterization aims to estimate parameters of partially observed linear causal models.
E
Limitations
One limitation of this work that our theoretical results are based on the assumption lincar gaussian causal models. When data is not linear gaussian, we have also conducted experiments to the performance of our method. It turns out that our method still performs well in the presence of certain extents of violation of normality and linearity. However, theoretical analysis under violation of linearity and normality would be interesting and the focus future work.
Broader Impacts
The goal of this paper is advance the field of machine learning. We do not see any potential negative societal impacts of the work.
24

---

Table 3: Performance under violation of linearity using leaky relu in MSE (mean (std)).
Metric
MSE up to group sign
Method
Estimator
Estimator TR
0.8 (close linear)
0.004
0.001
Case with 10k sample size
0.6 (quite nonlinear)
0.013
0.005
a
0.3 (very nonlinear)
0.046
0.027
4
4
X
la
X
X
X
X12
%
X;
Xa
X
X10
X
X
X
X,.
X15
X
X
Illustrative graph G1-
(b) Illustrative graph G2
4
X
X
X
La
L
K,
X
X
X
X
Xx1
X12
%
Xa
X
X
X
X12
(c) Illustrative graph Gs.
(d) Illustrative graph G.
X
A
(e) Illustrative graph Gs
Figure 10: Examples of graphs in the GS case. The parameters of them are identifiable up to group sign indeterminacy.
25

---

X
L
L2
X
X
X
X
X,
X
(a) Illustrative graph Go.
(b) Illustrative graph 97
X
X,
X
L
Us
L
4
X5
X
X
X
X,0
X,1
X
X
X
X,,
X
X
X,
X
X17
(c) Illustrative graph Ge-
(d) Illustrative graph Go
X
Lz
La
L
X
X
X
X1o
X
X12
X
X1S
X15
X
X,7
X
(c) Illustrative graph Gzo
Figure 11: Examples of graphs in the OT case. Parameters of them are Identifiable up to group sign and orthogonal transformation indeterminacy.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the claims stated in the abstract and introduction are the same what stated the theory and method part.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
discussion about the limitations can be found in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

All assumptions together with necessary definitions are provided in the main text and all complete proofs are provided in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The implementation details and the experimental settings are all provided in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
The code and the data will be publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipsscc//pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa///giiieee/eeeeebiiisssssss /Cc////oddiiisssseeeeeee ssinsoliccyyyyycyccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See the experiment section for the setting and details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The synthetic experiments were supported by error bars to show statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section S
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurrpps.co /ftiiiiiiittt///pppppp.o//aaa/aa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the
NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: is not related to this work. No such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the assets are properly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "DigIbYNSYw", "reviewer": "Reviewer_WKpu", "summary": "This paper introduces conditions under which DAGs can be recovered in the linear case where some nodes are observed and some are not. This DAG recovery involves computing edge weights between nodes in a causal graph. Nodes are allowed to be latent or observed, with varying types of edge weight indeterminacy depending on the structure of the causal system in question.", "questions": "- What are the implications of the diagonal covariance matrix $\\epsilon_{\\mathbf{V}_{\\mathcal{G}}}$? \n\n- Regarding, \"As variables are jointly Gaussian, asymptotically our observation can be summarized as population covariance over observed variable\". Wouldn't this statement also apply in finite samples under Gaussianity?\n\n- A major benefit seems to be identification of edge weights. If actual interpretation of the edge weights is going to be done in practice, group sign indeterminancy would limit applicability. Would it help to anchor the sign of one edge based on prior science? Guidance? \n\n- In Figure 4, are circulate nodes latent and nodes denoted by [LetterNumber] observed? If so, clearly articulate this in the figure label.", "limitations": "I see no major negative societal impacts.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "There is certaintly interest in estimating DAG structure in practice. The work here presents useful results for how and when that estimation may or may not occur given the structure of the causal system. The extension to latent variables is a contribution, although in practice, latent nodes might increase the already difficult task of interpreting inferred DAG structure. The misspecification analysis is useful and the presence of examples in the text helpful (with a few caveats below). \n\nI like the title. The framing is at its strongest when the paper articulates the general conditions under which identifiability can and cannot be achieved. As for whether a given causal system in practice meets assumptions for strongest identifiability is in the end, to my eyes, a very difficult question. \n\nOverall, the paper is a solid contribution, although I believe it could be improved (see below).", "weaknesses": "The text overall is generally well-written, with some caveats listed below. In my reading, the first half of the paper read more clearly than the second half. For example, I couldn't quite piece together from the discussion of estimation whether the estimated graph will be dense (all nodes connected to all other nodes, given the [seemingly?] continuous optimization being done, e.g., Eq 3. If all edges are connected to all other edges, then the relative usefulness seems to be weakened in that usually, investigators seek out a parsimonious representation of a causal system. \n\nI was also wondering what more established methods would yield as an empirical baseline (e.g., PC algorithm); currently, the Estimator-LM (estimator with Lagrange multiplers) is articulated as a baseline. This is one of the methods introduced in the paper. An external state-of-art baseline would be most informative. The authors state that \"no existing method...can achieve the same goal as ours.\" If this is because of the latent variable aspect, one could in principle restrict the MSE calculation to edges among observed nodes. In other words, perhaps there isn't a perfect analogue method but an imperfect comparison could be better than none. We could also get a visual comparison of the DAG among observed variables used in Figure 4 from some existing baseline methods for the Appendix. \n\nFinally, there is little discussion of uncertainty estimation. Uncertainty estimation in the observed DAG recovery case is hard, even more so here (presumably). \n\nThere are points where the text could, to me, use more clarity in the discussion: \n\n- Condition 1 seems relatively minimal and even intuitive. It seems very hard to know in practice if Condition 2 (line 193) holds or is even a minimal or very restrictive condition.\n\n- I appreciate the author(s)' inclusion of Example 2 and Example 3. I think the logic could be made clear, perhaps with additional shadings or labelings that would help us see which sets of nodes and edges are doing what work regarding Condition 1 and Condition 2. \n\n- I would make the \"pretty close\" language in 218 and 255 a bit more precise. Also, starting line 264. I would revise this from, \"it has considerable extents of necessity, and could be expected to serve as a stepping stone towards tighter and ultimately the necessary and sufficient condition for the field.\" to something that also is seomwhat more precise. Also, there is no guarantee that necessity+sufficiency will be found (or perhaps there is an impossibility), so would hedge this possibility somewhat. \n\n- Can you define what it means for \"QF and F\" to \"share the same support\" in this case? \"Support\" is often defined in causal inference settings as an event probability falling between 0 and 0; here, F is defined as the matrix embodying the causal edge coefficients, so I believe what is meant here is that QF and F do not share the same set of non-zero entries. Clarifying this would be helpful. I would also consider beginning with the example of indeterminacy before defining it to help the reader see your point intuitively before the formalization.\n\n- I would definition 7 into the main text. It is an important definition used multiple times and without it, it is hard to follow the atomic cover discussion. it's also a very short definition.\n\nMoreover, I would in general help the reader along by first explaining the concept before formally/technically defining it. Examples: \n\n- The term structure identifiability and parameter identifiability should be clearly defined before the terms are used. (I don't think I see a clear definition before use currently; I would move the paragraph beginning on line 171 up in the text, as it is a clear articulate of the point.) In a similar vein, I would explain what atomic covers are going to do before we jump into the definition on line 164. \n\nOther details would help this reader: \n\n- The MSE up to orthogonal transformation is an interesting metric. Some mention of how this optimization is done would be helpful.\n\n- I would add a sentence explaining whether GPU acceleration would or would not be helpful and why.\n\nI also noted several minor points listed here: \n\n- References used are inconsistent at times. Sometimes, we see reference to \"condition (i)\", others to Condition 1. \n\n- Line 212. Missing space \"identified upto the\" should read \"identified up to the\". This same occurs in line 140 (\"upto\" should read \"up to\") and in other parts of the text as well. \n\n- Line 147. \"indetermincay\" should read \"indeterminacy\". This typo occurs a few times in the text. \n\n- Line 129. Clarify what is meant by \"entails the same observation as that of...\" This also appears in line 142 (\"Entails the same observation\"). I assume this means something about implying the same probability distribution, but helping the uninitiated reader is usually appreciated. \n\n- Line 137. \"However, if we set f1,2 = 0, then the parameters are not identifiable. These rare cases of parameters are of zero Lebesgue measure so we rule out these cases for the definition of identifiability\" -> I would just say \"these presumably rare cases of parameters\". Probably some justification is needed to articulate why this should be rare in real causal systems. Does any prior literature speak to this?\n\n- Line 72. I believe \"the causal edge coefficient of the model\" should read \"the causal edge coefficients of the model\".\n\n- Capitalize \"Gaussian\" on line 286.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time dedicated to reviewing our paper, the insightful comments, and valuable feedback. Please see our point-by-point responses below.\n\n**Q1:** Whether the graph will be dense regarding Eq.3?\n\n**A1:**  We note that Eq.3 concerns the estimation of causal coefficients F, given the data and the structure G, and the entries of F that do not correspond to an edge in G are constrained to be zero during the optimization (as in lines 289-290). As the graph is given, sparsity constraints are not needed in Eq.3, which is in contrast to continuous-optimization-based structure learning methods such as NOTEARS.\n\nOn the other hand, you are totally right that we do expect that the given structure is parsimonious/sparse. For example,\nif a graph contains latent variables and all variables are fully connected, then according to our identifiability theory the parameters are provably not identifiable. Roughly, a sparse/parsimonious graph has a better chance of satisfying the condition for parameter identifiability.\n\n**Q2:** Is it possible to restrict the MSE calculation to edges among observed nodes to compare with any baseline that does not allow the presence of latent variables?\n\n**A2:** Thank you for your insightful suggestion. Yes, it is possible to \nrestrict MSE to edges among observed to compare with methods that do not allow latent variables. \nFor example, GES with BIC scores can be considered as it estimates all the linear edge coefficients during the calculation of likelihood. \nIn light of your suggestion, we conduct additional experiments to compare with GES using your proposed restricted version of MSE. Specifically, GES achieves restricted MSE of 0.41, 0.35, and 0.35 with 2k, 5k, and 10k sample size, while our Estimator-TR achieves 0.008, 0.002, and 0.001, respectively.\nIt is observed that, even when we restrict the MSE to edges among observed variables, methods that do not allow the presence of latent variables cannot perform well. The reason is that,\nif two observed variables have latent confounders, we cannot correctly recover the edge coefficients among them without explicitly modeling/considering the existence of latent variables. \nWe thank the reviewer again for the valuable comment and have added the additional result together with this discussion to the appendix.\n\n**Q3:** Discussion of uncertainty estimation?\n\n**A3:** Many existing works on the parameter identification problem focus on the asymptotic identifiability [18,19], and thus our theoretical analysis follows this spirit,  because this type of asymptotic identifiability result is needed before one can provide results on uncertainty estimation.\nAt the same time, uncertainty estimation is certainly possible under our framework. \nTo be specific, as we use maximum likelihood estimation for the parameters, some standard techniques can be readily used. For example, bootstrap can be employed to provide standard errors on linear coefficients. Chi-square test can also be done to examine the fitness of the model. Thank you for the valuable suggestion and we have added this discussion to our revision. \n\n**Q4:** What does it mean by QF and F share the same support.\n\n**A4:** Your understanding is correct. It means QF and F share the same set of non-zero entries. We have added this sentence to our revision as you suggested.\n\n**Q5:** Some mention of how MSE up to orthogonal transformation is done would be helpful.\n\n**A5:** MSE up to orthogonal transformation is calculated by solving the optimization problem in line 350. We use PyTorch with SGD to solve this problem where an orthogonal matrix Q can be directly parameterized and optimized. We thank the reviewer and have added it to our revision.\n\n**Q6:** GPU acceleration?\n\n**A6:** Yes, our optimization problem in Eq.3 is solved by gradient descent using pytorch (or any other automatic differentiation framework) and it can certainly be further accelerated by using GPU. A very related discussion can also be found in [33].\nOur current implementation is based on CPU as it is already fairly fast - it only takes around 2 minutes to handle a quite big graph that has 50 variables (as discussed in section 5.4). Thank you for the valuable suggestion and we have added this discussion to our revision.\n\n**Q7:**\nRegarding the rare cases of parameters, probably some justification is needed to articulate why this should be rare in real causal systems. Does any prior literature speak to this?\n\n**A7:** This is the typical setting in parameter identification literature where generic identifiability is concerned [5,19] (a similar spirit is shared by causal discovery literature that assumes faithfulness [48]). \nThese cases are rare in the sense that they have Lebesgue measure zero.\nIn real causal systems, if all the edge coefficients are randomly generated (from an absolutely continuous distribution), these cases are not a concern because they are of Lebesgue measure zero. At the same time, if edge coefficients in a causal system are deliberately or adversarially designed, then identifying these parameters using only observational data can become extremely difficult (similar to violation of faithfulness assumption in causal discovery in the sense that typical constraint-based algorithms would fail). \n\n\n**Q8:** Implications of the diagonal covariance matrix $\\Omega$ of $\\epsilon_{\\mathbf{V}_\\mathcal{G}}$\n?\n\n**A8:** Do you mean why it is diagonal? As our framework explicitly models all the latent variables, all the \n$\\epsilon_{\\mathbf{V}_i}$ \nare mutually independent, and thus $\\Omega$ is diagonal (in contrast, the ADMG framework does not explicitly model latent variables, and thus their $\\Omega$ is not necessarily diagonal).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TJyUf08NCe", "reviewer": "Reviewer_Ljpg", "summary": "This paper investigates the parameter identifiability of partially observed linear causal models, focusing on whether edge coefficients can be recovered given the causal structure and partially observed data. It extends previous research by considering relationships between all variables, both observed and latent, and the coefficients of all edges. The authors identify three types of parameter indeterminacy in these models and provide graphical conditions for the identifiability of all parameters, with some conditions being necessary. A novel likelihood-based parameter estimation method is proposed to address the variance indeterminacy of latent variables, validated through empirical studies on synthetic and real-world datasets, showing the effectiveness of the proposed method in finite samples.", "questions": "I have  few questions and comments for the Authors:\n\n*  Line 139: The authors mention that the indeterminacy of group sign is rather minor. if the parameters are identifiable only up to group sign indeterminacy, we still say that the parameters are identifiable. It would be useful to explain why group sign indeterminacy is a minor issue. It might actually depend on the underlying task, and there might be application scenarios where it is not insignificant. Some details on this would be useful in the main paper.\n\n*  The additive noise is assumed to be Gaussian. Suppose the noise follows another continuous distribution such as Gamma or Student-t distribution. Can we use the identifiability results? If not, what is special about the Gaussian noise here compared to other continuous noise distributions?\n\n* Referring to Proposition 1, some matrix inverses need to be calculated to compute the noise covariance matrices. Will the inverses always exist? If yes, can you elaborate on how? If no, how does that impact the application of Proposition 1?\n\n* In the experiments section, the authors demonstrate the effect of sample size on the mean squared error (MSE) for the parameter matrix \\( F \\). It would be useful to also show how accurate the estimates of the noise covariance matrix depending on the sample size used.", "limitations": "The authors clearly describe the problem setting and assumptions. I have mentioned the main limitation of the proposed method in the weaknesses section. I don't think there are any potential negative societal impacts of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors consider the problem of parameter identifiability in partially observed causal models with linear structural equations and additive Gaussian noise. This has not been studied in a similar way before in the literature, especially when observed and latent variables are allowed to be flexibly related. Specifically, observed variables are allowed to be parents of unobserved variables. The authors also provide graphical conditions that are sufficient for all parameters to be identifiable and show that some of these conditions are provably necessary. To address the scenario where noise covariance matrix is unknown, they propose a novel likelihood-based parameter estimation method and validate it with empirical studies on synthetic and real-world data.", "weaknesses": "The paper considers a restricted setting, allowing only linear relations between observed and unobserved variables and restricting the noise to be Gaussian. The theoretical guarantees hold under the scenario when the noise covariance matrix is unknown. Nonetheless, the authors mitigate this by providing an estimation scheme for the noise covariance matrix from limited data samples. The conditions of identifiability proposed are not fully necessary and sufficient; this is left for future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time dedicated to reviewing our paper, the insightful comments, and valuable feedback. Please see our point-by-point responses below.\n\n**Q1:**  Regarding group sign indeterminacy.\n\n**A1:** \nThe group sign indeterminacy is rather minor with reasons as follows. (i) In practice, we can always anchor the sign of some edges according to our preference or prior knowledge in order to eliminate the group sign indeterminacy.\nFor example, in Figure 4, \nif we expect that L2 should be understood as Extraversion instead of non-Exterversion, we can add one additional constraint during our parameter estimation such that the edge coefficient from L2 to E1 (\"I am the life of party.\") will be positive (as we believe E1 should be positively related to Extraversion). \n(ii) On the other hand, there are some application scenarios that are not influenced by the group sign indeterminacy,\nsuch as causal effect estimations between certain variables.\n\n\n**Q2:** What if the additive noise follows another continuous distribution?\n\n**A2:** Thank you for the insightful question. If we do not assume Gaussianity, the proposed asymptotic identifiability result still holds. The reason lies in that we only make use of the second-order statistics of the distribution and thus the additive noise can follow any other continuous distribution.\nWe thank the reviewer for the insightful question and have revised to make it clear in our revision.\n\n**Q3:** Will the inverses in Proposition 1  always exist?\n\n**A3:**  Yes, they always exist. \nThanks for asking this question which helps improve the clarity of Proposition 1. In light of your suggestion, we have updated Proposition 1 to state that all matrix inverses exist. We have added the proof to our revision with a sketch as follows.\n> *Sketch of proof.* Note that matrices $I-D$ and $I-F$ are invertible because structure $\\mathcal{G}$ is acyclic. This implies $\\det(I-F)\\neq 0$ and $\\det(I-D)\\neq 0$. Define\n$$\nU=\\\\begin{pmatrix}\nI & 0 \\\\\\\\\n-(I-D)^{-1}C & I\n\\\\end{pmatrix},\n$$\nwhich implies\n$$\n(I-F)U=\\\\begin{pmatrix}\nM & B \\\\\\\\\n0 & I-D\n\\\\end{pmatrix}\n$$\nand thus\n$$\n\\det((I-F)U)=\\det(M)\\det(I-D).\n$$\nSince $\\det(U)=1$ and $\\det(I-F)\\neq 0$, we have\n$$\n\\det((I-F)U)=\\det(I-F)\\det(U)\\neq 0.\n$$\nBy the statement above and  $\\det(I-D)\\neq 0$, we have\n$$\n\\det(M)=\\frac{\\det((I-F)U)}{\\det(I-D)}\\neq 0,\n$$\nwhich implies that $M$ is invertible. Similar reasoning can be used to show that $N$ is invertible.\n\n**Q4:** It would be useful to also show how accurate the estimates of the noise covariance matrix are.\n\n**A4:** We note that once the edge coefficient matrix $F$ is determined, the noise covariance matrix $\\Omega$ can also be uniquely determined - we have  $(I-F^T)\\Sigma_{\\mathbf{V}}(I-F)=\\Omega$ where the left hand side only depends on $F$ when variables have unit variance. Thus, a small MSE of $F$ usually implies a small MSE of $\\Omega$. In light of your suggestion, we conducted additional experiments to empirically validate this point: under the GS case, the MSE of $\\Omega$ using our Estimator-TR are 0.003, 0.001, 0.0004 with 2k, 5k, and 10k sample size, respectively. We thank the reviewer for the valuable question and have included this additional result in our revision.  \n\nWe genuinely appreciate the reviewer's effort and hope that your concerns/questions are addressed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HxKOTM8IhE", "reviewer": "Reviewer_PaWo", "summary": "This paper investigates the problem of parameter identification in linear causal models, which is important and well-studied task in causality. The authors examine models that explicitly include both observed and latent variables. The identification of parameters in such models has not been studied so far and the authors are the first to formulate this problem and provide the first results in this regard. The main achievements of the paper are the sufficient and necessary conditions for parameter identifiability.  Moreover the authors conducted empirical studies on both synthetic and real-world data to validate the proposed methods.", "questions": "Please provide the motivation for considering edges / parameters from observed to latent variables and edges between latent variables.\n\nSee also my questions above.\n\nIn Line 69: explain that d=n+m", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Thought causal structure learning in the presence of latent variables has been well studied in the literature, the parameter identification in models that explicitly include both observed and latent  variables -- as presented in the submission -- is new. The paper provides non-trivial sufficient and necessary graphical conditions for parameter identifiability and present them in the context of known graphical conditions for structure identifiability provided in [17].", "weaknesses": "The sufficient condition for parameter identifiability assumes that G, in addition to conditions (i) and (ii) in Thm. 1, satisfies conditions 1 and 2 presented in section 3.2. I agree that conditions 1 and 2 imply that the structure G can be identified (as shown in [17]) however the  sufficient conditions formulated in this way are overall very restrictive. It would be interesting to have sufficient conditions also in the case of structures which do not satisfy conditions 1 and 2: It can happen that G can be identified even if it does not satisfy 1 and 2 or the structure is provided by a researcher / theory. The authors do not discuss if there exist cases which can be identified but which do not satisfy 1 or 2.\n\nAlso, it is not clear what is the gap between the sufficient and necessary conditions.\n\nThe next issue is that the authors do not discuss what is the computational complexity of parameter identification in linear causal models, that explicitly include both observed and latent variables.\n \nIt is not clear to what extent the sufficient and necessary conditions proposed in Section 3 are useful for (numerical) parameter estimation discussed in Section 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time dedicated to reviewing our paper, the insightful comments, and valuable feedback. Please see our point-by-point responses below.\n\n**Q1:** It would be interesting to have sufficient conditions also in structures that do not satisfy conditions 1 and 2: G may be identified even if it does not satisfy 1 and 2 or the structure is provided by a researcher.\n\n**A1:** Thank you for the insightful question. We aim to explore conditions such that the whole causal model can be fully specified from observational data, and thus Conditions 1 and 2 have to be considered for structure identifiability. These conditions are not overly restrictive; rather, they are currently the most general or less restrictive ones for structure identifiability involving latent variables in the linear Gaussian setting, to our best knowledge [17].\n\nAt the same time, we also agree that in some cases the structure might be directly given by domain knowledge or experts, and the parameter identifiability for these cases can be interesting. We genuinely thank the reviewer for the suggestion and have added a related discussion to our revision, which can be summarized as follows. If the structure identifiability is not a concern (e.g., when G is given by an expert), a weaker sufficient condition for parameter identifiability can be used: Condition 1, along with conditions (i) and (ii) in Thm 1, are sufficient for parameter identifiability, and Condition 2 is not required.\n\n**Q2:** The gap between the sufficient and necessary conditions?\n\n**A2:** Theorem 1 provides a sufficient condition for parameters to be identifiable. The gap between this sufficient condition and a necessary and sufficient condition is further analyzed in Remark 1. Specifically, the sufficient condition in Thm 1 consists of two parts: condition (i) and condition (ii). In our paper, we have shown that condition (i) is provably necessary, meaning the gap arises only from cases that fall outside of condition (ii) yet remain identifiable. Characterizing these cases is highly challenging as it may involve tools from algebraic statistics, and thus we formulate our theorem 1 in the current form.\n\nWe appreciate the reviewer's insightful question and have revised the paper to include illustrative examples of these cases. We plan to address this gap in future research, and hope that these examples inspire further exploration in this area. After all, establishing a necessary and sufficient condition is always highly non-trivial and often requires significant time and multiple efforts (e.g., it takes around 10 years for the structure identification of latent linear non-Gaussian models).\n\n**Q3:**  Computational complexity of parameter identification.\n\n**A3:** The optimization in Eq.3 is solved by gradient descent, which involves evaluating the LogDet and matrix inverse (for the gradient) terms (similar to continuous causal discovery methods based on Gaussian likelihood [33]). According to [58], the computational complexity is $O(td^3)$, where $d$ is the number of variables and $t$ is the number of iterations of gradient descent. Note that the computational cost is largely independent of sample size as we only need to calculate the sample covariance once and save it for further use. Thank you for the valuable suggestion and we have added this analysis on complexity to our revision. As for the empirical runtime analysis, please kindly refer to sec 5.4, where our method is shown to be very efficient - e.g., it takes only around 2 minutes to estimate parameters for a graph that contains 50 variables.\n\n**Q4:** To what extent the sufficient and necessary conditions in Sec 3 are useful for (numerical) parameter estimation in Sec 4?\n\n**A4:** Given observational data and any structure, we can always employ our method proposed in Section 4 to get an estimation of parameters. However, whether the estimated parameters are meaningful depends on the specific given structure and we need to rely on the theoretical results developed in Section 3 to answer this question. Specifically, if the given structure satisfies the sufficient conditions proposed in Theorem 1, then we can conclude that the estimated parameters are meaningful in the sense that they will be the same as the true underlying parameters asymptotically (as in this case only the true parameters can maximize the likelihood). On the other hand, if a given structure does not satisfy the necessary conditions in Corollary 1 or 2, then it can be guaranteed that the true parameters cannot be recovered (by any estimation method as there is just not enough information). In such cases, multiple (infinite number of) parameter sets can yield the same maximum likelihood. We thank the reviewer for the insightful question and have added this discussion to our revision.\n\n**Q5:** The motivation for considering edges / parmeters from observed to latent variables and between latent variables\n\n**A5:** Thank you for the insightful question. From one perspective, if we do not explicitly consider the edge coefficients from observed to latent variables and edge coefficients between latent variables, then in many cases we cannot correctly recover the edge coefficients between observed variables either. From another perspective, the recovered edge coefficients that involve latent variables are themselves practically meaningful. Take our psychometric result in Figure 4 as an example. We may be interested in how Agreeableness influences Extraversion in human personality, even though none of them can be directly observed. In this case, the edge coefficient from L3 to L2 is informative; the value of +0.39 not only indicates a positive influence but also shows that the magnitude is considerable.\n\nWe genuinely appreciate the reviewer's effort and hope that your concerns/questions are addressed.\n\n[58] Toledo, Sivan. \"Locality of reference in LU decomposition with partial pivoting.\" SIAM Journal on Matrix Analysis and Applications, 1997.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r1PVC3E8fH", "reviewer": "Reviewer_kYam", "summary": "The manuscript proposes novel methods for learning linear structural equation models from partially observed data (i.e., allowing for latent variables). The authors provide graphical identifiability conditions for such models and describe an algorithm for learning structural parameters from data via gradient descent. Results on synthetic and real-world examples suggest that the method works well in practice.", "questions": "-I take it that the identiability conditions of Thm. 1 are untestable? Or perhaps they have some testable consequences in certain settings? If so, this would be very helpful for practitioners!\n\n-Would it be possible to perform inference on the learned parameters? For instance, could this method provide standard errors on linear coefficients? \n\n-Though I know the method is designed for the partially observable setting, I'm curious how it fares against competitors such as GES or PC when latent variables are absent?", "limitations": "Yes, in Appx. D. May be worth moving this to the main text in the final submission, however.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The manuscript is clear and well-written. It makes a meaningful contribution to a well-studied topic, which I think will be of great interest to many NeurIPS readers and the causality community more generally. The theoretical sections on identifiability and indeterminacy are well reasoned, with helpful examples along the way. The implementation is elegant and efficient, with compelling results on simulated and real-world data.", "weaknesses": "I was a bit confused by the discussion of necessary graphical conditions following Thm. 1. First, I would recommend avoiding terms like \"pretty close to...necessary\", which doesn't mean much. We get a bit more insight in Remark 1, where it's revealed that condition (i) *is* in fact necessary, but condition (ii) is not. We learn that if (ii) does not hold, then \"there are...some rare cases where parameters can be identified.\" Oddly, we don't see any examples of such structures (either in the main text or the supplement), or learn what \"rare\" amounts to here (presumably *not* Lebesgue measure zero?) Specific counterexamples would help a great deal! Perhaps some sort of disjunctive graphical condition could do the trick, e.g. condition (i) + [(ii) OR (iii)], where (iii) covers those purportedly \"rare\" structures that violate (ii) but are still technically identifiable. Alternatively, I would cut all discussion of necessity from this section and move it instead to a discussion section later in the manuscript. (I appreciate that space is tight here, but with an extra page in a final version this could be a more satisfying solution).\n\n\nMinor errata: \n-All instances of \"upto\" should be \"up to\"\n-It appears that references 46 and 47 are the same\n-It appears that references 5 and 19 are the same\n-The word \"Gaussian\" is occasionally uncapitalized", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time dedicated to reviewing our paper, the insightful comments, and valuable feedback. Please see our point-by-point responses below.\n\n**Q1:** Regarding writing suggestions about Thm 1, Remark 1, and other related discussions.\n\n**A1:** \nTheorem 1 provides a sufficient condition (which consists of conditions (i) + (ii)) for parameters to be identifiable. In Remark 1, we have shown that condition (i) is provably necessary.  Yet, (i)+(ii) is not a necessary and sufficient one, as there exist some cases that fall outside of condition (ii) yet remain identifiable (and they are not of Lebesgue measure zero).\nWe genuinely appreciate the reviewer's valuable suggestion and have revised to add multiple examples to illustrate these cases, as you suggested.\nAs for disjunctive graphical conditions,\nwe totally agree that it could be helpful, e.g., adding a condition (iii) to capture these rare cases such that (i)+(iii) could also be sufficient for identifiability.\nHowever, characterizing these cases is highly challenging\n as it may involve tools from algebraic statistics. \n Thus, we formulate our theorem 1 in the current form and plan to address this gap in future research. \nRegarding other writing suggestions such as moving the discussion of necessity to a section later and moving Appx. D to the main text, and correction of some typos, we \ngenuinely thank the reviewer for the valuable feedback and have revised them accordingly.\n\n**Q2:** Test of identifiability conditions will be helpful.\n\n**A2:** Yes, this would indeed be very helpful for practitioners. In this work, we focus on the problem of parameter identification, where the structure is assumed to be given. Thus, testing the conditions in Thm.1 is straightforward in our context.\n\nOn the other hand, we believe your question pertains more to structure identification, e.g., whether Conditions 1 and 2 in [17] can be tested.\n In practical applications, certain strategies can be employed to gain a preliminary understanding of whether the graphical conditions hold; one effective approach is to utilize domain knowledge. Yet, a rigorous test of these graphical assumptions still remains a significant challenge in the field of causal structure learning in general and will be a key area for future research. In this regard, researchers also focus on another critical question: identifying testable consequences when certain conditions are violated, as you mentioned. Specifically for Conditions 1 and 2, if certain kinds of violations exist, consequences can be detected,\nand a detailed discussion can be found in [17]. We thank the reviewer for the valuable question and have added this discussion to our revision.\n\n**Q3:** Would it be possible to perform inference on the learned parameters? For instance, standard errors on linear coefficients?\n\n**A3:**  Yes, it is possible to perform inference in our framework. To be specific, as we use maximum likelihood estimation for the parameters, some standard techniques can be readily used. For example, bootstrap can be employed to provide standard errors on linear coefficients and Chi-square test can also be done to examine the fitness of the model. Thank you for the valuable suggestion and we have added this discussion to our revision. \n\n**Q4:** How it fares against competitors such as GES or PC when latent variables are absent?\n\n**A4:** Yes, it is possible to compare our method with GES - although GES is primarily used for structure identification, it estimates all the linear edge coefficients during the calculation of likelihood. In contrast, constraint-based methods like PC cannot be compared.\nAsymptotically speaking, when latent variables are absent, the parameters estimated by our method will be exactly the same as that of GES, as in this case only the set of true parameters can maximize the likelihood. We also conduct additional experiments to check finite sample cases (5k and 10k data points are concerned) and the result is aligned with the asymptotic analysis - for graphs that have no latent variables, the parameters estimated by our method are nearly the same as that of GES (due to minor difference of implementation details of likelihood). \n\nWe genuinely appreciate the reviewer's effort and hope that your concerns/questions are addressed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates parameter identification in linear causal models including both observed and latent variables, and proposes necessary and sufficient conditions for parameter identifiability. This is an important problem that is clearly relevant for NeurIPS, and all reviewers have recommended acceptance.
Decision: Accept (poster)