Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper provides necessary and sufficient graphical conditions for full identification of linear causal models with latent variables. It introduces the so-called ‘bottleneck’ and ‘strong-redundancy’ condition in combination with bottleneck faithfulness that together ensure identifiability (up to irrelevant permutations/scaling). It also considers likelihood approaches to estimating the unique solution from data that exploit heterogeneity in the independent noise terms over multiple domains. Experimental results are reported to illustrate feasibility of the method.
main_review: Challenging and relevant problem that has been studied in many versions before. The main contribution of this paper is that it provides some generic graphical criteria that determine whether or not this identification problem is uniquely solvable, thereby bringing together several other approaches.

Drawback is that it is hard to asses to what extent the accompanying assumptions apply in practice or whether they are very restrictive or not. Naturally ‘linear’ is often assumed but not always valid. Also the need to know the number of latents beforehand makes it harder to apply in real-world scenarios. Often the goal is to infer what latent variables / constructs can be recognised/identified  from the observed distribution. Similarly, on applying the reconstruction estimation described in section 6, it is not easy to verify from the output obtained whether the assumptions actually did hold or not. And actually determining the full model F itself may involve an exhaustive search over graphs, or hope that a penalised log-likelihood approach will converge to the true optimum quickly … which is by no means guaranteed either. 
Perhaps the ‘full identifiability’ is more than what is actually needed. For example, a model with subpath Li -> Lj -> Xk would not satisfy the conditions, but the marginal over Li -> Xk would, and still be equally informative or useful to have. And for the reconstruction problem based on available X that would be perfectly fine.

Technical quality is good, with attention to detail and good efforts to interpret the sometimes quite abstract theoretical results. Clarity can be improved at some points where concepts are used before they are actually defined, but overall very readable for a challenging subject like this. Experimental evaluation could be a bit more extensive but is ok to illustrate some of the remaining challenges.

originality: well-known problem, but interesting and original approach
quality: high quality
clarity: ok with minor suggestions for improvement below
significance: not directly applicable as an off-the-shelf algorithm, but certainly a meaningful theoretical contribution that provides new insights

minor comments:
p3.82: typo ‘the the’
p3.93: ‘uncorrelated in each domain’ => does this mean they all *have* to change between domains? if so, then that is a rather strong and unrealistic setting, as in most cases only a subset of noise terms is likely to differ between domains / over time
p3.116: can the nodes in J and K themselves be part of the bottleneck from J to K? (ok later becomes clear)
p4,123-8: These two assumptions are at first sight quite artificial and hard to parse in terms of what graphs are excluded and what not. In other words, it is not easy to see how restrictive / realistic these assumptions are in practice. It would be helpful to give some hints/pointers to see what types of graphs generally satisfy these conditions. For example if the variables can be separated so that there are no directed paths from X to L, does the causal structure between the X influence the ‘unique minimal bottleneck condition’?
p4.133: explain/define ‘bottleneck faithful’ … (ok later found at p5.156, just refer here)
p6.192: weird character ‘one?’ in the pdf (or perhaps just my reader)
p6.213: ‘only recover limited information about the causal structure’ => true, but it would be interesting to see if starting from these to reduce the full search space to ‘local latent structure discovery’ would make the assumptions provided here much less restrictive in practice … and with it greatly enhance the applicability
p9.321-324: doesn’t L1 typically induce sparsity by driving certain small coefficients to zero?

limitations_and_societal_impact: no negative societal impact
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper provides two graphical conditions for identifying a linear causal model under non-Gaussian error or heterogenous errors across domains when a subset of variables are latent. Specifically, the paper introduces necessary (‘unique minimal bottleneck’ criterion and strong non-redundancy) and sufficient conditions (bottleneck faithfulness) for the identifiability. These conditions indeed form an identifiability condition.
main_review: 
originality:
The paper is original. The related work section is comprehensive enough to appreciate the original contributions of this paper.


quality:
Considering the readability, I was pleased to read the paper. Notations, texts, theorems, and explanations are all placed perfectly. Sections are organized coherently. Figures are simple yet informative. Technically, the authors made progress in identifiability via graphical and matrix-related understanding, connecting other existing conditions (as demonstrated in the related work section). 


clarity:
Is theorem 1 if and only if condition? It is currently written like sufficiency. Given that some of the space is left, it would be helpful if Section 4.2 contains figures that can explain the Lemmas in the section. Similarly, the section may have an algorithm (pseudocode?).


significance:
This paper provides necessity, sufficiency, and identification results (in addition to estimation). Since (theoretical) causality research focuses on the existence of clearly defined estimands, such conditions will be valuable resources to develop the causal theory further. 


limitation:
Although linearity is a strong assumption over the functions of causal models, since it allows us to study a class of causal models in depth, I would not consider the assumption restrictive. In the future, it would be desirable to provide some details about the estimation procedure for overcomplete ICA. 

limitations_and_societal_impact: .
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper is about statistical causal inference. More specifically, this paper is about causal discovery that infers causal graphs. The data used is observational data without intervention. All the variables are continuous.

This paper considers linear causal models with hidden variables. This setup can be written using an overcomplete mixing matrix as in (1). This overcomplete mixing matrix is known to identifiable up to the permutation and signs of the columns if the hidden variables are non-Gaussian and independent [4]. This is known in the field of independent component analysis. When the number o the hidden variables are the same as the observed variables, it is known that changing noise variance assumption is sufficient to make the overcomplete mixing matrix identifiable even when the hidden variables are Gaussian [16]. A contribution of this paper is Theorem 1 that extends the identifiability result [16] to overcomplete cases. 

Based on the two identifiability results, they further consider the sufficient and necessary conditions of the linear causal models after the overcomplete mixing matrix has been identified. They aim to identify the underlying causal model beyond the Markov equivalence class.

They further assume the sparseness of the causal adjacency matrix F, i.e., the sparsest causal model is true if multiple causal models give the same overcomplete mixing matrix M.

main_review: Originality: This paper tries to investigate the sufficient and necessary conditions for linear causal models with hidden variables, including linear causal models with hidden common causes and those with latent factors being causally related. This general framework would be new because those models have been investigated separately. 

The estimation method based on exhaustive search in Section 3 looks a naïve method. Another proposed estimation based on sparse regularization is proposed. The idea seems not so new. 

Quality: 
I’m not sure if the assumption on the sparseness on lines 106-111 is reasonable or effective in which application domains. I wonder if there is a reasonable scientific reason to think that the sparsest models are the truth.

Further, I’m interested in how the identifiability results change if the sparsest model assumption has been removed. I thought previous works [17-22] do not make this assumption. 

I would appreciate it if the authors make these points above clear.

The proposed estimation methods look not very fresh or very solid. One is a simple exhaustive search. It is not clear if the other regularization method is consistent or consistently selects the truth. In fact, they say this regularization method did not work well (lines 317-324).

I was not sure if an underdetermined version of [16] has been already considered in the literature of independent component analysis or signal processing.

Clarity: There are serval unclear things. 
1.	Line 116: A set B. Is B a set of what?
2.	Line 29: might not not - > might not 
3.	Line 133: Bottleneck faithfulness
Bottleneck faithfulness has not been defined at this point.
Figure 2: “is not a redundancy” What would be the definition of this?
Figure 3: Co-parental non-redundancy, parental non-redundancy
I couldn’t find their definitions in the paper. Ah, I see. I found them in the supplement.
4.	Line 192: at least one?
What is this ? here?
5.	In the supplement, on page 11, what Times of Times (i) means?
6.	In 6.1, were the artificial data generated under the assumption that the underlying model is the sparsest.
7.	How is the number of hidden variables selected?

Significance: 
This work could contribute to a deeper theoretical understanding of causal discovery methods in the presence of hidden variables, including hidden common causes and latent factors, by considering a general framework includes both classes of linear causal models with hidden common causes and latent factors. 
Relating to this, I have a question. Previous works [17-22] gave sufficient conditions. Did the authors of this submission find some other sufficient conditions for any of the models considered in [17-22]? If there are no other sufficient conditions, does this mean that the conditions given by [17-22] turned out to be also necessary?
limitations_and_societal_impact: More discussion on the assumption that the sparsest graph is the truth would be desirable.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper considers one of important problems that under what condition a directed acyclic graphical model, especially linear structural equatioon models, can be inferred without the causal sufficientcy assumption. Hence, the proposed method provide a novel approch that can cope with learning linear structural equation models even when there are several latent variables. The key conditions are (i) sparsest true model, (ii) minimal bottleneck, (iii) strong non-redundancy, (iv) bottleneck faithfulness. It tried to explain a lot of new concepts with examples, and to compare the proposed idea to the existing conditions such as faithfulness. 
main_review: I enjoyed the idea and I think the main problem of learning latent causal graphical models is very important; however, this paper should clarify some necessary definitions, concepts and examples; otherwise I cannot properly evaluate its originality and contribution. 

line 41/43/84: It is unclear that 
"\epsilon is a vector of independent noise variables", "\epsilon_i are uncorrelated within each domain, ...", and "\epsilon_i are not necessarily mutually independent..."

In the linear structural equation models (SEM) settings, is independence of \epilson_i required? Otherwise it may violate the markov condition and it seems to be a mixture of directed and undirected graphical model. 

(1) line 55: definition of the directed path says that (V_i) is a directed path from V_i to itself. Then, Pa(V_i) should include V_i. 

line 91, the conditions in Theorem 1 is not clear. It should clarify that "Suppose we have observed X generated according to the mixing procedure (4) in a number of domains, t = 1, 2, T"

If T =2, then is the considered model is X = Y M \epsilon_1 + (1- Y) M \epsilon_2 where Y is Bernoulli distribution with known/unknown probabilities?, X_j = \beta X_{Pa(j)} + Y_{1j} \epsilon_{j1} + (1 - Y_{1j}) \epsilon_{j2} where (Y_{1j}) are independent Bernoulli distribution with known/unknown probabilities, or X = M (p \epsilon_1 + (1- p) M \epsilon_2)? In addition, in the finite sample setting, is it assumed to be known what distribution of error (\epsilon_t) the sample follows? 

line 116, the definition of the bottleneck should be clarified when J and K are sets of nodes. 
"We say that a set B is a bottleneck from J to K if every directed path from J to K includes some b \in B."
line 121 From the definition, for each Vi, Ch(Vi) is a bottleneck from Ch(Vi) to X.

First of all, if J includes two nodes, saying J = {1, 2}, and K = {3}, the directed path from J to K is not defined. 
It is one of the key definitions to provide an important condition in this paper, it should be precisely defined.

e.g., I at first conjecture that it means that the directed path from j to k, for all j \in J and k \in K. Then, all directed paths are (1,3), (2,3). Hence, {3} is only bottleneck. However, then, it is contradictory to line 121. 

line 160, it is also unclear that "Rank(M_K^J) = 0 is a violation of classical faithfulness if there...". It is in fact not true that without considering the error variances of (\epsilon). The statement is true if all error variances are the same, otherwise there is no guarantees that "there is a path from J to K but the path coefficients cancel out so that the net effect of J on K is 0". 

In addition, the term "bottleneck faithfulness" might better to be modified, because it is completely based on the graph, the original faithfulness assumption is based on the graph and its probability distribution. Hence, the name "bottleneck faithfulness" might lead to incorrect understanding of the concept of the bottleneck faithfulness. 

line 179, it should be clarified that "Let V_k \to V denote the variables whose longest path to X has fewer than k nodes."
For example, 1->2->3<-4. Then, by the definition of V_k, we can see V_1 = {3}, V_2 = {2,3,4}, V_3 = {1,2,3,4}, while the longest path is 1->2->3. Hence, it should be the variables whose paths to X has fewer than k nodes. 

Figure 4, it is unclear that how Desc(X_2) = Desc(L_1) in all graphs. For instance, in G_1, Desc(X_2) = (X_2, X_3) and L_1 does not exist. 

line 285, it should be clarified that the expectation of the variable corresponding to a node is zero, somewhere in Section 2.
 
I agree that learning a SEM with latent variables is very difficult, and hence, it is worthy to investigate the necessary and sufficient condition for the identifiability. In addition, it is possible that I do not understand some parts of the paper and I amy not be familiar with the line of works. Howeve, without the concrete defintion of the model and concepts, I am pretty sure that common readers cannot understand the part that I fail to understand. Hence, it should clarify the model the paper focuses and it would be better to provide how to check the proposed conditions are satisfied from finite samples or prior information. 

Minor:
line 29 not not
line 82 matrix \mathcal{X}
line 173 equation (8) and does (8)
line 287 d-th domain might be t-th domain

******** Updated Review **********
Thanks for the kind answer. I increase the rate based on the answer. I hope the paper contains the model specified in term of math termonology. In addition, the statements that have potential misunderstanding are clarified. 

limitations_and_societal_impact: Not related.
review_title: null

Meta-review: The reviewers appreciated the relevance of the paper to the NeurIPS community and the strength of the technical results. 

Additionally, the authors did a good job of engaging with the reviewers to dispel most concerns. Hopefully, the discussion clarified to the authors which parts of the paper are less readable, and they can use this information to improve the writing.
Decision: Accept (Poster)