Latent Variable Identifiability in Nonlinear Causal Models with Single-domain Data under Minimality Condition

Authors: Jianing Li, Hongtao Zhang, Wei Chen, Xueqi Cheng
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Identifiability theory, Disentangled representation learning, Structural causal model

Abstract:
The identifiability of latent variables given observational data is one of the core issues in the field of disentangled representation learning. Recent progresses have been made on establishing identifiablity theories for latent causal models. However with much restrictions or unrealistic assumptions, their practicality on real applications are limited. In this paper, we propose a novel identifiablity theory for learning latent variables in nonlinear causal models, requiring only single-domain data. We prove that all latent variables in a powerset bipartite graph can be identified up to an invertible transformation, if the generation process of observable data is globally invertible, latent variables are independent, and shared latent variables entail minimal information. Experiments on synthetic data support the conclusions of our theory.

Decision: reject
Reviewer scores: 5, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1729589010041

thread_type:
official_review

<thread_object>
creator: reviewer_iyjm

<content>
summary:
This paper provides an identification result of latent variables in nonlinear causal models with single-domain data.

strengths:
1. The theoretical results presented in this paper are sound, the authors provide detailed proofs.

2. This paper is well-organized and easy to follow.

weaknesses:
1. The result of this paper is too weak, making it insignificant. To simplify the identification problem, it assumes that there is no direct path between observed variables. Even so, it can only identify latent variables in the PBG-SCM obtained by reducing the original SCM substantially. Notably, a PBG-SCM usually corresponds to a large number of SCMs. A latent variable in a PBG-SCM corresponds to the combination of several exogenous noises in the original SCM, which lacks physical meaning. In fact, even the number of latent variables in an SCM is not equal to that in the corresponding PBG-SCM. Therefore, identification of latent variables in the PBG-SCM is far from identification of latent variables in the original SCM, let alone causal relations between latent variables. I list two previous works for comparison. [1] also can only identify a simplified SCM rather than the original SCM, but it does not make the assumption that there is no direct path between observed variables. Moreover, the number of latent variables and causal relations between observed variables in the simplified SCM are same as that in the original SCM. [2] also makes the assumption that there is no direct path between observed variables, but it can identify latent variables and their causal relations.

2. In line 77, the author claims that "all these assumptions are mild, and necessary". First, the authors do not verify that all assumptions are mild, for instance, they are satisfied in most real-world cases or they are the same as or milder than assumptions made by related works. Specifically, I agree that the invertibility assumption is mild because it is used by most related works, but I cannot agree that the minimality assumptions are mild. Second, the authors only prove that minimality assumption is necessary in Proposition 5.1, but the other two assumptions are not proven to be necessary.

3. It seems that the identification algorithm based on auto encoder needs to know the dimension of $z$, but in real-world scenarios, this is usually unknown. Given an inaccurate dimension of $z$, the performance is not satisfactory.

4. Some minor concern: It world be better to give a proof sketch for Theorem 1 and provide some experimental results on real-world dataset.

[1] Hoyer P O, Shimizu S, Kerminen A J, et al. Estimation of causal effects using linear non-Gaussian causal models with hidden variables. International Journal of Approximate Reasoning, 2008, 49(2): 362-378.

[2] Kong L, Huang B, Xie F, et al. Identification of nonlinear latent hierarchical models. Advances in Neural Information Processing Systems, 2023, 36: 2010-2032.

questions:
According to [1], SCM consists of a series of assignments and a joint probability of exogenous noises, but in this paper, SCM is a triplet (G,F,P), the authors should list their references.

[1] Peters J, Janzing D, Schölkopf B. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615936415

thread_type:
official_review

<thread_object>
creator: reviewer_anxt

<content>
summary:
This work proposes an identifiability theory for nonlinear causal models with latent variables using only single-domain data. A minimality condition that has been implicitly used in most previous work has been formalized and leveraged for identification.

strengths:
1. The considered problem is important and of practical significance.

2. The assumptions have been discussed in detail with examples.

weaknesses:
1. The theory and methodology in this manuscript appear closely related to those in [1] for several reasons:

   - Both manuscripts decompose the original SCM into basis models, with very similar structures. For instance, Fig. 2 in [1] illustrates a basis model nearly identical to the one in this manuscript.
   - The identifiability theorems for the basis model share many similarities, differing primarily in the minimality assumption. While this manuscript makes the minimality condition explicit, [1] assumes known dimensions, implicitly ensuring minimality. Furthermore, the subspace span assumption in [1] is likely valid asymptotically, which may minimize substantial differences between the identifiability results in the two works.
   - For the general model, both manuscripts employ iterative identification of basis models to achieve global identifiability, following a similar approach.

Given the substantial overlap in the problem setting, theoretical results, and identification procedures, it may be helpful to further highlight the manuscript's unique contributions relative to [1], as the novel aspects are not yet sufficiently pronounced in the current version.


2. The notation could be clarified for better readability. For instance, $j\\&2^l$ in line 189 is difficult to interpret within the given context. 

3. There are some questions for both the proofs and experiments. Please kindly refer to the questions for details.


[1] Kong et al., Identification of nonlinear latent hierarchical models, NeurIPS, 2023

questions:
1. Could you please explain more on line 777, where it states that $\hat{v}_2$ does not change since the derivative of $v_2$ w.r.t. $s_1$ is zero?

2. The proof of Theorem 1 appears to establish only the nonexistence of certain edges, without addressing the existence of others. Including this additional aspect might provide a complete proof.

3. For evaluating general PBG-SCM identification, would it be more appropriate to use a metric that assesses component-wise identifiability, such as the mean correlation coefficient (MCC)? Since Theorem 2 establishes component-wise identifiability, relying on a metric like $R^2$, typically used for block-wise identifiability, may not fully capture the relevant evaluation criteria.

4. In all theorems, the generating functions are assumed to be invertible. However, in experiments, MLPs are used for the generation of synthetic data. Since MLPs are typically not invertible, it seems that the experiments cannot rigorously verify the theorems?

5. Could you please elaborate further on why previous methods require assumptions that there exists no connection among observed variables? I understand that this is necessary for the identification of latent structure, but not so sure about whether this is the case if we only aim to identify the latent variables. Perhaps elaborating it based on the concrete methods you mentioned in line 156 would be helpful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730634280624

thread_type:
official_review

<thread_object>
creator: reviewer_eyvz

<content>
summary:
Identifiability analysis of latent (causal) variables is a fundamental challenge in causal/disentangled representation learning. This work introduces a straightforward powerset bipartite graph, which defines an equivalence class of general causal models capable of generating the same observed distribution. We present identifiability analysis tailored to this specific graph structure and extend the analysis to provide deeper identifiability insights. Simulations are conducted to validate the theoretical claims.

strengths:
Identifiability analysis of latent variable models is both challenging and fundamental.

The identifiability analysis for this specific graph structure seems to be robust.

The relationship between the unknown nature of latent dimensions and the minimality condition is interesting.

weaknesses:
As a theory paper, I take the details of claims very seriously. However, I found some claims to be overly casual, lacking reasonable justification.

1) The claim that 'multi-domain data are usually hard to acquire' seems overly strong. I do not think this is the case. Additionally, the references cited to support this point (e.g., Matsuura & Harada, 2020; Creager et al., 2021) primarily propose methods for handling multi-domain problems rather than the difficulty of acquiring such data, and therefore do not support the claim.

2) The claim that 'any underlying structural causal model (SCM) can be reduced to an equivalent SCM with a powerset bipartite graph (PBG) structure' also seems overly strong. I understand that, in latent space, generating the same observed data can result from multiple graph structures, leading to a large equivalence class. This is a key reason why non-identifiability results often arise, and many works aim to address this challenge. Given this, it seems reasonable that some transformations between potential graph structures could result in a standardized structure. However, the claim that 'any underlying SCM can be reduced to a powerset bipartite graph' appears unsupported. At the very least, the authors should provide rigorous theoretical backing for this claim.

3) Additionally, as an initial step in this work, it would be helpful to clarify which graph structures can be reduced to a powerset bipartite graph, which cannot, and what the associated costs of this reduction might be. Furthermore, if the powerset bipartite graph structure is critical, it would be valuable to present real applications that justify its importance.

4) The identifiability results in Theorem 1 appear to be sound. However, some implicit assumptions should be explicitly stated. For example, in reviewing the proof for Theorem 1, I noticed that it presumes mutual independence across all dimensions of each latent variable (e.g., z[i]⊥z[j] for the vector z). However, this assumption—that each zi​ is mutually independent—is not clearly stated in Theorem 1 itself. It is crucial to clearly list all assumptions to avoid ambiguity.

5) In general, non-identifiability is common in nonlinear ICA. However, applying sparsity constraints to the mixing process from latent to observed data can achieve identifiability, as demonstrated in this work. Prior studies, such as Zheng, Yujia, Ignavier Ng, and Kun Zhang's 'On the Identifiability of Nonlinear ICA: Sparsity and Beyond' (NeurIPS 2022), provide in-depth analysis on leveraging sparsity for identifiability. A clear comparison between this work and prior approaches is essential but appears to be missing here, making it difficult to fully understand this work's unique contributions.

6) Again, the experiments on real data are not directly provided, which significantly weakens the claim regarding the importance of the powerset bipartite graph.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723054810

thread_type:
official_review

<thread_object>
creator: reviewer_xsdi

<content>
summary:
The paper's main contribution is the introduction of a novel identifiability theory for latent variables in nonlinear causal models, using only single-domain data. It proposes a reduction process converting general causal models into powerset bipartite graph structures, proving that latent variables can be identified under mild conditions: invertibility, independence, and a new minimality condition. This theory provides a framework for designing algorithms to uniquely identify latent variables in scenarios with limited data. One main issue is that it claims that every model can be converted to a powers bipartite graph, but did not give any proof. Another weakness is that in all the examples in the paper, observable variables are children of latent variables, which is quite restricted and unrealistic.

strengths:
The problem that the authors try to attack, i.e. latent causal discovery is crucial in causal discovery. It helps uncover hidden factors influencing observed data, enabling more accurate modeling of causal relationships. Identifying latent variables can improve understanding of complex systems, enhance predictive accuracy, and support tasks like disentangled representation learning, domain adaptation, and generalization to distributional shifts. Without addressing latent variables, causal models may miss key drivers, leading to incomplete or biased conclusions.

weaknesses:
There are two main issues of the proposed approach. 

(1) The conversion of SCM to a power-set bipartite graph is not proved to be correct.

(2) The assumption that observable variables are children of latent variables are too restrictive and inpractical.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732719777716

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Notification for updated paper

comment:
We thank all reviewers for your efforts. We have updated our paper (modified content is in blue color), and suggest future discussion on the new version. Noticeable modifications include:

1.	We add an experiment on controllable generation of images in Appendix A.9.
2.	We add a proof regarding the validity of the SCM reduction process in Appendix A.1.
3.	We add a figure explaining the numbering rule in our SCM reduction and general model in Appendix A.1.
4.	We update part of the proof of Thm. 1 for better clarity in Appendix A.3.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles the important problem of latent variable identifiability in nonlinear causal models using single-domain data, proposing a framework based on powerset bipartite graph structures. While all reviewers think the topic is significant, there are still some concerns, including insufficient rigor in claims, restrictive assumptions limiting generality, overlap with prior work, and reliance on synthetic data with limited real-world validation. The authors’ revisions address some concerns but leave others, particularly around novelty and practical impact, unresolved. While the paper presents promising ideas, it requires more rigorous refinement.