Causal Representation Learning from Multimodal Biomedical Observations

Authors: Yuewen Sun, Lingjing Kong, Guangyi Chen, Loka Li, Gongxu Luo, Zijian Li, Yixuan Zhang, Yujia Zheng, Mengyue Yang, Petar Stojanov, Eran Segal, Eric P. Xing, Kun Zhang
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: multimodal observations, identifiability, causal representation learning

Abstract:
Prevalent in biomedical applications (e.g., human phenotype research), multimodal datasets can provide valuable insights into the underlying physiological mechanisms. However, current machine learning (ML) models designed to analyze these datasets often lack interpretability and identifiability guarantees, which are essential for biomedical research. Recent advances in causal representation learning have shown promise in identifying interpretable latent causal variables with formal theoretical guarantees. Unfortunately, most current work on multimodal distributions either relies on restrictive parametric assumptions or yields only coarse identification results, limiting their applicability to biomedical research that favors a detailed understanding of the mechanisms.

In this work, we aim to develop flexible identification conditions for multimodal data and principled methods to facilitate the understanding of biomedical datasets. Theoretically, we consider a nonparametric latent distribution (c.f., parametric assumptions in previous work) that allows for causal relationships across potentially different modalities. We establish identifiability guarantees for each latent component, extending the subspace identification results from previous work. Our key theoretical contribution is the structural sparsity of causal connections between modalities, which, as we will discuss, is natural for a large collection of biomedical systems.
Empirically, we present a practical framework to instantiate our theoretical insights. We demonstrate the effectiveness of our approach through extensive experiments on both numerical and synthetic datasets. Results on a real-world human phenotype dataset are consistent with established biomedical research, validating our theoretical and methodological framework.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730121722422

thread_type:
official_review

<thread_object>
creator: reviewer_91p1

<content>
summary:
The paper addresses the identification of hidden causes underlying observed multimodal data. The authors propose a model that avoids previous limitations, specifically not requiring shared latent information across multiple modalities or assuming an exponential family distribution for hidden causes. The paper provides a theoretical identifiability analysis and presents experiments, including one on biological data.

strengths:
The paper presents meaningful relaxations of assumptions from prior work.

It is relatively well-written, considering the complexity of the theory and notation involved.

weaknesses:
There are issues with the main theoretical claims and their proofs.

The practicality of the current assumptions and their real-world applicability remains unclear.

The experiment using the human phenotype data does not justify the title or the highlights in the abstract.

questions:
### Theory

1. *Identification vs. Estimation*: In Theorems 4.2 and 4.4, identification and estimation are confused by statements like “We estimate the generating process…”. Identification is a model property and should be established before estimation is addressed. For a model $\mathcal{P}:=\\{p_{\phi}:\phi \in \Phi\\}$, we simply show that $\forall \phi_1, \phi_2 \in \Phi (p_{\phi_1}=p_{\phi_2} \implies \phi_1=\phi_2)$ to demonstrate identification.

2. *"Identification under Regularization"*: Particularly, Theorem 4.4 is problematic in stating that identification is achieved through regularization, which pertains to estimation. If regularization is essential, the model should be reformulated—like for ridge regression, the regularization can be treated as a sparse prior in a probabilistic model.

3. *Proof of Theorem 4.2*: The step to equation (14) seems to rely on $x=\hat{x}$. However, equal distributions do not imply equal values of random variables. If (14) is correct, the derivation should clarify how equal distribution leads to (14).

4. *Definition of $\tilde{g}_x$*: Strictly speaking, there is one $\tilde{g}_x$ for each $m$, and it’s unclear if these $m$ functions can be shown as the same function. If not, we are in fact making assumptions about $m$ functions.

5. *Proof of Theorem 4.4*: 1) Condition 4.3 mentions *undefined matrix $A$*. In the proof, equation (28) involves more than a single matrix,  as denoted by $A$ in Condition 4.3, and the definition of $d^*(U)$ does not seem to align with $d^*$ in Condition 4.3 (this questionable reasoning later is extended to $D$). This proof section is unclear both in notation and logic flow; we can not even see $A$ as a placeholder for the various matrices used in equation (28)! I suggest sorting out the involved matrixes (I believe, if the proof is not simply wrong, there should be several similar conditions for different matrixes), and a clear connection between the proof and assumptions should be seen. 2) *the regularization again*: the reasoning following equation (29) requires satisfaction of (29), but regularization (5) can only encourage, not guarantee this condition. This suggests the idea of “identification under regularization” itself does not work, not only an incorrect formulation as mentioned in the 2nd point.

### Implications
One core aspect of the causal model is that two modalities cannot share any direct causes, which is questionable. While $z^{(m)}$ can, and indeed is required to, affect $x^{(-m)}$ indirectly through $z^{(-m)}$, this seems to require hidden causes $z$ to be lower-level features. For example, “age” is a high-level feature affecting many modalities, like “sleep” and “eyes.” Thus, if the "sleep" modality includes "age" in $z^{sleep}$, then, to satisfy the "indirect" causes requirement, lower-level features $z^{eyes}$ would be needed to mediate the causal effects from “age” to “eyes", i.e., $d(z^{eyes})$ should be large enough to block all the causal effects from “age” to “eyes.” And, when we have many modalities and many hidden causes, this eventually requires most hidden causes to be low-level.

While the theoretical relaxations and real-world examples are appreciated, it’s unclear why, in the “sleep patterns” and “brain imaging” examples, your assumptions could be satisfied, though I can see previous ones are limited.


### Biological Application
The demonstration of the biological application lacks depth, and the claimed “strong interpretability” is overstated. For example, in the human phenotype dataset, how should we interpret the learned hidden variables? Does the method enhance our understanding of the data or its underlying mechanisms? How does the number of latent variables affect the results (possibly with very different causal graphs)?

Additional specific questions are: Why are causal relationships involving FRight and FLeft not symmetric? For example, why does FRight affect Cataract but not FLeft?
Why are there no Sleep variables that influence “Sleep efficiency”?
Shouldn’t ocular quality impact hand grip strength rather than the reverse?

More experiments should be conducted, both on the model side (e.g., testing various numbers of latent variables and other hyperparameters) and the data side (e.g., trying many different modalities). 


Overall, the authors could clean up theoretical developments, clarify the practical implications, and leave the biological application to future work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730518416331

thread_type:
official_review

<thread_object>
creator: reviewer_mbq3

<content>
summary:
This paper develops a theory for identifying the underlying structure of multi-modal data involving latent causal variables, based on the assumption of smooth invertible generating functions. It further investigates the sparsity of this causal structure and proposes an estimation method. Experiments conducted on both synthetic and real human phenotype data demonstrate the effectiveness of the proposed approach.

strengths:
- The paper is generally well-written.
- It includes an adequate literature review.

weaknesses:
- On real impact. As mentioned by the authors, knowing the number of underlying latent variables is unrealistic. 

- Missing implementation details, e.g., more details on data generation, network structure, optimization scheme, hyper-parameters, etc. see questions as well.

questions:
1. In my view, the example provided in Section 3 does not align well with the proposed model. What is your rationale for considering heart sizes and bone structures as latent variables?
2. In Appendix D.1, could you provide further details on the MLP architecture used for the numerical data generation process? Have you considered implementing an invertible neural network structure in your simulations? If you increase the number of data samples, do you expect the MCC to approach 1? I would appreciate seeing results on this, particularly illustrated with four variables (two latent and two observed).
3. The arrow from $z$ to the adjacency matrix in Figure 4 is unclear. Could you clarify this relationship?
4. How did you compute the KL divergence? Do you assume that $p(\gamma)$ follows a Gaussian distribution, with the encoder output providing the mean and variance?
5. When calculating the SHD, given that the indices of latent variables can be permuted, could this pose an issue?
6. In the context of the MNIST dataset, what causal relationship exists between the colored MNIST and fashion MNIST data?
7. How does the learned adjacency matrix A relate to the one derived from the PC algorithm?

Comment:
For the real-world application, I encourage the authors to explicitly present evidence from the literature demonstrating the success of their algorithm.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652335373

thread_type:
official_review

<thread_object>
creator: reviewer_uhwz

<content>
summary:
Authors develop new methods to identify patterns in complex multi-modal biological datasets. It improves upon previous work in two key ways: First, it uses a flexible mathematical framework that doesn't make rigid assumptions about how the underlying biological factors (latent factors) are distributed. Second, these factors can influence each other across different modalities. Simulation study and real-world data support authors's claim.

strengths:
1. The paper is well-written and motivated. 
2. Authors provide identifiability guarantees for each latent component. This is helpful to characterize the interactions among all latent
components across modalities for the biological applications. 
3. The assumptions of theoretical results look reasonable to me. 
4. Real-world dataset analysis is provided, demonstrating its usefulness.

weaknesses:
1. A1 indicated the neural network needs to be invertible. How do authors achieve this? Authors use normalizing flow, an invertible generative model, as a part of their network, what about others? Also what's the computation efficiency? 
2. In real world, user-defined number of latent variables can be biased. Have authors analyzed it?

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689366620

thread_type:
official_review

<thread_object>
creator: reviewer_jxdh

<content>
summary:
This paper provides a method that identifies the latent variables up to invertible component-wise transformations from multi-view data under the weak assumption that these latent variables exert partial influence on every modality and strong influence on one modality. The proposed approach relies on weaker assumptions compared to the existing works and yet provides strong results. The primary motivation for this approach is on multimodal health care data but the experiments are conducted on both synthetic and image datasets in addition to health care data.

strengths:
1.   The contribution is very clear and useful for future works on identifiability. More strengths included in the summary.
2.   The writing is lucid. The examples are cleverly used to contrast the proposed method against the existing works.

weaknesses:
I did not find any significant weaknesses. However, I do have a few questions for the sake of clarity. Questions in the following section.

questions:
1. It is assumed that the information of $z^{(m)}$ is preserved in its corresponding observation $x^{(m)}$ and it exerts sufficient influence on other modalities' observations $x^{(-m)}$. Now, consider the observation from another modality $x^{(k)}$ that has a similar behavior. How does $z^{(k)}$ not interfere in the identifiability of the subspace of $z^{(m)}$?
2. Is the considered setting a non-linear version of [10]? Can the authors add text comparing/contrasting the proposed approach with [10]?
3. Does "fully share" mean "retrieve without errors" in line 212? That's what I understood from the example that follows this statement in line 215.
4. Can the authors explain how the method is different from [9] apart from the latent variables being independent in [9]?Specifically, what does "part of the variables are directly observed and causal directions are given by default" mean in lines 335-338?
5. Can the authors add an intuitive description for Condition 4.3 as its implications are not clear from the main text? It is enough that this intuitive explanation is included in the appendix.


Typos and minor writing mistakes:

1. In line 78, what is $n$? Is it $z_i^{(m)}\rightarrow z_j^{(n)}$?
2. Is the co-domain of $G(z, \epsilon)$ correct in line 303?
3. Some issues with the references. E.g., "Non-parametric Identifiability of causal representations from unknown interventions" appeared in NeurIPS 2023. Sturma et al., (2023) cited twice in lines 126-127.


[1] Sturma et al., 2023, "Unpaired Multi-domain Causal Representation Learning", NeurIPS 2023

[2] Zheng et al., 2022, "On the Identifiability of Nonlinear ICA: Sparsity and Beyond", NeurIPS 2022
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695673874

thread_type:
official_review

<thread_object>
creator: reviewer_5apk

<content>
summary:
This paper studies the causal graph discovery problem from multiview data. Given observed multimodal data, the goal is to estimate the causal relationship of latent features between modalities. The authors study the data generation process where the observed multimodal data are independent given the latent variables. Under this assumption, the authors employ the encoder-decoder framework to decompose the latent factors and nuisance factors, and a sparsity regularization function to impose the sparse relationship between modalities in the latent spaces. They compare the proposed method with several baselines on simulated tasks and the results show improved mean correlation coefficient. Furthermore, the proposed method is applied to human phenotype dataset.

strengths:
This paper tackles an important and yet not well-addressed problem. It is well-motivated by biomedical applications. The proposed framework is more general compared to the prior work (as shown in Table 1).

weaknesses:
In general, I find several parts that need further clarification. (see the question parts)

My main concern is that the simulated tasks focus on low-dimensional data with simple sparse causal structures. It is not clear whether the method is scalable and can be generalized to more complex causal structures.

questions:
a. the paper assumes that the latent variables within the same modality form a DAG, could the author clarify why this assumption is necessary? Secondly, if this is needed, the loss function (6) does not enforce the latent variables within the same modality to be a DAG. How can we ensure such structures will be satisfied in the learning phase?

b. does the causal relationship between different modalities need to be DAG? If there are cycles, are we still being able to identify the latent variables?

c. I find the definition of condition 4.3 confusing. What are the connections between matrices A, G, T? and A seems to only denote the relationship between modality 1 and 2?

d. In Assumption A.1, what is smooth inverse?

e. Could the authors provide a guideline for practitioners on how to systematically choose the number of latent nodes?

f. In Theorem 4.4, does it require the regularization term to be scaled by some coefficient?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733312526195

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We are thankful for the reviewers' thoughtful feedback and insightful comments. To provide further clarity, we'd like to share a summary of the discussion phase.

All five reviewers acknowledged the strengths of our paper:
- **Importance of problem**: `Reviewer 5aPk` highlighted that we address an important yet underexplored problem, and our framework is more general compared to prior work.
- **Theoretical contributions**: `Reviewer JxDh` recognized our clear and impactful contributions on identifiability. The examples effectively contrast our method with existing works. `Reviewer uhwZ` recognized the proposed identifiability is crucial in biological applications, and assumptions were deemed reasonable. 
- **Relaxations of assumptions**: `Reviewer 91p1` appreciated the meaningful relaxations of assumptions from prior work. 
- **Real-world relevance**: `Reviewer uhwZ` noted that the real-world analysis demonstrated practical utility.
- **Good writing quality**: `Reviewer mBq3` noted that the paper is generally well-written. `Reviewer JxDh` appreciated lucid writing. `Reviewer 91p1` noted that the paper is relatively well-written given the complexity of the theory and notation.

While the reviewers recognized the above strengths, they also raised some concerns:
- **Scalability and generalization**: `Reviewer 5aPk` noted that the simulated tasks focus on low-dimensional data, raising concerns about the scalability and applicability of the method to more complex causal structures. `Reviewer uhwZ` and `Reviewer mBq3` raised concerns about the real impact of the user-defined numbers of latent variables. 
- **Implementation details**: `Reviewer uhwZ` questioned how to achieve invertibility and what is computational efficiency. `Reviewer mBq3` pointed out missing implementation details.
- **Theoretical claims**: `Reviewer 91p1` raised concerns about theoretical presentation and formulation.
- **Real-world applicability**: `Reviewer 91p1` questioned the practicality of the assumptions and their relevance to real-world scenarios. The analysis of human phenotype results was unclear to justify the paper's claims.

During the discussion phase, we carefully addressed each reviewer’s concerns. Below is a summary of the revised manuscript:
### Regarding Theory
- **Refinement of definitions and formulations** (`Reviewer 91p1`): We have revised our statement of the identification goal and notations, reformulated the regularization as a constraint on alternative specifications, and reformulated Condition 4.3.
- **Clarification of the theorems** (`Reviewer 91p1`): We have refined the technical presentation of our identifiability results to be consistent with the suggested framework and addressed a misunderstanding of the proof for Theorem 4.2.

### Regarding Experiment
- **More experimental evaluation** (`Reviewer 5aPk` and `Reviewer 91p1`): To address concerns about limited experiments, we have verified our framework on higher-dimensional simulation tasks in more complex scenarios. 
- **Evaluation of latent variable numbers** (`Reviewer uhwZ`, `Reviewer mBq3` and `Reviewer 91p1`): We have provided a guideline on how to choose the number of latent variables and conducted additional synthetic experiments to validate the effectiveness. 
- **Detailed implementation details** (`Reviewer uhwZ` and `Reviewer mBq3`): We have provided details on the data generation process, network architecture, training settings, and hyperparameter settings. We detailed how we achieved invertibility and empirically provided computational efficiency.

### Regarding Implications
- **Treatment of shared latent variables** (`Reviewer 91p1`): We have extended our framework to account for shared latent variables. This extension is validated both theoretically and experimentally.

### Regarding Biological Application
- **Improved biological interpretability** (`Reviewer 91p1`): We have clarified how the learned latent variables may correspond to known biological phenomena by referring to the existing literature. 

Eventually:
- **Reviewer 5aPk** raised no further questions and raised the rating.
- **Reviewer JxDh** replied that our response has answered the reviewer’s questions, and maintained a positive rating.
- **Reviewer uhwZ** might have been occupied during the discussion phase and was unable to engage in discussions.
- **Reviewer mBq3** raised further concerns regarding implementation details and we have provided further details accordingly.
- **Reviewer 91p1** appreciated our detailed response, and raised questions regarding the theorem.  We believe that it is a language barrier and the further concern is a misunderstanding of the proof framework we follow from the field of causal representation and nonlinear ICA. We hope our further response can resolve this question effectively.

We thank the reviewers and the ACs’ efforts in encouraging discussions. We hope this summary will assist all reviewers and ACs in making the final decision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a theoretical framework for identifiability conditions for causal representation learning from multimodal data. The reviews are largely positive, and agree this is a solid technical contribution to the community. I urge the authors to address a minor clarification required as noted by the reviewer 91p1.