Causal Representation Learning from Multimodal Biomedical Observations
Authors: Yuewen Sun
Lingjing Kong
Guangyi Chen
Loka Li
Gongxu Luo
Zijian Li
Yixuan Zhang
Yujia Zheng
Mengyue Yang
Petar Stojanov
Eran Segal
Eric P. Xing
Kun Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal observations
identifiability
causal representation learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Prevalent in biomedical applications (e.g., human phenotype research), multimodal datasets can provide valuable insights into the underlying physiological mechanisms. However, current machine learning (ML) models designed to analyze these datasets often lack interpretability and identifiability guarantees, which are essential for biomedical research. Recent advances in causal representation learning have shown promise in identifying interpretable latent causal variables with formal theoretical guarantees. Unfortunately, most current work on multimodal distributions either relies on restrictive parametric assumptions or yields only coarse identification results, limiting their applicability to biomedical research that favors a detailed understanding of the mechanisms.

In this work, we aim to develop flexible identification conditions for multimodal data and principled methods to facilitate the understanding of biomedical datasets. Theoretically, we consider a nonparametric latent distribution (c.f., parametric assumptions in previous work) that allows for causal relationships across potentially different modalities. We establish identifiability guarantees for each latent component, extending the subspace identification results from previous work. Our key theoretical contribution is the structural sparsity of causal connections between modalities, which, as we will discuss, is natural for a large collection of biomedical systems.
Empirically, we present a practical framework to instantiate our theoretical insights. We demonstrate the effectiveness of our approach through extensive experiments on both numerical and synthetic datasets. Results on a real-world human phenotype dataset are consistent with established biomedical research, validating our theoretical and methodological framework.

Published as a conference paper at ICLR 2025
CAUSAL REPRESENTATION LEARNING FROM MULTI- MODAL BIOMEDICAL OBSERVATIONS
Yuewen Sun 2,2 Lingjing Kong"* Guangyi Chen' Loka Li', Gongxu Luo',, Zijiann Li', Yixuan Zhang', Yujia Zheng", Mengyue Yang', Petar Stojanov', Eran Segal', Eric P. Xing Kun Zhang 'Mohamed bin Zayed University of Artificial Intelligence, "Carnegie Mellon University, 3 University of Bristol, "Broad Institute of MIT and Harvard
ABSTRACT
Prevalent in biomedical applications (e.g., human phenotype research), multi- modal datasets can provide valuable insights into the underlying physiological mechanisms. However, current machine learning (ML) models designed to ana- lyze these datasets often lack interpretability and identifiability guarantees, which are essential for biomedical research. Recent advances in causal representation learning have shown promise in identifying interpretable latent causal variables with formal theoretical guarantees. Unfortunately, most current work on multi- modal distributions either relies restrictive parametric assumptions yields only coarse identification results, limiting their applicability to biomedical re- search that favors a detailed understanding of the mechanisms.
In this work, we aim to develop flexible identification conditions for multimodal data and principled methods to facilitate the understanding of biomedical datasets. Theoretically, we consider nonparametric latent distribution (c.f., parametric as- sumptions in previous work) that allows for causal relationships across potentially different modalities. We establish identifiability guarantees for each latent com- ponent, extending the subspace identification results from previous work. Our key theoretical contribution is the structural sparsity of causal connections between modalities. which, as we will discuss, is natural for a large collection of biomed- ical systems. Empirically, we present a practical framework to instantiate our theoretical insights. We demonstrate the effectiveness of our approach through extensive experiments on both numerical and synthetic datasets. Results on a real-world human phenotype dataset are consistent with established biomedical research, validating our theoretical and methodological framework.
INTRODUCTION
Multimodal datasets provide rich and comprehensive insights into complex biomedical systems, offering the potential provide deeper understanding of physiological mechanisms. For example, the human phenotype dataset (Levine et al., 2024) contains measurements from multiple modalities, including anthropometric data, sleep monitoring, and genetic information. Proper analysis of such data can potentially uncover the underlying mechanisms that drive phenotypic diversity and disease susceptibility. leading to the discovery of novel molecular markers and the development of predictive models for disease. Recent advances in large-scale models have made it possible to exploit large biomedical datasets for various tasks such as protein structure prediction (Jumper et al., 2021; Lin et al., 2023), gene-disease association identification (Diaz Gonzalez al., 2023; Zagirova et al., 2023), and novel drug candidate discovery (Pal et al., 2023; Zheng et al., 2024b).
Despite the impressive performance of these models, their trustworthiness remains contentious issue (Zheng et al., 2023). A major concern lies in their lack of interpretability, which poses signifi- cant challenges in biomedical research and hinders the safe and ethical application of these models. For example, in clinical decision maaiinnnnmaiingggg (Hager et al., 2024), if the model recommends a specific
"Equal contribution.

---

Published as a conference paper at ICLR 2025
treatment plan for a patient, it is important for clinicians to understand the rationale behind the rec- ommendation. Without such transparency, it is difficult to trust the model's 's output or integrate these systems into critical decision- making workflows. Although several explainable models have been developed for multimodal datasets (Tang et al., 2023), this area remains largely underexplored.
Fortunately, recent advances in causal representation learning (CRL) (Schölkopf et al., 2021) have shown promise in identifying latent causal structures from raw observations, making it well-suited for biomedical applications. For example, plethora of CRL studies (Hyvarinen et al., 2019; Khe- makhem et al., 2020a; Zhang et al., 2024b; Buchholz et al., 2024; von Kügelgen et al., 2023; Zhang al., 2024a; Li et al., 2024c; Abuja et al., 2023) effectively utilize temporal information or domain indices to identify latent causal models and apply them in fMRI data. Recently, growing body of CRL. research has investigated multimodal distributions (Yao et al., 2023; Morioka & Hyvarinen, 2023; 2024; Daunhawer et al., 2023; Sturma et al., 2023; Gresele et al., 2020). These works leverage shared information across modalities to establish identifiability guarantees for latent variables (Yao et al., 2023; Morioka & Hyvarinen, 2024; Daunhawer et al., 2023). Despite these advancements, some aspects of these works are still limited. For instance, Von Kügelgen et al. (2021), Daunhawer et al. (2023); Yao et al. (2023) only focus on identifying latent subspaces that are directly shared by multiple modalities. In practice, however, many informative latent variables may influence multiple modalities indirectly through intermediate latent mechanisms. Moreover, such subspace identifiabil- ity loses track of the intricate causal influences among individual components, leading to a limited view of the underlying latent mechanism. Morioka & Hyvarinen (2024); Gresele et al. (2020); Morioka & Hyvarinen (2023) rely on specific assumptions about latent variable distributions (c.g., independence or exponential family). These constraints significantly limit their applicability for biomedical datasets that involve complex interactions among latent factors.
In this work, we aim to develop identification theory with multimodal biomedical datasets in mind, and design prin- cipled and interpretable models to facilitate analyzing such datasets. We assume that observations x(m) in each modal- ity m are generated by specific set of latent components (z/m²)) ): and allow for flexible causal relationships among la- tent components from potentially different modalities, such that 2/!)) 2/m) for m # n, i * j. shown in Figure 1. Theoretically, provide identifiability guarantees for each latent component 2!(²) thus generalizing the subspace iden- tification results in Yao et al. (2023); Daunhawer et al. (2023) while avoiding independence or parametric restrictions on the latent distribution p( (z(mm) m) as in Morioka Hyvarinen (2023; 2024): Gresele et al. (2020). In particular, we first show that any latent subspace (mm) can be identified as long as z (m) exerts sufficient influences on other modalities, which is weaker than assuming that (m)) is directly shared across multiple modalities as in Daunhawer al. (2023); Yao et al. (2023). Based on this subspace identification, we leverage the sparsity of the causal connections between modalities to further identify each latent component {2 }.- This notion of causal sparsity has been explored in recent work (Lachapelle et al., 2023; Xu et al., 2024; Zheng al., 2022) in other causal identification settings and has been shown realistic in many biomedical systems (Busiello et al., 2017; Milo et al., 2002; Babu et al., 2004; West et al., 2002; Banavar et al., 1999), as we will discuss further in Section 4.
Anthropometrics
Bload Pressure
Sleep
m
Morg Moodaite
POX
Fundus
Figure 1:
Multimodal data with
causal latent variables.
Empirically, we develop a theoretically grounded estimation framework to recover the latent com- ponents in each modality. Our model implements our theoretical conditions (in particular, con- ditional independence and sparsity constraints) on top of normalizing flow (Huang et al., 2018; Kobyzev et al., 2020) within the encoder-decoder framework. Extensive experiments on both nu- merical and synthetic datasets demonstrate its effectiveness. Most notably, our framework enables the identification of latent causal variables that capture complex biomedical interactions and facili- tates the analysis of potential causal mechanisms across modalities, which are important for clinical recision-making. The evaluation results on a real-world human phenotype dataset provide novel in- sights into the relationships between modalities, and the discovered causal relationships align with the findings from biomedical research, highlighting our contributions to the biomedical domain.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
ML models for biomedical research. For biomedical applications, ML models are developed to extract informative representations to facilitate downstream tasks, including DÑA sequence mod- eling (Zhou et al., 2024; Nguyen et al., 2023; Dalla-Torre et al., 2023), protein structure predic- tion (Jumper al., 2021; Lin et al., 2023), and disease detection (Zhou et al., 2023; Jang et al., 2024). The success of large language models (LLMs) has significantly advanced sequence model- ing for DNA, RNA, and proteins (Celaj et al., 2023; Shulgina et al., 2024; Nguyen et al., 2024; Li et al., 2023; Chen et al., 2023; Lin et al., 2023), yet these methods primarily operate a single modality, limiting their upplicability to the multimodal datasets, which are commonly encountered in biomedical research. Although several studies have explored integrating multimodal biomedical data (Garau-Luis et al., 2024; Pei et al., 2024; Taylor et al., 2022), these approaches often lack the- oretical guarantees, raising concerns about the reliability of their results. In this paper, we leverage causal principles to develop theoretically sound ML models for multimodal biomedical data, aiming to provide reliable and interpretable insights into complex biomedical systems.
Multimodal representation learning. Multimodal representation learning (Zhang et al., 2020; Manzoor et al., 2023) refers to the process of learning representations from multiple data modalities (e.g., text, image, audio) for specific tasks. Recent advances have leveraged contrastive learning techniques to improve the alignment of latent spaces across different modalities (Daunhawer et al., 2023; Wang al., 2022; Radford et al., 2021; Khosla al., 2020). Methods like CLIP (Radford et al., 2021) and Contrastive Predictive Coding (Oord et al., 2018) have demonstrated the ability to recover shared latent factors across modalities by (implicitly) maximizing mutual information be- tween representations. However, challenges remain in achieving finding modality-specift represen- tations, which requires novel upproaches that preserve the unique characteristics of each modality.
Identifiable CRL. CRL aims to identify high-level causal variables from low-level observations, integrating principles from both machine learning and causality (Schölkopf et al., 2021), 1), and can be viewed as extension of causal discovery (Spirtes et al., 2001; Li et al., 2024a; Luo et al., 2025; Li et al., 2024b; Ziu et al., 2024). CRL methods with identifiability guarantees can be classified based on the assumptions they impose, including sparsity constraints (Xu et al., 2024; Zheng et al., 2022; Zheng& & Zhang, 2023; Lachapelle et al., 2024), interreniionl/muti-diiriiioi et al., 2019; Khemakhem al., 2020a; Zhang al., 2024b; Kong et al., 2023; Buchholz al., 2024; von Kügelgen et al., 2023; Zhang et al., 2024a; Li et al., 2024c; Varici et al., 2023; Ahuja et al., 2023; Jiang& & Aragam, 2023), and of particular relevance to our work, multimodality (Yao et al., 2023; Morioka & Hyvarinen, 2023; 2024; Daunhawer et al., 2023; Sturma et al., 2023; Gresele et al., 2020). To provide a clearer comparison, Table 1 summarizes representative works in the multimodality category and highlights their differences from our work.
settings (Hyvarinen
Empirical CRL for multimodal applications. In contrast to the previously discussed works that emphasize identifiability, another line of multimodal research prioritizes practical applications in various domains, without addressing theoretical identifiability. Mao et al. (2022) assume inde- pendent latent variables and introduce a two-module amortized variational algorithm learn rep- resentations from medical images and biomedical data. Zheng et al. (2024a) develop a contrastive leamning-bal based approach to extract modality-speciffi and modality-invane invariiat representations from time-series tabular and textual data for root cause analysis. Rawls et al. (2021) leverage behavioral and psychiatric phenotyping alongside high-resolution neuroimaging data from the Human Con- nectome Project (Van Fssen et al., 2013), and perform greedy fast causal inference (Ogarrio et al., 2016) to investigate causal relations in alcohol use disorder. In contrast, our work establishes formal identification theory and integrates the theoretical insights into our estimation model.
Related work multimodal causal representation learning. This table considers whether method can accommodate more than two modalities, whether the latent variable distribution is nonparametric, whether it allows dependency among latent variables, and whether identifiability is component-wir wise.
Table 1:
Related work
> 2 Modalities Nonparam. Dist.
Latent Dependency Component-wiss Iden.
Gresele et al. (2020)
Von Kügelgen et al. (2021)
Daunhawer et al. (2023)
Yao al. (2023)
Morioka & Hyvarinen (2024)
Our work
V
3

---

Published as a conference paper at ICLR 2025
3 LATENT MULTIMODAL CAUSAL MODELS
Real-world biomedical datasets often integrate multiple modalities, each characterizing a unique yet interrelated aspect of the subject. For instance, the human phenotype dataset (Shilo al., 2021) consists of tabular data, time series, images, and text, capturing diverse biomedical measurements such anthropometrics, sleep monitoring, and genetic information. Understanding the latent factors behind each modality and their interplay can provide valuable insights into underlying biomedical mechanisms, ultimately facilitating the advancement of biomedical technologies. With this goal in mind, we formalize the multimodal data-generating processes as follows.
Figure 2: An illustrative exam- ple of the hypothesis space un- derlying the biomedical system.
-ata-geneggting processes. Let x == [x (2) x [)] be a set of observations/messsuremmsss
from M modalities, where (mm) E (m) represents the observation from modality m with dimensionality d(x("")). Let = (z("), [M] be the set of causally related latent variables underlying M modalities. Specifically, the data generation process (Figure 2) can be formulated as
(m) := m2(m) (Pa(z)")) ("))), c, )m)) (latent causal relations)
(1)
x !x(m) (m)) r7 (m))) (generating functions)
(2)
where Pa(-) denotes the parents of a variable. Since we allow for causal relations to exist within and across modalities, Pa(-) potentially includes latent variables across multiple modalities. The differentiable function encodes the latent causal graph connecting the latent components, and its
Jacobian matrix J 5. can be permuted into strictly triangular matrix. We denote c; (m) the exoge- nous variable for [(m)) where all exogenous variables are mutually independent. ny (m) represents domain-specific information independent of other components.
Example. In healthcare, different modalities capture complementary physiological aspects. A chest X-ray x (m) may reflect latent factors such as lung density, cardiac silhouette, and ribcage structure, represented by z!!"). These latent factors can causally influence those in other modali- ties, ( such as pulmonary function parameters (e.g., forced vital capacity) and cardiovascular biomarkers (e.g., left ventricular mass). These, in turn, may affect electrical activity recorded in an ECG, represented by x("), by modulating heart rate variability and conduction patterns.
Goal. As outlined previously, we aim to learn the latent variables underlying each modality and their causal relations. Formally, consider two specifications of the gettragenratiig process Eq. (1) and Eq. (2): == {gx(mm) (2(m) (atm))((((mm) ( |m== and ê := { (e)m), { M both which fit the marginal distribution p(x). Our objective is to show that, given the same value of x, each
estimated latent component 2!m) is equivalent to its true counterpart m! up to an invertible trans-
formation h. n(("), i.e., m(m) h nm!m) (=(m)). This component- identifiability ensures that latent components (e.g., gene types, nutrient levels) are disentangled from the observed measurements x while preserving their original information. Once component-wise identifiability is achieved, one can readily apply standard causal discovery algorithms (e.g., PC (Spirtes et al., 2001)) to the identi- fied components
e((m) to infer the graphical structures. The choice of structure learning algorithms can be tailored to the assumed graph class (e.g., potentially non-DAGs), and this step is orthogonal our contribution. These structures characterize the interactions between all latent components across modalities, which is particularly desirable for biomedical applications.
4 IDENTIFICATION THEORY
As motivated in Section 3, we address the component wise identifiability of latent components (m)
Please see Appendix for details on the notion of identifiability
4

---

Published as a conference paper at ICLR 2025
Remarks the problem. Identification for multimodal distributions often leverages the structure among the available modalities. However, component wise identification, especially in the general nonparametric setting, is challenging. Daunhawer al. (2023); Von Kügelgen et al. (2021); Yao et al. (2023) require certain information redundancy: the information of the latent variables should be fully shared and preserved by the observations of at least two modalities that is, we can express z(m) functions of x(m)) and x())) individually. Moreover, the identification can only be achieved up to subspaces (i.e., groups of latent components) determined by the sharing pattern. Often, how- ever, the latent components may not be fully shared by multiple modalities. For example, in health monitoring, while sleep monitoring data (e.g., sleep stages or duration) may not fully encode genetic predispositions, genetic factors may still influence sleep disorders, such as insomnia and circadian rhythm disruptions. In this case, the subspace identification may fall short of providing detailed interpretations of biomedical systems and the mechanisms encoded in the graphical structures over individual causal components.
For work that achieves component wise identifiability, Morioka & Hyvarinen (2023; 2024) assume that the latent distribution p( (m)) M m=1) follows exponential family form with additive causal influences from multiple parents, which may be restrictive in general cases. For instance, in brain imaging studies, fMRI data and EEG data capture different neural activities, and the interactions between brain regions are often highly nonlinear. Clearly, for general multimodal distributions (Figure 2), we cannot access the information redundancy assumed in Daunhawer et al. (2023); Von Kügelgen et al. (2021); Yao et al. (2023) and the nicely-behaved latent causal models in para- metric assumptions (Morioka & Hyvarinen, 2023; 2024).
Our high-level approach. We divide the problem into two parts: we first identify latent subspaces (mm) (Section 4.1) and further disentangle identified subspaces into components 2/!²) (Section 4.2). For the subspace identificatinn, we only assume that the information of the subspace (mm) is pre- served in its corresponding observation (m) and exerts sufficient influence on other modalities' ob- servations x( x(-m), thus weakening the redundancy assumption in previous work (Daunhawer al., 2023; Yao et al., 2023). For the component-wiss identification, we leverage a natural notion of struc- tural sparsity in the literature (Zheng et al., 2022: Lachapelle et al., 2024) the dependency among all the modalities should be explained with a minimal number of causal edges among latent sub- spaces {z (m) JM This allows us to further disentangle each subspace into components, without resorting to parametric assumptions (Morioka Hyvarinen, 2023; 2024).
Notations, We denote the dimensionality and the component indices of given argument with d(-) and 7(.), respectively. The notation -m represents the complement of modality m, while superscripts and subscripts enclosed in parentheses, such as (m), explicitly index modality m. We denote sub-matrices using the notation [.J,,c, where R and C are index sets corresponding to row and column selections, respectively. In this notation, setting Rt (or C') to indicates the inclusion of all indices along the corresponding dimension.
4.1 IDENTIFYING LATENT SUBSPACES
As previously discussed, now provide the subspace identifiability. Formally, we would like to show that the estimated latent subspace ((m) for any modality m and its true counterpart z (m) are equivalent up to an invertible map h(m)) (-), i.e., ((() = 1 (m)) (m(m)).
Given the data-generating process Eq. (2), the task is to remove modality-speciffi information ny m/mm) from the observational data (m) while retaining the latent variables (m)) causally related to other modalities. light of this, we express the relations between the latent variables z (m)) and the observation of its own modality x(m) and other modalities (-m) as Eq. (3).
m)) (x(mm) (z (m) (7()))(())) (m)) (-m)
.x mi (m) (f(-))),
where (-m) denotes all the information necessary to generate the complement group (-m) beyond z mm) 2 Consequently, (7") may admit causal/statistical relations with (m) We denote the joint
map of (xim) and fx(-m) g(mm) (z(m)) (7(1)), (mm)
We use differentiate (-m) from collection of modality-specific variables r defined in Fq. (2).
5

---

Interpretation and proof sketch. Theorem 4.2 states that one can disentangle the modality- specific information (T!) and the latent variables ((m)) contained in the observation (mm)) (which is a mixture of both). To achieve this, we leverage the fact that (m²) has no influence on other modal- Published as a conference paper at ICLR 2025
Condition 4.1 (Subspace Identifiability Conditions).
A1 [Smoothness & Invertibility]: The generating functions Ixim) and g(m) are smooth and have smooth inverse functions.
A2 [Linear Independencel: The generating function 9x(-n) is smooth and its Jacobian columns corresponding ((m)) (i.e., (J my ((x(m))) are linearly independent almost any
Discussion on the conditions. Condition 4.1-A1 requires that the information of the latent vari- ables 2 is preserved in its observation x(m),, so that the identification of latent variables is well- defined (Hyvarinen et al., 2019; Khemakhem et al., 2020a; Von Kügelgen et al., 2021; Kong et al., 2023; Yao al., 2023; Daunhawer et al., 2023). Since this holds for any modality m, the observa- tions x( (-m) should collectively preserve the information of other modalities z( ((-m).
Condition 4-1 formalizes the notation of a minimal connectivity over modalities: ((m) should also exert sufficient influence on other modalities 2(-m), so that the other modality observations (--m) could be informative to identify _!m) This condition excludes degenerate scenarios where the causal influences between modalities are nearly negligible and is equivalent to local invertibility of z 2(m), which is strictly weaker than the global invertibility assumption in previous work (Daun- hawer et al., 2023; Von Kügelgen et al., 2021; Yao et al., 2023) (e.g.. V = x² is locally invertible but not globally so), discussed earlier.
Theorem 4.2 (Subspace Identifiability). Let p(é { (g((m)))(((m)) p(é(m)) be two specifications of the data data-generating process in Eq. (3). Suppose that they generate identical observational distributions (i.e., p(x) = p(x)), O
0m) ( p(e! ((e p(((((m)))
and
① := 19x(m). 92(-m)
satisfies Condition 4. and @ satisfies Condition 4. 1-A. The latent subspace { for any group m
and its counterpart (m) are equivalent up to an invertible map h(m)()). Le., (m)) = (m) (z/m)).
ities x x while (m) has non-trivial influence on x(-)), as characterized in Condition 4.1-A2.
This crucial distinction provides sufficient footprints to disentangle these two subspaces for each modality, yielding the intended result.
4.2 IDENTIFYING LATENT COMPONENTS
Proceeding from the subspace identifiability (Theorem 4.2), we now further disentangle each sub- space into individual components (m) as outlined in Section 3. As foreshadowed, our key condition entails the sparsity of the graphical structures between modalities. Such dependency structures are captured in the generating function IN defined component-wis wise in Eq. (1), in particular its partial derivatives. We now introduce Condition 4.3, which facilitates component-we identification.
Additional notations. We denote the indices of the non-zero matrix entries by Supp(-). We denote the collection of partial
Alternatvve graph &
(=(")) as matrix func-
derivatives among all latent components 8z
tion Glzz e) E ()))))(())) We adopt diag(.) to denote matrices
consisting of equally-sized square matrices on its diagonal and define T to possess the structure T = diag(T).... TMr with
True graph G
invertible Tm. e (md( We denote the class of gener- alized permutation matrices of dimensionality d(z) as P(d(z)).
Figure 3: Denser graph G.
Condition 4.3 (Component Identifiability Conditions). Over the domain of (z, E), for any modality m and any T & P(((z)). we have
Σ
T
G (m))
Ta
G (Gmm))..))
(4)
m M
m M

---

Published as a conference paper at ICLR 2025
Discussion the conditions. Condition 4.3 stipulates sparse tross-modality causal connections among latent components z. Under this condition, if a latent component s!!²) is a function of two components 2 and 2/()) (when component-wis identification breaks down), the cross cross-modality
causal connections in G are guaranteed to be denser than those in G. We give a simple example to
aid intuition in Figure 3: for causal graph with three modalities, 211)
21() 21
(3) and 2/
22 suppose that x{²) is a non-trivial mixture of 2 and 22 and other components are
correctly identified, i.e., [=["] 22
=(1) e((2) =a(2)
233 = [=(") =2 M[= 22
(1)
=,) (2) (3) (3)
22
As a consequence, the alternative causal graph G would include additional edges #/2 + 21 and
x/3) + 2 giving rise strictly denser graph. In Theorem 4.4, show that this sparse struc- ture could give us the desired component-wise identifiability under a proper sparse regularization constraint. The availability of multiple modalities greatly improves the feasibility of such sparsity conditions, especially with a large number of modalities, because the entanglement is limited within a single modality (owing to Theorem 4. 1) and other modalities can be leveraged to provide space for sparse connections.
Sparsity conditions have been embraced by the causal representation learning commu- nity (Lachapelle et al., 2024; Moran et al., 2022; Fumero et al., 2023; Xu et al., 2024). Especially relevant to our work is Zheng et al. (2022). As discussed above, we are obliged to deal with causal structures among all latent variables. In contrast, Zheng et al. (2022) assumes the sparsity of the causal connections between the latent variables and the observed variables - the directions (from the directly observed. Notably, sparse
latent to the observed variables) are given and the children properties manifest in biomedical systems of our interest, including gene regulatory networks (Milo et al., 2002; Babu al., 2004; Nacher & Akutsu, 2013; Liu et al., 2011) 1), metabolic systems (West et al., 2002; Banavar et al.. 1999), and other living systems (Busiello et al., 2017), evidencing the plausibility of Condition 4.3 for biomedical applications.
Theorem 4.4 (Component-wise Identifiability). Let 0 = (f9a/m) Iatmo p(c ) } 1) and Ô == (10x (ytem)) ( ) M=) be two specifications of the data- generating process in Eq. (1) and
Eq. (2). Suppose that they generate identical observational distributions (i.e., p(x) = p(x)) and
satisfies Condition 4. I and Condition 4.3. If satisfies the following sparse regularization condition:
Σ ||GG)m) S
Σ M
(IG)lo)),
M
each component ((m and its counterpart *x()) are equivalent up invertible map h()), i.e.
a(m) (()) h(2/5)) under a permutation over (d(z(m))).
Interpretation and proof sketch. The key idea of Theorem 4.4 is that for sparse causal graphs (as characterized in Condition 4.3), the mixing of latent components in any modality would introduce unnecessary causal edges connecting the other modalities. As the sparsity regularization Eq. (5) selects alternative models that are not denser than the model 0, the mixing within each modality would be excluded. Consequently, each latent component z(m² is function of a unique component ((m) yielding the desired component-wise identifiability. 3j
Implications. In the context of biomedical applications, Theorem 4.4 indicates that under appro- priate constraints, each component 2 in our estimation uniquely captures the information of an intrinsic biomedical factor behind the medical measurements (e.g., genetic predisposition). There- fore, the learned representation enjoys strong interpretability under theoretical guarantees, which is often lacking in existing biomedical models. as noted in Section 2. Theorem 4.2 and Theorem 4.4 provide insights for practical model design, which we employ in our architecture in Section 5.
Shared latent variables. Certain applications may involve latent variables that are shared across modalities. In such scenarios, we can employ contrastive learning objectives and the associated the- oretical guarantees in previous work (Yao et al., 2023; Daunhawer et al., 2023; Von Kügelgen al., 2021) as a pre- precpesessing procedure and treat such shared latent variables as separate modalities in our implementation. Please refer to Appendix C.3, C.4, and E for detailed discussion and results.
7

---

Published as a conference paper at ICLR 2025
5 ESTIMATION MODEL ARCHITECTURES
Given the identifiability results, we further propose an estimation framework (shown in Figure 4) that enforces the proposed assumptions constraints to identify the latent variables in each modality.
Estimation framework. Given multi- modal observations (x(2), ".ti)....,,,.,, x(f)), the latent vari- ables and domain-specific information in modality m are inferred ((m) and fi(m) by individual encoders. The observations are then reconstructed with corre- sponding decoders as x(m), We enforce indepen- dence conditions by minimizing the KL divergence term DEL ((()())) J= (e); force the sparsity constraint by minimizing the L1 norm in the inferred adjacency matrix Å.
Figure 4:
(()
En(()
g(3)
(e
R())
Adjacency Matrix A
Flow
(M)
(M
(M)
(M
(M))
We en-
Encoder and decoder. Each modality x(m) is given as an input to the corresponding encoder and outputs the estimated latent (()) and domain-specific information f(m)) They are then concatenated and passed the corresponding decoder reconstruct the observations as {(()) The reconstruction
loss is calculated using the mean squared error (MSE) as LRccon
Σ E [|x(m) {
Conditional independence constraints. Given Eq (3), we enforce the conditional independence condition x!! (Tm) 1. (")) iz (mm) and the independence condition on (!(")) 1 (mm) by enforcing inde-
pendence among components in 7 [())(m) J.M Jmm { ) d(r)) Such equivalence is shown in Propo- sitions 5.1 and 5.2, and proof' are provided in Appendix B. Specifically, we minimize the KL diver- gence loss between the posterior and Gaussian prior distribution: Lind DKL (p(+)))N(( (o, I)).
Proposition 5.1. |Conditional Independence Condition] Let (()) and x( be two different multi- modal observations. z (m)) C z the set of block-identifying latent variables, and m/'m) C ni are domain speciffc information in modality m. We have x x(()) 1L x!") z((m) - mp(m) 1 Lr(())
Proposition 5.2. [Independent Noise Condition] Let z and r) be the block-identified latent variables and spmeii--pecific information, respectively, across all modalities. Denote c as the exogenous variables in the latent causal structure. We have 1 + 11 ll
Sparsity regularization. We use flow to estimate the exogenous variables e in Eq. (1) and imple- ment the causal relations through learnable adjacency matrix A. The binary values in Å represent the causal generation process between latent variables, e.g. A., indicates ł, is the parent of é, while Ai 0 means 2j dose not contribute to the generation of 2;. For each component &, we select its parents Pa(x:) based the adjacency matrix, and apply the flow transformation get é,
To encourage sparsity among the latent variables 2, we impose a regularization term on the learned adjacency matrix. Based on the sparsity assumption, the optimal causal graph should be the minimal one that still allows the model to accurately match the ground truth generative distribution. To achieve this, we reduce the dependencies between different components by adding a L1 penalty on the adjacency matrix, s.t., Lsp = |||]] |||-
Optimization.
The model parameters optimized using the combination objective:
L = D.Reeeo + Oind + Ctsp Csp
(6)
EXPERIMENT RESULTS
To evaluate the efficacy of our proposed method, we conduct extensive experiments on (1) numer- ical, (2) synthetic and (3) real-world datasets. In terms of the baselines, compare method
with: (1) BetaVAE (Higgins et al., 2017). which does not consider causal relations in the latent space. (2) CausalVAE (Yang al., 2020), which considers the causally related latent variables with
8

---

Published as a conference paper at ICLR 2025
single modality. (3) Multimodal contrastive learning (MCL) (Daunhawer et al., 2023), which covers the shared latent factors from multiple modalities. Throughout the experiments, we consider the following evaluation metrics: (1) Mean Correlation Coefficient (MCC) measures how well the estimated latent variables match the true ones, with an MCC of indicating perfect identifiability up component- wiiiee transformations. (2) R2 measures the proportion of variance in the ground truth latent that is explained by the estimated latent, with value of indicating that all variance is explained. (3) Structural Hamming Distance (SHD) compares graphs by their adjacency matrices, where a lower SHD indicates stronger similarity between graphs.
Tox Stst
00
Lakee aetalee
Late autalle
aariatie
(a) Causal comparison between esti- (b) Comparison of the identifiabil- (c) Identifiability result under
mated and graphs (SHD=0)).
ity result in different cases.
different sparsity ratios.
Figure 5: Numerical experiment results. (a) Successful recovery of the inter-modal causal graph.
(b) Baseline comparisons in different cases. (c) Result of sparsity ablation study.
6.1 NUMERICAL DATASET
Setup. In the numerical simulations, we consider three cases with different numbers of modali- ties and inter-modal causal relations. Case 15-dimensional observations across two modalities, each with two latent and one exogenous variable. Case 2: 20-dimensional observations across two modalities, each with three latent and one exogenous variable. Case 3: 30-dimensional observations across four modalities, each with two latent and exogenous variable. The nonparametric mixing function is simulated by a random MLP with LeakyReLU units, and the inter-modal latent variables sparse causally related. The detailed data-generation process is provided in Appendix D.I.
Results and ablation. Figure 5 shows the identifiability results in different cases, where the high MCC indicates the successful recovery of the latent variables. The inter-modal causal relations are successfully recovered (SHD==) and the causal comparison result in case is shown in Figure 5(a). The identifiability comparison results are shown in Figure 5(b) (MCL is not applicable in case due the two-modality constraint). CausalVAE requires additional supervision signals to establish identifiability, and MCL assumes content invariance and can only block bock-identify latent variables. In general, these baselines neither account for the multimodal setting nor the podality-speciff latent variables, and therefore do not recover the latent variables.
As an ablation study, we further show the consequences of violating the sparsity assumptions to validate our theorem. Based on case 2, we create four types of datasets with different sparsity ratios and report the MCC each scenario in Figure 5(c). The sparsity ratio represents the ratio of existing causal links to all possible causal links between modality-specific latent variables. A value of 0 indicates that the latent variables between modalities are fully connected, while higher values correspond to sparser connections. The result shows that identifiability can be better achieved with a higher sparsity ratio, and our framework outperforms other baselines in all scenarios.
6.2 SYNTHETIC DATASET: VARIANT MNIST
Setup. We manually create a variant of the MNIST dataset to encode causal relationships between different modalities, using colored MNIST (Arjovsky et al., 2019) and fashion MNIST (Xiao et al., 2017) as two different modalities. the image transparency. This horizontal position further serves as causal factor for the vertical position of the fashion items in the fashion MNIST, which influences image grayscale. This design ensures structured causal dependency across modalities while maintaining non-deterministic mapping. Further data descriptions are provided the Appendix D.2.
colored MNIST, horizontal position of the digit influences

---

Published as a conference paper at ICLR 2025
Results. Table 2 presents the results of the identifiability comparison, where higher MCC and R2 indicate better performance of our method. BetaVAR does not explicitly model causal relationships among latent variables, lead- ing to suboptimal recovery in our setting.
Table 2: The results of MNIST dataset.
MCL
BetaVAE CausalVAE
Ours
0.79 6e5 0.68 2-3
0.50 4 4e
0.90 1: 5
MCC 0.63 2 0.53 # te3
0.74 4: 2e
0.85 5
CausalVAE, which relies on additional supervision, fails to recover the latent variables effectively.
6.3 REAL-WORLD DATASET: HUMAN PHENOTYPE
The human phenotype dataset (Shilo et al., 2021) is a large-scale, longitudinal collection of pheno- typic profiles from diverse global population. It includes comprehensive human health data and provides a comprehensive view of health and disease factors. The dataset contains various types of participant information, categorized into tabular, time series, and image data. Specifically, it cludes health information across 30 modalities, such as blood tests, anthropometry, fundus imaging, etc. Detailed data descriptions can be found in the Appendix D.3.
Hand grip
[23] FRights
Medical Condition
[4]] hand's grip strength
[81] Cataract
(A2] fight hand' ario strength
(2)) FRightl
[62] Glaucoma
[A3] Dominant hand
[21]] Stees
[e3] Retinal detachment
[X3] Sleep
Sleep
D: Anthropometries
[es] Flight
[C]] Usygen seturation
[2]] Slesp
[01] Sex
Sieep effilency
[02] Age
[22] Rueh 3
[22]
[08] BkI
(22) Fleb?
Figure 6: Causal analysis results across different modalities, including hand grip, medical condi- tions, sleep. and anthropometrics. We ran the causal algorithm on all variables, but for clarity only report the causal relationships that have direct connections to the estimated latent variables.
In this work, we focus the time-series sleep monitoring dataset (Sleep) and the fundus imaging dataset for both left and right eyes (FLeft and FRight) to estimate the latent factors (Z1, 72, 73) underlying each modality. We applied the PC algorithm (Spirtes et al., 2001) to discover causal relationships between the estimated latent variables and other four additional tabular modalities (A. B. C, D), providing an implicit evaluation on the effectiveness. The result with direct causal relations is shown in Figure 6, where variables from the same modality have the same color and different modalities have different colors.
A key finding is that the discovered causal relationships are consistent with findings from medical research. For example, Sleep I shows direct causal relationship with Oxygen saturation, suggesting that sleep conditions may influence blood oxygen levels. This observation is consistent with previous studies (Wali et al., 2020). In addition, the fundus-related latent variables FRight and FLeft / have a direct causal relationship with Age, suggesting that aging plays an important role in changes in retinal health (Ege et al., 2002; Einbock et al., 2005). Interestingly, the fundus image of the right eye has direct causal relationship with the grip strength of the left hand, recently demonstrated in biomedical research (Bikbov et al., 2023; Qiu et al., 2020).
7 CONCLUSION AND LIMITATIONS
In this work, we develop a theoretically grounded framework for recovering latent causal variables from multi-modal observations. Extensive experimental results on synthetic and real-world datasets demonstrate the practical effectiveness of our approach. Limitations: Empirically, our framework assumes prior knowledge of the number of latent variables in each modality, which may be unreal- istic in real-world scenarios. Additionally, a detailed evaluation against the quantitative benchmarks used in biomedical models remains an area for future exploration.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGMENT
We would also like to acknowledge the support from NSF Award No. 2229881, AT Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract ROIHLI5980S and grants from Quris AI, Florin Court Capital, and MBZUAI-WIS Joint Program. The work of L. Kong is supported in part by NSF DMS-2134080 through an award Chi. P. Sto- janov was supported in part by the National Cancer Institute (NCI) grant number: K99CA277583- 01, and funding from the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard.
BIBLIOGRAPHY
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representa- tion learning. In International conference on machine learning, pp. 372-407. PMLR, 2023.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 1907.02999. 2019.
M Madan Babu, Nicholas M Luscombe, L Aravind, Mark Gerstein, and Sarah Teichmann. Struc- ture and evolution of transcriptional regulatory networks. Current opinion in structural biology, 14(3):283-25) 2004.
Jayanth R Banavar, Amos Maritan, and Andrea Rinaldo. Size and form in efficient transportation networks. Nature, 399(6732):1300 130-1123 1999.
Mukharram M Bikbov, Rinat M Zainullin, Timur R Gilmanshin, Ellina M Iakupova, Gyulli M Kazakhaeva, Songhomitra Panda-Jonas, Azaliia M Tuliakova, Albina Fakhretdinova, Leisan I Gilemzianova, and Jost B Jonas. Hand grip strength and ocular associations: the ural eye and medical study, British Journal of Ophthalmology. 107(10):167-11744 2023.
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general non- linear mixing. Advances in Neural Information Processing Systems, 36, 2024.
Daniel M Busiello, Samir Suweis, Jorge Hidalgo, and Amos Maritan. Explorability and the origin of network sparsity in living systems. Scientific reports, 7(1):12323, 2017.
Albi Celaj, Alice Jiexin Gao, Tammy TY Lau, Erle M Holgersen, Alston Lo, Varun Lodaya, Christo- pher B Cole, Robert Denroche, Carl Spickett, Omar Wagih, et al. An ma foundation model en- ables discovery of disease mechanisms and candidate therapeutics. bioRxiv, pp. 2023-09, 2023.
Ken Chen, Yue Zhou, Maolin Ding, Yu Wang, Zhixiang Ren, and Yuedong Yang. Self-supervised learning on millions of pre-mrna sequences improves sequence-based ma splicing prediction. bioRxiv, pp. 2023-01,202 2023.
Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314 1994.
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Hassan Sirelkhatim, Guillaume Richard, Marcin Skwark, Karim Beguir, Marie Lopez. and Thomas Pierrot. The nucleotide trans- former: Building and evaluating robust foundation models for human genomics. bioRxiv, 2023. doi: 10. 10.1100/2023.01.11 11. URL https:://wwwt biorxiv. www.vviooo/r//oooooooooooo httpd//.rr///oeteeeeeeeeeee coo//ot//tttttt/eeooeeeeeeo 2023/01/15/200
01. 01.11.11100.11111.11111111111111 123577779.
Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia Vogt. Identifi ability results for multimodal contrastive learning. In The Eleventh International Conference Learning Representations, 2023.
Armando Diaz Gonzalez, Kevin Hughes, Songhui Yue, and Sean T Hayes. Applying biobert to extract germline diseas associations for building knowledge graph from the biomedical literature. In Proceedings of the 2023 7th International Conference on Information System and Data Mining, pp. 37-42, 2023.
11

---

Published as a conference paper at ICLR 2025
Bernhard M Ege, Ole K Hejlesen, Ole V Larsen, and Toke Bek. The relationship between age and colour content in fundus images. Acta Ophthalmologica Scandinavica, 80(5):485 489, 2002.
Wilma Einbock, Andreas Moessner, Ute EK Schnurrbusch, Frank G Holz, Sebastian Wolf, and FAM Study Group. Changes in fundus autofluorescence in patients with age-related maculopathy. correlation to visual function: a prospective study. Graefe' 's Archive Clinical and Experimental Ophthalmology. 243:300- 305, 2005.
Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolà, Stefano Soatto, Bernhard Schölkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. Advances in Neural Information Processing Systems, 36:27682-27698, 2023.
Juan Jose Garau-Luis, Patrick Bordes, Liam Gonzalez, Masa Roller, Bernardo P. de Almeida, Lorenz Hexemer, Christopher Blum, Stefan Laurent, Jan Grzegorzesski Maren Lang. Thomas Pierrot, and Guillaume Richard. Multi-modal transfer learning between biological foundation models, 2024. URL https: //tp///aaiiv..a org/ abs /20..2406..2 14150.
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Uncertainty in Artificial Intelligence, pp. 217-227. PMLR, 2020.
Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, et al. Evaluation and mit- igation of the limitations of large language models in clinical decision-making. Nature medicine, 3009))2613-2622, 2024.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International conference on machine learning, pp. 2087-2087. PMLR, 2018
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Boa Jang, Youngbin Ahn, Eun Kyung Choe, Chang Ki Yoon, Hyuk Jin Choi, and Young-Gon Kim. A disease-specific foundation model using over 100k fundus images: Release and val- idation for abnormality and multi-disease classification on downstream tasks. arXiv preprint arXiv:2408.0877 08790, 2024.
Yibo Jiang and Bryon Aragam. Learning nonparametric latent causal graphs with unknown inter- ventions. Advances in Neural Information Processing Systems, 36:60468-660 00008866011,,,111111111111111181 2023.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-588) 2021.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International conference on artificial intelli- gence and statistics, pp. 2207-2217. PMLR, 2020a.
Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 33:12768-12778 2020b.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18666 8661-1877, 2020.
12

---

Published as a conference paper at ICLR 2025
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43 (11):3964-397, 2020.
Lingjing Kong, Shaoan Xie, Weiran Yao. Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial identifiability for domain adaptation. arXiv preprint arXiv:2306.065 10, 2023.
Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julien, and Quentin Bertrand. Synergies between disentanglement and sparsity: Gen- eralization and identifiability in multi-task learning. In International Conference Machine Learning, pp. 18171-18206. PMLR, 2023.
Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katic Everett, Rémi Le Priol, Alexan- dre Lacoste, and Simon Lacoste-Julien. Nonparametric partial disentanglement via mecha- nism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv preprint arXiv:2401.04800 2024.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmist// 1998.
Zachary Levine, Iris Kalka, Dmitry Kolobkov, Hagai Rossman, Anastasia Godneva, Smadar Shilo, Ayya Keshet, Daphna Weissglas- Volkov, Tal Shor, Alon Diament, al. Genome-wide association studies and polygenic risk score phenome-wide association studies across complex phenotypes in the human phenotype project. Med, 5(1)990-101, 2024.
Loka Li, Haoyue Dai, Hanin Al Ghothani, Biwei Huang, Jiji Zhang, Shahar Harel, Isaac Bentwich, Guangyi Chen, and Kun Zhang. On causal discovery in the presence of deterministic relations. In The Tiirt Annual Conference on Neural Information Processing Systems, 2024a.
Loka Li, Ignavier Ng, Gongxu Luo, Biwei Huang, Guangyi Chen, Tongliang Liu, Bin Gu, and Kun Zhang. Federated causal discovery from heterogeneous data. arXiv preprint arXiv:2402.1 13241, 2024b.
Sizhen Li, Saeed Moayedpour, Ruijiang Li, Michael Bailey, Saleh Riahi, Lorenzo Kogler-Anele Milad Miladi, Jacob Miner, Dinghai Zheng, Jun Wang, al. Codonbert: Large language models for mrna design and optimization. bioRxiv, pp. 2023-09, 2023.
Xiutian Li, Siqi Sun, and Rui Feng. Causal representation learning via counterfactual intervention. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 3234-3242, 2024c.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):11233 2023.
Yang- Yu Liu, JacnuJs Slotine, and Albent-László Barabási. Controllability of complex net- works. nature, 473(7346):667-77. 2011.
Gongxu Luo, Haoyue Dai, Boyang Sun, Loka Li, Biwei Huang, Petar Stojanov, and Kun Zhang. Gene regulatory network inference in the presence of selection bias and latent confounders. arXiv preprint arXiv:2501. 2025.
Muhammad Arslan Manzoor, Sarah Albarri, Ziting Xian, Zaiqiao Meng. Preslav Nakov, and Shang- song Liang. Multimodality representation learning: A survey on evolution, pretraining and its applications. ACM Transactions Multimedia Computing, Communications and Applications, 1033
Haiyi Mao, Hongfu Liu, Jason Xiaotian Dou, and Panayiotis V Benos. Towards cross- cross-modal modal causal structure and representation learning. In Machine Learning for Health, pp. 120-140. PMLR, 2022.
Ron Milo, Shai Shen-Or, Shalev Itzkovitz, Nadav Kashtan, Dmitri Chklovskii, and Uri Alon. Net- work motifs: simple building blocks of complex networks. Science, 29845794)8824-877, 2002.
13

---

Published as a conference paper at ICLR 2025
G Moran, D Sridhar, Y Wang, and D Blei. Identifiable deep generative models via sparse decoding.
Transactions on machine learning research, 2022.
Hiroshi Morioka and Aapo Hyvarinen. Connectivity-contastiii learning: Combining causal dis- covery and representation learning for multimodal data. In International conference on artificial intelligence and statistics, pp. 3399-3426. PMLR, 2023.
Hiroshi Morioka and Aapo Hyvarinen. Causal representation learning made identifiable by grouping of observational variables. In Forty-first International Conference Machine Learning, 2024.
Jose C Nacher and Tatsuya Akutsu. Structural controllability of unidirectional bipartite networks. Scientific reports. 3(1):1647, 2013.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, and Chris Ré. Hyenadma: Long-range genomic sequence modeling at single nucleotide resolution, 2023. URL https:t//aaaiiiiiaa org/ abs /20///30662 15794.
Eric Nguyen, Michael Poli, Matthew Durrant, Armin Thomas, Brian Kang, Jeremy Sullivan, Madelena Y Ng, Ashley Lewis, Aman Patel, Aaron Lou, et al. Sequence modeling and design from molecular to genome scale with evo. BioRxiv, pp. 2024-02, 2024.
Juan Miguel Ogarrio, Peter Spirtes, and Joe Ramsey. A hybrid causal search algorithm for latent variable models. In Conference on probabilistic graphical models, pp. 368-379. PMLR, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.0377 2018
Soumen Pal, Manojit Bhattacharya, Md Aminul Islam, and Chiranjib Chakraborty. Chatgpt or Ilm in next-generation drug discovery and development: pharmaceutical and biotechnology companies can make of the artificial intelligence- based device for faster way of drug discovery and development. International Journal of Surgery, 109(12)44382-433 2023.
Qizhi Pei, Lijun Wu, Kaiyuan Gao, Xiaozhuan Liang, Yin Fang. Jinhua Zhu, Shufang Xie, Tao Qin, and Rui Yan. BiotS+: Towards generalized biological understanding with iupac integration and multi-task tuning. arXiv preprint arXiv:2402. 2024.
Alejandro Pérez-Castilla, Amador Ramos, Beatríz Redondo, Fernández-Revellee Andrés, Raimundo Jiménez, and Jesús Vera. Determinant factors of intraocular pressure responses to a maximal isometric handgrip test: hand dominance, handgrip strength and sex. Current Eye Research, 46(1):64-70. 2021.
Zihan Qiu, Wei Wang, Yan Tan, Miao He, Langhua Wang. Yuting Li, Xia Gong, and Wenyong Huang. Associations of grip strength with retinal and choroidal thickness in patients with type diabetes mellitus without retinopathy: a cross-sectional study. BMJ open, 10(7):e036782, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 9748-8763. PMLR, 2021.
Eric Rawls, Erich Kummerfeld, and Anna Zilverstand. An integrated multimodal model of alcohol use disorder generated by data-driven causal discovery analysis. Communications biology, 4(1): 435, 2021.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109-533612-634 2021.
Smadar Shilo, Noam Bar, Ayya Keshet, Yeela Halmor-Barkan Hagai Rossman, Anastasia Godneva, Yaron Aviy, Yochai Edlitz, Lee Reicher, Dmitry Kolobkov, et al. 10 k: a large-scale prospective longitudinal study in israel. European journal of epidemiology, 36(11 1817 2021.
14

---

Published as a conference paper at ICLR 2025
Yekaterina Shulgina, Marena I Trinidad, Conner J Langeberg. Hunter Nisonoff, Seyone Chithrananda, Petr Skopintsev, Amos J Nissley, Jaymin Patel, Ron Boger, Honglue Shi, et al. Rna language models prediet mutations that improve ma function. bioRxiv, 2024.
M Slabaugh, Chen, Smit, and Glaucoma Today. Cataract surgery and iop. Glaucoma Today, pp- 17-8, 2013.
Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2001.
Nils Sturma, Chandler Squires, Mathias Drton, and Caroline Uhler. Unpaired multi-domain causal representation learning. Advances in Neural Information Processing Systems, 36, 2023.
Xin Tang, Jiawei Zhang, Yichun He, Xinhe Zhang. Zuwan Lin, Sebastian Partarrieu, Emma Bou Hanna, Zhaolin Ren, Hao Shen, Yuhong Yang, et al. Explainable multi-task learning for multi- modality biological data analysis. Nature communications, 14(1):2546, 2023.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: large language model for science, 2022. UtL wtw.v.iiiivv//eee
wwwpsvvooree org/ /tbs//2211. 2211.09085......
David C Van Essen, Stephen Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, Wu-Minn HCP Consortium, et al. The wu-minn human connectome project: an overview. Neuroimage, 80:62-79, 2013.
Burak Varici, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score- based causal representation learning with interventions. arXiv preprint arXiv:2301.08230, 2023.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 34:16451-16467, 2021.
Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Barein- boim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal represen- tations from unknown interventions. Advances in Neural Information Processing Systems, 36, 2023.
Siraj Omar Wali, Bahaa Abaalkhail, Tbrahim AlQassas, Faris Alhejaili, David W Spence, and Sei- thikurippu R Pandi-Perumal The correlation between oxygen saturation indices and the standard obstructive sleep apnea severity, Annals of thoracic medicine, 15(2):70-75, 2020.
Teng Wang, Wenhao Jiang, Zhichao Lu, Feng Zheng, Ran Cheng, Chengguo Yin, and Ping Luo. Vlmixer: Unpaired vision-language pre-training via cross-modal cutmix. In International Con- ference on Machine Learning, pp. 22680-22690. PMLR, 2022.
Geoffrey B West, Brian J Enquist, and James H Brown. Modelling universality and scaling. Nature, 420(6916):626 627, 2002.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv:/708.0777 2017.
Danru Xu, Dingling Yao, Sébastien Lachapelle, Perouz Taslakian, Julius von Kügelgen, Francesco Locatello, and Sara Magliacane. A sparsity principle for partially observable causal representation learning. arXiv preprint arXiv:2403.08833. 2024.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. arXiv preprint arXiv:2004.08697, 2020.
Dingling Yao, Danru Xu, Sébastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability. arXiv preprint S0ii:2311.00005. 2023.
15

---

Published as a conference paper at ICLR 2025
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data, arXiv preprint arXiv:2110.05428, 2021.
Diana Zagirova, Stefan Pushkov, Geoffrey Ho Duen Leung, Bonnie Hei Man Liu, Anatoly Urban, Denis Sidorenko, Aleksandr Kalashnikov, Ekaterina Kozlova, Vladimir Naumov, Frank W Pun, et al. Biomedical generative pre-trained based transformer language model for age-related disease target discovery. Aging (Albany NY), 15(18)9223, 2023.
Chao Zhang. Zichao Yang. Xiaodong He, and Li Deng. Multimodal intelligence: Representation learning, information fusion, and applications. IEEE Journal of Selected Topics in Signal Pro- cexsing, 14(3):478-493, 2020.
Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. Advances in Neural Information Processing Systems, 36, 2024a.
Kun Zhang, Shaoan Xie, Ignavier Ng, and Yujia Zheng. Causal representation learning from multi- ple distributions: general setting. arXiv preprint arXiv:2402.05005, 2024b.
Lecheng Zheng, Zhengzhang Chen, Jingrui He, and Haifeng Chen. Multi-modal causal structure learning and root cause analysis. arXiv preprint arXiv:2402.02357, 2024a.
Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh TN Nguyen, Lauren T May, Geoffrey I Webb, and Shirui Pan. Large language models for scientific synthesis, inference and explanation. arXiv preprint arXiv:2310.0799. 2023.
Yizhen Zheng, Huan Yee Koh, Maddie Yang. Li Li, Lauren May, Geoffrey Webb, Shirui Pan, and George Church. Large language models in drug discovery and development: From disease mechanisms clinical trials. arXiv preprint arXiv:2409.0448) 2024b.
Yujia Zheng and Kun Zhang. Generalizing nonlinear ica beyond structural sparsity. Advances in Neural Information Processing Systems, 36: 011333-133355 2023.
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. Advances in neural information processing systems, 35:16411-16422 2022.
Yukun Zhou, Mark A Chia, Siegfried K Wagner, Murat S Ayhan, Dominic J Williamson, Robbert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Court, et founda- tion model for generalizable disease detection from retinal images. Nature, (22(7981):156-4 2023.
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu, Dnabert- 2: Efficient foundation model and benchmark for multi-species genome, 2024. URL https: www.ii.org/aaaaaa.eae.aaoaeaaaaa abs http/bba////// 220.22306..000 15006.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process, In International Conference on Machine Learning, pp. 12979-12999 PMLR, 2021.
Klea Ziu, Slavomír Hanzely, Loka Li, Kun Zhang, Martin Takáč, and Dmitry Kamzolov. v dag: Projected stochastic approximation iteration for dag structure learning. arXiv preprint arXiv:2410.2386,, 2024.
16

---

Published as a conference paper at ICLR 2025
Supplementary Materials for "Causal Representation Learning from Multimodal Biomedical Observations"
CONTENTS
Notation and Terminology
18
Constraints in the Estimation Framework
18
Identifiability Theory
20
C.1 Proof for Theorem 4.2
20
C.2 Proof for Theorem 4.4
21
C.3 Extended Theorem 4.2 and its Proof
23
C,4 Extended Theorem 4.4 and its Proof
24
Experimental Details
27
D.I Numerical Dataset
27
D.2 Synthetic Dataset
27
D.3 Real-world Dataset
28
D.4 Evaluation Metrics
28
D.5 Detailed Discussion on Human Phenotype
29
Extended Experiment
29
F
Implementation Details
30
F1 Network Architecture
31
F22 Training Details
31
G Algorithm Pseudocode
31
17

---

Published as a conference paper at ICLR 2025
A
NOTATION AND TERMINOLOGY
We summarize the notations used throughout the paper in Table 3.
Index
m, n
Modality index Variable element index Component indices of a given argument
i.j
I()
d())
Dimensionality indices of given argument
Variable
(m)
Observatioo/meessuremenn in each modality Causally related latent variables in each modality
z (m)
(m)) m))
One specific observation in modality m, and the rest others One specific latent variables in modality m, and the rest of others Domain--peciiic information
(m) z mm)
77
Mutually independent exogenous variables Estimated latent variables over Reconstructed observation in modality m
2, (m)
Pa(2(m))
Set of direct cause nodes/parents of variable ((m)
Function and Hyperparameter
9=m
Causal function among latent variables Nonparametric mixing function in modality m Invertible mapping from true latent to the estimated latent Distribution function (e.g., Ps, is the distribution of 24)
(x(m)
h
p
O'Recon Ot]nd CtSp
Weights in the combination objective
Table 3: List of notations.
Notions of identifiability. Following the literature on ICA (Hyvarinen & Morioka, 2016; Hyvari- nen et al., 2019; Comon, 1994) and causal representation learning (Yao et al., 2023; Von Küggllgen et al., 2021; Daunhawer et al., 2023), we assume that the generating function 9x(m) (Eq. (2)) is an invertible map from (2(")), _/(m) to (m)) (Condition 4.1). Under this invertibility assumption, given the value of (m)), one can perfectly determine the value of (1"), which is essentially the posterior p(z|x) (a point mass here). Here, (m) is function of (m),, and the identifiability of gives rise to the result that (!n) values can be identified from x (m²)
In statistics, to show identifiability, start with equal distributions Pe. - -oe derive the equiv- alence of the parameters Φ1 Φ2. In our case, since the functions Ixim', 9x(mm) are invertible, one can reason about the relation between the two specifications [x(m) and fxim) through composi- tion := 9xtma Ix(m). For instance, h is the identity when the specifications are identical. Similarly, in this work, we start with equal values of x to establish the relation between z and 2.
B CONSTRAINTS IN THE ESTIMATION FRAMEWORK
Here provide the proofs for the constraints utilized in the estimation framework.
Proposition B.1. [Conditional Independence Condition] Let x(m) and x()) be two different multi-
modal observations. (m) C z are the set of block-identifying latent variables, and m/(m) C are domain-specific information in modality m. We have
(m) 11 (n)
(m) < 17 (mm) L (n)
Proof. Given the data generation process in Eq. (2), the following assumptions hold true for any m,n [M]: (1) (m) 1. (m)) (2) (()) Il (m); (3) n7 (m) xm))
18

---

Published as a conference paper at ICLR 2025
Sufficient condition.
Given LHS of Eq. (7), we have
p(x (m) x (n)
2 (m))
mmx(m)
(p((()) 2m())
p(x(m)) ((())) (m)))) p(x((m) x(()), x(()),(()) p(n)(m), splin) al())
det Ory(m)
(x(mm ||det Ory(1))
RHS
p(z(m))
p(z(m))
(x((n)
(m)
Onl(n)
= p(m)(m) 1/(1))
2 (m)) )
det
||det
Ox(m))
8x(1) (x)
(m)) p(x(mm)
(m)) p(x(m) 2 (m)
LHS
p(x(m))
p(z(m))
p(z(m))
p(y)(m) 2(()
On (m)
p(n)(n) [n] (m))
/der 8ry(1)
det {x(m))
p(z(m))
p(z(m))
Öx(m)
-(()/m) zz (z(m)))((("))) (m) |z(")) Idet Ory try(m)
(m)
(ny(n))
||det Ox(m))
(m)
Thus have
p(7 (m) (z(m)) = p() (2 (m)) (n( (2))) = p(npim), (p)lm),,(m))))) ,7(10) (((((!))(()))))) + m/(m) 1 r7(m)
Necessary condition. Given RHS of Eq. (7) and above conclusion, we have
p(x((m) (m) p() det (m)
)ry(n)
p(x(()) mm))) p(nn""))
0x (x(m)
Ox(n)
)p(n ydet Br
|det Orp(")
Multiplication (mx(mm) (m)) ((()))((((())))))))))))))))))() (m)) = (((n(")))(()"))))))))))))))))))
Ox(mj
0x(m)
= (m( mn(m)) Idet mm(mm) (m) (())(m), sn(m)) 2()))
Bry (m)(m) (m)
Ory(1)
p(x (m)
x(11) m())
@x(m)) |ldet
Idet
Ox (12) |ddet Ox((1)
0x(n) ((z(m))
p(x ((z(m))
= p(x(m)) )x(x (m)) p(x(m), x(1)) m)) + x (m)
1 x (1)
(m)
(8)
Proposition B.2 (Independent Noise Condition). Let z and n be the block-identified latent variables and Lomain-specify information, respectively, across all modalities. Denote e as the exogenous variables in the latent causal structure. We have
1 L
(9)
Proof. Given the causal function in Eq. (1), we have p(z) = ple) &
Sufficient condition. Suppose (2, li(,, and v 1. 2, have
p(z.n) = nddef de
dc
det de
de dz
= p(z)p(n) p(c, n) def
= p(c))(n))
= p(p,,
Oz
dz
Oz
p(E)p(n) p(e,n) 1 Le
(10)
Necessary condition. Suppose (z, 1)) hi(e, v)) and 1. e, we have
p(z. n) p(c, m)ldef dc
p(z,n) def dc Oz
pp(n) p(z,n) = p(z)p(n) = n Lz
Oz
(II)
19

---

Published as a conference paper at ICLR 2025
C IDENTIFIABILITY THEORY
C.I PROOF FOR THEOREM 4.2
We present the proof for Theorem 4.2. For ease of reference, duplicate Condition 4.1 and Theo- rem 4.2 below.
Condition 4.1 (Subspace Identifiability Conditions).
Al [Smoothness & Invertibility]: The generating functions Ix(m) and gfm) are smooth and have smooth inverse functions.
A2 [Linear Independencel:: The generating function Oxt my is smooth and its Jacobian columns corresponding to ((m²) (i.e., [JB.-mm)l..(ammm are linearly independent almost anywhere.
Theorem 4.2 (Subspace Identifiability). Lei 0 == ( (iaem),,(e!mm) ),p(),,(((e" ( and
i (0x(m)) Iet-m), m) p(((m)) p(c(-m))) be two specifications of the data- data-generating process in Eq. (3). Suppose that they generate identical observational distributions (i.e. p(x) = p(x)), aoy satisfies Condition 4. J, and @ satisfies Condition 4.1-A1. The latent subspace ((m) for any group m and counterport z ()) are equivalent up to an invertible map h h(m) (-), ie.,, (m)) (mm) (z (m)).
Proof. Given the generating processes in Eq. (2) and Eq. (I), can express any observed group x (m) and its complement (-m)
x xmm) as two views of the latent variables of group mi
x x(m) ( (z(m) .n (m))
(12)
( mm) = (-m (mm) ,)(-)), ),,
(13)
where (m) stands for exogenous variables for the group x (m) and 7)( represents all the infor- mation necessary to generate the complement group x( mm) beyond z (m).
Following the classic definition of identifiability, we define two specifications O (9x(m) Igim), , p(e(m)))
and Ô := (mx(m)) 92(m), (( that fit the observation distri-
bution p(x). To show the identifiability in terms of the functions in f and Ô, we show that given the same x (")) value the identifiability between (mm) and (mm)
Thus, the subspace identification is equivalent to show that for each group m, the estimated latent variable (m)) and the true counterpart are related via an invertible map h. i.e., ((m) h)(((m)
Eq. (12) and the invertibility of the map (z, 17 (m)) ry (())) + (x x x()))) (Condition 4.1-A1) give rise to an invertible map h (2(")) ((m)) (-m))) + (z (m)) .77 . (-)))
The matched observed distribution between the true and the estimated models for the generating process Eq. (13) yields that
mm) (z (m) (m). (/(-m)) = (2 f((m))
(14)
Plugging in h gives
( (z( (m) (((m)) = g( ( (m)))) (p(m) ;/( mm(=))]]]
(
(15)
where we adopt /(-) to indicate the indices of its argument.
For any i E [d(x(m)) and E (m))) take partial derivative w.r.t. mn on both sides of Eq. (15):
a [ 0gg ( afijtm)] m]
(16)
The left-hand side of Eq. (15) equals to because g(-m) is not function of (m))
20

---

Published as a conference paper at ICLR 2025
Therefore, expanding the right-hand side of Eq. (15) gives:
olg
alg(-m0)
0. (17)
kE (1) ah
be (x)
The first equality in Eq. (17) is due to the fact that (/( is function of (--m) and varying (mm) doesn't vary x (-m) (1)m) is a function of x(m) thanks to the invertibility g(m))), i.e. J 0.
øl m
Condition 4.1-A2 implies that the matrix (o m11:) has a full column rank. Therefore, its
8(k),
null space contains only a zero vector, which, together with Eq. (17), implies that a(a(m) 0. Consequently, given the generating process Eq. (12) and the invertibility of g(m) and g(m)) (Condi- tion 4. I), the estimated latent variable ((m) and the true latent variable (mm) are related via an invertible map, as desired.
Digtmi]
C.2 PROOF FOR THEOREM 4.4
We present the proof for Theorem 4.4. For ease of reference, we duplicate Condition 4.3 and Theo- rem 4.4.
Condition 4.3 (Component Identifiability Conditions). and any T P(d(z)), we have
Over the domain of (z, e). for any modality
Σ [M]
[G]
Ta [[(()))
Σ
G (mlm
(4)
D
m M
Theorem 4.4 (Component-wise Identifiability). Let 0 = (19x(t)) Iatms p( ( and Ô := ((0ggtm) im p (m) ) be two specifications of the data- getardegneratnng process in Eq. (1) and
Eq. (2). Suppose that they generate identical observational distributions (i.e., p(x) p(x)) and satisfies Condition 4. and Condition 4.3. If satisfies the following sparse regularization condition:
Σ m
Σ
[[(]]m)),o)llo
(5)
each component ((m) and its counterpart g(m) *7() are equivalent up invertible map h()), i.e.
=(m) x()) h((2(")) under a permutation over (d(z(m))].
Proof. Given Theorem 4.2, Condition 4.1 implies that the estimated group-wise latent variable (m) is related to the true variable (m) through an invertible transformation (mm), i.c.,
z (m)) (m) (z (m)).
(18)
follows that the Jacobian matrix Tos can be arranged into a block-diagonal matrix, in which diagonal block m corresponds to Jacobian matrix T Tegim). Then, the goal is to prove that these
mm)m) diagonal blocks are actually generalized permutation matrices, whose each column only contains one nonzero entry.
We divide the proof into several steps for the sake of exposition. At step 1, derive equiv-
alence relation between the estimation model (ğg. â.) and the true model (gz: J.). At step 2, we apply Theorem 4.2 the equivalence characterize the relation between the true and the estimated graph structure. At step 3, we leverage the sparsity condition (Condition 4.3) to reason about the
identifiability of each component 2 (m) for E [M] and i [d(=(m))).
21

---

Published as a conference paper at ICLR 2025
Step 1. The generating process in Eq. (1) and the subspace identification Eq. (18) imply
92 E) h o (2, c),
(19)
where h is defined as the Cartesian product of individual (m)) functions.
Taking partial derivatives w.r., 2, of both sides of Eq. (19) yields:
[G
T
OR
Tos G D2
(20)
Tee
Ds
Each T matrix is the Jacobian matrix consisting the corresponding partial derivatives. We use G ah to denote the derivatives from the function which encodes the dependence structure among components. The same applies to G As discussed above, the matrix T has block-diagonal 82
structure (after proper permutations) with block m corresponding to the Jacobian matrix of h (mm). Moreover, the matrix us strictly diagonal due to the generating function Eq. (1).
Step In this step, we simplify Eq. (20) to derive the relation between the estimated graph struc- tures and true graph structures encoded in G and G as respectively.
First, note that the Tee is also block-diagonal w.r.t. the groups. To see this, compute the
ar(m) 0e!m) ag(m) a3(m) a=!"). where we denote that output of 9. with
partial derivatives therein follows:
(N)
in the derivative. Due to the equivalent relation z = (Eq. (1)), we have 03(m) az!")
as! az m)
which is
zero for distinct groups + (Eq. (18)). It follows that
de (m))
0, m n.
(21)
0z (r)
Therefore, we have shown that Tox is block-diagonal w.r.t. the groups.
This structure allows us to simplify Eq. (20) to directly characterize the relation between the two graphical structures G d5 and G In partieular, since Te. is block-diagonal and Tax is diagonal, 8h
the off-diagonal blocks on the left-hand side of Eq. (20) are determined by G os Tas Therefore, follows from Eq. (20):
Ga Tog FF 05
ue G Ga B2
m. + n,
(22)
(m).(n)
82
(m).(x)
where we adopt subscripts (m) to denote the block for group On account of the block-diagonal structure Tos, the left-hand side of Eq. (22) can be expressed as follows:
G G Te (m).
G
Tes (.(n)
G d5
Tox (w))(m)
(23)
(m)),
(m)
Analogously, the right-hand side of Eq. (22) can be expressed as:
T
T
G
T (m).(m)
G dz (m)))n)
(24)
(m))nn)
(m))
(.)m)
It follows from Eq. (22), Eq. (23), and Eq. (24) that
G 22
Te
T (m))mm)
G us
(m).(n)
(n))(n)
(m).
G
Top 8x (m).(m)
G Uu
Tox na
(25)
(m)
(m))
(m).(n)
Eq. (25) relates the true off-diagonal (m + n) structure (m)).n)
G dzz
and its estimated counterpart
G
(m)
22

---

Published as a conference paper at ICLR 2025
We now reason about the component-win wise identifiability within each modality through the sparsity of the off-diagonal regions.
Step 3.
The component-we wwise identifiability is equivalent that each block sub-matrix generalized permutation matrix, each row/column of which contains only one nonzero element. Suppose that this was not the case, then it would follow from Eq. (25) and Condition 4.3 that
Toa is
(m))
Σ
Gem 02 (m).(n)
Σ
Tue 0z
Gaa oz (m))(n)
Tou 8z
(m).(m)
(m).(n)
[M]
myfne|M [M]
(26)
Σ
G Uz (m),6w)
Condition 4.3 [
which would violate the sparsity constraint Eq. (5).
That is, the component (m)) cannot functionally influence components in ((m)) other than z mm) Therefore, we have shown that each block sub-matrix
Tea 8x (m))(m)
is a generalized permutation
matrix. Consequently, have bijection =!m² h (2 (mm)). Since this holds for any group m
(m)
and any component we have arrived the desired conclusion.
C.3 EXTENDED THEOREM 4.2 AND ITS PROOF
We restate Theorem C.5 from Yao et al. (2023), which we invoke in our Theorem C.7. We drop the entropy regularization term in Yao et al. (2023), since we assume the invertibility of estimated functions ((m) directly.
Definition C.1 Viee-Specificc Encoders). The view-specific encoders R == (re 2g + Zs hkv consist of smooth functions mapping from the respective observation spaces to the view-specific latent space, where the dimension of the 4th latent space |S&] assumed known for all E V.
Definition C.2 (Selection). A selection operates between two vectors E {0, b Rd s.t.
a Ø = [by : 0j = 1,j E [d]
Definition C.3 (Content Selectors). The content selectors + = {ó(i, k)]vien,ev with glick) e {0, (1)) perform selection C.2 the encoded information: for any subset Vú2 [M] and view l Vi we have the selected representation: ((i, m) (1))) with [[ ()())))) for all
V, k, V..
Definition C.4 IiformationnShariii Regularizer). The following regularizer penalizes the bu- norm Ilo of the content selectors Φ: Reg( == = Evvev Σtev, |1 llo
Theorem C.5 View-SSpcific Encoder for Identifiability (Yao al., 2023)). Let R (9(m) } m== and & respectively be the generating functions and content selectors (Definition C.3) that solve the following constrained optimization problem:
min Reg($)
subject to:
R, Φ E arg min Laligrment (R, +).
(27)
where
Laligament (R, 2)
Σ V.EW
Σ mh MVVE
E
[ (i) mi) (g(m))) <(i,
1
keck'
(28)
Then for any subset of modalities V C [M] and any modality m Vi, d(i, m) 2 [g(m)]] the shared subspace z (fmev,m). m).
identifies
Definition C.6 (Reconstruction Loss). The following loss penalizes the deviation of the estimate and its corresponding counterpart in lg Trrecom Fx (x x).
23

---

Published as a conference paper at ICLR 2025
Additional notations. We slightly abuse the notation to denote both sets and vectors with bold symbols 2. Let ((m(m)) be the set of latent components shared by modality m and n, i.e., ((m(m)
z (mm) n z(n), Analogously, let z ((m))n) be the set of latent components in modality m that are not shared by n, i.e., mmm)))) == (m)) (m)).
and (ggtm) :00xm)):xxtmm)) 3) be two specifications of the generating process Eq. (3) with potentially shared variables z(m(n) over any two modalities m and n. Suppose that they both match the observational distribution p(x) f(xx and satisfy Condition 4.1. Then any subspace (m)) shared subspace 2 (mmm)) and their counterparts z. (mm), z mn))) are equivalent up to invertible maps.
Theorem C.7 (Generalized Subspace Identifiability). Let {(gx(m) mx)( )
Proof. We note that the latent model with shared latent variables across modalities can still be cast into Equation (3) and satisfies Condition 4.1. As a consequence, Theorem 4.2 gives us the subspace identification for each modality as in the disjoint case. Moreover, we can identify any blocks among modalities thanks to Theorem C.5. This concludes the proof.
C.4 EXTENDED THEOREM 4.4 AND ITS PROOF
Additional notations and discussion. The participation of multiple modalities requires a new definition of the shared blocks in since the sharing structure could be nested and various numbers of modalities could share one partition. We partition the entire latent space into disjoint blocks (z(") bbBB. whose components 2 have exactly the same modality membership M(2) := {m e [M]] : :2 z(m)). We define the (²(() as the smallest (the least components) identified block in z that contains (b)) In the two-modal case, we have B = {(m n),, (m \ nn), (n \ m)) and z ((m)m)) = =x(1))
We denote z (b1) x z (b2) if block (bb) is shared by a strict subset of modalities that share z ((bx)) ie., M(2(bi)) ç M(z((x)). Therefore, we have either (b) = (b)) (it is identifiable itself) or
(b)) < H(b) (0)) (it is not identifiable by itself but belongs to an identifiable block z ((b) together with more deeply shared block z (b) \ ())) We denote former blocks as b+ E B i.e., (b+))
z(b*), and the latter blocks as b E B = B B+. In the two-modal case, the modal- speciff blocks z (m)) z (x))m) are not identifiable themselves, and we have z (())))))) (m(nn)) and z (m)mm) (mann) and B = (m n n)} and B = {(m \ n), (n \ m)).
We note that all shared blocks (bb+) are identified. Thus their bijective indeterminacies are w.r., themselves, i.e., ((6)) + 2(0*), which implies the square shape of their indeterminacy matrices In contrast, the unidentifiable blocks 2b6²) can potentially receive the influence ).(b)). (b+))))()))
(T Dh
from all other blocks in its minimal block z "bb)", However, (b²)) do not influence the comple- ment block (b(6-) 2((=), since (b²)) x z (b²)), For instance, (mm))))) may receive influ- ences from (m(")) and z (m(m)))) Consequently, their associated non-trivial indeterminacy matrices are (T }.H((-) (c.g.. [Tgg) (m)) Thus, the indeterminacy matrix T can be expressed as T = Ton + Toff. where The matrix Ton contains all the on-diagonal square invertible matrices Tan == diag(T!
T (()))) and Tair contains all the off-diagonal elements potentially nonzero in the regions (b. (b) \ b) for b E B. We denote this class of matrix T as T. We denote a set of blocks E(b) whose memberships are either a strict superset or do not nest with b's membership
E(b) = (b E (b) - ()) V (M(z(*) & ((z A ( & MM(2(1)) }. In the two-modality case, we have F(m () = {(m n)), (n \ m)). The regions {(E(b), b) beB in the alternative graph & reveal identifiability of the latent variables. With these notations, we state the generalized component identification result in Theorem C.9.
Condition C.8 (Generalized Component Identifiability Conditions). Over the domain of (z, E), for any modality m, for T E T and T & P(d(z)), we have
Σ be (()
||T 16) [G] H)
( To
>
Σ
|G
(29)
bE BE E(B)
24

---

Published as a conference paper at ICLR 2025
Notice that the absence of the shared block (m Nn), Condition C.8 recovers Condition 4.3 where E(m) B {m) and H (mm \ (n) = m, and (n \ m) = n.
Theorem C.9 (Generalized
Component-win wise
Identifiabilityy. Let θ
and Ô = ((gimm) Iutor; p( (()(()) ( }=m)) be specifications of the data- generating process in Eq. (1) and Eq. (2) with potentially shared variables m/))) over any modalities m and n. Suppose that they generate identical observational distributions (i.e., p(x) = p(x)) and O satisfies Condition 4.1 and Condition C.8. condition:
(1xxtm); :alam) : ple (m
If 0 satisfies the following
Σ
Σ
| |G
(30)
BE BEB
RER
each component 2i
(m) and its counterpart 27() are equivalent up an invertible map h(-), i.e.
((1) s(m) = h( h(2((T)) under a permutation π over [d(z
This proof closely follows that of Theorem 4.4. We illustrate the key discrepancies as fol- lows.
Proof.
We start with only two modalities (m)) and (n)) for simplicity and then move on to general cases.
The structure of the indeterminacy matrix Taz. ship between Jacobian matrices:
Identical to Equation 20, we have the relation-
G
Te. D7
o2 G
Dx
T
(31)
The presence of the shared block (mmrm) alters the indeterminacy matrix Tax - instead of the disjoint 82
diagonal-block shape, Toz, the columns belonging to the shared variables z (m((n)) (shared between two modalities) are possibly nonzero over rows belonging to (m/(m) That is, the shared variables (mi')) can still mix in the estimates the two individual parts ((m)(n) and g(m))m) However, since we have identified the subspace of z (m)7n) its estimates would not contain information of the individual blocks z (m)) and (x)))) rendering the blocks 0g (g(minn)
Oz A = 0 and D8 (g(m)n)) 0.
(2inkm) km
The sparse connection among modalities. The reasoning in Step in the proof of Theorem 4.4 implies that the structure of the matrix Tes is consistent with that of the matrix Tes. That is, they dx
32 have zero block matrices at the same positions. In particular, since the subspace identifiability in Theorem C.7 implies that the estimated shared variable (mm(n)) and the modality-specific variable (()))))) are not influenced by the other modality-specifie variables (ma)))), the same applies to the estimated exogenous variable ((mrm) and é (n) This structure permits us to disregard Tex (an identity matrix) and Taz on the left-hand side of Eq. (31) when computing sub-matrix of the th
87
right-hand side product:
Ge Tex
Tos G ox (m))mm)))
(32)
(n).mm)n)
We further divide the block [(n), (m \ n)] into two blocks along their rows: [m n), (m n)] and [n \ (m) (m \ rı)] that represent the influence from (m))))) to (mmm) and ((m)m))
Expressing the block [(m nn), (m n)] on the left-hand side of Eq. (32) gives:
[ 0s Toe (m)))
G
Tea 02 (mm))))))
G
Tes d2
(mrn).:
(m)n)
(m)))..(m)))
(33)
Analogously, this block on the right-hand side of Eq. (32) can be expressed as:
T 1ix
Tex
G the
Teg
G
Uz
(m))))
(mrn),;
(.()))))))
(mMn))
(mmn))((mk)))
(34)
25

---

Published as a conference paper at ICLR 2025
Thus, have the equality for the block [(m Nm), (m n)]:
G u2
Tas de (m))))((m)n)
T the
G ue
dz
(m))))
(mar)),(mmm))
(m(n),(())))
G us
Tea 8s (min))
G (mmn))(m)a))
Tex D2 (m)))).(m))))))
(35)
(ml)))
This graphical relation identical that in Eq. (25).
However, the relation for the block [[n \ m), (m ni] between two modality-specific parts varies, due the potential mixing of the shared part into these blocks, which may increase the inbound edges (not outbound edges), as we show below.
For the block [n \ m2) (m n2] on the left-hand side of Eq. (32) gives:
G Tea 02 TR
G
Tes
G
Tge
(m)m))(m))))
(n)m))
(.0)))))
19 (m)m).((m)))
(m))))(()))) (36)
Unlike previous cases, the right-hand side of Eq. (32) for the block involves more than atomic blocks (i.e., it involves the entire modality (n)):
on de 8z
Toa 8x
the G
TTss de
G 8x
(37)
the
(mm))))(mm)))
(x)m):
8z
(m)))).())
(m))()))))
Then, follows from Eq. (36) and Eq. (37) that
[
Tox (m))))
Taz (m)))))(
G (m).(m))))
(m)())
[G ue_ (m))))
Toz de (x)
Goz die
oz
Tos du (m)(n).(m)px)
(38)
(m)))))))
We can observe that the existence the shared variables z (mfn)) divides the latent space into finer blocks ((m)))), z (m)rm) and (m"")) Eq. (35) and Eq. (38) reveal that the bijective indeterminacy relation hold over these finer blocks, exception for the non-square transition matrix T
(m)m))(()
on the right-hand side of Eq. (38). This is because that the shared part 2 (mf(n)) can potentially mix in 2 (m)m) so (n)mm) may receive edges inbound to (mnn))
Interplay among multiple modalities. In light of the graphical condition for the two-modality case (Eq. (35) and Eq. (38)), we can derive the conditions for the mulli-modallee case.
Specifically, we classify the blocks in the estimation graph G os into the following categories for two distinct atomic blocks b and b2.
Region Blocks by and by do not have nested memberships, i.e., M(z(1.) & M(z(12) and M(z(hz)) of M(z((1));
Region Block by has fewer memberships than block by: (bb)) z (ba);
Region 3 Block b, has more memberships than block by: (ba) < z (b1),
Eq. (35) and Eq. (38) reveal that the sparsity for Region I and Region 2 is informative, whereas Region 3 is not. This is because in these the inherent indeterminacy from the subspace identifiability within each modality (Theorem 4.2) will engage the product T az Tee in Eq. (31) in addition to the sparsity in the estimated graph G oz
82 8x
26

---

Published as a conference paper at ICLR 2025
Overall conditions. tion 4.3 as follows.
Consolidating all the considerations above, we re-define objects in Condi-
1. The indeterminacy matrix T Ton + Tor is not strictly block-diagonal: The matrix Ton contains all the on-diagonal square invertible matrices Ton == diag (Tb...... bn and Ten contains all the off-diagonal elements potentially nonzero in the regions (b, (b) \ b) for B. Also, the matrix multiplication becomes
Top
G ((())(6)
Toy as unified expression (M))
(6)
of Eq. (35) and Eq. (38).
2. The sub-matrices on which we impose the sparsity controls are exactly the union of Region 2 and Region 1, i.e., the complement of Region 3. We denote such a region as the function the block index (E(b), b) for each b B.
With these modifications, the rest of the proof follows exactly from that of Theorem 4.4.
D EXPERIMENTAL DETAILS
D.1 NUMERICAL DATASET
We use six numerical datasets in this paper, including three multimodal datasets that satisfy our assumptions and three that slightly violate the sparsity assumptions in the proposed theorems.
Multi-modality settings We generate = 10000 samples according to Eq. (1) and Eq. (2). Fol- lowing prior work (Von Kügelgen et al., 2021; Yao al., 2021; Zimmermann al., 2021), we generate observations using a multi-layer perceptron (MLP). Specifically, the mixing function modeled as three-layer MLP with randomly initialized weights and leaky ReLU activations, enabling y to represent general nonparametric mixing function. The causal noise terms are inde- pendently and identically distributed (i.i.d.), and the exogenous variables are mutually independent. Sparse inter-modality causal dependencies are randomly generated, ensuring that each modality's s latent variables maintain at least one causal connection with another modality.
Ablation settings For the ablation study, we generate modality observations under different sparsity ratios. Each observation is generated from three causally related latent variables and one exogenous variable. The sample size for each dataset is set to n 10000, and the dimensionality of the observations in each modality is d(x) 20. The sparsity ratio determines the extent of inter- modality connections among these latent variables. higher sparsity ratio leads to a sparser causal structure, meaning fewer causal connections between latent variables. Conversely, lower sparsity ratio yields denser causal matrix with more causal dependencies. For example, sparsity ratio of 0% indicates that all inter-modality latent variables are fully connected, whereas sparsity ratio of 50% implies that half of the possible causal edges are removed.
D.2 SYNTHETIC DATASET
Variant MNIST In real-world scenarios, the ground-truth latent processes are often unknown, making it challenging to evaluate model performance. To address this, we construct a synthetic dataset based on the real image dataset MNIST (LeCun, 1998) with known causal relationships, which supports the setting considered in our work. Our synthetic dataset consists of two modalities, each with latent variables that exhibit causal relationships. The design is flexible. The modalities could correspond to different MNIST variants, such colored MNIST (Arjovsky et al., 2019) or fashion MNIST (Xiao al., 2017). The causally related latent variables could be, for example, digit identity, image color, clothing category, image rotation, etc.
In order to make the synthetic setting more intuitive, we introduce an alternative setting: object position acts latent variable that influences the appearance of MNIST images. Across differ- ent modalities, such causal influence may vary. Furthermore, position in modality may causally influence position in modality 2, which aligns with the data generation process in work. For
27

---

Published as a conference paper at ICLR 2025
example, the horizontal position of a digit such the six directly influences the transparency of the MNIST image. This horizontal position then serves as a causal factor for the vertical position of shoes in the fashion MNIST. which in turn affects the grayscale intensity of the shoe image. To systematically evaluate the performance of our algorithm under different observational conditions, consider three variations in colored MNIST, where the digits are assigned one of three colors: red, green, or blue. These relationships are visually illustrated in Figure 7 (a) for clarity.
D.3 REAL-WORLD DATASET
In this paper, we consider three types of datasets, including image, time series, and tabular data. Visualizations of the image and time-series datasets are shown in Figure 7 (b-c).
6
6
6
6
b
6
6
6
6
6
b
6 Lo
6
6
6
G
EEATTRATT DTCT Shea 0110
Right eye
Horizontal Position
Vertical Position
Image Transperency
Grayscale Intersityy
SYORE WT dreep 100 HC
Colored MNIST
Fashion MNIST
Left eye
0:0
25500
50000
Synthetic dataset
(b) Fundus images
(c) Recordings the selected sleep channels
Visualization the datasets: (a) Synthetic dataset: Variant MNIST. (b) Real-world dataset: Fundus imaging shows the interior surface of the eyes. (c) Real-world dataset: Sleep moni- toring shows the time-series recording of sleep-related metries overnight.
Figure 7:
Fundus imaging is the visualization of the interior surface of the fundus, which includes structures such as the optic disc, retina, and retinal microvasculature. High-resolution images of the back of the eye essential for the diagnosis and monitoring of a variety of eye diseases and conditions.
For example, the retinal microvasculature, which consists of small blood vessels that supply blood to the retina, provides valuable information about eye health. Moreover, fundus imaging improve understanding of the underlying mechanisms of various eye diseases. It serves as a non-invasive tool assess the overall health of the microvascular circulation health and provides direct view of part of the central nervous system.
Sleep monitoring a time-series dataset collected over three consecutive nights that records various metrics including sleep stage, body position, respiratory events, heart rate, oxygen saturation, and snoring. This dataset focuses on obstructive sleep apnea (OSA), sleep disorder in which a person's breathing is interrupted during sleep due to the relaxation of throat muscles, causing upper airway obstruction. These interruptions often lead to loud snoring, reduced blood oxygen levels, stress responses, awakenings, and fragmented sleep.
This dataset is collected from a Home Sleep Apnea Test (HSAT), a non-invasive diagnostic method for sleep apnea. Patients wear portable device overnight to monitor their breathing patterns, heart rate, oxygen levels, snoring, and other sleep patterns. The dataset includes multiple channels, such as ACTIGRAPH for movement, HEARTRATE: DIST for heart rate, SPO2_WRIST for blood oxygen saturation, and SBORE_WY for snoring, capturing key aspects of physical activity and sleep patterns during the HSAT. The device calculates apnea-related indices, including the Apnea/Hypopnea Index (AHI), Respiratory Disturbance Index (RDI), and Oxygen Desaturation Index (ODI), as well as indices for diagnosing conditions such as atrial fibrillation.
D.4 EVALUATION METRICS
MCC: Mean Correlation Coefficient MCC is a standard metric used to evaluate the recovery of latent factors in causal representation learning. It measures the alignment between the ground-truth
28

---

Published as a conference paper at ICLR 2025
factors and the estimated latent variables. Specifically, MCC first computes the absolute values of the correlation coefficients between each ground-truth factor and each estimated latent variable. To account for possible permutations of the latent variables, the metric solves a linear sum assignment problem on the computed correlation matrix in polynomial time, ensuring optimal matching between the factors and their corresponding latent representations.
R2: Coefficient of Determination R2 is standard metric used to evaluate the goodness of fit in regression models. measures the proportion of variance in the dependent variable that is explained by the independent variables in the model. Specifically, R2 compares the residual sum of squares of the model with the total sum of squares and returns a value between and 1. A higher R2 indicates that the model explains larger portion of the variance in the data, with 1 representing perfect fit and O indicating that the model explains none of the variability.
SHD: Structural Hamming Distance SHD is a widely used metric for evaluating the accu- racy of graph structure recovery in causal discovery. It quantifies the difference between the true causal graph and the estimated graph. Specifically, SHD counts the number of edge modifica- tions additions, deletions, reversals required to transform the estimated graph into the ground- truth graph. This metric provides a simple yet effective measure of structural similarity, with a lower SHD indicating a closer alignment between the estimated and true causal structures.
D.5 DETAILED DISCUSSION ON HUMAN PHENOTYPE
Without learning such latent variables, cannot provide causal explanation between different modalities. The estimated model shows all causal influences involved, suggests the existence of hidden causal variables, and illustrates their relationships with each other and with observable data. Asymptotically, the learned adjacency matrix A corresponds to a graph within the Markov equiva- lence class given by the PC algorithm.
To interpret the learned hidden variables, we primarily refer to the existing medical literature, which supports their alignment with background knowledge, thereby adding validity to our results. For example, the latent variable FRight3 relates handgrip strength to fundus imaging, consistent with findings showing that handgrip strength correlates with intraocular pressure (IOP) (Pérez-Castill, et al., 2021), 1). In addition, the association between the cataract and changes in IOP (Slabaugh et al., 2013) is consistent with the findings of the model. These connections underline the physiological relevance of the learned hidden variable. Similarly, FRight1 and Fl.eft1, associated with fundus imaging and age estimation, are consistent with studics demonstrating age-related changes in fun- dus image color content (Ege et al., 2002). Another latent variable Sleepl associated with oxygen saturation and sleep metrics aligns with findings that oxygen saturation a strong predictor of obstructive sleep apnea (OSA) severity (Wali et al., 2020). This indicates that the model's latent variable effectively captures critical factors related to sleep disorders.
E EXTENDED EXPERIMENT
To further assess the robustness, scalability, and applicability of our proposed method, we conducted a series of extended experiments under more complex scenarios. These experiments aim to evaluate the performance under diverse latent variable configurations, varying sample sizes, and different structural assumptions, including non-DAG settings and shared latent variables.
Performance in complex scenarios. To evaluate the scalability and generalizability of our method to complex causal structures, we conducted additional experiments on nigher-dimensions simu- lated tasks with diverse configurations of latent variables and modalities. These setups introduce significantly more complex causal relationships between variables. Specifically, we consider three extended scenarios: (1) Five-mods, with 30-dimensional observations from five modalities with two latent variables and one exogenous variable per modality. (2) Six-mods, with 30-dimensional obser- vations from six modalities with two latent variables and one exogenous variable per modality. (3) Eight-mods, involving 30-dimensional observations from eight modalities with two latent variables and one exogenous variable per modality. The results, summarized in Table 4, show that our method consistently delivers robust performance under these challenging conditions.
29

---

Published as a conference paper at ICLR 2025
Impact of the number of latent variables. In real-world applications, the true number of latent variables is typically unknown, and arbitrarily predefining this number may introduce bias and de- grade model performance. In this section, discuss how our method can eliminate the redundant effect of the latent variables, and introduce a cros-validdtionn-aase method to determine the ap- propriate number of latent nodes. By manually setting range for the number of latent variables and selecting the one with the lowest validation loss, we ensure a principled approach that is both simple and widely applicable (Khemakhem et al., 2020b). Here we conduct synthetic experiments to validate its effectiveness. We followed the data generation process in Section D.I, where the ground-truth number of latent variables is two per modality. The results, as shown in Figure 8(a), demonstrate that our approach accurately recovers the correct number of latent variables.
Impact of sample size. To investigate the impact of sample size model performance, we con- ducted an additional experiment evaluating the MCC as the number of data samples increased. Fol- lowing the data generation process described in Section D.I, where the ground-truth number of latent variables is two for two modalities. We systematically increased the sample size from 10,000 to 40,000 and measured MCC and R2 accordingly. The results, presented in Figure 8(b), show consistent improvement in MCC as the sample size increases. This finding confirms the hypothesis that greater data availability enhances the model's 's ability to recover the underlying causal structure.
Latort Dimescions
10
Ldm1)
Lsm
Validation LO35
Mottic Volue
Ldm.5
0.5
LSRLE
Li
Trend Trend
NCC
10:00
20000
50000
60000
Epochs
Number of Samples
(a)
(b)
Figure 8: (a) Comparison of loss across different latent dimensions. (b) The effect of sample size.
Evaluation under non-DAG assumptions. The theoretical results in this paper do not strictly require the assumption of Directed Acyclic Graphs (DAGs) for latent variable structures within across modalities. To evaluate method under non-DAG settings, conducted synthetic experiments where cycles were introduced within and across modalities. Specifically, we followed the data generation process in Section D.I. and considered: (1) cyclic influence within modality; and (2) cyclic influence across modalities. Empirical results in Table 4 demonstrate that the presence of cycles does not hinder the identification of latent variables.
Discussion the shared latent variables. We present how to preprocess the current framework to accommodate shared variables across modalities and provide empirical results. The extended framework incorporates additional mechanism estimate the shared latent variable. Inspired by previous works (Yao et al., 2023; Daunhawer et al., 2023; Von Kügelgen et al., 2021). we incor- porate an additional contrastive loss to enforce similarity in the shared latent representations. To evaluate the effectiveness of this extension, we modify the data generation process in Section D.1 and allow for the existence of a shared variable across modalities. The results, summarized in Ta- ble 4, show that our method accurately recovers both shared and modality-specifi latent variables across different scenarios, confirming the theoretical guarantees of the extended framework.
F IMPLEMENTATION DETAILS
In this section, we provide details of the network architecture, including the optimization scheme and hyperparameter setting.
30

---

Published as a conference paper at ICLR 2025
Complex Scenarios
Non-DAG Settings
Shared Latent Variables
Metric
Five mods
Six mods
Eight mods
Cyclic within
Cyclic across
Two mods
Three mods
R2 MCC
0.89
0.97
0.83+ 1e-3
0.95# 1e-5
0.94
0.86 fe4
0.90+
0.84 3e-4
0.82 4e-4
0.91 Se-4
0.89 2e-4
0.92 1e-5
0.83 7
0.83 4e-6
Table 4: Extended experiment results across different experimental settings.
F.1 NETWORK ARCHITECTURE
We summarize network architecture below and describe it in detail in Table 5.
(1,2) Encoder and Decoder: The encoder transforms observations into latent representations, while the decoder reconstructs the inputs from the latent variables. The encoder-decoder design varies depending
the downstream task. For synthetic data, MLPs with leaky ReLU activation were used. For image data, CNN was used as the encoder, and Cony'T ConyTranspose2D as the decoder. LSTMs were used for time series data. Based on the universal approximation theorem, the model is theoretically able to approximating the underlying mixing function.
(3) Learnable Adjacency Matrix: The causal relationships are embedded in the learned adja- cency matrix, where the binary elements indicate whether specific pairs of vertices contribute to the generation of components, It initializes learnable matrix that captures these dependen- cies. During the forward pass, the matrix is processed to ensure a directional structure where only certain connections allowed based on threshold. This allows the model to learn sparse, meaningful relationships between the latent variables.
(4) Flow-based Transformation: The fow-basee transformation is implemented using an MLP to process the latent variable and a flow model for the transformation. The MI.P first extracts features from the latent variable, which are then used as input to the flow model, which applies an invertible transformation to the latent space, allowing the model to estimate the noise distribution.
F.2 TRAINING DETAILS
Optimization Scheme. The estimation framework was trained using the Adam optimizer on GPU, and the StepL. scheduler was used to reduce the learning rate periodically. The training process ran for maximum of 10000 epochs, with early stopping applied if the validation loss does not improve for 20 consecutive epochs. Random seeds were used to ensure reproducibility, and results were averaged across experiments, with variance reported.
The training loss combines multiple components.
Reconstruction loss: Mean squared error between reconstructed inputs and original data, KL divergence loss: Encourages estimated variables follow standard normal prior. Sparsity loss: An LI- Lo-morm penalty is applied to the adjacency matrix to enforce sparsity.
Hyperparameter. The hyperparameters = [[ilnd, aSSp d'ocon]] represent the weights assigned to each term in the composite objective function. For each dataset, they were tuned within appropriate logarithmic intervals, ensuring a balance between independence, sparsity, and reconstruction. For the experiments, the following settings were applied: = [le 1, 1e 22 1] for the synthetic dataset, le 2, 1e 3, 3. for the MNIST dataset, and a [le 1, 1e 2, 1] for the phenotype dataset.
G ALGORITHM PSEUDOCODE
The pseudocode for the proposed algorithm presented in Algorithm
31

---

Published as a conference paper at ICLR 2025
Configuration
Description
Output
1.1 MLP-Encoder
Encoder for numerical data
Input
Multi- modality observations
BS x d
Dense
h_dim neurons, Leaky ReI. h dim neurons, LeakyReLU
BS x h_dim
Dense
BS x h dim
Dense
Latent embeddings
BS x dim
2.1 MLP-Decoder
Decoder for numerical data
Input
Latent embeddings
BS x 1_dim
Dense
h_dim neurons, LeakyReLU h dim neurons, LeakyReLU Reconstructed observations
BS x h.dim
Dense
BS x dimi
Dense
BS x d_x
1.2 Image-Encoder
Encoder for image data
Input
Image input
BS x 3 x H x W
ResNer18
ResNet backbone, LeakyReLU
BS x h.dim
Dense
Latent embeddings
BS x Idim
2.2 Image-Decoder
Decoder for image data
Input
Latent embeddings
BS Ldim
Dense
h.dim neurons
BS x h.dim x H' x W
Conv TanTrrasppoeDD
Reconstructed observations
BS x 3 x H W
1.3 Time-series Encoder
Encoder for time-series data
Input
Multi-channel time-series data
BS x seq_len x n_channel
LSTM
Sequences into hidden representations
BS x h_dim
Output
Latent representation
BS x l.dim
2.3 Time-series Decoder
Decoder for time-series data
Input
Latent representation
BS dim
L.STM
Sequence into output features
BS x seqleen h_dim
Output
Reconstructed time-series data
BS x seq len x channel
3. Adjacency Matrix
Sparsity regularization
Input
Latent variables from encoders
BS x z.all
Masking
Lower triangular mask
7 all x all
Thresholding
Retain entries exceeding threshold
z_all z.all
Output
Learned causal adjacency matrix
z all x z all
4. Flow Transformation
Estimate the noise term
Input
Latent variables across modalities
BS x z z.all
Condition Input
Apply adjacency matrix to latent
BS x z_all x z_all
Flow Transformation
Apply transformation latent Estimated noise variables
BS x z.all
Oulput
BS x z.all
Table 5: Architecture details. BS: batch size, d_x: input dimension, I.dim: latent dimension in each modality, z alll:: latent dimensions across all modalities, h dim: hidden dimension, H/W: height/width of the input image, len: sequence length, n channel: number of channels.
32

---

Published as a conference paper at ICLR 2025
Algorithm Pseudocode for the proposed algorithm.
1: Input: Grouped observations {x((")) Jm
2: Output: Estimated latent variables (2(m) M
m==1
3:
4: # Random Initialization
5: Initialize encoders (En (m)) LM
Jm=1 and decoders (De(")) Jm=1 for each group
6:
7: # Encoder
8: Input: Grouped observations {x(m))
Jm
9: Output: Estimated latent variables (m)) for each group m
10: for each group m = to M do
11: Encode the current group latent and exogenous variables: 2(m), ./(mm) = En En!m (x(((((((())
end for
13: Concatenate latent representations: {
Im=1 = (1)) + 2 + + (
14: return Estimated latent variables and exogenous variables { 1/(mm) jm
16: # Flow-based Noise Estimation
Input: Estimated latent variables for each group (2(m) /mm
.((x) 18: Output: Estimated noise term (==1
19: Initialize adjacency matrix Å
20: Select the parents of latent variable based on the adjacency matrix
21: Pass through flow model to obtain estimated residuals és
22: Update the estimated causal graph based on the adjacency matrix with threshold Compute sparsity loss based on L1 norm
24: Compute the KI. divergence between [())(m) J ad(x) 25: return Estimated noise term 6;=1
ad(2) and Gaussian prior
4=1
26:
# Decoder
28: Input: Estimated latent and exogenous variables in each group (m(m), s/(m) ) M
29: Output: Reconstructed grouped features {x(m)) J m=
for each group m .M do
31: Decode (m)), (m)) to reconstruct features (mm) (m)) (m)) (2()) sf(m))
32: Compute reconstruction loss using MSE:
Recon MSE(X(m) (
(m)
33: end for
33


Reviews:
[{"review_id": "dWgzw97MEX", "reviewer": "Reviewer_5aPk", "summary": "This paper studies the causal graph discovery problem from multiview data. Given observed multimodal data, the goal is to estimate the causal relationship of latent features between modalities. The authors study the data generation process where the observed multimodal data are independent given the latent variables. Under this assumption, the authors employ the encoder-decoder framework to decompose the latent factors and nuisance factors, and a sparsity regularization function to impose the sparse relationship between modalities in the latent spaces. They compare the proposed method with several baselines on simulated tasks and the results show improved mean correlation coefficient. Furthermore, the proposed method is applied to human phenotype dataset.", "questions": "a. the paper assumes that the latent variables within the same modality form a DAG, could the author clarify why this assumption is necessary? Secondly, if this is needed, the loss function (6) does not enforce the latent variables within the same modality to be a DAG. How can we ensure such structures will be satisfied in the learning phase?\n\nb. does the causal relationship between different modalities need to be DAG? If there are cycles, are we still being able to identify the latent variables?\n\nc. I find the definition of condition 4.3 confusing. What are the connections between matrices A, G, T? and A seems to only denote the relationship between modality 1 and 2?\n\nd. In Assumption A.1, what is smooth inverse?\n\ne. Could the authors provide a guideline for practitioners on how to systematically choose the number of latent nodes?\n\nf. In Theorem 4.4, does it require the regularization term to be scaled by some coefficient?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper tackles an important and yet not well-addressed problem. It is well-motivated by biomedical applications. The proposed framework is more general compared to the prior work (as shown in Table 1).", "weaknesses": "In general, I find several parts that need further clarification. (see the question parts)\n\nMy main concern is that the simulated tasks focus on low-dimensional data with simple sparse causal structures. It is not clear whether the method is scalable and can be generalized to more complex causal structures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WI5YrkrcKt", "reviewer": "Reviewer_JxDh", "summary": "This paper provides a method that identifies the latent variables up to invertible component-wise transformations from multi-view data under the weak assumption that these latent variables exert partial influence on every modality and strong influence on one modality. The proposed approach relies on weaker assumptions compared to the existing works and yet provides strong results. The primary motivation for this approach is on multimodal health care data but the experiments are conducted on both synthetic and image datasets in addition to health care data.", "questions": "1. It is assumed that the information of $z^{(m)}$ is preserved in its corresponding observation $x^{(m)}$ and it exerts sufficient influence on other modalities' observations $x^{(-m)}$. Now, consider the observation from another modality $x^{(k)}$ that has a similar behavior. How does $z^{(k)}$ not interfere in the identifiability of the subspace of $z^{(m)}$?\n2. Is the considered setting a non-linear version of [10]? Can the authors add text comparing/contrasting the proposed approach with [10]?\n3. Does \"fully share\" mean \"retrieve without errors\" in line 212? That's what I understood from the example that follows this statement in line 215.\n4. Can the authors explain how the method is different from [9] apart from the latent variables being independent in [9]?Specifically, what does \"part of the variables are directly observed and causal directions are given by default\" mean in lines 335-338?\n5. Can the authors add an intuitive description for Condition 4.3 as its implications are not clear from the main text? It is enough that this intuitive explanation is included in the appendix.\n\n\nTypos and minor writing mistakes:\n\n1. In line 78, what is $n$? Is it $z_i^{(m)}\\rightarrow z_j^{(n)}$?\n2. Is the co-domain of $G(z, \\epsilon)$ correct in line 303?\n3. Some issues with the references. E.g., \"Non-parametric Identifiability of causal representations from unknown interventions\" appeared in NeurIPS 2023. Sturma et al., (2023) cited twice in lines 126-127.\n\n\n[1] Sturma et al., 2023, \"Unpaired Multi-domain Causal Representation Learning\", NeurIPS 2023\n\n[2] Zheng et al., 2022, \"On the Identifiability of Nonlinear ICA: Sparsity and Beyond\", NeurIPS 2022", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.   The contribution is very clear and useful for future works on identifiability. More strengths included in the summary.\n2.   The writing is lucid. The examples are cleverly used to contrast the proposed method against the existing works.", "weaknesses": "I did not find any significant weaknesses. However, I do have a few questions for the sake of clarity. Questions in the following section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tPXIEqsQt8", "reviewer": "Reviewer_uhwZ", "summary": "Authors develop new methods to identify patterns in complex multi-modal biological datasets. It improves upon previous work in two key ways: First, it uses a flexible mathematical framework that doesn't make rigid assumptions about how the underlying biological factors (latent factors) are distributed. Second, these factors can influence each other across different modalities. Simulation study and real-world data support authors's claim.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and motivated. \n2. Authors provide identifiability guarantees for each latent component. This is helpful to characterize the interactions among all latent\ncomponents across modalities for the biological applications. \n3. The assumptions of theoretical results look reasonable to me. \n4. Real-world dataset analysis is provided, demonstrating its usefulness.", "weaknesses": "1. A1 indicated the neural network needs to be invertible. How do authors achieve this? Authors use normalizing flow, an invertible generative model, as a part of their network, what about others? Also what's the computation efficiency? \n2. In real world, user-defined number of latent variables can be biased. Have authors analyzed it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ciCxjx0Lzw", "reviewer": "Reviewer_mBq3", "summary": "This paper develops a theory for identifying the underlying structure of multi-modal data involving latent causal variables, based on the assumption of smooth invertible generating functions. It further investigates the sparsity of this causal structure and proposes an estimation method. Experiments conducted on both synthetic and real human phenotype data demonstrate the effectiveness of the proposed approach.", "questions": "1. In my view, the example provided in Section 3 does not align well with the proposed model. What is your rationale for considering heart sizes and bone structures as latent variables?\n2. In Appendix D.1, could you provide further details on the MLP architecture used for the numerical data generation process? Have you considered implementing an invertible neural network structure in your simulations? If you increase the number of data samples, do you expect the MCC to approach 1? I would appreciate seeing results on this, particularly illustrated with four variables (two latent and two observed).\n3. The arrow from $z$ to the adjacency matrix in Figure 4 is unclear. Could you clarify this relationship?\n4. How did you compute the KL divergence? Do you assume that $p(\\gamma)$ follows a Gaussian distribution, with the encoder output providing the mean and variance?\n5. When calculating the SHD, given that the indices of latent variables can be permuted, could this pose an issue?\n6. In the context of the MNIST dataset, what causal relationship exists between the colored MNIST and fashion MNIST data?\n7. How does the learned adjacency matrix A relate to the one derived from the PC algorithm?\n\nComment:\nFor the real-world application, I encourage the authors to explicitly present evidence from the literature demonstrating the success of their algorithm.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well-written.\n- It includes an adequate literature review.", "weaknesses": "- On real impact. As mentioned by the authors, knowing the number of underlying latent variables is unrealistic. \n\n- Missing implementation details, e.g., more details on data generation, network structure, optimization scheme, hyper-parameters, etc. see questions as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IkIejbVwvT", "reviewer": "Reviewer_91p1", "summary": "The paper addresses the identification of hidden causes underlying observed multimodal data. The authors propose a model that avoids previous limitations, specifically not requiring shared latent information across multiple modalities or assuming an exponential family distribution for hidden causes. The paper provides a theoretical identifiability analysis and presents experiments, including one on biological data.", "questions": "### Theory\n\n1. *Identification vs. Estimation*: In Theorems 4.2 and 4.4, identification and estimation are confused by statements like “We estimate the generating process…”. Identification is a model property and should be established before estimation is addressed. For a model $\\mathcal{P}:=\\\\{p_{\\phi}:\\phi \\in \\Phi\\\\}$, we simply show that $\\forall \\phi_1, \\phi_2 \\in \\Phi (p_{\\phi_1}=p_{\\phi_2} \\implies \\phi_1=\\phi_2)$ to demonstrate identification.\n\n2. *\"Identification under Regularization\"*: Particularly, Theorem 4.4 is problematic in stating that identification is achieved through regularization, which pertains to estimation. If regularization is essential, the model should be reformulated—like for ridge regression, the regularization can be treated as a sparse prior in a probabilistic model.\n\n3. *Proof of Theorem 4.2*: The step to equation (14) seems to rely on $x=\\hat{x}$. However, equal distributions do not imply equal values of random variables. If (14) is correct, the derivation should clarify how equal distribution leads to (14).\n\n4. *Definition of $\\tilde{g}_x$*: Strictly speaking, there is one $\\tilde{g}_x$ for each $m$, and it’s unclear if these $m$ functions can be shown as the same function. If not, we are in fact making assumptions about $m$ functions.\n\n5. *Proof of Theorem 4.4*: 1) Condition 4.3 mentions *undefined matrix $A$*. In the proof, equation (28) involves more than a single matrix,  as denoted by $A$ in Condition 4.3, and the definition of $d^*(U)$ does not seem to align with $d^*$ in Condition 4.3 (this questionable reasoning later is extended to $D$). This proof section is unclear both in notation and logic flow; we can not even see $A$ as a placeholder for the various matrices used in equation (28)! I suggest sorting out the involved matrixes (I believe, if the proof is not simply wrong, there should be several similar conditions for different matrixes), and a clear connection between the proof and assumptions should be seen. 2) *the regularization again*: the reasoning following equation (29) requires satisfaction of (29), but regularization (5) can only encourage, not guarantee this condition. This suggests the idea of “identification under regularization” itself does not work, not only an incorrect formulation as mentioned in the 2nd point.\n\n### Implications\nOne core aspect of the causal model is that two modalities cannot share any direct causes, which is questionable. While $z^{(m)}$ can, and indeed is required to, affect $x^{(-m)}$ indirectly through $z^{(-m)}$, this seems to require hidden causes $z$ to be lower-level features. For example, “age” is a high-level feature affecting many modalities, like “sleep” and “eyes.” Thus, if the \"sleep\" modality includes \"age\" in $z^{sleep}$, then, to satisfy the \"indirect\" causes requirement, lower-level features $z^{eyes}$ would be needed to mediate the causal effects from “age” to “eyes\", i.e., $d(z^{eyes})$ should be large enough to block all the causal effects from “age” to “eyes.” And, when we have many modalities and many hidden causes, this eventually requires most hidden causes to be low-level.\n\nWhile the theoretical relaxations and real-world examples are appreciated, it’s unclear why, in the “sleep patterns” and “brain imaging” examples, your assumptions could be satisfied, though I can see previous ones are limited.\n\n\n### Biological Application\nThe demonstration of the biological application lacks depth, and the claimed “strong interpretability” is overstated. For example, in the human phenotype dataset, how should we interpret the learned hidden variables? Does the method enhance our understanding of the data or its underlying mechanisms? How does the number of latent variables affect the results (possibly with very different causal graphs)?\n\nAdditional specific questions are: Why are causal relationships involving FRight and FLeft not symmetric? For example, why does FRight affect Cataract but not FLeft?\nWhy are there no Sleep variables that influence “Sleep efficiency”?\nShouldn’t ocular quality impact hand grip strength rather than the reverse?\n\nMore experiments should be conducted, both on the model side (e.g., testing various numbers of latent variables and other hyperparameters) and the data side (e.g., trying many different modalities). \n\n\nOverall, the authors could clean up theoretical developments, clarify the practical implications, and leave the biological application to future work.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper presents meaningful relaxations of assumptions from prior work.\n\nIt is relatively well-written, considering the complexity of the theory and notation involved.", "weaknesses": "There are issues with the main theoretical claims and their proofs.\n\nThe practicality of the current assumptions and their real-world applicability remains unclear.\n\nThe experiment using the human phenotype data does not justify the title or the highlights in the abstract.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)