Beyond DAGs: A Latent Partial Causal Model for Multimodal Learning
Authors: Yuhang Liu
Zhen Zhang
Dong Gong
Erdun Gao
Biwei Huang
Mingming Gong
Anton van den Hengel
Kun Zhang
Javen Qinfeng Shi
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Representation Learning
Latent Variable Model
Disentangled Representation Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Directed Acyclic Graphs (DAGs) are a standard tool in causal modeling, but their suitability for capturing the complexity of large-scale multimodal data is questionable. In practice, real-world multimodal datasets are often collected from heterogeneous generative processes that do not conform to a single DAG. Instead, they may involve multiple, and even opposing, DAG structures with inverse causal directions. To address this gap, in this work, we first propose a novel latent partial causal model tailored for multimodal data representation learning, featuring two latent coupled variables parts connected by an undirected edge, to represent the transfer of knowledge across modalities. Under specific statistical assumptions, we establish an identifiability result, demonstrating that representations learned by MultiModal Contrastive Learning (MMCL) correspond to the latent coupled variables up to a trivial transformation. This result deepens our understanding of the why MMCL works, highlights its potential for representation disentanglement, and expands the utility of pre-trained models like CLIP. Synthetic experiments confirm the robustness of our findings, even when the assumptions are partially violated. Most importantly, experiments on a pre-trained CLIP model embodies disentangled representations, enabling few-shot learning and improving domain generalization across diverse real-world datasets. Together, these contributions push the boundaries of MMCL, both in theory and in practical applications.

Published as a conference paper at ICLR 2026
BEYOND DAGS: A LATENT PARTIAL CAUSAL MODEL FOR MULTIMODAL LEARNING
Yuhang Liu Zhen Zhang Dong Gong', Erdun Gao Biwei Huang',
Mingming Gong', Anton van den Hengel 122 Kun Zhang", Javen Qinfeng Shi "Responsible AI Research Centre, Australia
2 Australian Institute for Machine Learning, Adelaide University
Sccool of Computer Science and Engineering, The University of New South Wales *Halıciollu Data Science Institute, University of California San Diego
School of Mathematics and Statistics, The University of Melbourne
'Department of Philosophy, Carnegie Mellon University yuhang. liu0l@adelaicn edu.au
Project: https: htttp://sitts. http://sssssss.lll..... wwt.://wwww.yghhhaggggg////////a
ABSTRACT
Directed Acyclic Graphs (DAGs) are standard tool in causal modeling. but their suitability for capturing the complexity of large-scale multimodal data is questionable. In practice, real-world multimodal datasets are often collected from heterogeneous generative processes that do not conform to a single DAG. Instead, they may involve multiple, and even opposing. DAG structures with inverse causal directions. To address this gap, in this work, we first propose a novel latent partial causal model tailored for multimodal data representation learning, featuring two latent coupled variables parts connected by an undirected edge, to represent the transfer of knowledge across modalities. Under specific statistical assumptions, we establish an identifiability result, demonstrating that representations learned by MultiModal Contrastive Learning (MMCL) correspond to the latent coupled variables up to trivial transformation. This result deepens our understanding of the why MMCL works, highlights its potential for representation disentanglement, and expands the utility of pre-trained models like CLIP. Synthetic experiments confirm the robustness of our findings, even when the assumptions are partially violated. Most importantly, experiments disentangled representations, enabling few-shot learning and improving domain generalization across diverse real-world datasets. Together, these contributions push the boundaries of MMCL, both in theory and in practical applications.
pre-trained CLIP model embodies
INTRODUCTION
Recent advances in multimodal learning have demonstrated remarkable capabilities across vision, language, and beyond (Liang et al., 2024; Lymperaiou & Stamou, 2024; Li et al., 2024). Repre- sentative models, such as CLIP, achieve this by aligning different modalities through MultiModal Contrastive Learning (MMCL) (Radford et al., 2021). A crucial factor behind their success is that these models are trained on large-scale multimodal datasets, enabling them learn rich, high-quality cross-modal representations. Despite its remarkable empirical success, understanding the underlying mechanisms of multimodal learning is essential, not only to explain its current achievements but also to identify opportunities for further improvements (Liang et al., 2024). Recent works have also analyzed multimodal learning through the lens of latent causal models (Daunhawer et al., 2023; Yao et al., 2024; Gresele et al., 2020). These approaches examine the relationship between representa- tions learned by multimodal learning from observed data and the high-level latent causal variables underlying such data, a line of inquiry referred to identifiability analysis. By demonstrating that learned representations can, in principle, recover these latent causal variables, such analyses provide causality-
-uusality-rouuneed grounded explanation for the success of multimodal models. Crucially, most of these

---

Published as a conference paper at ICLR 2026
latent causal models rely the assumption that the latent causal variables follow Directed Acyclic Graph (DAG) structure. See Appendix A for more related work.
We argue that such a DAG assumption may be inappropriate for capturing the underlying generative processes of large-scale multimodal data, which underpin state-of- th-ttheartt multimodal models. This argument is supported by the following observation that large-scale multimodal data often arise from heterogeneous causal mechanisms that correspond to different, and sometimes even conflicting, DAG structures (Schölkopf al., 2012). For instance, in the context of text-image paired data, some pairs are generated through a text-to-image causal mechanism, where a textual instruction serves as the input from which the corresponding image is produced (Ramesh al., 2021). In contrast, some pairs arise from an image-to-texx pipeline, where images are first collected from the internet and subsequently annotated with descriptive text by experts (Sharma al., 2018). These two distinct causal mechanisms illustrate that large-scale multimodal data may arise from fundamentally opposite causal directions. Consequently, the common DAG assumption may be overly restrictive, failing to capture the diverse and sometimes conflicting generative processes underlying such data (see Sec. 2 for a detailed discussion). As a result, although prior works on identifiability analysis under DAG assumptions (Daunhawer et al., 2023; Yao et al., 2023; Gresele al., 2020) have provided valuable theoretical insights, they are often restricted to specific, small-scale multimodal data, where a DAG structure is sufficient to capture the underlying generative process. As a direct consequence of this modeling choice, these studies largely remain confined to simulation experiments, and offer limited guidance for applying advanced multimodal models trained on large-scale data, e.g., CLiP-like models, to real-world applications. To this end, this paper makes the following contributions:
A Novel Latent Partial Causal Model (Sec. 2). We propose a novel latent partial causal generative model, specifically designed for modeling the multimodal data generation process. Instead of relying on the DAGs assumption, our model introduces latent coupled variables, connected by undirected edges, to effectively capture transferable knowledge across different modalities.
Identifiability Guarantee (Sec. 3 and 4). We developed theoretical analyses specifically tailored to the proposed generative model, under certain statistical assumptions, showing that the representations learned by MMCL are related to the latent coupled variables up to a simple transformation, thereby providing a theoretical explanation for the success of MMCL.
Disentanglement Potential of MMCL (Sec. 5). Our theoretical results reveal the component- wise disentanglement potential of MMCL, which pushes the boundaries of how pre-trained models, such CLiP-like models, can be leveraged. To the best of our knowledge, this is the first work to provide guarantees for the component-ve wise disentanglement potential of MMCL.
Extensive Experimental Results (Sec. 6). We validate our theoretical findings under ideal conditions via simulations and demonstrate their robustness even when the underlying assumptions are partially violated. Extensive experiments on pre-trained CLIP model across various tasks, such as few-shot learning, domain generalization, and disentangled representation learning, on over 16 real-world datasets substantiate the practical effectiveness of our findings.
In summary, our work provides a principled explanation for the success of MMCL and, importantly, highlights its potential for learning disentangled representations. Although our theoretical findings rely on certain assumptions that may not be fully verifiable in practice, similar to most existing works on identifiability analysis, simulations demonstrate the robustness of our results even when these assumptions are partially violated. In addition, extensive experiments with pre-trained CLIP models across diverse real- world tasks provide strong evidence that the theoretical insights can translate into practical benefits. Taken together, these findings relax the conventional reliance on DAG assumptions in advanced MMCL, while maintaining applicability and effectiveness in rerl-world scenarios.
2 GENERATIVE MODEL: THE LATENT PARTIAL CAUSAL MODEL
In this section, we introduce a latent partial causal model that captures the generative mechanisms of multimodal data. Before presenting the model, we outline key observation about such data.
Diversity in generative process of large-scale multimodal data. We argue that real-world large- scale multimodal data often entails multiple, complex generative processes that may not be fully captured by single DAG structure. To illustrate this (see Figure 1), let latent variables z, and z,
2

---

Published as a conference paper at ICLR 2026
Figure I: Possible DAG structures underlying large-scale multimodal data: Left: A latent confounder influences both n and Zg. Middle: z; influences zu through an intermediate mediator b, serving as bottleneck for transferable knowledge. Right: A symmetric inverse relationship where 2₂ influences ze via b. These DAGs illustrate that a single DAG assumption may not hold when modeling large- scale multimodal data with heterogeneous generative processes.
denote shared semantic factors. For example, 2₂ may correspond to high-level visual concepts such object category scene type in an image, while z, may capture the semantic content of sentence, such as topic or intent. To model modality-specifie characteristics, we introduce additional latent variables m, and m,- For instance, m, may represent image-specific factors such background noise or visual artifacts, whereas m, could encode linguistic aspects such as sentence structure or grammatical patterns. Together, (Zg, m.) and (Z,, m,) generate the observed variables x (image) and t (text), respectively.
In the left DAG model of Figure 1, the latent confounder represents a shared source of variation that influences both latent variables 2, and Z,, which correspond to latent semantic factors generating the observed variables (e-g.. image) and t (e.g.. text), respectively. This confounder captures a common underlying context or concept connecting the two modalities. For example, if the image and text are related to the topic "sports," e could encapsulate this shared theme, influencing the generation of both the visual and textual data. The middle DAG depicts a structure where b represents transferable knowledge. Specifically, b serves as the bridge, deriving information from the text latent variable z and informing the image latent space z,. This scenario aligns with the generative process where text serves as a guiding input for image generation, e.g.. text-to-image generation. classical example is the MNIST dataset (LeCun & Cortes, 2005). In contrast, the DAG on the right represents image-guided text generation process, e.g., image captioning. Here, the high-level latent information in the image influences the high-level latent variable in the generated caption. A classical example is the CelebA dataset (Jiang et al., 2021).
Current advanced multimodal models, such as CLIP (Liang et al., 2022), are typically trained on vast collections of multimodal data, which may in fact arise from a mixture of the three scenarios illustrated Figure/ (potentially with additional DAG assumptions not depicted). In this context, restricting the generative modeling of large-scale multimodal data to single DAG structure may be inadequate capture the inherent diversity of worldworld multimodal dependencies.
The Proposed Latent Partial Causal Models. Instead of DAGS structure, we propose latent partial causal model, de- signed to represent the generative process for multimodal data, as illustrated in Figure 2. In it, the latent space is partitioned into two components, each corresponding to specific modal- ity. such as image and text. To capture unique characteristics within each domain, the model incorporates modality-spor specific latent variables, m, and m, In addition, to capture transfer- able knowledge between these modalities, the model introduces an undirected edge between the latent coupled variables, Zz and z,. Further, the observations are generated through dis- tinct processes that link the latent variables to the observed data. Specifically, images (x) are generated by the function gz (m,, 2,), while text (t) is produced by gi (m,, 2.). Besides the justification mentioned in Figure 1, this modeling approach also grounded in the intricate dependencies between modalities. For instance, the adage "a picture is worth thousand words" highlights the richness and detail of visual data, as supported by Gropper (1963); Hum et al. (2011). However, this perspective is not universally applicable, as Reinert (1976) argues that textual information often convey precise meanings. Similarly, Fidler et al. (2013) reinforces the complementary nature text, asserting that "a sentence worth thousand pixels" in its ability succinctly express complex ideas.
m,
7
mi
Figure 2: The proposed latent par- tial causal model. 2₂ and z, are latent coupled variables, and mg. m, are
3

---

Published as a conference paper at ICLR 2026
3 A FIRST LOOK: THE RECOVERY POTENTIAL OF MMCL
Given the proposed generative model, our goal is analyze how MMCL framework, trained with observed data x and t, can recover the true latent variables 2, and 2, up to simple transformation. Before this, we provide an intuitive motivation for why MCL is expected to achieve this. MMCL leverages a loss function designed to maximize similarity between embeddings of real paired data while minimizing similarity for incorrect pairs. The loss function is defined as Zhang et al. (2022b); Radford et al. (2021):
oo (E(O)))))))))) Σn e (L(x)))()))
E log (EC)))E)))))) cl EN (L())))))))
1
(I)
where d denotes a distance metric, e.g., cosine similarity on hypersphere or L.1 norm on convex bodies, is learnable temperature hyper-parameter, N denotes the sample size, which means that we have N positive pairs and N N negative pairs, fa denote the encoder on one modality x, i.e., image, similarly, f, denote the encoder another t, i.e., text. To understand the multimodal contrastive loss further, we investigate its asymptotics:
Theorem 3.1 (Asymptotics of L). For fixed T > O, as the sample size N co, the (normalized) multimodal contrastive loss converges
log E (e=d(((()))))))))))))) tmp(t)
lim L - -og N -2 N
E (x.t)=((x,t)
((())))).
E x-p(x))
log E [e (160,))))))))) x-p(xx)
E typ(t)
The proof is provided in Appendix B. This is generalization of Theorem in Wang & Isola (2020).
Insights into Latent Variable Recovery The loss function in Eq. (2) connects directly to two fundamental principles in latent variable recovery: Prior Matching and Information Preservation. These principles are crucial for methods like nonlinear independent component analysis (ICA) (Hyvärinen et al., 2001), which recover latent independent variables from observed data.
Prior Matching: This constrains the solution space using prior knowledge, addressing the non-uniqueness problem that often arises in latent variable recovery.
Information Preservation: This ensures that the solution space fully captures the complexity of the latent variables derived from the observed data.
Prior Matching The first term in Eq. (2) promotes alignment between representations of real data pairs across modalities, enforcing that one modality (i.e., text) acts as a prior signal for the other (i.e., image). Minimizing this term drives cross-modal alignment and incorporates prior knowledge, which key for recovering latent variables.
Information Preservation The last two terms in Eq. (2) are closely related to ensuring that the learned representations capture the full complexity of the latent variables. These terms can be approximated by optimizing the following expression (proof in Appendix C):
H
- (H(((((())
where H(.-)) denotes cross-entropy. The objective function in Eq. (2) is symmetric between x and Intuitively, ((.. (x)) and pt) are not equal, the solution deviates, increasing the objective value and introducing asymmetry in the last two terms. For the optimal solution, the two distributions must align. When pf, (x)) = (t)) the cross-entropy in Eq. (3) reduces to entropy, and f, and f, transform x and t into uniformly distributed random variables, Eq. (3) reaches its optimal value. This highlights the importance of finding transformations f, and f, that preserve information by fully capturing the latent variable structure.
4

---

Published as a conference paper at ICLR 2026
A Novel Unified Perspective on Contrastive Loss Previous research has primarily focused on contrastive loss in the context of single modality, emphasizing two main perspectives: 1) alignment- uniformity (Wang & Isola, 2020), which is closely related to prior matching, and 2) information preservation (Oord et al., 2018). However, these two perspectives have largely been treated separately. In this work, we offer novel insight by combining these two perspectives within the multimodal context for latent variable recovery. This insight motivates our belief that MMCL holds significant potential for recovering latent variables.
4 FROM POTENTIAL TO PRINCIPLES: IDENTIFIABILITY GUARANTEE
Given the initiative analysis in Section 3, which highlights the potential of MMCL for recovering latent variables, we now move forward to rigorous identifiability analysis, which provide theoretical guarantees that MMCL can indeed recover the true latent variables, by parameterizing the proposed latent partial causal model. We examine two distinct types of parameterization in latent spaces, hyperspheres and convex bodies, under specific assumptions, respectively.
4.1 IDENTIHIABILITY ANALYSIS ON HYPERSPHERE
On hypersphere, we parameterize the proposed latent partial causal generative models as following:
-(2x)=12171 - p(zz(zz) = Cj C (kz, X g. (Z., (z,, t ge m)),
(4)
where denotes the space of latent factors 2 and z, We assume that Z is the unit hypersphere gM-1 aligning with the commonly used normalization in constrastive loss. We do not enforce any further assumptions for my and m.. For ge and gi, we assume them to be invertible, and differentiable, ensuring the information in latent space can be recovered. In addition, we assume that p(2.,) follows a uniform distribution, and p(ze (2,)) follows von Mises-Fisher (vMF) distribution, considering the constraint of unit hypersphere. Given these assumptions, we first establish that the minimization of the cross-entrope Eq. (2) converges to a symmetric cross entropy, as follows:
Theorem 4.1. (L converges to the symmetric eross-entropy) Under the assumptions defined in Eqs. (4) for the proposed latent partial causal model, the necessary condition f, o g. = f, o go, denoted as h. for the optimal normalized multimodal contrastiveloss given by Eq. (2) leads the following reduction of the loss itself:
lim -- 2log N
21og| 2
E z.-p(z,)
E [H(p (z),
Qu (uz.)),.((zzz)))) [z)]]
N No
z=mp(a,)
(5)
where H the cross entropy, the conditional distributions qh (Zr(zz) and q (z)) are parameterized by the following:
(2rl2)))) -2 (b(z.)) (oolk.))((())))).... (n (m Cg(Znn) (h(l)))))
(6)
with
Cg(Zo)= = f
( hh((z))/))) ((b(t,)) dzzz) dz Cg(Zx) = f ( ((((x)) ((())))))
Refer to Appendix D.1 for proof. This is a generalization of Theorem I in Zimmermann et al. (2021).
Bridge Between Modalities By addressing key asymmetries arising from modality differences, such as modality-specific variables m, and m,, along with distinct generative processes ga and ge, we derive the result in Theorem 4.1. This result is pivotal as it establishes critical connection between MMCL and traditional mngal-modal contrastive learning. In particular, Theorem 4.1 enables the transfer of insights and results from single-modal settings to the multimodal context. As a result, we present the following corollary:
Corollary 1. By leveraging Theorem 4.1, the minimization of Eq. (5) identifies the latent variables Ze (and symmetrically, 21) up to a linear transformation. Specifically, the representations f, (x), learned by the minimization of Eq. (5), are linearly related to the underlying latent variables Zy in the proposed latent partial causal model, as follows: f,(x) Azg c, where A is orthogonal matrix and c. is constant vector.
For further details, Appendix D.2.
5

---

Published as a conference paper at ICLR 2026
Success of MMCL Corollary shows that minimizing Eq. (5) (or equivalently, the multimodal
contrastive loss in Eq. (1)) identifies the latent variables Zy (and symmetrically. z,) up to a linear transformation. This means that the representations fx(x), learned through MMCL, related to the latent variables 2; via a linear transformation, i.e., (x( = Azg + c. A similar result holds for 2ς- This finding highlights the effectiveness of MMCL, suggesting that its success in practical applications stems from its ability to recover latent coupled variables. This recovery preserves essential, transferable knowledge across modalities, enabling the learned representations to capture high-level transferable information while discarding model-speeitfe details. Such properties are key to the robustness and transferability of MMCL representations.
directly
4.2 IDENTIHABILITY ANALYSIS ON CONVEX BODIES
We now extend the previous identifiability result to convex bodies, e.g., the hyperrectangle [a1 b1 x [ams b.s]. On convex bodies, we parameterize the proposed generative models by the following:
(zzz) 12 p(z(/z)) (
ga (Z,, m.), t (e(,
where ò a distance metric induced by a norm. We consider a convex body in RM, denoted as Ze. where assume that p(z,,) follows uniform distribution, and the conditional distribution ((zz)z.) follows an exponential distribution. Again, we do not enforce any further assumptions for m, and m. For ga and gr. assume them to be invertible and differentiable mapping, ensuring information in latent space can be recovered. Given these assumptions, we have the following result:
Theorem 4.2. (L converges to the symmetric cross-entier (ross-entropy) Under the assumptions defined Eq. (7) for the proposed latent partial causal model, the necessary condition fy o gg fę go denoted as h. for the optimal normalized multimodal contrastiveloss given by Eq. (2) leads to the following reduction of the loss itself:
lim L - log N 2log Zel No
[H(p(zi)um) (h (Zel)e))) ==-D((.)
E
E ze-p(z))
(H(((Z)))e) ( [z))]
(8)
where H is the cross entropy, the conditional distributions by the following:
cintiiioonaldssrriuuuttnsssss 9h (zilzz) ( and (zz, (2)) parameterized
(2nl)))))) (2)2) ((b(2,)
(((.))ee((((...bba))))) b(()))) ,un(2.lzx) Cg - 6(h(zz)).(z.))
(9)
with
C,(Zi)= e- (5(((,.)) dz., Cg = f -(bt
Bridge Between Modalities In convex bodies, Theorem 4.2, introduced for the first time this work, plays key role in bridging MMCL with traditional contrastive learning by addressing the asymmetric challenges arising from modality differences. Building on this theorem, we have:
Corollary 2. The minimization of Eq. (8) in theorem 4.2 identifies the latent variables Z, (sym- metrically, z,) up permutation transfornation, i.e., the representations f, (x), learned by the minimization of Eq. (8), is related to the underlying by in the proposed partial causal model as follows: (x( = Pz, + c, where P is permutation matrix with scaling, is a constant vector.
For completeness, see details in Appendix E.2.
Success of MMCL Similar to Corollary on hyperspheres, Corollary 2 establishes that, on convex bodies, the representations f, fx) learned by MMCL are related to the true latent variables z, as (x) Pz, + c. This provides a foundation for the success of MMCL on convex bodies.
5 FROM PRINCIPLES TO PRACTICE: DISENTANGLEMERT IN CLIP MODELS
In theory. both Corollaries 1 and 2 suggest a disentanglement potential of CLIP-like models trained by MMCL, under the assumption that the variables in z₂ (and symmetrically, 2,) are mutually independent. we explore how these theoretical insights can be translated into practical guidance for the effective use of ClIP-like models.

---

Published as a conference paper at ICLR 2026
Corollary 1 shows that the representations (x) learned by MMCL are linearly related to the true latent variables zy via an orthogonal transformation, i.e. (x) = AzN + +c. This result holds under two key conditions: (1) the true latent variables are sampled from a hyperspherical latent space, and (2) the inference model, e.g., CLIP-like models, is trained in a hyperspherical inference space. Notably, CLIP-like models naturally satisfy condition (2), as they typically employ 1.2 normalization, constraining representations to the unit sphere. Therefore, under condition (1) holds, the representations from CLIP-like models can be passed through linear unmixing method (e.g.. FastICA (Hyvarinen, 1999)) to resolve the mixing matrix A, resulting in disentangled representations. worth to note that the geometry of the hypersphere. specifically the unit M 1-dimensional hypersphere, places an upper bound on the number of independent variables, i.e., M 1 at most.
shows that the learned representations f. (x) from MMCL, are already disentangled, i.e., fx(x) Pzz + c. This result mainly requires two conditions: (1) the true latent variables 2 are sampled from a convex body latent space, and (2) the inference space of the model is also constrained to convex body. However, CLIP-like models typically violate the second condition, as they operate in a hyperspherical inference space due to L2 normalization, even through we assume that the first condition may hold. Nevertheless, the insight from Corollary 2 remains useful with appropriate adjustments. In particular, the Corollary relies on the existence of an isometric mapping from the latent space to the representation space (see Eq. (48) in Appendix). Although global isometry from a convex body to a hypersphere is not feasible, it is reasonable to assume a local isometry between the convex body and small regions of the hypersphere. Based on this, we propose first applying Principal Component Analysis (PCA) to the representations f, (x). Then, FastCCA can be used to account for the orthogonal transformation introduced by PCA, enabling the extraction of the final disentangled representations. This PCA+ICA pipeline thus enables effective use of CLIP--ike models under the result of Corollary 2.
Unlike Corollary 1, Corollary
Remark 1. By leveraging the disentanglement capabilities of CLIP-like models, we can improve performance on tasks that benefit from disentangled representations, such as few-shot learning and domain generalization. This observation further motivates exploration of the disentanglement potential inherent in CLIP-like models across a broad range of downstream applications.
SYNTHETIC EXPERIMENTS AND REAL-WORLD EVALUATION
Synthetic Experiments In our initial experiments, we use synthetic data to validate our main identifiability results on hyperspheres and convex bodies, while also empirically assessing their robustness under significant violations of assumptions. We first sample p(z,) according to the distributions listed in Table 1. Additionally, we generate paired samples from the conditional distribution (Zzlz.z) following the distributions specified in the same table. Beyond hyperspheres, our experiments also consider bounded and unbounded spaces. Each experiment is repeated three times for every setting. For more details regarding experiments, refer to Appendix K.
To evaluate linear identifiability result in Corollary 1, we fit linear regression model between the ground truth 2, and representations f, (x) learned by MMCL and report the coefficient of determi- nation (R²). Further, to evaluate permutation identifiability result in Corollary 2, we employ the mean correlation coefficient (MCC) between the ground-truth 2, and representations fx(x) learned by MMCI. The first row in Table (left) and the first two rows in Table 1 (right), corresponding to the setting where the assumptions are satisfied, verify the identifiability results on hypersphere and convex bodies, respectively. Our empirical investigations have yielded a critical insight: discrepancies in the assumptions concerning marginal and conditional distributions, as well as the nature of the spaces (hypersphere and convex body), do not significantly impact performance. This robustness is demonstrated by the results detailed in Table (left) for the hypersphere space and Table 1 (right) for convex bodies. This might be attributed to the fact that the loss function described Eq. (2) predomi- nantly relies on expectation computations, inherently allowing for a wide range of approximations. If we can approximate the expectation calculations consistently across various distributions and spaces, it is reasonable that the identifiability results remain well within acceptable bounds.
Real-World Evaluation with Pretrained CLIP In real data, the true latent coupled variables are unknown. Therefore, we evaluate our theoretical findings from the perspective of disentanglement as discussed in Section 5. Again, we emphasize that, in contrast to previous studies on identifiability for MMCL (Daunhaver al., 2023; Yao et al., 2023; Gresele al., 2020), which rely on simulation
7

---

Published as a conference paper at ICLR 2026
Table I: Assessing identifiability up to linear (left) and permutation (right) transformations under varying assumptions. The first row in (left) and the first two rows in (right) represent settings that align with our assumptions in Corollary and Corollary 2, while the others show results for violated assumptions. S: Space, Sp: Sphere, U: Uniform, v: vMF (k = 1), L: Laplace (A 0.05), N: Normal 0.05), B: Box, Un: Unbounded, G: GenNorm (3 3).
Generative process
Model
p(z.)
p(z, (Z.)
S
g(2,(2.)
R²
Generative process
Model
Sp
U
99.5
Sp
U
L
Sp
V
99.4 = 0.2
S p(zz)
p(uala,)
g(z, lxc)
MCC
Sp
N
Sp
B
U
99.1 = 0.1
U
N
Un
N
90.3 =
B
97.2 #: 0.3
U
L
Un
N
92.2 0.3
B U
N
98.6a 0.2
U
L.
Un
G
99.1 0.4
U
N
Un
G
91.2
B
U
98.4* 0.1
Sp
N66
Sp
B
U
95.64 0.2
Sp
N
N
Sp
95.9 0.2
B
U
Un
96.44
Un
=)
N
Un
N
88.5
Un 1)
N
Un
N
89.2 0.2
8
5
(a) Smile
(b) Gender w/ Mustache
(c) Glasses
(d) Face Size
Figure 3: Disentangled Representations learned by combining pre-trained CLIP and FastICA. The results are aligned with our disentanglement findings.
experiments, this work validates identifiability through empirical analysis on real datasets and pre-trained CLIP model. This underscores the practicality of our theoretical contributions.
Disentangled representations for CelebA data According Section 5, we first extract represen- tations from the pre-trained CLIP model and then apply FastICA to these representations to achieve final representations for CelebA data (Liu al., 2015). We expect these final representations to exhibit clear signs of disentanglement. To validate this, we proceed to train decoder that reconstructs observational data using these extracted representations. Figure illustrates the effectiveness of our method through latent space traversals. Specifically, it visualizes changes in reconstructions as we traverse one dimension of the latent space at a time, showcasing out of 16 attributes uncovered by our approach. Additional results are available in Appendix H. This achievement not only validates our identifiability results, but also offers a new research line, i.e., learning disentangled representations by CLIP, or exploring how this disentanglement potential relate to the manipulation of pre-trained vision models, such as diffusion models.
Few-shot learning and domain generalization The goal of disentangled representations is to learn representations that transfer easily and robustly to downstream tasks, making them well-suited for few-shot learning and resilient to distribution shifts (Fumero et al., 2023). We thus focus on few-shot learning and domain generalization tasks further evaluate our disentanglement findings. We extract representations from a limited set of labeled samples using a pre- pre-trainne trained CLIP model, combined with FastICA align with the hypersphere, and with PCA followed by FastICA to align with convex body. These representations are then used to train a linear classifier. We evaluate the methods ImageNet (Deng al., 2009) for few-shot learning and test robustness ImageNet-V2
8

---

Published as a conference paper at ICLR 2026
Table Quantitative results for 2-shot learning and domain generalization by different methods. C: Linear Probe, 2: Linear Probe with FastICA, and @: Linear Probe with PCA and FastICA.
SOURCE
TARGET (IMAGENET-)
ENCODERS
METHODS
IMAGENET
V2
SKETCH
R
A
AVG.
RNS0
31.95
26.48
8.41
20.74
7.44
15.77
2
34.06
28.74
8.37
21.72
10.15
17.25
34.12
28.68
11.55
25.57
10.15
18.99
RN101
37.64
31.45
13.71
31.09
11.85
20.03
39.58
33.15
13.49
30.29
14.77
22.93
39.86
33.58
17.93
35.48
14.20
25.29
VIT32
1
38.23
32.00
16.17
33.67
12.88
23.68
40.21
33.97
16.54
34.79
15.72
25.26
39.34
33.44
19.02
36.98
14.69
26.03
VITI6
44.97
38.11
22.06
43.86
25.99
32.51
45.52
39.38
22.55
45.33
30.47
34.43
46.57
40.66
26.67
49.69
31.48
37.13
(Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-R (Hendrycks et al., 2021a). and ImageNet-A (Hendrycks et al., 2021b). Table 2 presents the performance metrics of the proposed methods in few-shot learning (the 'SOURCE' column) and domain generalization (the 'TARGET' columns). Analyzing the data in the 'SOURCE' column reveals that the proposed methods outperform the baseline approach, which trains linear classifier using representations directly obtained from pre-trained CLIP (i.e., the Linear Probe). This superior performance validates our disentanglement findings. The 'TARGET' column further reinforces the benefits of disentanglement, Refer to Appendix for more results.
Leveraging the Disentanglement Potential of CLIP-like Models for Few-Shot Learning the final experiments, we demonstrate how the disentan- glement potential pushes the boundaries of leverag- ing pre-trained models, e.g., CLIP. Recent progress shows that pre-trained CLIP's adaptability can be sig- nificantly improved with just few labeled training samples. The key to leveraging pre-trained CLIP for few-shot learning is effectively utilizing its ex- tracted representations from limited labeled data, as Tip-Adapter and Ada-eet methods proposed in the work (Zhang et al., 2022a). As we claim, leverag- ing disentangled potential of pre-trained CLIP can en- hance performance on tasks that rely on disentangled representations, including few-shot learning. There- fore, rather than using CLIP's raw representations, we apply FastICA to extract disentangled represen- tations for few-shot tasks. This be implemented plug-ane and-play way. As shown in Figure 4, incor-
80
70
Score (%)
65
60
Acapter
Ap-ccipter
To- Acaeter FastiCA
50
2
16
Number of labeled training examples per class
Figure Comparison of accuracy (%) achieved by different few-shot CLIP adap- tation methods across 11 datasets.
porating FastICA in the methods in (Zhang et al., 2022a), termed Tip-Adapter with FastICA and Tip-Adapter-P with FastICA, results in better performance, across 11 datasets. See Appendix J for more details.
7 CONCLUSION AND DISCUSSIONS
In this work, propose novel latent partial causal model for multimodal data that moves beyond the traditional DAG structure, using latent coupled variables connected by undirected edges to capture transferable knowledge across modalities. We establish theoretical link between this generative

---

Published as a conference paper at ICLR 2026
model and MMCL, showing that the representations learned by MMCL correspond to latent variables in the generative model, with linear and permutation transformations in hypersphere and convex body spaces, respectively. Our results provide the first thcoretical guarantees for the disentanglement capabilities of MMCI,, with applications in tasks like few-shot learning and domain generalization. Unlike prior simulation-based studies, work demonstrates the real-world utility of MMCL. and offers insights into leveraging pre-trained models like CLIP for disentangled representations. Our model challenges conventional DAG assumptions and provides flexible, practical framework that enhances the effectiveness of MMCL.
One of the main limitations of this work lies in the parametric assumptions, e.g., Eqs. (4) and (7), which may not strictly hold in real-worid applications. However, simulation experiments in settings where some of these assumptions are violated indicate that the theoretical results still largely hold (e.g., Table 1). Furthermore, empirical evaluations, including learning disentangled representations on face images (e.g.. Figure 3), few-shot learning and domain generalization on ImageNet-type datasets (e.g., Table 2), and few-shot learning across 11 cross-domain datasets (e.g., Figure 4), demonstrate the practical advantages our theoretical findings, providing additional support for their relevance.
ACKNOWLEEDMMENT
This project was partially funded by the Responsible AI Research Centre (Yuhang Liu, Zhen Zhang, Erdun Gao, Anton van den Hengel, and Javen Qinfeng Shi). Dong Gong was partially supported by the Australian Government through the Australian Research Council Discovery Early Career Researcher Award Australian Government through the Australian Research Council Discovery Projects (IP240102088). The authors also thank the anonymous reviewers for their constructive feedback.
Mingming Gong was partially supported by the
10

---

Published as a conference paper at ICLR 2026
Ethics Statement. This study complies with the ethical standards of ICLR. It relies exclusively public datasets and does not pose foreseeable negative impacts.
Reproducibility Statement. We provide thorough descriptions of the methodology and experi- ments. The code and instructions will be openly available once the paper is published.
REFERENCES
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representation learning. In International Conference Machine Learning, pp. 372-4077 PMLR, 2023.
Yuanhao Ban and Yinpeng Dong. Pre-trained adversarial perturbations. Advances in Neural Informa- tion Processing Systems, 35:1196-1209 1209, 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp.
pp. 446-461. Springer, 2014.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal representa- tion learning. arXiv preprint 22X3: 2022.
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf. and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. arXiv preprint arXiv:2306.00 02235, 2023.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia E Vogt. Identifiability results for multimodal contrastive learning. arXiv preprini urXiv:2303.09166, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 248-255. Ieee, 2009.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Sanja Fidler, Abhishek Sharma, and Raquel Urtasun. A sentence is worth thousand pixels. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1995-2002, 2013.
Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolà, Stefano Soatto, Bernhard Schölkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. Advances in Neural Information Processing Systems, 36: 27682-27698, 2023.
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Uncertainty in Artificial Intelligence, pp. 217-227. PMLR, 2020.
George I. Gropper. Why is picture worth thousand words? Audio Visual Communication Review, 11(4):75-95, 1963.
Michael Gutmann and Aapo Hyvärinen. Nonse-con estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
11

---

Published as a conference paper at ICLR 2026
Xiangteng He and Yuxin Peng. Fine- fine-grained image classification via combining vision and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5994 6002, 2017.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. JEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes mullii-mddl learning better than single (provably). Advances in Neural Information Processing Systems, 34:10944-10956. 2021.
Noelle J Hum, Perrin E Chamberlin, Brittany L Hambright, Anne C Portwood, Amanda C Schat, and Jennifer L Bevan. picture worth a thousand words: A content analysis of facebook profile photographs. Computers in Human Behavior, 27(5): 222--833, 2011.
A. Hyvärinen. Karhunen, and E. Oja. Independent Component Analysis. Adaptive and Cognitive Dynamic Systems: Signal Processing, Learning, Communications and Control. Wiley, 2001.
Aapo Hyvarinen. Fast and robust fixed-point algorithms for independent component analysis. IEEE transactions on Neural Networks, 10(3):626-634, 1999.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources, In Artificial Intelligence and Statistics, pp. 460-469. PMLR, 2017.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-433) 1999.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 859-868, PMLR, 2019.
Heinrich Jiang. Uniform convergence rates for kernel density estimation. In International Conference on Machine Learning, pp. 1694-1703. PMLR, 2017.
Yuming Jiang, Ziqi Huang, Xingang Pan, Chen Change Loy, and Ziwei Liu. Talk-to-edit: Fine- grained facial editing via dialog. In Proceedings of International Conference Computer Vision (ICCV), 2021.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autocncoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207-22111 PMLR, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658. PMLR, 2018.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005. URL https: aapi.... semant nt cachol a wtt. 60282629.
12

---

Published as a conference paper at ICLR 2026
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang. Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to purpose assistants. Foundations and Trends@ in Computer Graphics and Vision, 16(1-2):1-21,, 2024.
Paul Pu Liang, Amir Zadeh, and Louis-Philiipe Morency, Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 56(10): 1-42, 2024.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612- 17625, 2022.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212-220, 2017.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifying weight- variant latent causal models. arXiv preprint arXiv:2208. 14153, 2022.
Yuhang Liu, Zhen Zhang, Dong Gong, Erdun Gao, Biwei Huang. Mingming Gong, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Towards identifiable latent additive noise models. arXiv preprint arXiv:2403. 15711, 2024a.
Yuhang Liu, Zhen Zhang. Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifiable latent polynomial causal models through the lens of change. In The Twelfth International Conference on Learning Representations, 2024b.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Latent covariate shift: Unlocking partial identifiability for multi-source domain adaptation. Transactions on Machine Learning Research, 2025.
Yuhang Liu, Dong Gong, Yichao Cai, Erdun Gao, Zhen Zhang, Biwei Huang, Mingming Gong, Anton van den Hengel, and Javen Qinfeng Shi. I predict therefore am: Is next token prediction enough to learn human-interpretable concepts from data? In The Fourteenth International Conference on Learning Representations, 2026.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7086-7096, 2022.
Maria Lymperaiou and Giorgos Stamou. A survey on knowledge-enhanced multimodal learning. Artificial Intelligence Review, 57(10):284, 2024.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-graine' visual classification of aircraft. arXiv preprint 1306.5 151, 2013.
Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference on Artificial Intelligence and Statistics, pp. 4348 4380. PMLR, 2023.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXivv1807..3777 2018.
Naima Otberdout, Mohamed Daoudi, Anis Kacem, Lahoucine Ballihi, and Stefano Berretti. Dynamic facial expression generation on hilbert hypersphere with conditional wasserstein generative ad- versarial nets. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(2):848-88) 863, 2020.
13

---

Published as a conference paper at ICLR 2026
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505 IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot texi-to-image generation. In International conference on machine Pmlr, 2021.
learning, pp. 8821-8831
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 2019.
PMLR,
Harry Reinert. One picture is worth a thousand words? not necessarily! Modern Language Journal, pp. 160-168, 1976.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Confer- ence on Machine Learning, pp. 5628-5637. PMLR, 2019.
Bernhard Schölkopf, Dominík Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. causal and anticausal learning. In Proceedings of the 29th International Coference International Conference on Machine Learning, pp. 459-466, 2012.
Anna Seigal, Chandler Squires, and Caroline Uhler. Linear causal disentanglement via interventions. arXiv preprint arXiv:2211. 2022.
Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2556-2566 2018
Bernard W Silverman, Density estimation for statistics and data analysis. Routledge, 2018.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv: 121..0/22.04003 2012.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020.
Burak Varici, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.088 082.00000000. 2023.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self- self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 14:16111-1666) 2021.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 10506-10518, 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929-9939. PMLR, 2020.
Jianxiong Xiao. James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and patlern recognition, pp. 3495-3492. IEFE, 2010.
14

---

Published as a conference paper at ICLR 2026
Dingling Yao, Danru Xu, Sébastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability, arXiv preprint arXiv:2311.04056 2023.
Dingling Yao, Danru Xu, Sebastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability. In The Twelfth International Conference on Learning Representations, 2024.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, pp. 493-510. Springer, 2022a.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pp. 2-25. PMLR, 2022b.
Yaoyao Zhong, Weihong Deng, Jiani Hu, Dongyue Zhao, Xian Li, and Dongchao Wen. Sface: bigmoid-constraiine hypersphere loss for robust face recognition. IEEE Transactions on Image Processing. 30:2587-255 25998, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 130(9))2337-2348, 2022b.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pp. 19799.12990. PMLR, 2021.
15

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
A Related Work
17
B The Proof of Theorem 3.1
18
C Relation with Recovering All Information
18
D The Proof of identifiability on hypersphere
19
The Proof of Theorem
D.2
Identifiability result hypersphere
21
E The Proof of identifiability convex bodies
22
E.1 The Proof of Theorem 4.2
22
E.2 Identifiability result on Convex Bodies
24
F Differences from Previous Analysis for Mutilmodal Contrastive Learning
25
G Differences from Previous Analyses for Single-Modal Contrastive Learning
25
H More Results CelebA
26
More Results on ImygeNet-Tyype Data
26
J More Results on Few-Shot Learning Task
31
K Implementation Details
32
L Discussions FastICA vs. PCA Followed by FastICA
34
M Acknowledgment of LLMs Usage
34
N High-Level Discussion and Rationale for the used Assumptions
34
Rationale and Interpretation Assumptions Limitations and Practical Implications
34
35
O A Formal Definition of Disentanglement
35
Quantitative Validation on High-Dimensional Image
35
16

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Multimodal contrastive representation learning Multi- contrastive representation learning. driven by underlying transferable knowledge across modalities, aims to coalesce imputs from these diverse sources into a cohesive representation space. This is typically achieved using a symmetric version of the standard contrastive loss (Oord et al., 2018; Gutmann & Hyvärinen, 2010), a method designed to align accurate pairings while distinguishing incorrect ones (He & Peng, 2017; Radford et al., 2021). Although this approach has proven successful in a range of downstream tasks (Radford et al., 2021; Zhou et al., 2022a;b; Lüddecke & Ecker, 2022; Ban & Dong, 2022), there remains gap in our comprehensive theoretical and empirical understanding of the representations it learns. Recently, there has been growing interest in exploring multi-modal contrastive learning from various perspectives. For instance, the study by Liang et al. (2022) provides insights into the modality gap inherent in multi-modal contrastive learning. Similarly, the research presented by Nakada et al. (2023) establishes a link between general multimodal contrastive loss and SVD analysis. Additionally, Huang et al. (2021) posits that learning with multiple modalities can lead reduced population risk compared to using a subset of these modalities. Diverging from these approaches, our work delves into mudti-modal contrastive representation learning by examining its connection with generative models.
Past research has sought to comprehend the representations derived from standard cogll-mmdallty contrastive learning, examining them through the lens of alignment and uniformity (Wang & Isola, 2020), showing guarantees on the performance of the learned representations on the average classifi- cation task (Saunshi et al., 2019), or in terms of the identifiability of latent variables (Zimmermann et al., 2021; Von Kügelgen et al., 2021). Building on these foundations, our work takes a foreword step. We demonstrate that multi-modal contrastive learning can identify latent coupled variables, extending the insights from previous studies into the realm of multi- modalit...... Refer to Section G for more details.
Very recently, several studies have emerged, focusing on mudti-modal settings (Daunhawer et al., 2023; Yao et al., 2023). A clear distinction is that: the proposed model captures transferable knowledge across modalities by an undirected edge between latent coupled variables, while previous works often achieve it by introducing shared variables (Daunhawer et al., 2023; Yao et al., 2023). Notably, our modeling approach is more general, as it can be reduced to the shared variables used in previous works (Daunhawer et al., 2023; Yao et al., 2023) by enforcing an identical mapping on the undirected edge between latent coupled variables. Some of these works have only achieved partial identifiability of coupled variables (Daunhawer et al., 2023; Yao et al., 2023), specifically identifying latent content variables but not latent style variables. In contrast, our work achieves comprehensive identifiability results for all latent coupled variables, offering deeper level of understanding. Our research also diverges from the approach taken in Gresele et al. (2020) in two key ways: Firstly, we model ransferable knowledge across modalities using conditional distributions, whereas the latter utilizes identical variables for this purpose. Secondly, while Gresele et al. (2020) relies the premise that the mapping from the latent space to observations must be constrained by component wise corrupters to ensure identifiability, our findings do not necessitate such constraints. Refer to Section FF for details.
Nonlinear ICA and Causal Representation Learning Nonlinear Independent Component Analy- sis (ICA) aims to unravel latent independent variables from observational data that has been subject to a nonlinear mixture of these latent factors. However, as pointed out in the seminal work by Hyvärinen & Pajunen (1999), solving this problem is generally infeasible without specific underlying assumptions. A prominent direction in contemporary research leverages the concept of distribu- tional changes latent variables, which leads to the creation of multi-domain observational data. This approach has been extensively explored and developed in a series of studies (Hyvarinen & Morioka, 2016; 2017: Hyvarinen et al., 2019; Khemakhem al., 2020), each contributing to deeper understanding and more refined methodologies in the field of Nonlinear ICA. We build upon this body of research by incorporating co-occurrence patterns observed across multiple modalities. It is important to note the distinct difference between multi-domain and multi-modal approaches. The former typically implies a consistent mapping from the latent space to the observational space across all domains, whereas the latter accommodates different mappings for each modality. Additionally, while multi-domain approaches generally assume a totally shared latent variables across all domains, multi-modal methods allow for the existence of modality-specifff latent variables. This work is also
17

---

Published as a conference paper at ICLR 2026
distinct from prior studies causal representation learning (Von Kügelgen et al., 2021; Liu et al., 2022; Buchholz et al., 2023; Liu et al., 2024b: 2025; 2024a; 2026; Brehmer et al., 2022; Ahuja et al., 2023; Seigal al., 2022; Varici et al., 2023). While these works aim to address more general causal settings, achieving identifiability in such regimes typically requires substantially stronger assumptions, which can limit their applicability in real-world scenarios. In contrast, our approach is empirically grounded: extensive experiments with pre-trained CLIP models across diverse set of real-world tasks demonstrate practical benefits.
B THE PROOF OF THEOREM 3.1
Theorem 3.1. (The asymptotics of L) For fixed T > O, as the sample size N - co, the (normalized) multimodal contrastiveloss converges to
[e (((O)))))))))) (10)
lim L 2log N =2 No00
E [(t())) (xx,))((x,t)
E log E t-p(t)
x-p(x)
[e [E<((((h))()))))
E t-p(t)
log x-p(x)
This proof is done by mainly depending on the Continuous Mapping Theorem and the law of large numbers.
Proof.
N
lim L 2oog N N-20
Σ lim No
log
N
E
(((()))))))))
(E(M))))))
N
log
2log N,
2j= (((x)))))))))
N
N
log (((()))))))))))
lim
((f(())..
((((s)))((())))
log
2log
N
N
N N
(((())))))))
lim
d ([.()))
log
No00
N
(((())))))))))
2 N
log
log N)
2 log N
[e ((())))))))
2 E (xx))=((x.))
[((({()))..())))]+++
log E tmp(t)
x-x(x)
log E (((((s)))()))
t-p(t) x-p(x)
C RELATION WITH RECOVERING ALL INFORMATION
In this section, we show
log (=00(100))))))::]]]]]] x-p(x) txp(t)
log [ (=<((.(0)))(0))1+]]]
txp(t) x--((x)
x --H(((,((x) (F(
18

---

Published as a conference paper at ICLR 2026
Considering the symmetry evident in both the left and right sides of the equation, let us focus attention on the initial term on the left and its corresponding counterpart on the right.
E
log E
(xp(x)
t-p(t)
1 N
(((())))))))))
lim
log
Σ
(11)
N
j-1
8 lim
log PKDE (fg (x,)) + log ZKDE
(12)
No
N
F=1
= ((p((,,)))
+ log ZKDE,
(13)
Transitioning from Eq. (11) to Eq. (12), employ kernel density estimation, wherein the choice of kernel is influenced by the distance metric used. For instance, on a hypersphere, a von Mises-Fisher kernel is suitable, whereas on convex bodies, Laplace kernel aligns well with the L1 norm. In this context, log ZKDE represents the normalization constant associated with the kernel. The inherent symmetry in this setup allows us logically deduce the equation. Note that since here the bandwidth T can be optimized in MMCL, if true distribution is the same as the chosen kernel, Eq. (12) is equal to Eq. (11), i..., # in Eq. (12) can be =. Under certain conditions the kernel density estimation will converge to the real distribution, in that case in Eq. (12) can also be (Silverman, 201 8). Specifically, kernel density estimation converges to the true density (x) under the following assumptions:
The kernel K, 2 e d(((xx))/ is smooth, symmetric density function (e.g., Gaussian- like).
The number of samples oo, and the bandwidth - O, such that Ny oo.
The true density p(f is smooth (e.g., twice differentiable with bounded second deriva- tives).
These conditions ensure that PRDE (f, (x:)) converges the true density.
D THE PROOF OF IDENTIFIABILITY ON HYPERSPHERE
D.1 THE PROOF OF THEOREM 4.1
Theorem 4.1. (L converges to the symmetric cross-entropy) Under the assumptions defined in Eqq for the proposed latent partial causal model, the necessary condition f, gx = ge. denoted as h. for the optimal normalized multimodal contrastive loss given by Eq. (2) leads to the following reduction of the loss itself:
lim L- 2logog N
E
[H(p(z)zz)) Qh (z.(z.)))) =.p((.))
[H(((zu)z)) 9n [x)) (14)
2.-p(z.)
where H is the cross entropy, the conditional distributions on (Zi(zn) and ( (2r) are parameterized by the following:
In (zzl2)) (z.) (h((a))) h((x.//)) (((a))))))
(15)
(z./zz)) = Czxz)) (l (h()))))( h(xx))t)
(16)
with
Cz(2))= f C,(z,) = f (((a)))
To proof Theorem 4.1, first introduce the following Lemma.
Lemma 1. Consider the unit hypersphere space, given uniform prior p(z,), p(zz) = p(2z)=12) S RM denotes the space zz and conditional distribution p(z,(z,) C,))) exp (kzz, 2,), p(z.) follows uniform distribution.
=1211 where
19

---

Published as a conference paper at ICLR 2026
12|-1 f pl Z ) Cx) f exp (kzz zi dz,, then due to the unit hypersphere space, we have exp (kiz Zi ddz Cp(k)- As result, we obtain p(z.) 12
Proof. By Bayesian theorem, p(z)) / p(zs/zz (oZz)Zzz)pe (Zz))
Lemma 2. The normalized multimodal contrastive loss in Eq. (2) has an optimal global solution of 0, which can be attained under the following conditions:
h2 (m,, Zye= (e almost surely. for pair ((mz,zz) Ze), (m,, 2))), (CI),
h2 and be map (mg Zz) and (me 2,), respectively, to uniform variables on hypersphere, (C2),
Proof. First, is well known that traditional contrastive loss in single modality has an optimal global solution of log N (Oord et al., 2018; Tian et al., 2020), as result, the multimodal contrastive loss Eq. I has an optimal global solution of log N. For completeness, let focus on the first term in Eq. 1: ((LL))))
log ΣYIC (E(X)))()))
(17)
Under optimal contrastive learning conditions, the distance for positive pairs satisfies:
((((((x)...f)))))/ = 1, for negative pairs (x;, X;) where i * j: e <=<(t.(xx..(())))) e, where is a small value. As result, for each i, the denominator can be expressed as: 1 + (N - 1)c. Therefore, the first term in Eq. reduces to : - 1i=1log I-(N-1)E Clearly, when N is large, the first term in Eq. I equals to log Given that the second term is symmetric, we conclude that Eq. 1 has an optimal global solution of 2 log N. Therefore, Eq. 10 achieves global optimal solution of O. Moreover, this optimum is unique up to isometries and permutations. Minimizing the loss requires each positive pair to dominate its softmax denominator, which is only achieved when their embeddings are maximally aligned. Simultaneously, negative pairs must be mapped as far apart as possible under the bounded metric to minimize their influence. This configuration, tight positive alignment and maximal negative separation, is geometrically rigid: any deviation increases the loss. Thus, except for prreeerrrir transformations and index permutations, the solution is unique. Achieving the global minimum of Eq. 10 therefore necessitates maximizing the alignment of positive pairs. This occurs if and only if h (m, 2.2) hy (m,, 2t) almost surely, for real pair ((m,y 2z), (m, 2:)). (marked as (C1)). Thus, we obtain minimum solution of for the first term. Next, considering the remaining two terms in Eq. 10, detailed in Appendix C, we see an equivalent expression: (H((pp... [.. (x).
+ 2 log ZKDE- When both h, and h, map (m,, 2 and (m,, ), respectively, to uniform variables on hypersphere (marked (C2)), it reduces to (p (x)) + 2og ZKDE- Note that the entropy of a uniform distribution on the hypersphere @ is log( 2x M/2
(x),)(((()))) (t))) - H(((f p().
((1r/2y): ). where I' is the gamma function. Together with the fact that the 2n M/2 (i.c., log ZKDE), we
normalization constant of uniform distribution on hypersphere is log( [(M/2) arrive at the optimal solution of 0 for the last two terms.
Proof sketch The proof of Theorem 4.1 hinges on demonstrating the equality between the right- hand side of Eq. (14) and Eq. (10). Let us define h f, g. and h, f, ge. In Step using Lemma 2, we show that (1) f. ga f; gr, and (2) they are independent of the modality-seciiffc variables m, and m, In Step II, by defining h = f, g. f, gr and applying both the generative model from Eq. (4) and the inference model from Eqs. (15)-(16), we establish the theorem.
Step I Consider C1 in Lemma 2, e.g., h (m,, 2x)) h 2:) almost surely, for pair ((m,, Zz), (m,,,)) by differentiating it with respect m,, we have:
Oh (m,, zx) Oh, @h, dm
(18)
dm,
due to the independence between m2 and (m, 2:). Similarly, by differentiating it with respect to my we have:
Oh,mm,
Th ( 22)
(19)
8m.
Im:
20

---

Published as a conference paper at ICLR 2026
Based on Eqs. (18) and (19), conclude that both h2 and h, independent of the modality-
specific variables m, and my respectively, í.e., 1g(m;: Ze) (2,) and ( 2,) = h4 (zi). As result, we have h, (zy) distribution p(z,)zz) defined in Eq. (4). Note that this expression also holds true for 2, 2 (e.g..
(z,), for all real pairs (Z,, zz) sampled from the conditional
when 2, is sampled with the same value 2z), which implies (zz))) h, (z, As result, can obtain: hg h,.
Step According the results above: (m z,) h2 (z,). h, ze) (z,), and h. h,
from Step I, by defining h del h h4 can rewrite Eq. (10) as:
2
E (z (s.
((h(z.).b(z,)))
E x=-p(xx)
log E ze-p(x))
E Remp( (x))
log
E <xmp(z,))
(20)
We then connect the right-hand side of Eq. (14) with Eq. (20). To this end, since the two terms the right-hand side of Eq. (14) are symmetrical, we focus on one of the two terms for convenience, e.g.
E
H (v(2.)).)) 2n (Z/lzz)) Based on Lemma 1, it can be shown that:
(x.))
E pxpp(ma)
(H(r 9n (2)(2.))] (2)1))))))))))))
(21)
E ===p(
E ==n((al2z.))
[- log 9h (2 [z]]]
(22)
E
h(2z) h(2,)))))))))) / log C3 (zz)]
(23)
(2. 22) -p)...
E
(*.)-p(u,)) [log (z,))
(24)
(n. (.)((,, x.)
[log f ((())))
E
- (-h(zz) hh(z)/] (zz)-p(z.))
(25)
(z. (.))2.))
1 by Lemma 1, Eq. (25) simplifies to: Since p(2,) 12|-1 and p()))(2)))))))))))(()) = =121-1
and
[e ( + (26)
=- (2. (P).,.))
E
IE
log E z)mp(z,)
=.-p(z,)
On hypersphere space with radius r. due to [h(z,) h(2,) = =2r-2h (h(2)) ( Eq. 26 simplifies to:
[e (m((.))((x)))
E
E
log E 2:-p(z.)
(27)
(1)
2.mp(z.)
Similarly, for the second term in the right-hand side of Eq. (14), we can proof that:
E
E )x(...)))))
(x.)~((z))
(z,
(b(es))t))) + log Z].
E 2x=p(2))
log E ===p(z))
(28)
By combining Eq. (27) and Eq. (28), we can conclude the proof.
D.2 IDENTIFIABILITY RESULT ON HYPERSPHERE
Theorem 4.1 represents a adaptation of Theorem 1 from (Zimmermann et al., 2021) in the context of multi-modal setting. Specifically, within the confines of a single-modal framework, Theorem 4. is consistent with the findings presented in Theorem in (Zimmermann al., 2021). Consequently. this alignment allows us to employ Propositions I and from (Zimmermann et al., 2021) to demonstrate that the global minimization of the objective outlined in Eq. (5), specified in Theorem 4.1, identifies
21

---

Published as a conference paper at ICLR 2026
the latent variables Zz, well Zz. up linear transformations. For completeness, brief proof is provided herein, with comprehensive details available in the original work. Clearly, the global minimum of the cross-entropy between two distributions is reached if they match by value and have the same support. Therefore, for the optimal solution of the objective loss Eq. (14) in Theorem 4.1, have:
kz,) = 2h (zg(2e))
(29)
This expression also holds true for 2, 2x; additionally using that h maps from unit hypersphere to one with radius Viki we have:
e = (Zz) (b((x)) hhte))))
C Cy(
(30)
As the normalization constants are identical we get for all Zt,
kzz 21 = h(zz) hh(z)/TT
(31)
here we can see that maintains the dot product, which implies that h must be an orthogonal linear transformation by using Proposition 2 in Zimmermann et al. (2021). As a result, Theorem 4.1 supports the conclusion that the latent variables 2₂ (and z:) can be identified up to an orthogonal
linear transformation, í.e., the recovered latent variables f, (x) (note that h(z,) fx (x)), obtained by the minimization Eq. (5), is linearly related to the true Z as follows: fg (x) Azg + c, where A represents orthogonal matrix, and is a constant vector.
E
THE PROOF OF DEENTIFIBBILITY ON CONVEX BODIES
E.
THE PROOF OF THEOREM 4.2
Theorem 4.2. (C converges to the symmetric cross-entropy) Under the assumptions defined in Eq. (7) for the proposed latent partial causal model, the necessary condition fg o gg o go, denoted as h. for the optimal normalized multimodal contrastiveloss given by Eq. (2) leads to the following reduction of the loss itself:
[H(((z))a)), (m()))z)))) lim L- C-2log 2x-p(z.) N
E
E
[H(p(zc/2)), 9h (z. [2)]))))]
(2.)-p(z1)
(32)
where H the cross entropy, the conditional distributions qh (Zilzz) and q(zz (2)) are parameterized by the following:
2h (2.12..) -((c))))
(33)
Ih (z,(z,) = -ze -t.0
(34)
with
Cg(z))= f ((h(())) (((t( dzz. Cg(Zx) 6(b(()))) dz
Similar to the proof D.1, we first introduce the following Lemma.
Lemma 3. For random variables Zg E Z. and z; = Zo. assume that p(z.)) (==z== L/| if Z. and O otherwise, and assume that conditional distribution p(ze(za) C(zz) exp 8(zz,2))\\) AA). where 5 is symmetric metric induced by a norm, then p(Z:) converges to uniform distribution on Z, as 0,.
Proof. The proof can be done by the fact that as À + 0, the condition distribution p(zz(zz) converges to a delta distribution. resulting that p(z,) = p(zz). More specifically, as we will let À 0 in the procedure, it is notable that the normalize ((z,) actually depend on À and should be write as
22

---

Published as a conference paper at ICLR 2026
C(Zz, A) in more formal way. With simple integration trick, it would be straightforward to show that C(z,,,) A) can be decomposed as CZa,, A) x (zx)).
By definition we have
p(z))= = S Ja p(zx)p(zilze)
Sn n.
p((z))
('(zz) (-)(2, )
(35)
N
1
lim +00
Σ
(zz, exp (- -8(ze, ( 21)12), Vi, Ze, p(z))
C"
then obviously we have that
N
lim p(z)) = lim A-0
lim 2-0
C (Zz:) exp ( ò (Zur, 21)/2)
(36)
lim lim C' exp C'exp
2-0
where C² exp (- 8(0, z.) dzz, It is obvious that (36) can be viewed Kernel Density Estimation over samples Viz, p 2. and obviously lim,-o, p(z)) will converge to p(zz) (which is uniform distribution) under quite mild condition (for details of the convergence, refer to Jiang (2017)).
Proof sketch Similar to hypersphere, the proof of Theorem 4.2 can be done by demonstrating that the right-hand side of Eq. (32) is equal to the right-hand side of Eq. (10) on convex bodies. To achieve this, using Lemma 2, show that fz gz f, gt, and they are independent of the modality-speciffi variables m, and m,, respectively. Finally, by defining h = f, o og f₃ gr, and using the inference model (33) and (34), we obtain our result.
Step On convex bodies, and define he f o gu and h, = fg o ge. Consider CI in Lemma 2, e.g.. h, (m,, z,.) h, (m,,,, 2.) almost surely, for pair ((m,, 2,,), (m,,,)) Similar to Step in Appendix D.1, by differentiating it with respect to m, and m, respectively, we can conclude that both h2 and h, are independent of the modality-specific variables m and m,, respectively, iee., he Z2) = hh (z.) and h m, 2;) h,(2,), Further, since h2 (zx) h, (z;)) hold, for all real pairs (Ze, sampled from the conditional distribution p(z((z,) defined in Eq. (7), this expression also holds true for 2, Zg, which implies (z = h1 (Z.). As a result, we can obtain: h, h,
Step II According the results above: har k..) h (2x)). he(m,, 21) hr((z), and h, h
by defining h def f. g. f, og we can rewrite Eq. (10) as:
IE
[d(b(z.)..(2z))//)
E log E (n(d((())))(aa))))) 2(-p(z,)
(2., (.p(..)))
E log E ===)()))
(37)
2.-p(z,)
We then connect the right-hand side of Eq. (32) with Eq. (37). To this end, since the two terms the right-hand side of Eq. (32) are symmetrical, we focus on one of the two terms for convenience, e,g.
IF
HH (1((2)2)))) (2n)), 2n (2)|2=))] It can be shown that:
E ===p(a=)
[H(((z))z) 9h (z))2z))
(38)
E
[logg (z][z.]]
(39)
z =xx) (<<((1)(2.)
E (2, pp(....)
(8(h(z)) h1l / log (z,)
(40)
E (z, 2, J-p(s, 2
(5(h(z)). (())))))(Zz))))))))) ((2))/)
E [log (z,)))
(41)
(xp)))
E [log ʃ ((-d(tboss)
E
(6(h(z.), (z.)~p(z.)
(42)
(2, 2.) (2.2.2
23

---

Published as a conference paper at ICLR 2026
Since p(zz)= = =12 and p(z:) =| by Lemma 3, Eq. (42) is equal to:
[e Te-5((
E
(((b(z)..hzz)))
E Mamp(()))
log E 2e-p(z))
log 2cl
(z, (41
(43)
Similarly, for the second term in the right-hand side of Eq. (32), we can proof that:
E (H(((z))))) (nxz//z))) (n
E
b(z))/)
(44)
(z,)-p(z,)
(z, 2. pp(...)))
log ===p(z,) [ (b(( log [2c]-
+
=x-p(z,)
(45)
By combining Eq. (43) and Eq. (45), we can conclude the proof.
E.2 IDENTIFIABILITY RESULT ON CONVEX BODIES
Theorem 4.2 represents symmetrical adaptation of Theorem 3 from (Zimmermann al., 2021). This alignment allows us to employ Propositions 4, Lemma and Lemma A from (Zimmermann et al., 2021) to demonstrate that the global minimization of the objective outlined in Eq. (32), specified in Theorem 4.2, identifies the latent variables 2,. as well as 2z, up to linear transformations. For completeness, brief proof is provided herein, with comprehensive details available in the original work. Clearly, the global minimum of the cross-entropy between two distributions is reached if they match by value and have the same support. Therefore, for the optimal solution of the objective loss Eq. (10) in Theorem 4.2, we have:
p(z,2z,) = 9n (2,lz,),
(46)
This expression also holds true for 2, = Zn we have:
Cp(2m) (0,, Cg(zz)= ( h(x))))).(x)))/) CC (z.) Cg(z,)
(47)
As the normalization constants are identical get for all Z Zt,
8(2z, z.)
(48)
Then, by limiting 8 be an I metric for a 2 1, ar 2 or the a-th power of such an I metric, using the Theorems and 6 Zimmermann al. (2021), is composition of input independent permutations, sign flips and rescaling. In other words, Theorem 4.2 establishes that the latent variables 2y (and z,)
identifiable up permutation transformation, i.e., the recovered latent variable (x), obtained through the minimization of Eq. (8), is related to the true as follows: fx(x) = Pzz C, where P represents an permutation matrix with scaling, and is constant vector.
24

---

Published as a conference paper at ICLR 2026
F DIFFERENCES FROM PREVIOUS ANALYSIS FOR MUTILMODAL CONTRASTIVE LEARNING
This work differs from previous works focusing identifiability analysis for multimodal settings (Daunhawer et al., 2023; Yao et al., 2023; Gresele et al., 2020) across the following dimensions.
Modeling Setting This work proposes modeling transferable knowledge across modalities by latent coupled variables. In contrast, previous works (Daunhawer et al., 2023; Yao et al., 2023; Gresele et al., 2020) often achieve this by introducing the same/shared variables. The advantages of employing latent coupled variables are thoroughly justified in Section 2. Loosely speaking, From the perspective of model flexibility, the proposed model can considered a generalization of previous works. This generalization is apparent the proposed model scamlessly reduces to single-moda setting when the mixing functions from latent space to observed space are enforced to be identical, and specific variables are omitted.
Identifiability Results The identifiability results obtained in this work diverge from those found in previous works (Daunhawer al., 2023: Yao et al., 2023), both in terms of breadth and depth of identifiability, due to the introduction of the undirected edge between Zy and z,. a) Breadth of identifiability: Unlike earlier works that often achieve only partial identifiability of latent coupled variables 2u or Ze, e.g., latent content variables but not latent style variables (Daunhawer et al., 2023; Yao et al., 2023), our model extends this scope to ensure complete identifiability of latent coupled variables 2, and 2, b) Depth of Identifiability: In terms of depth, this work identifies latent coupled variables 2u and 2; up to linear or permutation transformations. As a result, after applying linear ICA method, can obtain component- wise identifiability, i.c., recovering independent latent variables up to permutation and scaling. This level of precision offers an enhancement over the block identifiability result in previous studies (Daunhawer al., 2023; Yao al., 2023), which only identifying latent variables up to a nonlinear invertible mapping, even for independent latent variables. The differences above in both breadth and depth of identifiability results enable us, for the first time, to unveil the component-wise disentanglement capabilities of multimodal contrastive representation learning.
Practical Significance in Real Applications Prior studies (Daunhawer et al., 2023; Yao et al., 2023; Gresele et al., 2020) have predominantly relied on simulation experiments, which often encounter a substantial gap between the assumptions made in theoretical analyses and the practical conditions of real-world applications. In contrast, our work bridges this gap by validating our theoretical findings using pre- rre-trinee CLIP models over 16 diverse real-world datasets. This empirical approach not only substantiates the practical effectiveness of our theoretical results but also demonstrates their applicability and robustness in real- world multimodal settings, highlighting significant departure from previous work in terms of real-world applicability.
G DIFFERENCES FROM PREVIOUS ANALYSES FOR SINGLE-MODAL CONTRASTIVE LEARNING
This work sets itself apart from prior studies focused the analysis of single-modal contrastive learning (Zimmermann et al., 2021; Von Kügelgen et al., 2021) in the following key aspects.
Problem Context Previous works primarily address single-modal scenarios, whereas our proposed model extends this framework to the more complex multimodal domain. This extension can be viewed as generalization of prior approaches. Specifically, our model naturally reduces to a single-modal setting when the mixing functions from the latent space to the observed space are identical, and certain variables are omitted. By expanding the scope to multimodal data, our approach addresses the limitations of prior studies and provides more comprehensive understanding of contrastive learning.
Technical Perspective Addressing multimodal settings requires significantly broader technical developments compared to single-modal analyses. To this end, we developed Theorem 3.1. which generalizes the asymptotic analysis of contrastive learning to the multimodal context, providing robust theoretical foundation. Bridging the gap between single-modal and multimodal settings
25

---

Published as a conference paper at ICLR 2026
also necessitated novel theoretical insights. For instance, Theorem 1 and Theorem 4.2 establish critical connections between multimodal contrastive learning and traditional single-mode frameworks, enabling unified understanding across these domains. These results not only expand the applicability of contrastive learning but also highlight the intricate dependencies introduced by multimodal data.
New Insights In the multimodal context, key challenge effectively modeling the connections between different modalities. This motivates the central insight of our work: latent coupled variables, linked by a unidirectional edge, provide foundation for exploring whether partial causal models are sufficient for multimodal learning. As highlighted in the introduction, we offer theoretical support for the success of multimodal contrastive learning, including guarantees for its disentanglement capabilities. From practical perspective, recommend refining representations from pre- pre-trained CLIP-like models rather than using them directly. Specifically, applying linear ICA methods, such as the hypersphere), combining PCA and FastICA (aligned with assumptions on convex bodies), can enhance performance on tasks that rely on disentangled representations. These insights not only validate the robustness of our theoretical findings but also emphasize their practical significance in real-world applications.
FastICA (aligned with assumptions
H
MORE RESULTS ON CELEBA
Figures 5 illustrate the 16 distinct disentangled representations obtained using pre-trained CLIP with FastICA. Interestingly, our method achieves competitive results compared to specialized disen- langlement techniques, such as FactorVAE (Kim & Mnih, 2018) and 3-TCVAE (Chen et al., 20.8). Specifically, FactorVAE identified disentangled attributes, while 3-TCVAE reported 15, whereas our approach successfully discerns 16 distinct disentangled representations.
is important to note that this comparison is not meant to position our method as a more effective disentanglement technique. Rather, our experiments are designed solely to validate our theoretical findings. We present this comparison to provide insight into the potential of leveraging CLIP for learning disentangled representations, therehy motivating future research in this direction. A particularly interesting avenue could exploring how disentanglement capabilities relate to the manipulation of pre-trained vision models, such diffusion models.
I MORE RESULTS ON IMAGENET-TYPE DATA
Table 3: Quantitative results for 16-shot transfer learning and domain generalization by different methods. Lin. (Linear Probe).
SOURCE
TARGET (IMAGENET-)
ENCODERS
METHODS
IMAGENET
V2
SKETCH
AVG.
RN50
LIN. P. FASTICA
55.36
45.45
18.22
34.09
12.52
27.77
57.82
47.78
LIN. P. W/ PCA AND FASTICA
57.37
47.67
20.39
38.76
12.89
29.93
RN101
LIN. LIN. P. W/ FASTICA LIN. P. W/ PCA FASTICA
60.98
50.36
25.80
46.61
18.64
35.35
61.86
$1.85
27.29
49.29
19.89
37.08
61.58
51.44
28.86 86
50.32
19.97
37.64
VIT32
60.76
50.92
37.15
LIN. P. W/ FASTICA LIN. P. w/ PCA AND FASTICA
61.94
51.95
30.30
51.82
20.81
38.72
62.00
$2.39
30.39
$1.61
20.96
38.84
VIT16
LIN. P. LIN. P. w/ PASTICA
67.17
$7.01
35.43
60.96
35.41
47,20
FASTICA
68.12
58.45
37.17
67.96
58.38
38.75
65.45
38.28
50.22
26

---

Published as a conference paper at ICLR 2026
+
(a) Smile
(b) Brightness
F
S
T
(c) Hair color
(d) Eye shadow
3
8
(e) Gender w/ Mustache
(1) Glasses
Figure 5: Disentangled Representations learned by combining pre-train CLIP and FastICA.
27

---

Published as a conference paper at ICLR 2026
3
4
P
E
Face width
(b) Face Size
6
6
T
+
(c) Background color
(d) Skin color
S
6
#
(c) Azimuth
(f) Background remove
Figure 6: Disentangled Representations learned by combining pre-train CLIP and FastICA.
28

---

Published as a conference paper at ICLR 2026
(a) Facc Height
(b) Hair Style
T
+
+
(c) Hair length
(d) Age
Figure 7: Disentangled Representations learned by combining pre-trat pre-train CLIP and FastICA.
Table 4: Quantitative results for 8-ohot transfer learning and domain generalization by different methods. Lin. (Linear Probe).
SOURCE
TARGET (IMAGENET-)
ENCODERS
METHODS
IMAGENET
V/2
SKETCH
AVG.
RN50
LIN. P. P. W/ FASTICA
49.33
40.83
15.06
31.23
10.99
24.53
51.99
43.58
34.35
26.56
LIN. P. W/ PCA AND FASTICA
51.42
42.93
17.28
35.53
12.33
27.02
RN101
LIN. P. LIN. P. W/ FASTICA LIN. P. PCA AND FASTICA
55.41
46.04
23.38
43.26
16.88
32.39
56.59
47.47
22.09
44.59
18.39
33.14
55.84
46.59
33.37
VIT32
LIN.
55.17
46.11
25,53
45.32
18.35
33.83
LIN. P. W/ FASTICA LIN. P. w/ PCA AND FASTICA
56.90
47.96
27.62
49.13
20.31
36.26
55.83
46.55
26.54
46.77
18.80
34.67
VIT16
LIN. P. LIN. w/ FASTICA LIN. P. w/PCA AND FASTICA
61.82
$2.34
32,26
55.93
32.63
43,29
63.55
54.81
61.54
38.21
47.29
63.47
$4.32
35.88
61.88
37.35
47,36
29

---

Published as a conference paper at ICLR 2026
Table 5: Quantitative results for 4-shot transfer learning and domain generalization by different methods. Lin. P. (Linear Probe).
SOURCE
TARGET (IMAGENET-)
ENCODERS
METHODS
IMAGENET
V2
SKETCH
AVG.
33.67
20.29
LIN. P. W/ FASTICA
44.10
36.07
12.75
30.15
11.64
22.65
AND FASTICA
35.38
12.29
RNI01
LIN.
48.23
39.53
18.80
38.10
14.32
27.69
LIN. P. W/ FASTICA LIN. P. W/ PCA AND FASTICA
49.43
41.02
17.49
39.33
15.25
28.27
49.01
40.25
19.26
39.71
14.75
28.49
VIT32
LIN. P LIN. P. FASTICA LIN. P. W/ PCA AND FASTICA
47.82
39.53
21.51
40.94
15.99
29.49
49.43
40.66
22.66
41.78
16.41
30.38
49.48
41,09
23.72
43.48
16.77
31.27
VIT16
54.30
46.06
LIN. P. w/ FASTICA AND FASTICA
56.65
48.18
28.27
$5.50
33.39
41.33
56.16
47.46
31.71
41,22
Table 6: Quantitative results for 1-shot transfer learning and domain generalization by different methods. Lin. P. (Linear Probe).
SOURCE
TARGET (IMAGENET-)
ENCODERS
METHODS
IMAGENET
SKETCH
A
6.55
LIN. P. w/ FASTICA
23.22
19.68
6.37
13.84
7.21
11.77
LIN. P. w/ FASTICA
24.06
20.26
6.85
17.54
8.05
13.18
RN101
LIN. P.
26.05
21.48
9.90
23.85
10.17
16.35
W/ FASTICA
LIN. P. w/ PCA AND FASTICA
28.50
24.17
11.63
26.38
12.28
18.62
VIT32
LIN. LIN. P. W/ FASTICA W/ PCA FASTICA
26.99
22.99
11.93
25.25
11.56
17.93
29.21
24.80
9.97
21.23
12,23
17.06
29.05
24.45
12.39
12.56
19.25
VIT16
LIN.
32.42
27.64
16.34
34.28
21.84
25.02
LIN. P. W/ FASTICA LIN. P. W/ PCA AND FASTICA
34.35
29.31
13.91
28.61
23.24
23.77
35.20
30.26
19.17
38.87
26.41
28.68
30

---

Published as a conference paper at ICLR 2026
J
MORE RESULTS ON FEW-SHOT LEARNING TASK
200
94-
62
20-
53
ecamples
treining esamples
(a) Caltech101
(b) DTD
(c) EuroSAT
Tarilaute of Peanica
Surtdapart
78-
Score (56)
Score
Sengle Pa
50-
1.2
16
1
16
1.2
0
16
Nurbbr of ilublled triiing examples per dass
Numb of loblled training csamples per class
Number of laboled wriniing examplis per class
(d) FGVCAircraft
(c) Food101
(f) ImageNet
100
70-
Sore (5)
Store 061
80
60
75-
70-
82-
50
Te Teudeaene ofeenca
05
00
50
labeled
ofueblled training
laboled
Oxford Flowers
(h) Oxford Pets
(i) Stanford Cars
72.5
70.0
75.0
67.5
72.5
10.5
675
00.0
05.0
575
Te Satude
10
2
28
Number of labeled training exa moles per class
Number of labsled trairing examples per dasss
(i) SUN397
(k) UCF101
Figure 8: More results on few-shot learning task: comparison of top-1 accuracy (%) achieved by various few-shot CLIP adaptation methods across 11 datasets, including ImageNet (Deng et al., 2009), 1altech101 (Fei-Fei et al., 2004), FGVCAircraft (Maji et al., 2013), UCF101 (Soomro et al., 2012), ETroSAT (Helber et al., 2019), Flowers 102 (Nilsback & Zisserman, 2008), StanfordCars (Krause et al., 2013), DTD (Cimpoi et al., 2014), Food101 (Bossard et al., 2014), OxfordPets (Parkhi et al., 2012), and, SUN397 (Xiao et al., 2010). The x-axis indicates the number of training examples per class. incorporation of FastICA notably enhances the performance of the original methods, Tip-Adapter and Tip-Adapter-F, proposed by (Zhang et al., 2022a).
31

---

Published as a conference paper at ICLR 2026
K IMPLEMENTATION DETAILS
We perform all experiments using the GPU RTX 4090, equipped with 32 GB of memory.
Synthetic Data We consider latent coupled variables 2, and 2,, each with dimensionality of 10. Additionally, we have modality-speciffi latent variables m, and m,, both set to a dimension of 5. The process begins with sampling from the marginal distribution p(zz), and the samples of modality- specific latent variables m, and m, are obtained by sampling from Gaussian distributions with zero mean and one variance. We then create real pairs by sampling from the conditional distribution p(2 a (z)) The observational data x and t are generated using two different Multi-Layer Perceptrons (MLPs). Specifically, we utilize MLPs comprising three hidden layers with leaky ReLU units and random weights. To ensure the invertibility of the MLP g. we carefully control the condition number of the weight matrices. For our encoders concerning both 2, and Zz, we adopt an MLP architecture with leaky ReLU units.
Evaluation: To evaluate the linear identifiability result established in Corollary I, we assess how well the learned representations 2 preserve the structure of the ground-truth latent variables zz up to a linear transformation. Specifically, we perform the following steps:
Learned Representations Extraction: We first obtain representations 2g learned by multi- modal contrastive learning.
2. Linear Regression Fitting: We fit linear regression model of the form:
2
Az + 6,
where Å is learned transformation matrix, is offset vector, and represents residual
errors.
3. Coefficient of Determination (R²²) Computation: We compute the R²² score, defined as:
R² Σ, Bzzzi (Azz,i 6)1 +0
Σ, 22,5 - 2
where 3, the mean of 2g This metric quantifies how well the learned representations can be linearly mapped to the true latent variables.
4. Analysis Under Different Assumption Violations:
We repeat the evaluation under settings that both satisfy and violate the theoretical assumptions, as listed in Table allowing us to empirically assess the robustness of the identifiability results.
By reporting the R² scores across different conditions, we quantify the extent to which multimodal contrastive learning successfully recovers the latent variables up to linear transformation.
to evaluate permutation identifiability result in Corollary 2, we employ the mean correlation coefficient (MCC) between the ground-truth 2, and representations (x) learned by multimodal contrastive learning. To compute MCC, we follow these steps:
We first calculate the correlation coefficients between all pairs of ground-truth source variables and representations learned by multimodal con- trastive learning. Specifically, for each pair of source component Zrr; and recovered latent component Žz.j. we compute the Pearson correlation coefficient:
Compute Correlation Coefficients:
Cov(zz.;:
Pi.j Ve,,;:un..
(49)
where Cov(..) denotes the covariance, and Uz..: and o2... are the standard deviations of the respective components.
2. Solve the Linear Sum Assignment Problem: Since the estimated components may be permuted relative to the ground-truth -ruth variables, we solve linear sum assignment problem to determine the optimal one-to-one mapping between the ground-truth and the learned representations. The goal is to maximize the total absolute correlation across all assigned pairs.
32

---

Published as a conference paper at ICLR 2026
TeLLUeBNCCov///asssosee21122 512, kernelsiz==l,strid==,,, http://psss////aaa/////////aaaaa pa
ReLLVBBNCCon/TTansssseZZll1Z 64, kemellsize=4, padding=0))
ReL.UBNNCoo//Tansooeel0..... 64, kernelsize=44 padding==
ReILUBBNNCCo//Tannnpo222dddd 32, kernelsize=,, stride padding----aa---gi----- padding=0))
ReILUUBBCCCo//Tasspps222222222 32, kernelsize=4, striddtei paddindn padding=()) Conv ConTTransposc2d(32, 3, kernelsize=4, stride=2, padding=1)
Table 7: Decoder for the image data.
3. Compute the Mean Correlation Coefficient (MCC): Given the optimal assignment the ground-truth variables to the learned representations, we compute the mean of the absolute values of the assigned correlation coefficients:
1 MCC
Σ (i,(()b d
(50)
where (i)) denotes the index of the assigned representation corresponding to the ith latent variable, and d is the number of latent variables.
A high MCC indicates that the learned representation closely match the true source variables, up to permutation transformations, thereby validating the identifiability of the learned representations.
Disentangled Representation Learning on CelebA To obtain disentangled representations for the CelebA dataset, we initially employ the FastICA implementation available in the scikit-learn software on the representations extracted from the pretrained ViT-B/32 encoder. Subsequently, we train the decoder, as outlined in Table 7, utilizing Mean Squared Error (MSE) loss.
Experiments of Linear Probe In our experiments with ImageNet-Type data, we utilized the PCA and FastICA implementations provided by scikit-learn. For proposed method, which combines PCA and ICA, we configured the number of components to 500 for PCA, and for FastCA, we set to 160 for 1, and 4-shot learning scenarios, and 200 for 8 and 16-shot learning scenarios. When employing ICA alone, we chose to use 300 components. For the proposed method with ICA only, we set number of components to 300. Following the setting of linear probe in CLIP, train logistic regression classifier using seikit-learn's 's L-BFGS implementation, with maximum 1,000 iterations. We determine the L2 regularization strength using a hyperparameter sweep on the validation sets over the range between 10 and 10 with 96 logarithmically spaced steps. To save compute required for the sweeps, we perform a parametric binary search and iteratively halves the interval around the peak until it reaches resolution of 8 steps per decade. The hyperparameter sweeps are performed on validation split of each dataset.
CastICA plug-and-play Tool. We incorporate FastICA in the framework proposed in (Zhang et al., 2022a) to enhance its ability for few shot learning. The framework consists of two primary modules: one keeps the zero-shot capabilities of pre-trained CLIP, ensuring effective utilization of prior knowledge, while the other, the cache module, constitutes the central contribution of the work. The cache module endeavors transfer knowledge from labeled training samples. Given the above, we integrate FastICA into the cache module, preserving the invaluable prior knowledge derived from the zero-shot abilities of pre-trained CLIP. For parameter settings in FastICA, we opted for 100 components for the majority of datasets. Specifically, we assigned 350 components for the ImageNet dataset, 300 components for the OxfordPets dataset, and 50 components for the AT dataset. A learning rate of 0.1 was employed for implementation. For the remaining parameter settings, we adhered to the specifications outlined by (Zhang et al., 2022a).
33

---

Published as a conference paper at ICLR 2026
L DISCUSSIONS ON FASTICA vs. PCA FOLLOWED BY FASTICA
Our theoretical findings are based on two distinct assumptions: one on the hypersphere (Sec. 4.1) and the other convex bodies (Sec. 4.2). Each of these assumptions motivates a corresponding practical method for real applications, namely FastICA, and PCA followed by FastICA, respectively. In practice, however, the true latent generative process is typically unknown, making it difficult to determine a priori which method is more appropriate. From an empirical standpoint, we observe that face image datasets, such as CelebA, tend align more closely with the hypersphere assumption. This observation is supported both by our experiments on CelebA, where learning disentang led representations under the hypersphere assumption improves performance, such as dynamic facial expressions generation, dynamic facial expression transfer(Otberdout et al., 2020), face recognition (Zhong et al., 2021; Liu et al., 2017). Moreover, consistent with the main motivation for learning disentangled representations, we find that the representations obtained under the hypersphere as- sumption lead to improved performance on related downstream tasks, further validating its practical usefulness.
M ACKNOWLEDGMENT OF LLMS USAGE
We acknowledge that large language models (LLMs) were used in this work only for word-level tasks, including correcting typos, improving grammar, and refining phrasing. No substantive content, results, or scientific interpretations were generated by LLMs. All scientific ideas, analyses, and conclusions presented in this manuscript solely the work of the authors.
N HIGH-LEVEL DISCUSSION AND RATIONALE FOR THE USED ASSUMPTIONS
Our identifiability analysis, like most theoretical works on latent variable recovery, relies on specific parametric assumptions about the underlying Data Generating Process (DGP) for 2r and 2,. While the exact DGP of large-scale multimodal data is unknown, these assumptions are essential for theoretical tractability and are motivated by prevalent machine learning practices and geometrical constraints.
N.1 RATIONALE AND INTERPRETATION OF ASSUMPTIONS
We introduce two sets of assumptions, primarily centered on the nature of the latent space geometry and the distributional modeling of the coupled variables.
Hypersphere Assumptions (Eq. 4)
Latent Space Geometry (S--1)); The assumption that the latent space resides on Hyper- sphere is motivated by consistency with models trained via MMCL. Specifically, modern architectures like CLIP typically enforce L2 normalization on their embeddings, which geometrically constrains the learned representations to lie on the unit sphere. Therefore, assuming the underlying generative factors are also on the hypersphere is a natural choice for space matching. Moreover, this geometry is inspired by prior work in Zimmermann et al. (2021), which demonstrates its potential for achieving disentanglement in the single-modal contrastive learning context.
Marginal Distribution p(z.,) as Uniform: This represents a maximum-entrops assumption. Essentially, in the absence specific prior knowledge, we assume that the distribution of the shared latent variables. Zg, is uniform across the latent space.
Conditional Distribution p(z,lz.)) as von Mises- Fisher (yMF) Distribution: The vMF distribution is the natural counterpart of the Gaussian distribution defined on sphere. Its parameterized form models the semantic coupling by formalizing the objective of MMCL: given factor Zy, the distribution expects its positive pair z, to be concentrated nearby with high probability. The alignment parameter kz;" 2, precisely quantifies the strength this shared semantie information across modalities.
The assumptions for the convex body (e.g., hyperrectangle) case provide an alternative geometric setting, often preferred in classic disentanglement works.
Convex Body Assumptions (Eq. 7)
34

---

Published as a conference paper at ICLR 2026
Latent Space (2,) Bounded Convex Body: This definition specifies a non-spherical, bounded space, which is typically crucial for ensuring identifiability and has been used in previous related works Zimmermann al. (2021).
Conditional Distribution p(z)(zz)) as Exponential Distribution: The mathematical form e-5(x,,2)/A models the coupling relationship by assuming the likelihood decays expo- nentially with the distance (6, a distance metric induced by a norm between the coupled variables, This implies that given Zy, the paired variable z₁ likely be found in its immediate vicinity.
N.2 LIMITATIONS AND PRACTICAL IMPLICATIONS
While these assumptions are theoretically sufficient for identifiability, as we have shown, their strict adherence in real-world scenarios is challenging to verify. This difficulty arises because the true data-generating process is unknown, making direct verification of conditions generally impossible. As is common in practice, performance gains on downstream tasks are therefore used as a surrogate to assess the plausibility of the theoretical assumptions. In particular, if the methods derived from identifiability theorems (e.g., using FastICA or PCA+FastICA to recover disentangled representations) consistently yield strong improvements across diverse downstream tasks as demonstrated in extensive experiments on few-shot learning and domain generalization- it is reasonable to infer that the underlying assumptions either satisfied or, more likely, approximated sufficiently well for the theory to be practically meaningful.
O A FORMAL DEFINITION OF DISENTANGLEMENT
Definition (Component-wise Disentanglement), A representation fx( (and symmetrically, fe(t)) learned by MMCL is defined as Component-wise Disentangled if two conditions are met:
1. Factor Independence (Prerequisite): The components of the underlying latent coupled variable z₂ (and z,) are mutually statistically independent.
2. Identifiability up to Trivial Transformation: The representation fx( (and symmetri- cally. ft)) is related to the true latent variable zu (and 2.) through a simple, invertible transformation up to constant c:
(x) Tzg + and symmetrically f((t)= c
where T (and T is a matrix of trivial ambiguity, specificully:
T is an orthogonal matrix (in the hypersphere latent space, Corollary 1).
T is permutation matrix with scaling (in the convex body latent space, Corollary 2).
This result guarantees that the shared, independent components of 2n (and z4) can be uniquely recovered by resolving the ambiguity T using post-hoc linear methods, such as FastCCA.
P QUANTITATIVE VALIDATION ON HIGH-DIMENSIONAL IMAGE
To provide more direct and quantitative assessment of our theory's disentanglement capabilities in higher-dimensional and more complex settings. we utilize the Multimodal dataset (Daunhawer et al., 2023), which provides paired image and text samples with complete ground-truth latent factors. We validate the identifiability of the shared latent variables (z,g and 2,) that jointly influence both the image and text modalities. For shared factors, its position (abject_xpp) xpos, object_ypppeeeeeeeep yyoss ob jecc_eepsss The remaining factors are treated modality-spediee see Daunhawer et al. (2023) for further details. The table below presents the R² scores for recovering the ground-trute --rttthh shared latent factors from both the image (z.)) and text (z,) factors.
consider the object's 's shape (7 discrete values) and
The image branch (z,)) achieves near cear-perfect recovery (R2 x 0.97), robustly validating our theoretical framework's ability identify and unmix latent factors from complex, high-dimensional inputs.
35

---

Published as a conference paper at ICLR 2026
Modality
Representation
R²² Score (Recovery of z)
Image
0.97 0.05
Text
Zt
0.75 # 0.04
Figure 9: A sample from Multimodal3Dddent (Left). The corresponding text is: 'The top-right of the image shows "tab:orange" colored head", Identifiability Scores (R²) (Right).
Recovery performance for the text representation (z+) is slightly lower (R² 0.75) (Similar observa- tions were also reported by Daunhawer al. (2023).), which we attribute primarily to the violation of the idealized continuous assumptions inherent the text modality-text factors (e.g., color) are often represented as discrete, named values, which conflicts with the continuous assumptions. Overall, these results suggest that our linear identifiability results extend effectively to high- sishddmmesiioal image data.
36


Reviews:
[{"review_id": "yQ4d3NBlxj", "reviewer": "Reviewer_QDaK", "summary": "The paper proposes a Latent Partial Causal Model for multimodal data (image and text). Within this framework, the authors establish identifiability guarantees between the learned multimodal contrastive representations and the underlying generative latent factors. Specifically:\n\n- When the true generative factors lie on a unit hypersphere, the learned representations are identifiable up to a linear transformation.\n\n- When the true generative factors lie in a convex body, the learned representations are identifiable up to permutation, scaling, and shifting of the true generative factors.\n\nThese results imply that multimodal contrastive learning (MMCL), such as CLIP, has the potential to learn disentangled and meaningful representations rather than arbitrary embeddings.\nIn experiments, the authors first validate these theoretical claims through synthetic experiments. They then apply their method to pretrained CLIP embeddings and demonstrate that the resulting representations improve performance across various downstream tasks.", "questions": "**Q1** Can the authors explain more about the aggregated functions h_x and h_t, and how they are used in the proof (Lines 964–977)? Is h_x(m_x, z_x) or h_x(z_x) actually h_x(m_x, z_x, m_t, z_t)? Also, are there smoothness or differentiability assumptions on h required for the proof?\n\n**Q2**. In Corollaries 1 and 2, the learned representations are an invertible linear transformation of the ground-truth vectors, or a permutation after scaling and shifting. **Does this mean the dimensionality of the learned representations must match the dimensionality of the ground-truth latent variables?** If so, this seems impractical in real-world settings (e.g., using a pretrained encoder like CLIP, where we do not know the latent dimensionality). **Can the theorem be generalized when there is a dimensionality mismatch between the learned representations and the ground truth? Does this mismatch affect the practical applicability of the method?**\n\n**Q3.** It appears that the proposed partial causal model (Figure 2) subsumes all the DAG structures shown in Figure 1 and can represent any setting where there is some correlation between $z_x$ and $z_t$. If so, why is this model considered “causal,” and why is it described as partial causal rather than a general latent-variable model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**S1.** The proposed model is interesting and suitable for multimodal settings.\n\n**S2.** The identifiability analysis is solid, and the techniques used in the proofs are interesting.\n\n**S3.** The downstream experiments using disentangled representations span several application areas, and the learned representations demonstrate better qualities compared to the baseline.", "weaknesses": "**W1.** The gap between the theoretical analysis and practical use is not clear. **The identifiability results rely on several assumptions** in Equations (4) and (7) (especially the probability density assumptions), along with assumptions “the latent space is the unit hypersphere” / “the latent space is a hyperrectangle.” **However, it is unclear whether the assumptions hold.** If I understand correctly, these assumptions are made on the ground-truth underlying generative factors. I suggest adding more discussion of what these assumptions mean at a high level (not just mathematically), why they are reasonable, and in which situations they may be violated.\n\n**W2**. There is a lack of a formal definition of disentanglement. The paper highlights that a key practical use of the theory is to obtain disentangled representations using a pretrained multimodal encoder (e.g., CLIP). However, the discussion in Section 5 is high-level and lacks mathematical grounding, which could lead to misunderstandings. For example, the intended disentanglement is among the components of $z_x$ (or $z_t$), not disentanglement between $z$ and the modality-specific noise $m$, correct? I suggest including a precise definition of disentanglement in the paper and discussing it more clearly.\n\n**W3.** The validation of the identifiability results is based on low-dimensional synthetic experiments, and the evaluation of disentangled representations on real-world data is done purely through downstream tasks. I suggest adding a semi-synthetic dataset (e.g., Causal3DIdent[1], Morph-MNIST[2]) that contains higher-dimensional ground-truth latent variables, so disentanglement can be evaluated directly rather than indirectly via downstream performance.\n\n[1] Von Kügelgen, Julius, et al. \"Self-supervised learning with data augmentations provably isolates content from style.\" Advances in neural information processing systems 34 (2021): 16451-16467.\n\n[2] Castro, Daniel C., et al. \"Morpho-MNIST: Quantitative assessment and diagnostics for representation learning.\" Journal of Machine Learning Research 20.178 (2019): 1-29.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gL6gkdiiMz", "reviewer": "Reviewer_Mb1E", "summary": "The paper proposes a latent partial causal model to generalize beyond traditional DAGs, capturing undirected multimodal dependencies and proving identifiability in contrastive learning frameworks like CLIP.", "questions": "Can the proposed method be tested on text domain datasets?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The extension of the causal structure in multimodal learning beyond DAGs is a valuable conceptual exploration.\n- The identifiability proofs are soundly constructed.", "weaknesses": "I don't have enough knowledge to evaluate this paper as I don't work in related domains. For me, it seems the paper did not empirically demonstrate that the learned representations encode genuine causal relations between modalities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fsjbPxz1eL", "reviewer": "Reviewer_9bGu", "summary": "The authors critique the standard assumption that a single Directed Acyclic Graph (DAG) can model the generative process of large-scale multimodal data. They argue that real-world data (e.g., image-text pairs) often result from heterogeneous processes, including conflicting causal directions like text-to-image and image-to-text. To address this, they propose a novel latent partial causal model. This model separates latent variables into latent coupled variables and modality-specific variables. The paper's core theoretical contribution is an \\textbf{identifiability result}: it proves that representations learned by MultiModal Contrastive Learning (MMCL) can recover the true latent coupled variables ($z_x, z_t$) up to a simple transformation.  This theory implies that models like CLIP learn disentangled representations, which are \"mixed\" by a simple transformation. The authors validate this practically by showing that applying post-hoc FastICA (for the hypersphere case) or PCA + FastICA (for the convex body case) to pre-trained CLIP embeddings improves performance on few-shot learning and domain generalization tasks.", "questions": "1. The central modeling innovation is the \"undirected edge\" between $z_x$ and $z_t$. However, the generative parameterizations in Eq. (4) and Eq. (7) both define a directed conditional probability $p(z_t|z_x)$. How does this parameterization formally represent an \"undirected\" relationship, as opposed to a standard directed model $z_x \\rightarrow z_t$? Is the \"undirected\" nature meant to imply that the model is a marginalization of a mixture of DAGs (like those in Figure 1), and that $p(z_t|z_x)$ is just one assumed form for this marginalized relationship?\n\n2. In the proofs of Theorem 4.1 and 4.2 (Appendices D.1 and E.1), a critical step relies on Lemma 2. This lemma states that the optimal solution is achieved when $h_x(m_x, z_x) = h_t(m_t, z_t)$ almost surely. By differentiating, the authors conclude that the optimal encoders must be independent of $m_x$ and $m_t$. This implies that the MMCL loss perfectly enforces the encoders to discard all modality-specific information. Is this a reasonable assumption for the proof, or is it an assumed condition for optimality that may not be achievable? How does the identifiability theory hold if there is minor \"leakage\" of $m_x$ into the representation?\n\n3. The paper offers two practical methods: FastICA for hyperspheres (Corollary 1) and PCA + FastICA for convex bodies (Corollary 2). Practically, how should one choose between these two methods? Is there an empirical test to determine the \"geometry\" of CLIP's latent space? Corollary 2 (convex bodies) implies the learned representation $f_x(x) = Pz_x + c$ is already disentangled. Why does this \"simpler\" case require a more complex solution (PCA + FastICA)? The paper states PCA is needed \"to account for the orthogonal transformation introduced by PCA,\" which is circular.\n\n\n4. The CelebA experiments show compelling semantic disentanglement (\"Smile,\" \"Gender\"). The theory, however, promises \"component-wise\" identifiability. Does this result imply that the true, independent latent factors for CelebA \\emph{are} these high-level semantic attributes? Have you considered quantitatively measuring disentanglement (e.g., MIG, DCI, SAP) by training a linear classifier on the disentangled representations to predict the ground-truth attributes?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper clearly identifies a valid and important weakness in existing causal models for multimodal learning. The observation that web-scale data is a mixture of generative processes (e.g., text-to-image and image-to-text) is sharp and provides a strong foundation for their new model.\n\n2. The connection between the abstract theory (identifiability of $z_x$) and a simple, practical recommendation (use FastICA on CLIP embeddings) is a significant strength. It makes the theory testable and useful.\n\n3. The experiments are very effective. Showing that a simple, off-the-shelf, post-processing step (FastICA) on a pre-trained CLIP model can improve performance across 16+ real-world datasets is a strong validation of the theory. The qualitative disentanglement on CelebA (Figure 3) is also very illustrative.", "weaknesses": "1. The paper's core motivation is replacing DAGs with a model featuring an \"undirected edge\" between $z_x$ and $z_t$. However, the actual mathematical parameterizations (Eq. 4 \\& 7) define a directed conditional probability, $p(z_t|z_x)$. This creates a mismatch between the conceptual claim and the formal execution.\n\n2. The proofs rely on the conclusion that optimal encoders must be \\emph{perfectly} independent of modality-specific variables ($m_x, m_t$). This is an extremely strong assumption, derived from a necessary condition for optimality, and is unlikely to hold perfectly in practice.\n\n3. The paper provides two methods (FastICA and PCA + FastICA) based on two different theories but offers no clear heuristic to determine which is appropriate for a given model or dataset.\n\n4.  The paper claims \"component-wise\" identifiability but validates it with \\emph{qualitative} and \\emph{semantic} disentanglement (e.g., \"Smile\"). It lacks standard \\emph{quantitative} disentanglement metrics (MIG, SAP, etc.) to support this strong technical claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uTqw1Mkei6", "reviewer": "Reviewer_LUiv", "summary": "The paper argues that a single DAG poorly captures large-scale multimodal generative processes and proposes a latent partial causal model with coupled visual/text factors ($z_x$, $z_t$) linked by undirected cross-modal dependencies. Under distributional and invertibility assumptions, it shows MMCL representations are identifiable up to simple transforms with respect to these latent variables, implying component-wise disentanglement potential. Using pretrained CLIP, few-shot and domain-generalization experiments report consistent gains across several real datasets.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-motivated. The author systematically argues that the generation process of large-scale multimodal alignment data has heterogeneity and may have opposite causal directions. The assumption of a single DAG is too limited and needs to be relaxed to describe \"partial causality\", which is consistent with the actual collection process of web scale text image paired data.\n\n- The writing is overall clear. The contribution and limitations of the theory proposed in the article are analyzed.", "weaknesses": "- The identifiability proved here is representation-level (up to simple transforms) rather than causal-effect identifiability.  I suggest the authors explicitly clarify this distinction, since the use of a \"causal graph\" and the term \"partial causal model\" may easily lead readers to misinterpret that causal-effect identification is being addressed.\n- According to the paper’s theoretical analysis, the identifiability and component-wise disentanglement properties of MMCL under the proposed latent partial causal model should not be specific to the visual branch. Yet the empirical evidence focuses almost exclusively on disentangling CLIP’s image features. If the theory is modality-agnostic, we would expect similar benefits when (i) disentangling the text branch and (ii) optimizing truly multimodal objectives.\n- Typos or Mistakes:\n  - Line 169, $\\tau$ should be a temperature hyper-parameter.\n  - In the caption of Table 2, there are two extra ①.\n  - Line 1730, Sec. ??.\n- No additional method or component is introduced to correspond to the theory; the paper relies on existing techniques (PCA and FastICA).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)