On the Value of Cross-Modal Misalignment in Multimodal Representation Learning
Authors: Yichao Cai
Yuhang Liu
Erdun Gao
Tianjiao Jiang
Zhen Zhang
Anton van den Hengel
Javen Qinfeng Shi
Venue: NeurIPS 2025 spotlight ()
Keywords: Representation Learning
Multimodal Learning
Identifiability
Latent Variable Models
Primary area: general_machine_learning
TL;DR: We theoretically formalize real-world misalignment in multimodal learning via latent-variable modeling, showing that learned representations inherently encode semantics invariant to selection and perturbation biases.

Abstract: Multimodal representation learning, exemplified by multimodal contrastive learning (MMCL) using image-text pairs, aims to learn powerful representations by aligning cues across modalities. This approach relies on the core assumption that the exemplar image-text pairs constitute two representations of an identical concept. However, recent research has revealed that real-world datasets often exhibit cross-modal misalignment. There are two distinct viewpoints on how to address this issue: one suggests mitigating the misalignment, and the other leveraging it. We seek here to reconcile these seemingly opposing perspectives, and to provide a practical guide for practitioners. Using latent variable models we thus formalize cross-modal misalignment by introducing two specific mechanisms: Selection bias, where some semantic variables are absent in the text, and perturbation bias, where semantic variables are altered—both leading to misalignment in data pairs. Our theoretical analysis demonstrates that, under mild assumptions, the representations learned by MMCL capture exactly the information related to the subset of the semantic variables invariant to selection and perturbation biases. This provides a unified perspective for understanding misalignment. Based on this, we further offer actionable insights into how misalignment should inform the design of real-world ML systems. We validate our theoretical findings via extensive empirical studies on both synthetic data and real image-text datasets, shedding light on the nuanced impact of cross-modal misalignment on multimodal representation learning.

On the Value of Cross-Modal Misalignment in Multimodal Representation Learning
Yichao Cai'
Yuhang Liu"t
Erdun Gao Tianjiao Jiang
Zhen Zhang
Anton van den Hengel Javen Qinfeng Shi
Australian Institute for Machine Learning The University of Adelaide, SA 5000, Australia
Abstract
Multimodal representation learning, exemplified by multimodal contrastive learn- ing (MMCL) using image-text pairs, aims to learn powerful representations by aligning cues across modalities. This approach relies on the core assumption that the exemplar image-text pairs constitute two representations of an identical con- cept. However, recent research has revealed that peal-wolld datasets often exhibit cross-modal misalignment. There are two distinet viewpoints on how to address this issue: one suggests mitigating the misalignment, and the other leveraging it. We seek here to reconcile these scemingly opposing perspectives, and to provide a practical guide for practitioners. Using latent variable models thus formalize cross-modal misalignment by introducing two specific mechanisms: Selection bias, where some semantic variables are absent in the text, and perturbation bias, where semantic variables are altered-
leading to misalignment in data pairs. Our theoretical analysis demonstrates that, under mild assumptions, the representations learned by MMCL capture exactly the information related to the subset of the semantic variables invariant selection and perturbation biases. This provides a unified perspective for understanding misalignment. Based on this, we further offer actionable insights into how misalignment should inform the design of real-world ML systems. We validate our theoretical findings via extensive empirical studies both synthetic data and real image-text datasets, shedding light on the nuanced impact of cross-modal misalignment on multimodal representation learning."
Introduction
Modern multimodal learning has achieved remarkable success in jointly modeling information from heterogeneous sources such as vision, language, and audio. Multimodal contrastive learning (MMCL) on paired data has emerged as dominant strategy for aligning modalities [55, 27, 72]. This approach models like CLIP, which attempt to identify a common representation space that maximizes the similarity of real image-text pairs while minimizing that of incorrect pairs [55]. One of the assumptions underpinning this approach is that the training pairs are aligned across modalities, meaning that they convey exactly the same semantic information [73, 45). This assumption, though convenient, often violated in real-world scenarios, where multimodal data inherently noisy or imprecisely paired [49, 50]. More critically, text taken from image captions typically only partially describes the paired image content, and often contains descriptive elements that are irrelevant or misleading, a situation referred to cross-modal misalignment. For example, in a Icage- video-text dataset, over 50% of the purportedly aligned clip-caption pairs were found be misaligned [49]. worth noting that the impact of cross-modal misalignment is mainly
exemplified by ission-and-languaal
Equal contribution. 'Coreepondencc to: yuhang.haannn www.iiiddelaaiiee edu. au. "The project page is available https://ylchhaoaa.. wwm/ www.://iiallgnmmnnn.nmmmm gith.b.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

concentrated in the text modality, as text captions, often derived from images, are more susceptible to semantic incompleteness and interpretive variability than the typically consistent visual semantics.
The misalignment discussed above has led to two seemingly opposing viewpoints. On one hand, misalignment is viewed as form of disruption that should be mitigated [40, 64, 72, 8, 65, 30, 76, 7. 47]. For example, cross-modal misalignment can result in "hallucination" in multimodal models [64, 35, 74]. It also provides weak, noisy, and even misleading supervision for multimodal pre- training [72, 82]. On the other hand, an alternative viewpoint suggests that multimodal representations may actually benefit from cross- cross-modal misalignment [71, 50, 6, 29]. For instance, linettuning the representations learned by CLIP through random text augmentation, which deliberately introduces misalignment in style-related information, can lead to more robust representations for zero-shot learning, few-shot learning, and even adversarial attacks [6]. This contrast raises crucial question:
How can we theoretically reconcile these two opposing views on misalignment, and, more importantly, determine which should guide practical applications?
In light of this, we offer a theoretical perspective that not only facilitates the understanding of misalignment but also provides guidance for real-world applications. Specifically, formulate the problem using a latent variable model (LVM), which depicts the underlying generative process of image-text data with misalignment, as shown in Figure 1. In it, the latent space consists of latent semantic variables representing factors common to both modalities (e.g., object shapes and colors), along with modality-specific subspaces that capture unique variations images and text. To make the LVM more adaptable to diverse real- -oold scenarios, we allow for an arbitrary causal structure among the latent semantic variables, providing flexibility in multimodal contexts. To model misalignment, we introduce two mechanisms: selection bias and perturbation bias. Both act on latent semantic information but differ in their effects. Selection bias determines which semantic information preserved in the text. For example, when describing an object, the text might preserve information about its color ("black") but omit its texture details or shape. On the other hand, perturbation bias introduces errors, such misannotating "black" (correct color) to "red" (incorrect color). Moreover, given their heterogeneity, we model the two modalities with separate generative processes.
Building on the proposed LVM (i.e., the generative model), we present a theoretical identifiability analysis within the MMCL framework (i.e., the inference model). We show, under mild assumptions, that the subset of semantic variables unaffected by selection and perturbation biases remain block- identifiable (Defn, 4.h)-that is, only the unaffected subset of semantic variables in the proposed LVM admits an invertible mapping to the representations learned by MMCL. In contrast, the remaining misaligned semantic variables affected by either bias are inherently excluded from the learned repre- sentations, irrespective of the latent causal structure among all semantic variables. This result provides a unified perspective on the seemingly opposing views discussed above. While misalignment can be problematic in tasks that rely on fully preserving semantic information to maximize downstream utility, it may paradoxically become beneficial in scenarios where robustness to distribution shifts is desired. In such cases, cross-modal misalignment naturally acts as a regularizer, encouraging models to focus on stable, invariant semantic factors shared across modalities.
Contributions. Our main contributions are presented below, with an in-depth discussion of related work available in App. B.
We propose an LVM for multimodal data generation that explicitly characterizes cross-modal misalignment through two mechanisms: selection bias and perturbation bias ($ 3).
We establish a general identifiability result, showing that MMCL recovers the subset of semantic variables unaffected by these biases, independent of the underlying latent causal structure ($ 4.1).
We extend this result to two practical scenarios, tasks requiring common representations and those targeting invariant representations, offering actionable insights into how misalignment should inform real-world applications ($ 4.2).
We empirically validate our theoretical findings through extensive experiments on both real- world and synthetic image-text datasets under various cross- modsl misalignment conditions. Additionally, we present a case study on pretrained CLIP models ($ 5).
For the sake of brevity, we use"cross modal misalignment" and "misalignment" interchangeably throughout the paper, the context clearly indicates that referring to misalignment between different modalities.

---

2 Preliminaries: Multimodal Contrastive Learning
Multimodal contrastive learning (MMCL) [55, 27, 81] aims learn joint representations by aligning paired samples from different modalities, i.e., t E T for text and x E X for images, while pushing apart unpaired (negative) samples. In practice. MMCL typically employs two modality-speciffe encoders, i.e., ft(t) for text and (x) for images which project observed paired data into a shared representation space. The learning objective is generally formulated as the following contrastive loss:
K
K
LYMEU (Fa: fi) = 2K
1
Σ
log Σ CH(T
Σ 1=1 Σ e calla(xs)..
log
where (x,, té are sampled paired data, K denotes the number of training pairs, is temperature hyperparameter controlling the concentration of the similarity distribution, and x(..-) denotes a similarity measure. Asymplotically. when K approaches infinity, and with 1 and the similarity function defined as the negative squared Euclidean distance, the objective in Eq. (1) reduces to [13]:
SsymAligaMacad (Ja, ft) (xx,,))>xx. [[(x(x) fc(t))la]
(H + (I.(t))).
(2)
where (-) denotes differential entropy [69, 68]. One of the main advantages of the asymptotic ob- jective in Eq. (2) is its suitability for theoretical analysis [48, 13, 78). At high level, Eq. (2) naturally decomposes into two intuitive terms [69]: the first term encourages minimizing the distance between paired samples, while the second term promotes maximizing the entropy of learned representations. Following prior works [48, 13, 78), we also adopt this objective our theoretical analysis.
A key feature of our work is its focus on the impact of cross-modal misalignment, in contrast to prior studies that assume perfectly aligned paired data. To model this, we introduce novel latent variable model ($ 3) that defines fundamentally different problem setting. Consequently, our theoretical analysis ($ 4) yields insights that diverge significantly from existing results, highlighting how misalignment influences the learned representations in multimodal contrastive learning.
3 Problem Formulation via a Generative Perspective
In this section, we introduce a novel latent variable model (LVM) formalize the underlying generative processes of image-text data, explicitly characterizing cross-modal misalignment ($ 3.1). Building on this LVM, we introduce the technical assumptions underlying image-text pairs under misalignment for subsequent analysis ($ 3.2). See App. A for a complete notation table.
3.1 A Latent Variable Model Characterizing Cross-Modal Misalignment
Figure I illustrates the proposed LVM. In the following, provide detailed explanation of the model from three aspects: the latent space, and the image and text generation processes.
Latent space. We partition the entire latent space 2 into three simply connected, open subspaces, i.e., = S x M2 Mr, where each defines the support of distinct group of latent variables. We denote the latent variables in Z as = (s, my, m.), where my, and m; lie in S, Mg. and Mr, respectively. Below, we describe the characteristics the latent variables s, mg. and m,:
S C R (Semantic variables): Latent variables capturing the semantic content of the data, i.e., information that is interpretable or describable through human knowledge (e.g.. object shape, color). We denote the index set of semantic variables II, : [n.,]* for future reference.
m, E M2 (Image-specific variables): Latent variables capturing non-semantic, image- specific factors that are independent of semantic variables (e.g.. camera noise or background artifacts).
m, M, (Text-specific variables): Latent variables capturing non-semantic, text-specific factors, independent of both semantic variables and image-specine variables m, (e.g., writing style or tone).
'Throughout this paper, we use the notation [d] to represent the set (1...., df for any integer > 1. For simplicity of notation, omit the dimensions of certain variables throughout this work, such mr.
3

---

selection bias
e
p
perturbation bias
color)
= | eizee
A large size cat.
S1g
o (shape, color)
m
Stp
Sig
SCG
m+
A cat that is black color
[colex) I (shape, color)
X
t
cat that is gray color, text under misalignment
imuge
text cuption
imuge
Figure 1: Illustration of the proposed latent variable model (left), with misalignment across modalities modeled via selection and perturbation bias. Image x is generated from semantic variables s and image-specific variables m, via the generative process Oz- The corresponding text t is generated by Jrie) which acts on a biased subset of semantic variables $1, = ( S1,)), influenced by selection bias 0 and perturbation bias p. along with text-specific variables m,- Selection bias omits 5 while perturbation bias marks subset I, S To whose components may be randomly replaced to form S1,- Example image-text pairs (right) illustrate the misalignment induced by these two biases.
key advantage of the proposed latent space structure is its flexibility and applicability to world scenarios. To this end, we depart from prior works that impose somewhat restrictive assumptions on latent structure. Specifically, unlike approaches that enforce certain fixed graphical structures among semantic variables (e.g., assuming content causally determines style) [13], methods based on nonlinear ICA that assume complete (or conditional on an auxiliary variable) independence among latent variables [28, 63], allow for arbitrary dependency structures among semantic variables since the true latent graph structure among semantic variables is often unknown in practice.
Image generation. An image x is generated from latent variables zz mg) via diffeomorphism (i.c., a bijection with smooth inverse function) S x M₁ + *::
Pm.:: 2, = (s. mz), x = (z.)
(3)
Here, Ps and Pm, are prior distributions over subspaces S and M₁. respectively; X defines an observation space that contains all generated images. This generative process formalizes that images fully encapsulate semantics, reflecting the fact that they are both informative and semantically rich.
Text generation. Unlike other multimodal data (e.g.. camera-LiDAd [37]) acquired via sensors, given an image, its corresponding text inherently exhibits flexibility in semantic richness and may also include distortions introduced by humans or captioning models [36, 75], leading to cross- modal misalignment. Here, formalize the root of such misalignment by introducing two types of biases: selection bias and perturbation bias. Accordingly, before formulating the text generation process, we first provide definitions of these types of biases.
Definition 3.1 (Selection bias 0). Let p+ (Ig) denote the set of all non-empty subsets of the index set Ig. defined as P (I,) == P(I.) \ {6). where P(.) denotes the power set. The selection bias is specific non-empty semantic subset II E p (1x). The complement II I lo denotes the omitted semantic subset.
defined as an integer index in the range [2 1), corresponding
Note that each uniquely determines a non non-empty semantic subset I, E p+ (Ix), which defines the semantic information to be expressed in the generated text. It also specifies a text generation mapping Jelo: St x M. - T(9), selected from a class of diffeomorphisms G.. Here, ( denotes an observation space that contains all the observed text ( under selection bias 0.
Definition 3.2 (Perturbation bias p). Let (To = P(TT) (To) denote the set of all proper subsets? of the selected index subset 1g. The perturbation bias is defined as an integer index in the range E [2 dol 1], corresponding to unique subset I, E (Ppop(Ta) subject to perturbation. The complement I To T, denotes the semantic subset that remains unperturbed across modalities.
"Without loss of generality, for a clarification for this choice of order. Again, we fix graded lexicographic order the proper subsets in Ppmp(Lo) throughout the paper.
graded lexicographic order (see Defn. C.I) over P (L.). See Clar. C.2

---

Example 3.1. Let the full semantic index set be 1, = =(shape, size, color), so that the set of non con-empty semantic subsets is P (I,) = {fshape), (size).... {shape, size, color)} Then, selection bias A = 5 corresponds to the fifth subset, i.e., I = (shape, color). with the omitted semantics given by I = (size). The corresponding text-generation mapping In:; uses only the selected semantic variables in Tg to generate text t The set of proper subsets of Tl is P Prrop (Tlg) = {0, (shape), (color)) ). Then, perturbation bias p = 3 corresponds to the subset Ip = [color). J. and its complement within 11g is Ii = (shape}. Under the combined biases 5 and p = 3, only shape is unbiasedly preserved, while color is subject to perturbation. A resulting text might be "A cal that is red color", even though the image shows a large-sized black cat.
Building upon the previously defined selection bias O and perturbation bias p. we now formalize the text generation process, explicitly characterizing the misalignment induced by these two biases. Consider image generated by Eq. (3), with associated semantic variables the index set llg is determined by O. For the corresponding text t we define the latent variables as Ze00) = (6t, : m²), where S1, represents the perturbed semantic variables under perturbation bias p. and m, denotes the text-specific latent variables. The text generation process is then formalized as:
(51, (1g)))))) where
$10= Par, 1s, m.
Pm:: t1
($1, m)); t = Ircon (z
where Pm, denotes the prior distribution over the latent subspace Mr, and gNe) is the diffeomor- phic mapping specified by the selection bias 0. The conditional ls,0,, explicitly characterizes misalignment in text generation arises through perturbations within selected semantic dimensions.
3.2 Model Assumptions for Theoretical Analysis
Based on the proposed LVM, we now present the assumptions underpinning our theoretical analysis:
Assumption 3.1 (Continuous positive densities). The latent variables Ma, and m, continuous and admit strictly positive densities, ie.,, Ps > 0, Pm. > O, and Pm; > O, almost everywhere (a.e.) on their respective supports S, MM and M,-
A Assumption 3.2 (Random perturbations). Given a selection bias O and perturbation bias consider a specific image-text pair (x, ) generated by Eq. (3) and Eq. (4), respectively. The conditional distribution Paa, 8. is defined via randomly sampled perturbation subset A C Ip. such that:
A A~PA PA: Pei, (5t, s, A) = 5 - $16(A) Paalaa (5 SA).
(5)
Here: (i) A is the random subset of semantic indices to be perturbed, with PA defined over P(T,). For every 1 E Tp. there exists at least one subset A C CTp such that 1 E A and PA(A) > 0: (ii) 8(-) denotes the Dirac delta function, enforcing that variables outside A remain unchanged; (iii) Paalsa
a smooth, strictly positive conditional density over SA S, where SA is the domain of S, and for each SA, the support of Asalaa includes non-empty open subset O C SA.
Remark 3.1. Note that Eq. (5) essentially implies that, in each (x, ()),, only a random subset A of T, undergoes perturbations, regardless of the underlying causal structure among latent semantic variables. The rationale is that la- tent semantic variables can only be modified indirectly by altering observations, rather than through direct interven- tion, unless the latent causal structure is fully identified. In the text modality, it occurs through the misassignment of certain content words specific image semantics during the captioning process. Unlike a direct intervention, this cross-modal misalignment does not propagate to descen- dant semantic variables, as illustrated in Figure 2.
modification
direct Intervention
82
do(,s k6)
81
82
FB
91
82
83
(5), 82.88 st)
ST (SL, Sc,
Figure 2: Text modifications affect only the semantics where content words are altered, while direct interventions act on latent semantic variables thereby propa- gating structural changes.
4 Identifiability Results for MMCL under Misalignment
In this section, we theoretically analyze how cross-modal misalignment impacts the identifiability of latent semantic variables in the proposed LVM, within the MMCL framework ($ 4).. Based on
5

---

these results, we further provide practical insights into how misalignment should be addressed in real-world applications ($ 4.2). Detailed proofs of the theoretical results are provided in App. C. To begin, we restate the definition of block-identifiabiliy [68] in the context of our problem setting:
Definition 4.1 bobockkldeniifiabittty..... A subset of latent semantic variables sT. E St with I' S Is. is said to be block-identifier identified by functions fa : X + R!Il and ft : T + l"l if the learned representations 2 E IR l' and 2, E R retain all and only the information contained in $i-. Formally, there exist invertible mappings hy : St- + RF1l s.t. 2g = 12 (8:-) and & = (s.-).
4.1 Identifiability of Latent Semantic Variables under Cross-Modal Misalignment
Building on the definition above, the formalization of the proposed LVM, and assumptions outlined in $ 3, we provide the following identifiability result:
Theorem 4.1 (Identifiability of latent semantic variables). Let (x, t() be image-text pairs drawn from the nata-generuting process described in $ 3, where x is generated uccording to Eq. (3) and ( is generated by Eq. (4). Further, suppose that Asms. 3.1 and 3.2 hold. Denote by St, the subset of semantic variables that annotated without bias in the text, and define its dimension as = [Tol - -ITpl- Let fx : XX + 10. and ft : T - (0,1)" be sufficiently flexible, smooth functions. Then, minimizing the loss LsyuniisMMsss in Eq. (2) over samples (x, t.") guarantees that fx and ft block-identify the semantic variables Stş in the sense of Defin. 4.1.
Remark 4.1. Thm. 4.1 formally establishes that, in the presence of cross-modal misalignment, the unbiased semantic variables St; that are shared across modalities can be effectively recovered, up to a block-wise indeterminacy, by minimizing the MMCL objective. contrast, components that are misaligned, specifically 8t, and Sig. are entirely excluded from the learned representations. We emphasize that this result holds regardless of any underlying latent graph structure among semantic variables. At its core, this result highlights the model's capacity to focus exclusively on the aligned semantic aspects of the data. Furthermore, modality-speciffc. non-semantic factors, i.c., my and m,, consistently discarded throughout the learning process. This further underscores the model's ability to extract meaningful, cross-modal semantic information while filtering out semantically uninformative noise-induced components. A detailed proof of Thm. 4.1 is provided in App. C.2.
4.2 Insights into Cross-Modal Misalignment for Practice
The above result is general and not limited to any specific problem context. We now consider two real-world application scenarios: (i) pretraining with large-scale data and (ii) invariant representation learning. The former aims to capture comprehensive semantic information to support a wide range of downstream tasks, while the latter focuses on learning robust representations for out-of-distribution (OOD) generalization. In what follows, we present corollaries and insights for both scenarios.
Corollary 4.1 (Identifiability of full latent semantic variables). Assume that Asms. 3.1 and 3.2 hold. Let the selection bias be A = 2n 1 and the perturbation bias be p == 1, such that the full set of semantic variables Is is selected, and the perturbable semantic subset is trivial, i.e., Ip = 0. Then, all semantic variables s are block-identified via smooth functions fa X (0, 1) and ft: T >> (0, 1)"², when minimizing LssinAiigMMudine
Insight 4.1 (MMCL pretraining on aurge-ccale data). Large-scale multimodal datasets (e.g., COCO [38], Conceptual Captions [62], LAION-5B [61]] [) often exhibit varying caption quality. Our analysis indicates that omitted or perturbed semantics are irretrievably lost in the learned representations, regardless of dataset size, although scale may mitigate sporadic misalignment by averaging its effects. Preserving a breadth of relevant semantic details is therefore crucial when pretraining foundation models, whose primary goal is to support diverse downstream tasks. As noted in Cor. 4.1, achieving this requires detailed and consistent annotation of image semanties. Consequently, improved caption control [36, 16, 14] is essential to avoid blind spots in semantic coverage.
Corollary 4.2 (Identifiability of invariant semantic variables). Assume that Asms. 3.1 and 3.2 hold. Consider an OOD setting in which a subset of semantic variables, Iino C lss, remains invariant between training and testing environments, while the remaining semantic variables, Tpar = II \ Tines undergo distribution shifts. the union of omitted and perturbable semantic variables under selection bias 0 and perturbation bias p coincides with the environment sensitive subset, i.e., Bvor L5 U Ulpp

---

then the invariant semantic variables S are block-identified via smooth functions fo X. - (0, 1) 1)Blool and ft T()) + (0, 1) by minimizing LyydtiggMMuaa
Insight 4.2 (Invariant representation learning). In tasks requiring robust OOD performance (e.g.- domain generalization) [57, 21, semantic variables that are vulnerable to distribution shifts can undermine generalization. As noted in Cor. 4.2, misalignment may, counterintutivelly enhance robustness by selectively omitting or perturbing these vulnerable variables. This suggests that MMCL may offer novel perspective on invariant representation learning [54, 32, 15], as auditing and curating text more precise and interpretable, since language is distilled from human knowledge.
Experiments
We conduct extensive experiments to validate our theoretical results, including numerical simulations ($ 5.1), a real-world image-text dataset with independent semantic variables ($ 5.2), a synthetic dataset with dependent semantic variables. ($ 5.3), and a case study with OpenCLIP models ($ 5.4).
5.1 Numerical Simulation
Experimental setup. We simulate data following the generative process in $ 3. Semantic variables
~ (0, E0,) (dim. 10) and codalltt-speccfff variables m (0, Em.) ). my (O, Em, (dim. 5) encode potential causal dependencies via their respective covariances Σ()- To study cross-modal misalignment, we independently vary: (i) Selection bias: controlling 1g = {1]..... [10], and (ii)
Perturbation bias: defining 1p = Ó, ..... [9], with Gaussian noise added to cach E .p with probability 0.75. In isolation, we fix Ip = 0 when varying Lg, and 14 = [10] when varying 4p. Image and text data generated using randomly initialized invertible MLPs gs and gra. We train two modality-spechie MLP encoders for 100,000 steps using the LsymAlignMastt loss in Eq. (2), with embedding dimension matching the number of unbiased semantic components. See App. D.1 for further details.
We conduct two experiments. (i) Semantic identification: A lightweight MLP regressor is trained to recover each ground-truth semantic component from MMCL learned representations; we report mean R2 over three seeds. (ii) Downstream performance: We evaluate learned representations on four tasks with targets y/, y2, V/s V1. defined nonlinear functions over semantic subsets [3]. [5], [7], [9]. We additionally binarize 3/2 for classification, and apply a heavy-tailed shift to dimensions {9, 10} of semantic variables to evaluate OOD generalization. See App. D.1 for full task design.
Identification of semantics. The results in Figure show that, under the independent latent variable scenario, unbiased semantic variables are clearly block-identified (R² ~ 1), whereas misaligned semantics due to selection bias are effectively discarded (R² = 0). In the dependent latent variable scenario, some misaligned semanties become partially predictable, reflecting inherent mutual predictability among strongly dependent variables [68, 78]. Modality-specinf variables are consistently omitted from the representations across all settings. Similar effects are observed under perturbation bias, as illustrated in Figure 11. Notably, although our theoretical results hold only up to invertible mappings, simple linear regression already achieves high Re scores, as demonstrated in Figure 12. These findings consolidate the identifiability results in Thmm 4.1. Additional analyses on misassigned encoding dimensions and combined bias effects, are provided in App. D.2.
G.0 00 m.-50 0.0
00 866 6.0 Sy 10
6.0
60 60 50 59 03
00 68
50 00
0.0 09 00
00
560000 66
00 02 56 6
60
00 00 50 GO 9.5
00 0. 5 5.6 &.
TO
20
10
1.0
10
DO
2.
L0
10 10
DO
6.0
1.0
Sa 0. 0O L.
1.6
D.0 0.0 0
1.
1.0
0.1
1.0
L.
D1 9.2
1.
LO
1.0
5..60 na
1.0
0.0 0.0 0
2.0
12
2.2
1.0
0.2 94
1.0
L.0
1.
1.
1.0 2.0
B2 0.5 2
L.
2.0 2.0
56.0.0 0
0.3
2.0 2.0
0.0 0.0
30
LA
13
0.1 02
61 0-2
0.5
1.0
0.0 0.
5A 5
02
0.5
52 0.2
6.5
0.5
1.0
Su 0. 0.
0 0.0
1.5
0.0 0.0
3.0 C.0 0.5
1.0 3.0
0.2 0.1 9:3
2. 0.6 0.0 1.0
3.0
6. 0.1
8:3
84
0.
0.6 0
2.0 1.0
Sue 5.0
0.3
0.0 O.
0.0 D SJD U. 3.0 0.0 E.0
U.L 0.2
0.2
9.4
0.5 0.8
2.0
BLL 0.2
0.4
0.3 0.3
[2] 1811 (5) 16) 170 181 OF
1 121
(4) [5) (6) (71 181 190)
3 [21 (31
(4) (5) [61 TP1 191E10T
1 [2]
13) [4] [9] [6] [-1 [8] [ON10
selection bias, 2
selection bias, &;
selection bias, 2.
selection blas,
Figure 3: Mean R² scores under selection bias settings. From left to right: predictions based on 2, and 2, with independent latent semanties, followed by those with dependent latent semantics.
7

---

Regression, Independent
Regression, Dependent
Classificattonn, Indspendent
Classification, Dependent
1.0
1.0
1,0
0.8
0.8
5.5
0.6
3.6
02
0.2
185h
3/2
60
(4](5) (811111
0.0
(1)12) (8119555
(1)121(5) (11666661 (91]00 selection blas, 2,
(1) [4T (8T00
selection bias, &
selection bias, &
selection bias, 2,
Figure 4: Downstream performance of pretrained 2 under selection bias. Left: in-distribution (ID) regression performance. Right: ID classification and outoof-distribution (OOD) generalization.
Downstream performance. As shown in Figure 4, retaining more semantic information during pretraining significantly enhances in-distribution regression performance, consistent with Cor. 4.1. Conversely, under distribution shift scenarios, accurately identifying invariant semantic variables essential for robust cutt-f-diistrbutioo generalization. In this setting, introducing appropriate selection perturbation biases effectively removes variables sensitive distribution shift, supporting the result in Cor. 4.2. Additional results on the effects of perturbation bias are provided in App. D.3.
5.2 MPI3D-Complex: Real-World Dataset with Factorized Latent Variables
Experimental setup. The MPISD>CCompex dataset [19] con- tains real-world images annotated with seven mutually inde- pendent, discrete ground-truth factors: object color (color), shape (shape), size (size), camera height (cam.) ), background color (back.) ), horizontal position (hori.) .), and vertical posi- tion (vert.). We treat hori. and vert. as image-specific Figure 5: MPI3D-Comp. samples. factors, and the remaining five as semantic variables. See Fig- ure 5 for example images from the HPI3D-Comple semantics using content-wond word mappings and three manually designed templates. To simulate mis- alignment, we introduce: (i) selection bias by progressively increasing the subset To of included semantic indices: D: color, @: [color, shape). 3: (color, shape, size), D [color, shape, size, cam. ). a all five; (ii) perturbation bias by fixing To (color, shape, size, cam., back. } and varying Ip from D: 0 to 3: shape, size, cam., back. in reverse order, replacing selected values with alternatives at 90% probability.
dataset. Text is generated from the ground-trui truth
Training is performed using the loss LAMAL defined in Eq. (l), following the formulation in [13]. Performance is evaluated using the average Matthews Correlation Coefficient (MCC) across three random seeds, based on the prediction of all image latent variables from the learned image and text representations using linear and nonlinear MLP decoders, respectively. Full details on dataset attributes, text generation, bias configurations, and architectures are provided in App. E.I.
Results. Figure 6 presents the nonlinear MCC results with learned representations. The findings demonstrate that, even with discrete latent variables, misaligned semantic variables across modalities, whether due selection or perturbation biases, are consistently excluded from the representations (MCC-a = 0). In contrast, unbiased semantics are well recovered, with MCC scores predominantly approaching and all values 2 0.8, reinforcing our theoretical findings. Further investigations into MCC using linear classifiers and ablation studies on encoder dimensionality are provided in App. E.2.
color 1.00
1.00
1.00
1.00
100
1.00
1.00
1.00
1.00
0.99
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
0.99
shape-0.02 0.00
1:00
0.98
0.81
0.91
0.03
1.00
0.89
0.77
0.90
0.00
1:00
0.98
0.82
0.93
0.00
1.00
0.90
0.77
0.92
MCC
09
0.96
0.96
cam. 0.02
0.00
0.04
1:00
100
0.02
0.00
0.02
1.00
1.00
0.00
0.00
0.00
0000
1.00
0.00
0.00
0.00
1.00
1,00
back.
0.01
0.01
0.00
100
0.01
0.00
0.01
0.00
1:00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.01
0.00
1.00
hori.-
0.00
0.00
0.00
0.00
0.00
0.00
vert. 0.00
0.00
0.00
0.00
0.00
0.00
0.00 0.00
0.00 0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
selection blases, i,
perturbation blases, 2,
selection blases, 2.
perturbation blases. à
Figure 6: Mean MCC scores under misalignment settings. Left to right: Image features 2g with selection and perturbation bias settings, text features 2, under the same bias settings.
5.3 Causal3DIdent: Semi-Synthetic Dataset with Structured Causal Latent Variables

---

Experimental setup. We further conduct experiments on the Causal3Dident dataset [83, 68. 13. 78]. symi-synthetic dataset allowing explicitly defined causal relations among la- tent variables. Images are generated from 10 latent variables: 3 discrete (shape, x_pos, y-pos) and 7 continuous (color, s s_pos, s_ s_color, b_color, alpha, beta, gamma). We treat Figure 7: Causal3IDleent samples. the rotation angles (alpha, beta, gamma) image-specific, and the remaining as semantic variables following a causal graph among latent variables (Figure 17). See Figure 7 for example images from the dataset. For text, discretize color- attributes (color, s c_coo, b_color), retain continuous s_pos and simulate partial loss in spotlight infor- mation when generating text. Text descriptions rendered using five templates, based latent semantic variables under selection and perturbation bias. Specifically, selection bias settings (Tl) vary from (shape) to the set of all seven semantic indices, with I = 0; perturbation bias settings progressively perturb subsets T from 0 to (x_pos, y-pos, s_pos, color, s_ color, b_ b_color) in reverse order, with full semantic selection.
Training paradigm mirrors that of MPI3D-Complex, We evaluate using R² for continuous and MCC for discrete latent variables, averaged over three seeds. See App. F.1 for full experimental details.
Results. Figure 8 presents the prediction performance of nonlinear MLP classifier regressor trained on learned image representations. We observe that unbiased semantic variables shared across modalities, whether continuous (e.g., s_pos) or discrete (e.g., shape, x_pos, y-pos), are reliably recovered by MMCL across all settings, with predictive performance approaching perfect (R2 s 1). For semantic variables that continuous in the image modality but discretized in the text modality, prediction performance shows some degradation (e.g.. color in selection setting c or perturbation setting 2), yet still achieves relatively high R² scores. emage-speciffe
variables are consistently excluded from the learned representations, as indicated by π² = O. Likewise, semantic variables omitted due selection perturbation bias are generally discarded. For instance, in selection setting ! or perturbation setting ②. only shape remains predictable. When factors such as x x_pos or y-pos included, other dependent semantic variables, such as color, become partially predictable (e.g.. in selection setting 2). Similarly, identification of s_pos enhances predictability of s s_colol or and color,, reflecting the latent causal structure. Overall, the results on the Causal3DIdent dataset further support our theoretical findings. Analyses of the text representations are provided in App. F.2.
alpha-
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.0
alpha- 0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.0
beta
0.00
0.00
0.00
beta.
0.00
0.00
0.00
0.00
gamma 0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.8
gamma 0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.8
MCC
shape 1.00
0.91
0.62
0.84
0.89
0.96
shape 1:00
0.98
0.92
0.86
0.85
0.93
0.90
x_pos:
0.04
1.00
1.00
1.00
1.00
1.00
1000
0.6
pos
1.00
1.00
1.00
1.00
1.00
1.00
y_pos 0.05
0.08
1.00
1.00
1.00
1:00
1.00
y_pos 0.05
0.05
1.00
1.00
1.00
1.00
1.00
0.89
0.4
s_pos-
0.00
s_color: 0.00
color- 0.00
0:40
0.00
D.79
0.00
0.46
0.79
0.39
0.84
0.86
0:69
0
0. 84
0.2
s_color- 0.00
color- 0.00
0040
0.00
0.78
0.00
0.48
0.78
0.84
0:48
0.87
0 69
0.70
0.96
0.2
b_color 0.00
0.00
0.00
0.61
b cocolr. 0.00
0.00
0.00
0.61
0.62
0.65
-0.0
0.0
1
selections settings
perturbations settings
Figure 8: Predicting semantic variables under misalignment using image features. R² is reported for continuous factors and MCC for discrete factors. The predictions pattern align with our theoretical results, though extra dimensions may be predictable due to statistical correlations.
5.4 Case Study: Zero-Shot Evaluation of OpenCLIP Model Representations
We further validate our theoretical findings through a comprehensive zero-shot evaluation case study on OpenCLIP models [24], foundation MMCL model pretrained on the LAION-400M dataset [61]. To achieve this, we develop a structured taxonomy comprising 146 visual concepts, organized into 15 distinct concept groups with corresponding abbreviations for clarity in plots and analyses: Animal, Clothing, Color, Food, Object, Role (i.c., human roles), Scene, Vehicle, Weather, Texture, POV (i.c., point of view of the photo). Emot. (i.c.. emotion mental state), Postproc. (i.c.. prot-processing). Trait (i.c., trait judgment by appearance), and Stere. (i.c., stereotypes). For each concept, we collect dataset of up to 200 images from Flickr by prompting with concept names, available under CC licenses. See the concept taxonomy and data collection details in App. G.I.
hhttps://www flickr.com com/

---

Concept coverage in LAION-400M captions. We begin by analyzing the concept coverage of LAION-400M captions, measuring the average coverage rate, defined the percentage captions that explicitly mention each concept or its synonyms, averaged by concept group. As shown in Figure 9, concept groups display substantial variation in average coverage: common concepts exhibit relatively high coverage (e.g., Object: 1.63%, Color: 2. 2.16%), while some concepts are rarely mentioned (e.g., S 0.0003%, Postproc.: 0.0118%). Importantly, due to the large scale of the dataset, no concept group exhibited 0% coverage, indicating that complete omission by selection bias unlikely. Thus, the coverage rate serves as an approximate indicator of selection bias across concept groups.
Zero-shot evaluation of OpenCLIP models. Figure 10 shows the zero-shot FI performance of OpenCLIP across concept groups. Consistent with theoretical findings, OpenCLIP models, regardless of scale, perform well on frequently captioned groups (e.g., Animal, Object), but their performance declines sharply for under-captioned groups (e.g., Trait, Emot.). Concept-level confusion matrices further reveal systematic misclassifications particularly within low-coverage groups (see App. G.3). Notably, omitting captions for sensitive concepts such Stere. may be intentional to avoid biased knowledge. In contrast, under-captioned yet valuable concepts like Texture and Emot. should be learned, suggesting need for more targeted captioning strategies.
Distribution
Medlan
Moan value
3
2.1022
2
1.6288
1.17447
0.8100
0.8114
0,5363
0.5284
0.4882
0.1470
0.0835
0.0003
Color
Object
Clothing
Food
Vebicle
Somn Woather Animal
Role
Textaree
Traic
POV
Emot. Postproc. Stare.
Figure 9: Concept coverage by group in LAION-400M captions. See App. G.2 for intra-group details.
100
$7.5
20,111.0
VIT-B-32
21.
V4T-L-4
Macroo F1 Score (%)
76375.8
40
36.8
37.1
29.01.
20
Object
Postproc.
Figure 10: Zero-shot F1 score of OpenCLIP model trained on LAION in each concept group.
6 Conclusion
In this work, we present a formal analysis of cross-modal misalignment multimodal data pairs within the MMCL framework, examining its impact on the learned representations. We demonstrate that contrastive multimodal encoders retain only the unbiased shared semantic variables, while discarding misaligned latent variables. When image-text pairs exhibit misalignment due to selection or perturbation biases, the joint embedding prioritizes consistent content, while omitting altered or missing aspects. This trade-off fundamental: perfectly aligned text captions preserve rich semantic detail, whereas selective or biased text can enhance domain invariance by filtering out distribution-senssine sensstive factors. Our experiments, conducted across simulations and image-text datasets, empirically validate these theoretical findings. These insights highlight the need for multimodal learn- ing frameworks that mitigate misalignment or leverage beneficial biases to enhance representation learning in world settings. More discussion of these implications is provided in App. I.
10

---

Acknowledgments and Disclosure of Funding
The authors gratefully acknowledge support from the Responsible AI Research (RAIR) Centre (J. Q. Shi and Z. Zhang), the Centre for Augmented Reasoning (CAR) (Y. Liu and E. Gao), and the Commonwealth Bank of Australia through the CommonBank Centre for Foundational AI Research (A. van den Hengel). This support was essential to the completion of the research presented in this publication. The authors also thank the anonymous reviewers for their constructive feedback.
References
[1] J.BB. Alayrac, A. Recasens, R. Schneider, R. Arandjelović, Ramapuram, J. D. Fauw, L., Smaira, S. Diele- man, and Zisserman. Self-supervised multimodal versatile networks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 6666/34957722 3495727.
2020. Curran Associates Inc. https://dl. acm. httg/doi/10... /tto://...../////////1
[2] M. Arjovsky, I.. Bottou, Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint (eXiv:/907.0289) 2019. attps://arxie org/
[3] L. W. Barsalou. Grounded cognition. Annu. Rev. Psychol., 59(1):617 645, 2008, https://www2. psychology www.lloggg.......o...oooooooo... wttp///accltyy/ooolnggoooooo
Baaeeee/aarallluu pdf.
E. M. Bender and B. Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics. 6:587-604 2018. https: sttpss//accannthollls
www//og-////.o 1041/.
[5] V. Bogachev. Measure Theory. Number Volume in Measure Theory. Springer Berlin Heidelberg, 2007.
https://books: google com, aw/books Cid= C5mmTTC.
[6] Y. Cai, Liu, 7. Zhang, and J. Q Shi. Clap: Isolating content from style through contrastive learning with augmented prompts. In Eumopan Conference Computer Vision, pages 130-147. Springer, 2025.
https://doi org/10. 19000000777770 001-000--7775 721677264-4444 -_8.
[7] A. Chartsias, G. Papanastasiou, C. Wang. S. Semple, D. E. Newby, R. Dharmakumar, and S. A. Tsaftaris. Disentangle, align and fuse for multimodal and semi-supervised image segmentation. IEEE transactions on medical imaging, 40(3):881 1792, 2020, https: http:///oooooo/////// 1ww.91199TTMM /MM. 2020 3036584.
[8] C. Chen, J.
Qi. X. Liu, K. Bin, R. Fu, X. Hu, and P. Zhong. Weakly misalignment- adaptive feature aligmment for uavs-bos bass- multimodal object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26836-26845, 2024. httpa://doi. http:///ooo..r//11/// 1109/ CVPR52733. 2024. 025334.
[9] B. O. Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. http: / //ww. blender org.
[10] T. M. Cover. Elements of information theory. John Wiley & Sons, 2006. htttps//doi.oorg111 047174882X.
[11] K. Crawford. The atlas of Al: Power, politics, and the planetary costs of artificial intelligence. Yale University Press, 2021. https: bttpss:/bbokss google.
[12] G. Darmois. Analyse des liaisons de probabilité. In Proc. Int. Stat. Conferences 1947, page 231, 1951. attps://doi. org/10 10. 2307 0000//10110 11011511.
[13] I. Daunhawer, A. Bizcul, E. Palumbo, A. Marx, and E. Vogt. Identifiability results for multimodal contrastive learning. In The Eleventh International Conference on Learning Representations, 2022. https://openrevvew nwt/ meet/forrmmiddD 2ku
[14] N. Ding, C. Deng, M. Tan, Q Du, Ge, and Q. Wu. Image captioning with controllable and adaptive length levels. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, https://ieeexploor ieee. .eeee..rg/oooeeeeeee pmgg/dccumen/1101100
[15] C. Eastwood, A. Robey, S. Singh, J. Von Kügelgen, H. Hassani, G. J. Pappas, and B. Schölkopf. Probable domain generalization via quantile risk minimization. Advances in Neural Information Processing Systems, acm. prg/doi/10 wtt.///...........eeeeeeeeeeee/e 5555/3600270. 3601531.
1117740-11358. 2022. sstps://dd)
[16] H. Fang. S. Gupta, F. landola, R. K. Srivastava, L. Deng. P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, et al. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1473- 1482, 2015. https: //doi. org/ / 10. 1109//109CC CCVP.. 2015. 2015.7298754 7298754.
11

---

[17] I. Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411 -337, 2020. httpa:// springer cww/article//1..ooooo 100000000000000000000000000 020-00000-099-222- 09539-2.
[18] E. Gao, H. Bondell, S. Huang. and Gong. Domain generalization content factors isolation: a two- level latent variable modeling approach. Machine Learning, 114(88), 2025. https: //linkk http://innnk..iing......
com/article/ carrtcce11000 10. 1007/ s00994- 0240004---0000 0204-06777-40 -.
[19] M. W. Gondal, M. Wuthrich, D. Miladinovie, F. Locatello. M. Breidt, V. Volchkov, J. Akpo, O. Bachem, B. Schölkopf, and S. Bauer. On the transfer of inductive bias from simulation the real world: new disentanglement dataset. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bau Buc, E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019. https: //dl: acm. org/ htt///oi/100//// /0. 3555/3454282 34556977.
[20] Y. Gu and G. Xu. Partial identifiability of restricted latent class models. The Annals of Statistics, 48(4):2082-2107, 2020. https: //tps::///di.... org/10..//.. -914/19-001188.
[21] A. Hatcher. Algebraic Topology. Algebraic Topology. Cambridge University Press, 2002. https: //books. google.. com. au/books" Sooosss/ddDGGGGIIIIVVC
[22] K. He, X. Chen, S. Xie. Y. Li, P. Dollár, and R. Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16009-1000,, 2022. https:/ /doi: htt////..orr1...1......../ httppp///aaa..oo//////////////// wttp///...11////////52500
01553..
[23] A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-439, 1999. https://doi httpp://ddi.oo//1...../ wttg///..///////...e 1016/50893----- 6080 (98) 00100004400000
[24] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, Dave, V. Shankar, H. Namkoong, Miller, Hajishirzi, A. Farhadi, and L., Schmidt. Openclip, July 2021. https: / //doi. //doi.org/1e 10. 5281/zenodo 5143773.
[25] L. Itti and C. Koch. Computational modelling of visual attention. Nature reviews neuroscience, 2(3); 194- 203, 2001. https://www http://///w...tureeem..me natura. com /tmm/arrccceee/5055000.
[26] E. T. Jaynes. On the rationale of maximum-entropy methods. Proceedings of the IEEE, 70(9):939-952,
1982. https://doi. http:///oo.o///../// 1109/PROC.1990 12425.
[27] C. Jia, Y. Yang. Y. Xia. Y.TT. Chen, Z. Parekh, H. Pham, Q. Le, Y.HH. Sung. Z. Li, and T. Duerig, Scaling up visual and vision-language representation learning with noisy supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021. https://pproceedigg mlr press/ vv39// jia21b.html
[28] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen, Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217 PMLR, 2020. https: htttps::/ppoceeeingg. mlr. press/ v108/ /kh khem200
[29] B. Kim, Y. Jo, J. Kim, and S. Kim. Misalign, contrast then distill: Rethinking misalignments language- image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages (6722 October 2023. https: //doi. erg/10 10. www0///C/111000... 2023.. .02442.
[30] B. Kim, J. Kim, Y. Jo, and S. H. Kim. Expediting contrastive language-image pretraining via self-distilled encoders. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2732 2740, 2024. sttps:///ol. org/ 10. 1609/ aaai. v3813. 28052.
[31] L. Kong. S. Xie, W. Yao, Y. Zheng, G. Chen, P. Stojanov, V. Akinwande, and K. Zhang, Partial identifiability for domain adaptation. arXiv preprint arXiv:2306.066. 10, 2023. https://proceediing. mlr .rress/ v162/kong22a.
[32] D. Krueger, E. Caballero, J. Jacobsen, A. Zhang, Binas, D. Zhang, R. Le Priol, and A. Courville. Out-of-distribution generalization via risk extrapolation (rex). In International conference on machine learning, pages 5815-5826. PMLR, 2021. https://proceeding mlr. w.x.ppess/vvle 1t11919//// /rueger2aa krueger21a, pdf.
[33] S. LLchapellee, Rodriguez, Y. Sharma, K. Everett, R. Le Priol, A. Lacoste, and S. Lacoste- Juliin. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference on Causal Learning and Reasoning, pages 428 484, PMLR, 2022. https: : http://pppeeviieeeeeee evieerreeeev.... wtt.:ee//toooooo forum?? F.dddHsFFekd_oo.
12

---

[34] S. Lavoie. P. Kirichenko, M. Ibrahim. M. Assran, A. G. Wilson, A. Courville, and N. Ballas. Modeling caption diversity in contrastive vision-language pretraining. Proceedings of Machine Learning Research, 235:26070-26088 2024. httpp: https://proceedings mlr. prwss/ vv2///23/////// 1235//3131ie2225 htmll.
[35] S. LLnng H. Zhang. G. Chen, X. Li, S. Lu, C. Miao, and L. Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 13872-13882, 2024. https://githen
[36] Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. In International conference on machine leurning. PMLR, 2022. setpss//proceediin
mlr. ppwss/v16e//si 2ttii/2nnn22i 12..222...
[37] T. Liang, H. Xie, K. Yo, Z. Xia, Z. Lin, Y. Wang. T. Tang. B. Wang, and Z. Tang. Bevfusion: A simple and robust lidar-camera fusion framework. and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 10421-10434. Curran Associates, Inc., 2022. https: // http///ppperreeeeeeeeeeee net/Eormmiid=zzDrrre
Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho,
[38] T.YY. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt V 13, pages 740-755. Springer, 2014. https:
// /linkk cot.geer.commeeooee http:/igrr..oo/aaaaareeeeeeeeeee /00. 1007 10007/978-331999 100--339--1020220 0000001428.8...
[39] P. Lippe. S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and S. Gavves, CITRIS: Causal identifiability from temporal intervened sequences. In K. Chaudhuri, S. Jegelka, L. Song. C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 13557-13603. PMLR, 17-23 Jul 2022. https: html.
/ ippoceediing. mlr. www..l................... httpa//v62/1ipppppe
[40] Y. Liu, K. Zhang, X. Ren, Y. Huang, J. Jin. Y. Qin. R. Su, R. Xu, Y. Yu. and W. Zhang. Alignrec: Aligning and training in multimodal recommendations. In Proceedings the 33rd ACM International Conference on Information and Knowledge Management, pages 1503-1512, 2024. https:/ //doi.////iii 000/10.1100 1145/ 3627673. 3667966..
[41] Y. Liu, Z. Zhang, D. Gong, M. Gong, B. Huang, A. v. d. Hengel, K. Zhang, and J. Q. Shi. Identifying weight- variant latent causal models. arXiv preprint arXiv:2208. 14153, 2022. https://arras orw.oogg/..............o... 2208 14153.
[42] Y. Liu, Z. Zhang, D. Gong, M. Gong, B. Huang, A. v. d. Hengel, K. Zhang, and J. Q Shi. Identifiable latent neural causal models. arXiv preprint 2rXiv:2403.1571, 2024. https: //arziv..////////// wttpp//////eeeeoeooe/ p/dd//240.155 15711.
[43] Y. Liu, Z. Zhang, D. Gong. M. Gong, B. Huang. A. van den Hengel, K. Zhang, and J. Q. Shi. Identifiable latent polynomial causal models through the lens of change. In The Tweiftth International Conference on Learning Representations, 2024. https:/ hatpp:///pppreveeeee oppnnevvieee net htt///orrrddiaat000000
[44] Y. Liu, Z. Zhang, D. Gong, M. Gong, B. Huang, A. van den Hengel, K. Zhang, and J. Q. Shi. Latent covariate shift: Unlocking partial identifiahility for multi-source domain adaptation. Transactions on Machine Learning Research, 2025. https: hetps://openreviey
[45] Y. Liu, Z. Zhang, D. Gong, B. Huang, M. Gong, A. d. Hengel, K. Zhang, and J. Q. Shi. Revealing multimodal contrastive representation learning through latent partial causal models. arXiv preprint 24Xiv:2402.06223, 2024. https://arre org/ abs/ 222...06222....
[46] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 11144122. PMLR, 2019. https://proceediing aemiiiinnns.m........mmmmmr mmr. mlr.press/ v97/1ocatello19a. html.
[47] F. Lx, X. Chen, Y. Huang, L. Duan, and G. Lin. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2554-2562, 2021. https:/ //doi. otg/10. 1109/ CVPR46437 2021. 002558.
[48] Q Lyu. X. Fu, W. Wang, and S. Lu. Understanding latent correlation- based multiview learning and self-supevisioo: An identifiability perspective. In International Conference on Learning Representations. 2022.
13

---

[49] A. Miech. D. Zhukov. J.-B. Alayrae, M. Tapaswi, 1. Laptev, and J. Sivic. Howtol00m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2630-2640, 2019. https:/ htops://www... /www dii ens. fr/ willow/researcchhoottooo
[50] R. Nakada, H. I. Gulluk, Z. Deng, W. Ji. J. Zou, and L. Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference on Artificial Intelligence and Statistics, pages 4348-4380. PMIRR 2023. https://proceeding: mlr. html.
[51] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 107.:0807,3374,, 2018.
wwt///r////////eeeeeeee /1077 111118077777 03748.
[52] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. HAZIZA,
F. Massa, El-Nouby, M. Assran, Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. httpe://openrewe
net//oorumiid-a6SSttae
[53] J. Pearl. Causality. Cambridge university press, 2009.
cs. www.cla.a.....omm..m.........ala edu/B0OK-20 -
[54] J. Peters, P. Bühlmann, and N. Meinshausen, Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5)997-1012, 2016. https://doi. org/10. 1111/rssb. httt://rabb........... 12167.
[55] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 3748-8763. PMLR, 2021. attps://proceedding mlr press/ v 1139/aadford21e
[56] P. Reizinger, S. Guo, F. Huszár, B. Schölkopf, and W. Brendel, Identifiable exchangeable mechanisms for causal structure and representation learning. The Thirteenth International Conference on Learning Representations, 2025. https://cpenrevieee net/ fowmmmddd400mm41rrrrrr
[57] M. Rojas-Carulla, B. Schölkopf, R. Turner, and J. Peters. Invariant models for causal transfer learning. Journal of Machine Learning Research, 19(3)):1-34, 2018. https://jmlr http:/ppprr/v/111--eeeee html.
[58] Rudin. Principles of Mathematical Analysis. International series pure and applied mathematics. McGraw-HHill 1976. https::/bbole http://ooooo.ollleee.e// com. au/books htt/bbokssidd=kwore iiddkkggPAAAAAAA
[59] M. Ruzhansky and M. Sugimoto. On global inversion of homogencous maps. Bulletin of Mathematical Sciences, 5:13- 18, 2015. https://doi org/ 1007 1
[60] B. Schölkopf, Locatello, S. Bauer, N. R. Ke, N. Kalchhrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEFE, 109(5))612-634, 2021. https://dol org//0........../ 1109/ JPROC.2021.00 3058994.
[61] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-Sb: An open large-scale dataset for training generation image-text models. Advances in neural information processing systems, 15:25278-25294 2022, https: //openreview. net wwt//eerurrttteeeee -x3.mmnddd--------
[62] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned. hypernymed, image alt-text dataset for automatic image captioning. In I. Gurevych and Y. Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguistics. https: // //aclanthology.oo www.ooog//////. 1238//"
[63] P. Sorrenson, C. Rother, and U. Köthe. Disentanglement by nonlinear ica with general incompressible-fow networks (gin). In International Conference on Learning Representations, 2020. https://openrerrie. net//foruniidrryeeeeee
[64] Z. Sun. S. Shen, S. Cao, H. Liu, C. Li, Y. Shen, C. Gan, L. Gui, Y--. Wang, Y. Yang. et al. Aligning large multimodal models with factually augmented rlhf. In Annual Meeting of the Association for Computational Linguistics, 2024, https:/ h h/ts:////aaaaaoogg/ycy org/2024 findings--ant acl. 775. pdf.
14

---

[65] T. Tao, G. Wang. Y. Lao, P. Chen, J. Liu, L. Lin, K. Yu. and X. Liang. Alignmif: Geometry-aligned multimodal implicit field for lidar-camera joint synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21230-21240, 2024.
[66] V. Vapnik. Principles of risk minimization for learning theory. In J. Moody, S. Hanson, and R. Lippmann. editors, Advances in Neural Information Processing Systems, volume Morgan- Kaufmann, 1991. https:
/ //l1. acm. wwt//oi/to//ooooooooooo 100. 2555/2986916... 2987018.
[67] von Kügelgen, M. Besserve, L. Wendong, L. Gresele, A. Kekić, E. Bareinhoim, D. Blei, and B. Schölkopf. Nonparametrie identifiability of causal representations from unknown interventions. Advances in Neu- ral Information Processing Systems, 36, 2024. noteld-mk3uTiD
[68] J. Von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf. M. Besserve, and F. Locatello. Self- supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451-1646; 2021. https: htttss///ppenrevier..o net/teruumndeeeeeeeee p0o0Dt.
[69] T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939, PMLR, 2020. https: / http:e///prcceeingg.. mmm.eeeeddinne...........m..e... mlr. ggress/v119/wan///
[70] B. L. Whorf. Language, thought, and reality: Selected writings of Benjamin Lee Whorf. MIT press, 2012.
[71] C.-E. Wu, Tian, H. Yu, H. Wang, P. Morgado. H. Hu, and L. Yang. Why is prompt tuning for vision- language models robust to noisy labels? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3348--55497 2023, https: //doi. org/10. 1109/ 00009//IC1110000 2023. 014400.
[72] J. Wu, Y. Liang, H. Akbari, Z. Wang, C. Yu, et al. Scaling multimodal pre-training via cross- modality gradient harmonization. Advances in Neural Information Processing Systems. 35:36161 -31177,,, 2022. https: // /ptp///eppeeeeeeee//e/e/ee net/forumiddd/eaaeeee id77huttttl1QQ7.
[73] Y. Xia, H. Huang. J. Zhu, and Z. Zhao. Achieving cross modal generalization with multimodal unified representation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 63529-63541. Curran Associates, Inc., 2023. https:/ //pp:///////eee///////eeee net. wtt.mm/mdddttttoooooaooooeeeeeee Ioooo..
[74] Y. Xic, G. Li, X. Xu, and Y. Kan. V-dpo: Mitigating hallucination in large vision language models via vision- vision-guided direct preference optimization. In Findings of the Association for Computational Linguisties: EMNLP 2024, pages 13258-13273, 2024. https: http:///aaaaaaaaaaaoooce . org/ 2024. f findings- mereinnnggenll....m 775. pdf.
[75] H. Xu, P.Y. Huang, X. Tan, C.-F. Ych, J. Kahn, C. Jou, G. Ghosh, O. Levy, L. Zettlemoyer, W.-t. Yih, et al. Altogether: Image captioning via re-aligning alt-text. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19302-19318, 2024. https: sttps://aclanthoooogg .rr// 2024. eenee-mamceee. main. 1075. 1075.pdf.
[76] K. Xuan, I.. Xiang, X. Huang, L.. Zhang, S. Liao, D. Shen, and Q. Wang. Multimodal mri reconstruction assisted with spatial alignment network. IEEE Transactions on Medical Imaging, 41(9):2499-2500 2022. https: //dos. org/ 10. 1109/TMI. 2022. 3164050.
[77] D. Yao, D. Rancati, R. Cadei, M. Fumero, and F. Locatello. Unifying causal representation learning with the invariance principle. In The Thirteenth International Conference on Learning Representations, 2025. https://openreevew net/forumminee 11tkiidkkkkkkkxxppp
[78] D. Yao, D. Xu, S. Lachapelle, S. Magliacane, P. Taslakian, G. Martius, J. von Kügelgen, and F, Locatello. Multi-view causal representation learning with partial observability. arXiv preprint arXiv:2311.00 1.04066. 2023.
[79] M. Yarom, Y. Bitton, S. Changpinyo, R. Aharoni, J. Herzig, O. Lang, F. Ofek, and I. Szpektor. What you see what you read? improving text-image aligmment evaluation. Advances in Neural Information Processing Systems, 36: 16:1991-1619, 2023. https:///ppereeviww
[80] K. Zhang, S. Xie, I. Ng. and Y. Zheng. Causal representation learning from multiple distributions:
A general setting. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume PMLR, 21-27 Jul 2024, https:
235 of Proceedings of Machine Learning Research, pages // //proceedings nggs........ mlr. www.prrsssveee press/v235// 2teess/225//hhag2bbr zhaag24brrrr... httml.
15

---

[81] Y. Zhang. H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pages 2-25. PMLR, 2022. https:/ https://ppoceedinng
mlr www.spreessvvvee http//11222/eeeee 2rres//11822haag222a.
[82] Y. Zhang, Li. L. Liu, and W. Qiang. Rethinking misalignment in vision-language model adaptation from causal perspective. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. https:// https/////peeeeie..eoeoo www.ee.wweettttrrr//wwwwwbooooee
[83] R. Zimmermann, Sharma, Schneider, M. Bethge, and Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979 -199900 PMLR, 2021. https: / mlr. press/ prress/v399 v139/ / zimmermann211
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have outlined the background and introduced our main contribu- tions tions-including the problem formulation, theoretical results, and empirical evaluation-in the abstract and introduction, aligning them with the core sections of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See App. for a detailed discussion of limitations and future directions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification: The assumptions are formally stated in $ 3, and the complete proof is presented in App. C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The overall experimental setup is described in Expanded details for numerical simulations are provided in App. D.I, while App. E.1 and App. F.1 contain the full setup for the image-text experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code is available at https://anonynnn wcw.::osss.opp.....cccccc.cc.e
crossmodal hagprndddlliisllagggnnnte-eeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ Saaiiddee/CoeSSbmiisieee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nipp.cc..oooo wwt.cc/cccccccuiicoooooooooo gut////dde///e/// /Coee//mmdeabiiicc iemniissionnn ssionnolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setup described in the main paper, with full details provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We run each experiment with three random seeds and report the mean per- formance in heatmaps. Standard deviations are visualized as shaded areas or eтror bars in plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See App. H for detailed information about the computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:::///nurrps.... httpp////eeapp..pppppii/i/////// CthicsCuudeellees
Answer: [Yes]
Justification: This work complies in all respects with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:[Yes Yes]
Justification:
We discussed broader impacts of our work in App. 1.2.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Our work does not involve the release of high-risk models or datasets. The models and data used in this study are focused on theoretical analysis without known potential for misuse.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All third-party assets used in this paper-Suue such as code, datasets, and mod- els-are are properly credited, with licenses and terms of use explicitly acknowledged.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww/coo//a/o///o datastss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This work does not introduce any new assets. All experiments conducted using existing publicly available datasets and models, which are properly cited and used in accordance with their licenses.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects and therefore does not require IRB or equivalent approval.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research does not involve LLMs as any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: bttps://neuute ctw.iipppcccooocceeeeeeeeeeeeeee cc//oonferencce//222//LL.. for what should or should not be described.
23

---

On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (Appendix)
Contents
Notation and Terminology
25
Related Work
26
C Proofs
27
C.I Lemmas
27
C.2 Proof of Theorem 4.1
29
C,3 Proof of Corollary 4.1
33
C.4 Proof of Corollary 4.2
34
D Numerical Simulation Details
35
D.I Detailed Experimental Setup
35
D.2 Additional Identification Results
36
D.3 Additional Downstream Results
38
Experiment Details on MPI3D-Complex Dataset
38
E.1 Detailed Experimental Setup
39
E.2 Additional Results
40
Experiment Details on Causal3DIdent Dataset
F.1 Detailed Experimental Setup
F.2 Additional Results..
44
Details on OpenCIIP Case Study
46
G.I Concepts Taxonomy and Data Collection
46
G.2 Details Caption Frequency Statisties
47
G.3 Zero-Shot Evaluation Details
49
H Computation Resources
49
Discussions
52
L.1 Limitations and Future Directions
52
I.2 Broader Implications
53
24

---

A
Notation and Terminology
Table 1 provides summary of the notations and terminologies used throughout the paper.
Table I: Notations and terminologies used throughout the paper.
Observation and Latent Spaces
Image observation space (C 1Rd))
T
Text observation subspace under selection bias 0
S
Latent semantic space (C R"*)
Mr
Image-specific non-semantic latent space
M, Index set of semantic variables: [m.]]
Text-specific non-semantic latent space
I
law Iver
Index subset of semantic variables that remain invariant under distribution shift
Index subset of semantic variables that vary under distribution shift
Mappings and Functions
Differomorphic generative mapping for images: S x MM + N Differomorphic generative mapping text under selection 0: St, x Me + T(0) A diffcomorphism function class where 9,(0) selected from Image encoder: X' + (0, 1)" with specified n
g,003 Ge
fz
Text encoder: T (0, 1)" with specified
Loss Functions
Lonaci (fe, ft)
Symmetric InfoNCE loss for MMCL (Fq. (1))
CyynAigghMssa (Jx fa)
Alignment and entropy maximization loss (Eq. (2))
Notations for Cross-Modal Misalignment
0
Selection bias, integer realization in the range [2" 1]
pt (IL)
The of non-emply subsets of I. A selected semantic subset indexed by 6, Ig E (T+)
Io
T
Omitted semantic subset under 6, IŹ = IL llo
Perturbation bias, an integer realization in the range [2 ol 1]
Prop (Ho)
The set of all proper subsets of Is A subset of Is subject to perturbation indexed by p. II, PImnp(Io)
I.
I
Subset that always unbiased under p. Iţ ll \ Ip
Pa 10,0,p
Perturbation conditional distribution, reflecting misalignment (Eq. (5))
A C 1p
random subset of semantic variables subject perturbation, drawn from PA
Distributions and Operators
Psy Pma Pm, H(-)
Distributions semantie, imagc-specific and text-specific variables, respectively Differential entropy
5())
Dirac delta function
Random Variables
Image observation sampled from X
(9))
Text observation under selection view 0, sampled from T
8
Latent semantic variables in S
Selected latent semantic variables for generating text Omitted latent semantic variables when generating text
S:ş
Si,
Perturbable latent semantic variables for generating text Unbiased latent semantic variables within the selected part Image-specific non aon-eemattic variable in M Text-specific non-sena non-semantic variable in MM
m.
m:
Combined latent variables for images: (s, m2)
2,(0)
Combined latent variables for text under 0: (8L, m,)
25

---

B Related Work
Theoretical multimodal (multi-view) contrastive learning. Recent work has sought to formalize the theoretical foundations of multimodal and multi-view representation learning, particularly under contrastive objectives. Wang al. [69] decompose the InfoNCE loss [51] into alignment term, which pulls positive pairs together, and a uniformity term, which encourages dispersion over a hypersphere- the groundwork for subsequent analysis. Zimmermann et al. [83] show that contrastive objectives can invert the data-generating process, while Liu et al. [45] extend this result to multimodal settings. However, these approaches often rely on strong assumptions about latent distributions or manifold structures, which limits their practical applicability. A complementary line of work, such as [68], demonstrates that contrastive learning with data augmentation can recover shared content without requiring such strong parametric assumptions. This has been extended to MMCL by [48, 13], and Yao et al. [78] further investigate identifiability under partial observability in multi-view settings. Distinct from prior work, we do not assume fixed content-style decompositions, causal directions, or perfectly aligned pairs. Instead, we analyze MMCL. with image-text pairs under cross-modal misalignment and systematically examine its impact on representation learning.
Vision-Ianguage models and perspectives misalignment. Multimodal contrastive learning (MMCL) has achieved significant empirical success, particularly in aligning visual and textual modalities using models such as CLIP [55] and ALIGN [27]. These successes are partly attributed to the use of massive training corpora, e.g., LAION-5B [61], which are substantially larger than those used for vision-only foundation models [22. 52]. However, real-world multimodal datasets are often imperfectly aligned and noisy [49]. Existing empirical methods [1, 34] typically treat such misalignment as label noise, employing strategies such as multiple-instance learning or dataset refinement [1] to mitigate its impact. While some recent work suggests that contrastive models are robust to certain forms of structured misalignment [50], Others suggest augmenting text as proxy for changes in visual content [6]. Our work suggests that cross-modal misalignment can act as either a barrier or a bridge, depending on the application. Unlike [50], which assumes linear representations without modeling the generative process, our analysis is grounded in realistic latent variable model that provides a deeper understanding of misalignment from a data-generating perspective.
Identifiability in latent variable models. Identifiability analysis addresses the fundamental ques- tion of whether the learning process can uniquely recover the latent generative structure or distribution underlying the observed data. This problem has been extensively studied in the context of nonlin- ear independent component analysis (ICA) [23, 46, 28, 63] and causal representation learning (CRL) [41, 43, 80, 77, 44]. In either setting, full identifiability. typically up to permutation, is rarely achievable in practice without strong assumptions, such as access to interventional data. Conse- quently, recent works have focused on partial identifiability [20, 31, 18] or relaxed equivalence classes, such as identifiability up to linear transformations [83, 45] or up to cooup-wis//blockk indeterminacy [68, 78], which can offer sufficient guarantees for specific tasks or applications. In the context of multimodal representation learning, several recent studies have explored identifiability results [48, 13, 45], but largely neglect the presence of cross-modal misalignment. In contrast, our work explicitly models misalignment and adopts a block-identifiabliii definition to characterize the extent to which semantic factors can be recovered up to an invertible mapping.
Invariant representation learning. Invariant representation learning (IRL) seeks to learn represen- tations that remain robust under distributional shifts between environments [2, 15, 18), particularly in settings where empirical risk minimization (ERM) [66] fails to generalize out of distribution. In the absence of such challenge, ERM is sufficient for in-distribution prediction, rendering the objective of IRL unnecessary. From a causal perspective, learning invariant representations, or more ambitiously, autonomous mechanisms [60], requires observing sufficient variations in the latent factors underlying data [41]. Such variability can be introduced through interventional data [39, 33), exchangeability assumptions [56], or the use of auxiliary variables such as domain indices [80, 42]. However, direct interventions on latent variables are typically infeasible in real-world observational data, and auxiliary variable methods often require access to a large number of diverse environments to ensure identifiabllity-
assumption that is often challenging to satisfy in practice. Our work sheds light on an alternative approach: by leveraging the inherent flexibility of text supervision, we show that manipulating biases, specifically through selective omission or semantic perturbation in text, can serve as controllable proxy for environmental variation.
26

---

C
Proofs
C.1
Lemmas
Before proceeding with the proof, we first establish the following lemmas.
Lemma C.1 (Global Minimum of CsymstiinMuate)) In the context of the proposed LVM as outlined in $ 3.1, and Asms. 3.1 and 3.2, the global minimum of
1 2
LomddiggMaadd far fi) (x,t(0))~ppp. 2())
(6)
This minimum can be attained by the following pair of smooth functions:
f = d (9. 1)3 : X + (0, 10", ft = d T (0, 1)",
(7)
(8)
where:
9s and gese) denote the true underlying generative mappings for images and paired text, respectively, as described in $
The operator (1)s, extracts the components corresponding to the unbiased semantic variables (i.e., unaffected by the selection bias nor the perturbation bias), with [Tg. being their dimensionality.
d (di....., dr) defined via the Darmois construction [12, 23, 68], where for each i E [m]]
d.(s:;) = CDF; - sTg, St5.1e-11),
with CDF, denoting the conditional cumulative distribution of $ig. given (t..
Proof of Lem. C.I. We prove that the candidate functions f and ft in Equations (7) and (8) yield SsymAligMMxxtal (f2 fi) = O. Substituting these candidate functions into the loss in Eq. (6), we have
SsymAlignMasten (f2,
[I.2 - (t
(x. Jm
(f)()x)) + (J))t))))
By the invertibility of the generative processes Is and grco (see 3), we may change variables to express the expectation over the latent variables:
SsymAliguMestn (f² (f2 fi)
E (z (0) Pr (())
[id((i:) d(s;:))lz
1 (d((s)))) H (d((:)))
where Siş and št. denote the preserved unbiased components of the semantic variables across image- text pairs, respectively.
We now show that these unbiased semantic components are identical across modalities almost everywhere (a.e.). By Asm. 3.2, for any image-text pair the text is generated via random perturbation process that modifies only subset A Ip of the activated semantic variables. Specifically, recall Eq. (5), the perturbation density is defined as
pkr ps.0 (1
s, A) = 8( - SIA(A) Psals, (SA SA)
Since A C Ip. it follows that the indices in Ić are subset of those in Hs A; that is, IE S Ho A4 Thus, the Dirac delta in the above expression enforces that
$ig
Stş almost surely (a.s.) Vs Ps. S1, Pai, 1a.0,p
27

---

under selection bias & and perturbation p. regardless of the particular perturbation set A at each time.
Further, by the properties of the Darmois construction [12), the mapping d transforms Stş, into uniform distribution over (0. 1)" (with n = (Tg). Since the uniform distribution is the unique maximum entropy (i.e., zero) distribution on bounded domain (under no further moment constraints) [26, 10], the entropy terms in the loss are maximized. In the formulation of SymAlignMaxdis this maximal entropy precisely cancels any potential reduction in the loss, ensuring that
SynmAlgnMMasnn (5z 0.
Therefore, the global minimum SymmAlignMat is achieved at 0 by the given function pairs fz and fi completing the proof.
Lemma C.2 (Uniformizing Mapping Preserves All Information). Let 2 + be a smooth map between simply connected, open C² manifolds u, V.V S 12. Suppose that is random variable taking values U with a smooth probability density that is strictly positive a.e.. If the pushforward h(u) is uniformly distributed V. then h is global diffeomorphism: in other words, h is bijective and depends on every component of u.
Proof of Lem. C.2. Let Pu : .44 - IR and pv V + denote the probability density functions of and v, respectively. Since h(u), the change-of-variable formula (refer to, e.g., (5)) yields
= p.))= det ((h)(u)
where u is any preimage of v under h. By assumption, v is uniformly distributed on V; that is, there exists constant > 0 such that
pv(v)= for all v E V.
Thus, for any with = huu) we obtain
C Pu Pu(u)
det (())
Since (u( is strictly positive a.c. on 21 it follows that
det det ((h)
0 a.s.
Vu Pu:
or equivalently, det (u) + 0 a.e.. By the Inverse Function Theorem (see, e.g., [58]), this implies that h is local diffcomorphism.
Moreover, since L/ and V are simply connected, open C³ manifolds, standard covering space theory (refer to, e.g., the discussion around Theorem 1.38 in [21]) implies that h is covering map. The uniformity of Pv forces to be surjective (otherwise, some points in V would have zero density, contradicting uniformity). Since any covering map from simply connected space is trivial, h must be equivalent to the identity covering. In other words, is a homeomorphism onto and hence both injective and surjective (i.e., bijection).
Finally, the fact that the Jacobian determinant nonzero a.e. guarantees that h depends on all components of if any component were omitted, the rank of the Jacobian would drop, contradicting non-singularity. Furthermore, by the Global Inverse Function Theorem (refer to, e.g., [59]), the inverse of h is smooth.
In summary, h is global diffeomorphism from U onto V. Consequently, it preserves all information of u: every variation in u is reflected in v = h(u), and u can be uniquely and smoothly recovered from v. This completes the proof.
We do not claim originality for this result due to its fundamental nature in topology and measure theory; rather, detail here tool for subsequent arguments.
28

---

C.2 Proof of Theorem 4.1
We now proceed to prove Thm. 4.1. To begin, we restate the theorem for clarity:
Theorem 4.1 (Identifiability of latent semantic variables). Let (x, t(0) be image-text pairs drawn from the data-gen gettrannerg process described in $ 3, where x is generated according to Eq. (3) and th is generated by Eq. (4). Further, suppose that Asms. 3.1 and 3.2 hold. Denote by St; the subset of semantic variables that annotated without bias in the text, and define dimension as = =Tol 1T Let Ja X + (0.1)* and St. T - (0, 1) be sufficiently flexible, smooth functions. Then, minimizing the loss LssnniisMasss Eq. (2) over samples (x, t. guarantees that fa. and Jt block-identify the semantic variables 81g in the sense of Defi. 4.1.
Proof of Thm. 4.1. The proof is organized into the following five steps:
First, we show that the objective function SsynAiggMatt (fz. fi) achieves a global minimum value of O. At this minimum, any pair of smooth functions fa and ft satisfying this condition must exhibit invariance across modalities. This invariance condition ensures that the learned image representations and text representations must align across all positive x and ( pairs.
Next, we prove that minimizing LsymAlignMaccnn inherently eliminates any dependence of the learned representations representations are restricted to the dependence on latent semantic variables.
moddllttysspeciff variables m, or m,, This ensures that the
By contradiction, we further establish that any contribution from the omitted semantic variables induced by selection bias 0, i.e., Sig, would violate the invariance condition established in Step 1. This guarantees that the representations exclude the dependence on omitted semantic variables.
We then establish the exclusion of perturbed semantic variables influenced by perturbation bias p, i... Si,, from the learned representations, also by contradiction.
Finally, we demonstrate that the optimized mappings, which compose with the underlying generative functions, are invertible with respect to the learned representations and the true unbiased semantic variables 81.- This ensures that the representations block-identify the preserved unbiased semantic variables, thereby concluding the proof.
Step (Global minimum and invariance condition). Let fa : X - (0, 1) and ft T()) - (0, 1)" be any smooth functions attaining the global minimum. Define the smooth mappings:
h
fz gar ho fe o gre0
Since all terms in LSymAAignMantn non-negative, and its global minimum is 0 by Lem. C.1, each term in C SamAliinMa must vanish a.s. for any pairing (x, to)) Thus, minimizing SymAAiggMMnnn leads to:
E
[115((x)= - f.(t
E
(11)x(2)) - he (Z)10) ||2) = 0, (9)
(x,+(9)
(2212 (0)) JuP Pa (,,
H(L(x))) H (1 (2x)) 0,
(10)
H()(t()) H (he((e)))) = O.
(11)
From Eq. (9), it follows that
hz(2))))) ht, (z.z)
(z.z).
a.s. Vzg Ps Pmm, Zette Pei, s.9.p Pmn:
(12)
which ensures alignment of representations a.s. for any pair (x, t))). The substitution of expectations for the image modality in Eq. (9) is valid because x follows the pushforward distribution of z, under the deterministic diffeomorphism Iz (Eq. (3)); similar argument applies to the text (Eq. (4)).
Equations (10) and (11) imply that h2 and h, map the latent variables z, and 2,c0) onto uniform distributions over (0, 1)" (with n = (I since their differential entropy equals to zero
29

---

Step 2
(Exclusion of sppecfft- variables). We now show that the smooth functions ha and h, depend only on latent semantic variables (with further exclusion components in later steps) and not modality-specifet variables m, or m,,
Since 2y (s. m.) and (, ( m.) the latent variables generating images and paired text t respectively, the data-generating process in 3 implies the following independence properties:
(cl) is independent of 2,0::: This means changes in m, do not influence 2,00). Moreover, since the text generation process Jo00 is independent of m. follows that:
hy ($1. m., m = h4 (5 m
(13)
(c2) m, is independent of Zz: This means changes in m, do not influence 2g. Similarly, since the image generation process Iu independent of m,, it follows that:
h (s. Ma, m,) = h (s, ma).
(14)
Combining Equations (12) and (13), we have: (s, m.) he (St, m my); a.s. Yzz Ps Pm. Zet0) Ps |..0. Pm,
(15)
Consider small perturbation m. M. + S, where ||s|| is arbitrarily small but remains within the open space My. Since changes in m, do not influence he by statement (cl), we obtain:
( my m. + s) = h4 (5 m, m
(16)
Since m, is independent of s. perturbations in m, does not alter the semantic variables, and Pm, > 0 a.e. over M₂ by Asm, 3.1. Thus, substituting
(s, m + (s, m. + s)
in Eq. (15) and combining with Eq. (16), get:
(s, m. + 5) = hr (s, my).
By the smoothness of ag (inherited from the smoothness of 92 and fz), taking < -> gives: h my + +s) he m.) 0.
Öh
lim 0m, S0+
Thus, ht is independent of m
symmetric argument applies to m. m, + m. s with a small enough perturbation, then by Eq. (14) in statement (c2), h my + s) remains unchanged. The invariance condition in Eq. (12) then forces (,,, m, + s) to remain constant w.r.t. s, showing that h2 is independent of my Therefore, the learned representations satisfy:
hr (s), a.s. Vz Ps Pmo: hz(2,,0)) = hy (St,); a.s. V2000) ~ Ps., 1s.pp Pm,
(17)
(18)
Step 3 (Exclusion of omitted semantic variables). We now establish that the function ht is independent of Sig. where Iī = Ig \ llg. In other words, the learned representations do not contain information about the omitted semantic variables that are absent in the corresponding text.
Using the invariance condition in Eq. (12), together with the independence of modality-speciff non- semantic variables in Equations (17) and (18), have the following updated invariance condition:
hs)(s) h=(5:.); a.s. Vs ~ Ps: St, Pat, js.
(19)
Next, we show by contradiction that hy independent of sig. Suppose, for the sake of contradiction. that there exists function hy = ff 9x which depends on at least one component of the omitted semantic variables sig. Formally,
Ô h (si,, Ös
31 Tý, (sf, si.) E ES, such that
*
30

---

By the C' continuity of hg. guaranteed by the smoothness of both Iu and f2, and that S is an open space, it follows that
En 0: +
\C(s. (igh)))))))) is strictly monotonic on (si 7) sf + r)),
where sig denotes all components of sig except st-
Since ps a.e. on S, we can find distinct realizations of latent semantic variables
(s (sT (ix) (si, (st, sig)n)) with 87 E (of 7, st); of E (07, of + n)
(20)
that correspond to two different image observations, such that hg(si, :igg)())) * (g(s (6f sigr())))
(21)
However, combining Eq. (19), we have
h (s7 sigho)) he(s), (s sigve)) ho(
(22)
where sr, represents the perturbed semantic variables si, introduced by selection bias (with the exclusion of perturbed components further addressed in the next step).
Equations (21) and (22) thus contradict each other. Hence, such function hy cannot exist. Conse- quently, hy must be independent of Sig. Formally,
hg (2z) = h2 (sL,), a.s. Vzg ~ Ps Pm.
(23)
Clarification C.1 (Causal Interpretations). (i) Justification for the existence of distinct points in Eq. (20). This follows from the assumption Ps a.e. on S by Asm. 3.1. From latent SCM perspective [53, 67], even if specific semantic component 5; in Big is the ancestor node of some other semantic components in S1,, the strict positivity of ps ensures that the exogenous noise variables are well-defined. Thus, for different values of 81, there exist corresponding noise values that keep si, remaining fixed. (ii) What if the unknown causal structure is Sty - Si ? The potential causal influence from Sty to SL, does not resolve the contradiction. Independence here means that, once 1 set, there is no direct functional path from Stş the representations (st.).. ), i.c., the causal influence among them is fully accounted for by the realized value of St,,
In summary, these arguments show that h is genuinely independent of Sig. even allowing for arbitrary unknown causal interactions among the latent semantic variables.
Step 4 (Exclusion of perturbed semantic variables). We now demonstrate that both representations are independent of st, and št, respectively, as a consequence of the contradiction between the invariance condition and the random perturbations introduced by perturbation bias.
First, we refine the invariance condition by excluding omitted semantic variables as established above. Combining Equations (12), (18) and (23), we obtain:
h (St,)
hy St.)
a.s. Vs ~ Ps. ST,
Pip S,0.p
(24)
Next, we show that he must be independent of $1, by contradiction. Suppose, for contradiction, that there exist some function hf = fi o Jro; which depends on at least on component of the perturbed semantic variables St,- Formally,
31 E Ip: sig) such that (h Stg
0.
0s;
By the C'l continuity hy guaranteed by the smoothness of fj and guis, for some sufficiently small > O, we have the following inequality:
(g((5 Si,)), # NC((sT . sig) V&T E ($] - 81), of E (87, +))
(25)
the other hand, by the pairing conditions in Asm. 3.2, there exists at least one subset C Tp of perturbed semantic variables such that / A and ( > 0. Pick one such set and call it A. Define
31

---

the realization of latent semantic variables corresponding to the image of this pair as ((rA)) (here, we omit the latent semantic components that have already been excluded in previous steps). By Asm. 3.2, we know that AA SiAA a.c., that is, (SI,AA, siy) (As Sig a.c.. Thus, we can rewrite the corresponding realization of textual semantic variables s = -si, sip) as ( si,,)).
Further, also by Asm. 3.2, there exists non-empty open subspace O^ C SA such that Paals (-1s)) is strictly positive on OĄ- Since the perturbed random variable S is a realization within this open subspace, we know lies in OA and OA non-empty. Moreover, because PsAla (-15)) is smooth and strictly positive on OA, there exists a sufficiently small > 0 such that
Psa|sa (SA)s2) O, VSA (1))()) x (5] 1,, s7 +m), with (1) } x (5) -1, $7 +mi) S AA.
Thus, with a positive probability guaranteed by the above conditional, for the realization of image semantic variables (s), va),)), we can construct two distinct realizations of perturbed semantic
variables for generating different text (due to Ps > 0 a.e.):
where
E (5] - 1/2 B)), of (5i, 5) + 1/22 with 1/2 = min(n, m)).
Based on the invariance condition established in Eq. (24), we have the following equalities:
hf ((s). s),
This is contradicted by the inequality established in Eq. (25), which implies that such a ht cannot exist. Consequently, any he minimizing the loss must be independent of the perturbed semantic
variables s i.e.
(Sr.) he (Srg), a.s. Vi,
Pai, 1s,8,p-
(26)
Updating the invariance condition, and combining Equations (24) and (26), we obtain:
hg (St,) = he(stg), a.s. V ~ Ps.
(27)
The exclusion of Sr, from image representations h2 (8:..) can be demonstrated by a similar procedure to Step 3, namely, that excluded semantic components from one modality cannot exist in another view, regardless of the latent causal structure among St,- Specifically, fixing the value of Stş and varying ST, within a small region, we can sample distinct semantic variables (due to Ps > 0 a.c. over S by Asm. 3.1). The smoothness of ht then leads to an inequality if it depends on any component in Ip. This inequality contradicts the alignment condition established in Eq. (27). Thus, hg must also be independent of St,
Overall, due to the exclusion of modality-specifee variables (m, and m.). omitted semantic variables ($t;) and perturbed semantic variables (st,) introduced by selection and perturbation biases for generating text, we now have the following equalities:
hz = hr (s1;),
a.s. Vzz ~ Ps Pm.
(28)
he(2,,o)) hy (stg),
a.s. Vz ls..,e.0.p Pm.
(29)
Step 5 (Preservation of all unbiased semantic variables). Based on Equations (28) and (29), we define the learned image and textual representations as
2 h2 (81;). ), 2, = h1((s,)), with 2, E (0,1)", 2, E 1)".
According to Equations (10) and (11), we know that the learned representations in both modalities
uniformly distributed over (0, 1) By directly applying Lem. C.2, it follows that the learned representations 2Ø (and also 2,) include all and only the information of the unaltered semantic components St: a.s., and that hy (and similarly h₂) is invertible. Consequently, the true modality- shared semantic variables Siş are block-identified by Sa and St in the sense of Defn. 4.1.
Thereupon, the proof of Thm. 4.1 is complete.
32

---

C.3 Proof of Corollary 4.1
We now proceed to prove Cor. 4.1. To begin, we restate the corollary for clarity:
Corollary 4.1 (Identifiability of full latent semantic variables). Assume that Asms. 3.1 and 3.2 hold. Let the selection bias be = 2 - and the perturbation bias be p = 1, such that the full set of semantic variables Is is selected, and the perturbable semantic subset trivial, i.e., Ip Then, all semantic variables s are ieeek-idetiiiied via smooth functions fa : :X + (0, 1) and Jt T + (0, 1)",, when minimizing LspmauigMMuaFar
Proof. As we have fixed graded lexicographic order over the range of 0, that is, over P* (Ila) as defined in Defn. 3.1. Then, setting 0 2 corresponds to selecting the full set of semantic variables for text generation, i.c., 1p Ig.
Furthermore, have similarly fixed graded lexicographic order over the range of p. i... over Prrop(ls (T&s,, as defined in Defn. 3.2. Given that To = Is. setting = implies that all semantic variables preserved without perturbation during the generation of the corresponding text t
Under these assumptions, and by Asm. 3.2, the perturbing subset A is always trivial because Ip is trivial. Consequently, we have
ps ( = ö(s s)
with To = Is, II =0 a.s. Vs ~ Ps:
which indicates that almost surely.
Now consider any pair of smooth functions fz : X - (0, 1) and fro) T (0, 1), that achieve the global minimum of the loss in Eq. (2), i.e., yield a value of zero. From Step 1 and Step 2 of the proof of Thm. 4.1, it follows that:
ha
(x)) (2x)
h4 (2,(0) h. (s) hs(s)
Vzz Ps Pm,
Zele) Pisp 0s Pinc
Since both the omitted and perturbable index subsets trivial may directly apply Lem. C.2, which implies that the full semantic vector s is block-identified by both fz and ft-
This completes the proof.
Definition C.1 (Graded lexicographic order). Let S be a finite set with a total order (e.g.. S (1.2..... ..). The graded lexicographic order on the non-empty subsets of S is defined as follows: (i) Subsets are ordered first by increasing cardinality (i.e., all subsets of size k precede those of size + 1). (ii) Within each group of equal cardinality, subsets are ordered lexicographically: that is, {i,, ... < (i..2......Ja if there exists an l such that Jm for all m < l and ig < je.
Clarification C.2 (Fixed graded lexicographic order). The graded lexicographic order over subsets of semantic indices is fixed solely for notational consistency and clarity. For example, given semantic variables s (81, 82, $a) with index set ll = [1,2,3), the graded lexicographic order over the non-empty powerset p+ (Is) yields:
P (IL) {{1), (2), (3), (1,2), (1. (2, (1),,3)}
This ordering imposes constraints the underlying latent structure and is adopted without loss of generality. Since the true permutation of latent variables is unidentifiable, any consistent order can be used index subset-valued parameters such 0 (selection) and p (perturbation). Importantly, both and act indirectly through text observations rather than directly operating on the latent space. The fixed order simply determines how each value of p maps to subset of semantic indices. For instance, omitting color in a caption corresponds to removing the associated latent variable-which index of encodes this depends on the fixed ordering. Thus, adopting graded lexicographic order ensures a reproducible indexing scheme without limiting model generality.
33

---

C.4 Proof of Corollary 4.2
We now proceed to prove Cor. 4.2. For clarity, we restate the corollary below:
Corollary 4.2 (Identifiability of invariant semantic variables). Assume that Asms. 3.1 and 3.2 hold. Consider an OOD setting in which a subset of semantic variables, Line C ll., remains invarians between training and lesting environments, while the remaining semantic variables, Bear 1l Tanns undergo distribution shifts. the union of omitted and perturbable semantic variables under selection bias 0 and perturbation bias p coincides with the then the invariant semantic variables Sianv are block-identifier via smooth functions fo X 7 (0, 1) Hooo and ft T + (0, 1) (0, by minimizing SgmdtiigMMaann
cetoonmmen----nsttiii subset, i.e., Dvaar 1% U p>
Proof. Under the OOD setting, and without loss of generality. let the subset of semantic variables susceptible to distribution shift be denoted by Ivar = Lvar U 12 Suppose the index set associated with selection bias given by 1.e 1.ar U Hivv and that the perturbation bias acts on Lp = 4 That is, the subset 1 entirely omitted by the selection mechanism (i.e., excluded from ll.)., while the subset Mar is included but remains vulnerable to perturbation, as determined by p.
Given this structure, we directly apply the argument from the proof of Thm. 4.1. The omission of variables in Lenr together with the perturbation of variables in I ensures that only the invariant subset Hiw is both selected and unperturbed. Therefore, the invariant semantic components St... alock-identified via smooth functions
fz X' + (0, 1)! and f + (0, 1)
which attain the global minimum of the alignment objective. This concludes the proof.
34

---

D Numerical Simulation Details
We provide additional details on the numerical simulations that are not fully covered in $ 5.1. Specifically:
In App. D.I, we outline the experimental setup, including hyperparameters, model architectures, and the construction of downstream tasks.
In App. D.2, present additional experiments that further validate our theoretical findings.
In App. D.3, we analyze downstream performance under various perturbation bias settings.
D.1 Detailed Experimental Setup
Latent space construction. We define latent space comprising a semantic subspace S C I2 and two coodality-specifie subspaces Mr [ RS Variables are sampled from Gaussian priors:
(O, E.), m, ~ (,,, Em,), m,
~ m N0 E,,)
For independent semantic variables, Eig = Too yields a factorized standard Gaussian, aligning with nonlinear ICA assumptions [28, 63]. For dependent semantics, we sample Eg from Wishart distribution Who(T, 10) (identity scale, 10 degrees freedom), introducing latent dependencies as in [68, 13, 78J. Modality-specific covariances Mm, and Em sampled similarly from (1, 5), the Wishart distribution with identity scale and degrees of freedom.
Selection and perturbation biases. Given the 10-dimensional semantic space, the total number of nonempty subsets 210 1023, defining the full range selection bias configurations For tractabilityy, choose 10 representative selection subsets Lg. detailed in Table 2.
To isolate perturbation bias, we fix full selection (0 1023) and evaluate 10 representative perturba- tion subsets Ip, as listed in Table 3. Perturbations are introduced by independently modifying each semantic variable s; I, with probability 0.75 using additive Gaussian noise:
(0, Eg,i);
where the joint noise covariance Σ, is sampled from Wio(I, (I, 10). Perturbations are applied solely to the text modality, aligning the proposed LVM and assumptions stated 3.
In addition to the isolated bias settings, we consider a joint bias scenario to investigate compounding effects. Specifically, we select I (corresponding 0 968, excluding the last semantic indices), and apply perturbations to the first two indices (I, = [2]. correspnding to 12).
Results across these 20 representative configurations (10 selection settings, 10 perturbation settings) and the joint bias case serve to validate the theoretical findings under a range of misalignment settings.
Table 2: Selection bias settings and selected semantic indices.
0,
56
176
386
638
848
968
1013
1023
Io
{1}
2
[3]
[4]
[5]
[6]
[7]
[9]
[10]
Table Perturbation bias settings and perturbable semantic indices.
0
1023
1
12
57
177
387
639
849
969
1014
4p
0
(1)
5
[6]
[8]
[9]
Parameter settings. We parameterize the generative functions 9N and 9200) using randomly initial- ized 3-layer invertible MLPs, following prior work [13]. Invertibility is enforced by maintaining a condition number threshold of 1e-3 for each layer.
The encoding functions far and fe are implemented as 7-layer MI.Ps and optimized using the Adam optimizer. For MMCL training, use batch size of 6144, learning rate of 1e-4, and train
35

---

for 100,000 iterations. The loss function is given by Eq. (2), with Euclidean distance used as the similarity metric and temperature parameter set to 1.0. To ensure training stability, gradients are clipped using maximum 2-norm of 2.
with three distinct random seeds, and report averaged R values, clipped to the interval [0, 1] for interpretability.
All experiments
Downstream tasks design. To evaluate pretrained representations under various bias conditions, we construct several downstream tasks. Specifically, four regression tasks are created by generating labels using complex nonlinear functions f,, each applied different subsets of the semantic variables:
11 = fu (8(3))
1/2 mm (S|a)),
y/ = (S:7)) 1/4 - fn (S19))
Each function Fuc) F IR includes quadratie, cubic, pairwise, and triple-wise interaction terms, as well as sinusoidal, logarithmic, and exponential transformations. The full formulation for semantic vector Sig] is:
d
fv (s(a)
= 1=1
2 +
0.3
Σ
0.5
Σ
8,3; +.2
Σ
8,3ySk
1=1
ISKjSd
1sKgt<kEd
+0.7)
(sin(8;) + cos(s:)) + 0.4
log(1 + (8:1) + 0.4
|sll
For the classification task, labels obtained by binarizing 32 its median value, which serves as the decision boundary. To simulate distribution shifts the observations x, we apply a skewed, heavy-tailed transformation semantic dimensions 9 and 10:
s sign(s))
|,
for (9,
For both downstream tasks, we fix the pretrained encoders and evaluate the quality of the learned representations using two-layer MLP probing model. We generate 20,480 samples as the evaluation set for training the regressors and classifiers, along with an additional 20,480 samples as the in-distribution test To assess OOD generalization, we generate another 20,480 samples from the shifted latent space as the OOD test set.
The regressors are trained using Mean Squared Error (MSE) loss, and the classifiers use Cross- Entropy loss, both with a learning rate of 10-3 and trained for 10,000 steps. We report classification performance using the average Matthews Correlation Coefficient (MCC). Importantly, in the OOD setting, we perform no adaptation or fine-tuning; the classifier is evaluated directly to test the generalization capability of the pretrained representations.
D.2 Additional Identification Results
Results under perturbation bias. As shown in Figure 11, the results of predicting true latent semantic variables under various perturbation bias settings exhibit similar trends to those observed under selection bias. Modality-specifii variables are consistently discarded, unbiased semantic vari- and misaligned semantic variables become partially identifiable
ables are faithfully block-identife in scenarios with dependent latent variables demonstrating a consistent pattern across both image and text modalities. These findings further support and reinforce our theoretical analysis.
Linearity of learned representations. We further analyze the linearity of the identified semantic representations by reporting the R² scores obtained from linear regression applied the learned features for predicting the true latent variables. As shown in Figure 12 (selection bias) and Figure 13 (perturbation bias), the performance of linear regression closely mirrors that of nonlinear regression reported in Figure 3. This strong correspondence suggests that the relationship between the learned representations and the true latent semantic variables is approximately linear, indicating that the identified representation subspace is nearly linear.
36

---

m, 6.6 o
3.5
5.6
6.0 6.0
5.6 6.0 0. 6a
6.0
06
03
6a 62
0.0
on
m--00 0
3-0
5.0
8.8
9.0
5-8
9-0
00 50
0.0 6.0 0
D-3
5-6
06
0.
5-
5.6 60
0. 0.
02
00
0.3
60 on
68 0.5
0.0
0.0
0.3
5.6
6.0 on 3n sa
51-5.0
0.0
0.
5.0
D.0 D.O
B.0
0.
8303
Sx
0.0 0.0
3.0
8.
08
33
20 1.
0.00
0.5
0.2
6.0
5.0
6.0
10
19
5a.0.0 00
6.0
6.
00
BE 10
0.0 0.0 0.0
0.0
3.6
00 T
1.0
0.1 0.2
08
0. 0.2
0.3
10
53-6.0 0.
3.5
8.0
60
D
1.0
2.
0.0 6.0 0
0.3
3.0
3.0
0.1
05 10100010
61 0
1.
$4-0.0
LO
0.0 0.0000000
10
2.0
02003
2.000000 1.0
02
100000000 2.0
5..5.0
1.0
5a.60
0: J/
0. REB
1.0 1.0
0.
1a 16
1.0
10
1.6 1.0
1
1000000
R.D 1.
1.2
1. 1.0
3.0 2.0
A.0
3.0
2.2
1.
1.0
2.0
3.0 1.0
DA 2- 10 3.0 2,0
1.0
[9]
DT (6) [5] [4] (21 R
[9] [H (1 (6) 1S] [4]
(3)T))(1)) (1) Ø
(91 [H] [F] [6] [ST [A] (2)
[9] [8] P11 [4] [5] (2)
perturbation bias,
perturbation bias,
perturbation bias,
perturbation
&.
Figure 11: Mean R2 scores under perturbation bias settings. Left to right: predictions using representations 2 and â, under independent latent semantic variables, followed by those under dependent latent semantic variables.
m, 5.0 8.6 0. 08
30 D. 5.0
00
00
00 on 5a
56 9.5
00
D. &O
m.....
D. 00 00 30
50
00 03
0.0
00 50
DA 0.
03 3.
5.0 DO
LR L0
1O001
1D13
100
101000
5. 10
0.0
0.1
5. 0.0 0O L.
10
2.8
0.0 0.0 00
1.00
0.1 0.2
0X
1.0
LM0
LO
03
2.3
DE 9.2
L.
6.9
5.000 00
2.0
0.D U.D 0
1.0
2.6 1.0010000 1.0
0.2
1.0
1. 1.0
BL2 84 LO L.
50.00
1.0
0.0
10
0.1
0.0
Sa 5.0 0.0 0.0
0.0 0.5
1.6
0.0 0.0 00
0.
3.0 C/
2.0
0.2
0.5
0.2 0.2
0.5
0.0
5-500 0.0 0.0
0.0 0.2 2.5 0.0 00
2.5
0.0 0.0 0. 00
2.0 C 0.5
2.0
0.2 0:3
0.6 0.7
1.0 2.0
B.2 0.1
0.4
0.5 0.0 0:
0:1 2.0 2.0
5.0-5.0 0.0 0.2 2.0
0.0
2.0
0.1 0.2
2,8
2.0
DLL
[2] (311)))) (51 16) 17) (61
191(10)
2 121133
[3] [G) (71 (8) (0]
1 121
(31
[5]
[6] 19011
1. [21
13) [4] [51 [6] [71 (9]100
selection bias, &,
selection bias, &
selection bias, &
Selection bias, &
Figure 12: Evaluating linearity of learned representations under selection bias. Left to right: predic- tions using representations 2y and 2, under independent latent semantic variables, followed by those under dependent latent semantic variables.
Ablation studies. We perform two ablation studies to further examine the robustness of our theoretical findings.
First, we investigate the impact of assigning an incorrect representation dimension. Specifically, we consider scenario in which the true dimension of the selected semantic variables is 3 (with selection
La (3), but we intentionally set the representation dimension to 5. As shown in Table 4, in the independent case, all selected semantic variables are successfully preserved, while omitted semantic variables are effectively discarded. In contrast, the dependent scenario yields significantly different patterns of Rt2 scores compared to those obtained using the correct representation dimension (see Figures 3 and 12). These results suggest that redundant representation dimensions tend to encode exogenous noise, potentially introducing unnecessary complexity into the learned representations.
Second, we explore the joint effect of selection and perturbation biases by defining scenario with selection bias I [8] and perturbation bias T = [2). Results presented in Table demonstrate that when both biases coexist, their effects on semantic identification remain consistent: semantic variables that are either omitted or perturbed are discarded, while unbiased semantic variables those that are selected and yet unperturbed-ant are reliably preserved in the learned representations.
Together with previous results, these findings further reinforce our theoretical conclusions in Thm. 4.1.
Table 4: The effect of using an incorrect encoding size. The representation size is set 5, whereas the true dimension should be 3. The biases are defined as La = [3] and II = 0.
R2 of Predicting Latent Semantic Variables under I, [3 and 1, 0
Setting
Reps.
81
82
$3
84
85
96
87
88
8g
810 my
m.,
linear
2,
0.98
0.96
0.98
0.01
0.03
0.03
0.02
0.04
0.01
0.02
0.02
0.00
independ.
4,
1.00
1.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.04
2,
0.98
0.99
0.99
0.00
0.02
0.02
0.03
0.04
0.00
0.00
0.00
0.00
non-lin.
2, 1.00
1.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.06
2,-
0.98
0.99
0.99
0.14
0.23
0.16
0.10
0.22
0.42
0.36
0.01
0.00
linear
dependent
2, 1.00
1.00
1.00
0.13
0.20
0.13
0.09
0.20
0.41
0.34
0.00
0.01
2,
0.99
1.00
0.99
0.13
0.22
0.16
0.15
0.20
0.43
0.36
0.02
0.00
non-lin.
1.00
1.00
1.00
0.13
0.20
0.13
0.09
0.20
0.42
0.35
0.00
0.05
37

---

m, 6.0 on 3.6
5.6
6.0
6.0
5.6
6.0 0. 6a
6.0
06
mh-00 0
2-2
5.0
8.8
0.8
5-0
9-0
00 50
0.0 6.0 0
0-3
5-6 06 00
5-
5.6 60
0. 6.
02 00
00
68 0.5 6.0
0
9.3
5.6
on
s0 sa
51-5.0
6.
5.0
6.0
B.
8303
Sx 5.0
8.0
0.
0.2
6.0
10
5a.0.0 00
5.0
60
00
BED
0.0 0.0
0.9
3.6
00 EU
1.0
0.1
1.0010000 03
0.3
63
55-0.0 00
3.5
8.0
60
D
2.
0.0 6.0000000 0 0.3
2.
0.1
05
10000000000
61 02
10
$6-0.0
0.0 0.0 0
2.0
0200
3.000.000000
02
5.65.0
5a.60
0: 10 3.0
0O RTB
1.0
0.
1a
L0
1.6
10000
R.D 1.
LD
5.0 2.0
1.0
3.0 3.0 1.0
1.0
(9) [8
DI (8) [5] [4] [5] (21 R
[O [8 (6 1S) [AT [3]
(55T22(L (1)
[91 [8] [6] 11111111111 [ST [S (2)
[9] [8]
[P]
F11111 [4] [5] (2)
perturbation bias,
perturbation bias,
perturbation bias,
perturbation
Figure 13: Linearity of learned representations under perturbation bias. Left to right: predictions using representations ğ and 2, under independent latent semantic variables, followed by those under dependent latent semantic variables.
Table 5: Coexistence of both selection and perturbation biases. The biases defined Lo [2).
[8] and
I,
Rf of Predicting Latent Semantic Variables (L.
[8), 1p
[2]
Setting
Reps.
82
85
56
810
111
R₂
0.00
0.01
0.97
0.98
0.97
0.95
0.99
0.98
0.00
0.00
0.00
0.00
linear
independ.
2,
0.00
0.98
0.98
0.99
0.98
0.00
0.00
2₂
0.00
0.01
0.98
0.98
0.98
0.98
0.96
0.99
0.98
0.00
0.00
0.00
non-lin.
2,
0.00
0.00
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.00
0.00
0.00
2,
0.67
0.57
0.97
0.99
0.99
0.98
0.99
0.97
0.63
0.63
0.00
0.00
linear
dependent
2,
0.64
0.53
0.98
0.97
0.99
0.98
0.99
0.98
0.61
0.60
0.00
0.00
0.68
0.57
0.99
0.99
0.99
0.99
0.99
0.99
0.64
0.64
0.00
0.00
non-lin.
2,
0.65
0.53
0.99
0.99
0.99
0.99
0.61
0.00
D.3 Additional Downstream Results
We further report downstream task performance under varying perturbation bias settings. Specifically, the preserved semantic variables are sequentially reversed-staiee sttrtiing from semantic index 10 and incrementally expanding until the full semantic set included. The results shown in Figure 14 indicate that, in general, semantic variables critical to downstream tasks must be preserved in the learned representations achieve high performance. This observation holds across both independent and dependent latent semantic settings.
1.0 Regression, Independent
Regression, Dependent
Classification, Independent
Classification, Dependent
1.6
1.0
0.6
3.5
3.4
244
02
52
02-
[M]0)1 [H]])]]))] perturbation Blas, R,
[M]1/166) perturbation blas, 9,
(2)) perturbation blas,
perturbation blas, &,
Figure 14: Downstream performance of pretrained representations 2, under perturbation bias. Top: in-distribution (ID) regression performance. Bottom: ID classification and outto--distributtin (OOD) generalization.
E Experiment Details on MPI3D-Complex Dataset
We provide additional details on the MPI3D-Comple dataset that are not fully covered in $ 5.2. In App. E. we comprehensively describe the experimental setup, including a dataset overview, selection and perturbation bias configurations used generate text, model architecture, and training parameters. App. E.2, we present additional results, including the use of a linear classifier for predicting latent factors and an ablation study encoder dimensionality.
38

---

E.1 Detailed Experimental Setup
MPI3D-Complex dataset. MPI3D-Complex [19] contains 460,800 real-world images of resolution 64 x 64 x 3, spanning all combinations of seven mutually independent, discrete factors (see Table 6). We designate horizontal and vertical positions (hori., vert.) as image-specific due to their low- level visual nature, while treating the remaining five as semantic variables for representation learning and evaluation purposes.
Text latent factors. Text descriptions are generated from the ground-truth semantic attributes using content- mappings (see Table 6). Under unbiased settings, all five semantic attributes are included. For text-specific variation, a generation template is chosen via a discrete latent variable m, (niform([3) (3)), each corresponding to a different human- human-written written sentence structure.
Table Latent variables MPI3D-Complex and corresponding content words in text.
Factor Name
Distribution
Content Words
Object color (color)
Uniform( (0,...,.3)
yellow, green, olive, red
coffee-cup, tennis-bally croissant, beer-cup
Object shape (shape)
Uniform (f0, (O,.....))
Object size (size)
Uniform( (0, 13)
small, large
Camera height (cam.)
Uniform ( {0, (0,....2)) 2})
top, center, bottom
Background color (back.)
Uniform ( 2})
purple, sea-green, salmon
Horizontal axis (hori.)
Uniform( {0, ..... 39})
(image-specifie factor)
Vertical axis (vert.)
Uniform( {0,.............. (O.....,39) 39]) (image-specific factor)
Text generation under misalignment settings. We generate text for each image under various selection and perturbation bias settings investigate the effects of misalignment. To ensure computa- tional tractability- since exhaustively enumerating all possible configurations is both infeasible and unnecessary-wo adopt progressively incremental strategy for introducing biases into the latent semantic variables, as outlined in Table 7.
In selection bias settings (where the perturbable set I, is empty), textual descriptions include only the content words corresponding to a subset of the true image semantic variables. We define five incremental settings: ① (color), ), 2 (color, shape), 3: [color, shape, size), D: [color, shape, size, cam. ), 3 all five attributes. The specific text generation templates associated with Jete; under each selection setting detailed in Table 8.
In perturbation bias settings (where all semantic indices are selected), we apply random substitutions to subset II. For each factor in Ilp. its text value is replaced with randomly sampled alternative with probability 0.9. Five perturbation configuration are used: @: ②: (back.) ), 3: (cam., back.) ).
4 (size, cam., back. J, 5 (shape, size, cam., back.
Table 7: Misalignment settings for text generation of MPI3D-Complex dataset. Setting Selection Bias, Lo
Perturbation Bias, II,
(color)
0
[color, shape)
[back.]
(color, shape, size]
(cam., back.
(color, shape, size, cam.)
(size, cam., back.
(color, shape, size, cam., back.
(shape, size, cam., back.
Training details. For each setting, the dataset partitioned into training, evaluation, and test subsets in fixed ratio of 44,720 23,040 23,040. Across all configurations, train the image and
39

---

Table Text generation templates for MPI3D-Complex dataset for different selection settings.
Setting
Text Generation Templates for Each Selection View
"An object colored [color)."
"It has a (color) appearance." "Something with [color]."
"A (shape) that is (color)."
"The (color) (shape)."
"An object shaped like a (shape), colored (color)."
"A ( size] |shape] in (color)."
"[color), size], (shape)."
"The object is size), shaped as a (shape), and colored (color)."
"A size]
shape (hapel in (color), seen from (cam. )."
"Viewed from ( [cam. J. (color), size) (shape)." "A size (shape) with (color), perspective: (cam. )."
"A size] { (shape) in [color). viewed from (cam [back.
5
"From ccam. ). you see a (color), size) [shape), [back. }."
"A size] (shape), (color). placed [back. ), observed from (cam.)."
text encoders for 200,000 steps using the training subset, averaging results over three random seeds. The training objective is the multimodal contrastive loss LuMCL defined in Eq. (1), and optimized using the Adam optimizer with an initial learning rate of 10-5 a batch size of 256, and temperature parameter 7 = 1.0.
For image encoding, input dimensionality of 100. The output dimensionality adjusted according to the number of unbiased semantic factors under each bias setting.
ResNet18 backbone followed by fully connected layer with fixed
For text encoding, we tokenize text using the nltk. PunktTokenizer, following the procedure in [13]. Tokenized sequences are transformed into two-dimensional one-hot embeddings and pro- cessed using a convolutional neural network (CNN) with variable number of layers, determined by the shape of the tokenized input. The output dimensionality of the CNN is configured to match that of the image encoder, ensuring compatibility in the joint representation space.
The encoding size is generally set to match the number of unbiased semantic dimensions, i.e., dim( However, when this number less than we set the encoding size to ensure minimal representational capacity. An ablation study on this design choice is provided in the following section.
Evaluation metrics. After training the representations, we freeze the encoders and train both a linear classifier (logistic regression) and nonlinear classifier (a two-layer MLP with ReLU activation) for each setting and each image latent factor. Classifiers are trained on the evaluation subset for 10,000 steps.
Performance assessed on the test set using the Matthews Correlation Coefficient (MCC), computed separately for each latent factor. MCC is chosen for its robustness in evaluating binary classification performance under class imbalance.
E.2 Additional Results
Linearity of learned representations. We evaluate the linear separability of the learned representa- tions by reporting MCC scores for predicting each latent factor using a linear classifier. As shown in Figure 15, and in comparison to the nonlinear results in Figure 6, the findings reveal that certain latent semantic variables-ae most notably size are not linearly embedded in the learned representation space when training image-text pairs with MMCL.
In contrast, factors such color, shape, cam., and back. exhibit strong linear separability, suggesting that these semantie variables are linearly represented by the learned image and text encoders.
40

---

color 1:00
1:00
0.99
1.00
1:00
0.99
1.00
0.98
shape-0.0 0.00
0.89
0.87
0.70
0.70
0.00
0.89
0.79
0.67
0.69
0.00
0.90
0.88
0.70
0.70
0.00
0.89
0.79
0.67
0.70
MCC
size-0.03 0.00 0.00
0.56
029
021
0.00
0.00
0.50
0.41
0.30
0.00
0.00
0.59
0.7
010
0.00
0.00
0.52
0.41
0.29
0.00
1.00
0.00
1:00
back.-0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.01
0.00
1.00
0.00
0.00
0.00
0.00
1.00
hori. 0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
vert. 0.000
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1
80
0
$
8
@
2 8
selection biases, R,
perturbation biases, 2,
selection blases, &
perturbation biases, &,
Figure 15: Evaluating linearity of learned representations under misalignment. Left right: image features 2. under selection bias, image features 2g under perturbation bias, text features 2, under selection bias, and text features 2, under perturbation bias.
Ablations on the encoding size. We conduct an ablation study on the encoding size by evaluating a selection bias setting in which only the semantic attribute (color) ) is selected for text generation. All other training parameters are kept consistent with those used in our main experiments, except for the encoding dimensionality.
As shown in Figure 16, in contrast to training with purely numerical data, learning from image-text data exhibits sensitivity to the choice of encoding size. Specifically, when the encoding size is set to I 1-exactly matching the number of perfectly aligned semantic dimensions-the image encoder tends to be under-optimized, resulting in high variance across runs. However, increasing the encoding size leads to more stable and reliable performance.
Notably, in the presence of independent latent factors, the additional (redundant) encoding dimensions do not appear to capture misaligned semantic variables, suggesting that excess capacity does not harm identifiability in this setting.
Linear Model (LogReg)
Nonlinear Model (MLP)
1.0
1.0
0.8
0.8
0.6
color
-
back.
0.6
color
back
MCC
shape
shape
hon.
0.4
size
cam.
vert.
0.4
size
cam.
ventt.
0.2
0.0
0.0
Encoding Size
Encoding Size
Figure 16: Ablation study on encoding size under selection bias setting where only the (color) attribute is included in text generation. Left: average MCC over three runs using a linear classifier. Right: average MCC using a nonlinear classifier.
F Experiment Details on Causal3DIdent Dataset
We provide additional details the Causal3DIdent dataset that are not fully covered in $ 5.3. In App. F.I, we offer a comprehensive description of the experimental setup. including the image and text latent factors, the image generation process, the design of selection and perturbation bias settings for text generation, and training configurations. In App. F.2, we present supplementary results, including analyses of the learned text representations and assessments of the linearity of the learned representations.
F1 Detailed Experimental Setup
Image latent factors and image generation. Following prior work [83, 68, 13, 78), we utilize the Cidsal3DDdent dataset to synthesize images from a predefined latent causal structure. Images are generated using the Blender renderer [9], which applies a complex rendering function parameterized by 11 input variables. In our configuration, the object's z-position fixed, leaving 10 latent factors that govern image generation.
41

---

These include 3 discrete oarjables-ojeet shape (shape), and object positions along the horizontal (x_pos) and vertical (y-pos) axes axes-and 7 continuous variables: object color (color), spotlight position (s_pos) and color (s _s_colo), background color (b_ (b_color) and the three object rotation angles (alpha, beta, gamma). We treat the rotation angles (alpha, beta, gamma) as image-specific latent variables, while the remaining factors are considered semantic latent variables, structured according to the causal graph shown in Figure 17.
We synthesize 80,000 samples for MMCL training, 10,000 samples for classifier or regressor training, and another 10,000 samples for test-time evaluation. Images are rendered resolution of 128 128 x 3.
latent semantics
image-specific factors
shape
x_pos
y_pos
$_pos
s_color
alpha
beta
color
b_color
gamma
images
Figure 17: Latent causal model governing image generation in the Causal3Dident dataset. Rectangular nodes represent discrete latent random variables, while elliptical nodes denote continuous ones. Object shape (shape), horizontal and vertical position (x_pos, y-pos), object color (color), spotlight position and color (s_pos, s_color), and background color (b_cool________ color) are treated as latent semantic variables shared across modalities and potentially subject to misalignment. contrast, the rotation Ingles-appha, alpha, beta, and gamma considered image-specific latent factors.
Text latent factors. We discretize the continuous variables color, s_ color, and b color using COLORS for color, 1SS4_COLORS for s_color, and XKCD COLORS for b_ color. While color remains continuous in the underlying latent representation, we simulate partial information loss for the spotlight position (s_pos) during the generating mapping form text. As result, the generated textual descriptions of s_pos do not constitute an information information-pprserrinng preserving transformation.
sampled image semantic variables mapped to distinct color palettes: TABLEAU_COLORS
To introduce text-specific variation, we employ five manually designed templates generate text from the latent factors under each bias setting, adapting the text rendering pipeline from [13]. A complete list of latent factors and their types is provided in Table 9.
Text generation under different misalignment settings. Following the MPI3D-Complex experi- ments, we explore a series of incrementally increasing selection and perturbation bias configurations to introduce varying degrees and types of cross-modal misalignment. These settings enable sys- tematic investigation how different forms of alignment impact representation learning. Each configuration indexed using circled numerals and summarized in Table 10.
For the perturbable semantic variables each perturbation setting, we randomly sample the cor- responding text semantic values. Specifically, for discrete variables such x_pos and y y_pos,
42

---

Table Latent factors of Causal3Dldent and corresponding content words in text.
Factor Name
Image Modality
Text Modality
Content Words
teapot, hare, dragon, cos,
shape
(anform((O.....555
armadillo, horse, head
x_pos
Uniform((O, 1, 1,2))
Uniform({0, 1,2))
left, center, right
y-pos
Uniform( (0. 1,2))
Uniform (0, 1,2))
top, mid, bottom
northwest, northeast, center, southwest, southeast
8_po8
Uniform([0. (0,1])
Uniform (0. 1))
color
[(x_poe + y-poe) + Unniform([. IJ)
Up to 10 colors
Color names SEBLEA_COLORRE
8_color
{a__oo + Uniform 1}})
Up to 147 colors
Color names in C994_COLORS
b_color
{ + e_color Uniform((. 1)))
Up to 954 colors
Color names XKCD_COLORS
alpha
Uniform(Uu. (U, 1)))
beta
Uniform( 00, 1))
gamma
Uniform( 1))
phrase
Uniform 5)))
sample uniformly from the set {0, 1, 2); for continuous variables, we sample uniformly from the interval [0, 1].
Table 11 provides the text generation templates associated with each selection setting. Representative image- texttee-etttteeeee text pairs generated under different selection and perturbation bias configurations are shown in Figure 18.
Table 10: Misalignment settings for text generation of Causal3DIdent dataset.
Setting Selection Bias,
Perturbation Bias, L,
(shspe)
{x_pos, y-poe, s_pos, color, s_color, b_color)
(shape, x_pon)
(y-pos, s_pos, color, s_color, b_color)
(shape, x_poe,
{s_pos, color, s_color, b_color)
(shape,
s_pos)
(color, s_color, b_color)
(shape, x_pos, y-pos, s_pos, calor)
(s_color, b_color)
(shape, x_pos, y-pos, s_pos, color, s_color]
[b_color]
(shape, x_pos, y y-pon, pop,,, calor, s_color, color)
Training details. Across all experimental settings, we train the image and text encoders for 100,000 steps on the training subset, using three different random seeds to ensure robustness. The training objective is the multimodal contrastive loss LAMCU defined in Eq. (1), and optimized using the Adam optimizer with an initial learning rate of x 10 a batch size of 256, and a temperature parameter 1.0.
For both the image and text encoders, we adopt the same architectures used in the MP13D-Complex experiments. The encoding dimensionality is adjusted according to the bias setting: for selection settings ( through D the encoding sizes are set to 3, 3, 4, 5, 5, 6, and 7, respectively; for perturbation settings ! through D. the encoding sizes are assigned reverse order: 6. 5, 5, 4, and 3.
Evaluation metrics. After training the representations, freeze the encoders and, for each bias setting, train both linear classifier (logistic regression) and a nonlinear classifier (a two-layer MLP with ReLU activation) for each discrete latent factor. Similarly, for continuous latent factors, train both a linear regressor and a nonlinear regressor (a two-layer MLP with ReLU activation). All classifiers and regressors are trained for 10,000 steps using the evaluation subset.
We assess the predictive performance of the learned representations by evaluating their ability to recover the ground-truth latent factors corresponding to their respective modalities. This evaluation accounts for the fact that some semantic variables may appear in discrete or continuous form, depending on the modality and rendering process.
43

---

Selection settings
dragon is In the Image.
At the right of the image, there is object the form of dragon.
The bottom-right of the image shows dragon.
At the bottom-right image, an object the center.
dragon, illumination from
"tab:purple" dragon is at the battom-right the Image, Illuminated by a light from the center.
At the bottom-right of the image, there is "tab:purple" object in the form of dragon, with iilumination from a "lime" spotlight the center.
"tab:purple" dragon is the bottom-right of the image, illuminated "lime" spotlight from center, against "xked:bbuu/lpuple" background.
Perturbation settings
At the mid-center of the image, there is "tab:cyan" object in form teapot, standing out under "dodgerblue" spotlight from the center, against "xked:minty green" background.
At the mid-center image, there "tab:cyan" object under "dodgerblue" spotlight from the center, against a "xkod:fire engine red" background.
teapot, standing
At the mid-center of the picture is teapot in "tab:cyan" color, bathed in a "lawngreen" spotlight coming from the center, with "xkcd:fire engine red" background.
image, there "tab:olive" object under "blue" spotlight from the center, against "xkcd:pink red" background. teapot, standing out
At the mid-center
At the mid-center of the picture is teapot in "lab:orange" color, bathed in a "red" spotlight coming from the northeast, with "xked:strong blue" background.
"tab:olive" teapot is at the top-center of the Image, Illuminated by "springgreen" spotlight from the center, against "xkodiorange" background.
At the bottom-right of the picture teapot in "tab:olive" color, bathed "red" spotlight coming the northwest, "xkcd:bright magenta" background.
Figure 18; Example image-text pairs from Causal3DIdent under different selection bias settings. The left panel shows randomly selected images; the right panel presents the corresponding text from top to bottom, each generated under selection settings ( to ? with no perturbations, and perturbation bias ① ! with full selections.
Prediction performance is measured on the test set using the Matthews Correlation Coefficient (MCC) for discrete factors and the coefficient of determination (R²) for continuous factors. Metrics are computed separately for each latent factor.
F.2 Additional Results
Results of text representations. We now turn to the analysis of the text representations learned by MMCL. As shown in Figure 19, we observe patterns similar those found in the image modality. In particular, discrete latent semantic pariables-such identified, provided they are unbiased and consistently aligned across modalities. A notable case is s_pos, which is a continuous latent factor in both modalities but is mapped to only five discrete tokens in the text observations, rendering the text generation process non non-invertible. Despite this lossy transformation, the model achieves relatively high R²² suggesting that the learned text repre- sentations remain strongly influenced by the alignment objective, even when semantic information is partially lost.
shape, x_pos, and y-pos-
reliably
For other latent semantic variables that are continuous in the image modality but discretized in the text modality such as color, color,, and b_color-www observe varying degrees of performance degradation. This drop in performance is likely attributable not only to the quantization continuous values but also to semantic ambiguity introduced during text generation. Notably, all three attributes correspond to different aspects of color, yet they may be described using overlapping vocabulary drawn from distinct color palettes. For instance, tab: cyan from TABLEAU LABLEAUCCLLOSS
COLOORS refers to object
44

---

Table 11: Text generation templates for Causal3Dldent dataset under different selection settings.
Setting
Text Generation Templates
"A (shape) is visible."
"A [shape) is in the image."
"The image shows a (shape)."
"The picture is {shape)."
"There is an object in the form of (shape)."
"A (shape) is visible, positioned at the (x_pos) of the image."
"A [shape] J is at [x_pos} the image."
"The (x_pos) of image shows (shape)."
At the [x_pos] of the picture is (shape)."
At the (x_pos) } of the image, there is object in the form of (shape)."
"A (shape) is visible, positioned at the jy_pos]-(x,, [x__]os of the image." "A (shape) is the (poo)-(x__pos of the image."
"The -y-ppos)(x__pp poa] of the image shows a (shape)."
At the [y_pos]-tt [x_pos] of the picture is [shape]."
"At the [y-pos [x__os) of the image, there is an object in the form of (shape)."
"A (shape) is visible, positioned at the (_.pos)(x_pos]_ with spotlight shining from {s_pos)."
"A (shape) is at the (__poss-(x_pos)) illuminated by a light from (s_pos)."
"The [y_pos) -oc.pos)s(___p (x__oos shows (shape), highlighted by a light from (s_pos}." "At the | (xx_pooa] is a (shape), under a light from (s_pos)."
"There is a [shape) [y-pos]-(x_pos lit from (s_poa)."
"A (shape) (color) color is visible (y_pos) |-xx_pss] with spotlight from (s s__os)."
"A (color) (shape) is at (y_pos)-(x_pos), lit from (s_pos}."
"The area [y-poaa-(xx_po_ shows a [color] (shape), under light from (a_pos)."
A [color) (shape) is illuminated at (7-poa)-(x_pos_ from (s_pos)."
"A (color) object shaped like (shape) is lit from (a_poa)."
[color) [shape] lit by [s_color) spotlight from [s_pos], J. at (x_pos)-(x_pos)." "At [y_possj((__pos, [color] (shape) is under a (s_color) light from (s_pos)."
"The (shape) is (color), under a [s_color) light at [s_pos]."
A [color) (shape) under (s_color) spotlight [s_pos), located
"A [color) (shape) stands under (=_colrr) light from (e_pos)."
"A (color) (shape) under (s_ [__col))) spotlight [s_pos), with (b_color) background, at (y_pos))(x__oo_."
"At [y_pos)-[x_po_ a (color) (shape) ) is under a (s_color) light from (s_pos], against a [b_color] background."
"A [color] (shape) appears (y-pos)-(x_pos]] lit by [s_color] from (s_poa), with bb_color) background."
"The seone shows a (color) (shape) under (s_color) lighting at (s_pos), with a (b_ (b_color) backdrop."
"A (color) object shaped like (shape), under (s_color) spotlight at (s_pos), with [o_oo) background."
color (color), cyan from CSS4_COLORS XKCD_COLORS COLORS indicates background color (b_coo co_colr).. Despite referencing different latent variables, these tokens all contain the word cyan, which is tokenized identically by nltk. Pattkk.Pnktookkk PunktTokenizer resulting in ambiguity in the text observations. These findings highlight the importance of using distinct and unambiguous content words when representing semantically different concepts in multi- modal learning lhariing-paricularryy when the text modality is not treated merely as an auxiliary input for visual representation learning.
COLORS describes spotlight color (s_ color,, and xkcd: cyan from
Interestingly, the identification of x_pos and y-pos the text representations does not lead to improved predictability of color, in contrast to what is often observed in the image modality. This aligns with our theoretical expectation that perturbation biases disrupt the underlying causal structure in the image latent space.
45

---

Regarding the text-specific factor phrase, we find it to be partially encoded in the learned represen- tations. This contrasts with the image-specific continuous factors, which are consistently omitted. The partial identifiability of phrase is likely attributable to its discrete nature, which violates the conditions typically required for modality-specifft factors to excluded consistent with findings reported in [13]. Moreover, this effect appears more pronounced under selection bias settings, partic- ularly when the encoding dimensionality exceeds the true dimensionality of the unbiased semantic subspace.
Overall, the behavior of the learned text representations provides empirical support for our theoretical analysis, even under conditions where certain modeling assumptions are relaxed or violated.
1.0
phrase
0.79
0.34
0.45
0:37
0.41
0.09
0.11
phrase. 0.17
0.12
0.05
0.07
shape
1.00
0.96
0.35
0:97
0.38
0.99
0.97
0.8
shape 1.00
1.00
0.97
0.96
0.94
0.99
0.96
0.8
/CCC
x_pos-
1.00
1.00
1.00
1.00
1.00
1.00
x_pos
0.03
1.00
1.00
1.00
1.00
1.00
1.00
Y_Pos-
1.00
1.00
1.00
1.00
1.00
0.6
Y_pos 0.03
0.00
1.00
1.00
1.00
1.00
1.00
0.6
s_pos
0:92
0,88
0.88
0.81
s_pos: 0.00
0.00
0.00
0.94
0.94
0.89
0.86
0.4
color.
0.74
0.71
0.68
0.4
color: 0.06
0.00
0.00
0.01
0 60
0.70
0.97
s_color.
0.16
0.14
s_color 0.02
0.00
0.00
0.00
0.00 0.15
0.16
-0.2
b_color.
0:35
0.2
b_color 0.00
0.00 0.00
0.00
0.00
0.00
0.45
8
60
de
0.0
selections settings
perturbations settings
Figure 19:
Prediction of latent semantic variables under misalignment with text features. R²2 for continuous and MCC for discrete factors. Left: Selection bias. Right: Perturbation bias.
Linearity of learned representations. We assess the linear separability of the learned image and text representations by evaluating the performance of linear classifiers and regressors trained to predict each latent factor. As shown in Figure 20 and Figure 21, and in comparison to the nonlinear prediction results, the findings indicate that not all latent semantic variables are linearly embedded in the representations learned through MMCL. In particular, the semantic factor color consistently demonstrates poor linear predictability. suggesting that it is encoded nonlinearly in both modalities. Conversely. factors such as x_pos and y-pos exhibit strong linear separability in certain settings, indicating that these semantic variables are more directly captured in the latent space of both the image and text encoders. Overall, these results suggest that the linearity of the learned representations dectoddde and shaped by both modality-specife encoding strategies and the underlying structure of the input data highlighting an important direction for future investigation.
alpha 0.00
0.00
0.00
0.00
0.00
0.00
0.00
alpha- 0.00
0.00
0.00
0.00
0.00
0.00
0.00
beta- 0.00
0.00
0.00
0.00
0.00
0.00
0.00
beta 0.00
0.00
0.00
0.00
0.00
0.00
0.00
gamma. 0.00
0.00
0.00
0.00
0.00
0.8
gamma. 0.00
0.00
0.00
0.00
0.00
MCC
shape 1.00
0.89
0.41
0.52
0.58
0:60
0.65
shape 1.00
0.92
0.58
0.65
0.60
0.75
0.81
x_pos:
1.00
0.85
1.00
0.6
x_pos-
0.99
0.90
0.6
y_pos: 0.00
0.01
0.87
0.89
0.90
0.84
0.89
y_pos. 0.00
0.01
0.73
0.92
0.95
0.98
0.91
s_pos 0.00
0.00
0.00
0.88
0.55
0.80
0.4
0.00
0.00
0.00
0.95
0.91
0.68
0.86
0.4
color: 0.00
0.36
0.58
0.57
color. 0.00
0.22
0.52
0.56
color 0.00
0.00
0.00
0.45
0.28
6.67
0 80
-.2
s_color 0.00
0.00
0.00
0:48
4:
0 0:68
0.89
-.2
b_ colorr
b. Socolrr-
-0.0
-0.0
4
@
+
selections settings
perturbations settings
Figure 20: Evaluating linearity of image features under misalignment settings. R for continuous and MCC for discrete factors. Left: Selection bias. Right: Perturbation bias.
G Details on OpenCLIP Case Study
G.1 Concepts Taxonomy and Data Collection
Concepts taxonomy. To validate the representations of pretrained OpenCLIP models, we curated a set of 146 concepts organized into 15 distinct concept groups. Each concept group is treated as a separate solution space, and we aim to ensure that the concepts within each group are mutually exclusive. Table 12 presents all the concepts used in the case study. For simplicity, abbreviations are used in the main text, with their full meanings provided in the concept group column.
46

---

phrase-0.01
1.0
phrase
0.00
1.0
shape
1:00
0.8
shape
0.8
MCC
8_pos
x_pos
Y_pos
0.6
y_pos
0.6
s,pes
0.4
s_pos
s_color
s. s_color
b_c _oclorr-
b_color
-0.0
@
selections settings
perturbations settings
Figure 21: Evaluating linearity of text features under misalignment settings. R for continuous and MCC for discrete factors. Left: Selection bias. Right: Perturbation bias.
Table 12: Taxonomy of concept categories with abbreviations, grouped by conceptual relevance.
Group
Explanation
Concept Name
Animal
animals
dog. cat, horse, bird, elephant, giraffe, cow, zebra, rabbit, duck shirt, pants, dress, shoes, hat, jacket, skirt, tie, hoodie, socks red, bluc, green, ycllow, black, white, orange, pink, gray, brown pizza, burger, sandwich, salad, cake, coffee, tea, heer, ice cream, noodles chair, table, phone, laptop, car, bottle, bag, cup, backpack, television chef, teacher, athlete, doctor, engineer, artist, pilot, firefighter, police officer, lawyer
Clothing
clothing
Color
colors
Food
food
Object
object
Role roles
Scene
scenes
beach, kitchen, forest, street, park, office, bedroom, classroom, stadium, playground bus, truck. airplane, train, motoreycle. bicycle, boat, van, taxi, scooter sandstorm, drought, rain, snow, fog, rainbow, storm, moonlight, overcast, frost
Vehicle
vehicles
Weather
weathers
Texture
texture
glossy, matte, rough, smooth, fuzzy, silky, grainy, wrinkled, slippery, furry close-up. wide shot, top-down, low angle. high angle, aerial, shallow-depth, long exposure
POV
viewpoint
Emot.
mental state
tired, focused, surprised, proud, shy, bored, confused, excited, thoughtful, nervous
Postproc.
post- post-proessing
hdr, desaturated, color-filtered werrsharrenned saturated, sepia, heavily edited, bokeh, tilt-shift
Trait
moral bias
easy-going, cold-hearted, evil, criminal, immoral, pure, corrupt, hero, cheater, deserving
Stere.
stereotypes
exotic look, ethnic look, slacker, aggressive look, gangster look, terrorist look, illegal look, privileged, ghetto look
Data collection. We collected up to 200 images per concept by using the concept names as search queries on Flickr, filtering for content licensed under Creative Commons. To ensure relevance, all images were manually reviewed, and those deemed unrelated to the queried concept were removed. The remaining images were retained for subsequent experiments. Table 13 summarizes the number of images retained for each concept. It is important to note that some concept groups, such those related to stereotypes, include terms that may reflect societal biases. do not endorse or perpetuate these biases; rather, these concepts are included solely to assess whether pretrained models encode or reproduce such biased associations their representations.
G.2 Details on Caption Frequency Statistics
To assess caption coverage frequency for each concept, we first enumerated plausible synonyms for every concept name. We then computed the percentage of captions in the LATON-400M dataset that included either the original concept term or one of synonyms. Figure 5.4 presents the group-wir wosu average coverage rates.
47

---

Table 13: Concept-wise image counts grouped by concept category for zero-shot evaluation.
Group
Concept
Count
Group
Concept
Count
Group
Concept
Count
Animal
dog bird
124
Animal
112
Animal
horse
109
Animal
Animal
elephant
Animal
giraffe
Animal
cow
88
Animal
zebra
81
Animal
rabbit
77
Animal
duck
65
Clothing
shirt
113
Clothing pants
104
Clothing
dress
101
Clothing
shoes
96
Clothing hat
92
Clothing
jacket
Clothing
skirt
Clothing tic
78
Clothing
hoodic
Clothing
socks
Color
130
Color
blue
127
Color
green
119
Color yellow
114
Color
black
111
Color
white
109
Color orange
97
Color
pink
91
Color
gray
88
Color brown
82
Food
pizza
burger
sandwich
Food salad
101
Food
cake
Food
coffee
93
Food tea
90
Food beer
85
Food ice cream
82
Food noodles
78
Emot. tired
58
Emot.
focused
55
Emot. surprised
51
Emot.
proud
49
Emot.
shy
45
Emot. bored
Emot.
confused
Emot.
excited
Emot. thoughtful
39
Emot.
nervous
36
Trait
easy- going
29
Trait
cold-
27
Trait evil
25
Trait
criminal
24
hearted
Trait
immoral
Trait
pure
Trait
corrupt
Trait hero
18
Trait
cheater
17
Trait
deserving
15
Object chair
120
Object
table
115
Object
phone
112
Object laptop
110
Object
108
Object
bottle
105
Object
bag
Object
cup
Object
backpack
Object television
90
Postproc.
hdr
33
Postproc.
desaturated
32
Postproc.
color-
31
Postproc. over-
30
Postproc.
saturated
28
filtered
sharpened heavily edited chef
Postproc.
sepia
Postproc.
Postproc
bokeh
Postproc.
tilt-shift athlete
22
Role
88
Role
teacher
86
Role
85
Role
doctor
83
Role
engineer
80
Role
artist
Role
pilot
74
Role
firefighter
71
Role
police offi-
Role
lawyer
Scene
beach
cer
Scene
kitchen
115
Scene
forest
110
Scene
street bedroom
107
Scene
park
104
Scene
office
102
Scene
98
Scene
classroom
Scene
stadium
Scene
playground
Stere.
exotic
Stere.
ethnic
Stere.
slacker
look
look
Stere.
aggressive
18
Stere.
gangster look privileged
17
Stere.
terrorist
16
look
look
Stere.
illegal
Stere.
Stere.
ghetto
look
look
Texture
glossy
47
Texture
matte
45
Texture
rough
43
Texture
smooth
41
Texture
fuzzy
40
Texture
silky
38
Texture
grainy
Texture
wrinkled
Texture
slippery
33
Texture
furry
Vehicle
bus
106
Vehicle
truck
Vehicle
airplane
102
Vehicle
train
100
Vehicle
motorcycle
97
Vehicle
bicycle
93
Vehicle
boat
89
Vehicle
86
Vehicle
taxi
83
Vehicle
scooter
80
POV
close-up
55
POV
wide shot
top-down
POV
low angle
POV
high angle
47
POV
acrial
POV
shallow-
42
depth
POV
long expo-
39
Weather
sandstorm
66
Weather
drought
63
sure
Weather
rain
Weather
snow
Weather Weather
fog
Weather
rainbow
53
Weather storm
moonlight
47
Weather
overcast
45
Weather
frost
42
48

---

To explore
variability, Figure 22 provides a more granular view of concept-level coverage. It reveals that even within a single concept group, the degree caption coverage can vary substantially across individual concepts. As expected, commonly observable concepts, such object types, colors, and food items, tend to appear more frequently in captions. In contrast, abstract or subjective concepts, such as mental states, post-processing artifacts, moral judgments, and social stereotypes, are significantly underrepresented. This discrepancy may arise from several factors: ethical concerns leading annotators to avoid certain terms, unconscious omission of nuanced details, general challenges in describing abstract visual content.
G.3 Zero-Shot Evaluation Details
To probe the representations of pretrained OpenCLIP models, we evaluate two variants trained on the LAION-400M dataset: ViT-B-32 and ViT-L-144 Using the collected image samples, we perform zero-shot analysis by employing the concept names as text prompts. Each concept group is treated as a distinct solution space. To mitigate the impact of sampling bias the test dataset, we report macro-averaged Fl scores at the group level.
As shown in Figure 10, concept groups that are frequently captioned-defined here those with coverage rates exceeding 0.15%, are generally well represented in the multimodal contrastive learning (MMCL) models, with Fl scores above 65%. In contrast, concept groups that are severely under- captioned, often due to selection bias or ethical omission, exhibit poor representation, with Fl scores falling below 40%. The results are consistent across different model scales, further reinforcing our theoretical findings, particularly the impact of selection bias on the learned representations.
Figure 23 presents the intra- gora-grouppooo confusion matrix from the zero-shot evaluation using the OpenCLIP ViT-B-32 model trained on the LAION-400M dataset. As clearly shown, well-captioned concept groups exhibit strong diagonal patterns, indicating accurate mreictioons-weeeee concept groups affected by selection bias suffer from substantial misclassification.
This misclassification has multifaceted implications. For certain under-captioned yet potentially valuable concepts, such as Texture (texture) and Emot. (mental state), the poor performance suggests need for contrastive learning (MMCL) pretraining. Conversely, for sensitive concepts like Trait (moral judgement) and Stere. (stereotypes), which ideally should not be encoded, this under-representaaion may be desirable. In these cases, the inherent biases the captioning process may function as epistemic filters that implicitly align model representations with human ethical standards.
comprehensive and semantically rich captioning pipelines in multimodal
H
Computation Resources
All experiments were conducted on a serehppeffomance computing cluster equipped with 4xNVIDIA A A100 GPUs (40 GB each), running CUDA 12.2 and driver version 535. 161.07. The system also included an AMD EPYC 7313 16-core processor and 503 GB of RAM. For the numerical simulations, we trained over 120 models in total, requiring approximately 70 GPU-hours across GPUs. On the MPI3D-Complex dataset, we trained 36 models, consuming approximately 27 GPU-hours. For the Causal3DIdent dataset, we trained 42 models, which required roughly 25 GPU-hours across 4 GPUs. Additionally, we generated 100,000 synthetic images for the Causal3DIden dataset using Blender. Rendering was performed over four days on a separate workstation equipped with an AMD Ryzen 7700X 8-core processor (4.50 GHz) and single NVIDIA RTX 4090 GPU (24 GB).
49

---

Animal
Color 2.182%)
1.198%)
1.7
30-
122
110
15
666
600
Hebba
derk
deoute
60
Pilte
60
0.0245)
0.094%)
0/8
0.02
beger cense
para miig
cheeter
Postproc: (mean: 0.012%)
Objject (mewn: 1.62999
3.032
Role [mewn: 0.215%)
Peoentage
3.12
2002
k00
uaterta
briik
Insttoreed
cake Blerdd
enyle
earproodd
silldd
interdio lespece
alter
Scene (mean: 0.536%)
Tosture Imoen:: 0.147%
6.7
3.000
66-
Pucestoge
2.026
30024
3,0000
pmiggged
deckee
ettaie doik
sudd bugk
spstew Rook
Bespthy book
txreeiie Book
Blegal lork
alotto book
pest
Alnd
dereef
plar mord
sappery
uradled
0.031%
Vahicl (mean: 0.611%)
Weather (meen:: 0.528%
0.04
255
0.02
Aurial
chrmap wide ab
high ansaee
fore ample
Lopsboo
60
applece
muderesa wonter
Figure 22: Statistics of concept-level coverage within each group in LAION-400M captions.
50

---

MADOLA HOOPS
124
140
150
DAN
EX0
15
LOME
Preti Lake
21
LKG
PNO UNN
inddcdd Lbbel
14
TOME
EST
17
Figure 23: Confusion matrix for zero-sho shot prediction using OpenCLIP ViT-B-32 of each group.
51

---

I Discussions
In this section, reflect the limitations of the current study, propose future research directions informed by our findings, and discuss broader implications.
L1 Limitations and Future Directions
Estimating dataset biases. Our study focuses on formalizing the effects of semantic misalignment, but does not directly address the empirical estimation of dataset-specific biases such as selection and perturbation bias. These biases are often latent and difficult to observe, particularly in large-scale, web- curated datasets. Yet, identifying and quantifying them is essential for improving interpretability. data quality, and downstream robustness. This task involves challenges in defining concept taxonomies, determining annotation granularity, and choosing evaluation metrics. Our framework suggests practical strategies: selection bias can be diagnosed via concept coverage statistics over predefined vocabulary, while perturbation bias may be estimated by comparing dataset captions with those generated by strong image-to-ted models, using semantic similarity or factual consistency metrics. We view the development of robust bias identification pipelines, such as those explored by [79], as a valuable direction for future work.
Expanding the taxonomy of misalignment. Multimodal data presents a wide range of potential misalignments across modalities. In this work, we deliberately focus on semantic misalignment in image-text pairs, a particularly prevalent and influential form of bias in vision-language learn- ing. This type of misalignment central to current multimodal research due to its relevance in large-scale pretraining paradigms and real-world applications (e.g.. CLIP, ALIGN), By explicitly modeling selection and perturbation biases, our framework captures common semantic omissions and distortions that significantly affect representational quality. Extending the analysis to other forms of misalignment, such as temporal lag in lideo-language data, modality-specife dropout, or ambiguity arising from multi-entity co-occurrence, poses additional challenges and may require substantially different modeling assumptions and techniques. We view our proposed abstraction as tractable and principled foundation for understanding semantic misalignment and believe it can work as building- block for future explorations on this direction.
Modeling cross-modal semantic synergy. Multimodal representations often exhibit semantic synergies arising not only from the alignment of shared content but also from emergent meaning generated through cross-modal integration. At theoretical level, this aligns with ideas in grounded cognition:
cognition (and by extension, semantic representation) is rooted in the integration of per- ceptual, motor, and introspective modalities via simulation [3]. This parallels our formal distinction between two forms of synergy: i) Some semanties may be deterministically inferable across modali- ties-such as emotional tone from facial expression, which our latent-space alignment framework readily accommodates. 2) Other higher-order semantics (e.g., sareasm, irony, humor) emerge through non-additive, nonlinear interactions between modalities and thus require modeling joint-modality latent variables. While our current theory does not explicitly model these emergent semantic phe- nomena, establishing semantic invariance under structured misalignment is a necessary precursor. Extending our framework to capture structured, emergent cross- cross-modal modal semantics remains a promising avenue for future research.
Formulating missing data in the latent space. Our current framework addresses misalignment arising primarily from fixed selection and perturbation biases in textual annotations. However, real- world datasets, particularly large-scale, user-generated corpora such as LAION-5B [61], often display random and unstructured semantic omissions. Extending our latent variable model to accommodate such random missingness (e.g., missing completely at random or missing not at random) poses both theoretical and practical challenges. Future research should examine the identifiability consequences of randomly missing semantic variables and establish conditions that ensure partial or probabilistic recovery of latent factors remains feasible.
Linearity multimodal representations. Although our theoretical analysis guarantees identifia- bility of unbiased semantic factors up to general invertible transformations, empirical observations suggest learned representations are often approximately linear with respect to the underlying seman- tics. This motivates important open question: under what conditions can identifiability up to
52

---

linear transformation be rigorously established, without imposing overly restrictive assumptions on data geneeattng processes or latent structures? Clarifying this issue not only theoretically significant but also practically beneficial, as linear representations enhance interpretability and sim- plify downstream applications. Future studies might investigate training objectives, regularization strategies, or architectural inductive biases explicitly designed to encourage linearity. Bridging this empirical regularity with robust theoretical guarantees represents promising research frontier.
1.2 Broader Implications
Linguistic relativity and epistemic constraints in multimodal AL. Our findings offer computa- tional support for a contemporary perspective on linguistic relativity [70], which posits that language shapes human perception and conceptualization. In multimodal AI, linguistic supervision implicitly determines which aspects of the visual world are foregrounded, and which are omitted. Selection and perturbation biases in textual annotations thus act epistemic filters, constraining the conceptual space that models can represent and generalize over. This reframes dataset design as both an epistemic and normative act: inclusion or omission in annotations encodes a stance salience, relevance, or appropriateness. Two practical insights follow. First, achieving generalizable representations requires supervision that faithfully captures the intended semantic scope without systematic omis- sions. Second, curating data to intentionally include or exclude ethically sensitive or socially salient content provides a mechanism for aligning learned representations with human values [4, 11]. When annotation is treated as an epistemic commitment, dataset design becomes a central tool for shaping both semantic fidelity and ethical alignment.
Human annotation biases as signals of implicit value judgments. Beyond explicit annotation content, human annotation errors, such as consistent omissions mislabels, reveal subtle but powerful behavioral signals. These errors are not uniformly random; rather, they concentrate in dimensions that humans intuitively deprioritize under limited attention [25]. Core attributes like risk or intentionality are rarely mislabeled, while peripheral or low-salience details are more frequently neglected. Such patterns implicitly encode value hierarchy over semantic factors. This perspective invites a reinterpretation: rather than discarding annotation errors as noise, we can study them as reflections of human value structure. Systematically neglected factors often carry little perceived cost when missed, suggesting lower social or cognitive importance. These regularities form a behavioral prior, informing models not only what remains invariant to environment changes, but what is valued. Incorporating such cues enables a richer form of alignment grounded in human preferences, priorities, and ethical sensibilities [17].
53


Reviews:
[{"review_id": "Yj8v2bqx6P", "reviewer": "Reviewer_1M1L", "summary": "This paper provides a theoretical framework to understand the impact of cross-modal misalignment in multimodal contrastive learning (MMCL), focusing on image-text pairs. The authors identify two opposing views in existing literature: one that treats misalignment as noise to be filtered, and another that suggests it can be beneficial for learning robust representations.\nTo reconcile these views, the paper introduces a latent variable model (LVM) that formalizes misalignment through two mechanisms affecting the text modality, namely selection bias and perturbation bias.\n\nThe main theoretical contribution is a proof that MMCL, without any modification, inherently learns representations that are \"block-identifiable\" with the subset of semantic variables that are invariant to these biases. In other words, the model learns to represent only the semantic content that is consistently and accurately shared across both modalities, while systematically discarding misaligned (omitted or perturbed) information and modality-specific noise.\n\nThe authors then derive two key practical insights from this result: (1) To build comprehensive foundation models, one must minimize misalignment by creating detailed and faithful captions. (2) For tasks requiring out-of-distribution (OOD) generalization, misalignment can be strategically introduced to filter out spurious or environment-sensitive variables, thereby improving robustness. These theoretical claims are validated through extensive experiments on a synthetic dataset, the MPI3D-Complex dataset, and the Causal3DIdent dataset, showing that the model reliably identifies the unbiased, shared semantic factors.", "questions": "- The entire theoretical framework rests on the generative functions being smoothly invertible. However, this is rarely true in practice. It's seemingly also violated in the paper's own experiments (e.g. Causal3DIdent). If the theory holds even when its assumptions are violated, does this suggest the theorem is not capturing the essential mechanism at play?\n\n- The proof relies on Assumption 3.2, where textual perturbations do not propagate to causally dependent variables. Given that real language often has its own internal causal logic, how do you justify this simplifying assumption, and how would your identifiability guarantees degrade if it were violated?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- **Timely Unification**: The paper tackles a crucial and confusing issue in multimodal learning—the dual role of misalignment—and offers a clear, unifying theoretical perspective. The ability to explain why misalignment can be both harmful and beneficial depending on the end goal is a significant conceptual contribution.\n\n- **Strong Theoretical Result**: The core identifiability theorem (Thm. 4.1) is powerful. Demonstrating that a standard learning objective like the InfoNCE loss implicitly performs this sophisticated semantic filtering is a non-obvious and important finding. The proof that this holds regardless of the latent causal structure is particularly strong.\n\n- **Clear Practical Implications**: The paper successfully bridges theory and practice by providing actionable advice for practitioners. The corollaries on pretraining vs. robust OOD learning give clear guidance on data curation strategies, which is highly valuable for the community.\n\n- **Thorough Empirical Validation**: The experiments are well-designed and systematically test the paper's claims. The use of three different datasets provides convincing evidence for the theoretical results. The ablations further strengthen the conclusions.\n\nWeaknesses:\n- **Oversimplified Model of Misalignment**: While the formalization of selection and perturbation bias is a strength, it represents a highly structured form of noise. Misalignment in large, web-scraped datasets (e.g., LAION) is likely to be far more chaotic, containing unstructured factual errors, complex paraphrasing, and ambiguous references that don't fit neatly into the proposed categories. The experiments, which use template-based text, also do not capture this complexity.", "was_revised": true, "final_justification": "As written in my original review and the follow-up response, while I still have reserved opinions about the applicability of the theorems due to its assumptions, I acknowledge that this work makes interesting and novel contributions. I would therefore keep my rating.", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of our theoretical contributions, practical implications, and empirical validation. Below, we respond to your insightful questions and further clarify key modeling choices and assumptions raised in your comments.\n\n---\n#### **Oversimplified Model of Misalignment**\n> *\"While the formalization of selection and perturbation bias is a strength, it represents a highly structured form of noise. Misalignment in large, web-scraped datasets (e.g., LAION) is likely to be far more chaotic, containing unstructured factual errors, complex paraphrasing, and ambiguous references that don't fit neatly into the proposed categories.\"*\n\nWe agree that real-world misalignment is diverse and often unstructured, particularly in large-scale web-scraped corpora. However, our goal is to provide a principled and analytically tractable abstraction that captures the two dominant sources of semantic misalignment in vision-language data: (1) omission of relevant semantics (modeled as selection bias), and (2) inclusion of distorted or stochastically perturbed semantics (modeled as perturbation bias).\n\nThese two categories subsume a broad spectrum of real-world issues, including ambiguous references (which omit disambiguating context $\\rightarrow$ selection bias) and unstructured, random factual errors ($\\rightarrow$ perturbation bias). As for linguistic-side distortions such as complex paraphrasing, we treat them as modality-specific factors (e.g., stylistic variation, denoted $\\mathbf{m}_t$ in our formulation). We also explicitly acknowledge this modeling limitation in Appendix H.1.\n\nCrucially, this abstraction allows for formal reasoning about how such misalignment affects representation learning, which would be intractable under unconstrained or ad hoc noise models. As such, we view our framework as offering first-order, foundational insights into the effect of semantic misalignment rather than a full taxonomy of noise processes. Extending the framework to incorporate richer or data-driven misalignment structures remains a valuable and promising future direction.\n\n---\n> *\"The experiments, which use template-based text, also do not capture this complexity.\"*\n\nOur original experiments were designed to validate the core theoretical claims under controlled settings. To further assess the practical relevance of our theory on real-world, large-scale data (and to also address Reviewer mroJ’s related concern), we include a case study using OpenCLIP trained on LAION-400M. Specifically, we evaluate zero-shot image-to-text matching across 146 visual concepts, grouped into 15 semantic categories (e.g., color, object). For each concept, we collected up to 200 CC-licensed Flickr images and estimated the frequency of their appearance in LAION-400M captions as a proxy for selection bias. \n\nWe report the average F1 scores using OpenCLIP ViT-B/32 and ViT-L/14:\n\n||Color|Object|Clothing|Food|Vehicle|Scene|Weather|Animal|Role|Texture|Trait|Viewpoint|Emotion|Postprocess|Stereotype|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Average Coverage (%)|2.1622|1.6288|1.1747|0.8140|0.6114|0.5365|0.5284|0.4682|0.2147|0.1470|0.0935|0.0312|0.0238|0.0118|0.0003|\n|ViT-B/32 F1 Score (%)|81.1|87.7|84.6|86.6|90.0|97.0|76.3|97.6|69.2|33.2|21.2|29.4|19.5|41.8|34.8|\n|ViT-L/14 F1 Score (%)|79.0|89.5|86.6|91.0|90.1|94.6|75.9|98.9|66.4|37.1|19.4|31.0|20.1|44.1|40.2|\n\nThe results show that visual concepts with extremely low caption coverage yield dramatically worse F1 scores, aligning with our theoretical prediction. This suggests that our theoretical framework remains predictive even under realistic misalignment in large-scale models. We will include this study in the final version to strengthen the empirical grounding of our work. \n\n---\n#### **Smooth Invertibility Assumption**\n> *\"The entire theoretical framework rests on the generative functions being smoothly invertible. However, this is rarely true in practice. It's seemingly also violated in the paper's own experiments (e.g. Causal3DIdent).\"*\n\nIt is correct that our theory assumes smooth invertibility, which may not strictly hold in practice. Nevertheless, this assumption is standard in identifiability theory [1–3], enabling formal guarantees under well-defined conditions. Without it, identifiability is generally unprovable or requires stronger assumptions on other aspects.\n\nThat said, empirical robustness beyond this assumption is often observed. For example, even in discrete settings with non-injective mappings, such as Bernoulli observations from logistic models, approximate identifiability can still emerge in practice [4]. We are aware of such results and acknowledge them as empirical support for our findings, especially in settings like Causal3DIdent, where our theory remains predictive despite relaxed assumptions.\n\n[1] *Von Kügelgen, Julius, et al. \"Self-supervised learning with data augmentations provably isolates content from style.\"   NeurIPS 2021*.  \n[2] *Brehmer, Johann, et al. \"Weakly supervised causal representation learning.\" NeurIPS 2022*.  \n[3] *Zhang, Kun, et al. \"Causal representation learning from multiple distributions: a general setting.\" ICML 2024*.  \n[4] *Khemakhem, Ilyes, et al. \"Variational autoencoders and nonlinear ica: A unifying framework.\" AISTATS, 2020*.\n\n---\n> *\"If the theory holds even when its assumptions are violated, does this suggest the theorem is not capturing the essential mechanism at play?\"*\n\nNo, this does not imply that the theory fails to capture the essential mechanism. Rather, it highlights that the mechanism identified, namely, alignment on unbiased shared semantics via contrastive learning, holds beyond the idealized conditions needed for proof. The theory provides a principled explanation for this mechanism and clarifies when and why it should be expected to hold, even if real-world success extends further than the formal boundary.\n\n---\n#### **Assumption 3.2**\n> *\"The proof relies on Assumption 3.2, where textual perturbations do not propagate to causally dependent variables. Given that real language often has its own internal causal logic, how do you justify this simplifying assumption?\"*\n\nAssumption 3.2 posits that textual perturbations do not causally propagate to other latent variables. We recognize that natural language can exhibit internal causal structure. However, this assumption is motivated by two practical considerations grounded in how image captions are produced: \n\n- Intentional perturbation of unwanted semantics (e.g., due to sensitive or ethical concerns). In some settings, annotators may deliberately perturb certain sensitive factors, such as age or religion. These perturbations target specific factors, even if those factors are causally related to others in the observational modality. For example, although age may correlate with clothing color in real-world photos, perturbing age in captions does not imply that clothing color must also be modified. This supports our assumption that misalignment mechanisms can be selectively applied without inducing causal propagation.  \n\n- Human perceptual limitations. Annotators often aim to produce accurate and detailed captions, but due to cognitive or attentional constraints, they may inadvertently perturb certain semantic concepts. These perturbations often follow patterns—for example, certain semantic factors (like material and shape) may be jointly perturbed more frequently than others, forming entangled perturbation patterns. In such cases, while semantic entanglement exists within the perturbable set, the perturbation pattern does not necessarily follow the causal laws of real observations, but instead reflects human cognitive nuances. Crucially, such patterns do not causally propagate to other semantics. This fits naturally into our modeling: the range of the perturbable subset can be interpreted as a restricted subset of the selected semantic powerset $\\mathcal{P} _ {\\text{proper}}(\\mathbb{I} _ \\theta)$, capturing typical co-perturbation patterns without requiring assumptions on the latent causal pathways underlying observational data. \n\nIn both cases, Assumption 3.2, grounded in practical rationale, provides a pragmatic abstraction that enables us to analyze recoverable semantics without relying on structural assumptions about the latent causal graph or modeling human cognitive nuances.\n\n---\n> *\"... and how would your identifiability guarantees degrade if it (i.e., Assumption 2) were violated?\"*\n\nIf perturbations were allowed to causally propagate across latent semantic factors, i.e., like those interventional settings, the distinction between biased and unbiased semantics would become less sharply defined, as the injected bias could indirectly affect additional, descendent variables (and those descendent variables also become biased due to causal propogation). In this scenario, the learned representation may no longer cleanly recover the truly unbiased semantics, but instead reflect a blurred or partially contaminated mixture, depending on how far the perturbation propagates, due to we can not determine the intrinsic representation dimensionality without knowing the true latent causal strucutre. Thus, formal guarantees would require strong assumptions on the latent causal graph and the dynamics of perturbation spread (whether certain latent factor is dominated by its exogenous variable), both of which would substantially increase theoretical complexity and reduce the generality of the model. By contrast, our current abstraction deliberately avoids these commitments, allowing us to make latent-structure-agnostic, tractable statements about semantic recovery. \n\n---\nWe hope our reply clarifies your concerns, and thank you again for your time."}, {"review_id": "EKiDTFPl5v", "reviewer": "Reviewer_iYK6", "summary": "This paper analyzes how cross-modal misalignment affects multimodal contrastive learning (MMCL). A latent variable model with selection and perturbation biases is proposed to characterize misalignment, revealing which semantic factors remain identifiable. Theoretical results are validated on real and synthetic data, showing that moderate misalignment can enhance generalization by acting as a regularizer.", "questions": "1.\tCan the proposed theory be extended beyond contrastive learning frameworks to settings like multimodal generation or joint classification? In particular, is it applicable to asymmetric tasks such as image-to-text generation?\n2.\tIn practical scenarios, how can the strength of selection or perturbation bias be estimated? Are there learnable or calibratable metrics that can quantify the degree of misalignment?\n3.\tThe paper suggests that misalignment can act as a regularizer, but what are the precise conditions under which this becomes beneficial? At what point does misalignment turn from helpful bias into detrimental noise suppression?\n4.\tMany existing works (e.g., CLIP with hard negative sampling or caption filtering) aim to mitigate misalignment. Can such techniques be theoretically or empirically integrated with this framework? Are they compatible in principle?\n5.\tIs it possible to design corrective mechanisms based on estimated selection or perturbation bias to improve text generation or data sampling? Could such targeted interventions form a promising direction for future work?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "S1.\tThe paper proposes a latent variable model with selection and perturbation biases to precisely characterize cross-modal misalignment, offering strong theoretical grounding.\nS2.\tThe paper establishes a block-identifiability framework that reconciles seemingly conflicting views on whether misalignment is harmful or beneficial, providing practical guidance for real-world applications.\nS3.\tThe paper validates the theory through comprehensive experiments on both synthetic and real-world datasets (e.g., MPI3D, Causal3DIdent), covering misalignment severity, variable dependencies, and downstream tasks.\n\nW1.\tThe theoretical analysis is tightly coupled with the MMCL framework (contrastive learning + symmetric max-entropy loss), limiting its applicability to other paradigms such as generative modeling or multi-task learning.\nW2.\tWhile the modeling of selection and perturbation biases is representative, it does not account for more complex forms of misalignment such as temporal desynchronization, modality dropout, or entity mismatch.\nW3.\tThe downstream tasks are simplified and mainly focus on semantic variable prediction or synthetic targets, lacking evaluation on real-world tasks like visual question answering, retrieval.\nW4.\tThe paper asserts that perturbation bias does not structurally propagate, but it lacks a deeper analysis of how misalignment may affect causal pathways or downstream generalization in practical systems.\nW5.\tImplementation and training details are somewhat underreported; while the core theory is solid, missing descriptions of hyperparameters, baselines, and robustness tests limit reproducibility.", "was_revised": true, "final_justification": "The authors’ response, including their experimental results and clarifications, has addressed part of my concerns. Therefore, I have decided to raise my score.", "rebuttal": "We appreciate the thoughtful assessment and the recognition of our contributions to the theoretical depth, identifiability framing, and empirical validation. We address the remaining concerns and discussion points below.\n\n---\n> **W1** *\"The theoretical analysis is tightly coupled with the MMCL ... limiting ... other paradigms such as generative modeling or multi-task learning.\"*\n\nWe would like to clarify that our study focuses on identifiability analysis under MMCL, as stated in the abstract, introduction, and theoretical sections. This choice is principled: MMCL with InfoNCE is a widely adopted paradigm in vision-language learning (e.g., CLIP), offering conceptual clarity and practical significance. Centering on a single objective is also consistent with prior work in identifiability theory, such as nonlinear ICA with likelihood-based modeling [1] and unimodal contrastive learning [2].\n\nWhile our framework is developed within MMCL, its insights into how semantic misalignment shapes representation learning remain broadly relevant. Applying these ideas to downstream tasks such as generation or multi-task learning would require integrating decoding mechanisms or task-specific objectives—directions we view as promising but beyond the current scope of this theory-focused work.\n\n[1] *Khemakhem, Ilyes, et al. \"Variational autoencoders and nonlinear ica: A unifying framework.\" AISTATS, 2020*.  \n[2] *Zimmermann, Roland S., et al. \"Contrastive learning inverts the data generating process.\" ICML, 2021*.\n\n---\n> **W2** *\"While the modeling of selection and perturbation biases is representative, it does not account for ... misalignment such as temporal desynchronization, modality dropout, or entity mismatch.\"*\n\nWe appreciate this observation and agree that multimodal data presents a variety of misalignment forms. Our study specifically focuses on semantic misalignment in image-text pairs, which is a setting of growing practical relevance and theoretical interest due to its prevalence in large-scale pretraining pipelines (e.g., CLIP, ALIGN). By modeling selection and perturbation biases, we aim to capture common patterns of omission and distortion in this widely used modality pair.\n\nWe acknowledge that other types of misalignment, such as temporal desynchronization, modality dropout, or entity mismatch, are also important and may require different modeling assumptions and methodological tools. Our current abstraction serves as a focused and tractable step toward formalizing semantic misalignment. Extending the framework to encompass broader or more heterogeneous modalities is an exciting and valuable direction for future work.\n\n---\n> **W3** *\"... downstream tasks ... simplified ... focus on semantic variable prediction or synthetic targets, lacking evaluation on real-world tasks like visual question answering, retrieval.\"*\n\nWe respectfully clarify that our downstream evaluations were deliberately designed to be controlled, interpretable, and causally grounded, in order to rigorously validate our theoretical predictions and isolate the effects of selection and perturbation biases. While VQA or retrieval can offer support, they also introduce extra supervision or inductive biases beyond our theory, which may confound interpretation of results. As emphasized in Corollary 4.1 and 4.2, our findings offer principled guidance for improving data quality regarding model versatility and robustness in practical multimodal systems. Extending the framework to complex downstream tasks is an important but orthogonal extension for future work.\n\n---\n> **W4** *\"... perturbation bias does not structurally propagate ... lacks deeper analysis of how misalignment may affect causal pathways or downstream generalization ...\"*\n\nWe appreciate your interest in the causal implications of misalignment. Our assumption that perturbation bias does not causally propagate (Assumption 3.2) is a deliberate and principled modeling choice. In practice, such perturbations often reflect textual distortions—e.g., omission or rewriting of certain attributes by annotators—without intervening on the visual content. This abstraction allows us to isolate the effects of semantic misalignment without committing to specific causal structures among latent variables.\n\nRegarding downstream generalization, we directly address this in Corollary 4.2. When biased semantics align with environment-sensitive factors, MMCL implicitly filters out these spurious signals, thereby enhancing generalization. We empirically validate this in Section 5.1: under domain shifts, domain-sensitive factors lead to degraded performance when preserved, but robustness improves when they are filtered (by intentional perturbation), as predicted by our theory.\n\n---\n> **W5** *\"... implementation and training details ... underreported ... missing hyperparameters, baselines, robustness tests ...\"*\n\nWe respectfully disagree with this assessment. Comprehensive implementation and training details are reported throughout the paper and appendices:\n - Section 5 outlines the experimental setup, datasets, and evaluation procedures.\n - Appendices D.1, E.1, and F.1 include full model specifications, training hyperparameters, and evaluation settings.\n - All reported results are averaged over three random seeds, and standard deviations are included in performance plots to assess robustness.\n\nSince our focus is on validating theoretical claims under controlled conditions, benchmark comparisons are not applicable. However, we provide a fully reproducible anonymous codebase with all configurations and scripts for transparency.\n\n---\n> **Q1** *\"Can... extended beyond contrastive learning frameworks ... such as image-to-text generation?\"*\n\nYes, with caveats. While our work focuses on MMCL, the core principle, that semantic misalignment influences which representations are learned, applies beyond contrastive learning. Extending the theory to generative tasks (e.g., image-to-text) would require incorporating decoding mechanisms, which lies outside our current scope. Nevertheless, we believe the framework offers foundational insights into how alignment quality shapes learned semantics, and may inform future efforts toward improving fidelity and safety in generative settings.\n\n---\n> **Q2** *\"... how can ... bias be estimated? Are there... metrics ... quantify ... misalignment?\"*\n\nYes, with caveats. While our study does not implement bias estimation mechanisms, the proposed framework suggests plausible directions for approximating misalignment:\n - Selection Bias: One can define a target concept vocabulary (e.g., based on human annotations or a domain ontology) and compute the frequency with which each concept appears in image captions. The aggregate coverage can serve as a proxy for semantic omission, though it depends on the quality and granularity of the chosen vocabulary.\n - Perturbation Bias: This is more nuanced, as it involves distortions rather than omission. A possible approach is to generate reference captions using strong image-to-text models (e.g., fine-tuned BLIP or GPT-4V) and compare them to dataset captions using approximate semantic similarity or factual consistency metrics. While these metrics are only proxies and may themselves be noisy, they can help identify candidate distortions or inconsistencies at scale.\n\n---\n> **Q3** *\"... misalignment can act as a regularizer, ... under which this becomes beneficial? ... misalignment turn from helpful bias into detrimental noise suppression?\"*\n\nThis is a valuable question, and we address it formally in Corollary 4.2. Misalignment becomes beneficial when it systematically affects environment-sensitive (i.e., non-robust or domain-specific) semantics, allowing contrastive learning to filter out unstable cues and focus on domain-invariant features, thus acting as an implicit regularizer.\n\nImportantly, the key factor is which semantics are biased, not how strongly. Mild omission of task-relevant features can degrade performance, while even strong distortion may help if it targets spurious factors. The tipping point lies in the semantic identity of the biased subset, not the absolute noise level.\n\n---\n> **Q4** *\"Can ... CLIP with hard negative sampling or caption filtering ... integrated with this framework ... compatible in principle?\"*\n\nYes, both techniques are compatible with our framework and align with the insights in Corollary 4.1. Caption filtering directly reduces selection and perturbation bias by removing imcomplete or distorted captions, improving semantic alignment across modalities. Hard negative sampling, while not addressing misalignment per se, enhances semantic discrimination and complements alignment by refining decision boundaries. Together, these practices support the theoretical predictions of our model and can be viewed as practical extensions.\n\n---\n> **Q5** *\"Is it possible to design corrective mechanisms ... to improve text generation or data sampling? ... promising direction for future work?\"*\n\nYes, and we agree this is a highly promising direction. Since misalignment introduces structured inductive biases, learning corrective mechanisms that estimate and mitigate these patterns can meaningfully improve data quality and downstream performance. One practical approach is to use high-quality image-to-text models (e.g., BLIP, GPT-4V) to generate reference captions and compare them with existing captions. Semantic divergence between the two can serve as a proxy signal for selection or perturbation bias. While current models may underperform on rare or subjective concepts, human-in-the-loop feedback or filtering can further enhance correction quality. We believe semi-automatic pipelines for mitigating misalignment will become increasingly feasible as tools improve.\n\n---\nWe thank you again for the thoughtful feedback. Points like Q3–Q5 point to valuable extensions beyond our current scope, and we will incorporate relevant discussion in the final version."}, {"review_id": "vWXaY4DSRT", "reviewer": "Reviewer_HLeK", "summary": "This paper proposes a framework for characterizing multimodal misalignment through a latent variable model for the underlying generative process of image and text. This model describes two types of potential misalignment: selection bias, where only a subset of the latent variables are used in generating text, and perturbation bias, where the ground truth semantic information is corrupted to generate erroneous text. Under the LVM framing, the paper demonstrates that multimodal contrastive learning recovers the unbiased semantic variables but excludes the misaligned components. Experiments on synthetic numerical data verify this theoretical finding, as well as experiments on a real-world dataset with seven mutually independent ground-truth factors and a synthetic dataset with causal relations between latent variables.", "questions": "Questions:\n1. How might practitioners identify selection and perturbation biases in their dataset?\n2. In a situation where the amount of bias varies per sample (such as because of differences in human annotations), how would this affect the representations learned?  \n3. When might misalignment be useful to capture synergistic interactions in multimodal data? For example, sarcasm is difficult to predict given unimodal data, but is easier given multimodal context, such as a video.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The problem of understanding when multimodal misalignment is desirable is important and novel.\n2. The paper is well-written and the intuition behind the theoretical findings is well-explained.\n3.Experiments on both simulated numerical data and real-world modalities support the theoretical findings.\nWeaknesses\n1. Experiments are limited to image and text.\n2. In a real world setting, it is often unclear to what extent there is selection and perturbation bias. This work does not seem to address how one would quantify these biases, which would be relevant to practitioners.", "was_revised": true, "final_justification": "The authors have addressed my questions on the practical aspects of the work -- how it may be extended to modalities beyond vision and text, how practitioners may identify selection and perturbation bias, and how synergistic interactions between modalities that yield new information can be identified. I maintain my positive rating.", "rebuttal": "Thank you for reviewing our work and your feedback. We appreciate your recognition of our theoretical framing, clarity, and empirical validation. While some of your excellent questions extend beyond our current scope, we see them as exciting directions built on our foundation. We respond to each point below.\n\n---\n> *\"Experiments are limited to image and text.\"*\n\nWe appreciate the suggestion. Our focus on image-text pairs is a deliberate and principled choice aimed at isolating the core phenomenon of interest: semantic misalignment in multimodal learning. Image-text is among the most widely studied modality pairings in large-scale representation learning (e.g., CLIP, ALIGN), and it offers practical advantages for analyzing semantic filtering—due to its scale, accessibility, and interpretability.\n\nImportantly, our framework is modality-agnostic and applies to any setting involving two views: one being a high-dimensional entangled observation (e.g., image), and the other a structured auxiliary modality that embeds human conceptual knowledge, such as text. This makes language a natural fit for modeling human-induced biases like semantic selection and perturbation.\n\nWe select image-text pairs not for their exclusivity, but because they allow controlled, interpretable evaluations of theoretical predictions, using well-established datasets like MPI3D and Causal3DIdent. Other modalities (e.g., depth maps, audio data, tactile tensor grids) are compatible with our framework in principle, but currently may lack the structured datasets and semantic annotations necessary for rigorous, controllable analysis. Exploring these modalities is a valuable direction, though outside the current scope of our theory-driven study.\n\n---\n> *\"In a real world setting, it is often unclear to what extent there is selection and perturbation bias. This work does not seem to address how one would quantify these biases, which would be relevant to practitioners.\"*\n\nWe appreciate your emphasis on this practical point. Our work is primarily focused on formalizing the impact of semantic misalignment on representation learning and identifiability, rather than on empirically quantifying dataset-specific bias. That said, we fully acknowledge that estimating the extent and structure of such biases (particularly in large-scale, web-curated datasets) is an important challenge for improving interpretability, data quality, and downstream robustness.\n\nWe view this as a challenging but valuable problem, and a natural next step that builds on our theoretical contributions—representing a complementary direction beyond the current scope of this study.\n\nFor preliminary directions, we refer to our response to your next question. We will incorporate a discussion of this topic in the final version.\n\n---\n> **Q1** *\"How might practitioners identify selection and perturbation biases in their dataset?\"*\n\nThis is an great and timely question. While our current work does not implement empirical detection mechanisms, our framework suggests several actionable directions for practitioners:\n\n- Selection Bias: Practitioners can begin by defining a target concept vocabulary (e.g., derived from ontology or human annotation) and computing the caption coverage rate for each concept across the dataset. Averaging coverage across categories may help reveal systematic underrepresentation or omission of certain semantic types.\n\n- Perturbation Bias: Since this bias involves semantic distortions, one practical approach is to generate reference captions using a strong, well-aligned image-to-text model (e.g., fine-tuned BLIP or GPT-4V), and compare them with existing captions. Metrics such as semantic similarity, factual consistency, or entity-level precision/recall can serve as approximate proxies for quantifying distortion or hallucination.\n\nThese techniques involve design choices and approximations, but they offer a useful starting point for diagnosing selection and perturbation biases in real-world multimodal datasets—ultimately supporting more reliable and semantically grounded vision-language model training.\n\n---\n> **Q2** *\"In a situation where the amount of bias varies per sample (such as because of differences in human annotations), how would this affect the representations learned?\"*\n\nWe appreciate this insightful observation. The impact of instance-level bias variation depends on the type of misalignment:\n\n- Selection Bias: If the subset of semantic variables included in the text varies across samples, it introduces a missing-not-at-random (MNAR) pattern in the latent space, a well-known challenge in missing data theory, particularly in the context of latent semantic representations. In this case, the selection bias parameter $\\theta$ becomes a sample-dependent random variable, potentially governed by annotation conventions, cultural norms, or attentional cues, especially in large-scale web data. Modeling this rigorously would require specifying a distribution over selection subsets, possibly incorporating structured priors, a challenging but important direction that we identify for future work. We explicitly acknowledge this limitation in Appendix H.1, and emphasize that our current theory provides a principled abstraction that can be extended to such sample-specific stochastic selection scenarios.\n\n- Perturbation Bias: Our framework already accommodates sample-level variation in perturbation. As noted in Assumption 3.2, we do not assume that all samples share the same perturbed semantics. Instead, we allow a random subset $A \\subset \\mathbb{I}_\\rho$ with non-zero probability $p(A) > 0$, such that perturbations occur stochastically across the dataset. This formulation captures instance-level variation in perturbation patterns, and our identifiability result holds under this generalization.\n\n---\n> **Q3** *\"When might misalignment be useful to capture synergistic interactions in multimodal data? For example, sarcasm is difficult to predict given unimodal data, but is easier given multimodal context, such as a video.\"*\n\nThank you for this excellent and thought-provoking question. It points to two distinct cases of multimodal synergy:\n\n- Deterministic cross-modal mappings:\nSome semantics that initially appear non-shared (e.g., facial expression $\\rightarrow$ emotional tone)  can be deterministically inferred across modalities. Our framework naturally accommodates this case: deterministic cross-modal mappings (i.e., $z^{(2)} = f(z^{(1)})$) can simply be absorbed into either modality’s encoder, thus preserving semantic alignment. Hence, although we present a shared latent semantic space for conceptual clarity, our identifiability guarantees fully extend to these scenarios.\n- Emergent semantics requiring interaction-specific latent variables:\nIn contrast, phenomena such as sarcasm, irony, or humor arise from non-additive, modality-interactive effects, e.g., the incongruity between neutral text and exaggerated prosody or facial expressions. Such semantics may not be inferable from a single modality alone, and explicitly modeling stable, interaction-specific latent variables becomes necessary. Importantly, we believe these multimodal interactions exhibit structured and recurring patterns rather than arbitrary noise, and our principled abstraction provides a foundational framework upon which future interaction-driven models can build.\n\nThus, while our current framework does not directly model these interactions, by isolating invariant semantics under structured misalignment, it sets the stage for capturing emergent multimodal meanings. We see this as an exciting complementary direction and will incorporate this discussion into the final version.\n\n---\nWe hope our reply clarifies your concerns, and thank you again for your time."}, {"review_id": "PiA9UfEjpG", "reviewer": "Reviewer_t7nT", "summary": "Leveraging multimodal contrastive learning (MMCL) and a latent variable model (LVM), the authors investigate the problem of cross-modal misalignment and propose a unified perspective by introducing two key mechanisms: selection bias and perturbation bias. Through a thorough theoretical analysis, they demonstrate that MMCL learns representations that retain only the semantic information invariant to these biases. The work provides both theoretical foundations and practical insights into how misalignment impacts multimodal representation learning.", "questions": "1. On page 3, two loss functions are introduced: L_MMCL and L_SymAlignMaxEnt. However, in the experimental setup (page 7, section 5.1, line 262), L_SymAlignMaxEnt is used for the first experiment, while L_MMCL is used in the second experiment (page 8, section 5.2, line 298). It would be helpful to clarify the rationale behind choosing different loss functions for these experiments, and to specify any differences in their expected outcomes or suitability for each setting.\n\n2. In Figure 4 (page 7), the X-axis is somewhat unclear. Adding a more detailed explanation in the figure description on page 8 would greatly improve clarity for readers trying to interpret the results.\n\n3. According to the experiment setups (lines 261, 301, and 324), it appears that all models are based on simple MLP architectures. I’m curious whether the proposed theoretical insights hold under more complex models, such as transformer-based architectures. Exploring this would strengthen the practical relevance of the findings.\n\n4. On page 3, in the section on Latent Space (lines 121–122), it is stated that text-specific variables are independent of semantic variables $s$, with “grammar” given as an example. However, some grammatical features—such as verb tense—can carry semantic meaning. This raises the question of whether certain text-specific variables are inherently entangled with semantic information. A brief discussion or clarification would be valuable here.", "limitations": "I suggest including a discussion on the limitations of the assumptions required by the proposed theorems.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper presents a comprehensive theoretical analysis of the cross-modal misalignment problem, offering a unified perspective that reconciles two contrasting views: (1) that misalignment can lead to hallucinations and degraded performance in multimodal models, and (2) that deliberately introducing misalignment—particularly in style-related information—can enhance robustness in representation learning.\n2. The explanation of the core idea is clear, and the illustration in Figure 1 makes the concept easy to understand.\n3. There are extensive experiments conducted to verify the proposed theorems.\n\nWeaknesses: \n1. Since this paper aims to unify opposing views on cross-modal misalignment—and one of these views argues that misalignment can lead to “hallucination” in multimodal models—it would be valuable to include a more detailed analysis, preferably theoretical, of the hallucination aspect to strengthen the argument.\n\n2. In the theoretical results, the assumption that the generative mappings are diffeomorphisms (e.g., as stated on page 4, line 147) may be overly idealized and unrealistic in practical settings. A discussion of this assumption’s implications and possible relaxations would enhance the paper’s applicability.", "was_revised": true, "final_justification": "The concerns I raised have all been addressed. I believe the author has put significant effort into this work, and from my perspective, it is worthy of acceptance.", "rebuttal": "We sincerely thank you for your thoughtful review and recognition of our work’s theoretical depth, clarity, and significance. Your constructive suggestions on modeling assumptions, latent structure, and presentation are valuable and will help strengthen the paper. Below, we address your questions point by point.\n\n---\n> **W1** *\"... one of these views argues that misalignment can lead to 'hallucination' ... it would be valuable to include a more detailed analysis ...\"*\n\nWe thank you for surfacing this important line of inquiry. While our current theoretical framework does not explicitly model hallucination, we view it as a consequential practical symptom of unresolved cross-modal misalignment. In MMCL-based models, hallucination can manifest in two prominent scenarios:\n\n- When important visual attributes are omitted or rarely mentioned (selection bias) or inconsistently described across samples (perturbation bias), the resulting representations may fail to retain these factors. In practice, if the learned representation has higher capacity than the subspace of invariant semantics, the remaining dimensions may be filled with spurious or dataset-specific correlations. These can surface as hallucinated or non-factual semantic information during generation or matching.\n\n- When key semantic variables are consistently mislabeled in captions (e.g., both sea lions and seals captioned as \"seal\"), MMCL can learn falsely grounded associations—that is, associations that are consistently aligned across modalities but semantically incorrect. These may propagate into downstream tasks, producing outputs that reflect dataset artifacts rather than visually grounded meaning.\n\nIn essence, hallucination arises when training captions lack semantic fidelity, either through omission or distortion, especially for concepts expected to generalize at test time. As emphasized in Corollary 4.1, high-fidelity alignment is crucial for learning grounded, reliable representations, offering an implicit explanation for when hallucination is likely to occur.\n\nA rigorous theoretical treatment of hallucination would require extending our framework to include decoding mechanisms and generative modeling dynamics. While non-trivial, we view this as an exciting direction for future work and will include this discussion in the final version.\n\n---\n> **W2** *\"... the assumption that the generative mappings are diffeomorphisms ... may be overly idealized ...\"*\n\nWe agree that the assumption of diffeomorphic generative mappings can be idealized in practice. We adopt this assumption primarily for theoretical tractability, especially without assuming parametric latent distributions or structural priors. Below, we discuss both its implications and potential relaxations:\n\n - Implications: The diffeomorphism assumption ensures that the true latent semantic factors are recoverable from the observed modalities via a well-behaved inverse. This structural regularity is essential for identifiability analysis; without it, semantic recovery becomes ill-posed or approximate under stricter assumptions. Our use of this assumption follows established precedents in nonlinear ICA and causal representation learning [1,2,3], where assumptions on diffeomorphistic generative process are common to obtain rigurous identifiability guarantees.\n\n - Possible relaxations: While assuming diffeomorphic generative mappings provides clean theoretical guarantees, we acknowledge that this assumption could be relaxed in future work. For instance, one may consider weaker conditions such as injectivity or explore models with discrete latent factors combined with measure-preserving or piecewise invertible mappings. However, relaxing the diffeomorphism assumption typically necessitates more advanced mathematical machinery and may require stronger inductive biases to maintain theoretical rigor. These relaxations offer promising directions but also increase complexity and may reduce generality.\n\nImportantly, our experiments show that deep models still perform well in settings where the diffeomorphism assumption does not strictly hold (e.g., on MPI3D-Complex dataset). This suggests the assumption serves as a sufficient but not necessary condition for recovery, common in many theory-driven studies.\n\nThank you again for raising this point, and we will incorporate this discussion on both implications and possible  relaxations into the final version of the paper.\n\n[1] *Von Kügelgen, Julius, et al. \"Self-supervised learning with data augmentations provably isolates content from style.\" NeurIPS 2021*.  \n[2] *Brehmer, Johann, et al. \"Weakly supervised causal representation learning.\" NeurIPS 2022*.  \n[3] *Zhang, Kun, et al. \"Causal representation learning from multiple distributions: a general setting.\" ICML 2024*.\n\n---\n> **Q1** *\"... two loss functions are introduced ... to clarify the rationale behind choosing different loss functions ...\"*\n\nWe appreciate your question and clarify that the use of two different loss functions, $\\mathcal{L}_ {\\small \\mathrm{MMCL}}$ and $\\mathcal{L}_ {\\small \\mathrm{SymAlignMaxEnt}}$, is intentional and corresponds to the distinct goals of the respective experiments:\n\n- In Section 5.1, we employ the loss $\\mathcal{L} _ {\\small \\mathrm {SymAlignMaxEnt}}$ to validate our theoretical analysis, which is derived from $\\mathcal{L}_{ \\small \\mathrm{MMCL}}$ under idealized conditions such as infinite observational data (see Page 3, line 92–93). This loss is designed for analytical tractability and reflects the formal identifiability guarantees of our theory (Page 3, line 98). As is standard in identifiability studies related to contrastive learning (Page 3, line 98), such idealized objectives provide a principled lens to rigurously analyze the effects of misalignment on multimodal representation learning.\n\n- In Sections 5.2 and 5.3, we turn to practical evaluation on realistic image-text datasets, where the canonical MMCL loss is adopted (Page 3, line 89). This loss, used in systems such as CLIP, ALIGN, reflects standard training objectives for multimodal contrastive learning and allows us to examine how our theoretical insights manifest in real-world scenarios.\n\nThe core difference between the two losses lies in the geometry of the representation space, particularly in how they normalize and align semantic manifolds. Despite this difference, our theoretical framework targets block-wise semantic identifiability up to invertible transformations, rather than exact functional recovery, which remains compatible with both objectives.\n\nWe will further clarify this design choice in the final version, and thank you for bringing attention to this important detail.\n\n---\n> **Q2** *\"In Figure 4 (page 7) ... unclear ... figure description on page 8 would greatly improve clarity ...\"*\n\nThank you! We will revise the caption for clarity in the final version.\n\n---\n> **Q3** *\"... all models are based on simple MLP ... whether the proposed theoretical insights hold under more complex models, such as transformer-based architectures ...\"*\n\nWe appreciate this thoughtful question. The identifiability guarantees in Theorem 4.1 are derived under general assumptions and are not tied to any specific architecture, as long as the model class is sufficiently expressive to minimize the MMCL objective.\n\nWe deliberately use simple MLPs in our experiments to validate the theory under controlled conditions. MLPs are standard in identifiability studies, and in the context of our study, they allow us to isolate the effects of semantic misalignment without confounding factors introduced by architectural inductive biases. In contrast, more complex models (e.g., CNNs or Transformers) may benefit performance on practical datasets, but also involve heavier parameterization of complexity that may obscure the mechanisms under study—especially in limited-sample regimes.\n\nWhile our theory remains valid for such architectures, understanding how architectural inductive biases interact with misalignment dynamics is an important and orthogonal direction, and we acknowledge this as valuable future work.\n\n---\n> **Q4** *\"... text-specific variables are independent of semantic variables $s$, with “grammar” given as an example ... whether certain text-specific variables are inherently entangled with semantic ...\"*\n\nWe appreciate this thoughtful observation. The term *\"grammar\"* may have unintentionally overstated the independence between linguistic form and semantics. Our intention was to refer to text-specific variables that influence expression but are not visually grounded, such as stylistic phrasing or rhetorical choices. We agree that certain grammatical features—such as verb tense or modality—can carry semantic content, particularly in textual contexts.\n\nIn our formulation, if such features contribute to visually grounded meaning, they are included as part of the shared semantic variable $\\mathbf{s}$. If they do not correspond to observable visual attributes but are statistically correlated, they can be interpreted within our framework as a form of selection bias (with the modalities mirrored), i.e., information present in text but omitted in the image modality. This reflects a modeling simplification to keep the exposition concise, rather than a fundamental gap, as the asymmetry is already accounted for by our selection bias formalism.\n\nTo prevent misinterpretation, we will revise the explanation of terminology in the final version, replacing “grammar” with “stylistic or expressive form”. We appreciate your careful reading.\n\n---\nWe hope our reply clarifies your concerns, and thank you again for your time."}, {"review_id": "NKkO1rASoY", "reviewer": "Reviewer_mroJ", "summary": "This work proposes a Latent Variable Model (LVM) for resolving Cross-Modal Misalignment problem, which explicitly characterizes misalignment through two mechanisms: Selection bias and Perturbation bias. It makes the theoretical analysis is made to validate the  the representations learned by MMCL framwork can capture the semantic variables invariant to selection and perturbation bias. Experiments on synthetic and real-world image-text datasets (e.g., MPI3D-Complex, Causal3DIdent) demonstrate the efectiveness of the proposed method.", "questions": "See weakness part.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. A clear and reasonable framwork is proposed to disentangle the unbiased and biased factors for cross-modal representation learning, where sufficient theoretical analysis is given to surport the framwork. \n2. The source code is attached that enhance the reproducibility and reliability of this work. \n\nWeakness:\n1. This study solely does not include experimental designs to apply this framework to more common downstream tasks (e.g., image-text matching), which undermines the work's persuasiveness and limits its broader impact.\n\n2. This experiments of this paper only validates the efficetiveness of the proposed framework on distinguishing the aligned and misaligned factors of iamge-text pairs, whilst not designing experiments tp apply this framework to resolve more prevalent related downstream tasks, such image-text matching. It weakens the persuasiveness and broader impcat of this work.", "was_revised": true, "final_justification": "The additional experiments and discussion address my concerns, and I maintain a positive score.", "rebuttal": "We thank you for your review and for recognizing the value of our theoretical framework and reproducibility. We address your comments and clarify our contributions below.\n\n---\n> **W1** *\"This study solely does not include experimental designs to apply this framework to more common downstream tasks (e.g., image-text matching), which undermines the work's persuasiveness and limits its broader impact.\"*\n\nWe appreciate your concern regarding conventional downstream tasks such as image-text matching. However, we would like to clarify that the primary goal of this paper is not to introduce a new training method or optimize task performance, but to provide a principled theoretical framework for understanding how selection and perturbation biases shape semantic representations in multimodal contrastive learning. Specifically, our work offers a formal identifiability analysis of how multimodal misalignment filters semantic content during learning.\n\nTo support this goal, our original experiments are intentionally designed around controlled, factorized datasets (e.g., MPI3D, Causal3DIdent) where ground-truth latent variables are known. This setting allows for direct empirical validation of the theoretical claims in Theorem 4.1 and its corollaries, without introducing additional confounding factors.\n\nNonetheless, to address your concern regarding practical applicability, we conducted an additional illustrative analysis using OpenCLIP trained on LAION-400M. In this case study, we perform zero-shot image-to-text matching across 146 visual concepts grouped into 15 semantic categories (e.g., color, object, emotion). For each concept, we collected up to 200 CC-licensed Flickr images and measured the frequency of concept mentions in LAION-400M captions to estimate selection bias. We report the average F1 scores using OpenCLIP ViT-B/32 and ViT-L/14:\n\n||Color|Object|Clothing|Food|Vehicle|Scene|Weather|Animal|Role|Texture|Trait|Viewpoint|Emotion|Postprocess|Stereotype|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Average Coverage (%)|2.1622|1.6288|1.1747|0.8140|0.6114|0.5365|0.5284|0.4682|0.2147|0.1470|0.0935|0.0312|0.0238|0.0118|0.0003|\n|ViT-B/32 F1 Score (%)|81.1|87.7|84.6|86.6|90.0|97.0|76.3|97.6|69.2|33.2|21.2|29.4|19.5|41.8|34.8|\n|ViT-L/14 F1 Score (%)|79.0|89.5|86.6|91.0|90.1|94.6|75.9|98.9|66.4|37.1|19.4|31.0|20.1|44.1|40.2|\n\nThese results reveal a clear trend: visual concepts with low caption coverage suffer from significantly lower retrieval performance, consistent with our theoretical predictions about the role of semantic fidelity and coverage in MMCL. This provides further evidence that our framework is applicable and predictive in real-world settings. We plan to include a brief version of this additional case study in the final version to strengthen practical relevance.\n\n---\n> **W2** *\"This experiments of this paper only validates the efficetiveness of the proposed framework on distinguishing the aligned and misaligned factors of iamge-text pairs, whilst not designing experiments tp apply this framework to resolve more prevalent related downstream tasks, such image-text matching. It weakens the persuasiveness and broader impcat of this work.\"*\n\nWe appreciate your perspective and agree that further downstream evaluations can provide additional insight into practical relevance. However, we would like to clarify that the primary objective of this work is not to introduce a new training pipeline or method related to downstream tasks, but to develop a principled theoretical framework that explains how multimodal misalignment shapes learned representations in MMCL, with a focus on identifiability under selection and perturbation biases.\n\nOur contributions lie in the domain of representation learning theory—formulating and proving identifiability results under semantic misalignment. To connect this theory to practice, we derive implications via Corollaries 4.1 and 4.2, which illustrate how different forms of misalignment can help or hinder generalization depending on the semantics affected. These insights offer conceptual guidance for dataset and method design in real-world multimodal learning.\n\nWhile we conducted a case study in our previous response validating pretrained OpenCLIP representations, the design of downstream systems, such as image-text matching methods informed by our theoretical framework, is a valuable direction but remains outside the scope of this work. We see it as a promising avenue for future research building upon our theoretical foundation.\n\n----\nThank you again for your time and for motivating this additional clarification and analysis. We hope that our responses address your concerns."}]

Author rebuttal:

Meta-review: This paper introduces a theoretical framework for cross-modal misalignment in multimodal contrastive learning, formalizing it as selection and perturbation biases. Reviewers agreed the work is novel and impactful, with clear theory and supportive experiments, further strengthened by the additional OpenCLIP–LAION case study in rebuttal. While concerns remain about simplifying assumptions and broader empirical validation, the reviewers found the clarifications convincing. Overall, the paper makes a meaningful theoretical contribution and is recommended for acceptance.
Decision: Accept (spotlight)