Towards the Causal Complete Cause of Multi-Modal Representation Learning
Authors: Jingyao Wang
Siyu Zhao
Wenwen Qiang
Jiangmeng Li
Changwen Zheng
Fuchun Sun
Hui Xiong
Venue: ICML 2025 poster (2025)
Keywords: Multi-Modal Representation Learning
Causal Sufficiency and Necessity
Primary area: general_machine_learning->representation_learning

Abstract: Multi-Modal Learning (MML) aims to learn effective representations across modalities for accurate predictions. Existing methods typically focus on modality consistency and specificity to learn effective representations. However, from a causal perspective, they may lead to representations that contain insufficient and unnecessary information. To address this, we propose that effective MML representations should be causally sufficient and necessary. Considering practical issues like spurious correlations and modality conflicts, we relax the exogeneity and monotonicity assumptions prevalent in prior works and explore the concepts specific to MML, i.e., Causal Complete Cause ($C^3$). We begin by defining $C^3$, which quantifies the probability of representations being causally sufficient and necessary. We then discuss the identifiability of $C^3$ and introduce an instrumental variable to support identifying $C^3$ with non-exogeneity and non-monotonicity. Building on this, we conduct the $C^3$ measurement, i.e., $C^3$ risk. We propose a twin network to estimate it through (i) the real-world branch: utilizing the instrumental variable for sufficiency, and (ii) the hypothetical-world branch: applying gradient-based counterfactual modeling for necessity. Theoretical analyses confirm its reliability. Based on these results, we propose $C^3$ Regularization, a plug-and-play method that enforces the causal completeness of the learned representations by minimizing $C^3$ risk. Extensive experiments demonstrate its effectiveness.

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Jingyao Wang" 12 Siyu Zhao 2* Wenwen Qiang" 12 Jiangmeng Li
Changwen Zheng' 12
Fuchun Sun 13
Hui Xiong*
Abstract
1. Introduction
Multi-Modal Learning (MML) aims to learn ef- fective representations across modalities for ac- curate predictions. Existing methods typically focus on modality consistency and specificity to learn effective representations. However, from a causal perspective, they may lead to representa- tions that contain insufficient and unnecessary in- formation. To address this, we propose that effec- tive MML representations should be causally suf- ficient and necessary. Considering practical issues like spurious correlations and modality conflicts, we relax the exogeneity and monotonicity assump- tions prevalent in prior works and explore the concepts specific to MMI,, i.e., Causal Complete Cause (C²). We begin by defining C which quantifies the probability of representations being causally sufficient and necessary. We then dis- cuss the causal identifiability of C and introduce instrumental variable to support identifying 3 with non-exogeneity and non-monotonicity. Building on this, we conduct the C measure- ment, i.e., C risk. We propose twin network to estimate it through (i) the real-world branch: utilizing the instrumental variable for sufficiency, and (ii) the hypothetical-woot branch: applying gradient- based counterfactual modeling for neces- sity. Theoretical analyses confirm its reliability. Based these results, we propose C3 Regular- ization, a plug-and-play method that enforces the causal completeness of the learned representa- tions by minimizing 3 risk. Extensive experi- ments demonstrate its effectiveness.
The initial inspiration for artificial intelligence is imitate human perceptions which are based on different modalities (Jordan & Mitchell, 2015; Mahesh, 2020), e.g., sight, sound, and touch. Each modality provides unique information with distinct statistical properties (Baltrušaitis et al., 2018; Wang et al., 2023c). A fundamental aspect of human perception is the ability to simultaneously integrate data from different modalities to understand the world (Xu et al., 2023). Multi- modal learning (MML) (Huang et al., 2021a; Ge et al., 2023) has emerged promising approach to emulate human sen- sory perception. It aims to learn good representations from multiple modalities that can achieve accurate decisions.
In this context, a key question arises: what defines a "good" MML representation? Existing methods typically define good MML representations from two perspectives: modal- ity consistency and modality specificity (Zhang et al., 2023a; Ge al., 2023; Radford al., 2021; Fan et al., 2024; Dong et al., 2024). The former emphasizes extracting modality- shared semantics related to primary events within the MML task. This type of method (Liang et al., 2022a; Xia et al., 2024; Abdollahzadeh et al., 2021) aims to obtain unified representations by mapping features from different modal- ities into common embedding space. In contrast, the second perspective believes modality specificity captures the distinct statistical properties of each modality, reflecting different aspects of the primary events. This type of method (Dong et al., 2024; Yang et al., 2022; Frost et al., 2015; Zhou et al., 2021) decomposes features within each modal- ity into modality-specific and modality-shared components, learning all shared components while applying distance con- straints modality-specifft features enhance diversity.
However, from a causal perspective (Ahuja et al., 2020; Koyama & Yamaguchi, 2020), these methods may result in the learned representations being insufficient or unnecessary. Specifically, sufficiency indicates that the use of the repre- sentations will establish the label, while necessity indicates that the label becomes incorrect when the representations are absent (Pearl, 2009). Figure 1 provides an example. If the MMI. model only focuses on causal sufficiency, it will lose important modality-specifii semantics, affecting gener- alization; if the model only focuses on causal necessity, the
"Equal contribution IInstitte of Software Chinese Academy of Sciences, Beijing, China University of the Chinese Academy of Sciences, Beijing, China 'Tsinghua University, Beijing, China "Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou), China Department of Com- puter Science and Engineering, The Hong Kong University of Science and Technology Hong Kong SAR, China. Correspon- dence to: Wenwen Qiang <qiangwenwen wiretteewe@@sss...... ac.>..ac.cn>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Training
Testing
202 Wang et al., 2024) undermine monotonicity. These challenges render the conditions of traditional causal suffi- ciency and necessity discussion inapplicable.
"duck paws" sufficient but unnecessary
To address this, in this paper, we relax the above assump- tions to explore the causal sufficiency and necessity in MML., ensuring the quality of the learned representation. Firstly, we propose the definition of causal sufficiency and necessity, i.c., causal complete causes (C3), for MML based on (Pearl, 2009). It reflects the probability that the learned represen- tation is causally complete by estimating the probability of label change after an intervention on the representation, given two conditions of observations. One condition is for sufficiency, and another is for necessity. Then, we analyze the causal identifiability of C', which allows us to quantify C using the observable data in practice. Unlike previous studies, we propose instrumental variable to ensure esti- mating the segmented effect of C without spurious correla- tions, thereby relaxing the assumptions of exogeneity and monotonicity. Based on this, we propose the measurement of C3 using the twin network, i.., C3 risk, where a low C risk means that the learned representations are causally complete with high confidence. The challenge of C mea- surement lies in eliminating the spurious correlations in the sufficiency evaluation and modeling the counterfactual data required for the necessity evaluation. The proposed twin network addresses this through (i) the real-world branch: us- ing the proposed instrumental variable to eliminate spurious correlations in the representation, and (ii) the hypothetical- world branch: using provable gradient- based adjustments to model the counterfactuals. Through theoretical analyses, we prove the reliability of the twin network and provide the performance guarantee of the C risk. Based on these theoretical results, we propose Causal Complete Cause Reg- ularization (C*R), a plug-and-play method to learn causal complete representations by constraining their C' risks.
A duck standing on surface with striking orange webbed paws..
A duck swimming the lake with its wings folded, the feathers...
"wings" necessary but insufficient
duck flying with its wings spread and striking orange webbed paws..
A little bird flying with its wings spread, has blue feothers and dark groen...
Figure I: Example of causal sufficiency and necessity in "duck" classification task (See Section 2 for more analyses).
decisions may be made incorrectly based on the background, affecting discriminability. Correspondingly, considering the aforementioned two types of MML methods: (i) those fo- cused on modality consistency may cause the model to overlook important modality-specifie semantics; (ii) those focused on modality specificity may incorrectly capture ir- relevant information. Thus, existing MML methods (Dong al., 2024; Lu et al., 2022) without causal constraints may fail to satisfy sufficiency and necessity, affecting model per- formance. The experiments in Section 6.2 further prove this (Figure 3 and Table 1): (i) the representations learned by existing methods have much lower correlation scores with sufficient and necessary causes than the proposed method with causal constraints; (ii) after constraining the learned representations, the performance of existing methods are significantly improved. The analyses in Section 4 also em- phasize the importance of causal sufficiency and necessity. Thus, inspired by (Pearl, 2009; Yang et al., 2024), we pro- pose that a good MML. representation must be both causally sufficient and necessary, i.c., causally complete.
The main contributions are as follows: (i) We propose the definition, identifiability, and measurement of the causal complete cause (Ct) concept for MML without the assump- tions of exogeneity and monotonicity. These results provide an effective way, i.e., C3 risk, to estimate the sufficiency and necessity of the learned representation for robust and ac- curate MML. (Section 3). (ii) We theoretically demonstrate the effectiveness and reliability of the proposed C3 risk. In- spired by this, we propose CRR, which can be applied to any MML model to learn causal complete representations with low C risk (Sections 4 and 5). (iii) Finally, we conduct extensive experiments on various datasets that prove the effectiveness and robustness of CRR (Section 6).
Noticeably, in practical MML applications, constraining the causal sufficiency and necessity of learned representations remains challenging. Specifically, the related discussion in existing works (Yang et al., 2024; Pearl, 2009) is based on the exogeneity and monotonicity assumptions (Definition D.1 and Definition D.2 in Appendix D.3). Exogeneity refers to the scenario where the influence of the external intervention on the conditional distributions is negligible when the causal representation variable is exogenous rela- tive to the label variable, while monotonicity illustrates the consistent, unidirectional effect on the label of the represen- tation. However, in practice, non-trivial corner cases often arise: the inseparability of MML semantics leads to spurious correlations (Figure 2), and cross-modal conflicts combined with nigh-dimensional nonlinear interactions (Huang et al.,
2. Problem Analysis
Problem Settings Let X, V, and 2 represent the input space, label space, and latent space, respectively. Consider
2

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
MML task involving data (4r E X' x V, where x E X' denotes the sample and y E Y denotes the label. All the data are sampled from the same joint distribution Pxy over X' x y. The training dataset is defined as Dr = {(r,, 34) 3:) and the testing dataset is defined as Die- Here, each sample xi = {=/"" ...... _ ( contains K modalities. The objective of MML is to obtain robust model fo W g that performs well on the unseen test dataset Die. Here, the g: X' + Z is the feature extractor and W: 2 + y is the classifier. The specific implementation is illustrated in Appendix G. and the table of notations is shown in Table 4.
Fe
observable variable
O
unobservable variable
X
X
Y
true correlations
FS
FS
spurious correlations
Figure 2: Structural Causal Model (SCM) for MML. Left: causal generating mechanism, Right: the learning process.
prediction. This is also the goal we explore in this study, More analyses are provided in Appendix D.2.
From the perspective of data generation (Suter et al., 2019; Hu et al., 2022b), each sample is considered to be generated using set of generating factors, e.g., color, shape, etc. In the context of MML, each sample contains generating factors from all modalities. From a causal perspective (Pearl, 2009), these generating factors can be divided into task- related factors, i.e., causal factors Fo. and task-independent factors, i.e., non-causal factors F.- Among them, the causal factors are related to the task label, supporting accurate predictions (Deshpande et al., 2022). Thus, the goal of fo can be re-established as learning representations that contain all causal generating factors F, from the data without F,,
Causal Analysis of MML We construct Structural Causal Model (SCM) for MML based on the causal generat- ing mechanism (Suter et al., 2019; Deshpande et al., 2022) in Figure left. In this SCM, Y and X denote the label vari- able and corresponding generated data variable in the MML task. F. and F, represent the distinct sets of generating factors that are causally and non-causally related to Y. Each generating factor corresponds to semantic of the MML task, c.g., color, shape, background, modality indicator, etc. Since both F. and F, represent high-level knowledge of the data but only F. is causally related to Y, we could naturally define the MML task label variable Y as the cause of the Fe. i..., Y Fo. There no connection between F2 and Y. Following (Hu al., 2022a; Deshpande et al., 2022), the sample in each modality is generated simultancously using all the generating factors, including factors caused from the label and unobservable variables, e.g., environmental effects. The unobservable variable results in the generating factors F., which are non-causally related to Y. Then, we get both F. - X and F, + X. Thus, we obtain Figure 2 left. Based this, we further construct an SCM to discuss the learning process of MML, as shown in Figure 2 right. It can be viewed as the inverse process of the causal gener- ating mechanism. Based on Figure 2 left, an ideal MML model should only utilize causal generating factors F, and be invariant to any intervention on non non-causal generating factors Fg. However, in practice, F. and F, in the learned representation may be coupled (Dong et al., 2024; Li et al., 2023). This leads to the model potentially learning based on non-causal F., There exist spurious correlations between F, and Y, i.c., we get the additional F, + Y in Figure 2 right.
Example of Causal Sufficiency and Necessity As shown in Figure 1, assume the goal of this MML task is classify the "duck", the analysis is conducted under specific task and data conditions. The training data of this task contains three modalities, i.e., image, text, and audio, and all the modalities have the feature "duck paws". Then, the models that learn representations based on consistency may capture the features of "duck paws", and can establish the label "duck" based "duck paws". However, the model may make errors in another "duck" scenario where the MML samples do not contain "duck paws", e.g., "duck swims on the lake" (upper right in Figure 1). This suggests that the learned representation contains sufficient but unnecessary causes: the label "duck" can be predicted using the current representation (Line Left). but may not work in another scenario (Line Right). Similarly, the models may also learn necessary but insufficient representations, e.g., the rep- resentations with feature "wings". Take the Line 2 of Figure 1 as an example, "duck" must have "wings", but the samples with "wings" may also correspond to another label, e.g., "bird" (lower of Figure 1), leading to incorrect predictions. Under this context, a sufficient and necessary feature can be "flat duck bill" as its presence indicates "duck" and every "duck" sample includes it. Briefly, for sufficient and neces- sary causes, they ensure that the learned representations not only reflect the features targeted the "duck" label, but also the features that the "duck" label must have. Thus, a good representation must have both causal sufficiency and neces- sity, i.e., causal complete causes, for accurate and robust
The presence of spurious correlations leads to inaccuracies in MML. Specifically, the learned MML representations may fall into four scenarios: (i) sufficient and necessary: contains all Fo: (ii) necessary but insufficient: contains part of Fe; (iii) sufficient but unnecessary: contains all F. with Fs; (iv) insufficient and unnecessary: only contains F.. See Appendix D and G for more analyses. To ensure accurate and robust MML, in this paper, we aim to propose a method- ology that constrains the model to learn representations that contain only causally sufficient and necessary information.
3

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
3. Causal Complete Cause
sary but insufficient causes, sufficient and necessary causes, and insufficient and unnecessary causes (Appendix D.1 for detailed analyses). We aim to constrain the C* score of the learned representation, i.c., extract sufficient and necessary causes, to achieve robust and accurate MMI..
To access the learning of causal sufficient and necessary rep- resentations, in this section, we first provide the definition of causal sufficiency and necessity in MML, i.e. Causal Com- plete Cause (C9). Next, we discuss the identifiability of C which ensures the quantification of C through observable data in practice, even in corner cases (non- monnoonotty and non-exogeneity). Finally, we give the measurement of C i... Ca risk, to measure the probability of whether the learned MMI. representationn is causally complete.
3.2. Causal Identifiability of C
Since it is difficult to obtain all samples in the multi-modal data distribution, especially in real systems, e.g., the coun- terfactual data in the definition of C* is difficult to obtain (Kusner et al., 2017: Morgan & Winship, 2015), calculating the probability of C is still challenging issue. To access the calculation of C3 based on observable MMI data, we discuss the causal identifiability of C3 in this section.
3.1. Definition of C
To learn representations of causal variable F. for MML that is with both causal sufficiency and necessity, based on (Pearl, 2009; Yang et al., 2024), we introduce the concept of the probability of Causal Complete Cause (C9) as follows:
Identifiability refers to the ability to uniquely infer causal effects from observable data under given assumptions (Pearl, 2009). For MML settings, means the causal factors within the representation Z can uniquely determine the label Y from the sample X. To ensure reliable estimation, we con- strain the C score of the learned representation Z extracted from X, ensuring its identifiability. In previous studies (Tian & Pearl, 2002; Yang et al., 2024), identifying causal proba- bilities typically assumes that statistical data are derived un- der exogeneity and monotonicity conditions (Definition D.1 and Definition D.2 in Appendix D.3). Exogeneity refers to where the external intervention's 's impact on the conditional distribution P P(Y[Z) is negligible when the variable Z is exogenous to Y. This ensures the learned Z to recognize F, without being affected by F.. Monotonicity, on the other hand, describes the consistent, unidirectional effect of the representation Z on Y. This ensures that the impact of the learned Z on the labeling decision will not be reversed as other factors change. However, in practice, non-trivial cases often arise: the inseparability of generating factors in the latent space leads to spurious correlations, i.c., the learned representations Z contain non-causal F, (Figure 2); Mean- while, modality conflicts and the sigh-dimensional nonlin- ear interactions (Wang et al., 2024; Huang et al., 2021b) un- dermine monotonicity. These challenges render traditional identifiability conditions inapplicable. To address this, we relax the above assumptions and explore the identifiability of C assessing the calculation of the C score.
Definition 3.1 (Probability of Causal Complete Cause (C*)). Denote that the data and corresponding label variables of the given multi-modal data distribution are X and Y, while the variable of the learned MML representation is Z. Let the specific implementations of representation variable Z as e and e, where e denotes the implementation that results in the accurate label prediction Y y. and ē * e denotes the implementation resulting in Y A y. The probability that Z is the causal complete cause of Y can be defined as:
3 (Z)= := P(Ydoozza) = Z = a, Y + v) P(Z Y v)
Sufficiency
+ P(Ydoizzo v (= P(Z= = Y, v).
Nucossity
where P(Y do = y 1 E,Y + y) denotes the probability of Y = y when force Z to be another specific implementation e via do-operator do(Z. = c). when giving observations Z = č and Y * y with probability P(Z = e, Y # y). Similarly, the second term corresponds to the case that where the observations are Z. e and Y y, the probability of Y becomes incorrect when force Z = C.
Definition 3.1 indicates that when Z with a high C score, it has a high probability of being the causal complete cause of Y. The two terms in E correspond to sufficiency and necessity, respectively. Specifically, the sufficiency term P(ooo(z=) = y Z = č, Y 7 y) means that even when Z= and Y * y. intervening to set Z crsults in a high probability of Y y. indicating that Z has sufficient causal effect on the occurrence of Y = y. Correspondingly, for necessity term P(Y # 7. e. Y = y), even when Z = and Y = y, intervening to set Z = č results in Y # y. indicating that Z = e is necessary for the occurrence of Y = y. Under this condition, the learned representation can be divided into sufficient but unnecessary causes, neces-
Consider the effects of spurious correlations, i.e., P(Y # P(Y Z cc, we first relax the exogene- ity assumption (with definition and analysis in Appendix D.3). Based on Definition 3.1, we extend Theorem 9.2.15 in (Pearl, 2009) to MML for C3, obtaining:
Theorem 3.2 (Causal Identifiability under Non-Exogeneity). Given the MML model fe. where the label variable Y is influenced by the causal factors F. and non-causal factors Fg. Assume fe satisfy the Positive Markovian assumption (Tian & Pearl, 2002), the probabilities of C (P(Yavizze) =
4

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
y: YdolZ=e) + 4)) is identifiable and can be estimated via:
3.3. Measurement of C
(²) = P(Yaotzme)) - P(Yda(zea)
(2)
Based on Definition 3.1 and Theorem 3.3, we provide the measurement of C in this section, i.., C risk, to estimate the C score of the representation distribution P(Z|X = x) inferred from X on the multi- data distribution Pxy. When the learned representation obtains less necessary and sufficient information, the C* risk will be higher. Specifi- cally, we first model the instrumental variable V based on Theorem 3.3. It aims to make the MML model learn rep- resentations Z. that only contain the causal factors F. for crrision-making while ensuring the identifiability of the C score assessment (Theorem 3.4). Next, based on Defini- tion 3.1, we provide the C33 measurement of the learned Z, i.e., C risk (Theorem 3.5). We employ a twin network (Figure 5) with a real-world branch (extract causal repre- sentation using the proposed instrumental variable) and a hypothetical-woorld branch (modeling counterfactual data with gradient-based adjustments), thereby modeling the suf- ficiency and necessity terms. Appendix D.4 establishes the reliability of the proposed twin network.
where Y is monotonic relative to 7. with Ydo(Zze) = y/ A
Ydo(Z=e)= = y is false or Yao(Z=o) = y A Yan(Z=e) = y is false, and the model fo satisfies local invertibility.
Theorem 3.2 establishes that, under the monotonicity con- dition, CO can be estimated using observable multi-modal data, thereby quantifying C The local invertibility (Propo- sition D.3) states that the model can uniquely recover the distribution of & from the conditional distribution of Y given its parents Pa(Y). Specifically, by modeling the effects of spurious correlations (F, on Y) as non-causal pathways in the SCM, interventions with the do-operator, such as P(Y dolz=e)) and P(Y do(z=a)), facilitates reliable causal ef- fect estimation. For counterfactual terms (Z = 2) involved in the necessity probability, the model Le imposes locally reversible constraints to ensure identifiability, as discussed in (Galles& & Pearl, 1998). Under these conditions, the proba- bility of causal sufficiency C and the probability of causal necessity C can be estimated as follows:
Firstly, as shown in Section 2, we establish the model So using the feature extractor y and the linear classifier gX)
C3 Ca (Z) P(Yduiz-o)) P(YaoiZ-a)
(3)
1-P(Z=o)
W Rd + y on the overall representation
C. (Z) P(Yauz=a)) P(Yaoizme))
(Z =Z= + F.,) obtain the label Y = g))n X, we use the instru- To infer F. from observable data
(4)
P(Z-o)
Next, according to the second paragraph of Subsection 3.2, considering the cross-modal conflicts with high-dimensional nonlinear interactions in practice we relax the assumptions of monotonicity. The key idea lies in introducing an instru- mental variable V (Caner& Hansen, 2004) to estimate the piecewise effects of conditional distributions within the C definition. Meanwhile, V satisfies V TT F, with control- lable impact on Fz, i.e., V F + Y. Then, we have: Theorem 3.3 (Causal Identifiability under Non-Monotonic- ity and Non-Exogeneity). Given Ta that learns the causal effect Z- + Y. The representation variable Z E.Fo+E.Fs where [.) is the weight matrix. Introducing an instrumental variable V satisfying P(Y| Z, V) = P(Y| Z), we have:
mental variable V. Here, we focus on how V is modeled to constrain Z only contains F. for decision-making while satisfying the conditions in Theorem 3.3. We propose:
Theorem 3.4 (Instrumental Variable V in MML). For mul- timodal input X E Rx representing K modalities each with a feature dimension of d, instrumental variable V is introduced to extract causal generating factors Fe while reducing the influence of non-causul factors F.. V is gener- ated from X using self-attention mechanism with:
exp(sin) X,Wv, E k= exp(ssn)
K
(6)
q ky -all: k21
(2) = / P( = 2 = e, V v) 0
where
813
Vd
(5)
where qi = XXWQ and kj = X,WK are the query and key i- and j-th modalities' features, Wo, WK. Wy E dddd are linear projection matrices, mapping query, key and value respectively. Then, V satisfies (i) P(Z V) # P(Z); (ii) P(F, V) = P(F.); and (úi) P(Y Z, V) = P(Y F.).
- P(Y= = y Z= ج V = v) P(V = de.
Theorem 3.3 posits that the estimation of C is achievable without the dependence on exogeneity and monotonicity, en- abling the quantification of O3 in the absence of counterfac- tual data. The detailed discussion is provided in Appendix D.3, and the proofs of the above theorems are provided in Appendix A.I and A.2, respectively. According to this theorem, the practical estimation of (33 hinges on how the instrumental variable V is modeled and how the sufficiency and necessity terms of C3 are constrained under V, also the main focus in the next subsection, i.e., measurement of C3
In Theorem 3.4, we propose using the self-attention mecha- nism to model the instrumental variable V for C Specifi- cally, the alignment score Sij is to capture inter-modal in- teractions for labels. Then, by strengthening the correlation between V and Z, the learned representation Z is guided to approach F. while weakening the correlation with F.. Fur- thermore, consider inter-moda misalignment, l|q: ky 12
5

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
with weight o is introduced as a distance-basee based adjustment to diminish the impact of modalities with large feature dis- tances on attention weights. It satisfies three conditions: (i) Relevance: V manipulate Fz; (ii) Independence: V is condi- tionally independent from Fs; and (iii) Exclusion: the influ- ence of V on Y is completely indirectly realized through F-- The detailed proofs are provided in Appendix A.3.
4. Performance Guarantee with C' Risk
In this section, we conduct theoretical analyses to establish the connection between the proposed C risk and MML generalization performance, proving its effectiveness.
Before discussing generalization, we first illustrate the em- pirical and expectation risks of the MML model fo on train-
ing and test data, i.e., RC (fo) (Eq.8) and R (fo) E(z.g)~Pxy (lo Fe.s) * y.l + pla(WW Fe.) = yi]. Then, we provide the following performance guarantee:
Next, based on these results, we provide the measurement of C3, i... C risk. Specifically, we first utilize V constrain fo. effectively distinguishing F. from F., and ensuring the identifiability of Ca based on Theorem 3.4. Obtaining Ze that only contains F.. we then define sufficiency risk (when Z = c, the label is not y) and necessity risk (when Z = č, the label is y) to construct the C risk based on Definition 3.1 and 5q The key challenge lies in the counterfactual data, ie.. Ze : Z = č, is difficult to obtain (Kusner et al., 2017; Morgan & Winship, 2015). To address this, we propose twin network comprising: (i) the real-world branch to branch
Theorem 4.1 (Performance Guarantee via Ca), Denote the hypothesis set H. of linear classifier W E H C {h: : IR + y}, let H = Pdimm (f&x h E H}). Then with probability at least - 6, and training samples that sampling from empirical distribution (x. y). we have
(mm(1/))
(Jo) RCO()O) S (P)) (fo)
(FO) MA((H) + +V
(9)
2H
obtain Z. with V, and (ii) the hypothetical to obtain Ze under a provable gradient- gradient-based adjustment, ensuring Ze stay close to the original observable distribution (Appendix D.4). These branches share network structures and parameters, maintaining a mirrored correspondence that enforces causal consistency (Pearl, 2009). Thus, get:
Here, R(H) is the Rademacher complexity of the hypothesis class H. and M is a finite constant.
Theorem 4.1 derives the upper bound on the generalization error of C risk, relating its empirical error on the training data to its performance unseen samples. Specifically, the generalization error is bounded by the sum of the empirical error, the Rademacher complexity of the hypothesis class of the linear classifier, and a confidence term that controls the probability of exceeding the bound. This result provides theoretical performance guarantee for optimizing MML models with C risk. See Appendix A.5 for detailed proofs.
Theorem 3.5 (C' Risk). Given N observable samples {(x,, W) drawn from the multi-modal distribution Pxy. and the model fo with feature extractor g(.) and linear classifier W, we define the following twin-nelwork:
Zo,i = 9="(x;;; vi),
Zo, 5 -Ze, Dir
where A = 20, ( (0(W v:):
5. Learning Causal Complete Representations
s.t. MKL (o.. 20.1) Vi,
where 9" (x,, v1) is the calibrated g with U; using Kull- back Leibler divergence (xu(-), ie., with loss Lu = B....)) Pu.evpkulg(a)) (g(a())()),,))))))))) vz), to estimate the causal rep- resentation Ze.i in the reul-world branch, Du arises from the gradient-bar based intervention with ~ V for the hypo- thetical world. If fo converges to an optimal solution under a Monte Carlo approximation of V, then by using:
Based on the above theoretical results, we propose plug- and-play method, Causal Complete Cause Regularization (C*R), which is built upon the Cs risk to extract causal complete representations from observable multi-modal data.
Specifically, first introduce the constraints of the instru- mental variable V (Theorem 3.4) to make the MML model So learn causal representations while satisfying the identi- fiability in practice. Then, we minimize the C risk of the learned representations (Theorem 3.5) to ensure the causal completeness of the learned representation, i.e., causal suffi- ciency and necessity. In summary, the objective of CR is a combination of the above two-step objectives, which can be embedded in various MML models. It can be expressed as:
RC + [ 2..)) # vll + plooWT = mi). (8)
the discrepancy beiween NC and the ideal causal com- plete quantity approaches zero with high probability. Here, a denotes the signum function, p(-) denotes an indicator function which is equal to the condition in p(-) is true.
min RC + +Xuf + AjoLfe fo
Theorem 3.5 presents method to measure the causal com- pleteness of learned representations based on observable data. It leverages instrumental variables V and a twin net- work to address the issues of spurious correlations in MML and the unavailability of counterfactual data in practice. The detailed proofs and analysis are provided in Appendix A.4.
(10)
where Xu and Xye denotes the importance weights, the three terms: (i) RC (Eq.8) constrains the model to learn a causally complete representation, i.c., one that exhibits minimal sufficiency and necessity risk; (ii) f... built upon
6

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Table 1: Performance comparison when 50% samples are corrupted with Gaussian noise, i.e., zero mean with the variance of N. "(N, Avg.)" and "(N, Worst.) denotes the average and worst-case accuracy. The best results are highlighted in bold.
Mathed
(0,Avg.)
Wontt. I (10,Avg.)(00)) Woost
NYU Depth V2
[0.Arg.)
(0.Arg.(0,WWrs.. (0,Worst.) (10.Avg.) (10. (...Ar) (0, Warst. (101,Agg.) (1B.Arg.)(11.))))) Warst. (an,Wr (0.Avg.) (G), (U.. (10.Avg.1 (oot.))).))
SUN RGB D
FOOD 101
MVSA
CLIP 1San ol., 2023)
69.32
68.29
51.67
48.54
56,
54,75
35.65
32.76
55.24
54.20
52.12
49.31
62.48
61.22
31.64
28.27
ALIGN (Ria al. 2021)
66.43
64.31
45.34
42.42
57.5
56.26
38.43
35.13
86. 14
55.00
50.85
63.25
30:55
MaPLe (Khanak al., 2023)
71.26
69.27
52.98
45.73
62:44
61.76
34,51
30.29
90.40
$6.28
53.16
40.21
77.43
75.36
413.72
38.82
CoOp
2022a)
67.48
66.94
49.43
62
58.
56.31
39.67
35,45
58.33
$5.10
55.24
51.01
74.26
73.61
42.58
37.29
VPT (Jia al., 2022a)
62.16
61.21
41.05
37.81
54.75
53.92
37,48
29.81
83.80
$2.00
49.01
65.87
64.98
32.79
29.21
Laie fusion (Wing et 2016)
69.14
65.35
51,99
44.95
62.09
60.55
47.33
64.60
90.69
90.58
$5.00
55.
76.88
74.76
55.16
47.78
ConcatMMLL (Zhang al., 2021)
70.30
69.42
53.20
47.71
61.90
61.19
45.64
42,95
89.43
53.79
56.02
54.33
75.42
75.33
53.42
50,47
AligsMMI. (Wang ot 2016)
70.31
68.50
51.74
44.
61.12
60.12
44.19
38.1
55.11
52.76
74.91
72.97
52.71
47.03
Concatioon (Zhang et 2023c)
49.64
43.66
31.43
29.87
41.25
40.54
26.76
24.27
70.77
70.68
35, 68
34,92
64,09
62.04
45.40
40,95
ConcaREET (Zhang al., 20236)
70.56
29
88.
64.74
46.12
MMTM (Joer et 2020)
71.04
70.18 18
52.28
46 18
61.72
60.94
46.03
44.28
89.75
$9.43
57.91
54.98
74.24
73.55
54.63
49.72
IMC (Han et al. 2020)
71.06
09.57
53.36
49.23
60.68
60.31
45.66
41,60
S9.86
$9.80
61.37
61.10
74,88
71.10
00.36
53.37
LCKD (Wang
2023b)
68.01
66.15
40.56
56.43
43.21
42.45
85.32
84.26
47.43
44.22
62,44
62.27
43.52
38.63
UniCODE (Kia all. 2024)
70.12
68.74
44.78
42.77 79
59:21
58.55
46.32
42:21
88. 39
87.21
51.28
47.95
66.97
65.94
48.34
42,95
SimMMDG (Dong et al., 2024)
71.34
70.29
45.67
44.83
60.54
60.31
47.86
45.79
$9.57
$8.43
$2.55
50.21
67.08
66.35
49.52
44.01
MMBT (Kiela al. 2019)
67.00
65.84
47.24
56.91
43.28
56.75
56.21
78.50
78.01
QMF (7hing et at., 2023c)
70.09
68.81
55.60
$1.07
62.00
61.30
48.58
47.50
92.92
92.72
62.21
61.76
78.07
76.30
61.28
57.61
CLIP+CRR
76.54
75.12
56,73
52.90
62.31
58.71
41.59
37.52
92.93
91.80
59.77
57.54
69.61
68.64
39.58
35.89
77.07
74.45
59.94
55.95
66.21
40.12
37.34
94.
60.63
46.07
81.19
Late IR
73.26
71.63
57.21
50.98
64.84
63:25
53.35
50.43
94.
92.34
65.27
59.02
83.77
79.79
62.14
$2.50
LCKD+O*R
77.14
75.12
50.11
47.98
60.97
60.14
47.23
46.21
90.89
90.14
$4.48
51.16
66.78
65.67
49.28
42.84
SLMMMDGIC*K MMBT+C0R
75.32
74.61
49.99
47.22
64,58
52.69
51.70
92.24
91.14
57.32
53.56
71.01
73.74
71.83
54.35
52.57
61.47
$9.99
48.42
46.07
94. 25
93.90
60.41
60.11
87.76
81.64
62.12
58.93
QMF+C"R
77.58
74.95
59.72
39.18
67.35
65.84
$2.26
51.28
94.87
93.79
66.45
63.69
83.13
81.98
66.66
64.51
Table 2: Performance with missing modalities on BraTS. The brackets "0" indicate the effect changes after introducing C and "o" indicate the availability and absence of the modality for testing. The best results are highlighted in bold.
Modalities
Enhancing Tumour
Tumour Core
Whole Tumour
FUTI The T2
HMIS HVED RSeg mmFmm LCKD LCKD+CCR [MMS HVED RSeg mmFm I. LCKD LCCDD*C*R HMIS
HVED RSey mmnFmm I. LCKD LCKD+*CR
11.78
23.80
25.69
39.33
45.48
49.81 (44.33) 26.06
57.90
53.57
61.21
72.01
76,65 (14.64)
52,48
84,39
85,69
86,10
89.45
91.62 (+2.17)
10.16
8.60
17.29
32,53
43.22
49. (-6.01) 37.39
33.90
47.90
56.55
66.58
72.18 (+5.60) 57.62
49.51
70.11
67.52
76.48
82,39 (+5.91)
0
62.02
57.64
67.07
72.60
75.65
80.50 (+4.85)
65.29
59.59
76.83
75.41
83.02
88.06 (+5.04)
61.53
53.62
73.31
72.22
77.23
81.93 (++,770)
0
9
25.63
22.82
28.97
43.05
47.19
54.13 (+6.94) 57,20
54.67
57.49
64.20
70.17
77.32 (47.15) 80.96
79.83
82.24
81.15
84.37
90.78 (+6.41)
10.71
27.96
32.13
42.96
48.30
54.16 (44.86) 41.12
61.14
60.68
65.91
74.58
79.83 (45.25) 64.62
85.71
88.24
87.06
89,97
93.63 (43.66)
66.10
68.36
70.30
75.07
78.75
82.98 (+4.23) 71.49
75.07
80.62
77.88
85.67
89.74 (+4.07) 68.99
85,93
88.51
87.30
90.47
93.91 (+3.44)
30.22
32.31
33.84
47.52
49.01
56.12 (+7.11) 57.68
62.70
61.16
69.75
75.41
82.57 (+7.16) 82.95
87.58
88.28
87.59
90.39
95.48 (+5.09)
66.22
61.11 69.06
74.04
76.09
81.76 (+5.67) 72.46
67.55
78.72
78.59
82.49
88.32 (+5.83) 68.47
64.22
77.18
74.42
80.10
87.03 (+6.96)
32.39
24.29 32.01
44.99
50.09
56.03 (+5.94) 60.92
56.26
62.19
69.42
72.75
78.78 (46.03) 82,41
81.56
84.78
82.20
86.05
92.33 (+6.28)
o
67.83
67.83 69.71
74.51
76.01
83.97 (+7.96) 76.64
73.92
80.20
78.61
84.85
93.57
(+8. 72)
82.48
81,32
85
82.99
86.49
94.40 (+7.91)
68.54 68.60 70.78
75.47
77.78
(+5.06)
76.01
77.05
81.06
79.80
85.24
90
(45.22)
72.31
86.72
88.73
87.33
90.50
95.51 (+5.01)
0
31.07 32.34 36.41
47.70
49.96
56.25 (+6.29) 60.32
63.14
64.38
71.52
76.68
82.69 (+6.01) 83.43
88.07
88.81
87.75
90.46
96.23 (+5.77)
68.72 68.93 70.88
75.67
77.48
83.90 (+6.42) 77.53
76.75
80.72
79.55
85.56
92.39 (46.83)
83.85
88.09
89.27
88.14
90.90
96.78 (45.88)
69.92
67.75
70.10
74.75
77.60
82.54 (+4.94)
78.96
75.28
80.33
80.39
84.02
89.43 (15
83.94
82.32
86.01
82.71
86.73
91.73 (+5.0m
70.24
69.03
71.13 77.61
79.33
86.36 (+7.03) 79.48
77.71
80.86
85.78
85.31 91.43 (26.12) 84.74
88.46
89.45
89.64
90.84
95.41 (14.57)
Theorem 3.4, employs the instrumental variable V to guide the model toward the causal representation, correspond- ing to the real-world branch; (iii) f,r stems from the countcrfactual construction condition in Eq. 7, specifically Lie = A Σm" MKLL (Ze,i, 2o..), ensuring that Zo,; remains within the feasible region. Thus, by minimizing the above objective, the C3 score of the learned representation will be higher while maintaining the required conditions for practi- cal implementations. Then, it makes the learned representa- tion causally sufficient and necessary with high confidence.
6.1. Experimental Settings
Datasets We select six datasets: (i) scenes recognition on NYU Depth V2 (Silberman et al., 2012) and SUN RGBD (Song et al., 2015) with RGB and depth images; (ii) image- text classification on UPMC FOOD101 (Wang et al., 2015) and MVSA (Niu et al., 2016) with image and text; (iii) seg- mentation considering missing modalities on BraTS (Menze et al., 2014; Bakas et al., 2018) with Flair, TI, Tlc, and T2; and (iv) synthetic MMLSynData (see Appendix D.5).
Implementation Details We use a three-layer MLP with activation functions (Clevert et al., 2016) as the representa- tion learner. The hidden vector dimensions of each layer are specified as 64, 32, and 128, while the learned representa- tion is 64. For optimization, we employ the Adam optimizer (Kingma & Ba, 2015) with Momentum and weight decay
6. Experiments
In this section, we conduct extensive experiments on various benchmark datasets to verify the effectiveness of &R More details and experiments are provided in Appendix E-H.
7

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Boxplet of
oreghtersc
91
Abiation Study on FooD
Label
0.9
03
6.8
0.7-
Coreegtton
loes
0.5
TMc
THe
With tastff, The
ConcusiciT
Wit are mostumeddel dT and Tidt
Win tea meerg wesainen and
0.2
0.3
01
Objective
Boxslot NC
0.5
Boxplet of se
95-
Ablation Study BraTS Whole Tumour with Missing Modalities
92
45-
93
2.4
94
23
Corre
Tic
TMC
Concenhor
2.3
Maaic NC
0.1
90
Objoctiv
Objective
Figure 3: Evaluation for the property of learned representa-
Figure 4: Ablation study of C (performance when remov- ing different regular terms). See Appendix H for details.
tions (SNC, SC, NC, and SP). See Appendix H for details.
set at 0.8 and 10-1 The initial learning rate is established at 0.1, with the flexibility for linear scaling as required. Ad- ditionally, we use grid search to set the hyperparameters A 0.75 and Aye = 0.4. All experimental procedures are executed in five runs via NVIDIA RTX A6000 GPUs. Codes can be found in our Github repository'.
experiments: (i) construct four types of MML data, i.e., suf- ficient and necessary (SNC), sufficient but unnecessary (SC), necessary but insufficient causes (NC), and spurious corre- lations (SP) following (Yang et al., 2024) (see Appendix H for more details); then (ii) evaluate their correlation with the learned representation based on the distance metric (Jones et al., 1995), i.e., the higher the score, the stronger the cor- relation. The results are shown in Figure 3 (see Appendix D and H for details). The representations learned by C have a higher correlation with SNC and lower with SP. This proves that CR can learn causal complete representations more effectively, while other methods are hard to achieve.
6.2. Results
Performance and robustness analysis To evaluate the ef- fectiveness of CRR, we record the performance change across various MML baselines after introducing CR under Gaussian noise (for image modality) and blank noise (for text modality) following (Han et al., 2020; Zhang et al., 2023d; Ma et al., 2021). We record both the average and worst-case accuracy. The results are shown in Table 1. We can observe that CR achieves stable improvements in both the average and worst-case accuracy. This proves the supe- rior effectiveness and robustness of CRR.
Ablation Study To evaluate the effect of each item in C (Eq.10), we evaluate the performance change on LCKD+CAR. We record the performance changes before and after the model introduced specific terms. We consider the standard and corner case (missing modality), and con- duct experiments on FOOD 101 and BraTS. The results are shown in Figure 4. We can observe that each item plays an important role. See Appendix H for more experiments, including the analyses about model efficiency and param- eter sensitivity. We provide brief summary about all the experiments conducted in this paper, as shown in Table 3.
When faces the problem of missing modalities Consid- ering the missing modality issues, we evaluate the perfor- mance of C$R and several strong baselines (Wang et al., 2023b; Bakas et al., 2018; Zhang et al., 2022b) on all 15 pos- sible combinations of missing modalities on BraTS. From Table 2, we can observe that (i) CR brings sigmificant per- formance improvements; (ii) C3R can reduce the learning gap for the representations different modal semantics, i.e., reducing the accuracy gap of learning on the difficult- to-identify Fl and TI modalities and the easy Tlc. This demonstrates the superiority of Ca and the advantage of causally complete representation in missing modality issues.
7. Related Work
Multi-modal learning aims to learn good representations through multiple modalities for accurate prediction. Re- cently, multiple methods (Xu et al., 2023; Jia et al., 2022b; Wang al., 2023c; Fan al., 2024) have been proposed to solve MML tasks through modality consistency, e.g., tokenize diverse modalities into sequences and utilize Trans- formers for joint learning (Bao et al., 2022; Wang et al., 2022), whereas CLIP (Radford et al., 2021; Fan et al., 2024), ALIGN (Jia et al., 2021), etc. employ distinct encoders for each modality and utilize contrastive loss to synchro-
Learning causal complete representations To evaluate COR's ability extract causal complete causes, conduct
'Codes can be found https:/ htapps///gthhan gigghub.cco.. WangJingyao0" wwapayyyyo00MMMaa /Mt/ttitiiiiia -Modal -Base
8

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
nize features. These methods align features from different modalities into the same space capture the primary events.
References
Abdollahzadeh, M., Malekzadeh, T., and Cheung, N.-M. M. Revisit multimodal meta-learning through the lens of multi-task learning. Advances in Neural Information Processing Systems, 34: 14632-14644, 2021.
Another type of work (Jiang al., 2023; Dong et al., 2024; Wang et al., 2023a; Wu et al., 2020) proposed to addition- ally focus on modality specificity. They divide the MMI. features into modality-sharee and modality-specific compo- nents and learn separately. However, all these methods may result in learning insufficient or unnecessary information. Meanwhile, they rely strong assumptions like semantic alignment (Akbari et al., 2021; Wang et al., 2023c; Lu et al., 2022; Zhang et al., 2023b) and may be limited to ideal data (Ge et al., 2023; Zhang et al., 2023a), affecting model perfor- mance. For causal sufficiency and necessity, (Pearl, 2009) first proposed the related concepts, and (Yang et al., 2024) applied it to domain generalization with assumptions in an ideal environment. The differences between our method- ology and previous causal-related works include problem settings, theorems, implementations, learning objectives, etc. (More comparison and analyses are provided in Ap- pendix D.6). We explore the MML-specific concepts of causal completeness without strong assumptions, ensuring the effectiveness of the learned representations.
Ahuja, K., Shanmugam, K., Varshney, K., and Dhurandhar, A. Invariant risk minimization games. In International Conference on Machine Learning, pp. 145-155. PMLR, 2020.
Ahuja, K., Mahajan, D., Wang, Y., and Bengio, Y. Interven- tional causal representation learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 372-407. PMLR, 2023.
Akbari, H., Yuan, L., Qian, R., Chuang, W.-H., Chang, S.-F., Cui, Y., and Gong, B. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34: 24206-24221 2021.
Arslan, Y., Atabek, A., Hulagu, T., and Şahinöz, S. Expec- tation errors, uncertainty, and economic activity. Oxford Economic Papers, (7(3):6346660 2015.
8. Conclusion
We explore the MML-specific concepts of causal sufficient and necessary without exogeneity and monotonicity assump- tions. To measure whether the learned MML representation is causally complete, we present the definition, identifiabil- ity, and measurement of C3 with theoretical support. Based on these results, we propose plug-and-play method CaR to promote MML learning causal complete representations. Extensive experiments demonstrate its effectiveness.
Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shinohara, R. T., Berger, C., Ha, S. M., Rozy- cki, M., al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv: 18lt::22110029 2018.
Baltrušaitis, T., Ahuja, C., and Morency, L.-P. Multimodal machine learning: A survey and taxonomy. IEEE trans- actions on pattern analysis and machine intelligence, 41 (2):423-443, 2018.
Impact Statement
This paper presents work whose goal to advance the field of Machine Learning and Multi-Modal Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O. K., Aggarwal, K., Som, S., Piao, S., and Wei, F. Vlmo: Unified vision-language pre-training with mixture-of- modality-experts. Advances in Neural Information Pro- cessing Systems, 25:32897-32911 2022.
Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable comments. This work was supported in part by the National Natural Science Foundation of China (Grant No. 62406313), in part by the National Key R&D Program of China (Grant (0.2023YFF00250001, in part by the National Natural Science Foundation of China (Grant No.92370204), in part by the guangdong Basic and Applied Basic Research Foundation (Grant No.2023B15113 1515120057) in part by Guangzhou-HKon HKUSST (GZ) Joint Funding Pro- gram (Grant No.2023A03100088 Education Bureau of Guangzhou Municipality.
Berger, M. and Gostiaux, B. Differential Geometry: Man- ifolds, Curves, and Surfaces: Manifolds, Curves, and Surfaces, volume 115. Springer Science & Business Me- dia, 2012.
Blumenthal, R. M. An extended markov property. Transac- tions of the American Mathematical Society, 85(1):52-72, 1957.
Botev, Z. I., Kroese, D. P., Rubinstein, R. Y., and L L'Ecuyer, P.
The cross-entropy method for optimization. In Handbook of statistics, volume 31, pp. 35-59. Elsevier, 2013.
9

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Brehmer, J., De Haan, P., Lippe, P., and Cohen, T.S. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319- 38331, 2022.
Intervention-MICCA. 2019: 22nd International Confer- ence, Shenzhen, China, October 13-17, 2019, Proceed- ings, Part II 22, pp. 74-82. Springer, 2019.
Fan, L., Krishnan, D., Isola, P., Katabi, D., and Tian, Y. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36, 2024.
Caner, M. and Hansen, B. E. Instrumental variable estima- tion of threshold model. Econometric theory, 20(5): 813-843, 2004.
Frost, R., Armstrong, B. C., Siegelman, N., and Chris- tiansen, M. H. Domain generality versus modality speci- ficity: The paradox of statistical learning. Trends in cognitive sciences, 19(3):1 2015.
Chen, C., Dou, Q., Jin, Y., Chen, H., Qin, J., and Heng. P.-A. Robust multimodal brain tumor segmentation via feature disentanglement and gated fusion. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part III 22, pp. 447-445 Springer, 2019.
Frydenberg, M. The chain graph markov property. Scandi- navian journal of statistics, pp. 333-353, 1990.
Galles, D. and Pearl, J. An axiomatic characterization of causal counterfactuals. Foundations of Science, 3:151- 182, 1998.
Chicco, D., Warrens, M. J., and Jurman, G. The coeffi- cient of determination r-squared is more informative than smape, mae, mape, mse and rmse in regression analysis evaluation. PeerJ Computer Science, 7:e623, 2021.
Ge, Y., Ren, J., Gallagher, A., Wang, Y., Yang, M.-H., Adam, H., Itti, L., Lakshminarayanan, B., and Zhao, J. Improving zero-shot generalization and robustness of multi-modal models. In Proceedings of the IEEF/VVF Conference on Computer Vision and Pattern Recognition, pp. 1003-111001, 2023.
Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential lin- ear units (elus). In Proceedings of the 4th International Conference Learning Representations, 2016.
Dandekar, A., Zen, R. A., and Bressan, S. A compara- tive study of synthetic dataset generation techniques. In Database and Expert Systems Applications: 29th Inter- national Conference, DEXA 2018, Regensburg, Germany, September 3-6, 2018, Proceedings, Part II 29, pp. 387- 395. Springer, 2018.
Goldberger, Gordon, and Greenspan. An efficient im- age similarity measure based on approximations of kl- divergence between two gaussian mixtures. In Proceed- ings Ninth IEEE International conference on computer vision, pp. 487 IEEE, 2003.
Gong, C., Wang, D., and Liu, Q. Alphamatch: Improv- ing consistency for semi-supervised learning with alpha- divergence. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13683 13692, 2021.
De Boer, P.-T., Kroese, D. P., Mannor, S., and Rubinstein, R. Y. A tutorial on the cross-entropy method. Annals of operations research, 134:19-67, 2005.
De Lara, L., González-Sanz, A., Asher, N., Risser, L., and Loubes, J.-M. Transport-baid based counterfactual models. Journal of Machine Learning Research, 25(136):1-59, 2024.
Han, Z., Zhang, C., Fu, H., and Zhou, J. T. Trusted multi- view classification. In International Conference on Learn- ing Representations, 2020.
Deshpande, S., Wang, K., Sreenivas, D., Li, Z., and Kuleshov, V. Deep multi-modal structural equations for causal effect estimation with unstructured proxies. Ad- vances in Neural Information Processing Systems, 35: 10931-10944, 2022.
Havaei, M., Guizard, N., Chapados, N., and Bengio, Y. Hemis: Hetero-modal image segmentation. In Medical Image Computing and Computer-Assisted Intervention- MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part 11 19, pp. 469-477. Springer, 2016.
Dong, H., Nejjar, I., Sun, H., Chatzi, E., and Fink, O. Simm- mdg: A simple and effective framework for multi-modal domain generalization. Advances in Neural Information Processing Systems, 36, 2024.
Hershey, J. R. and Olsen, P. A. Approximating the kull- back leibler divergence between gaussian mixture mod- els. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP volume 4, pp. IV-317. IEEE, 2007.
Dorent, R., Joutard, S., Modat, M., Ourselin, S., and Vercauteren, T. Hetero-modal variational encoder- decoder for joint modality completion and segmentation. In Medical Image Computing and Computer Assisted
Hu, L.. Chen, Z., Zhao, Z., Yin, J., and Nie, L. Causal inference for leveraging image-text matching bias in
10

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
multi-modal fake news detection. IEEE Transactions on Knowledge and Data Engineering, 35(11):11141-11112 2022a.
Kiela, D., Bhooshan, S., Firooz, H., Perez, E., and Testug- gine, D. Supervised multimodal bitransformers for classi- fying images and text. arXiv preprint 2019.
Hu, Z., Zhao, Z., Yi, X., Yao, T., Hong. L., Sun, Y., and Chi, E. Improving multi-task generalization via regularizing spurious correlation. Advances in Neural Information Processing Systems, 35:1 1450-1 1446, 2022b.
Kim, T., Shin, S., Yu, Y., Kim, H. G., and Ro, Y. M. Causal mode multiplexer: A novel framework for unbiased mul- tispectral pedestrian detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26784-26793, 2024.
Huang, Y., Du, C., Xue, Z., Chen, X., Zhao, H., and Huang, L. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34:10944410099 10956, 2021a.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
Huang. Z., Niu, G., Liu, X., Ding, W., Xiao, X., Wu, H., and Peng, X. Learning with noisy correspondence for cross-modal matching. Advances in Neural Information Processing Systems, 11-29406-29411 2021b.
Koyama, M. and Yamaguchi, S. When is invariance use- ful in an out-of-distribution generalization problem? In Proceedings of the 37th International Conference Ma- chine Learning (ICML 2020), 2020.
Jia, C., Yang. Y., Xia, Y., Chen, Y.TT., Parekh, Z.,, Pham, H., Le, Q. Sung, Y.-H., Li. Z., and Duerig, T. Scaling up visual and lision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 1904-4916 PMLR, 2021.
Kusner, M. J., Loftus, J., Russell, C., and Silva, R. Counter- factual faimnsss. Advances in neural information process- ing systems, 30, 2017.
Jia, M., Tang, L.,, Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022a.
Lebanon, G. Bias, variance, and mse of estimators. Georgia Institute of Technology. Atlanta, 2010.
Li, Y., Wang, Y., and Cui, Z. Decoupled multimodal dis- tilling for emotion recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6631-6640, 2023.
Jia, Z., Cai, X., and Jiao, Z. Multi-modal physiological signals based auuezze-and-excitaion network with do- main adversarial learning for sleep staging. IEEE Sensors Journal, 22(4):3464-344 3471, 2022b.
Liang, V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. Mind the gap: Understanding the modality gap in multi-moda contrastive representation learning. Ad- vances in Neural Information Processing Systems, 35: 17612-17625, 2022a.
Jiang Q., Chen, C., Zhao, H., Chen, L., Ping, Q., Tran, D., Xu, Y., Zeng, B., and Chilimbi, T. Understanding and constructing latent modality structures in multi-modal representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7661-7671, 2023.
Liang, V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- vances in Neural Information Processing Systems, 35: 17612-17625, 2022b.
Jones, T., Forrest, S., et al. Fitness distance correlation as a measure of problem difficulty for genetic algorithms. In ICGA, volume 95, pp. 184-192, 1995.
Lu, J., Clark, C., Zellers, R., Mottaghi, R., and Kembhavi, A. Unified-io: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations, 2022.
Jordan, M. and Mitchell, T. M. Machine learning: Trends, perspectives, and prospects. Science, 349(6245):255-225 2015.
Joze, H. R. V., Shaban, A., Iuzzolino, M. L., and Koishida, K. Mmtm: Multimodal transfer module for enn fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13289-13299, 2020.
Ma, H., Han, Z., Zhang, C., Fu, H., Zhou, T., and Hu, Q. Trustworthy multimodal regression with mixture of normal-inverse gamma distributions. Advances in Neural Information Processing Systems, 34:6881 118881 2021.
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113-19122, 2023.
Mahesh, B. Machine learning algorithms-a review. Interna- tional Journal of Science and Research (Intrne, 9(1 ((1):381-30) -366, 2020.

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Marmolin, H. Subjective mse measures. IEEE Iransactions on systems, man, and cybernetics, 16(3):486-489, 1986.
Sun, Q., Fang, Y., Wu, L.., Wang, X., and Cao, Y. Eva- clip: Improved training techniques for clip scale, arXiv preprint arXiv:2303. 15389, 2023.
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J.. Wiest, R., et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging, 34( (4(100)1193 1924444992,,, 2014.
Sun, Y., Kong, L.., Chen, G., L., Luo, G., Li, Z.., Zhang, Y., Zheng, Y., Yang. M., Stojanov, P., Segal, E., Xing. E. P., and Zhang, K. Causal representation learning from multimodal biological observations. In Proceedings of the 13th International Conference on Learning Represen- lations, 2025.
Morgan, S. L. and Winship, C. Counterfactuals and causal inference. Cambridge University Press, 2015.
Suter, R., Miladinovic, D., Schölkopf, B., and Bauer, S. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning, pp. 6056- 6065. PMLR, 2019.
Niu, T., Zhu, S., Pang, L., and EI Saddik, A. Sentiment analysis on multi-view social data. In MultiMedia Model- ing: 22nd International Conference, MMM 2016, Miami, FL, USA, January 4-6, 2016, Proceedings, Part II 22, pp- 15-27. Springer, 2016.
Tangirala, A. K. Principles of system identification: theory and practice. CRC press, 2018 8.
Norton, J. P. An introduction to identification. Courier Corporation, 2009.
Tian, J. and Pearl, J. general identification condition for causal effects. In Aaaifiaai, pp. 567-573, 2002.
Panaretos, V. M. and Zemel, Y. Statistical aspects of wasser- stein distances. Annual review of statistics and appli- cation, 6:405-431, 2019.
Tsai, Y.-H. H., Wu, Y., Salakhutdinov, R., and Morency, L.PP. Self-supervised learning from a multi-view perspec- tive. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020), 2020.
Pearl, J. Causality. Cambridge university press, 2009.
Piotroski, J. D. and So, E. C. Identifying expectation er- rors in value/glamour strategies: A fundamental analysis approach. The Review of Financial Studies, 25(9))2841- 2012.
Vallender, S. Calculation of the wasserstein distance be- tween probability distributions on the line. Theory of Probability & Its Applications, 18(4):784-786, 1974.
2012.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763, PMLR, 2021.
von Kügelgen, J. Identifiable causal representation learning: Unsupervised, multi-view, and multi-envinonment arXiv preprint arXiv:2406. 13371, 2024.
Wachter, S., Mittelstadt, B., and Russell, C. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31:841, 2017.
Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalch- brenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE, 109(5): 612-634, 2021.
Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L.,, and Carneroo, G. Multi-modke learning with missing modality via shared-specifie feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15878 15587, 2023a.
Shlens, J. Notes on kullback-leibler divergence and likeli- hood. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR 2014), 2014.
Silberman, N., Hoiem, D., Kohli, P.,, and Fergus, R. Indoor segmentation and support inference from rgbd images. In Computer Vision-ECCV 2012: 12th European Confer- ence on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part 12, pp. 746-760. Springer, 2012.
Wang, H., Ma, C.,, Zhang, J., Zhang, Y., Avery, J., Hull, L., and Carneiro, G. Learnable cross-modal knowledge dis- tillation for multi-modal learning with missing modality, International Conference on Medical Image Comput- ing and Computer-Assister Intervention, pp. 216-226. Springer, 2023b.
Song, S., Lichtenberg, S. P., and Xiao, J. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 567-576, 2015.
Wang. J., Wang, Z., Tao, D., See, S., and Wang, G. Learn- ing common and specific features for rgb-d semantic segmentation with deconvolutional networks. In Com- puter Vision-ECCV 2016: 14th European Conference,
12

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Amsterdam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part V 14, pp. 664-679. Springer, 2016.
Zhang, C., Chu, X., Ma, L.,, Zhu, Y., Wang, Y., Wang, J., and Zhao, J. M3care: Learning with missing modalities in multimodal healthcare data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2418-2428, 2022a.
Wang, J., Mou, L.,, Ma, L.,, Huang. T., and Gao, W. Amsa: adaptive multimodal learning for sentiment analy- sis. ACM Transactions Multimedia Computing, Com- munications and Applications, 19(3s)11-11, 2023c.
Zhang, J., Greenewald, K. Squires, C., Srivastava, A., Shan- mugam, K., and Uhler, C. Identifiability guarantees for causal disentanglement from soft interventions. Advances in Neural Information Processing Systems, 36, 2024.
Wang, J., Ren, Y., Song, Z., Zhang, J., Zheng, C., and Qiang, W. Hacking task confounder in meta-learning. In Proceedings of the Thirty-Thire International Joint Conference on Artificial Intelligence, IJCAI '24, 2024.
Zhang, Q. Wang, Y. and Wang. Y. On the generalization of multi-modal contrastive learning. In International Con- ference on Machine Learning, pp. 41677-41693, PMLR, 2023a.
Wang, P., Yang, A., Men, R., Lin. J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. Ofa: Unifying architec- tures, tasks, and modalities through a simple sequence- to-sequence learning framework. In International Con- ference on Machine Learning, pp. 2331--23340. PMLR, 2022.
Zhang, Q., Wu, H., Zhang, C., Hu, Q., Fu, H., Zhou, T., and Peng, X. Provable dynamic fusion for low-quality multimodal data. In International conference machine learning, pp. 41753-41769, PMLR, 2023b.
Wang, X., Kumar, D., Thome, N., Cord, M., and Precioso, F. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). pp. 1-6. IEEE, 2015.
Zhang, Q., Wu, H., Zhang, C., Hu, Q., Fu, H., Zhou, J. T., and Peng. X. Provable dynamic fusion for low- low-quality multimodal data. In International conference on machine learning, pp. 11753-47769, PMLR, 2023c.
Wang, Y. and Jordan, M. I. Desiderata for representation learning: a causal perspective. J. Mach. Learn. Res., 25 (I), January 2024. ISSN 1532-4435.
Zhang, Q., Wu, H., Zhang, C., Hu, Q. Fu, H., Zhou, T. and Peng, X. Provable dynamic fusion for low-quality multimodal data. In International conference on machine learning, pp. 41753-41766 PMLR, 2023d.
Wu, F., Jing, X.-Y., Wu, Z., Ji, Y., Dong, X., Luo, X., Huang, Q., and Wang, R. Modality-speciff and shared generative adversarial network for cross-modal retrieval. Pattern Recognition, 104:107335, 2020.
Zhang, Y., Yang, J., Tian, J., Shi, Z., Zhong, C., Zhang. Y., and He, Z. Modality-aware mutual learning for multi- modal medical image segmentation. In Medical Image Computing and Computer Assisted Intervention-MICCAA 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part 24, pp. 589-599. Springer, 2021.
Xia, Y., Huang, H., Zhu, J., and Zhao, Z. Achieving cross modal generalization with multimodal unified representa- tion. Advances in Neural Information Processing Systems, 36, 2024.
Xu, P., Zhu, X., and Clifton, D. A. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Zhang, Y., He, N., Yang, J., Li, Y., Wei, D., Huang, Y., Zhang, Y., He, Z., and Zheng, Y. mmformer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation. In International Conference on Medical Image Computing and Computer-Assiited Intervention, pp. 107-117. Springer, 2022b.
Yang, D., Kuang, H., Huang, S., and Zhang, L. Learn- ing crodality-specifii agnostic representations for asynchronous multimodal language sequences. In Pro- ceedings of the 30th ACM International Conference on Multimedia, pp. 7708-1717 2022.
Zhao, J., Li, R., and Jin, Q. Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 2608-2618 2618, 2021.
Yang, M., Zhang, Y., Fang, Z., Du, Y., Liu, F., Ton, J.-F., Wang, J., and Wang, J. Invariant learning via probability of sufficient and necessary causes. Advances in Neural Information Processing Systems, 36, 2024.
Yao, D., Xu, D., Lachapelle, S., Magliacane, S., Taslakian, P., Martius, G., von Kügglggen, J., and Locatello, F. Multi- view causal representation learning with partial observ- ability. In Proceedings of the 12th International Confer- ence on Learning Representations, 2024.
Zhou, T., Fu, H., Chen, G., Zhou, Y., Fan, D.-P., and Shao, L. Specificity preeffiityy-reservinng rgb-d saliency detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4681 1691,,,91, 2021.
13

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Appendix
This supplementary material provides results for additional experiments and details to reproduce our results that could not be included in the paper submission due to space limitations.
Appendix A provides proofs and further theoretical analysis of the theory in the text.
Appendix B provides the pseudo-code of incorporating C into the MML models.
Appendix C provides the detailed notations and the corresponding definitions for this work.
Appendix D provides discussions of how to better understand C score, and the three parts of learned invariant representations, ie., sufficient but unnecessary, necessary but insufficient, and sufficient and necessary causes.
Appendix E provides the additional details of the benchmark datasets.
Appendix F provides the additional details of the baselines for comparision.
Appendix G provides the additional details of the implementation details.
Appendix H provides the full results and additional experiments for the evaluation of C*
Note that before we illustrate the details and analysis, we provide a brief summary about all the experiments conducted in this paper, as shown in Table
Table 3: Illustration of the experiments conducted in this work. Note that all experimental results are obtained after five
rounds of experiments.
Experiments
Location
Results
Performance and robustness analysis
Section 6.2 and Appendix H.1
Table 1, Table 6, and Ta- ble 7
Performance comparison when faces the problem of missing modalities
Section 6.2 and Appendix H.2
Table
Performance of learning causal complete repre- sentations
Section 6.2, Appendix D.5, and Ap- pendix H.3
Figure 3 and Figure 6
Ablation Study about the effect of each item in the C objective
Section 6.2 and Appendix H.4
Figure 4
Experiment of model efficiency
Appendix H.4
Figure
Experiment of parameter sensitivity Visualization of Causal Complete Cause
Appendix H.4
Figure 8 and Figure 9
Appendix H.5
Figure 10 and Table 8
Performance When Facing Noise
Appendix H.6
Table
Role of Distance Loss
Appendix H.7
Figure
A. Proofs
In this section, we provide the proofs of (i) the 8 identifiability under non-exogeneity (Theorem 3.2). (ii) the O identifiability under non-exogyneity and non-monotoyycity (Theorem 3.3) (iii) the modeling of instrumental variable Z in MML (Conditions in Theorem 3.4), (iv) C risk with twin network (Theorem 3.5), where the reliability and completeness of the twin network demonstrated in Appendix D.4, and (v) the performance guarantee for C risk (Theorem 4.1).
14

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
A.1. Proofs of Theorem 3.2
Proof. Based on Definition 3.1, (Z) defined
C (Z) (z) = P do(Z=o) = 2
v) 2
+P (Yao7a) F c. = :)
P(Z = c, Y-
Expand the conditional probabilities into joint probabilities over marginal probabilities:
- V. -,
P
P P(Yao(Z=)) = 2=
P + y)
P (Ydol(Z=)) + y, c,
P ((Yo(ZZ=)) * y Z = Y Y= v) =
c Y
Substituting these into (Z):
C (Z) = P (Yántz=e) v, v) + P (Y # v. v,
Considering that the monotonicity assumption states:
Y do((mm)) = y do(7 y
This means there are no individuals for whom Y is under Z - è but changes to o when Z - c. Therefore:
P P( = y do)(z=)) 3)
This eliminates the second term in (Z):
P P(Yáo(Z=)) + y. c,, y)=0 =0 =0
Thus, the simplified (Z) becomes:
y, ट, +
Considering the causal model, Ydo(Z=c) can be decomposed into two parts:
Individuals for whom Y * y when = ē, but Y when intervened to = C.
Individuals for whom Y = when Z = ट, and Y remains when intervened to Z = c.
Thus:
P ( dodo = v) = (z) + P P( = U, Ydo(z=o) = v)
According to the monotonicity assumption:
Ydo(Z=o) y + Ydo(Z=o)= =y
Therefore:
P (Ydn(7=)) = y. dd = P (YYdo(z=e) = v)
Substituting this into the previous equation:
P =)) = C²) P(Yooiz-) =
Solving for (Z) from the above equation:
(Z)= = P(Ysooz-)) = v) P (Y do =
This is precisely the formula presented in the theorem.
15

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
A.2. Proofs of Theorem 3.3
Proof. Under spurious correlations and non-monotonicity, one cannot simply take
P(Y Z c) Z
as valid measure of causal effect. The key idea lies in introducing an instrumental variable V (Caner & Hansen, 2004) to estimate conditional distributions and piecewise effects. An instrumental variable V helps "decouple" F, from unobserved Fs, allowing us to recover or approximate the true intervention distribution P(Yao(z)). Specifically, if
P(Y ZZ.V P(Y Z))
then V is conditionally independent of Y given Z. By stratifying over V, we can estimate the conditional interventional distribution and aggregate (integrate) across values of V to obtain the true causal effect.
Recall the key idea in causal inference:
P(Ydo(Z=o)= -) = P(Yuoozzz)
[ = y V
Since V is assumed to provide no direct influence on aside from through Fe, we can drop the do-operator in the conditional:
P(Ydo(Z=o) =) 2
P(Y 7. c, =) v) P(V = v) dv
| P(Y = y Z= P(V = v) PiY Z. = P(Y Z)).
Likewise,
P(Ydolz=0)= = P(Y y 2, v) P(V v) de.
Recall the definition of C
(z(Z) = P(Ydo[Z=)) = = P(Ydo(z=a) = v).
By substituting the integrals,
(² =
P(Y = = - -V = = dv
P(Y -/,==v= = P(V v)
V=v= P(Y = y -=cV = v) P(Y = y Z a = =i] P(V = v) du.
Thus, get:
(² = / P(Y
|(( = = e, V - P = y z = ē, V = v) P(V = dv.
Hence, even under non-monotonicity and non-exogeneity. if there exists an appropriate instrumental variable V satisfying P(Y 2, = P(Y Z), we can stratify on V and integrate out P(V = v)) to recover an identifiable estimate for C (Z).
A.3. Analyses of Theorem 3.4
Proof. In multimodal learning setting, we denote the causal generating factors across modalities as F. and the label- irrelevant interference factors as F., and our goal is to construct an auxiliary variable 2 via a self-attention mechanism. After training, this z should sufficiently aggregate information from F.,. weaken as much as possible the influence F,, and avoid providing any extraneous intervention pathway for the label Y. Briefly, in this subsection, we illustrate how achieve the three conditions Theorem 3.4, (i) P(Z. V) # P(Z); (ii) P(F, V) P(F.); and (iii) P(Y Z, V. P(Y Fe).
16

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Building upon the earlier stage, now provide more detailed formulaic explanations, especially regarding how gradients computed.
First, we start from the classical self-attention formula. Let
4 W G=XyWKs 1/ = Wv,
where X2 Rdd represents the imput features of the i-th modality, and Wo: WK, Wv Rxx are learnable linear projection parameters. To generate the auxiliary variable for the i-th modality, we define the scoring function:
2 ky Vd
1. - k2ll?
Sij
and then apply a Softmax to obtain the attention weights
exp(sis)
Oäj
> m= exp exp(sim)
Subsequently, we construct
2 = Σ Oij (1z))
by taking the weighted sum. In this formulation, F. can be viewed as the "common dimension" across all modalities that are strongly correlated with the label, whereas F, corresponds to each modality's 's internal style or noise dimension. In order for u to primarily collect F. while excluding Fs, we rely on the objective-driven gradient updates during training. If we let 0 (Wq. WK, Wv) denote all projection matrices and possibly other network parameters, then each iteration 's update rule can typically be abstracted as
Oc
0syj
0
Σ (1.1)
0s:
00
where 17 is the learning rate, and L is the model's overall error or loss function used measure how well the network predicts the label y on the current batch (or satisfies other constraints, such as preferences for causal representations). By applying the chain rule, we can further expand
Oc
OC doij da:z
OC daip
+
Osi
Osij
Daip
Os.j
p
Bay and dase reflect the competitive relationships of attention distribution across different modalities, and ac. where U8es
Bae; corresponds to how the specific attention weight impacts the task other constraints, whether positively negatively, Next, consider dayg Since
dag
00
k2 Vd
k2ll² = (X,Wo) a (((,WW)))) -(X, Vd
a
its gradient with respect to 9 can be further decomposed into
Us aw Q
Jsy OWk
0sij
dwv
Note that 2y = XyWv appears only in the weighted sum, not in the scoring function, aw the final output v, fy still influences 8c des," performs this type of gradient backpropagation, if certain (i,j) features are only close in terms of F, and do not help real label prediction, then HC. tends to be negative very small; conversely, if (i,j) align well in terms of F. and bring better Bay
88, O. However, in generating
so 8c will likewise propagate back to Wy. When the network repeatedly De,
ac predictions, dey will be overall positive or large, encouraging the score to increase. Consequently, after many iterations of
gradient descent, the scores Sij corresponding to F. are continuously reinforced, shrinking || - k; further and boosting
17

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
the dot product 2 ky resemblance will gradually get suppressed by the long-term gradient updates, lowering di and reducing their contribution to 7.
vd so that auj is raised to a higher level. Meanwhile, pairs that initially had high scores only due to F.
By a similar logic, in meeting the exclusion requirement, if some (i,j) combination can potentially bypass Fg and directly influence label Y. but fails to get consistent positive reinforcement in the loss function (or even triggers negative regularization), then the result of backpropagation will be to suppress such shortcut-related attention weights, preventing them from accumulating through repeated updates. If F. alone suffices to determine Y, then additional influences on the label from such detours not favored by the loss, so the parameters shift against this overstepping, ultimately making the learned attention distribution pay little heed to noncausal pathways.
In this manner, once training converges, the instrumental variable V comes to reflect common features aligned with Fe. which explains why observing V updates our inference about F., and ensures P(Z. V) * P(Z). Meanwhile, couplings with F,, which represent mere style or noise, are gradually eliminated from the scoring, leaving z nearly independent of F Lastly, given Z, V does not convey additional pathways to Y, since detouring around F. has no sustainable benefit in the gradient. From the perspective of the gradient update formula, each positive or negative adjustment applies directly sij and aij. ultimately shaping into representation that strongly consolidates the causal features while masking non-causal interferences and refiaining from interfering directly with the label.
Hence, by the end of training, we naturally obtain an auxiliary variable V that satisfies the following properties: it is positively correlated with F., largely insensitive to Fs, and does not offer extra intervention for label Y. The formulaic derivation and gradient-based explanation clearly illustrate why simply introducing the above self-attention scoring and relying on gradient-driver optimization can distill a robust representation of Z with minimal interference in a multimodal setting, i.e., achieving (i) P(Z V) * P(V); (ii) P(Fs| V) P(Fx); and (iii) P(Y (2,V)= P(Y Fe).
A.4. Proofs of Theorem 3.5
Proof. The main proofs can be divided into three steps: (i) from Markov property to the validity of local intervention; (ii) reliability of the twin network: why gradient-based intervention + KL constraint can approximate causal intervention; and (iii) justification of ca risk and corresponding error analysis.
Before establishing the main proof, we begin by illustrating the background of the problem and the statement of the Theorem. Specifically, we operate under multimodal distribution Pxy. with an observable sample set {(r,, 34) ) (244, y/) ~ Pxy, where x; here denotes a concatenationfjoin form of features from multiple modalities, and y is the label. We then consider a learning model fe = W o g(.), where g(-) is a feature extraction network and W a linear classifier. In order to approximate the causal representation Ze during training, we introduce a calibration or regularization loss in g. In particular, for each sample vi, we draw random variable v; V, which can be regarded an auxiliary or instrumental distribution, and define Ze,- g" (z,, vz), where g" denotes the modified feature extractor incorporating the calibration/regullaiittoon that approximates the causal representation.
Next, for gradient-based intervention with KL constraint, in this causal representation space Zo, we introduce a small perturbation A,, setting
A = VZn, ((o(WTTe.)) v1).
thereby defining the second branch of the twin network:
Ze,i = 2n, + A,
To ensure that this local intervention does not diverge too far from Zo.is we impose the Kullback-Leible (KL) divergence constraint:
MKl. (Za,i, Zns)) S E,
which means we only allow gradient interventions within small neighborhood, ensuring that Ze.; remains probabilistically similar Ze,i. Then, we define the empirical quantity
1 N N
RCa =
Σ ((o(WTZe..) + vi) + p(o(WT = =.)]
18

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
where p(.) is an indicator function, taking value 1 when the condition is true and 0 otherwise, and (-)) -)) is the signum function. In concise terms, R simultaneously counts the classification errors in the real branch Ze,; and the instances where the intervention branch Ze,i incorrectly retains the original label yi instead of flipping it.
Theorem 3.5 states: If Ja converges to an optimal solution under the Monte Carlo approximation {up V), and for each sample the perturbation A; satisfies the above KL constraint, then with high probability in the large-sample limit, No converges to the true causal complete quantity.
Part Formalizing the Markov property (Frydenberg, 1990; Blumenthal, 1957), we assume in the multimodal context that the causal representation Ze satisfies a Markov property: given Ze, the label Y is independent of other interference variables Far symbolically (Y F.) Z- or in causal graph terms, back-door paths (Pearl, 2009) are blocked by Z-c- When we perform an intervention on Ze (be it global or local), it does not compromise the remaining independence conditions involving F.. If, for sample vi, we feed it into g to obtain Zz, as the real-world branch, and then apply a small gradient perturbation to get Ze,; as the hypothetical-worlk branch, the Markov assumption ensures that modifying Ze alone will not inadvertently bring Fs, into play or disrupt other independent factors. Consequently, the concept of "twin" is feasible at the formulaic level. The difference between Ze,i and Ze,i precisely gauges how the label Y, if it truly depends only on Ze. should respond to local change.
Part II
Let the loss function
(((W Zo,e), v1)
be denoted by
L,(2,,)) = ((o(W Zoss); vi)
Then
A: - -- Ze,, Le (Zo,)) is small step in the negative gradient direction on the Zu,i space. We define
Ze,i = Zeei + A..
To keep this perturbation local, we require
MKL (Zo,i, Ze,s) S E.
thereby bounding the scale of |||,|| so that Zo.; remains in the KL-neighborhood of Z.,..
Viewing Z- directly influencing Y, changing Ze Ze, in causal model is akin to imposing do(Z. Ze,s). In practice, performing a complete change in Ze for multimodal environment can be too difficult, or insufficient data may exist to estimate it (Liang et al., 2022b). By enacting local gradient A; while ensuring 7z,; and Zi.; remain within finite KI. distance, we can approximate mild causal intervention and test whether the model is sensitive to label changes that should happen. If the model truly captures the correct causal relationship, then it should detect and reflect these label flips. The reliability and completeness of the twin network proved and discussed in Appendix D.4.
Part III Recall the proposed 8 risk, we have:
RC
Σ ((oo(WZe.)) # v.) + Ze. = =v)] N
1
For brevity, set
h; = P + vi) + p Zo,)) = -w)),
Then, we get:
1 N
ho0
h4-
N
19

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
In theory, if we had fully correct causal model, could define ideal quantity
RC ((xx)))Pxx p((...) }]
where the p(....) events reflect correct label under the real branch? and incorrect label under the intervention branch? etc. Because the Markov property allows local interventions on Fe alone and the Monte Carlo set {u,} helps calibrate g", we expect that, as N- >> 20,
Rea
R 0 (with high probability).
Based on this, can decomposing the enror, i.c., define:
8, = h;
h ideal
where Aideal represents the indicator's value under the truly causal complete scenario for sample i. Then
RC RR
6
To show that this difference converges to zero as N 00, one needs argue that under the KL constraint and the Markov assumption, can be made very small on average, and its variance is controlled. Applying Hoeffding's 's inequality Markov's inequality on the i..dd samples (ra, Vı. 0.) then ensures convergence with high probability.
When || is small and MKl. (Za, Zu,,) S e, the local intervention around Ze,e will not drastically alter the sample's label distribution. In particular, assuming fe is well-optimized, two things can happen:
If (T Ze,)) was correct, then small gradient shift aW Ze.i) likely stops predicting the same label (assuming it is designed to flip it).
Conversely, if Ze,t was incorrect, then the adjusted Ze,s can likely correct the label.
Since this agrees with how a genuinely causal intervention would flip or not flip the label, 8, can be suppressed a very small expected value (Arslan et al., 2015; Piotroski & So, 2012). With i.i.d. sampling and concentration inequalities, we get:
N
6
+ 0 (in probability),
implying SCa + RC Hence, it shows that in multimodal setting under the Markov assumption, performing controlled gradient-based intervention on Zu, to obtain Ze, based on Hoeffding inequality, and defining Ro from these two branches (real vs. intervention), yields estimate converging the genuine causal complete quantity in the large-sample limit. Thus, the proof of Theorem 3.5 is established:
RCC
RCa8
as N + oo, with high probability.
A.5. Proofs of Theorem 4.1
Proof. We will need the following definition for this proof:
(0,n y) min (h(2, v) (z,8²) 0l
(11)
of
where 0 > is an arbitrary constant. Observe that E [1phtow]s)]] I [ee since the inequality Po. (v. y) S (a(z, y) holds for all (x, y) E x y:
(0, v) min (M(x, - h (x. + 01 =yy
(12)
S min (h(,, y) h (x, + v=y=
(13)
y'
min (h(x, y) (x,v')) = (n(x,
(14)
y'Fy
20

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
where the inequality follows from taking the minimum over smaller set. Now, let H ((x,)) + - h H) and H { 0 a h: h } With probability at least for all h E H.
1 do (po, (.., + 29m (H) +
log
E p (Po, w)M =
V
(15)
2m
Since Lugo S (u) for all R, the generalization error is a lower bound on the left-hand side, and we can write:
1 m S
log 1 2m
Rº (fo)
m Σ
p (Pa.6 (xr, + 2 (H) + V
(16)
Fixing 0 = 2p, we observe that Dp (Pe. (z) = 2 (pn (I 84)). Indeed. either aa, (I Bi) = Ph (T v.) or
Pa.h (Ni, Wi) 2p ph (z), wi), which implies the desired result. Furthermore, Rm (H) Rm (H) since D, is L-Lisschitz function. Therefore, for any ô > O, with probability at least - 6, for all h E E::
RCdh (fo)
( S R (fe) + 2 R (PL + V
log b 2m
(17)
and to complete the proof it suffices to show that Rm (Ht < 2 (H))
Here (H) can be upper-bounded as follows:
Rm (H)
1
E
sup BEW
oe
lo (wi,
max (h (Ii. 2p1 vn))
(18)
m
S,
3
1
m
sup
osh vi)
E
sup NEW
o max (h (z.,.) 2p1 y=m))
(19)
m
S
mEW
m SSa
y
i=1
Now we bound the first term above. Observe that
E
sup h
o (I Vi)
E
sup
oih (Ii v) Ln=
(20)
hEW
i=
vÊl
S
E
sup hEH
oz (xz v) 11
(21)
m
yet
c where & 2- 1y - 1. Since & (-1, we have that σ; and o,e; admit the same distribution and each of the terms of the right-hand side can be bounded as follows:
E
sup
Σ
og (f,,1)
2
(22)
PE7
yE4
1
E
sup
o (2i,8)
+
(23)
KEH
2
1=1
E
sup
gresh (Ii,
+
E
sup heH
o (zi, v)
(24)
2m
hEW
2m
1=1
i=1
< Rm
(H))
(25)
Thus, can write LES (suphen Em oc (Ite, w:)]
km (HL). To bound the second term, we derive
1 E
SuPkER
max, (/ (2r,0) - 2ply=m.) <
E
sup bEH
a (h (zi, y) 2pp
(26)
m 8.a
S.a
DEl
21

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
and since Rademacher variables mean zero, we observe that
m
m o, (xxr,
m
sup
as (h (at, v) 2p1 v=v
E
sup NEH
2p >
of y=
(27)
S.a
KER
S.a
m
sup KEH
ozh (ri, y)
S Rm (IT
(28)
S.a
which completes the proof.
B. Pseudo-Code
The pseudo-code of incorporating C into the MML models is shown in Algorithm 1.
Algorithm 1 Pseudo-Code of MML with C
Input: MML data distribution Pxy; Randomly initialize MML model So with feature extractor y and a linear classifier W; Hyperparameters Ag and Afe
Output: MML model fo
1: Sample training datasets Drr = [(r)), 0f)را=1 with N samples from Pxy
Data Sampling
2: for each iteration do
3: Calculate the molti-modal representation via Z, = =x)
4: Obtain causal Žz, = g (Z, 02) from Zj based on the V via Eq.6 >. Extract Causal Representation for Sufficiency
Model Counterfactual Representation for Necessity 6: Update the MML model fo using the original MML loss with C in Eq.
5: Obtain counterfactual Ze, Ze + A via Eq.7
> Update MML Model
7: end for
8: return solution
C. Table of Notations
We list the definitions of all notations from the main text in Table 4.
D. Discussion
D.1. More Details of C Definition
Recalling Definition 3.1, the learned representation of existing MML models can be divided into:
Sufficient but Unnecessary Causes: P(Ydo(Z=e) Z E, Y + u) > 0 and Piz = č Y = v) > 0.
Necessary but Insufficient Causes: P(Ydo!Z=mo) v) = 0 and P(Y do
7 = Y v) 0. Sufficient and Necessary Causes: P(Ydo(Z==) y Z E Y ÿ) and P(Y do(7=r)
Insufficient and Unnecessary Causes: P(Ydotz=e) Z 2, Y + y) and P(Ydo(Z=z) v)
For the four parts of Z, provide the detailed illustration:
Sufficient but unnecessary: Z results in effeet Y, yet the presence of Y does not definitively imply that Z is the cause;
Necessary but insufficient: the occurrence effect Y confirms that the cause is Z, but Z alone is not guaranteed to produce Y:
Sufficient and necessary: the presence effect invariably indicates cause Z, and the presence of Z invariably results in Y.
Insufficient and unnecessary: Even Z exists, it does not guarantee that will occur. Similarly, even occurs, cannot be determined that Z, is the cause of Y because there are other possible causes.
22

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Table 4:
The definitions of notations.
Notations
Detinition
Notations of Data
X.V, and Z
The input space, label space, and latent space The sample z: X and the label E y/ The joint distribution over X' x y The training dataset with N samples
(2,0) E N' x y PxY Der (1I:
DIe (1z Bi) [-
The dataset with Nr, samples Fach sample, which contains K modalíties
21 10: ,..., }
Notations of Model
Wegg 9: - 2
The MML model
The feature extractor of MML model fa The classifier of MML model fo
W: -
Notations of Variables
Y.Y Z
Variables of the sample and corresponding label
Variable the learned representation. i.c., E.F. + E,F.
Fe, F3 E. E
Variables of the causal and non-causal generating factors
Weight matrices of F. and F, that make up % The instrumental variable satisfies V F, with a controllable impact Fe Y
V
Notations of C and C Learning Objective
RC
C risk
Lo Dje X Aze Ze,t Ze A;
The loss based on the instrumental variable V, designed to guide the model toward the causal representation
The feature extraction loss term based on counterfactual constructions, ensuring remain within the feasible region The weights for C, and Ly-
Real extraction version of for sample z, where the condition is e (real-world branch) Counterfactual version of Z for sample where the condition is e (hypothetical-won wooldd branch)
Gradient- based adjustment for counterfactual data
(-L())
Kullback- Leibler divergence
Signum function
p(-)
Indicator function which is equal to if the condition in pf-) is true.
8i
The alignment scores between the i-th and j-th modalities Linear projection matrices for query, key. and value in the modeling of V
Wo Wx. Wy
Query vector for the i-th modality Key vector for the j-th modality
Weighting factor for distance-based adjustment l1g. ky 12 in modeling of V
For the derivation of Eq.3 and Eq.4 (Z) can be decomposed into sufficiency and necessity components. Specifically, following (Pearl, 2009): (i) when Z & occurs with probability 1 - P(Z c), its contribution reflects the sufficiency effect
C3 (2) required for Z = c; (ii) when Z = c occurs with probability P(Z = c). its contribution reflects the necessity effect
ne. (Z) of Z = ē on Y. Therefore, we have (Z) = (1 P(Z = c)) C (Z) + P(Z = c) C (Z). By normalizing C (Z) according to their respective weights, we obtain:
(2) C CZ,(Z) 1 1- 1-P(Z= c)
P(Yáo(Z=o)) - P(Ydo(Z=o)
P(Y P(Yoo(Z=)) P() (o))))))))
(29)
1 -P(Z=
1 1-P(Z=0
(²( P(Yáo(Z=o)) P(V ((Vd((Z=)) p(z=c)
= P(Ydo(Z=e)) - -P(Ydoizzo)
ch = P(Z c))
(30)
P(Z =
D.2. Example to Understand C Score
We have provided an example of causal sufficiency and necessity in MML tasks in Section 2. Inspired by (Yang et al., 2024), we also provide specific numerical calculations to further explain this example. This task aims to identify the label of "duck" using three modalities, i.c., image, text, and audio. Using this example, we further use probability to explain causal sufficiency and necessity. The illustration of this example shown in Figure 1.
23

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Example of Causal Sufficiency If the learned representation is represented by the variable Z (taking binary values 0), we use it predict the label of whether being "duck". If the learned representation contains the information of "duck paws". it is a sufficient but unnecessary cause because the MML data containing "duck paws" must have a "duck", However, might not contain "duck paws", e.g., duck swim in the lake. Assuming that P(Y do(Z==1) 1) = 1) = 1 and P(Y do = 0) = 0.5, P(Y = 1) = 0.75, P(Z. = 1, Y = 1) = 0.5, P(Z. = 0, Y = 0) = 0.25, P(Z. = 0, Y = 1) = 0.25. Then, following (Pearl, 2009), for the probability of sufficiency and necessity, we obtain:
sample
P(Yáo(z=1) = 1|Y - 0, 2 0) = P0
1-0.75 P(Y=1/==1) =
P(Y=0,Z=0)
(31)
P(Y = Y = 1, 1) P(Y=1) P(Ya))000))))
P(Y=1Z==1 0.5
0.73-0.5
P(Y=1Z=11)
where the first line represents the probability of sufficiency and the second line represents the probability of necessity. Thus, the learned representation contains "laugh (positive)-good (positive)-rising tone (positive)" has probability of being the sufficient but unnecessary cause.
Example of Causal Necessity If the learned representation contains the information of "wings" Z. (taking values and O, where means "wings"), predict Y ("duck" 1, other label 0). Since "duck" must have "wings" but if give sample with "bird", it may also have "wings". Assuming that P(Y do(Z=1) 1) = 0.5 and P(Y do(z=0) 0) = 1, P(Y = 1) = 0.25, P(Z 1, Y = 1)- = 0.25, P(Z 0, Y = 0) = 0.5, P(Z = 0, Y 1) 0.25. Then, for the probability of sufficiency and necessity, we obtain:
P 10 1|Y = 0,Z = 0) ((Ya) 0.5
P(Y=UZ=0)
(32)
P(Y = 0 = 1,2 1) P(Y=I)- P(Y=-)-P(
P(Y=1Z=1)
where the first line represents the probability of sufficiency and the second line represents the probability of necessity. Thus, the learned representation has a probability of being the necessary but insufficient cause.
If we only focus causal sufficient causes, we will lose important modality-specific information and affect generalizability; if we only focus on causal necessary causes, then decisions will be made incorrectly based on background knowledge, affecting discriminability. For sufficient and necessary causes, the probability of sufficiency or necessity should be 1. Together they ensure that the MML model can learn a representation that not only reflects different modal information, i.e., contains the semantics of all three modalities, but also targets primary events, i.e., the "duck" label.
D.3. Discussion of Identifiability
For detailed analyses, in this subsection, we provide the definitions of exogeneity and monotonicity in MML settings, and illustrate the importance of the identifiability proposed in Subsection 3.2.
Firstly, we propose the definitions of exogeneity and monotonicity for C3 based on (Pearl, 2009) as follows:
Definition D.1 (Exogeneity). Variable Z is exogenous relative to variable Y if and only if the Y would potentially respond to conditions e or č is independent of the actual value of Z. The intervention probability is identified by conditional probability P(Ydo(Z=o) = y) = P(Y = v/Z = c).
Exogeneity (Definition D.1) means that when Z is exogenous to Y, external interventions (e.g., environmental disturbances or other unobserved factors) have a negligible impact on the conditional distribution of Z and Y. In other words, Z. as a causal variable is not easily influenced by external environments or noise, ensuring that its causal effect on Y remains stable across different modalities (image, text, audio, etc.). This assumption ensures the learned representation Z only contains causal generating factors Fe without spurious correlations caused by F,. It requires the model to retain relatively reliable predictive inference capability even under cross-modal cross-domain distribution shifts.
Definition D.2 (Monotonicity). Variable Y is monotonic relative to Z if and only if Ydo(Zmo) = 7 A do = y is false or Zdo(Z=a) = y A Ydo(Z=e) = G is false, where ō # y. For probabilistic formulations, P(Ydo(Z=e) = y, Ydo(z=a) # y) = 0 or (()oo(z=)) # y, Yáo(Z=o) = v) = 0.
24

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Monotonicity (Definition D.2) illustrates the consistent, unidirectional effect on Y of representation Z. If Z increases (or decreases) along some dimension, Y correspondingly and consistently increases (or decreases). Such monotonic constraint helps the model capture Z's causal effect on Y more effectively, so that even if different modalities depict Z slightly differently, the overall causal direction remains consistent, improving the learned representation's 's generalization across modalities and scenarios. The assumption ensures that changes in all data variables have the same trend effect on the results. It simplifies the causal structure and is a common assumption for complex scenarios such as MML.
However, in practical MML settings, non-trivial cases often arise: for instance, the inseparability of causal and non- causal semanties leads spurious correlations (as depicted in Figure 2), and cross-modal semantic conflicts combined with digesiiimnsional nonlinear interactions undermine monotonicity. These challenges render traditional identifiability conditions inapplicable. More specifically, enforcing exogeneity and monotonicity during model training can introduce multiple issues. First, it may oversimplify or deviate from real-world distributions, leading to large errors. As Figure 2 shows, real-world Fg often cannot be perfectly resistant to external interventions, and the spurious correlations between F, and Y will break exogeneity. In such cases, forcing exogeneity can lead to the model learning spurious correlations, undermining its applicability and accuracy. Likewise, if monotonicity does not hold in reality (e.g., certain modalities exhibit nonlinear or time-varying relationships with Z), the model may be biased or misjudge the direction of Z's effect on Y. Moreover, some systems have feedback loops in which changes in Y can affect F. or other variables (e.g., user behavior and recommendation outcomes on social media), and imposing strictly one-way causal or monotonic relationships can ignore these dynamic interactions and distort the model. Additionally, from performance perspective, if exogeneity and monotonicity assumptions are rigidly applied during training, the model may fail to generalize once deployed to new environments or tasks that do not meet these assumptions. Such "hard" constraints may lead the model to over- or underestimate the actual contribution of F. to Y, resulting in decision risks. Therefore, the discussion on identifiability in Subsection 3.2 is crucial to ensure accuracy and generalizability of MML models in real-world applications.
D.4. Reliability and Completeness of the Twin Network
Considering that the intervention value ē does not necessarily come from the same distribution as (Pearl, 2009), we define the intervention variable Z has the same range as Z, where ē comes from its distribution P(Z|X = z)). Correspondingly, the estimated distribution is defined as P(Z|X- = z) and P(Z|X z)). Generally, the C risk is formally defined as:
RC == P(n.gi-Pxy Pex Maa c) # v) b Be 2) = v]],
(33)
Sufficiency
Necessity
where Wi is the above invariant predictor, p(-) denotes an indicator function which is equal to if the condition in p()) is true, otherwise equals O. However, as described in the main text (Subsection 3.3), it is difficult to directly obtain the evaluation of necessity. since the counterfactual data ē is hard to acquire. Therefore, we employ twin network to separately model the real-world branch for sufficiency evaluation and the hypothetical-world branch for necessity evaluation. The twin network for C3 estimation in MML is shown in Figure 5. Then, we get:
N
RCa
1
Zo.) # md + plo = =vi]
(34)
where 2 = g" Uz), Zo, Z., 2.,
2o, (O(W Zoni), 91).
This equation satisfies PKL (Z.., 2c,t) Se, Vi, also illustrated in Theorem 3.5. In this subsection, we discuss and prove the reliability and completeness of the proposed twin network.
The core idea of the twin network to generate two representation branches for the same input æ, which are close but slightly different, thereby modeling the real world and the hypothetical world in parallel. In this work, we aim to use the twin network to separately model causal sufficiency and necessity, corresponding the real world and the hypothetical world: (i) the real world evaluates whether the extracted representation, i.e., Z = can yield the ground-truth label; (ii) the hypothetical world evaluates whether the extracted representation, under the intervention Z. = ē, can still yield the ground-truth label. The key challenge lies in modeling counterfactual data Z = č.
Note that the true counterfactual effect involves unobserved outcomes, making direct modeling difficult (Pearl, 2009). Existing work typically adopts the "minimal change" principle to estimate counterfactuals, which is proven to align with
25

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
true counterfactual effects (Pearl, 2009; Kusner et al., 2017): to study the causal effect of a variable factor on the outcome, one should avoid making large modifications to other irrelevant factors. Instead, only the parts most sensitive to the outcome should be adjusted, ensuring that the original instance and its counterfactual counterpart remain as similar as possible, apart from the change in the target variable. For instance, Chapters III-V in (Wachter et al., 2017) demonstrate that by making only minor adjustments to the treatment variable while preserving the distribution of other covariates, the counterfactual samples maintain the original data's characteristics, ensuring accurate counterfactual effect estimates. Based on these results, we leverage gradient intervention to satisfy the "minimal change" principle for counterfactual effect estimation with theoretical guarantees. Inspired by this, we propose to first use instrumental variables extract the causal representation Zee from input æ: in the real-worle branch. Simultaneously, in the hypothetical world, we compute Ze = Z + A along the gradient direction A, constructing a relative "small-step" change strictly in the dimensions relevant to the causal mechanism. This approach allows Zec to serve as approximate counterfactual representation for the same input. The core idea lies in: (i) constraining the gradient direction to ensure that counterfactual modeling does not deviate from the true implementation Z in the latent space (Berger & Gostiaux, 2012); (ii) applying minimal changes to satisfy the "minimal change" " principle of counterfactual modeling. Under the Markov assumption (Frydenberg, 1990; Blumenthal, 1957), once Zn is given, any disturbance variables will be independent of Y. Therefore, if Z, needs flip, it is sufficient to make small-step modifications to Ze until the label changes, aligning with the essence of counterfactual reasoning: I slightly modify Ze. will Y remain the same or change? As long as these small perturbations are enough to alter the model's output, they effectively simulate the counterfactual condition of the prediction.
To assess whether the generated counterfactual data adhere to the principle for accurate estimation, we develop a distribution consistency test with Wasserstein distance Dw. i.c., whether the distribution of the covariates matches that of the original data. The lower Dw shown below proves that our method satisfies the principle. Besides, we also conduct a toy experiment on LCKD and NYU Depth V2 to demonstrate credibility. We follow (Galles & Pearl, 1998) make manually curated examples and select the recently proposed transport-based baaed counterfactual modeling method (De Lara et al., 2024) as another baseline. Table 5 shows the advantages of our method, i.c., superior accuracy and lowest computational cost.
Table 5: Model performance and calculation overhead
Model
Accuracy
Calculation Overhead
C
77.6
1x
manually curated example
71.2
4.1x
Transport-based
77.2
2.3x
We further provide an illustration of the twin network in Figure 5. The left (orange) region contains nodes X, Fe, F,,, Y to represent the causal structure of the real world, while the right (purple) region contains X", Fr, Y to indicate a hypothetical (counterfactual) scenario. Both branches connect to the overhead nodes D, V. where D denotes the variable for data generation and V denotes the instrumental variable, meaning that they share certain background and content variable but can diverge in some causal nodes. Following (Pearl, 2009), in causal inference or counterfactual analysis, if we aim to compare how the world actually is (the real scenario) with how the world would be under certain interventions (the sypothetical/counterfacuuu scenario), we naturally create two similar branches in a diagram, sharing part of the background or parent nodes (such as D, V) but intervening in certain causal nodes (F:: or X*). The twin network shows two conditions of the same model.
Based on the above analyses, we propose the following propositions to expand the ability to identify causal effects:
Proposition D.3 (Local Invertibility). Let M be a twin network that operates on exogenous/background variables s. Suppose that in every local neighborhood of M, the mapping P(Y Pa(Y)) + P(s) is injective, i.e. from the conditional distribution of Y given its parents Pa(Y', one can uniquely solve for the distribution P(s). Then, any counterfactual statement in M is uniquely identifiable from the observational data.
Proposition D.3 tells us that, in each neighborhood, specifying Pa(Y) = enforces a unique distribution of s. So once we move Pa(Y') from the real- world value to an intervened value s' (the do(x') operation), that new local condition forces a unique s. If, hypothetically. local invertibility did not hold, multiple distinct distributions over s could give the same conditional P(Y Pa(YY= = =), leading to multiple solutions for the counterfactual. But since local invertibility excludes
26

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
D
real world
V
hypothetical world
observable variable
unobservable variable
true correlations
Y
Y
spurious correlations
Figure 5: Twin Network for Ca estimation in MML.
that possibility, the solution for each neighborhood is unique-hence identifiability. Thus, under local invertibility, any local or global counterfactual query in the twin network can be computed from observational data.
Proof. A general counterfactual statement or in causal inference (for example, If X had been x', then Y would have been y often demands integrating over all possible exogenous-variable configurations s. Formally, evaluating
P(a)= =
/ P(a] s) P(s) ds
in principle calls for specifying how is distributed and how it changes under different local conditions.
If in neighborhood N,, we know the conditional distribution P(Y Pa(YY v) for some value v, the local invertibility assumption states that precisely one distribution over s leads to that conditional. Symbolically,
P(Y Pa(Y) = o) + PP(s) Yuiqque-
Hence, specifying Pa(Y) fixes how s must be distributed.
In computing counterfactual probability, e.g. P(Y, = y'), one typically modifies certain parent values Pa(Y) - o' and then integrates out Because local invertibility provides single assignment to in the local region, we obtain a unique predicted outcome distribution. There is no ambiguity about which exogenous states correspond to that local condition.
Since any condition on Pa(Y) in the local domain yields a unique exogenous distribution, any if-then statement about local changes Pa (Y) can be pinned down. As long as the twin network M remains locally invertible all relevant neighborhoods, we can piece together all such local analyses conclude that any finite or infinite combination of counterfactual statements is identifiable.
Proposition D.4 (Markov Identifiability). If the twin network M is semi-Markovian, and furthermore satisfies graphical criteria (Pearl, 2009), then the quantities C3, O and Ca are all identifiable within such causal model, and the causal effect can be uniquely determined from the topology of the graph.
Proposition D.4 shows that whenever the graph meets the semi-Markovian plus the cited conditions, the causal effect and the individual-leve necessity/suffiiinney measures are identified from the topology alone. Accoding (Norton, 2009), if the confounders appear in specific ways (with certain back-door or front-door paths, or no cyclical bridging in the unobserved structure), we can factor the joint distribution into purely observational data pieces partial adjustments. This also extends to multi wooli queries needed by C3, C3 and C3 Meanwhile, if there were multiple expressions consistent with the same graph, we would lose identifiability (Tangirala, 2018). But the theorems ensure a single final expression for each of C C su and C3 establishing that these cross-world probabilities cannot be ambiguous.
D.5. Multi-modal Representation Learning on Synthetic Data
To assess the effectiveness of the proposed CRR in capturing the critical causal information that serves as both sufficient and necessary causes, we have constructed synthetic data that encompasses four distinct categories variables following (Yang et al., 2024), ie., sufficient and necessary causes (SNC), sufficient but unnecessary causes (SC), necessary but insufficient causes (NC), and spurious correlations (SP). Among them, the first three are the information contained in the invariant
27

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
representation currently learned in standard cases, while the last one is artificially constructed and used for evaluation in practice.
Specifically, we first construct different multi-modal data distributions, then generate labels from the known data distributions, and finally constrain their correlations to achieve the construction of different causes. We assume the task contains three modalities and construct the following generating functions. Note that the first three correspond to the three parts of the invariant representation learned by the current MML methods, i.e., sufficient but unnecessary, necessary but insufficient, and sufficient and necessary causes, while the last, i.e. spurious correlations, is for evaluation.
Sufficient and Necessary Cause (SNC) Each modality of the sufficient and necessary cause is generated according to a Bernoulli distribution following (Yang et al., 2024), i.e., SNC; ~ (((a), ) a = 1, 3, where & represents the Bernoulli distribution parameters corresponding to different modalities, e.g., SNC; ~ B(0.5), & = 0.5. The data label yi is generated based on the sufficient and necessary cause, where y/ SNC; ((E)). e.g.. SNC, ((0.15) when &a = 0.5. Since Y is generated from each modality corresponding SNC {0, 13. The probability of the sufficient and necessary cause is P(Y = OJdo 0)) P(Y = 1]do(SNC = 1)).
Sufficient but Unnecessary Cause (SC) Sufficiency indicates that the presence of a representation aids in establishing the accuracy of the label, i.e., SC - Y. According to the definition of C3,, when SF is defined as fsc(SNC). the distribution of intervention P(Y]do(SC = SC,)) is determined by the conditional distribution P(Y]SC- = SC₃), where P(Y]do(SC: = SC,)d SNC = sf PCYISNC)PPSSNC(ss(SSN) = SC,)d SNC. Even Y
SC,)) f P(Y (SNCCcs is only generated from SNC, the sufficient cause SC is exogenous relative to Y. Meanwhile, according to the identifiability results of Ca the sufficient but unnecessary cause SC E {0.1} 1 in synthetic data has the same probability with SNC, expressed as P(Y = 1|Co(SC = 1)) = P(Y = 1 [SNC = 1))- However, it has a lower probability of P(Y = O]do(SC- = 0)) compared to P(Y = Oldoo = 0)). To determine the value of SC,, we use a transformation function fsc : {0, 1} - {0, 1 to derive SC; from the sufficient and necessary cause value SNC, for each modality. The generation process of SC; can be expressed as: (i) SC,- ((G)) when SNC, 0: and (ii) SC; = SNC, when SNC, 1.
Necessary but Insufficient Cause (NC) Necessity indicates that the label becomes invalid when the representations are absent, i.e., NC 4 Y. which reflects the general situation of different modal information, i.e., contains the semantics of all modalities. Similarly. we get: PNN = NC,)) f P(Y ((Y]d((SNC) (( (SSNC) = NC,) SNC NC,) ) SNC. Based on the definition and identifiability outcomes of C the cause that
I P is insufficient but necessary exhibits an equivalent probability to P(Y Ojdo(NC 0)) as P(Y = OJdo(SNC 0)), yet it diminishes the likelihood of P(Y = 1 doCC =)) compared to P(Y = [|NoCSNC = 1)). To determine the value of NC, we employ a transformation function fNC : {0. 1} + {0, 1} to derive NC, from both sufficient and necessary cause SNC,. The process of generating NC, is outlined below, and NC, serves as the cause of y: NC, = fSc(SNC.)) == SNC, * B(E)), = 1, 2, 3 where & equals 0.5, 0.7, and 0.9, respectively.
Spurious Correlations (SP) Spurious correlations indicate data in the learned representation that is irrelevant to the decision, e.g., background information and noise. Although the learned invariant representations do not contain noise information and false correlations are not separately considered in the scenarios involved in this study because the model satisfies the corresponding identifiability through the four constraints corresponding to exogeneity and monotonicity, we still construct relevant data to examine whether false correlations are learned in reality. We introduce an additional variable that exhibits spurious correlation with both the sufficient and necessary causes. The level of spurious correlation is determined by a parameter denoted as w. The generation process for this variable is defined as SP; = w SNC, 1. + (1 w) 1), where ((0, 1) represents a Gaussian distribution, 1a represents a vector of ones of dimension d, and d is set to 5 in the synthetic generation process. As w increases, the strength of the spurious correlation within the data sample intensifies. For our synthetic experiments, we choose x = 0.1 and = 0.7 to explore different levels of spurious correlation. Note that the SP data here is artificially constructed and has been used for evaluation, and it involves the corner case, e.g., anti-causal and confounder, which has been discussed for evaluation.
Construction of MMLSynData for Evaluation After obtaining the above functions, we now introduce how to generate the synthetic dataset for evaluation (the experiments of "Learning causal complete representations" in Subsection 6.2) following (Yang et al., 2024). Firstly, we introduce a nonlinear transformation to generate the multi-modal samples x from the variables (SNC,, SC,, NC,, S where each sample consists three modalities (functions with different parameters).
28

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Initially, we create a temporary vector with Gaussian noise, i.e., v [SNC; * 17 SC, + ld. NC, Ia,SP; * la] + +N(0,, 0.4), where La represents a vector of ones of dimension d, and (0, 0.4) denotes the Gaussian noise with mean O and variance 0.4. Next, following (Dandekar et al., 2018), we use two functions, i.e., vi (v) subtracts 0.5 from each element of v if is greater than 0, otherwise, it sets the output to O; and (2(e) adds 0.5 to each element of if v is less than 0, otherwise, it sets the output to O. Finally, the vector v is generated using the sigmoid function applied element-wise to the product of vn (v) and (v)),, i.e., v = sig (v) - (2))... For the training phase, we generate 1,000 multi-modal samples for training, while generating 200 samples for evaluation. We call this MML synthetic dataset (MMLSSyData)).
D.6. Comparison and Uniqueness Discussion
"Causal sufficiency and necessity" is an important concept in causal theory, which is proposed in Book "Causality" (Pearl, 2009). Recently, (Yang et al., 2024) applied it to domain generalization. To clearly highlight the novelty of our work and differentiate it from existing studies, we outline the key distinctions.
Uniqueness of the C Concept and Modeling We introduce the concept of Probability of Causal Complete Cause (C*), which serves foundational element in our framework for MML. For definition and extension: C quantifies the likelihood that a representation Z is both necessary and sufficient cause for the label Y. It extends the PNS concept of (Pearl, 2009) to accommodate the complexities of MML., involving distinct feature extraction and fusion mechanisms, while (Yang et al., 2024) is for domain generalization. They both based on the concept of (Pearl, 2009) but focus on different problems. Causal Identifiability: We propose identifiability of C in real-world MML systems, relaxing the strong assumptions of Exogeneity and Monotonicity that previous studies (Zhang et al., 2024; von Kügelgen, 2024; Yang et al., 2024) relies on. We consider additional conditions specific to MML, such as handling missing modalities and confounders, thereby enhancing the robustness of causal inference in multimodal settings. Meanwhile, we introduce instrumental variables to ensure the identifiability of the model even in corner cases, e.g., in the presence of spurious correlations and noise. Next, for measurement and modeling, we introduce the Ca risk as a metric to estimate the Ca score of multimodal representations on unseen test datasets. We directly model the sufficiency and necessity of the learned representations via a provable twin network, while (Yang et al., 2024) replacing the necessity with counterfactual modeling via a theoretical upper bound based on monotonicity; (Schölkopf et al., 2021) presents a review for causal inference; (Sun et al., 2025; Yao al., 2024; Ahuja al., 2023) mainly focus on identifiability under different settings and problems, e.g., weak supervision and partial observability, instead of the concept of causal sufficiency and necessity; (Wang & Jordan, 2024) aim to construct measures of non-spuriousness and disentanglement, where the exploitation of causal necessity and sufficiency concept is to align it with non-spuriousness and "invoke" the corresponding measure in (Pearl, 2009) to construct the measure of son-spuriousness. Our framework is able to adaptively capture complex patterns and potential relationships in the data without relying on pre-set assumptions. Meanwhile, it can construct a more complex model structure without being limited by the high- nigh-dimensional and nonlinear conditions.
Constraining Causal Completeness Our methodology for enforcing causal sufficiency differs significantly from that of previous theory causal-related works (Yang al., 2024; Kim et al., 2024; Sun et al., 2025; Yao et al., 2024; Ahuja et al., 2023). For the objective function, as outlined in Section 5, our C3 R objective incorporates both effectiveness and reliability. The effectiveness component addresses causal completeness of the learned representation, with the generalization guarantee, while the reliability component ensures mitigates the effects of spurious correlations and the accuracy of counterfactual modeling (for necessity) through additional constraints. is inspired from the proposed measurement. The objective of (Yang et al., 2024) relies on the proposed generalization bounds, where the foundation is different. For implementation and experimental Settings: Our approach employs distinet feature extractors for each modality, followed by MLP layer for fusion, tailored to the diverse data distributions inherent in MML. This contrasts with (Yang et al., 2024), which utilizes a shared feature extractor suitable for domain generalization across different domains. Furthermore, our experimental results demonstrate that C3 R consistently improves performance across various MML scenarios.
In summary, while both our work and previous works (Yang et al., 2024; Sun et al., 2025; Yao et al., 2024; Ahuja et al., 2023; Brehmer et al., 2022) draw inspiration from key concepts causal theory, they diverge significantly in their problem settings, motivations, theoretical foundations, optimization strategies, empirical validations, etc. Our introduction of causal sufficiency and necessity tailored for MML, the novel 3 concept, and the comprehensive C R framework collectively advance the field of MML beyond the scope of existing domain generalization approaches.
29

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
E. Benchmark Datasets
In this section, we briefly introduce all datasets used in our experiments. In summary, the benchmark datasets can be divided into four categories: (i) scenes recognition on two datasets, i..., NYU Depth V2 (Silberman et al., 2012) and SUN RGBD (Song et al., 2015) datasets, with two modalities, i.c. RGB and depth images; (ii) image-text classification on two datasets, ie., UPMC FOOD101 (Wang al., 2015) and MVSA (Niu et al., 2016) datasets, with two modalities, i.e., image and text; (iii) segmentation when consider missing modalities on the BraTS dataset (Menze et al., 2014; Bakas et al., 2018) with four modalities, i.e., Flair, TI, Tlc, and T2: and (iv) MMLSynData mentioned in Appendix D.5 which is an MML synthetic dataset that used to evaluate whether learned representations contain causal sufficiency and necessity. The composition of the data set is as follows:
NYU Depth V2 (Silberman et al., 2012) is an indoor scene dataset captured by New York University using Microsoft Kinect's RGB and depth cameras. It includes 1,449 labeled RGB and depth images across 464 distinct indoor scenes from three cities, along with 407,024 unlabeled images.
SUN RGBD (Song et al., 2015) is a scene understanding dataset released by the Vision Robotics Group at Princeton University. It comprises 10,335 RGB-D images of indoor scenes, which are pairs of color and depth images. These images were captured using four different types of 3D cameras, including the Intel RealSense, Asus Xtion, Kinect v1, and Kinect v2. Each image in the dataset has been meticulously annotated with 2D polygonal segmentation and 3D bounding boxes.
UPMC FOOD101 (Wang et al., 2015) is comprehensive food recognition dataset consisting of 101,000 images across 101 categories. Each category features 750 training images and 250 test images. Notably, the images are stored in JPEG format and are uniformly resized to maximum dimension of 512 pixels.
MVSA (Niu et al., 2016) is a multimodal biometric dataset that includes a variety of biometric samples such as fingerprints, iris, face, and hand shapes. It is designed to support research in the fields of biometric recognition and security applications. The MVSA dataset typically comprises a rich collection of samples with image and text modalities.
BraTS (Menze et al., 2014; Bakas et al., 2018) aims to segment specific areas within brain tumors, which are identified as the enhancing tumor (ET), the tumor core (TC), and the entire tumor (WT). The dataset is composed of 3D multi- modal MRI scans of the brain, featuring modalities such as Flair (FI,, T1, TI contrast-enhanced (Tlc), and T2, all of which come with corresponding ground-truth segmentations. It includes a training set of 285 cases and an evaluation set of 66 cases. While the ground-truth annotations for the training cases are accessible to the public, those for the validation set remain undisclosed.
MML.SnDaat aims to analyze whether the learned representations contain causal sufficiency and necessity. It contains four types of data, i.c., sufficient and necessary causes (SNC), sufficient but unnecessary causes (SC), necessary but insufficient causes (NC), and spurious correlations (SP). Each type of data uses 250 MML samples for training and 50 groups for evaluation. The construction details and corresponding functions are described in Appendix D.5.
F. Baselines
For comprehensive evaluation of the proposed CRR, we select 5 types of comparison baselines for evaluation, which covers almost all types of MML baselines including (i) large model and foundation model for MML,, i.e., CLIP (Sun et al., 2023), ALIGN (Jia et al., 2021), CoOp (Jia et al., 2022a), MaPLe (Khattak et al., 2023), and VPT (Jia et al., 2022a); (ii) classic MML methods, i.e., RGB (Zhang al., 2023c). Depth (Zhang al., 2023c), Late fusion (Wang et al., 2016), ConeatMML (Zhang et al., 2021), and AlignMML (Wang et al., 2016); (iii) strong unimodal baselines and the corresponding multi-modal methods, i.e., Bow (Zhang et al., 2023c), Img (Zhang et al., 2023c), BERT (Zhang et al., 2023c), ConcatBow (Zhang et al., 2023c), and ConcatBERT (Zhang et al., 2023c); (iv) recently proposed and SOTA MML. methods, i.e., MMTM (Joze et al., 2020) and TMC (Han et al., 2020), LCKD (Wang et al., 2023b), UniCODE (Xia et al., 2024). SiMMMDG (Dong et al., 2024), MMBT (Kiela et al., 2019), and QMF (Zhang et al., 2023c); and (v) MML methods that specifically designed for missing modalities, i.e., HMIS (Havaci et al., 2016), HVED (Dorent et al., 2019), RSeg (Chen et al., 2019), mmFm (Zhang al., 2022b), and LCKD (Wang et al., 2023b).
30

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Table 6: Full results (with error bars) of scenes recognition performance on NYU Depth V2 (Silberman et al., 2012) and SUN RGBD (Song et al., 2015) with RGB and depth images. "(N, Avg.)" and "(N, Worst.) denotes the average and worst-case accuracy. The results show the error of accuracy by executing the experiments randomly 3 times on 40 randomly selected hyperparameters. The best results are highlighted in bold.
NYU Dajch V2
SUN RGB-DD
Method
(0.Avg)
(0.WomL)
(10.Avg)
(l0.)ons))
(0.Avg)
(0.Worst)
(10.Avg)
(10,WomsL)
CLIP (Sun et al., 2023)
69.3210.33
68.2910.36
51.6710.42
48.5410.36
56.2411..51
54.7310.31
35.6510.17
32.7610.51
ALIGN (lia et. 2021)
06.43:0.33
64.33=0.32
45:2442:47
42:42:0.88
57.32+0.52
06.26:20.36
38. 18.43=0.42
35 15.13=0.52
MaPLe (Khattak et 2023)
71.2610.32
69.27.0.35
42.98.0:45
48.7310.37
61,7610,33
34,5110,42
30.2910.51
CoOp (Ji at 2022a)
07.48:0.34
0639400.32
49.43=00.4
45.62=0.32
59.36*90.50
56.31
39.67=0.41
45.55
VPT (Jia al. 2022a)
62.16+0.36
61.21-0:31
41 05-044447
37.81-0.32
54.72+0.54
53 32.2240.32
33.48+0444
29 29 +0.51
RGB (Zhang
2023<)
63.3310.35
62.5110.32
45.4610:47
42.2010.38
56.7610.51
56,5111,36
429110005
11.02.0..1
Depth (Zhang et al., 20236)
02.65:0.337
0..34
44.13:00.88
35.93:00322
51,99
51.3220.32
35.63+0.40
33.07=0.544
Late fasion (Wang at at. 2016)
09.14+0.35
68 0.30.
S1 41.99-0.99
44.95-0.32
62.09+0.588
60,55+0.38
47.33+0446
44 00+007
ConcatMML. (2hang al. 2021)
70.30:00.2
0.342:0.36
00:00:
47
61,90:052
61,19=0,37
45,64=0,41
00
AligoMMI. (Wang or al., 2016)
70.31+0.33
68 0.330
51.74-0.42
14+19-0.39
12+-2-
60.12+0.33
441940447
12+05.1
Bow (Zhang
2023)
61.3810.35
09.231..36
37 47.98
34.2110.88
51.37.00.50
54.1110.34
19.071..17
36.4310.53
ing (Chang al., 2023c)
43.27:0.33
0.396:0.34
29
28.53-0.77
36.26:10.54
05,26:20,36
21,32=0,400
20.31
BERT (Zhing ot al., 20233)
65. 65.31+00.36
03.23-0.32
36.5--0333
57 98+-1.52
56. 56.74+0.335
42.51+041
58 53+037
Concatoow (2hmng al., 2023c)
19.6410.33
0,366.0.34
41
29.8710.30
11,2510,54
40,54.1..32
26.7610.49
24 27.27.0.0
Con (Zhang et 2023c)
70.56.10.34
0.33300.36
44.5220.44
43.2910024
39,7611..52
58.92.0,34
45,8510,422
41.76.10.54
MMTM (Jees al., 2020)
71.0410.36
00.18
12.28
46.18.0.33
61.72.0..6
60,9110.311
46.0310.44
44 28.10
IMC IHan et 2020)
31.06:0.34
09.57.0.32
53.3620.42
49.2320,99
60.08:10.54
00.31:0.30
45.66:10.66
01.00
LCKD (Wang 20233g
68.0100.31
66.15-0.34
42.31+0.45
40
56.43+0.56
56.32+0,32
43,21+0:99
42.43-0.54
UMCCODE (Nia al., 2024)
70.12:1..3
68 0.
44.78.0.48
42
59.21.10.55
58.5510.366
46.12.10.47
42.21
SimMMDG (Dong
2024)
71.3410.32
70.29.0.31
45.67
44.3200.99
60,5110.50
60.3110.37
43.8610.43
45.7910,56
MMBT (Kiels al. 2019)
67.00+0.35
0.33
49.59=0.41
47.24-0.38
56.91+0.51
16.18+0.36
43.28+0047
46+0..1
QMF 12hang et al., 2023c)
70.0910.30
08,8120,34
55
0.2042
51.022034
62.0920.50
61.3020.36
48.58:10466
47 07,50.
CLIP+C*R
76.54=0.89 (47.22) 75.12: 20 (36.83) 56.73
(65 $2,90
(+4.30)
62
(+6.37)
58.71 (+3.98)
41 1919
37,52:0078 (+4.26)
ALIGN-OOR
71.92-0996 (45.49) 70.33+0001 (46.00) 52.59=1 (+7
51,
26. (-8 84) 62 09-09-111 (+5.67)
61
(45.69)
(47.655
41,95-1 (46.82)
MaPLe+C"R
17.07:21.16 16 (+5.81)74445:11 74.45.1 23 (+5. (8) 58,94.1 1.17 (+5 96)
55.95
(-7.23 66 21 L885 (+3.77)
65 51 (+3.75)
(+5.61)
37,34:11 00 (+7.05)
ER
73.26-1005 05 (44.12) 71.62-0077 (4-3.27) 57.21-0 85 (45.22) 50, 100.9--1.51
64 84-1 (-2.75)
63 25-25-13
53.35-0811
50.43-1
ConEIMMLL+C*R
75.97.11.51 (+5.67) 75.95L0..0 (+6.53) 60.32.1 (3 (+7 (2) 55.02.0. (-7
67.
17.17.1..0 2.20 (+5.27) 66 73. 1.30 (+5.54)
$2.28 12.28.111448
(+6.5-)
50.42.. (+7.17)
Bow+C"R
65.77:20.92 (+-1.39) (5.22:6:1.42 (+5. 99) 44.82 64 (+6.84) 1.38 (+8.64) 64)
58 15:1.41 [+4.78)
57.52=0.67
(+3.41)
47.57 (+8.30)
43,85: 142 (++1.42
LCKD&C'R
77.14-1.66 62 (+9.113 75.12=1 62 (+8.97)
1
47.98-0-99
60.97-1
54)
60 14-0 (43.82)
47.23-+11111
(14.02)
46.21-1 (43.78)
U
76.52::1 42 (+6.40) 74,30 70 (+5.6) 65) 51. 1 41 (+6.73) 89 11 1.05
65 78 .4 (+6.57) 64.49411110
1.6 (45.94)
51 1.52 (+5.10)
40.70:11 (+7.49)
SLMMMDG+CRR
1.22
(+3.99))4..6111
74.61=1.00 (+-1.22 49, 19.99 24 (+L.22)
47. 22..2211.. 50 (-2 65)
65 1001 (+1.96)
54 1...S.1.44 (+1.27)
52.69
(--183)51.70011)
(+5.91)
MMBT+<"R
73.74
(+6.74) 71.82.1122 (+5.96) (8)98) 54.35
(+1.76)
52.57
67 (+1.56
(+3.81)
(+5.1/4)
16.0016677111 (+6.61)
OMP+C*R
77.58 1.37
(+7.49) 74,95 (46.14) 99.72 60
(+4.12) 59. 18
.5
(+8.
(1)
67.
05.35
-1 1.12
64
1.26 (+4.54)
52.26.t 1.59 (+3.68)
51.28:11 (+3.78)
G. Implementation Details
For the basic MML model, we follow the commonly used structure mentioned in (Zhang et al., 2023d; Xu et al., 2023) or the corresponding official code. For the model architecture of the causal representations learner, we use a three-layer Multilayer Perceptron (MLP) neural network with activation functions designed following (Clevert et al., 2016). The dimensions of the hidden vectors of each layer are specified as 64, 32, and 128. It can be embedded after the feature extractor of any MML model, ensuring the causal completeness of the learned representations by learning a learnable matrix based on Eq. 10 that is consistent with the size of the representations. Moving on to the optimization process, we employ the Adam optimizer to train our model. Momentum and weight decay are set at 0.8 and 10-4, respectively. The initial learning rate for all experiments is established 0.1, with the flexibility for linear scaling as required. Additionally, we use grid search to set the hyperparameters >, 0.75, and xye- 0.4. Note that specially construct corresponding ablation experiments for the selection of these parameters, as described in Appendix H. Experimental results show that the model can maintain relatively stable performance various data sets using different hyperparameter settings.
For evaluation, the training dataset is randomly split as training and validation datasets, the hyperparameters are selected on the validation dataset, which maximizes the performance of the validation dataset. The overall accuracy results are evaluated on the test dataset rather than the validation dataset. All experimental procedures are executed using NVIDIA RTX A6000 GPUs, and all experimental results are obtained on the basis of five rounds of experiments.
H. Full Results and Additional Experiments
In this section, we provide the full results and analyses of the experiments in Section 6 and additional experiments which can only be supplemented in the appendix due to space limitations. Specifically, we first provide full results and additional details of "Performance and Robustness Analysis" " (the first experiment in Subsection 6.2 with Table 1), including performance on all MML baseline methods and more analysis conclusions (Appendix H.1). Then, we provide the additional details
31

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Table 7: Full results (with error bars) of image-text classification performance on UPMC FOODI01 (Wang et al., 2015) and MVSA (Niu et al., 2016) with image and text. "(N, Avg.)" and "(N, Worst.)" denotes the average and worst-case accuracy. The results show the error of accuracy by executing the experiments randomly times on 40 randomly selected hyperparameters. The best results are highlighted in bold.
FOOD 101
MVSA
Method
(0.Avg)
(0.WomL)
(10.Avg)
(10, (on.)))))
(0.Arg)
(0.Worst.
(10.Avg)
(10,Woms))
CLIP (Sun et al.3
85.2411.331
01.30.0.34
52.1210.51
49.3110.33
62.48.10.33
61.22.1.37
31.6410.58
28.2710.51
ALIGN (lia at. 2021)
06.1420.32
0.33
53.21=0,55
5085=0447
63.25
62. 62.0920.332
30,5520.56
26.44
MaPLe (Khattak et 2023)
90.4010.32
53.1610..66
40.2110.47
77.4310.32
75.36.10.30
13.72.0.56
38.52.0.52
CoOp (Ji at 2022a)
88.33=0.30
85. 0.3322
55 04:06
51.01:048
74.26+40.35
73.61=0.34
42.58:00.56
57 37.29:00558
VPT (Jia al. 2022a)
83.89+0.31
02.00-0:35
0.552
49701-0477
65.87+0.56
64 04.37
52.79+0.50
29 29.21 +0.59
RGB (Zhang 2023c)
63.5110.30
51.3210.55
45
69.2610.32
12.0.31
51,1310.55
47.2610.51
Depth (Zhang et al., 20236)
01.37
01.33
46 46:29:0..77
43.5720422
67,5240.39
06.76:2..34
43.7720.51
38. 03.58
Late fasion (Wang at at. 2016)
90.69+0.30
90.58-0.11
56.00+0.52
55,77-0447
76.88+0.35
74.76+0.338
55, 05.16+0..66
07.78+0.52
ConcatMML. (2hang al. 2021)
89:43:0.34
0.33
44
75.4240.33
75,33=0,30
53.42=051
5,
AligoMMI. (Wang or al., 2016)
88.26+0.30
88. 88.11-0.38
45
52.76-0.42
74.91+0.36
72.97+0.34
52.1++0.55
47 07.03-
Bow (Zhang
2023)
62.5010.36
02.32210:34
41.55
41,4100.8
18.7910.34
35.0510.32
41.5710.50
32, 18.0.33
ing (Chang al., 2023c)
64.6240.32
33.00
32
64.1240.37
62,04
45,00:00,66
00
BERT (Zhing ot al., 20233)
86.46+0.37
06.42-0.35
43 08
43.56=00.22
+5.61+0.300
7476+-0.35
47.41+0.55
45 96++0.11
Concatoow (2hmng al., 2023c)
70.77.0..35
0.
68
34.92-0.41
64.09 17
02,0400.35
45.1010.56
40,9510.54
Con (Zhang et 2023c)
08.20:10.36
0..81
49.36.10.54
47.79
65.5911.38
64.7410,36
46.12.0.56
41.81
MMTM (Jees al., 2020)
59.7510.33
09.4310.39
57,9L10.52
$4.96.00446
74.24110.38
73.5510.34
54,6310,50
49.72100.6
IMC IHan et 2020)
39,86:20.35
0,
61.37
61.1020433
74.88810.00
71.1010.31
60.3610.55
53.37
LCKD (Wang 20233g
85.32+0.36
04.26-0.34
43.4--
44.22-043
62,44+0.30
62,27+0.34
43.52+0.59
03.33-0.54
UMCCOD (Nia al., 2024)
88, 08.39.1..36
57.21
51 0805
47.95.0.41
66 66.97.1..39
65.9410.35
48.34:10.58
42.95110.54
SimMMDG (Dong
2024)
89.5710.38
08.43
52.55
50.3110.42
67,0810.35
66.3510.39
49:5210.58
44.0110.52
MMBT (Kiels et al. 2019)
91.52+0.37
91. 01.38=0.36
56.75=0.55
56.21=0.00
78.50+0.34
78.04+0.99
55.35=0052
02:2270:5
QMF 12hang et al., 2023c)
92.9220.32
02.7220.355
02.21
61.7620.40
78.0720.39
76,30.0.31
61,2810,55
$7.61
CLIP+C*R
92,93=10004 04 (#77.99) 91, S00:1.41 (+77.60)
59,77
(+7.65)
57.54
(+8
23)
(+
68
64
(+7.42) 39,
00.58811.5 1.52 (+7.94)
35 89: (+7.62)
ALIGN-OOR
90.91=0995 (14.77) 90. 13+ (+5.13)
58
74
(35.5))
57, 96.6-+1177 1,78 (-7.
(+5.40)
46)
67
21+111111 (14.52)
37.26---1111
(46.71)
33,60=1
(+7.16)
MRPLe+C"R
94.38
(33.98933.5113
1.55::1.555 (+7.200)
60.63: 59 (+7.4 (7) 46.07 10.69
(-5 86) $1.19.0.50
(+3.76)
81 51.51 511.94 (56.15)
(5)54992..10.000 (+5.60)) 45.98 14 (-7.16)
ER
94 34.09-0 (4-3.40)92.25 92.092222---5
65.27-1
59. 59.02+0
80.77-0.66
79 75-0 (+5.03)
62. 14-00 7-
52,50-0095
(14.72)
ConEIMMLL+CR
94.48_00.73 (+5.05) 94,36 20 (+5.57) 60,91 23 (+1.3) 89) 59, L1.1 (-5 (3) 70 75.65 (+4.53) 78 04 (+3.51) 59 00.36..1 (+5.9-)
57,66 11111101111 (1.7.1)
Bow+CCR
96.61:11.29 (+-,111) 89.51 (+6.99)
46.44
57 (+-1.4) (9) 48.62
(+7.21)
$4.89 0O (-6.
10) 43, 00 (+7.71)
47 47.93 107 (+6.36)
37.67:00.71 (+3.49)
LCKD&C'R
90.80=1 22 (+5.57))90
90. 14+11
54,48-1 21 (47.05) (5)
16-0 (16.94) 66 78-667
68 67
49
(45.766
(14.21)
U
91. 01.76:00669 (+3.37) 89, 099.74:10.066
(+2.53 53) 54, 00 (+3.3 51) 52, 10
(-4 19) 70 49.00.1 (+3.52) 67 96 0. (+2.07) $2.56
(+4.22)
47.55 (+1
SLMMMDG+CRR
92024=1107 (+2.67))
11, (+2. (22.1))
57.32 53 (-4.7 77) 53. 53.56
(+3.25)
73 62 22 (+6.54)
71.111.1 (+1.66)
51 10.80 (+2.13)51.072000
(+7.06)
MMBT+<"R
04.25.0.73 (+2.73)995
93.
(+2.52)
60.41
90) 82
(+1.26)
(+6.77) 58.93.111111111
(+6.711
QMP+C*R
94.87=1 1.00 (+1.95)933))
(+1.07)
66.
63 (+4
(4)
63.09
1.38 (+1.93)
83 03 (+5,06)
81 0. (+5.68) 66.66:0.551
(+5.38) 64.51
1.08 (+6.90)
and analysis of "When faces the problem of missing modalities" (the second experiment in Subsection 6.2 with Table 2). Next, provide the full results and more analysis of "Learning causal complete representations" (the third experiment in Subsection 6.2 with Figure 3), including the detailed settings, the visualization and more analysis of the correlation between different methods and different causal causes under different spurious degrees. Finally, we provide the full results and additional experiments of the ablation study, including the experiments about model efficiency, trade-off performance, and parameter sensitivity.
H.1. Full Results and Additional Details of Performance and Robustness Analysis
Due to space limitations, we provide part of the experimental results in Table 1 of the main text, which contains typical methods of all categories of baselines mentioned in Appendix F. In Table 6 and Table 7, we provide comparison baselines for all baselines. Specifically, Table provides the experiments about scene recognition on NYU Depth V2 (Silberman et al., 2012) and SUN RGBD (Song et al., 2015) datasets. Table 7 provides the experiments about image-text classification on UPMC FOOD101 (Wang et al., 2015) and MVSA (Niu et al., 2016) datasets. From the results, we can observe that (i) CR achieves stable improvements in both the average and worst-case accuracy on almost all the comparison baselines, including both the foundation model and all types of MML baselines; and (ii) regardless of the data scale, C can bring obvious and stable performance improvements, with an average increase of more than 5%.. This proves the superior effect and robustness of CaR
H.2. Full Results and Additional Details of Performance with Missing Modalities
Due to the complexity of data in real systems, MML methods face the dilemma of missing modalities (Zhao et al., 2021; Zhang et al., 2022a). This problem severely affects model performance, especially when important modes are missing. Therefore, in order for the model to have good practicality, it is very critical to be able to maintain stable performance in the face of missing modes. To evaluate the performance of the proposed C when facing missing modalities, we constructed
32

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
0.9
0.7
0.7
0.7
INC
0.4
CovaSSERT
64
cancussent
0.4
ConcatSER
0.2
0.2
task scht
0.3
ods
Merhets
Merteds
Methods
Douplet
Bouglot of
0.8
0.5
0.7
0.5
Comeatton
Comestoo
0.5
6.3
Thk:
0.4
ConcuESER
0.
CovoutEET
0.3
6.3
Methoca
Methhesa
Mothoca
Methods
(a) Spurious correlation degree Dap 0.3
(b) Spurious correlation degree Dap 0.6
Figure 6: Evaluation for the property of learned representations (identification for SNC, SC, NC, and SP) with different spurious correlation degree Dsp-
comparative experiments on all 15 possible combinations of missing modalities on BraTS. Specifically, based on BraTTS, we construct fifteen combinations of different modes following (Wang et al., 2023b). Then, we recorded the performance of Cº and several strong baseline models after five runs.
From the results illustrated in Table 2, on Enhancing Tumour. Likewise, Tlc and Flair from the Tumour Core and Whole Tumour contributed the most. (ii) The model has been significantly improved after the introduction of CRR, especially when important modes are missing, e.g., Fl on Enhancing Tumour. The results prove that our method can improve the performance of the model in the scenario of missing modality and improve the stability.
observe that (i) using F1 and T1c, the model performs much better than others
H.3. Full Results and Additional Details of Causal Complete Evaluation
In order to verify whether our method CR actually extracts causal necessary and sufficient representations, we construct synthetic data set called MMLSanData (as mentioned in Appendix D.5) to conduct comparative experiments. Specifically, we first built MMLayn which is synthetic data set for MML scenarios containing four types of data, i.e., sufficient and necessary causes (SNC), sufficient but unnecessary causes (SC), necessary but insufficient causes (NC), and spurious correlations (SP). Each category contains 250 sets of training data and 50 of data. Next, based on the above 1200 sets of data, set different degrees of spurious correlation Dep for comparative experiments, including Dsp 0.3 and Dap 0.6. It is worth noting that the result shown in Figure 3 in the text is D.p = 0.6. Then, we choose SOTA and classic MML methods to compare with the basic MML framework after the introduction of C*R, where basic represents simple MML learning framework based on the Conv4 backbone network and a classifier following (Wang et al., 2016). We record their correlation with four different types of data in MMLSynData.
The results of x.p 0.3 and D.p 0.6 are shown Figure Combined with Figure 3, the results show that compared with other methods, the correlation with real data (such as SN, SF, and NC) higher after the introduction of C*R, and the correlation with spurious correlation (SP) has lower score. For example, when Dap 0.3, obtain the average distance corelations of SNC. SC, NC, and SP as 0.91, 0.71, 0.69, 0.13 respectively. Furthermore, when we set Dap to a larger value of 0.6, the distance correlation between the MML method and false information is almost unchanged after introducing C*R while other methods are difficult to achieve this. The results show that when there are more spurious correlations in the data, C still tends to capture valid information from the real data to extract sufficient and necessary causes.
H.4. Full Results and Additional Details of Ablation Study
To evaluate the effect of the model and understand how C*R works well, we constructed a series of experiments, including (i) the effect of each item in the C objective as shown in Subsection 6.2; (ii) trade-off performance about the model efficiency and accuracy after introducing C$R; (iii) parameter sensitivity about the three hyperparameters in the CRR
33

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Parameter Size (M)
1M
LCKD&C QMF+C'R
Accuracy (%)
SiMMMDG-CCR
Late fusionn
CoMKaEMML
Late fusion
Training Time (h)
Figure 7: Trade-off Performance of different methods on (0,AAgg)) NYU Depth V2.
78
76
76
Accuracy(5))
Accuracy
75-
78
73
72
72
Ag
An
Figure 8: Parameter sensitivity about X on (0.Avg.) NYU Depth V2.
Figure 9: Parameter sensitivity about Afe on (0.Avg.) NYU Depth V2.
objective, i.e., A and Aye- In this section, we provide details about the latter two experiments and results, where the results of the first experiment are shown in Figure 4.
Model efficiency Since CR is plug-and-play module, in order ensure its practicability, explore the balance between model performance and efficiency. We compare the trade-off performance of multiple baselines before and after using our CR with the same Conv4 backbone. The results illustrated in Figure 7 show that introducing CR achieves great performance with acceptable computational cost.
Parameter sensitivity We determine the hyperparameters the regularization term the experiment based on the performance of the validation samples. Specifically, for each experimental scenario, we test the impact different values of X, and Afe model performance. The range of these values is set between [0.3, 0.9]. In each scenario, first use grid search to screen the parameters with difference of 0.05. After screening the optimal interval, we screened the parameters with difference of 0.01 and recorded the final average results.
The results are shown in Figures 8-9. From the results, we observe that when X, = 0.75, and Aço- = 0.4, the results are better which is also our choice. Meanwhile, the model effect does not change significantly under different parameters, which illustrates the stability of the parameters and the convenience of adjustment in reality.
34

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
ruce am
(a) Heatmap of Sample
(b) Heatmap of Sample 2
(c) Heatmap of Sample 3
(d) Heatmap of Sample 4
(e) Heatmap of Sample 5
(f) Heatmap of Sample 6
(g) Heatmap of Sample
(h) Heatmap of Sample
Heatmap of Sample 9
Figure 10: Nine sets of visualization results on the BraTS dataset. For qualitative demonstration, we used the model after five iterations calculate the importance weights of the resized (32 x 32) sampled data and obtained the above heat map. In the heat map, the score of each area represents the probability that it belongs to a causal complete cause.
35

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Table 8: Effect of the extracted representation. This experiment aims to analyze the practical guidance of C R and how this representation mirrors the properties in the example. The "Original" means that the representation is directly from the extractor of the method; "Reduce" means we reduce the weights of high-weight elements which are more likely to be causal complete and directly input them into the frozen classification head; "Increase" means we increase the weights of low-weight representation elements twice the original weights and normalize them.
Method LCKD LCKD+C R LCKD+C R LCKD+C*
Representation
(ccuracy(9))
Original
79.30
Original
85,39
Reduce
83.73
Increase
84.02
Table 9: Performance when faces noise on NYU Depth V2. We follow the same experimental setting in Section 6.2. We apply a mask to 30% of the data area. Then, we selected MMBT and QMF as the baselines to calculate the performance changes before and after the introduction C
Method
Accuracy(%)
MMBT 62.89
QMF
64.72
MMBT+C R
71.26
QMF+CRR
75.49
H.S. Visualization of Causal Complete Cause
In addition to the examples of causal completeness provided in Figure and Appendix D, we construct more qualitative demonstrations for real datasets. Specifically, for the five datasets involved in the experiment, we randomly sample a set of training samples from each dataset and visualize, i.e., caleulate the importance heatmap of sufficient and necessary causes based on the constructed 8 R. The results are shown in Figure 10.
Besides this, we conduct a series of visualization experiments to further elucidate the practical guidance provided by C*3 R and to examine how this representation reflects the underlying properties in the example. Utilizing the BraTS dataset as a case study, we consider four modalities, namely F1, T1, Tlc, and T2, and adopt LCKD as the baseline method. The results are presented in Table 8. Specifically, we first randomly select five sets of LCKD to obtain both the results and the representations with the weight matrix. Subsequently, we employ the same settings to visualize the feature representations after integrating C observing that the weights altered and the accuracy is significantly enhanced. We hypothesize that this improvement arises because C R calculates the probability of representations belonging to causal factors and subsequently re-weights the matrix. To further validate this hypothesis, we reduce the weights of high-weight elements and directly input them into the frozen classification head, resulting in a decrease in accuracy. Conversely, when we amplify the weights of wei-ht representation elements to twice their original values and normalize them, the accuracy also declines. Thus, this example demonstrates how C3 R effectively guides the representation in these datasets, where elements with high weights correspond to the causal factors of the selected data.
data and input them into
H.6. Performance When Facing Noise
One reason of why we conduct experiments of "Performance and robustness analysis" is to consider the impact of data noise and damage. The difference between the "worst result" and the "average result" under the same setting may be caused by noises. The results in Table Table 6, and Table 7 show that C R steadily improves the effect of MML and greatly reduce the gap between the worst and average results. Meanwhile, we also considered more serious data defects such as missing modality, and the results (Table 2) reached similar conclusions. These results prove that (i) C R is effective and robust, and (ii) noise will damage multi-modal representations but hard to affect causal representations learned by CR.
To comprehensively evaluate the effect of C in the presence of noise, we further provide a toy experiment to intuitively evaluate the impact of noise on C R. Specifically, we choose NYU Depth V2 as the benchmark dataset and apply a mask to
36

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
Datasets
Standard 101 Dataset with Missing Modalities: BraTs (F1 and T1)
05
Functions
Accuracy
MSE
85
Cross-Entropy
KL
Wasserstein
4.0
5,5
7.0
Training Hours
Figure 11: Trade-off Performance of with different distance loss functions.
30% the data area. Then, we select MMBT and QMF as the baselines to calculate the performance changes the models before and after the introduction of C R. The results are shown in Table 9. The results show that (1) noises affects the performance of the MML baselines with the degradation exceeding 5%; (2) even in the presence of noises, the performance of MML after the introduction of C R only decreases by less than 2% compared with the original (Table 1), needless to say that is may be due to the mask blocking important feature areas. This further shows that C R has good robustness.
H.7. Role of Distance Loss
In this subsection, examine the role of loss functions for distance in the objective of C. The goal of this optimization is to evaluate the distance between Z,, and Zo. with the choice of loss function directly affecting performance. We evaluate four common distance loss functions-Mern
Squared Error (MSE) (Tsai et al., 2020), Cross- Cross-Entroopy (De Boer et al., 2005), KI. Divergence (Hershey& & Olsen, 2007), and Wasserstein Distance (Panaretos & Zemel, 2019)-by analyzing their performance and training time on FOODI01. The loss functions are defined as follows:
Mean Squared Error (MSE) (Tsai al., 2020) measures the average squared differences between predicted and true values across modalities. is simple to compute but sensitive to differences between modalities:
n Σ byy, --1-.11.. 7i
MSE(y, Y)
where y and ý are the true and predicted values across modalities, || || is the Euclidean distance, and is the sample size.
Cross-Entropy (De Boer et al., 2005) calculates the difference between true and predicted joint distributions, capturing uncertainty between modalities:
CB(y, 8)
Σ Vij log Wij.
where y and ý denote the true and predicted joint probability distributions, and m is the number of modalities.
KL Divergence (Kullback-Leibler Divergence) (Hershey & Olsen, 2007) measures the difference between two joint probability distributions, offering a balance between accuracy and computational cost:
KL(PIIQ) = ΣΣ P(i.)) log P(i.j) Q(i,))
m
5=1
where and Q are the true and predicted joint distributions.
Wasserstein Distance (Panaretos & Zemel, 2019) reflects the geometric distance between joint distributions but is computa- tionally expensive:
WD(P. = inf 7EI((P:Q)
PIx YI]].
37

---

Towards the Causal Complete Cause of Multi-Modal Representation Learning
where P and Q are distributions, [I,,, Q) represents the set of joint distributions coupling P and Q.
From empirical analysis, KL divergence achieves the best trade-off between accuracy and efficiency, as shown in Figure 11. We evaluate the effect of LCKD+C R by introducing different distance metrics. We select the standard MML benchmark dataset, FOOD 101, and dataset with missing modality, i.e., tumour core on BraTS (with FI and Tl available), for evaluation. The results show that it provides higher accuracy in shorter training times compared to MSE and cross-entropy, while Wasserstein distance achieves comparable accuracy, its computational cost is significantly higher, making KL divergence the more practical choice.
From theoretical perspective, to better use C in MML models, the distance metric used in Theorem 3.5 should include: (i) the ability to capture subtle differences between distributions accurately, (ii) utility in ensuring stable and efficient convergence to the global optimum during optimization, (iii) applicability to a wide range of complex distributions, and (iv) computational efficiency. KL divergence excels in these aspects, as evidenced by three key features (Hershey & Olsen, 2007; Goldberger et al., 2003; Shlens, 2014). First, KL divergence is non-negative and equals zero only when two distributions are identical, aligning with intuitive notions of difference (Gong et al., 2021). This property ensures its stability in capturing subtle differences, fulfilling criteria (i) and (iv). Second, KL divergence convex, increasing the likelihood of convergence the global optimum rather than becoming trapped in local minima, especially in high-dimensional settings (Hershey & Olsen, 2007). This characteristic addresses criterion (ii). Finally, effectively quantifies information loss and uncertainty, making it highly versatile for diverse applications (Goldberger et al., 2003), including self-rewarding learning tasks, thereby satisfying criterion (iii).
extension of information entropy, KL divergence
In contrast, alternative metrics exhibit significant limitations. MSE, rooted Euclidean distance, is overly sensitive to outliers and disregards the non-negativity and normalization properties of probability distributions (Marmolin, 1986; Chicco al., 2021; Lebanon, 2010), failing meet criteria (i) and (iii). Cross-entropy, specific case of KL divergence, struggles with continuous distributions and non-one-hot true distributions (De Boer al., 2005; Botev al., 2013), limiting its ability to precisely measure complex distributions (i and iij.. While Wasserstein distance captures overall shape differences between distributions, its high computational cost and dependence on smoothness conditions make it unsuitable for high-dimensional problems (Panaretos & Zemel, 2019; Vallender, 1974), falling short of criterion (iv).
In summary, KL divergence strikes an optimal balance between theoretical robustness and computational feasibility, adhering to the properties for twin network. This balance enables better model generalization and reduced training costs, making the preferred metric for practical applications.
38


Reviews:
[{"review_id": "D9hIE1G7W3", "reviewer": "Reviewer_ZspQ", "summary": "This paper proposes Causal Complete Cause Regularization (C³R) Risk, a metric that quantifies the likelihood that a learned representation is causally complete. A lower C³ Risk indicates that the representation satisfies both causal sufficiency and causal necessity, meaning that spurious correlations have been removed, and all essential causal factors are retained.\n\nIn conventional Multi-Modal Learning (MML) methods, two major issues arise. 1) Spurious correlations exist in sufficiency evaluation, leading to unreliable representations. To address this, the paper introduces Twin Network’s real-world branch, which leverages a self-attention-based instrumental variable to ensure that representations do not rely on non-causal factors, effectively mitigating spurious correlations. 2) For necessity evaluation, the challenge is that it requires observing label changes when a representation is absent, which is impossible in real-world data. \n\nThe proposed method overcomes this limitation using Gradient-Based Counterfactual Modeling in Twin Network’s hypothetical-world branch. Since directly removing a representation is not feasible, the model instead adjusts gradients in the loss function to guide the representation toward a counterfactual direction, generating an approximation of counterfactual representations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer ZspQ's feedback and the time and effort dedicated to the review. We are also grateful for the recognition of our work and sincerely hope the following responses can eliminate the concerns.\n## Response to W1 & Q3\n- The true counterfactual effect involves unobserved outcomes, making direct modeling difficult (Pearl, 2009). Existing work typically adopts the \"minimal change\" principle to estimate counterfactuals, which is proven to align with true counterfactual effects (Galles & Pearl, 1998; Kusner et al., 2017). For instance, Chapters III–V in Wachter et al. (2017) demonstrate that by making only minor adjustments to the treatment variable while preserving the distribution of other covariates, the counterfactual samples maintain the original data’s characteristics, ensuring accurate counterfactual effect estimates. Based on these results, we leverage gradient intervention to satisfy the \"minimal change\" principle for counterfactual effect estimation with theoretical guarantees (Appendix D.4). To assess whether the generated counterfactual data adhere to the principle for accurate estimation, we develop a distribution consistency test with Wasserstein distance $D_w$, i.e., whether the distribution of the covariates matches that of the original data. The lower $D_w$ shown below proves that our method satisfies the principle.\n- According to the suggestions, we conduct a toy experiment on LCKD and NYU Depth V2 to demonstrate credibility. We follow (Galles & Pearl, 1998) to make manually curated examples and select the recently proposed transport-based counterfactual modeling method (Lara et al., 2024) as another baseline. The table below shows the advantages of our method, i.e., superior accuracy and lowest computational cost.\n|Method|Acc|Calculation overhead|$D_w$|\n|-|-|-|-|\n|$C^3$R|77.6|$1\\times$|0.9|\n|manually curated|77.4|$4.9\\times$|0.7|\n|Transport-based|75.2|$2.3\\times$|2.6|\n## Response to Q1\nWe have conducted experiments to validate \"$C^3$ risk serves as an accurate indicator\":\n- Appendix H.5: We calculate the heatmaps of samples using $C^3$ risk in Fig.7, where high-weight elements correspond to low $C^3$ risk. If low $C^3$ risk reflects causally sufficient and necessary factors, then reducing the weights of these elements would degrade model performance. Table 7 shows that reducing the weights of low $C^3$ risk elements indeed lowers performance, confirming that $C^3$ risk is an accurate indicator.\n- Section 6.2: We conduct experiments on MMLSynData, which includes four types of generated data, i.e., sufficient and necessary causes (SNC), sufficient but unnecessary causes (SC), necessary but insufficient causes (NC), and spurious correlations (SP). By evaluating the correlation between the representation learned by minimizing $C^3$ risk and SNC, we can validate whether low $C^3$ risk refers to causal sufficient and necessary causes. Figures 3 & 6 indicate that $C^3$R significantly increases the correlation between representations and SNC, proving the accuracy.\n- Section 6.2 & Appendix H: If $C^3$ risk can be the indicator, then $C^3$R should lead to performance improvements by minimizing $C^3$ risk. Tables 1-8 show that $C^3$R achieves significant performance gains across all baselines, proving effectiveness.\n## Response to Q2 & Comments\nWe have conducted experiments on datasets with \"high spurious correlations\" and provide the outline below:\n- Quantitative (Section 6.2, Appendices D.5 and H.3): We conduct experiments on MMLSynData, which contains SNC, SC, NC, and SP as mentioned in \"Response to Q1 (2)\". Different degrees of $D_{sp}$ are set to control the level of spurious correlations. By evaluating the correlation between the learned representation and SP/SNC, we can validate the effectiveness of the corresponding model in mitigating spurious correlations/learning causal complete representation. Figures 3 and 6 show that even with high spurious correlation ($D_{sp}=0.6$), $C^3$R markedly reduces the correlation with SP (0.4 -> 0.1) while significantly increasing the correlation with SNC (0.5 -> 0.9),  proving advantages.\n- Qualitative (Appendix H.5): As in \"Response to Q1\", the visualization shows that $C^3$R learns causally complete representations and effectively eliminates spurious correlations.\n## Regarding Essential References Not Discussed\nWe sincerely thank the suggestion and carefully reviewed the paper, finding differences with ours in problem, theory, method, experiments, etc. (cited and added in Section 7). Kim et al. aim to address modality bias in multispectral pedestrian detection tasks, e.g., \"prediction on ROTX without thermal features\". They conduct SCMs for analysis and propose CMM for the tasks. Differently, we focus on the concepts of causal sufficiency and necessity and are the first to explore them in MML. We propose a new theoretical framework and a plug-and-play method to learn causal complete representations, which can also be embedded in CMM.", "claims_and_evidence": "Table 1 and Table 2 demonstrate the effectiveness of applying C³ Risk to various benchmark datasets, including NYU Depth V2, SUN RGBD, FOOD 101, MVSA, and BraTS. The results show that incorporating C³R improves performance across scene recognition, image-text classification, and segmentation tasks, particularly by enhancing both average and worst-case accuracy, thereby increasing model robustness.\n\nAdditionally, Figure 4's Ablation Study experimentally verifies the contribution of each component in the proposed loss function. The study confirms that these components play a crucial role in improving model performance by evaluating performance degradation when removing C³ Risk, the instrumental variable, and counterfactual representation modeling. The results indicate that the proposed loss function is effective in learning robust and causally complete representations.", "methods_and_evaluation_criteria": "The proposed C³R method and evaluation criteria are well-aligned with the goal of learning causally complete representations in Multi-Modal Learning (MML). The introduction of C³ Risk effectively quantifies causal sufficiency and necessity, while the Twin Network architecture provides a structured approach to estimating counterfactual effects. The use of instrumental variables further helps disentangle causal from spurious factors, which is essential for robust multimodal learning. \n\nThe evaluation strategy is comprehensive, covering six benchmark datasets spanning classification and segmentation tasks, including NYU Depth V2, SUN RGBD, FOOD 101, MVSA, and BraTS. The robustness analysis under Gaussian noise (image) and blank noise (text) strengthens the credibility of the results, ensuring that C³R improves both average and worst-case accuracy. The ablation study further confirms that each component of C³R contributes meaningfully to performance.", "theoretical_claims": "The proposed methods and evaluation criteria are well-aligned with the problem, as they have been quantitatively validated across various MML methods and diverse MML tasks.", "experimental_designs_or_analyses": "The soundness and validity of the experimental designs were verified. \n\nIn Table 1, the proposed method was validated by comparing it with recent approaches across various MML benchmark datasets, including NYU Depth V2, SUN RGBD, FOOD 101, and MVSA, which cover different tasks. Additionally, the ablation study confirmed the effectiveness of each regularization term.", "relation_to_broader_scientific_literature": "The key contributions of this paper are closely related to existing research in multi-modal learning (MML). \n\nBy adopting the concepts of sufficiency and necessity from Pearl (2009), it introduces a novel approach—Causal Sufficiency and Causal Necessity. which significantly advances MML learning.", "essential_references_not_discussed": "Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detection, CVPR 2024", "other_strengths_and_weaknesses": "* Strengths:\n\nThe paper effectively identifies the limitations of existing causal representation learning research and introduces a novel approach that ensures both causal sufficiency and necessity, which is highly compelling. \n\nAdditionally, the use of Gradient-Based Counterfactual Modeling to disentangle causal factors and approximate counterfactual effects without generating new data is particularly interesting, as it allows for counterfactual reasoning without access to actual counterfactual samples.\n\n\n* Weaknesses:\n\nWhile the Twin Network generates counterfactual representations, there is a lack of direct validation on how well these estimated counterfactuals align with true counterfactual effects. \n\nA comparison with manually curated counterfactual examples or other counterfactual modeling approaches would enhance the credibility of the proposed method.", "other_comments_or_suggestions": "The paper claims that instrumental variables are used to mitigate spurious correlations, but there is a lack of quantitative experiments measuring how much spurious correlation has actually been reduced. \n\nA more compelling validation would include quantitative and qualitative comparisons of representations before and after applying C³R on datasets with a high presence of spurious correlations.", "questions_for_authors": "1. How can C³ Risk be directly verified as an indicator of causally complete representations? Specifically, what experimental approaches could be used to validate that a lower C³ Risk truly corresponds to representations that satisfy both causal sufficiency and necessity?\n\n2. If you have conducted experiments using datasets with a high concentration of spurious correlations, could you provide details on the quantitative and qualitative comparisons of representations before and after applying C³R?\n\n3. Are there any quantitative or qualitative methods to evaluate how similar the counterfactual representations generated through Gradient-Based Counterfactual Modeling using the Twin Network are to actual counterfactual representations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uk3QMt3rcI", "reviewer": "Reviewer_fkSa", "summary": "This paper addresses the problem of multi-modal representation learning from a causal perspective. It analyzes the insufficiency and redundancy of information across multiple modalities. The authors propose a novel concept termed Causal Complete Cause ($C^3$), supported by identifiability guarantees under weaker assumptions (non-exogeneity and non-monotonicity). A twin network is introduced to estimate the $C^3$ measurement, and extensive experiments are conducted to validate the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer fkSa's constructive feedback and the time and effort dedicated to the review. We are grateful for the recognition of our work and sincerely hope the following responses can eliminate the concerns.\n## Response to W1 \nWe sincerely appreciate the suggestions and have refined Fig.1 accordingly. The examples in Fig.1 are based on specific tasks and data conditions (L129-155). A sufficient and necessary feature can be \"flat duck bill\" as its presence indicates \"duck\" and every \"duck\" sample includes it. We also added the textual modality, e.g., \"A duck has a flat bill and orange webbed feet standing with its wings folded\" as suggested for better clarification.\n## Response to W2\nWe agree “in the absence...” but would like to kindly clarify that the SCMs in Fig.2 are conducted under different settings, aiming to illustrate potential confounding issues in MML instead of true data-generating direction (L129-155). Left is for how MML sample $X$ is generated based on causal generating mechanism; Right is to align with the practical MML process, where factors are typically coupled for predicting $Y$. We apologize for any ambiguity that may caused by the caption and have refined it accordingly.\n## Response to W3\nWe sincerely appreciate the suggestions and would like to clarify that our concept of causal sufficiency (Definition 3.1) is adapted from Definition 9.2.2 in Pearl (2009), i.e., “setting x would produce y where x and y are in fact absent”. It aligns with the mentioned “the absence of latent confounder”, as we utilize instrumental variable and twin network to eliminate confounders (Sections 3.3 and 4), aiming to satisfy causal sufficiency. We further emphasized it in Section 3.1 according to the valuable suggestion.\n## Response to W4, Theoretical Claims 1&2\n- We have illustrated “local invertibility” in Proposition D.3 and will further elaborate on it immediately following Theorem 3.2 as suggested. Briefly, it states that the model can uniquely recover the distribution of exogenous variable $s$ from the conditional distribution of $Y$ given its parents $Pa(Y)$.\n- We have provided a brief explanation about non-exogeneity, e.g., $P(Y_{do(Z=c)}) \\neq P(Y \\mid Z=c)$ in L185-191 with detailed in Appendix D.3. We will further refine this immediately following the theorem.\n- Compared to Theorem 3.2, the non-monotonicity in Theorem 3.3 allows the effect of $Z$ on $Y$ to vary in direction or intensity. The key modification is introducing $V$ for piecewise estimation through integration (Eq.5).\n## Response to W5\nWe sincerely appreciate the suggestions and provide a brief derivation following Theorem 9.2.15 in (Pearl, 2009): $C^3(Z)$ can be decomposed into (i) when $Z\\neq c$ occurs with probability $1-P(Z=c)$, its contribution reflects $C^3_{su}(Z)$ for $Z=c$; (ii) when $Z=c$ occurs with $P(Z=c)$, its contribution reflects $C^3_{ne}(Z)$ of $Z=\\bar{c}$ on $Y$. By normalizing, we get $ C^3_{su}(Z) = \\frac{C^3(Z)}{1-P(Z=c)}$ and $C^3_{ne}(Z) = \\frac{C^3(Z)}{P(Z=c)}$ as Eq.3 and Eq.4. We will add it in Appendix A.1 for better understanding.\n## Response to W6 & Theoretical Claim 3\nThe intuition is twofold:\n- Aligning with the modeling objectives of $V$ (L220–248): The self-attention mechanism, dynamically measures the alignment score between modalities, can emphasize the important features for $Y$ across modalities (i.e., achieves higher scores on $F_c$) while downplaying $F_s$. Intuitively, using self-attention-based $V$ on $Z$ with distance penalties satisfies the goal that \"constrain $Z$ only contains $F_c$\". \n- Higher efficiency: Although custom-designed networks can also achieve the above goal, the self-attention mechanism is typically more lightweight without substantially increasing model complexity.\n## Response to Q1\nThe learned representations can \"be mapped to interpretable latent concepts\". Such a representation must both fully explain the decision (sufficiency) and be indispensable (necessity), revealing the most important factors behind predictions. For example, a causally complete representation captures all critical factors for classifying \"happy\", e.g., visual (upturned mouth corners, wide-open eyes) and audio cues (high pitch) where removing \"upturned mouth corners\" may result in \"angry\". \n## Response to Q2\nThe results in Table 2 (missing modality) and Table 8 (data noise) show that $C^3$R achieves stable performance improvements in domains with real-world data complexities.\n## Regarding Essential References Not Discussed\nWe appreciate the suggestion and carefully reviewed the papers (cited and supplemented in Section 7). Schölkopf et al. present a review for causal inference; the rest four mainly focus on identifiability under different settings and problems, e.g., weak supervision and partial observability. We focus on the causal sufficiency and necessity concept, aiming to explore and model it for MML, where the problem, theory, method, and experiments are all different.", "claims_and_evidence": "Yes. This paper is mainly discussing the definition (explained in Section 2 & 3.1), identifiability (explained in Section 3.2), and measurement (explained in Section 3.3) of the novel causal complete cause ($C^3$).", "methods_and_evaluation_criteria": "Yes, the proposed method is the causal complete cause ($C^3$), which makes sense for multi-modal representation learning, and the evaluation criteria is mainly the classification accuracy, which is also reasonable.", "theoretical_claims": "Yes, the key theoretical claims that I checked include Theorem 3.2 (Identifiability under Non-Exogeneity), Theorem 3.3 (Identifiability under Non-Monotonicity and Non-Exogeneity), and Theorem 3.4 (Modeling Instrumental Variable V in MML).  \n\n- In Theorem 3.2, what do you mean by \"local invertibility\"?\n\n- how to interpret that non-monotonicity in Theorem 3.3, compared to the monotonicity requirement Theorem 3.2? What is the key modification such that monotonicity is not required anymore.\n\n- In Theorem 3.4, what is the intuition of choosing self-attention mechanism, instead of other methods?", "experimental_designs_or_analyses": "Yes. The experimental design is sound and valid, the experimental evaluation is comprehensive, covering 17 baseline methods and 6 datasets across various tasks including scene recognition, image-text classification, and segmentation.", "relation_to_broader_scientific_literature": "This paper presented the definition, identifiability, and measurement of $C^3$ with theoretical support, without exogeneity and monotonicity assumptions. This is a relaxed and quantifiable framework, compared to previous work.", "essential_references_not_discussed": "In general, this paper has covered most related work. It would be more comprehensive if also comparing and discussing the following disentangled/causal representation learning papers, to name a few:\n\n- Schölkopf et al. \"Toward causal representation learning\", Proceddings of IEEE, 2021.\n\n- Brehmer et al. \"Weakly supervised causal representation learning\", NeurIPS, 2022.\n\n- Ahuja et al. \"Interventional causal representation learning\", ICML, 2023.\n\n- Yao et al. \"Multi-view causal representation learning with partial observability\", ICLR, 2024.\n\n- Sun et al. \"Causal representation learning from multimodal biological observations\", ICLR 2025.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- This paper is well-written and clearly-organized. \n\n- The $C^3$ concept is novel and intriguing. The authors provide a new measurement backed by identifiability guarantees, along with a thorough analysis of sufficiency and necessity.\n\n- The experimental evaluation is comprehensive, covering 17 baseline methods and 6 datasets across various tasks including scene recognition, image-text classification, and segmentation.\n\n**Weaknesses and Comments:**\n\n- The example in Figure 1 remains somewhat ambiguous. The distinction between sufficient and necessary features in the image domain is unclear. Including an additional example that explicitly illustrates features that are both sufficient and necessary would be helpful. Additionally, the figure should more clearly reflect the elements of multi-modality.\n- In Figure 2, both anti-causal (Y → X) and causal (X → Y) directions are discussed. However, in the absence of spurious correlations, either direction could represent the true data-generating process. The comparison between an anti-causal mechanism (true, without spurious correlation) and a causal mechanism (learned, with spurious correlation) may be misleading.\n- Terminology clarification: The term causal sufficiency in Section 3 is typically used in causal inference to mean the absence of latent confounders. It would be clearer if the authors explicitly clarify their intended meaning in the context of this paper.\n- In Theorem 3.2, the terms local invertibility and non-exogeneity are not clearly defined. Providing a brief explanation or formal definition immediately following the theorem would improve clarity.\n- The derivations of Equations (3) and (4) are not clearly explained. A more detailed explanation or derivation would help the reader better understand the methodology.\n- What is the intuition of choosing self-attention mechanism particularly for generating instrumental variable V?", "other_comments_or_suggestions": "There is no obvious typo so far.", "questions_for_authors": "- Can the learned representations be mapped to interpretable latent concepts, or are they inherently abstract?\n- How does the method perform in domains with real-world data complexities, such as missing data, or measurement error?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bWc7Nc2dIH", "reviewer": "Reviewer_eaJB", "summary": "The paper explores causal completeness in multi-modal representation learning, addressing issues where existing methods may capture unnecessary or insufficient information. It introduces the Causal Complete Cause (C3) framework, which ensures learned representations are both sufficient (contain all necessary information) and necessary (exclude irrelevant details). The authors propose a twin-network approach using instrumental variables and counterfactual modeling to estimate and enforce C3, leading to a new regularization method (C3R). Experimental results show that this approach improves robustness and accuracy, especially in scenarios with spurious correlations and missing modalities", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer eaJB's constructive feedback and the time and effort dedicated to the review process. We are also grateful for the recognition of our work and sincerely hope the following responses can eliminate the concerns.\n## Response to W1\nWe appreciate the suggestions and apologize for any misunderstandings. To distinguish it from traditional identifiability, we change the word to \"Causal Identifiability.\" According to Pearl (2009) and Lee & Bareinboim (2020), causal identifiability refers to establishing the effect of intervening on a set of variables on another set, using observational or interventional distributions under given assumptions. It permits the inclusion of interventional terms if they are uniquely determined from the observational distribution (Shpitser, 2012). Based on these, Theorem 3.2 is derived. When the model satisfies local invertibility, we can uniquely recover the variable distribution from the observed data, thereby ensuring causal identifiability and estimation of $C^3$.\n## Response to W2\nThe experiments on computational cost in Appendix H.4 (Figure 7) indicate that introducing $C^3$R increases the computational cost by less than 1.3x compared to the original.\n## Response to W3\nWe provide an outline to explain how it \"handles hidden confounding\":\n- Theoretically: The \"hidden confounding\" issues correspond to the relaxation of the exogeneity assumption, which requires satisfying causal identifiability of $C^3$ in the environment that contains hidden confoundings. To achieve the relaxation, we introduce an instrumental variable $V$ to eliminate confounding effects, thus achieving causal identifiability regardless of whether hidden confounding exists (Section 3.2). We model $V$ using the improved self-attention mechanism, which employs alignment scores to capture causal factors $F_c$, theoretically constrain $Z$  to only contain $F_c$, and eliminate confounding effects (L220-257 and Appendix A.3).\n- Methodology: Based on theoretical results, we constrain the causal completeness of the representations with $C^3$ risk. The concept of causal completeness also means without \"hidden confounding\" in causality (Pearl, 2009). We propose $C^3$R to constrain it by reducing $C^3$ risk. It is achieved through a twin network: the real-world branch uses $V$ to adjust $Z$ so that it only contains $F_c$ for accurate prediction, achieving causal sufficiency and eliminating hidden confounding; the hypothetical branch employs gradient-based counterfactual modeling to further calibrate the causal factors, constraining causal necessity and further filtering out hidden confounding (L258-307 and Appendix D.4). Table 1-8 prove its effectiveness.\n## Response to Minor 1\nWe sincerely appreciate the suggestions and would like to provide an outline for illustration (further emphasized in Section 3.1):\n- Meaning of Causal Sufficiency: Our concept of causal sufficiency (Definition 3.1) is adapted from Definition 9.2.2 in Pearl (2009), i.e., “setting x would produce y in a situation where x and y are in fact absent” which is considered the original definition of causal sufficiency. It also aligns with another type of illustration in causality literature, e.g., “the absence of latent confounder”, as we utilize instrumental variable and twin network in Sections 3.3 and 4 to eliminate confounders, aiming to satisfy causal sufficiency.\n- Clarification on Assumptions: Previous works rely on an exogeneity assumption, assuming there are no confounders in environments. In contrast, we relax these assumptions to account for possible confounders in practice (L80–99 and Appendix D.3). It does not alter the meaning of causal sufficiency; it means that we do not assume that there is no confounding in the environment, but introduce instrumental variables to eliminate the impact of confounding to relax the assumption.\n## Response to Minor 2\nWe sincerely appreciate the suggestions and have carefully reviewed the mentioned paper. Below is a brief illustration of the differences (cited and supplemented in Section 7). Wang & Jordan aim to construct measures of non-spuriousness and disentanglement. Their exploitation of causal necessity and sufficiency concept is to align it with non-spuriousness and “invoke” the corresponding measure in Pearl (2009) to construct the measure of non-spuriousness. We focus on modeling the concept of causal sufficiency and necessity itself in MML. We relax the exogeneity and monotonicity assumptions that previous works depend on, including Wang & Jordan, and propose a new method to measure and constrain MML-specific causal completeness. Thus, although both works draw inspiration from Pearl (2009), they differ significantly in problem, theoretical framework, methodology, and experimental validation.\n## Regarding Comments\nWe appreciate the reviewer's suggestions and have polished the text accordingly, i.e., changing “with both” to “be both”, “conduct” to “construct”, and “a” to “an”.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I only checked proof of theorem 3.2.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "I am not aware of the literature in multi-modal representation learning.", "essential_references_not_discussed": "I can not certainly comment on this because I am not aware of the literature in multi-modal representation learning.", "other_strengths_and_weaknesses": "> Strengths\n1. The paper is written well and easy to understand.\n2. Even though the idea of causal sufficiency and necessity is borrowed from literature, applying it to multi-model learning shows.\n\n> Weaknesses \n1. In theorem 3.2, the the term obtained after identification still contains interventional terms. This is in contrast to traditional identifiability where the final result is statistical estimand instead of an estimand containing interventional terms. It may be good to use another word instead of \"identifiability\".\n2. Because the proposed method is used to improve the existing methods as shown in Table 1, what is the additional computational cost of the proposed method adds to the existing methods?\n3. In real-world data, there is a high chance that hidden confounding exists between causal and spurious feature, how does the proposed method handle such scenarios?\n\n> Minor issue:\n1. It is crucial to highlight the difference between causal sufficiency assumption used in causality literature to avoid confusion.\n2. How are the ideas related to the paper: https://arxiv.org/abs/2109.03795 ? The ideas of causal sufficiency and necessity of representations has been studied in the paper. It is crucial to discuss this paper in related works.", "other_comments_or_suggestions": "> Typos:\n\n1. Line 78: Looks like there is a grammatical error in the sentence.\n2. Line 156: Shouldn't it be \"we construct a structural causal model\"?\n3. Lin2 121 (right): Should be \"an SCM\".", "questions_for_authors": "Please see weaknesses above. Addressing those points is crucial for strengthening the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses key limitations in Multi-Modal Learning (MML), where representations may be influenced by spurious correlations or lack causal relevance. It introduces the concept of Causal Complete Cause, C^3, requiring that learned representations be both causally sufficient and necessary. To operationalize this, the authors propose C^3 Risk, a metric quantifying the failure to meet causal completeness. They develop a Twin Network to estimate and minimize this risk. The resulting C³R regularization is a plug-and-play method that can be integrated into existing MML pipelines.

The reviewers pointed out the paper’s main weaknesses, which are mianly in clarity and technical precision. For example, the illustrative examples (e.g., Fig. 1) are ambiguous, with insufficiently clear distinctions between sufficient and necessary features, and a lack of explicit multi-modal elements. Some mechanisms of network components are not properly clarified. 
Some key concepts, such as causal sufficiency, local invertibility, and non-exogeneity, are not clearly defined or aligned with standard causal inference terminology, which may cause confusion. Several important derivations lack sufficient explanation. 

The authors have addressed the main concerns in the rebuttal, and all reviewers have converged toward positive recommendations, with two accepts and one weak accept. AC agrees and recommends accepting this paper based on the consensus from reviewers. 

The authors are encouraged to revise the paper by clarifying the raised issues and incorporating relevant clarifications from the rebuttal.
Decision: Accept (poster)