Plug-and-play Feature Causality Decomposition for Multimodal Representation Learning
Authors: Ye Liu
Zihan Ji
Hongmin Cai
Venue: NeurIPS 2025 poster ()
Keywords: Representation Learning
Causal Inference
Multimodal Learning
Primary area: deep_learning

Abstract: Multimodal representation learning is critical for a wide range of applications, such as multimodal sentiment analysis. Current multimodal representation learning methods mainly focus on the multimodal alignment or fusion strategies, such that the complementary and consistent information among heterogeneous modalities can be fully explored. However, they mistakenly treat the uncertainty noise within each modality as the complementary information, failing to simultaneously leverage both consistent and complementary information while eliminating the aleatoric uncertainty within each modality. To address this issue, we propose a plug-and-play feature causality decomposition method for multimodal representation learning from causality perspective, which can be integrated into existing models with no affects on the original model structures. Specifically, to deal with the heterogeneity and consistency, according to whether it can be aligned with other modalities, the unimodal feature is first disentangled into two parts: modality-invariant (the synergistic information shared by all heterogeneous modalities) and modality-specific part. To deal with complementarity and uncertainty, the modality-specific part is further decomposed into unique and redundant features, where the redundant feature is removed and the unique feature is reserved based on the backdoor-adjustment. The effectiveness of noise removal is supported by causality theory. Finally, the task-related information, including both synergistic and unique components, is further fed to the original fusion module to obtain the final multimodal representations. Extensive experiments show the effectiveness of our proposed strategies.

Plug-and-play Feature Causality Decomposition for Multimodal Representation Learning
Ye Liu Zihan Hongmin Cai*
School of Future Technology
South China University of Technology
Guangzhou, China 511442
fyliu03, hmcai aiscut.edu..co ftjizihan&mail. scut. edu. cn
Abstract
Multimodal representation learning is critical for a wide range of applications, such as multimodal sentiment analysis. Current multimodal representation learning methods mainly focus on the multimodal alignment or fusion strategies, such that the complementary and consistent information among heterogeneous modali- ties can fully explored. However, they mistakenly treat the uncertainty noise within each modality as the complementary information, failing simultaneously leverage both consistent and complementary information while eliminating the aleatoric uncertainty within each modality. To address this issue, we propose plug-and-play feature causality decomposition method for multimodal represen- tation learning from causality perspective, which be integrated into existing models with no affects on the original model structures. Specifically, to deal with the heterogeneity and consistency, according whether can be aligned with other modalities, the unimodal feature is first disentangled into two parts: modality-invarian (the synergistic information shared by all heterogeneous modali- ties) and modality-specific part. To deal with complementarity and uncertainty, the modality-specine part further decomposed into unique and redundant features, where the redundant feature is removed and the unique feature is reserved based on the backdoor-adjustmnn causality theory. Finally, the task-related information, including both synergistic and unique components, further fed the original fusion module to obtain the final multimodal representations. Extensive experiments show the effectiveness of our proposed strategies.
The effectiveness of noise removal is supported by
Introduction
Multimodal representation learning is the basis of the downstream tasks, such as multimodal sentiment analysis [39; 13: 9; 15] and text-image classification [18; 11]. Due to the feature heterogeneity among different modalities [36], the consistent and complementary information among modalities [12] are crucial for effective multimodal representation learning- The consistent information refers to common features that present in multiple modalities, facilitating alignment of modalities within a common representation space. On the other hand, the complementary information refers to distinctive features presenting in only one modality but is still useful for the overall understanding, It enriches multimodal models, enabling them to learn more comprehensive and nuanced representations. Numerous existing methods have been proposed to effectively explore them. For example, [13] captures the consistent information via maximizing mutual information among different modalities, thereby improving multimodal fusion performances. [20] applies the contrastive learning to enhance The corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the feature representation capability of consistent information on both label-level and instance- [5] performs dynamic late fusion by predicting the generalization error upper bound of each modality, exploiting the heterogeneous complementary information in each modality.
However, due to the variations in data acquisition processes and sensor characteristics, multimodal data contain distinct types of uncertainty noise in each modality. We carry out qualitative analysis with the help of structural causal model (SCM) [25] to explain it. Figure 1 describes the SCM of multimodal data of a single modality. The consistent and complementary information are gathered from the task-related factor, from where the synergistic and unique features are to be constructed respectively. The synergistic feature is the shared consistent information among different modalities, which is aodality-invariant And the unique feature is the complemen- tary information that is modality-specifft However, due to the modality uncertainty noise factor resulted from the aforemen- tioned cues, the redundant task-agnostic information is also mixed in the modality-speciff feature, which would affect the construction of unique feature. For example, in image-text task, the task-related parts are the ones that contribute to the final task prediction. The synergistic parts are both included in the image and the text modalities, which jointly contribute to the final task prediction. The unique parts are the semantic con- tents that only appear in image modality (or text modality) and don't ion't appear in text modality (or image modality). Both synergistic and unique factors are task-related and derived from the task-related factor. The redundant part is the uncertainty noise within one modality, such as the typos in text modality or quality of image, which is derived from the noise factor. The multimodal data store these information in their different format (the specific factor). but they are still associated in high-level semantic (the synergistic factor). Existing methods often overlook this unimodal uncertainty noise, mistakenly treating it as complementary information, which hampers multimodal fusion performance [11]. Moreover, several researches have shown that removing uncertainty in unimodal features helps to obtain more accurate and robust multimodal representations. For example, [11] assumes the unimodal features are sampled from a multivariate Gaussian distribution and tries to remove the aleatoric uncertainty by estimating their latent distributions. [38] applies causal intervention on the amplitude information after the Fourier transformation of different modality samples, and enforces model on high-level semantic information. Nevertheless, these methods typically focus solely on consistent, complementary, or uncertain information, or combination of two of these aspects, with very few approaches effectively capturing all three types simultaneously.
T M
1/2
di-opensoor
Figure I: Structural causal model for the n-th multimodal data of the m-th modality. T: task-related factor. M: modality uncertainty noise factor. S. synergistic fea- ture. hT modality-specific infor- mation. u unique feature. r. mpeeaiit--ppecific task-agnostic in- formation. Yn task annotation.
Inspired by [24; 45], we propose plug-and and-play Feature Causality Decomposition (FCD) method to address the heterogeneity and alcatoric uncertainty of multimodal data. Ideally, the task-related information within each modality should be fully utilized to perform the multimodal representation learning, whilst the task-agnostic information should be eliminated. Thus, FCD is designed to take the original unimodal features as inputs and decompose them into their synergistic, unique and redundant components. To achieve this, FCD first employs Causality Components Decomposition (CCD) module on each unimodal feature to decompose it into the modality-invarian and modality- specific parts. The modality-invariant part plays a role of synergistic component, which is capable of capturing the consistent information shared across heterogeneous modalities. FCD employs another module named Synergistic Distribution Alignment (SDA) to narrow the space difference and align the synergistic component from all modalities via a purameter-sharing MLP constrained by Sinkhorn divergence [10]. Then, CCD further extracts the unique component from the modality part based on the teckdoor-adjustmee uncertainty noise mixed in the modality-specinee part can be removed via the backdoor-adjue cceuutor-aduuttmentt under some specific conditions. Moreover, contrastive loss among the redundant components from different modalities is utilized to constrain the modality-specific characteristic, effectively handling the uncertainties that vary across modalities. Different from existing multimodal learning methods [9; 13; 44; 45], FCD is plug-and-play that is more generalized and achieves the theoretical support from causality and probability perspectives. Extensive experiments on 9 existing multimodal methods and 5 multimodal datasets prove the effectiveness of FCD.
[25]. We theoretically [16; 25] prove that the unimodal

---

Visual Enc
Visual Enc
CCD
Fusion
Dec
bll
SDA
Fusion *
Dec
M
Prodiction
Prediction
Textual
Textual
Enc
Enc
CCD
Tood
Tout
FCD
(a) Intermediate Fusion Pipeline
(b) Intermediate Fusion Pipeline with FCD
Take image and text modalities as examples. "Enc" and "Dec" stand for encoder and prediction head, respectively. Arrows are the forward paths. (a) The original pipeline of multimodal intermediate fusion model. (b) The multimodal intermediate fusion pipeline involving FCD module, which takes unimodal features as inputs and outputs the synergistic (pink double-line shaft arrows). unique (green triple-line shaft arrows) and redundant (gray dashed arrows) components.
Figure 2:
2 Related Works
2.1 Multimodal Representation Learning
Multimodal representation learning has gained increasing attention with the rapid advancement of social media and multimedia technologies [31; 11]. Moreover, feature extraction plays a crucial role in multimodal representation learning, and numerous studies have been conducted. Recent efforts have explored various strategies for effective multimodal learning. [11] mitigate alcatoric uncertainty within each modality via latent distribution modeling, followed by dynamic integration and VIBebased fusion [1]. [20] performs token-level fusion with contrastive learning to align cross- modal representations. [13] reduce modality discrepancy via mutual information minimization and introduce CPC [28] to preserve modality-invan -invariatn features. [9] treat fusion as a progressive process and employ a hierarchical contrastive framework to retain semantic consistency across fusion stages. These works hardly considered the heterogeneity and aleatoric uncertainty of the multimodal data simultaneously. In addition these works, many works other tasks, such as multimodal learning with miss miss-modality [35], semantic segmentation [44] and multimodal domain generalization [8], focused the coddaity-speciffic-inn
features but ignored the unimodal uncertainty noise. Recently, [45] noticed the uncertainty noise within each modality and proposed to remove it through cross-attention mechanism. Unlike [45], provide simpler yet more generalized plug-and-play module whose effectiveness is also supported by causality and probability theory.
2.2 Causal Inference in Deep Learning
Causal inference [25] has attracted massive attention in multimodal tasks, where causality is defined as the relation between an event and the cause that gives rise to it [38]. It improves the robustness and generalization of models by debiasing the specific spurious correlation [6; 15; 23]. [23] used
the backdoor and frontdoor causal intervention to eliminate the noise and the spurious correlation between input multimodal data and the ground truth in visual question answering task. [6] proposed to remove the influence of word frequency in textual modality by backdoor-adjustmee they also argued that the authenticity of the news should not be inferred only based on image features, and proposed utilize the indirect effect of the image feature via counterfactual inference. (38] considered the modality-specin characteristic as the amplitude information of the Fourier transformation, and proposed to remove the modality noise by perturbing the amplitude component and generate a counterfactual sample to perform counterfactual inference.
Besides,
3 Preliminaries
Given a multimodal dataset 'D with N samples in M modalities, and each sample pair is denoted as (X yn), where X is the n-th data sample the m-th modality, yo is the label of the n-th sample. Yn E IR is real-valued number in regression tasks, while it is a discrete number, i.e., Yn E (1,...., K} with being the number of clusters classification tasks. In this paper, we specifically focus multimodal intermediate fusion model, example with visual and textual
3

---

modalities is shown in Figure 2 (a), Multimodal intermediate fusion usually involves three steps, which can be formulated as follows.
In O (z.: (Doc): Zn = FFusc (Z .2n FFsss E R
s.t. n, et (X,:; O E 18d" {1, M
where (:; Enc) is the encoder for the m-th modality with parameter O Fruse(:: Oruee is the fusion function with parameters Puse. and (De)) the predictive function with parameters Dpec The optimization function of the regression and classification tasks is usually represented as:
N
Regression tasks: L = Leask Lree
Mal + Lreg
NV
(2)
K
Classification tasks: L Lrssk + Liee
1y =k log Iin + Luee
where the indicator function uy = 1 if and only y,, equals to k. otherwise =y 0. Dn the prediction of the n-th sample. Vnk is the predicted probability that the n-th sample belongs to the k-th category, and feg the regularization term. Different freg employed in different existing works. For example, series of contrastive learning loss terms at different fusion phases are proposed in [9].
4 Methodology
The overview of our proposed method, Feature Causality Decomposition (FCD), is shown in Figure (b). FCD is plug-and-play module, which can be integrated into any intermediate fusion model for multimodal learning defined in Section 3. It is located between the unimodal encoder and the original fusion module, which takes the unimodal features {2." J features!
m== input and feeds the processed
{ AM mm to the original fusion module.
FCD contains two main submodules for each modality, i... Causality Components Decomposition (CCD) and Synergistic Distribution Alignment (SDA). CCD first separates the unimodal feature z into three aforementioned components, one of which can be aligned with the same part from other modalities (i.e., the synergistic component, pink double-line shaft arrows in Figure 2 (b), denoted as s ). The modality-seciffi task-related unique component is illustrated green triple-line shaft arrows in Figure 2 (b) and denoted as u The last one is the redundant component (the gray dashed arrows in Figure 2 (b), denoted r, which is the modality uncertainty noise. Then, s is aligned by the SDA module. Finally, FCD aggregates the two tesk-related component, i.e., synergistic and unique features, and feeds the aggregated feature 2n the original fusion module F
4.1 Causality Components Decomposition Module
The illustration of CCD is shown in Figure 3. Inspired by [45], CCD first disentangles z,!" in Equation (I) to the modality-speciin part h and the modality-invarian part s via two MLPs T and F.m.
h = Fil (z. OK"), S F.th (2 0
(3)
where 8ThH and ATh are parameters of the F Rd
- d and F Rdd - Rdd respectively.
As discussed in Section 1, current methods may mis- takenly treat features with uncertainty noise the part of modality-specific information h In other words, h contains both mpdelity-specifii task-related com- ponent n. (complementary information) and redun- dant component rT. Therefore, the core problem is how to reserve u and eliminate r from the hm simultaneously. From the causality perspective, n is confounded by h. As shown in Figure 1, there exists a backdoor path between T and yo, which is confounded by the modality-spciffe feature hm u + h -> r/ + yu- To eliminate the causal effect of the redundant component r in the backdoor
n
MLP
URD
Mi
MLP
IaNoNCE
r."0"
Figure 3: The illustration of the pro- posed Causality Components Decomposi- tion (CCD) module.

---

path, backdoor-adjuee adjjutmeent is commonly used [25; 23; 15], which is usually formed as the do(-). It performs causal intervention on the confounder factor to remove its spurious causal effect. In CCD, bockdoor--djjstmees is also employed to remove the causal effect of the confounder in backdoor path.
Theorem 4.1. Given be extracted by Fi". the conditional mutual information (CMI) between u. and Yn defined as [(, yn (n)h))) Assuming u, is extracted from hT by a measure- measure-preserrvng bijective function Fm then maximizing the expectation of conditional mutual information for all possible hn iee., Ep(by) (T( Yn [h))), ]. is equivalent to maximizing the mutual information [do(u yn), where do- op-rraaaoo do(.) ) the employed backdoor-adee aadiutore,,dssmmmnt, E is the expectation, and P is the probability distribution function.
The proof [16; 25; 21] Theorem 4.1 can found Appendix Section A. Theorem 4.1 indicates that if the extraction function F is measure-presservng bijective, then effectively extracting the task- related feature :, from the modality-speciffi feature h while separating out the modality noise r. via backdoor adjustment equivalent maximizing the mutual information between the annotation and u after causal intervention. When u,," is extracted from h by Fmb. i.e., u, Fmb (h,"). CCD is able to extract u from h by optimizing the following:
M
min LCMI
def.
Ep (Pann)[[(uT" [T(,,' Un [!
NM
12
(4)
log P(u yo (h
T(u Un (h)) Pppug bgr)
P(ug her P(yn h
However, the true distribution of h, is not assessable, which impedes the calculation of Equation (4). More importantly, directly optimizing Equation (4) without adjustment ignores the existing backdoor path, which would spuriously bring task-agnostic information towards annotation prediction. Based on Theorem 4.1, we can transform Equation (4) to Equation (5) when Fm a measure presurrvin bijective function and bockdoor-adjustmen is applied to ensure the mixed uncertainty noise can be removed.
M
min LMi
[do(u Un)
NM
n=1
(5)
M
P(do(u?) Un)
Σ
E
B (ww) log P(do(u?)) P(
NM
m=11
According to [28; 22; 9]. CCD applies InfoNCE loss as its lower bound in practice. Hence, we can transform Equation (5) to Equation (6) to perform the mackdoor-adjustmen and reserve deconfounded modality-specific task-related feature (o
1 NM
LM RJ LInfONCE =
LInfONCE yn)
#= m=1
(6)
N M
ci /n)/T
cos( m Sn//T
Σ
Σ
log
gcos (r /// + log
2NM
E cas i Soa/T
m=1m=l
where in E Rd and yn E Rd are linearly projected from do(um)) and yn. with normalization, respectively. is the temperature parameter. And cos(.,)) is the cosine between two vectors.
On the basis of Theorem 4.1, effective extraction of u from h whilst separating r apart requires that the extraction function Fm is measure-presse ppessurr-r bijective function. According to [46; 45], we propose binary disentangle module Unique Redundant Decompose (URD), which is guaranteed to satisfy the measure-preservine bijective condition.
3 W (k)
u
Im
URD-
m h
U
s.t. u1 n(k)
FURD-a (h") (()) (RRDa))
where the scaler wi 12 is linearly projected from the dimensional concatenation of T n(k)
and hm
((k)
Fm a
dd pdd is MLP for decomposition with parameter 0
Fi
FURD. R d
+ g is
5

---

a feed foward net with parameter URD- and n(k) is recursively deduced as:
h k
h (R))
(8)
h
Σ
u ((1)* h:
Practically, we apply SVD on the weight matrices of F URD- and F where the 0 diagonal elements of the singular value matrix Σ are set 10 The pseudo code is in Appendix Section B.
On the other hand, the modality-specific fask-agnostic component r, should have significant dif- ference with rm (m' * m) to constrain the modality discrimination of h To achieve it, we employ InfoNCE [28] to perform contrastive learning between the redundant features from different modalities of the same sample an unsupervised optimization target.
N
M
C cos cos (P F2///
def.
min Dis
log
NM
n=1 1==1
mT
where Fm E Rd is linearly projected from r with normalization. By applying LMi and LDs. the extraction codaalit--specffee feature h mixed with unique and redundant features constrained. With the optimizing of LMI and DDs. the unimodal uncertainty noise mixed in the poddllty-sppccinn feature can be effectively removed.
4.2 Synergistic Distribution Alignment Module
In order to ensure that the synergistic component sh captures the shared and consistent knowledge across modalities, Synergistic Distribution Alignment (SDA) module is proposed perform the feature space align- ment, which is illustrated in Figure 4.
8
Fa
Layer Noem
2
To achieve the alignment, SDA first linearly projects S TRd to subspace with d dimension shared by all modalities:
Jun
Layer
ST proj (s gm E Rdd
(10)
m = (1,..,,M s.t.
Figure The illustration of Synergistic
where 0mg is the parameter of the m-th modality pro- jection function proj _Rd + Idd With all modalities in the same dimension, 8 of each modality able to
Distribution Alignment (SDA) module.
propagate forward to the parameter-sharin MLP FSDA Rd + Rd to achieve the space alignment:
( OSDA) s.t. (1..... M)
(11)
where OSDA is the parameter of MLP FSDA. To preserve the original model's structure and dimension- ality, SDA linearly projects h back to the original space
P (, O"proj) s.t. m (I,...,. M)
(12)
where 6Pppeoi is the parameter of projection layer Fm r-proj Rdd > at in the m-th modality. To reduce the impact of back propagation on the gradient of model and the original semantic information shift[14, 14; 33], we apply shortcut between S and a with layer normalization:
2 LayerNorm(S," + + s.t. m = (1. M
(13)
The key target of SDA module is to effectively constrain the extracted S/ contains the shared information among different modalities. However, the true distributions of s are unreachable. Although there have been works assuming them to be multivariate Gaussian distributions and trying to meet this assumption by applying Kullback-Leibler (KL) divergence [11; 19], [2] has pointed out the poor stability of KL-divergence for gradient descent. Inspired by [10; 26; 17], we employ the
Sinkhorn-divergene defined in Equation (15), a variant of the optimal transport distance, to measure the difference between the distribution of latent embeddings from two modalities.

---

Definition 4.1 (Bi-modalities Batch Discrete Probability), Given a batch size No. the latent feature vectors in the my -1 and my-th modalities are {5, Ji=l and { ز== respectively. Let AN. denotes the probability simplex of R and (ai,".. .aN.] E AN., b b= (b, lbi,... bN..] E A N.. de; refers to point mass located at coordinate s E R2 We have the bi-modalities batch discrete probability:
arogmi 3=
by Oame
s.t. m1, m2 (1..... M3 and my + m2
(14)
Based on Definition
1, according to [10; 26; 17], the Sinkhorn-divergence based term is defined:
min LSDA
def.
2
Σ
Laiv (m11, m2)
M(M-1)
(15)
s.t. Laiv (m1, m2) OT(a, 3) 0.5(OT(a, + (T(,
Here, OT(,,)) ) is the total optimal transport cost between two distributions solved by the regular optimal transport defined as (a, 33) min < T, M >r, where M E N *N.NN, is the cost matrix with each element calculated with cosine similarity, i.e., M.j
cos( 3 3 ... >F is the Frobenius dot- dot-product. The constrain II B) = {T R Σ Tūj b3. Σ, TT, = a enforces to have a, B as its marginals. T the transport plan matrix to be optimized by Sinkhorn algorithm [7].
By minimizing Equation (15), the shared and consistent information across modalities can be retained. Finally, the two relktee components u and 2,, are aggregated by:
z m u
(16)
4.3 Training Loss Function
To sum up, the optimization target of our proposed FCD can be formed as:
CTCD AI XzCDis
AgLSDA
(17)
By combining the original loss function (Equation (2)) and Equation (17), the total loss of an effective multimodal representation learning model with plug-and-pla FCD without changing model structure is:
Loverall = + LICD
Lins + freg AI + AzLDis + Aa
(18)
Original Loss
5 Experiments
5.1 Datasets and Evaluation Metrics
Datasets. We design and conduct our experiments on 5 widely used multimodal datasets, i.e. CMU- MOSI [41], CMU-MOSEI [42], MSVA-Single [27], UPMC-Food101 [3], and HFM [4]. Please refer to Appendix Section C for more descriptions about these datasets.
Evaluation Metrics. Following [9], report experimental results CMU-MOSI and CMU-MOSEI datasets with the mean absolute error (MAE), Pearson correlation (Corr), binary classification accuracy (Acc-2) and weighted Fl score (FI). Following [11], report the results on the remaining datasets with accuracy (Acc) and weighted Fl score (F1). Please refer to Appendix Section C for more details about these metrics.
5.2 Experimental Settings and Implementation Details
Our experiments are conducted on NVIDIA RTX 4090 24GB GPUs with PyTorch [29] framework. We control the system environment for all experiments to be consistent and reproduced all the base comparison experiments using the hyper-parameters reported their original papers. Please refer to Appendix for more experiments, including sensitive analysis and computational overhead analysis.
7

---

Table 1: Quantitative results of SOTA methods. The left side of "/" in Acc-2 and F1 is computed as "negative/non negative moon-exclude 0)" and the right side is computed as "negative/positive (exclude 0)". All results scaled (x100). Better results in each compared method are highlighted in bold.
DATASETS
CMU-MOSI
CMU-MOSEI
METHODS
MAE
CORRT
AcC-21
F11
MAEJ
CORRT
ACC-2
FI
BASE
72.51
79.44
82.80/84.8
$2.78/84.33
54.04
75.65
83.26/85,
83.40/84.
SELF-MM MM
OURS
68.29
80.62
84.95/87.09
84.93/87.11
52.49
76.98
85. 85.19/86.08
85.12/85.77
BASE
73.81
78.17
83. 83.24/85 06
83.21/85.00 09
55.34
75.18
82. 82.14/84,76 76
82, 62.49/84, 666
MMIM
80.91
84.55/85.98
84.56/86.03
54.41
76.83
83.97/85.72
83,98/85.74
70.05
79.59
83.97/86.11
83.75/85
54,21
76.68
80 8..883/84.9
81.40/84.00 9666
MCL-MCF
OURS
69.29
80.29
84.99/87.00 .04
84.84/86,90
53.68
77.82
84.40/85.55 553
84.41/85.27
BASE
72,80
79.57
8.3. 131.67/85.00 06
73.61/85.04
$3.96
75.99
82, 98/8844/40
$3.09/844.1
ATCAF
OURS
69.65
80.32
1/86.13
84.01/86.09
53.13
77.38
84.78/85.55
84.73/85.22
85. $5.75/88.06
51.95
80.74
$5.46/87.00 0
85, 15..9/86.88
MMML
OURS
59.86
87.55
88,48/90.24
88.44/90.23
50.01
81.78
86.63/88.08
86.63/87.99
The main hyper- apper-parameten, in this paper consist of two parts, i.e., the typer-parametent in each multi-modal intermediate fusion method (e.g., learning rate, batch size) and the ones of FCD (A1, X2 As). The hyper-parameet Section E, and please refer to the original papers for other ayper-parameten of each base model. Moreover, we report the computational overhead and complexity analysis in Appendix Section F. We fix the temperature parameter 7 0.07 in InfoNCE loss for the most common cases.
of FCD and their sensitive analysis are summarized in Appendix
5.3 Quantitative Studies
To evaluate the effectiveness of our proposed FCD, we select 5 recently proposed state-of-the-art
(SOTA) multimodal sentiment analysis methods as our evaluation baseline on CMU-MOSI and
CMU-MOSEI datasets, i.e., Self-MM [39], MMIM [13], MCL-MCF [9], AICAF [15], and MMML [37]. And multimodal fusion methods on the remaining datasets are compared: MMBT [18]. CLMLF [20]. MVCN [36]. and URMF [11]. All these methods are intermediate fusion methods and please refer Appendix Section for more details.
The results on CMU-MOSI and CMU-MOSEI datasets are shown in Table and the results
on the other 3 datasets are reported in Table 2. The Base experiments are conducted on the corresponding datasets using the official imple- mentations and the hyper-parameters reported in their papers. Then we add FCD to each method and reproduce them again with the orig- inal
Table 2: Quantitative results of SOTA methods on MVSA-Single (MVSA-S), 1P (Food101), and HFM datasets. All results are scaled (x100), Better results in each compared method are highlighted in bold.
fixed, which are repre- sented by "Ours". By doing so, we are able to control the uniqueness that affects the experi- mental results and improve their credibility.
DATASETS
MVSA-S
FOOD101
HFM
METHODS
AcC
Acc
F1
Acc
F1
BASE 74.76
74.76
91.22 91.22
MMBT OURS 76.30
76.30
91.72
91.72
From Table and Table 2, we can find that
BASE 70.67
69.03
84.77
84.76
our proposed FCD is able to improve the per- formance among various evaluation metrics on
CLMLF OURS 72.22
71.62
85.55
85.52
BASE 72.44
71.25
84.89
84,90
all tested regression tasks and classification tasks. This indicates that the heterogeneity and aleatoric uncertainty of the multimodal data should be considered simultaneously to generate valid multimodal representation. Specifically, MMML achieves the best performances on both CMU-MOSI and CMU-MOSEI datasets after ap- plying FCD, which may caused by the superior behavior and complexity of MMML. has the
MVCN
OURS 75.33
74.78
85.31
85.32
BASE 72.25
71.11
92.27
92.26
URMF
OURS 74.57
74.22
92.94
92.96
longest training overhead per epoch (see Appendix Section F). On the other hand, MMBT has the highest Ace and F/ on MVSA-S dataset. URMF performs better than MMBT on Food101 dataset on both Ace and Fl metrics. CLMLF shows the better performance than MVCN on HFM dataset
8

---

on both tested metrics. The improvements of methods are various. This may be caused by the different unimodal encoding approaches, which directly determines the quality and semantic richness of unimodal features and thereafter affect the effectiveness of FCD.
There exists significant difference with other methods that Self-MM considered both consistent and complementary information in multimodal data. It took the heterogeneity of multimodal feature space into consideration and proposed to shift annotations in their unimodal feature space. This proves that the consistent and complementary information within multimodal data should be effectively excavated at the same time, which also coincides with our idea.
Besides, compared with other methods, the Corr of MCL-MCF with FCD has less improvement on both CMU-MOSI and CMU-MOSED datasets. This may be caused by the assumption of MCL-MCF where the fusion is considered as progressive process. In this situation, the consistent information plays the dominant role and the aggregated complementary information is gradually weakened the fusion process progresses.
5.4 Ablation Studies
To evaluate each component used in FCD, ablation studies are con- ducted. Specifically, we conduct ablation experiments on term fMI LDss and LSDA by setting A1 = 0, X = and Às = O, respectively. Besides, we also evaluate the effectiveness of in- volving unique component T Equation (16), where only syn- ergistic feature 2TTh is used for fusion. And the shortcut Equa- tion (13), where 8 directly fused with u.. The results can be seen in Table and the results of Base and Ours are also shown in the first line and the last line, respectively.
Table Ablation studies results on CMU-MOSI dataset with Self-MM. The ablation terms are marked as "V" if they are not ablated, otherwise they are marked as "_"
Cxn Cas CADA EQ. (16) EQ. (13) MAE CORR ACC-2
F1
72.51
82.80/84.30
82.78/84.33
70.76
79.65 82,94/84.91
82.85/84.88
/
68.83
80.21
84.42/86.53
84.39/86.19
8
/
v
68.66
80.47 83, 33.38/85.98
83.15/85.8 84
69.45
80.33
83.24/84.91
83.19/84.91
84.02/86.13
83.94/86.07
68.29
80.62
84.95/87.09
84.93/87.11
When λ 0, the mutual information loss between u and yn. would not be added the total loss function. In other words, the backdoor-adjie adjaasmuuunttttt is not applied to effectively decompose the redundant feature r and the unique feature u." The unimodal uncertainty noise may be still mixed in the complementary information. Meanwhile, since the unique feature is modality sopalitt-specifiic, the task-related information may be also mixed in r resulting in the worst performance in all ablation cases. This case proves that CCD can extract task-related information from the mixed features.
When A2 0, the discriminative information between r is neglected. In this case, h extracted by Fn may be not modality-speciiff In this case, the complementary information may not be fully exploited, resulting task-related information loss.
When Aq = 0, there is no constrains on the synergistic information alignment. It means there is no supervision the aligned feature space, which means the features are probably not modality- -nnvaiiat and contains moodllitysspecinn information.
When removing the unique component i Equation (16), i.e., 2. = 2m compared with the full FCD, the performance worse due to the complementary information loss. Therefore, the unique component is significant for multimodal representation learning.
When the shortcut between s and s is removed, the gradient information and original semantic knowledge may be lost. Thus, the shortcut path is able to retain the original information.
5.5
Case Study
To validate the corresponding parts of task-related components St and u in the original data and the interpretability of FCD, we employ Grad-CAM [30] to visualize the decomposition results of CLMLF on MVSA-S dataset. The results is shown in Figure

---

From Figure 5, we can find that there is a correlated semantic similarity shared by the synergistic components of differ- ent modalities (e.g., building collapse). On the other hand, the scattered bricks on the ground from visual modality and "fire" from textual modality provide the unique information, which related to
Visual
Original
Synergistiic
Unique
Rodge
Avemin
is
closed
Ridge Averta
elosed
the annotation Negative. This indicates that FCD is able to capture the synergis- tic and unique information. Besides, for both modalíties, there may exist impor- tance overlap between synergistic and unique information, such as the word "building" is both more important in tex- tual modality and both more attentions paid on the buildings in the distance of the visual modality. For synergistic information, focuses on the semantic consistency to construct the semantic cor- respondence between different modali-
after
partial
building
after
partial
building
Textual
collapse
collupse
electrical
Saturday
night
Saturday
Syncristic
Unique
Amnotation: Negative
Figure 5: The Grad-CAM of task-related components s and u extracted by CLMLF trained on MVSA-S. The upper part of the figure is the original, synergistic and unique data of visual modality. And the lower half of the figure is the synergistic and unique data of textual modality.
ties. For unique information, it focuses on the unimodal contextual information learned from the whole dataset. Taking the "building" in textual modality as an example, it may experience more concurrent together with negative words like "disasters" or "collapse". This also proves the necessity of simultaneously excavating the consistency and complementary information.
5.6 Sensitivity of Batch Size
Since the LMI and LpS loss functions used in the FCD are essentially contrastive learning, whose ef- fectiveness may be affected by the number of nega- tive samples provided by the batch size, we validate the sensitivity of batch size. We conduct this exper- iment on Self-MM equipped with FCD on CMU- MOSI dataset. Practically, the batch size doubles from to 128.
Table 4: The sensitivity of batch size Self-MM (with FCD) on CMU-MOSI dataset.
BATCH SIZE
MAE
CORR
AcC-2
F1
67.83
80.34
14/74/86.96
84,74/87.0
16
68.02
79.86
84.43/86.27
84.48/86.31
32
68.29
80.62
84.95/87.09
84.93/87.11
The results are shown Table 4. The results show that as the batch size increases, the model achieves consistently better performance across various met- rics. One possible reason is that larger batch pro-
68.11
80.27
15/18/87.41
85. 16/87/87
85.74/87.85
85.71/87.83
vides a wider range of negative samples for the contrastive loss, which makes the optimization of Lyu and LDss more effective. With more diverse negatives, the model can better separate similar but distinct representations, leading to stronger feature discrimination and better overall performance.
Conclusion
In this paper, we propose a plug-and-play module, Feature Causality Decomposition (FCD), to solve the existence of heterogeneity and alcatoric uncertainty within multimodal data by decomposing the unimodal feature into its synergistic, unique and redundant components from causality perspective. Based on whether it can be aligned with other modalities, FCD first uses Causality Components Decomposition (CCD) module to disentangle the unimodal feature into two parts: modallty-speciiff and modality-invanl invariant components, which contains the synergistic information shared by various heterogeneous modalities. Then fackdoor-adjustmee and retain the task-related component in the modality-speciffc part, it is optimized by maximizing the mutual information between the unique component and the annotation. Besides, the Sinkhorn divergence is employed to narrow the difference of synergistic embeddings among modalities. Extensive experiments prove the effectiveness of FCD.
is applied to remove the redundant information
10

---

Acknowledgements
This work partly supported by the National Science and Technology Major Project (0660))) the National Natural Science Foundation of China 022306118, 62325204, U2IA20520), the Fundamental Research Funds for the Central Universities (2025ZYGZZR054) Innovation Fund (2023RY03)..
China University
References
Alemi, A.A., Fischer, I,, Dillon, J.V., Murphy, K.: Deep variational information bottleneck. In: International Conference on Learning Representations (2017), https::/pppeereeiiiie www..eeeeeo.nneeete.eeee.. HyytuiiBeeeeyeeeeeggeeee
[2] Arjovsky, M., Chintala, S., Bottou, Wasserstein generative adversarial networks. In: International conference on machine learning. pp. 214-223. PMLR (2017)
[3] Bossard, L., Guillaumin, M.. Van Gool, L.: Food-101-mintle discriminative components with random forests. In: Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13. pp. 446-461 Springer (2014) 7
[4] Cai, Y., Cai, H., Wan, X.: Multi- modal sarcasm detection in twitter with hierarchical fusion model. In: Proceedings of the 57th annual meeting the association for computational linguisties, pp. 2506-25111 (2019)
[5] Cao, B., Xia, Y., Ding, Y., Zhang, C., Hu, Q:: Predictive dynamic fusion. In: Forty-first International Conference on Machine Learning (2024), https: http:////eeeeeeeeeeeeeeeeeeeeeee net net//Eorumiid-LLy@@rrrrr 2, 27
[6] Chen, Z., Hu, L., Li, W., Shao, Y., Nie, L.: Causal intervention and counterfactual reasoning for multi-modal fake news detection. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 627 627-638 (2023) 3
[7] Cuturi, M.: Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural informa- tion processing systems 26 (2013)
[8] Dong, H., Nejjar. L., Sun, H., Chatzi, E., Fink, Simmmdg: A simple and effective framework for multi-modal domain generalization. Advances in Neural Information Processing Systems 36, 78674-78695 (2023) 3
[9] Fan, C.. Zhu, K., Tao, J., Yi, G., Xue, J., Lv. Z.: Multi-level contrastive learning: Hierarchical alleviation of heterogeneity in multimodal sentiment analysis. IEEE Transactions Affective Computing (2024) 1.
3, 4, 5. 7, 23, 24
[10] Feydy, J.. Séjouréé, T., Vialard, F.X., Amari, S.i., Trouvé, A., Peyré, G.: Interpolating between optimal transport mmd using sinkhorn divergences. In: The 22nd International Conference on Artificial Intelligence and Statisties. pp. 2681-2690. PMI.R (2019) 2, 6
[11] Gao, Z., Jiang, X., Xu, X., Shen, F., Li, Y., Shen, H.T.: Embracing unimodal alcatoric uncertainty for robust multimodal fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition. pp. 26876-26885 (2024) 1, 2, 3, 6, 7, 8, 23, 24,
[12] Guan, W., Wen, H., Song, X., Yeh, C.H., Chang. X., Nie, L.: Multimodal compatibility modeling via exploring the consistent and complementary correlations. In: Proceedings of the 29th ACM international conference on multimedia. pp. 2299-23900 2307 (2021)
[13] Han, W., Chen, H., Poria, Improving multimodal fusion with hierarchical mutual information maxi- mization for multimodal sentiment analysis. In: Moens, M.F., Huang, X., Specia, L., Yih, S. S.t.t. (eds.) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9180- 9192. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). htips://doi.org/100 723/ 1, 2, 3, 8, 23,
http:/02...mmpmmmmi........ https: : //aaaaaaaaanoooyyyy.eoooo org /20212221122 emnlp-main
[14] He. K., Zhang, X., Ren, S., Sun, Deep residual learning image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016)
[15] Huang, C., Chen, J., Huang, Q. Wang, S. Tu. Y., Huang. X.: Atcaf: Attention based causality-awarc fusion network for multimodal sentiment analysis. Information Fusion 114, 102725 (2025) 1, 3, 5, 8, 23, 24

---

[16] Jaynes, E.T.: Probability theory: The logic of science. Cambridge university press (2003) 2, 5,
[17] Ji, 7., Tian, X., Liu, Y:: Affakt: hierarchical optimal transport based method for affective facial knowledge transfer in video deception detection. Proceedings of the AAAT Conference on Artificial Intelligence 39(2), 1336-1344 (Apr 2025). http:://oo.org/11... 1699/aaii../ 1609/aaii.33912.32222 https://ojs...... aaai. wtg/index.php/AAAe ptp/AAAI/ wttet/icceeeeeeee/e/e/eeee 6,7
[18] Kiela, D., Bhooshan, S., Firooz, H., Perez, E., Testuggine, D.: Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv:1909.0299) (2019) 1, 8, 24
[19] Kingma, D.P., Welling, M.: Auto-encoding variational bayes (2022), https: http://wxxvvvv...aaaa./ 6114 6
[20] Li, Z., Xu, B., Zhu, C., Zhao, T.: CLMLFa contrastive learning and multi-layer fusion method for multimodal sentiment detection. In: Carpuat, M., de Marneffe, M.C., Meza Ruiz, L. (eds.) Findings of the Association for Computational Linguisties: NAACL 2022. pp. Linguistics, Seattle, United States (Jul 2022). hups://doi.org 10. 18653/v 12022..inding---aael.... 175, https: //aclanthology.oo wwg.//loooooooooooooooeee 2022. findings- naacl. 175/ 1, 3, 24
Association for Computational
[21] Lin, R., Hu, H.: Multi-task momentum distillation for multimodal sentiment analysis. IEEE Transactions Affective Computing (2023) 5
[22] Liu, N., Wang, X., Wu, L.. Chen, Y., Guo, X., Shi, C.: Compact graph structure learning via mutual information compression. In: Proceedings of the ACM web conference 2022. pp. 1601-1610 (2022) 5
[23] Liu, Y., Li, G., Lin, L.: Cross-modal causal relational reasoning event-level visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(10). 11624-1166 (2023) 3,5
[24] Martínez-Sánetee Á., Arranz, G., Lozano- A.: Decomposing causality into its synergistic, unique, and redundant components. Nature Communications 15(1), 9296 (2024) 2
[25] Neuberg, L.G.: Causality: models, reasoning. and inference, by judea pearl, cambridge university press,
2000. Econometric Theory 19)), 675-685 (2003) 2, 3,5,5, 22
[26] Nguyen, T.T., Luu, A.T.: Improving neural cross lingual abstractive summarization via employing optimal transport distance for knowledge distillation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 11103-1 1111--1111111111 (2022) 6, 7
[27] Niu, T., Zhu, S., Pang, L.., El Saddik, A.: Sentiment analysis on multi-view social data. In: MultiMedia Modeling: 22nd International Conference, MMM 2016, Miami, FL, USA, January 4-6, 2016, Proceedings, Part 11 22. pp. 15 27. Springer (2016) 24
[28] Oord, v.v.d. Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.0377 (2018) 3, 5,6
[29] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradhury, J., Chanan, G., Killeen, T., Lin, 7.,, Gimelshein, N., Antiga, L., Desmaison, A.. Kopf, A., Yang. E., DeVito, Z., Raison, M., Tejani. A., Chilamkurthy, S.. Steiner, B., Fang. L.. Bai, J., Chintala, library. In: Wallach, H. Larochelle, H., Beygelzimer. A., d'Alché- Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32. pp. 8024-8035. Curran Associates, Inc. (2019) 7
Pytorch: An imperative style, higg--peformance deep learning
[30] Selvaraju. R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra. D.: Grad-cam: Visual explanations
from deep networks via gradient- based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618-626 (2017)* 9
[31] Tian, H., Tao, Y., Pouyanfar, S., Chen, S.C., Shyu, M.I.:: Multimodal deep representation learning for video classification. World Wide Web 22, 1325- 1341 (2019) 3
[32] Tsai, Y.H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.P., Salakhutdinov, R.: Multimodal trans- former for unaligned multimodal language sequences. In: Proceedings the conference. Association for computational linguistics. Meeting. vol. 2019, p. 6558. NIH Public Access (2019) 24
[33] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, Jones, L., Gomez, A.N., Kaiser, ł.,, Polosukhin, L.: Attention all you need. Advances in neural information processing systems 30 (2017)
[34] Walters, P.: An introduction to ergodic theory, vol. 79. Springer Science & Business Media (2000) 23
[35] Wang, H., Chen, Y., Ma. C., Avery, J., Hull, L., Carneiro, G.: Multi-modal learning with missing modality via chared-speciffe feature modelling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 15878 15887 (June 2023) 3
12

---

[36] Wei. Y.. Yuan, S., Yang, R.. Shen, L.. Li, Z., Wang, L., Chen, M.: Tackling modality heterogeneity with multi-view calibration network for multimodal sentiment detection. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 5240-5252 (2023)
1, 8, 25
[37] Wu, Z., Gong, Z.. Koo, J., Hirschberg. Multimodal multi-loss fusion network for sentiment anal- ysis. In: Duh, K., Gomez, H.. Bethard, S. (eds.) Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 3588- 3602. Association for Computational Linguisties, Mexico City, Mex- ico (Jun 2024). https://doi.co http///..r///..... 0rg/10..8633/11/202222 16653/11/022.mmaccooong 197, https: /atpaa:///aaaaaaaaaaaaaeeaaaaaaa aalaaaaaaatoooyyyyyoooooo wwwllooyoooooooa/eeeeee org/2024 naacl long. 197/ 8, 24
[38] Yan, J., Deng, C., Huang, H., Liu, W.: Causality-ivvarian interactive mining for cross-modal similarity learning. IERE Transactions on Pattern Analysis and Machine Intelligence (2024) 2, 3
[39] Yu. W., Xu, H., Yuan, Z., Wu, J.: Learning coddality-specife representations with supe-supe multi- task learning for multimodal sentiment analysis. In: Proceedings of the AAAI conference on artificial intelligence. vol. 35, pp. 10790-10797 (2021) 1, 8, 24
[40] Zadeh, A., Chen, M., Poria, S., Cambria, E., Moreney, L.P.: Tensor fusion network for multimodal sentiment analysis. arXiv preprint 1nXiv:1707.077: (2017) 24
[41] Zadeh, A., Zellers, R., Pincus, E., Moreney, I.P: Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint urXiv:1606.06255 (2016) 7
[42] Zadeh, A.B., Liang, P.P., Poria, S., Cambria, E., Morency, L.P.: Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 2236-2246 (2018) 7
[43] Zhang, Q., Wu, H., Zhang, C., Hu, Q., Fu, H., Zhou, J.T., Peng, X.: Provable dynamic fusion for low-quality multimodal data. In: International conference machine Icarning. pp. 41753-41769, PMLR (2023) 23,
27
[44] Zheng, X., Lyu, Y., Wang, L.: Learning modality-agnostic representation for semantic segmentation from
any modalities. In: European Conference Computer Vision. pp. 146-165. Springer (2024) 2,
[45] Zhou, Y., Liang, X., Chen, H., Zhao, Y., Chen, X., Yu, L.: Triple disentangled representation lcarning for multimodal affective analysis. Information Fusion 114, 102663 (2025) 2, 3, 5,
[46] Zou, X., Yan, Y., Xue, J.H., Chen, S., Wang, H.: Leamm-to-dccompose cascaded decomposition network for cross-domain few-shot facial expression recognition. In: European Conference Computer Vision. pp. 683-700. Springer (2022)*
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims are made in the abstract and introduction accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Proof' are in appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We will make the code public after acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We will make the code public after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececccccccccct http://///iiieeeeeeeuuuuiiiiiiii ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental details are given.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The experimental results don't contain error bars etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We give the computational platform.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerr..p.eeeeee cc cccpublic//eeeccccccecccccceeeea www.ehhiiiiieeeeeeeeeeeeeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't have these contents.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the works in our paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't provide new assets with human.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https htt////eerpppp.pe http://aaap..c///oeeeeeeeeeeeeee 22/ps///ee.oeeeeeeee/22///////// for what should or should not be described.
20

---

A Proof of Theorem 4.1
We first give the following lemma, which describes the probability distribution between two variables X and Y.
Lemma A.1. For two variables X, defined on the measurable seis X, y with the same measure, and the probability distributions are P(X), P(Y). If = F(X), where FF is peasure-preserring bijective funcion, we have P(X) = P(Y).
Proof. Since F is measure-pressrrving bijective function, we have: X) P(X E X) = P(F x)= P(Y e F(X)) = P(Y
(19)
Corollary A.1. For two variables X, Y from measurable sets X, y with their measures vxx) and m())) Yf F(X) and F is measure--resserving bijective function, there exists:
P(X,Y) P(X))(Y - P(Y))) - F(X))
(20)
when vx(2)= μd))).
where 8(t) is the standard Dirac delta function which has the following properties for E
1. 8(t) 0 for all t + 0
2. _ 8(t)dt 1.
Proof. Based on Bayes' theorem, we have:
= P(X))((X))
(21)
Since F bijective function, Y is uniquely determined by X, according to [16], P(Y]X)
5(Y F(X)). Based on Lemma A.I, we have:
P P(X) P(X)8(Y F(X)) P(Y)8(Y F(X))
(22)
Corollary A.2. For three variables X, Y and Z defined on the measurable sets X, and Z, and the joint probability distributions are P(X, Z). P(Y. Z). I = =((),, where F is a measure- preserving bijective function, X is independent with Z, and is also independent with Z, we have P(X, 7) = P(Y, Z).
Proof. Let p(X) is the measure X. Since = =(X), where function, then X' and ) have the same measure, i.e., m(x²) (())). Besides, the measures of the joint domains X' Z, y can formed as μ(('' x 2) and μ()² x 2). respectively. Based on the Fubini's 's Theorem, we have:
measure-preserving bijective
x(x))() (00)(()))())((()-))))))))))))))) = p(y Z)
(23)
Thus, the measures of joint domains X' x 2 and y x Z are the same. Since F is a measure- seaaure-preserving bijective function, is independent with Z, and Y is also independent with Z, based Lemma A.1, we have:
P((X22) (X' Z)) P(X E X)P(Z)
(24)
= PY)
P(Y e F(X)) = P(Y, ( x Z))
Corollary A.3. For three variables X, Y, Z from measurable sets X, Y, Z, where X is independent with Z and Y is also independent with Z. IfY =FXX and F is a measure-preserring bijective function, there exists:
P(X,Y,2)= 2)8(Y - F(X)) P(Y, 2)8(Y - F(X))
(25)
when u(x) μ())..
21

---

Proof. Based on Bayes' theorem, we have:
P(X, Y2 P(X, PXX 2)
(26)
Since F is bijective function, Y is uniquely determined by X. Thus, X is independent with Z and is also independent with Z, then we have:
P( 2) P(Y,X, P(Y, P(Y,)) P() 2) P(X)P(Z) P(X)
According to Corollary A.I and Corollary A.2, and P(Y (X) is Dirac delta function 8(Y - F(X)). we have:
P(X, Y, 2) P(X. 2))
(27)
P(X, Z)P X) = P(X, 2)0(Y - F(X)) 2)8Y -F())
F(X))
Based on Corollary A.1 and Corollary A.3, we can prove the Theorem 4.1 as follows.
Proof. For the hidden features h 4 and the ground truth you we are seeking to maximizing the expectation of the conditional mutual information between u, and the ground truth ye. when h, is given. From the probability perspective, the h and u, are both independent with the given ground truth yn- Based the definition of conditional mutual information and the Bayes' theorem, have:
PM D
max p(bgg) [L(u [b = max Ep(bp) Bp(up on bn. log P(up bog P(an ber)
Plu (b
max
Jaa
PP("" Mn h los
Plu (he (a (brg
dhl"" du db?""
P(u Bn biy)
mas
gat
gam
P(h yot bi yPo bill)
P(ug [Ige)) P(y (ogr) dymdh, du."" dh
(28)
P(u . Un hgg (P("")
mas
aaa Piunlll (ol bi (MT))OP P( Sou big)) log
Plu (ht) P(ya (hg (h(g) dyychh" du,"" dh
Jeam ( P(o"))( bi P(u b Hog
Plug big)
max
Jaan
P(ug bg P(y (bt))
dy th dul" dh
Note that the distribution of the ground truth P(yx) doesn't change with h thus we have that causal inter-
Plyn hr) = P(y.). Besides, based on the core idea of backdoor-adjustmas vention is applied for the confounder cut off the causal relation between the treatment and the confounder [25]. we apply the causal intervention on h to turn it into its counterfactual form ha and cut off the causal relation between h and u From Corollary A. and Corollary A.3, the Equation (28) can be further transformed to Equation (29).
gaa Jeam Jeam P(NT)) in by PP) P bi log P(u 2h (n)) J (h OTL))
P(ug) (u F (hym:: (1 duy dh du th
Camal Insevvstion
max Jaam Jyam from ((b"")) P(valu"" bi y P( bill
P(ug vn). du ddo(u"" jubj"" P(ug))
max
aam
P(b (Pun P(b, bin ) dh
far P(u hi Jha log
P(ull vn) dandu"" P(u)))(())
ggam P(u. (Pddo 1 log P(do(u vn)
max
Pido(u ()P(vn)
dy, ddo(u""))
fram P(do(u"))) un) P(do(u Mt) ()dooug))))o. donddo(u
max
max ((do(ul)) um).
(29)
where Fin (:; O0 is the bijective decomposition function with parameter 0mh (the URD module in our paper). do(-) is do-operator for causal intervention backdoor-adjustmen mn the h after causal intervention. From Equation (28) and Equation (29), Theorem 4.1 holds.
D
22

---

B Pseudo Code for Training Process
We first give the training process to show the reversibility of LRD-d and F URD-I
Algorithm 1: Enforce Full Rank in PyTorch style
Input: Multimodal model with FCD Model
for 1 to M do
7/ Get parameters of linear layers in URD
(iRDDa: (URD. get (Model.cCD.URD, Linear);
// SVD
U URRD-- Em V
URD-- URD- SVD (0 URD-a (eiDg---eiight;: weight); URRD-F Im
4
URD- URD-F f- SVD (OUR.. weight);
// Enforce Full Rank (Invertible)
Im LURD-d < torch.r.... maximum (EURD-d: 1e-9);
Fm morch..aximut (DURD-r le 1)= LURD
// Reconstruct
URD-a t-nn. Parameter (U URD-d URD
URD : weight f-n. Parameter 2 VUk
URD.)):
end
Algorithm 2: Training process of our process method in PyTorch style
Input:
Multimodal dataset D, Multimodal model with FCD Model, Optimizer opt, Number epochs N
for e i (1. No do
Loverall foodel..orward(D)
Bowerat.backward O)
opt.step O)
Apply Algorithm with model as input;
end
Besides, to achieve measure-preserving, we apply the following constraint on the weight matrix of a and FüRDA Note that the nesting of two measure-press pressurrp functions is still measure- preserving [34].
Algorithm 3: Measure-preserving Linear Layer Forward
Input: Feature vector
Output: Forward output vector x'
// Get weight matrix and bias
W +- 0 weightt;
b <- @ .bias;
// Construct skew symmetric matrix
3 A + WTWT:
W Etorrh.matrix_ertt (A);
// Linear forward
x XW b
C Introduction to Datasets and Evaluation Metries in Our Experiments
CMU-MOSI and CMU-MOSEI are popularly used in Multimodal Semantic Analysis task [9; 13] with 3 modalities. Each sample from both of these datasets is annotated with a sentiment value ranging from -3 (strongly negative) to +3 (strongly positive), indicating the polarity and relative strength of the expressed sentiment within each sample. The former one contains 2199 utterance video segments taken from 93 YouTube, and the latter one contains 22,856 utterance video segments [11]. Table 5 summarizes the training, validation and test subset splits following [15; 43].
23

---

Table 5: Datasets splits (train, validation (val), and test) in our experiments.
DATASET
TRAIN
VAL
TEST
OVERALL
CMU-MOSI
1284
229
686
2199
CMU-MOSEI
16326
1871
4659
22856
UNVSA-SINGLY
1555
518
519
2592
UPMC FOODI01
62971
5000
22715
90686
HFM
19816
2410
2409
24635
MSVA-Single, UPMC Food101, and HFM datasets only have two modalities. MVSA-Single is a commonly used text-image sentiment dataset collected from Twitter. It has 3 categories: positive, neutral and negative with 1398, 724 and 470 samples, respectively [27]. UPMC Food101 dataset is usually used for multimodal image classification [1]], which has 101 categories with about 100k images. HFM has two categories, i.e., positive and negative.
We report our results with the mean absolute enror (MAE), which is calculated by averaging the absolute value between the predicted and ground truth; Pearson correlation (Corr) quantifies extent to which predictions deviate from a linear relationship: binary classification accuracy (Acc- 2) and weighted Fl scores (F1) are computed for both the negative/non-negatio (non-exclude 0) [40] and negative/positive (exclude 0) [32], which is achieved by filtering our the samples whose annotation is O. Following [11], we report the results on the remaining datasets with accuracy (Acc) and weighted F1 score (F1).
D Introduction to Quantitative Experiment Methods
Due to the page limit, we introduce the methods incorporated in our quantitative experiment here.
Self-MM [39] focused on restricted ability in capturing differentiated information within each modality due the unified multimodal annotation, and proposed self-supervised multi-task learning method to generate unimodal annotation. Besides, Self-MM shifts the generated annotation according the relative distance the class center in each modality space.
MMIM [13] designed an MI based method to preserve critical task-related information that flows from the original input to the fusion representation. It employed a tight lower bound MI and estimated lower bound via likelihood maximization and Gaussian Mixture Model (GMM). Then Contrastive Predictive Coding (CPC) loss was employed to retain the modality-invarian information in fused represetation.
MCL-MCF [9] considered that fusion is progressive process, and provided hierarchical structure to maximize the maintenance of semantic information during different fusion level via contrastive learning. Besides, MCL-MCF used 1-D convolutional layers fuse features at different level.
AICAF [15] started from casual inference perspective. blocked the back-door path between text modality and target annotation via front-door adjustment. Then it applied counterfactual reasoning the attention matrix integrated in cross-attention fusion process to improve the fusion robustness.
MMML [37] proposed a Multimodal Multi-Loss Fusion Network that integrates pretrained audio and text encoders, cross- and self-attention mechanisms, and multi-loss training to enhance sentiment analysis. The model achieved state-of-thed of-the-arr art performance on various datasets, demonstrating the effectiveness of multimodal fusion and contextual modeling.
MMBT [18] proposed multimodal fusion method for text-image classification based on bitransformer. It jointly finetuned the pretrained unimodal encoders by mapping image embeddings to textual token space.
CLMLF [20] focused on the token level multimodal fusion and employ contrastive learning to align the representations from different modalities.

---

MVCN [36] focused on the challenge of modality heterogeneity in multimodal tasks, and proposed filter redundant visual features based on sparsemax mechanism. Besides, calibrated feature shift in representation space by minimizing the intra-class discrepancy.
URMF [11] adopted a multivariate Gaussian distribution to represent spotty semantic instances in noisy latent space and tried eliminate the impact of unimodal aleatoric uncertainty to perform robust multimodal fusion via estimiting the Gaussian distribution behind features.
E Hyper-parameter Setting and Sensitive Analysis
The hyper-parameters setting of FCD used in different base methods are reported in Table 6. Since FCD can be integrated into any multi-modal intermediate fusion method, the hyper-parameters may be different with each other. We search the appropriate hyper-parameters with two steps: 1) scale search: ranging each typer-parametee in {0.5, 0.05, 0.005, 0.0005, 0.00005). 2) fine-grained search: ranging each hyper-parameter in the scale that achieves the best performance in step 1). For example, if \1 0.05 achieves the best performance in step 1), we then range A from 0.01 to 0.09 in step 2).
Table 6: paperr-arrametess (A1, X2 and Aς) settings in our experiments.
METHOD
DATASET
A1
A
METHOD DATASET A
Aa
CMU-MOSI 0.08
0.05
0.005
MVSA-S
0.005
0.5
0.05
SELF-MM
CMU-MOSEI
0.003
0.009
0.08
MMBT FOOD101
0.5
0.005
0.5
CMU-MOSI 0.08
0.07
0.9
MVSA-S
0.02
0.0004
0.02
MMIM
CMU-MOSEI
0.6
0.04
0.0003
CLMLF
HFM
0.5
0.5
0.005
CMU-MOSI 0.01
0.3
0.09
MVSA-S
0.05
0.05
0.005
MCL-MCF
CMU-MOSEI 0.007
0.0003
0.8
MVCN
HFM
0.5
0.0005
0.1
CMU-MOSI 0.09
0.0005
0.3
MVSA-S
0.5
0.07
0.09
ATCAF
CMU-MOSEI 0.02
0.0004
0.007
URMF FOOD101
0.05
0.0005
0.05
CMU-MOSI 0.005 0.05
0.05
MMML
CMU-MOSEI 0.05 0.0005
0.05
Additionally, we report the sensitive analysis of several methods: Self-MM, MMIM, MCL-MCF and AICAF on CMU-MOSI dataset. We fix other syper--prameters the value in Table when specific hyper-parameter are ranging in the corresponding scale of magnitude.
Figure 6 shows the variation tendencies of FCD's prediction performance with the changing of values for the hyper-parameters in Equation (17), i.e., (1) A1: the weight for LMI which supervises the extraction of unique feature without redundant information, (2) A2: the weight for Lais which keeps the discriminative information between redundant features and further constrains the modality-speciffc feature extraction, and (3) >g: the weight for LSDA which controls the quality of modality-invarian information extraction and synergistic feature alignment. From Figure 6, there seems to be no obvious common trend among different methods for each hyper-parameter. This may be caused by the way how FCD cooperates with each method. Generally, MAE shows an opposite trend of change compared to other metrics. For most cases, there exists a significant peak (or valley for MAE) of each metric, representing the suitable value. When MAE reaches a valley, other metrics reach peaks and vice versa. However, for A of A1CAF, there exists a peak for MAE and valleys for other metries, which is significantly different with other cases. This may be caused by an inappropriate amplitude of change, where a more inn-grained typee-parameter search is required. Therefore, FCD needs careful haper-parameter searching achieve its greatest potential.
F Computational Overhead Analysis
To further analysis the effectiveness of FCD, we give the following computational overhead analysis to discover training time cost brought by FCD. We conduct this experiment employing Self-MM, MMIM, MCL-MCF, AICAF, and MMML. as the base models to calculate the training overhead per epoch (seconds). The results are shown in Table 7.
25

---

Self-MM
MMIM
MCL-MCF
AICAF
0..0
0.91
0.90
135
0.7
0.76
012
0.70
270
052
021
530
0.55
335
0.85
0.4-
MAE
Corr
830
0.80
2,
676
0.30
Fi(aD)
072
0.32
Ace(ni)
648.
ods
031
9.20
800
034.
382
030-
030-
1/
6/7-
0.7-
0774
0.68-
0100 Dieo a dos Dan bice a00
80:
86
08
800 1337 OIST DOW 316 010
05
65
as
Figure 6: Sensitive analysis of thyper-parameters Å1. Ag and λ with Self-MM, MMIM. MCL-MCF and ArCAF. The horizontal coordinate axis denotes the hyper-parameter rangings and the vertical coordinate axis denotes the evaluation metrics. "n0" in (1(0)) and Acc(n0) stands for the exclusion of O when these two metrics are calculated.
Table 7:
The computational time overhead per epoch (seconds) of Self-MM, MMIM, MCL-MCF, AICAF, and MMML.
METHOD
DATASET
BASE (w/o FCD)
OURS (w/ FCD)
CMU-MOSI
3.92
5.1240.47 51.13+4.37
SELF-MM
CMU-MOSEI
37.14#3.72
CMU-MOSI
18.73=1.30
24.98 1,90
MMIM
CMU-MOSEI
97.74#4.61
131.57
CMU-MOSI 21.19=1.43
27.45+2.39
MCF-MCL
CMU-MOSEI
242.30+7.87
327.76410.84
CMU-MOSI 17.26+1.69
21.95+1.70
ATCAF
CMU-MOSEI 200.61+9.15
250.68+9.97
CMU-MOSI
228.52+11.77
239.12
MMML
CMU-MOSEI
1558.47+107.74
2570.14490.22
In Table 7, we report the average value and the standard deviation duration train one epoch on CMU-MOSI and CMU-MOSEI datasets when FCD is applied (Ours (w/ FCD)) or not (Base (w/o FCD)). From Table 7, we can find that the training time of each epoch is different between different methods. This may be caused by the original structure and computation complexity of each base method. When FCD is applied, the increment is also different. This may be caused by the various hidden dimensions, the default batch sizes and the number of modalities (e.g., MMML only has two modalities).
G Broader Impacts and Future Works
FCD is plug-and-play module that can integrated into any existing intermediate multimodal models to handle the unimodal uncertain noise whilst makes full use of the task-related information.
26

---

We believe that FCD can bring more attentions to current multimodal representation learning commu- nity about handling both of the task-related features (i.e. the synergistic and unique features) and the unimodal uncertainty noise. However, the quality and semantic richness of unimodal feature not fully explored. In the training phase of Self-MM and MMMI,, we find that the prediction performance of each unimodal is quite different. Although there have been researches, such as [43; 5], that engage on estimating the reliability of unimodal prediction, it still remains to be excavated when unimodal uncertain noise is removed for better intermediate fusion. In this case, how much task-related information can unimodal features provide should be considered. make our effort toward this situation to overcome the issues that the quality and semantic richness of unimodal features are various.
the future, we will
27


Reviews:
[{"review_id": "aJmnGRK38G", "reviewer": "Reviewer_B7dA", "summary": "This paper introduces a plug-and-play module named \"Feature Causality Decomposition (FCD),\" which aims to address a core challenge in multimodal learning: how to differentiate and utilize valuable complementary information while simultaneously eliminating modality-specific noise. To achieve this fine-grained decomposition, FCD primarily relies on several key modules and constraints. The Causality Components Decomposition (CCD) module first performs an initial separation of the raw feature z into a modality-invariant part (i.e., the synergistic feature s) and a modality-specific part h. The Synergistic Distribution Alignment (SDA) module then employs pairwise Sinkhorn divergence to align the s features from all modalities to enforce their consistency. The most critical step occurs in the processing of the modality-specific part h. The model applies backdoor adjustment theory, using a Unique Redundant Decompose (URD) submodule to further decompose h into u and r. This process is guided by a mutual information-based loss function, L_MI, which aims to maximize the association between u and the task label, thereby isolating task-irrelevant components into r. Experiments on multiple datasets and existing models demonstrate the effectiveness of this decomposition strategy.", "questions": "1. The paper reports consistent performance improvements across multiple baseline models. To assess the stability of these results, could the authors clarify whether the data in the tables are based on single runs or are averages of multiple independent experiments? Reporting the mean and standard deviation over several runs is necessary to establish the statistical significance of the results.\n\n2. Can the feature decomposition guarantee that h contains only specific information and s only synergistic information? The model uses indirect loss functions to guide the information separation between the s (synergistic) and h (specific) branches. This raises a question: is this mechanism sufficient to guarantee that there is no information leakage between the two? For example, is it possible for the modality-specific branch h, in order to optimize the final task, to also learn and encode some cross-modal synergistic features? Without an explicit constraint to prevent such cases in the current model, how can the purity of the decomposition be guaranteed?\n\n3. The data in Appendix F demonstrates the computational overhead of FCD for 2-3 modalities. Considering that the L_SDA loss is based on pairwise comparisons, its complexity grows quadratically (O(M^2)) with the number of modalities M. How is the scalability of the method envisioned for scenarios with many more modalities (e.g., M > 5)? Would this growth in computational complexity become a bottleneck for its application?\n\n4. The L_MI and L_Dis loss functions used in the model are essentially contrastive learning, whose effectiveness is often related to the number of negative samples provided by the batch size. The paper does not include a sensitivity analysis for this hyperparameter. Could the authors provide the specific batch sizes used in the experiments and comment on the FCD's performance sensitivity to this parameter?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses an important and subtle problem in multimodal learning: how to distinguish modality noise from effective complementary information. Its core contribution lies in moving beyond the traditional shared-specific binary decomposition. It introduces a causal inference framework to propose a more fine-grained, three-part decomposition (synergistic, unique, redundant). \n\n2. The paper's methodology is well-designed and technically sound. The model's architecture is clearly defined (e.g., CCD, SDA, URD modules), and it employs mature techniques from the field (e.g., Sinkhorn divergence, InfoNCE loss) to achieve its objectives. The experimental validation is thorough; by applying the FCD module to 9 different state-of-the-art models across 5 standard datasets, the paper provides substantial empirical evidence for the method's effectiveness and its \"plug-and-play\" generalization ability. \n\n3. The paper is well-structured and clearly written. The authors effectively use diagrams (such as the causal graph and model architecture) to illustrate their complex theoretical model and workflow, which greatly aids the reader's understanding of the core ideas.\n\nWeaknesses:\n\n1. A main concern of the method is that its separation of synergistic (s) and specific (h) features relies entirely on the indirect guidance of downstream loss functions. The model lacks an explicit mechanism (e.g., an orthogonality constraint or a mutual information minimization loss) to actively prevent information leakage between these two branches. Therefore, the \"purity\" of the decomposition beomes a core assumption that is not directly verified, which to some extent undermines its theoretical rigor.\n\n2. Although the experimental coverage is broad, it still needs further statistical analysis. All performance results are reported without standard deviations or significance tests. This makes it difficult to determine to what extent the reported performance gains are stable and reproducible, rather than being a result of random fluctuations during model training.\n\n3. The design of the Synergistic Distribution Alignment (SDA) module has a notable limitation. As it relies on computing pairwise Sinkhorn divergence between modalities, the computational complexity may grow rapidly.", "was_revised": true, "final_justification": "Thank you for the rebuttal. I appreciate the clarifications regarding feature decoupling (Q2) and the added sensitivity analysis (Q4).\nHowever, my primary concern about experimental rigor (Q1) remains. The claim of \"no stochastic mechanism\" implies a fixed seed, yet the rebuttal neither confirms its consistent use across baselines nor provides the necessary multi-run results. This lack of statistical validation raises significant concerns about potential cherry-picking, especially on larger datasets where such low variance would be highly unusual. I also concur on the computational overhead challenges (Q3).\nAs a result, I will maintain my original rating.", "rebuttal": "Thank you very much for your thoughtful and constructive comments, which would be greatly helpful to improve the quality of our work. \n\n----\n\n**Q1.** The standard deviation over several runs of our experiments.\n\n**A1.** We conducted multiple independent experiments and found that each experimental result is basically the same. This is because we followed the hyperparameters reported in each base methods or their official scripts. And there is no stochastic mechanism designed in our approach. So the standard deviations are pretty small and they were not indicated in the paper. We are sorry for the unclear statement and will refine our description of the experimental design and results.\n\n|Method|MAE|Corr|Acc-2|F1|\n|-|-|-|-|-|\n|Self-MM|0.00|0.00|0.00/0.00|0.00/0.00|\n|MMIM|0.00|0.00|0.00/0.00|0.00/0.00|\n|MCL-MCF|0.00|0.00|0.00/0.00|0.00/0.00|\n|AtCAF|0.00|0.00|0.00/0.00|0.00/0.00|\n|MMML|0.00|0.00|0.00/0.00|0.00/0.00|\n\nThese are the standard deviations of 5 independent experiments on CMU-MOSI dataset.\n\n----\n\n**Q2.** The guarantee of separation between the s (synergistic) and h (specific) branches (purity).\n\n**A2.** The guarantee of separation between the $\\mathbf{s}$ and $\\mathbf{h}$ branches is the direct optimization targets, i.e., the item $\\mathcal{L}\\_{\\text{FCD}}$ (Eq. 17).\n\n- (1) On the one hand, to separate the synergistic feature $\\mathbf{s}$, $\\mathcal{L}\\_{\\text{SDA}}$ (Eq. 15) is employed to optimize $\\mathcal{F}\\_{\\text{s}}$ and SDA module. It constrains $\\mathbf{s}$ to be mapped into a common representation space.\n\n- (2) On the other hand, to separate the modality-specific feature $\\mathbf{h}$, the other two optimization targets are employed, i.e., $\\mathcal{L}\\_{\\text{Dis}}$ (Eq. 19) and $\\mathcal{L}\\_{\\text{MI}}$ (Eq. 6). These two items jointly supervise the extraction of $\\mathbf{h}$, ensuring it’s not only modality-specific but also task-related information.\n\nSince the unique feature $\\mathbf{u}$ is mixed in $\\mathbf{h}$, there exists a complex high-level semantic overlap between $\\mathbf{s}$ and $\\mathbf{h}$, because $\\mathbf{s}$ and $\\mathbf{u}$ describe the same object (the task annotation is the same for a single data point). It is tricky to directly separate them since the semantic contents of different samples are various. If we directly add constraints (like orthogonality constraint), the two output may become task-related and task-agnostic. In this case, we cannot guarantee the task-related part contains both complementary and consistent information, which may cause information distortion.\n\nInspired by [1], we use two parallel MLPs to map the original feature to the synergistic $\\mathbf{s}$ and the modality-specific $\\mathbf{h}$ with the supervision of $\\mathcal{L}\\_{\\text{FCD}}$, as we mentioned above. Although there are no regularization terms between these MLPs, the output of them are still constrained by $\\mathcal{L}\\_{\\text{FCD}}$, ensuring them effectively extracting the corresponding features.\n\n[1] Ying Zhou et.al. Triple disentangled representation learning for multimodal affective analysis. Information Fusion 2025.\n\n----\n\n**Q3.** The computational overhead of pairwise comparison L\\_SDA.\n\n**A3.** Your concerns are very reasonable and worth improving upon. The purpose of designing this pairwise comparison loss function is to reduce the differences between any different modal distributions, in order to ensure that the extracted synergistic features are indeed shared among $M$ modalities. As you have pointed out, this strict constrain will cause the $O(M^2)$ complexity. To improve it, we can randomly sample two modalities in each iteration. Then only one Sinkhorn divergence between two distributions is calculated to achieve approximate alignment effect.\n\n----\n\n**Q4.** The sensitivity analysis of batch size hyperparameter.\n\n**A4.** We reproduced and conducted our experiments based on the batch size given in each compared paper or their scripts. Since FCD is a plug-and-play module and we aim to validate the performance under the original experimental settings, we did not tune the batch size (or other base hyperparameters) in our experiments. Here, we conduct the sensitiveity analysis on Self-MM with FCD on the CMU-MOSI dataset and the results are as follows.\n\n| Batch Size | 8 | 16 | 32 | 64 | 128 |\n|-|-|-|-|-|-|\n| MAE | 67.83 | 68.02 | 68.29 | 68.11 | 67.94 |\n| Corr | 80.34 | 79.86 | 80.62 | 80.27 | 80.57 |\n| Acc-2 | 84.74/86.96 | 84.43/86.27 | 84.95/87.09 | 85.15/87.41 | 85.74/87.85 |\n| F1 | 84.74/87.01 | 84.48/86.31 | 84.93/87.11 | 85.16/87.42 | 85.71/87.83 |\n\nThis experiment result shows that larger batch size may enhance the performance of the model because of the increment of negative sample diversity in contrastive losses. We would add the results to the camera-ready version if accepted."}, {"review_id": "azdf9KmYEH", "reviewer": "Reviewer_KESk", "summary": "This paper proposes a plug-and-play feature causality decomposition method for multimodal representation learning, addressing the limitations of existing approaches that fail to distinguish between complementary information and aleatoric uncertainty noise within modalities. The method disentangles unimodal features into modality-invariant (synergistic information shared across modalities) and modality-specific components, with the latter further decomposed into unique and redundant features using backdoor adjustment to eliminate noise while preserving task-relevant information. The resulting synergistic and unique features are then integrated into existing fusion modules, with causality theory supporting the noise removal process. Extensive experiments demonstrate the effectiveness of this approach in enhancing multimodal representations.", "questions": "1. The size of this plug-in module appears to be relatively large, with a considerable number of parameters. Typically, plug-in modules should maintain a compact design without excessive parameters. Could you clarify how the parameter scale of the proposed FCD module compares to that of the base model?\n\n2. The author should clearly explain why the URD is a measure-preserving bijective function and what \"measure-preserve\" specifically refers to in this context. In mathematics, \"measure\" has a strict definition. What exactly does \"measure\" refer to here?\n\n3. The rationality of the assumed information model in Fig. 1 requires further elaboration. Specifically, it should be clarified whether common multimodal data aligns with the proposed SMC model. The authors could enhance the justification by providing concrete examples to illustrate the model's validity. For instance, in image-text multimodal tasks, which factors are task-related, which are synergistic features, and what are the other features mentioned in Fig1.\n\n4. The motivation and necessity of SDA are not sufficiently justified. Is such a complex design truly necessary for SDA? If replaced with a simpler strategy—for instance, using a shared MLP across all modalities—would the performance degrade significantly?\n\nAdditional comments:\n\n1. Some typos: line 226 and 338\n\n2. The authors should more clearly and directly clarify what kind of claim or what kind of conclusion their theorem supports.\n\n3. The font of Definition should be different from the main contexts.\n\n4. For the case study, it is suggest to select and analyze challenging cases where prior methods proved ineffective. By demonstrating that the proposed approach succeeds in these scenarios—particularly when aligned with the SCM hypothesis—we can highlight its critical role. Specifically, these cases should strongly support the hypothesis, illustrating that without it, the model would fail. Conversely, if the hypothesis is not rigorously validated, there remains a risk that the method's effectiveness could be coincidental rather than causally linked to the proposed mechanism.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThis article introduces a novel Structural Causal Model (SCM) assumption for multimodal data, providing an innovative approach to feature representation. The proposed method includes a feasible feature disentanglement technique, which is rigorously supported by both theoretical analysis and experimental validation. The framework demonstrates strong potential for advancing multimodal data modeling.\n\nWeaknesses:\nWhile the proposed SCM assumption is promising, the article lacks a detailed rationality analysis of this assumption. Additionally, the presentation could be refined to enhance clarity and precision. Further improvements in explanatory depth and linguistic expression would strengthen the work.", "was_revised": true, "final_justification": "The authors have addressed my concerns and provided clarifications that improved my understanding of the paper's contributions and core principles. I appreciate their efforts in revising the manuscript based on my earlier feedback. However, I have some reservations regarding the extent of the revisions. The original submission contained several key points that required clarification or updates. While the authors have resolved these issues, I am uncertain whether making substantial revisions to a conference paper at this stage is appropriate. If these critical points are not adequately incorporated into the final version, it may affect readers' ability to fully grasp the work. While I maintain my score, I remain generally positive about this work.", "rebuttal": "We greatly appreciate the time and effort you have taken to review our work, and we believe your suggestions have significantly improved the quality of the manuscript. \n\n----\n\n**Q1.** The parameter size of FCD.\n\n**A1.** There are only several MLPs integrated in FCD and the parameter sizes of them depend on the hidden dimensions of the output of modality encoders. \n\nSpecifically, for each modality, CCD module has 4 MLPs with the input size $d^m$ and the output size $d^m$ ($\\mathcal{F}^m\\_{\\text{h}}$, $\\mathcal{F}^m\\_{\\text{s}}$, $\\mathcal{F}^m\\_{\\text{URD-f}}$ and $\\mathcal{F}^m\\_{\\text{URD-d}}$). So the parameter scale is $O({d^m}^2)$. For SDA module, there are M MLPs that map each unimodal feature to a common space and then map to the original dimension. Then parameter scale is $O(d^2+d\\sum\\_{m=1}^{M}d^{m})$.\n\nSince the base methods have their unique designs, there is no exact parameter size ratio between FCD and base model. We report the training time cost in Appendix F, which may also reflect the parameter size relationship between the FCD and the original model. It can be seen that the growth rate brought by FCD is relatively large for models with original short training times. Conversely, the growth rate brought by FCD is relatively small.\n\nTherefore, the FCD parameter scale is associated with the base model unimodal encoder output dimensions. If the output dimensions are large, the FCD parameter size would be larger than those whose output dimensions are small.\n\n----\n\n**Q2.** The measure-preserving bijection of URD and the ''measure'' in our paper.\n\n**A2.** We technically apply SVD and construct skew symmetric matrix to constrain URD. We give the pseudo code to train FCD in Appendix B. Specifically, every singular value in $\\Sigma$ is enforced to be greater than $10^{-5}$ to ensure that the weight matrix of the linear layer in URD remains invertible. Then, the skew symmetric matrix with exponential map is constructed to achieve the measure-preserving characteristic. Since the nesting of two measure-preserving functions is still measure-preserving, the URD is also a measure-preserving bijective function. Please refer to Appendix B for more details of the training process and further description of measure-preserving bijective URD.\n\nWe specify that URD should be ''measure-preserving'' to make Lemma A.1 in Appendix A hold. In Lemma A.1, if $\\mathcal{F}$ is not measure-preserving, it may change the probability distribution of variables. For example, if $\\mathcal{F}$ is not measure-preserving, let $Y=2X$ and $X$ is uniform distribution $P(X)=1/2$. In this case, the probability distribution of $Y$ should be $P(Y)=1/4$, Lemma A.1 do not hold. Therefore, the ''measure-preserving'' ensures that the probability distributions of variables are the same. And the ''measure'' here refers to the size of subsets within a stochastic variable's domain, allowing probability to be assigned to events.\n\n----\n\n**Q3.** Further elaboration of SCM (Fig. 1).\n\n**A3.** As we stated in Introduction (Section 1) and Related Works (Section 2), previous researches mainly focused on better utilization of complementary and consistent information or mitigating aleatoric uncertainty within each modality. We summarize previous related works and empirically abstract the unimodal feature into 3 types of factor (synergistic, unique and redundant).\n\nFor example, in image-text task,\n\n- The task-related parts are the ones that contribute to the final task prediction. The synergistic parts are both included in the image and the text modalities, which jointly contribute to the final task prediction.\n\n- The unique parts are the semantic contents that only appear in image modality (or text modality) and don't appear in text modality (or image modality). Both synergistic and unique factors are task-related and derived from the task-related factor.\n\n- The redundant parts are the uncertainty noise within one modality, such as the typos in text modality or quality of image, which is derived from the noise factor. \n\n- The multimodal data store these information in their different format (the specific factor), but they are still associated in high-level semantic (the synergistic factor).\n\nIn case study (Section 5.5), we visualize the synergistic and unique parts in both image and text modality, which can more directly understand the corresponding regions of different types of features in the original sample.\n\n----\n\n**Q4.** The motivation and necessity of SDA.\n\n**A4.** SDA module is used to constrain the extraction of synergistic feature from each modality. The synergistic feature represents the consistent information that presents in multiple modalities, facilitating alignment of modalities within a common representation space. Without SDA, there is no explicit strategy to constrain the features are able to be aligned within a common representation space, which causes the failure of synergistic feature extraction.\n\nSimilar to your idea, we also employed a shared MLP across all modalities ($\\mathcal{F}\\_{\\text{SDA}}$, see Fig. 4). To further align the features, we propose $\\mathcal{L}\\_{\\text{SDA}}$ (Eq. 15) to explicitly optimize $\\mathcal{F}\\_{\\text{SDA}}$ and guide features from different modalities towards a common representation space. In our ablation study (Section 5.4), we set the coefficient of $\\mathcal{L}\\_{\\text{SDA}}$ ($\\lambda\\_3$) equals to $0$ to validate the effectiveness of $\\mathcal{L}\\_{\\text{SDA}}$, where only the parameter-sharing MLP is employed and there are no constrains on the synergistic information alignment. From Table 3, we can find that the performance decline without this explicit constraint. Therefore, it is necessary to explicitly supervise the module by $\\mathcal{L}\\_{\\text{SDA}}$ (Eq. 15). Please refer to Section 5.4 for more details.\n\n----\n\n**Additional comments:**\n\n**Q1.** Some typos: line 226 and 338\n\n**A1.** Thank you again for your meticulous review. We will double check the details in the article.\n\n----\n\n**Q2.** The authors should more clearly and directly clarify what kind of claim or what kind of conclusion their theorem supports.\n\n**A2.** Theorem 4.1 states that if the extraction function $\\mathcal{F}^m\\_{\\text{mb}}$ is a measure-preserving bijection, then effectively extracting the task-related feature $\\mathbf{u}^m\\_n$ from the modality-specific feature $\\mathbf{h}^m_n$ while separating out the modality noise $\\mathbf{r}^m\\_n$ via backdoor adjustment is equivalent to maximizing the mutual information between the annotation and the $\\mathbf{u}^m\\_n$ after causal intervention. Based on this, URD extracts the unique feature (u) via Eq. 6 where the unimodal noise is simultaneously removed.\n\n----\n\n**Q3.** The font of Definition should be different from the main contexts.\n\n**A3.** Thank you for your suggestion. We will revise it in our camera-ready version if accepted.\n\n----\n\n**Q4.** For the case study, it is suggested to select and analyze challenging cases where prior methods proved ineffective. By demonstrating that the proposed approach succeeds in these scenarios—particularly when aligned with the SCM hypothesis—we can highlight its critical role. \n\n**A4.** Thanks for your rigorous suggestion to the case study. We will conduct this experiment and revise it in our camera-ready version if accepted."}, {"review_id": "fFZREwHH3s", "reviewer": "Reviewer_hx2Y", "summary": "This paper proposes a plug-and-play Feature Causality Decomposition (FCD) framework for multimodal representation learning. The method performs a two-stage decomposition process: first, it separates unimodal features into modality-invariant and modality-specific components; second, the modality-specific component is further decomposed into unique and redundant parts using a backdoor-adjustment mechanism grounded in causal inference. Extensive experiments are conducted on two benchmark datasets for multimodal sentiment analysis (MSA), as well as three additional datasets, to validate the effectiveness of the proposed approach.", "questions": "1. How to introduce the FCD to the existing work, given that previous work has its unique design. Which layer or block is for the 5 methods for MSA? Is there some extra effort to tune the hyperparameters? If so, what is the motivation behind tuning the hyperparameter?\n\n2. Whether the number of modalities affects the performance or the use of FCD?\n\n3. What is the difference between FCD and methods following the disentanglement paradigm? like DMD[1] and its follow-up papers.\n\n[1] Decoupled Multimodal Distilling for Emotion Recognition, CVPR 2023.", "limitations": "yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n1. The paper addresses a critical problem in multimodal representation learning—namely, the challenges of redundancy and conflict across modalities. Effectively decomposing multimodal representations remains a central research challenge, and the proposed method directly tackles this issue.\n\n2. The proposed approach is conceptually straightforward. By introducing an additional FCD loss to guide the learning process, the method integrates seamlessly with existing architectures and enhances performance without introducing significant complexity.\n\n3. The experiments are comprehensive and convincing. The use of CMU-MOSI and CMU-MOSEI—two widely adopted benchmarks in multimodal sentiment analysis (MSA)—demonstrates the effectiveness of the method in standard settings. Moreover, the evaluation on additional classification tasks further supports the generalizability of the approach, which is a notable strength.\n\nCons:\n\n1. Potential missing citation for Theorem 4.1: It appears that Theorem 4.1 is derived from prior work, but no citation is provided. Proper attribution is important for clarity and academic integrity.\n\n2. There are several textual and formatting typos throughout the paper. For example, line 267 mentions five methods, while Table 2 only presents four. Additionally, there are typographical errors in lines 290 and 318. These inconsistencies may affect the readability and perceived rigor of the work.\n\n3. The ablation study and case study are conducted on different datasets, which may reduce the coherence of the analysis. It would strengthen the paper to conduct both studies on the same dataset. For instance, providing a case study on the MOSI dataset—similar to Figure 5—could offer more consistent and interpretable insights into the effectiveness of each component.", "was_revised": true, "final_justification": "The authors address most of my concerns and give the details of my questions. I think this work is meaningful and prefer to accept it if it can revise the content in the rebuttal and add some recent related work to the performance comparison, especially the decoupled-based framework, which is highly related to the proposed method in this paper.\n\n[1] Li et al. \"Decoupled multimodal distilling for emotion recognition.\" CVPR 2023.\n[2] Wang et al. \"DLF: Disentangled-language-focused multimodal sentiment analysis.\" AAAI 2025.", "rebuttal": "We truly value your expertise and the careful consideration you've given to our work, which has helped us refine our approach.\n\n----\n\n**Cons (Weaknesses)**\n\n**W1.** Citation of Theorem 4.1.\n\n**A1.** We are sorry for this careless. Theorem 4.1 is based on the casual inference [1], information theory [2], and probabilistic theory [3]. We will formally add the necessary citation in the camera-ready version if accepted.\n\n[1] Judea Pearl. Causality. Cambridge university press 2009.\n\n[2] Ronghao Lin et.al. Multi-Task Momentum Distillation for Multimodal Sentiment Analysis. Affective Computing 2024.\n\n[3] E. T. Jaynes. Probability theory: The logic of science. Cambridge university press 2003.\n\n----\n\n**W2.** Typos in the paper.\n\n**A2.** Thanks again for your careful review. We will revise these typos and recheck the content of the paper.\n\n----\n\n**W3.** Different datasets in ablation study and case study.\n\n**A3.** The reason why we conducted these experiments on different datasets is mainly because of the characteristics of the datasets themself. We didn’t conduct case study on CMU-MOSI dataset because the visual modality data are videos, which can not be directly displayed in the paper. Thus, we perform the ablation study on the MVSA-S using CLMLF and the results are as follows.\n\n|$\\mathcal{L}\\_{\\text{MI}}$|$\\mathcal{L}\\_{\\text{Dis}}$ |$\\mathcal{L}\\_{\\text{SDA}}$|Eq. 16|Eq. 13|Acc-2|F1|\n|-|-|-|-|-|-|-|\n||√|√|√|√|70.89|69.49|\n|√||√|√|√|71.56|70.70|\n|√|√||√|√|71.78|70.95|\n|√|√|√||√|71.11|69.43|\n|√|√|√|√||71.78|71.06|\n|√|√|√|√|√|72.22|71.62|\n\nFrom the table, we can find that it has the similar tendency with the ablation study on CMU-MOSI using Self-MM. Eq. 16 and Eq. 13 are the ablation cases for task-related features fusion and shortcut in SDA module. The results also show the collective effect of our designs, which is consistent with Section 5.4.\n\n----\n\n**Questions**\n\n**Q1.** Introduce FCD to existing works and the motivation behind tuning the hyperparameter.\n\n**A1.** As you mentioned, previous works have their unique designs. So we abstract them into a unified process framework (Fig. 2 (a)), which mainly consists three parts (unimoda encoder, fusion module and prediction head). FCD is designed to decompose the synergistic, unique and reduntant features from unimodal representations, so it is attached between the unimodal encoders and the fusion module (Fig. 2 (b)).\n\nSince FCD is a plug-and-play module, we only tune the hyperparameters involved by FCD and fixed the original ones. We report the hyperparameter search process in Appendix E. We first ranging each hyperparameter in {$0.5, 0.05, 0.005, 0.0005, 0.00005$} to find the suitable scale, then a more fine-grained search is performed in this suitable scale to find the final results. Please refer to Appendix E for more details about the tuning process.\n\n----\n\n**Q2.** The impact of the number of modality on FCD.\n\n**A2.** The number of modelity would significantly affect the training time cost because of the pairwise comparison $\\mathcal{L}\\_{\\text{SDA}}$, which has $O(M^2)$ complexity. The purpose of designing this $\\mathcal{L}\\_{\\text{SDA}}$ is to reduce the differences between different modal distributions, ensuring that the extracted synergistic features are mapped into the same representation space and indeed shared among $M$ modalities. Moreover, the calculation of Sinkhorn divergence may also become time-consuming as $M$ increases. To improve it, we can randomly sample two modalities in each iteration. Then only one Sinkhorn divergence between two distributions is calculated to achieve approximate alignment effect.\n\n----\n\n**Q3.** The difference between FCD and methods following the disentanglement paradigm. like DMD[1] and its follow-up papers.\n\n[1] Decoupled Multimodal Distilling for Emotion Recognition, CVPR 2023.\n\n**A3.** FCD not only focus on the disentanglement of modality-invariant and modality-specific information (as DMD[1] did), but also explicitly considers the unimodal uncertainty noise contained in each modality caused by data acquisition processes and sensor characteristics. In short, FCD takes the modality-invariant (consistent information), modality-specific task-related (complementary information) and modality-specific task-agnostic (unimodal uncertainty noise) into consideration simultaneously. Then, based on causal inference, we propose to remove the unimodal noise mixed in the modality-specific feature to obtain the complementary information. Meanwhile, the consistent information is also extracted. After obtaining both consistent and complementary information, the task-related information can be better utilized. Moreover, FCD is a plug-and-play module that can be integrated to existing methods to improve their performances by eliminating the unimoda uncertainty noise.\n\n[1] Yong Li et.al. Decoupled Multimodal Distilling for Emotion Recognition, CVPR 2023."}, {"review_id": "Ql2oOnsVXl", "reviewer": "Reviewer_Qg9n", "summary": "This paper introduces a novel module, Feature Causality Decomposition (FCD), designed to enhance multimodal representation learning by explicitly modeling and separating synergistic, unique, and redundant components of unimodal features. Drawing from principles of causal inference, particularly backdoor adjustment, the method aims to remove modality-specific task-agnostic noise while preserving task-relevant information. FCD integrates two core submodules: Causality Components Decomposition (CCD) and Synergistic Distribution Alignment (SDA). The proposed method is plug-and-play and can be inserted into existing intermediate fusion models without structural modifications. The authors evaluate FCD on five benchmark datasets across various tasks and report consistent performance improvements over several state-of-the-art (SOTA) methods.", "questions": "* Could the authors clarify whether this SCM was constructed based on empirical data, prior knowledge, or intuition? Additionally, can you discuss how sensitive your method is to potential misspecifications of this causal structure?\n\n* Empirical evaluations on larger-scale datasets (e.g., MIMIC-CXR) is encouraged to demonstrate the scalability", "limitations": "This paper specified in the checklist that they discussed limitations but it was not found in the main text nor in the appendix. From what I can see the scalability is a significant limitation", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The authors provide a theoretical framework connecting backdoor adjustment (from causal inference) with mutual information optimization, supporting their decomposition strategy.\n\n* Applying causality to multimodal learning is an interesting trend, and the proposed study is well motivated.\n\n* Experiments on conventional multimodal learning datasets (CMU-MOSI, MOSEI, Food101, etc.) and base models. Ablation studies and Grad-CAM visualizations add interpretability and support the claimed benefits of each component.\n\n**Weaknesses**\n\n* Despite the sufficient motivation, the topic of feature disentanglement on backdoor adjustment is not novel. See [1] for an example of similar work on backdoor adjustment on graph representation learning. Therefore, the methodological contribution is limited. \n\n* While the paper leverages causal language and theoretical constructs, the actual role of causality remains largely conceptual. The core assumption—that modality-specific noise can be removed via backdoor adjustment—is not empirically validated. The SCM and do-operator usage is not grounded in real causal discovery or data-generating processes.\n\n* Following the previous comment. The assumed causal structure (e.g., the existence and confounding role of modality-specific features) is not validated via data-driven methods such as causal discovery, nor is any synthetic experiment conducted to test causal robustness. This undermines the practical credibility of the theoretical claims.\n\n* Scalability is limited for empirical evaluation as the authors only performed experiments on CMU---MOSI-like datasets, which are small in scale. Empirical evaluations on larger-scale datasets (e.g., MIMIC-CXR) are encouraged to demonstrate the scalability\n\n[1] Sui, Yongduo, et al. \"Causal attention for interpretable and generalizable graph classification.\" Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. 2022.", "was_revised": true, "final_justification": "Thanks for the rebuttal. I agree that validating real causal structure would be difficult and I also acknowledge the effectiveness of causal debiasing. The authors' rebuttal demonstrates the principle behind the proposed framework. Hence I will increase my score. However, I would also encourage the authors to perform experiments on modern high-dimensional (not only large sample size) datasets such as MIMIC-CSR, which will greatly enhance the empirical evaluation part, particularly the scalability.", "rebuttal": "We are truly grateful for your insightful suggestions and are pleased to provide our responses.\n\n----\n\n**Weaknesses**\n\n**W1.** Backdoor-adjustment has been widely used (novelty is limited).\n\n**A1.** Backdoor-adjustment has been widely used in various kinds of tasks, including graph representation learning, multimodal sentiment analysis [1] etc. The backdoor-adjustment itself has been proposed for a long time [2], which usually performed as a tool for debiasing. Many researchers have dedicated to employ it for removing the causal effect of confounders in their systems, such as generating counterfactual samples, working together with frontdoor-adjustment etc. So do we. However, different with them:\n\n- (1) We utilize it in our theoretical derivation process for removing the causal effect of unimodal uncertainty noise on the task prediction.\n\n- (2) We just utilize it in the theoretical level to avoid the potential irrationality and poor diversity of counterfactual samples. For example, in [3], the backdoor-adjustment is employed to generate counterfactual samples via replacing the background of samples within a batch. The diversity of generated samples are highly related to the dataset scale, where only backgrounds contained in the dataset can be used for intervention. Different with them, we apply backdoor-adjustment on the theoretical derivation process without explicitly generating counterfactual samples, which can avoid the potential impact of dataset scale (the diversity of confounder).\n\n- (3) FCD is a plug-and-play module that can be implemented to the existing methods for unimodal uncertainty noise removal.\n\nIn short, FCD is a different representation learning plug-and-play module compared with existing ones. Just like the Bayes’ theorem in probabilistic theory [4], the backdoor-adjustment is a basic conception in causality theory.\n\n[1] Changqin Huang et.al. AtCAF: Attention-based causality-aware fusion network for multimodal sentiment analysis. Information Fusion 2025.\n\n[2] Judea Pearl. Causality. Cambridge university press 2009.\n\n[3] Sarah Rastegar et.al. Background no more: Action recognition across domains by causal interventions. Computer Vision and Image Understanding 2024.\n\n[4] Jaynes, E.T.: Probability theory: The logic of science. Cambridge university press 2003.\n\n----\n\n**W2.** The actual role of causality remains largely conceptual.\n\n**A2.** We aim to extract the complementary and consistent information whilst removing the uncertainty noise mixed in the unimodal features. As it has been widely proved effective for defounding (debiasing), backdoor-adjustment and causality are also employed in our work. However, we aim to provide a generic plug-and-play module. To this end, we hope to simplify the actual complexity of FCD as much as possible through theoretical analysis and deduction. Therefore, we propose Theorem 4.1, which proves that the unimodal uncertainty noise can be removed by Eq. 6 based on backdoor-adjustment under certain conditions. Although it seems to be “conceptual”, the actual roles of causality and backdoor-adjustment are vital in our paper.\n\n----\n\n**W3.** Validation of causal structure. \n\n**A3.** Please refer to the A1 in Question below.\n\n----\n\n**W4.** Experiments on larger-scale dataset.\n\n**A4.** Please refer to A2 in Question below.\n\n----\n\n**Questions**\n\n**Q1.** The motivation of SCM construction.\n\n**A1.** Previous researches mainly focused on the better utilization of the complementary and consistent information (such as fusion strategy, dynamic fusion, etc.), or mitigating aleatoric uncertainty within each modality. We summarize the previous related works and empirically abstract the unimodal feature into 3 types of factor (synergistic, unique and redundant) corresponding to work [1], which proposed these concepts and validated them from both theory and demo experiments.\n\nWe acknowledge your suggestion that data-driven causal structure validation helps to improve the persuasiveness of the proposed SCM, though we note that this direction has rarely been explored in the existing literature [2-5]. To validate the causal structure of our proposed SCM and the correspondence with [1], following [1], we conduct the experiments using Self-MM on CMU-MOSI testing set, where the normalized information increment is used as metric. Please refer to [1], for their theoretical derivation and explanation.\n\nWe varify the causal relationship between $\\mathbf{z}$, $\\mathbf{h}$, $\\mathbf{s}$, $\\mathbf{u}$, $\\mathbf{r}$, $\\hat{y}$. We first store variables of each modality, and then apply PCA on the hidden embeddings ($\\mathbf{z}$, $\\mathbf{h}$, $\\mathbf{s}$, $\\mathbf{u}$, $\\mathbf{r}$) along the dimension axis. Therefore, all vectors are reduced in dimension to scalars. Then, we apply Causal Additive Models algorithm for causal discovery. We repeat this procedure for 1000 times to calculate edge confidence. Since figures are not allowed in rebuttal period, we simply describe the result as follow:\n\n||$\\mathbf{z}$|$\\mathbf{h}$|$\\mathbf{s}$|$\\mathbf{r}$|$\\mathbf{u}$|$\\hat{y}$|\n|-|-|-|-|-|-|-|\n|$\\mathbf{z}$|0.00|0.75|0.83|0.55|0.31|0.23|\n|$\\mathbf{h}$|0.19|0.00|0.31|0.72|0.83|0.40|\n|$\\mathbf{s}$|0.23|0.27|0.00|0.20|0.31|0.80|\n|$\\mathbf{r}$|0.29|0.45|0.37|0.00|0.38|0.27|\n|$\\mathbf{u}$|0.42|0.55|0.47|0.39|0.00|0.78|\n|$\\hat{y}$|0.34|0.37|0.54|0.23|0.34|0.00|\n\nEach element of the table stands for the frequency of edge occurrence in repeated experiments. The higher the frequency, the stronger the causal relationship between two variables. It is apparently that synergistic $\\mathbf{s}$ and modality-specific task-related $\\mathbf{u}$ have stronger causal relationship with the target $\\hat{y}$. And the overall results indicate the causal effectiveness of our proposed SCM structure.\n\n[1] Álvaro Martínez-Sánchez et.al. Decomposing causality into its synergistic, unique, and redundant components. Nature Communications 2024.\n\n[2] Yongduo Sui et.al. Causal attention for interpretable and generalizable graph classification. SIGKDD 2022.\n\n[3] Changqin Huang et.al. AtCAF: Attention-based causality-aware fusion network for multimodal sentiment analysis. Information Fusion 2025.\n\n[4] Yanan Zhang et.al. Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective. NeurIPS 2024.\n\n[5] Weixing Chen et.al. Cross-Modal Causal Intervention for Medical Report Generation. TIP 2025.\n\n[6] Peter Bühlmann et.al. CAM: Causal additive models, high-dimensional order search and penalized regression. The Annals of Statistics 2014.\n\n----\n\n**Q2.** Empirical evaluations on larger-scale datasets (e.g., MIMIC-CXR).\n\n**A2.** We evaluated our method on different datasets with different scales (CMU-MOSI with $2199$ samples, UPMC Food101 with nearly $10^5$ samples, please refer to Appendix. C for more details.). Besides, as we mentioned in Weaknesses A1, since FCD doesn’t acutally generating counterfactual samples, the impact of dataset size is theoretically reduced. \n\nAlthough we tried to apply for MIMIC-CXR dataset as son as we can, it requires certain Certification (the CITI program) and verification to access the dataset and it is too large to donwload and train, which may delay the rebuttal process. Practically, following CMCRL [1], we use the preprocessed dataset, and conduct the quantitative experiment on CMCRL. Because of the complexity of CMCRL and the scale of dataset, the experiment is still ongoing. We will update the results as soon as they come out.\n\n[1] Weixing Chen et.al. Cross-Modal Causal Intervention for Medical Report Generation. TIP 2025."}]

Author rebuttal:

Meta-review: This paper introduces a plug-and-play Feature Causality Decomposition (FCD) module for multimodal representation learning, aimed at separating task-relevant synergistic and modality-specific features from redundant noise via causal inference (backdoor adjustment). FCD integrates seamlessly with existing fusion models and demonstrates consistent performance gains across nine state-of-the-art models and five benchmark datasets.

Reviewers praised the paper for addressing an important problem with a theoretically grounded framework, clear architecture (CCD, SDA, URD modules), and thorough experiments. Concerns included potential feature leakage, missing statistical analyses, SDA computational complexity, and novelty of backdoor adjustment. The authors’ rebuttal clarified these issues, providing standard deviations, explicit loss-based feature separation, strategies to reduce SDA complexity, and distinctions from prior disentanglement methods.

Given its originality, strong empirical validation, and effective response to reviewer concerns, the paper meets NeurIPS standards for technical rigor and impact.
Decision: Accept (poster)