Title: Plug-and-play Feature Causality Decomposition for Multimodal Representation Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Representation Learning

Causal Inference

Multimodal Learning
Abstract: Multimodal representation learning is critical for a wide range of applications, such as multimodal sentiment analysis. Current multimodal representation learning methods mainly focus on the multimodal alignment or fusion strategies, such that the complementary and consistent information among heterogeneous modalities can be fully explored. However, they mistakenly treat the uncertainty noise within each modality as the complementary information, failing to simultaneously leverage both consistent and complementary information while eliminating the aleatoric uncertainty within each modality. To address this issue, we propose a plug-and-play feature causality decomposition method for multimodal representation learning from causality perspective, which can be integrated into existing models with no affects on the original model structures. Specifically, to deal with the heterogeneity and consistency, according to whether it can be aligned with other modalities, the unimodal feature is first disentangled into two parts: modality-invariant (the synergistic information shared by all heterogeneous modalities) and modality-specific part. To deal with complementarity and uncertainty, the modality-specific part is further decomposed into unique and redundant features, where the redundant feature is removed and the unique feature is reserved based on the backdoor-adjustment. The effectiveness of noise removal is supported by causality theory. Finally, the task-related information, including both synergistic and unique components, is further fed to the original fusion module to obtain the final multimodal representations. Extensive experiments show the effectiveness of our proposed strategies.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel module, Feature Causality Decomposition (FCD), designed to enhance multimodal representation learning by explicitly modeling and separating synergistic, unique, and redundant components of unimodal features. Drawing from principles of causal inference, particularly backdoor adjustment, the method aims to remove modality-specific task-agnostic noise while preserving task-relevant information. FCD integrates two core submodules: Causality Components Decomposition (CCD) and Synergistic Distribution Alignment (SDA). The proposed method is plug-and-play and can be inserted into existing intermediate fusion models without structural modifications. The authors evaluate FCD on five benchmark datasets across various tasks and report consistent performance improvements over several state-of-the-art (SOTA) methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The authors provide a theoretical framework connecting backdoor adjustment (from causal inference) with mutual information optimization, supporting their decomposition strategy.\n\n* Applying causality to multimodal learning is an interesting trend, and the proposed study is well motivated.\n\n* Experiments on conventional multimodal learning datasets (CMU-MOSI, MOSEI, Food101, etc.) and base models. Ablation studies and Grad-CAM visualizations add interpretability and support the claimed benefits of each component.\n\n**Weaknesses**\n\n* Despite the sufficient motivation, the topic of feature disentanglement on backdoor adjustment is not novel. See [1] for an example of similar work on backdoor adjustment on graph representation learning. Therefore, the methodological contribution is limited. \n\n* While the paper leverages causal language and theoretical constructs, the actual role of causality remains largely conceptual. The core assumption—that modality-specific noise can be removed via backdoor adjustment—is not empirically validated. The SCM and do-operator usage is not grounded in real causal discovery or data-generating processes.\n\n* Following the previous comment. The assumed causal structure (e.g., the existence and confounding role of modality-specific features) is not validated via data-driven methods such as causal discovery, nor is any synthetic experiment conducted to test causal robustness. This undermines the practical credibility of the theoretical claims.\n\n* Scalability is limited for empirical evaluation as the authors only performed experiments on CMU---MOSI-like datasets, which are small in scale. Empirical evaluations on larger-scale datasets (e.g., MIMIC-CXR) are encouraged to demonstrate the scalability\n\n[1] Sui, Yongduo, et al. \"Causal attention for interpretable and generalizable graph classification.\" Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. 2022.\n\nQUESTIONS:\n* Could the authors clarify whether this SCM was constructed based on empirical data, prior knowledge, or intuition? Additionally, can you discuss how sensitive your method is to potential misspecifications of this causal structure?\n\n* Empirical evaluations on larger-scale datasets (e.g., MIMIC-CXR) is encouraged to demonstrate the scalability\n\nLIMITATIONS:\nThis paper specified in the checklist that they discussed limitations but it was not found in the main text nor in the appendix. From what I can see the scalability is a significant limitation", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a plug-and-play Feature Causality Decomposition (FCD) framework for multimodal representation learning. The method performs a two-stage decomposition process: first, it separates unimodal features into modality-invariant and modality-specific components; second, the modality-specific component is further decomposed into unique and redundant parts using a backdoor-adjustment mechanism grounded in causal inference. Extensive experiments are conducted on two benchmark datasets for multimodal sentiment analysis (MSA), as well as three additional datasets, to validate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. The paper addresses a critical problem in multimodal representation learning—namely, the challenges of redundancy and conflict across modalities. Effectively decomposing multimodal representations remains a central research challenge, and the proposed method directly tackles this issue.\n\n2. The proposed approach is conceptually straightforward. By introducing an additional FCD loss to guide the learning process, the method integrates seamlessly with existing architectures and enhances performance without introducing significant complexity.\n\n3. The experiments are comprehensive and convincing. The use of CMU-MOSI and CMU-MOSEI—two widely adopted benchmarks in multimodal sentiment analysis (MSA)—demonstrates the effectiveness of the method in standard settings. Moreover, the evaluation on additional classification tasks further supports the generalizability of the approach, which is a notable strength.\n\nCons:\n\n1. Potential missing citation for Theorem 4.1: It appears that Theorem 4.1 is derived from prior work, but no citation is provided. Proper attribution is important for clarity and academic integrity.\n\n2. There are several textual and formatting typos throughout the paper. For example, line 267 mentions five methods, while Table 2 only presents four. Additionally, there are typographical errors in lines 290 and 318. These inconsistencies may affect the readability and perceived rigor of the work.\n\n3. The ablation study and case study are conducted on different datasets, which may reduce the coherence of the analysis. It would strengthen the paper to conduct both studies on the same dataset. For instance, providing a case study on the MOSI dataset—similar to Figure 5—could offer more consistent and interpretable insights into the effectiveness of each component.\n\nQUESTIONS:\n1. How to introduce the FCD to the existing work, given that previous work has its unique design. Which layer or block is for the 5 methods for MSA? Is there some extra effort to tune the hyperparameters? If so, what is the motivation behind tuning the hyperparameter?\n\n2. Whether the number of modalities affects the performance or the use of FCD?\n\n3. What is the difference between FCD and methods following the disentanglement paradigm? like DMD[1] and its follow-up papers.\n\n[1] Decoupled Multimodal Distilling for Emotion Recognition, CVPR 2023.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a plug-and-play feature causality decomposition method for multimodal representation learning, addressing the limitations of existing approaches that fail to distinguish between complementary information and aleatoric uncertainty noise within modalities. The method disentangles unimodal features into modality-invariant (synergistic information shared across modalities) and modality-specific components, with the latter further decomposed into unique and redundant features using backdoor adjustment to eliminate noise while preserving task-relevant information. The resulting synergistic and unique features are then integrated into existing fusion modules, with causality theory supporting the noise removal process. Extensive experiments demonstrate the effectiveness of this approach in enhancing multimodal representations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThis article introduces a novel Structural Causal Model (SCM) assumption for multimodal data, providing an innovative approach to feature representation. The proposed method includes a feasible feature disentanglement technique, which is rigorously supported by both theoretical analysis and experimental validation. The framework demonstrates strong potential for advancing multimodal data modeling.\n\nWeaknesses:\nWhile the proposed SCM assumption is promising, the article lacks a detailed rationality analysis of this assumption. Additionally, the presentation could be refined to enhance clarity and precision. Further improvements in explanatory depth and linguistic expression would strengthen the work.\n\nQUESTIONS:\n1. The size of this plug-in module appears to be relatively large, with a considerable number of parameters. Typically, plug-in modules should maintain a compact design without excessive parameters. Could you clarify how the parameter scale of the proposed FCD module compares to that of the base model?\n\n2. The author should clearly explain why the URD is a measure-preserving bijective function and what \"measure-preserve\" specifically refers to in this context. In mathematics, \"measure\" has a strict definition. What exactly does \"measure\" refer to here?\n\n3. The rationality of the assumed information model in Fig. 1 requires further elaboration. Specifically, it should be clarified whether common multimodal data aligns with the proposed SMC model. The authors could enhance the justification by providing concrete examples to illustrate the model's validity. For instance, in image-text multimodal tasks, which factors are task-related, which are synergistic features, and what are the other features mentioned in Fig1.\n\n4. The motivation and necessity of SDA are not sufficiently justified. Is such a complex design truly necessary for SDA? If replaced with a simpler strategy—for instance, using a shared MLP across all modalities—would the performance degrade significantly?\n\nAdditional comments:\n\n1. Some typos: line 226 and 338\n\n2. The authors should more clearly and directly clarify what kind of claim or what kind of conclusion their theorem supports.\n\n3. The font of Definition should be different from the main contexts.\n\n4. For the case study, it is suggest to select and analyze challenging cases where prior methods proved ineffective. By demonstrating that the proposed approach succeeds in these scenarios—particularly when aligned with the SCM hypothesis—we can highlight its critical role. Specifically, these cases should strongly support the hypothesis, illustrating that without it, the model would fail. Conversely, if the hypothesis is not rigorously validated, there remains a risk that the method's effectiveness could be coincidental rather than causally linked to the proposed mechanism.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a plug-and-play module named \"Feature Causality Decomposition (FCD),\" which aims to address a core challenge in multimodal learning: how to differentiate and utilize valuable complementary information while simultaneously eliminating modality-specific noise. To achieve this fine-grained decomposition, FCD primarily relies on several key modules and constraints. The Causality Components Decomposition (CCD) module first performs an initial separation of the raw feature z into a modality-invariant part (i.e., the synergistic feature s) and a modality-specific part h. The Synergistic Distribution Alignment (SDA) module then employs pairwise Sinkhorn divergence to align the s features from all modalities to enforce their consistency. The most critical step occurs in the processing of the modality-specific part h. The model applies backdoor adjustment theory, using a Unique Redundant Decompose (URD) submodule to further decompose h into u and r. This process is guided by a mutual information-based loss function, L_MI, which aims to maximize the association between u and the task label, thereby isolating task-irrelevant components into r. Experiments on multiple datasets and existing models demonstrate the effectiveness of this decomposition strategy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper addresses an important and subtle problem in multimodal learning: how to distinguish modality noise from effective complementary information. Its core contribution lies in moving beyond the traditional shared-specific binary decomposition. It introduces a causal inference framework to propose a more fine-grained, three-part decomposition (synergistic, unique, redundant). \n\n2. The paper's methodology is well-designed and technically sound. The model's architecture is clearly defined (e.g., CCD, SDA, URD modules), and it employs mature techniques from the field (e.g., Sinkhorn divergence, InfoNCE loss) to achieve its objectives. The experimental validation is thorough; by applying the FCD module to 9 different state-of-the-art models across 5 standard datasets, the paper provides substantial empirical evidence for the method's effectiveness and its \"plug-and-play\" generalization ability. \n\n3. The paper is well-structured and clearly written. The authors effectively use diagrams (such as the causal graph and model architecture) to illustrate their complex theoretical model and workflow, which greatly aids the reader's understanding of the core ideas.\n\nWeaknesses:\n\n1. A main concern of the method is that its separation of synergistic (s) and specific (h) features relies entirely on the indirect guidance of downstream loss functions. The model lacks an explicit mechanism (e.g., an orthogonality constraint or a mutual information minimization loss) to actively prevent information leakage between these two branches. Therefore, the \"purity\" of the decomposition beomes a core assumption that is not directly verified, which to some extent undermines its theoretical rigor.\n\n2. Although the experimental coverage is broad, it still needs further statistical analysis. All performance results are reported without standard deviations or significance tests. This makes it difficult to determine to what extent the reported performance gains are stable and reproducible, rather than being a result of random fluctuations during model training.\n\n3. The design of the Synergistic Distribution Alignment (SDA) module has a notable limitation. As it relies on computing pairwise Sinkhorn divergence between modalities, the computational complexity may grow rapidly.\n\nQUESTIONS:\n1. The paper reports consistent performance improvements across multiple baseline models. To assess the stability of these results, could the authors clarify whether the data in the tables are based on single runs or are averages of multiple independent experiments? Reporting the mean and standard deviation over several runs is necessary to establish the statistical significance of the results.\n\n2. Can the feature decomposition guarantee that h contains only specific information and s only synergistic information? The model uses indirect loss functions to guide the information separation between the s (synergistic) and h (specific) branches. This raises a question: is this mechanism sufficient to guarantee that there is no information leakage between the two? For example, is it possible for the modality-specific branch h, in order to optimize the final task, to also learn and encode some cross-modal synergistic features? Without an explicit constraint to prevent such cases in the current model, how can the purity of the decomposition be guaranteed?\n\n3. The data in Appendix F demonstrates the computational overhead of FCD for 2-3 modalities. Considering that the L_SDA loss is based on pairwise comparisons, its complexity grows quadratically (O(M^2)) with the number of modalities M. How is the scalability of the method envisioned for scenarios with many more modalities (e.g., M > 5)? Would this growth in computational complexity become a bottleneck for its application?\n\n4. The L_MI and L_Dis loss functions used in the model are essentially contrastive learning, whose effectiveness is often related to the number of negative samples provided by the batch size. The paper does not include a sensitivity analysis for this hyperparameter. Could the authors provide the specific batch sizes used in the experiments and comment on the FCD's performance sensitivity to this parameter?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}