Title: Amplifying Prominent Representations in Multimodal Learning  via Variational Dirichlet Process
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal learning

Bayesian nonparametric methods

medical images
Abstract: Developing effective multimodal fusion approaches has become increasingly essential in many real-world scenarios, such as health care and finance. 
The key challenge is how to preserve the feature expressiveness in each modality while learning cross-modal interactions.
Previous approaches primarily focus on the cross-modal alignment,
while over-emphasis on the alignment of marginal distributions of modalities may impose excess regularization and obstruct meaningful representations within each modality.
The Dirichlet process (DP) mixture model is a powerful Bayesian non-parametric method that can amplify the most prominent features by its richer-gets-richer property, which allocates increasing weights to them.
Inspired by this unique characteristic of DP, we propose a new DP-driven multimodal learning framework that automatically achieves an optimal balance between prominent intra-modal representation learning and cross-modal alignment. 
Specifically, we assume that each modality follows a mixture of multivariate Gaussian distributions and further adopt DP to calculate the mixture weights for all the components. This paradigm allows DP to dynamically allocate the contributions of features and select the most prominent ones, leveraging its richer-gets-richer property, thus facilitating multimodal feature fusion.
Extensive experiments on several multimodal datasets demonstrate the superior performance of our model over other competitors.
Ablation analysis further validates the effectiveness of DP in aligning modality distributions and its robustness to changes in key hyperparameters.
Code is anonymously available at https://github.com/HKU-MedAI/DPMM.git


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Dirichlet Process-based multimodal learning framework (DPMM) that adaptively balances intra-modal representation and cross-modal alignment. By modeling each modality as a Gaussian mixture and using the richer-gets-richer property of the DP to weight features, the method emphasizes the most informative components. Experiments show DPMM outperforms existing approaches, with ablation studies confirming the robustness and effectiveness of the DP mechanism.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe proposed approach is conceptually novel, leveraging a shared Dirichlet Process prior to implicitly align multimodal representations.\nWeaknesses:\n1. Unclear Exposition: The lack of detailed mathematical formulations significantly hinders the reader’s understanding of the model design and underlying assumptions.\n2. Unclear Theoretical Justification of “Implicit Alignment”: The claim that the DP structure can \"automatically achieve an optimal balance between intra-modal representation and cross-modal alignment\" lacks theoretical support. While the DP prior emphasizes frequent components within each modality, it does not inherently ensure alignment across modalities. Without a shared latent space or explicit alignment objective, the observed alignment appears incidental rather than structurally guaranteed, and may not generalize well across datasets or domains.\n3. Missing Comparison with Explicit Alignment Models:The experiments do not include comparisons with state-of-the-art alignment methods such as contrastive multimodal models (e.g., CLIP, ALIGN, or CMML), which include direct semantic matching between modalities. This makes it difficult to assess whether the proposed implicit mechanism is truly competitive.\n\nQUESTIONS:\n1. Could you clarify how \"implicit alignment\" is achieved in your framework without a shared latent space or explicit alignment objective? As currently presented, the alignment appears to be an incidental byproduct of training rather than a mechanism that is theoretically or structurally grounded.\n2. Would you consider adding experiments to directly validate that the DP structure facilitates cross-modal alignment, rather than simply enabling modality fusion? Clear empirical evidence is needed to distinguish alignment from general feature integration.\n3. Please consider including comparisons with recent state-of-the-art explicit alignment models, including those published after 2024. This would provide a more convincing evaluation of how the proposed implicit alignment method performs relative to current best practices in cross-modal learning.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel Dirichlet Process Mixture Model-based multimodal learning framework that addresses the balance between intra-modal feature expressiveness and cross-modal alignment. By leveraging the richer-gets-richer property of the Dirichlet Process, the model is designed to amplify the most prominent features across modalities while still enabling effective modality alignment. The framework is validated on multiple multimodal datasets, including large-scale clinical datasets and general multimodal benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- I think the use of the Dirichlet Process to dynamically allocate feature importance across modalities is an innovative angle that contrasts with typical cross-modal alignment methods that often suppress intra-modal feature strength. The probabilistic formulation with gradient-preserving sampling provides a natural way to handle missing modalities without relying on simple imputations or discarding data.\n- I appreciate that the model is validated across four multimodal datasets, including both healthcare and general-purpose benchmarks, providing a convincing demonstration of generalizability.\n- Through ablation, the paper explores the role of key components (alignment losses, fusion modules, concentration parameters), strengthening confidence in the design choices.\n\n### Weaknesses\nDespite the compelling contributions, several aspects of the paper need further attention to fully validate its claims.\n- The disentanglement between the effects of alignment and feature amplification is not sufficiently isolated, leaving it unclear whether performance gains stem primarily from the DP-driven amplification or from improved alignment.\n- The baseline selection for missing modality comparisons is limited, with some relying on simplistic zero-imputation, which underrepresents more competitive recent methods like generative imputers or retrieval-based completion.\n- While the theoretical background on the Dirichlet Process is sound, the paper does not rigorously justify why the richer-gets-richer property is always optimal for multimodal tasks, especially when task-relevant signals may not necessarily be statistically dominant. \n- It seems to me that the qualitative analysis remains superficial. A deeper, more interpretable feature analysis is needed to verify that the features amplified by the DP process are semantically meaningful and not just artifacts of the probabilistic modeling.\n\nQUESTIONS:\nThe questions below are related to what I mentioned in the strengths and weaknesses section.\n\n1. Can you provide a more rigorous ablation to disentangle the effects of alignment and amplification?\n2. Can you compare DPMM to more competitive missing modality baselines?\n3. Can you provide more interpretable qualitative examples?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Dirichlet-Process Mixture Multimodal Learning (DPMM) to avoid representation degradation caused by excessive inter-modality alignment in multimodal learning. It treats the latent representation distribution of each modality as a Gaussian-mixture model (K components) and generates the mixing ratios via a Dirichlet process with a stick-breaking construction, leveraging its rich-get-richer property to emphasize important features.\nFor missing modalities, gradient-preserving sampling (GPS) performs probabilistic imputation while keeping gradients intact.\nThe proposed method outperforms state-of-the-art approaches on four datasets, including clinical (MIMIC-III/IV) and general-domain (CMU-MOSI, POM) benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is very well structured and easy to read.\n- The method makes effective use of the rich-get-richer characteristic of the Dirichlet process, and the proposed GPS mechanism for handling missing modalities is markedly more effective than simple imputation, representing a substantial contribution to the community.\n- Experiments show consistently strong performance compared with the latest multimodal integration methods, validating the effectiveness of the proposal.\n\nWeaknesses:\n- As the authors themselves note, constraining each mixture component’s covariance matrix to be diagonal may limit expressiveness for modalities with strong cross-feature correlations. The authors should clarify the range of multimodal data for which this assumption remains valid.\n- Although the method claims to scale via the Dirichlet process and stochastic variational inference (SVI), there is no discussion (or quantitative evidence) of potential computational bottlenecks introduced by the stick-breaking process. Including such discussions or comparing training times with existing methods would strengthen the practical impact.\n\nQUESTIONS:\n- I would like the authors to address the weaknesses I pointed out.\n- Appendix E shows that replacing the Dirichlet process with learnable but fixed mixture weights yields only modest performance gains. This raises the question of how the results would fare on other datasets.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}