Amplifying Prominent Representations in Multimodal Learning  via Variational Dirichlet Process
Authors: Tsai Hor Chan
Feng Wu
Yihang Chen
Guosheng Yin
Lequan Yu
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal learning
Bayesian nonparametric methods
medical images
Primary area: general_machine_learning
TL;DR: We propose a novel DP-driven multimodal learning framework that automatically optimizes the balance between prominent intra-modal representation learning and cross-modal alignment.

Abstract: Developing effective multimodal fusion approaches has become increasingly essential in many real-world scenarios, such as health care and finance. 
The key challenge is how to preserve the feature expressiveness in each modality while learning cross-modal interactions.
Previous approaches primarily focus on the cross-modal alignment,
while over-emphasis on the alignment of marginal distributions of modalities may impose excess regularization and obstruct meaningful representations within each modality.
The Dirichlet process (DP) mixture model is a powerful Bayesian non-parametric method that can amplify the most prominent features by its richer-gets-richer property, which allocates increasing weights to them.
Inspired by this unique characteristic of DP, we propose a new DP-driven multimodal learning framework that automatically achieves an optimal balance between prominent intra-modal representation learning and cross-modal alignment. 
Specifically, we assume that each modality follows a mixture of multivariate Gaussian distributions and further adopt DP to calculate the mixture weights for all the components. This paradigm allows DP to dynamically allocate the contributions of features and select the most prominent ones, leveraging its richer-gets-richer property, thus facilitating multimodal feature fusion.
Extensive experiments on several multimodal datasets demonstrate the superior performance of our model over other competitors.
Ablation analysis further validates the effectiveness of DP in aligning modality distributions and its robustness to changes in key hyperparameters.
Code is anonymously available at https://github.com/HKU-MedAI/DPMM.git

Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process
Tsai Hor Chan"
Feng Wu" University of Hong Kong fengsu96@connect. hku hk
University of Pennsylvania
Tsaihor. Bean@PennMedicing upenn. edu
Yihang Chen
University of Hong Kong
yihangc@connect.. hku hk
Guosheng Yin University of Hong Kong gyin@hku. hk
Lequan Yu
University of Hong Kong 1qyu@hku hk
Abstract
Developing effective multimodal fusion approaches has become increasingly sential in many real-world scenarios, such as health care and finance. The key challenge is how preserve the feature expressiveness in each modality while learning cross-modal interactions. Previous approaches primarily focus on the cross-modal alignment, while over-emphasis on the alignment of marginal distri- butions of modalities may impose excess regularization and obstruct meaningful representations within each modality. The Dirichlet process (DP) mixture model is powerful Bayesian non con-parametric method that can amplify the most prominent features by its richer-gets gerr---eet-rrrrrr property, which allocates increasing weights them. Inspired by this unique characteristic of DP, we propose new DP-driven multimodal learning framework that automatically achieves an optimal balance between prominent intra-modal representation learning and cross-modal alignment. Specifically, we assume that each modality follows mixture of multivariate Gaus- sian distributions and further adopt DP to calculate the mixture weights for all the components. This paradigm allows DP to dynamically allocate the contributions features and select the most prominent ones, leveraging its richer- seter--gts-riccee prop- erty, thus facilitating multimodal feature fusion. Extensive experiments on several multimodal datasets demonstrate the superior performance of model over other competitors. Ablation analysis further validates the effectiveness of DP aligning modality distributions and its robustness changes in key hyperparameters. Code is anonymously available at wttttps:// /tttps:///ithbb..
git
Introduction
Multimodal learning aims aggregate information from multiple modalities to generate meaningful representations for downstream tasks. It has been widely explored in the context of vision-language
"Equal contribution.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

models [11, 9]. vidio-visual applications S 35 20 image-video models [ 12 and healthcare applications [50 18. For example, multimodal learning has been applied to various healtheare tasks such as clinical outcome prediction 61 50 report generation 42 4 and clinical trial
site selection 45] One of the main tasks multimodal learning to fuse information from different modalities, which can be divided into early, joint, or late fusion strategies [21]. The joint fusion strategy is the most popular multimodal fusion paradigm [21] due to its strong ability to capture structural inter-modality interactions. It aims to align the distributions of different modality representations, creating a unified representation that effectively captures these interactions and preserves meaningful cross-modal information for downstream prediction tasks [8]
However, due to the inherent heterogeneity of different modalities (e.g., medical images, med- ical reports, electronic health records), properly aligning the distributions of various modalities for fusion remains a challenging problem. De- spite the success of existing alignment strategies, such as contrastive learning [47] and adversarial alignment [31] most of them primarily focus on aligning features while neglecting the important intra-modality representations within each modal- ity. As a result, this leads to weaker fused rep- resentations, ultimately limiting the performance of downstream tasks and reducing the generaliz- ability and robustness of multimodality models. Consequently, there is strong need for a novel approach that can not only effectively align the fea- ture distributions but also preserve the important intra-modal representations.
Tast-Falass Teaur
B
Heckery
Figure 1: Dirichlet process based modality align- ment (top right) vs. conventional modality align- ment (bottom right). The Dirichlet process al- locates more weights to capture the prominent features (c.g.. *, 4) while conventional methods inevitably shrink the intra- features during alignment.
Bayesian nonparametric methods have shown great success in representation learning due their flexibility and capability in density estimation. Specifically, the Dirichlet process (DP) has been frequently adopted since it can highlight the most significant features in the covariates with its richer-gets-richer property [60]. Figure presents an illustrative example of how DP can amplify the feature signals for each modality. Despite its capability, the potential of DP in tackling multimodal representation learning is still under-explored. Moreover, the issue of unpaired observations (i.e., missing observations on some modalities) presents significant challenge in multimodal learning. Most existing methods operate under the assumption that all modalities are available for every ob- servation. However, in real-world scenarios, is common for certain modalities, such as medical images or reports, to be absent for some patients due to clinical or administrative reasons in health care (54). The existing solutions either discard these observations or impute simple values [18] (c.g.. zeros) to address the missing modality problem. However, the feature signals in each modality after learning the alignment objectives are shrunk, leading to weaker feature representations and worse imputation performance. Therefore, it is necessary to amplify the intra- modall representations to generate representative features for the observations with unpaired observations.
In light of the above challenges, we propose a novel Dirichlet process-driven multimodal learning framework, namely DPMM (Dirichlet Process Mixture Multimodal Learning), to tackle the multimodal fusion paradigm from a probabilistic perspective. We formulate the multimodal feature distribution as a Gaussian mixture model whose weights are allocated by DP, such that prominent features can receive increasing attention. Our contributions can be summarized as: (1) We introduce novel multimodal learning framework via the Dirichlet process to amplify the feature signals while properly aligning them across modalities. We treat the joint distribution as a DP mixture model to allow DP to select the most useful features using its setherr-ett-richer property. (2) We adopt stochastic variational inference to efficiently optimize the proposed Dirichlet process model, which overcomes the scalability limitations of traditional optimization algorithms based on Markov chain Monte Carlo (MCMC) and enables the scalability of our model to large-scale datasets. (3) We utilize the learned marginal distributions as the representation generator to accurately impute the missing observations with the prominent features highlighted by DP. This enhances the features supplied to training and downstream tasks and hence leads to improved performance. (4) Empirical results on four

---

Modulity Aligument
Modality Fasion
Objective Function
Variational Teaining
Mealaay at Sealtey Medally
Encoder
Tack-sperifie
Concatconnt
Less
Modale
MLP
Fncader 2
ELBO
Negative
Lag:
Encoder M
FM
Gradiont Sampling
21
22
ZM
Mictue Wrights
Roum Pait (uolt-
Mhxdd Distribution
[1-Ao]
Stick-breaking Process
Mixture Gaussians
Dirichlet Process Alignment
Figure Overview of our proposed DPMM framework. We first imput the data to the encoders obtain latent embeddings 2,...., MM We then compute the mixture weights {Tmx m é [M]] k [K]} using the stick-breaking formulation of the DP. The weighted likelihood then computed with
the mixture weights and the multivariate Gaussian mixture assumption. Finally, by integrating the task-specific loss, the evidence lower bound (ELBO) for training the encoders can obtained. To
handle unpaired samples during training, we adopt gradient- podieent-reseevvng sampling which can impute the missing observations while the gradient can still backpropagated.
multimodal datasets demonstrate the superior performance of our framework and ablation analysis further corroborates the effectiveness DP in modality alignment and feature amplification.
Related Works
Cross-modal Alignment. Cross- alignment aims to make representations of different modali- ties comparable and aligned a shared latent space 22 [22] and has been widely applied in various multimodal tasks, such image-text retrieval 42 image captioning (63) and multimodal classifi- cation [10]. Some existing works focus on learning a shared representation using fusion strategies, such concatenation
45 without alignment. Despite being effective for certain tasks, these methods may yield suboptimal results when dealing with highly heterogeneous modalities
[18 46 transformer-bas based mechanisms [61]
(54) To address this issue, some works use contrastive learning align
representations by maximizing the similarity between different modalities [4 48). Other studies variational inference methods 44 62 10], such variational autoencoders (VAEs), align
representations by minimizing the Kullback -Liblerr (KL) divergence between latent distributions of different modalities. However, these methods often emphasize the shared information between modalities, which may compromise the preservation of crucial poodality-specinee features that are necessary for downstream tasks.
Deep Bayesian Nonparametric Methods. Bayesian nonparametric methods, including Gaussian 16
process (GP) 440 30
27 4T, Dirichlet process (DP) 60 23 2]. and Pólya tree [44 36),
have shown significant success in parameter estimation as they perform searching in an infinite dimensional parameter space. Among these methods, DP has been extensively applied clustering and density estimation for its picher-gets-richee
property. In light of the rapid development of deep learning, many works [43 2 16] introduce variational inference to render traditional Bayesian nonparametric methods to large scale. However, the capability of deep DP in multimodal learning still undervalued. Recently, there are some works applying DP to multimodal learning [3] 53) For example, MoM-HDP [53] is a multimodal topic model via a hierarchical Dirichlet process which

---

extends its latent Dirichlet allocation (LDA) counterparts [ However, these methods are limited topic modelling problems, where their assumptions are difficult to be generalized other scenarios.
Learning with Missing Data. Traditional multimodal learning assumes all modalities are available, while in reality, some observations may be missing, e.g., missing medical images or reports in clinical data. Late fusion is common strategy to address missing modalities by aggregating predictions [56] or latent space representations [45] from the available modalities. Despite its effectiveness, late fusion treats each modality independently and lacks interactions between them. Some existing works focus on extracting shared information across modalities [6 (6(54]88) However, learning such shared representations can be challenging, particularly when the modalities are highly different in distributions (e.g.. tabular data and image data). Many approaches attempt to preserve model performance via modelling the relationships between them (59 50] or generating global representation for the missing data [18]. Other methods assume that the missing modality follows a certain distribution, imputing the missing values using the mean or mode of that distribution [63]. Despite their successes, the distributional assumptions emphasize more on the structural interactions between the modalities, while highlighting the prominent features signals is less considered. The resulting feature generator would produce weaker signals for missing modalities, which leads to less representative features.
3 Methodology
3.1 Preliminaries
Dirichlet Process. Denoted DP G), the Dirichlet process is random probability measure on the sample space X, such that for any measurable finite partition S of X, denoted S (B.) K
(BB)),
~ Dir(1,G( ( (,G(,)(())))))
where G is the base probability measure, 7) is the concentration parameter and Dir(-) denotes the Dirichlet distribution.
Multimodal Learning. Given the multimodal training dataset Dur {(x
(9) x it . H
where an (for ,...., M) denotes the i-th observation of the m-th modality and ())) is the corresponding label, the goal is to train multimodal neural network fe(() with parameter O such that the model achieves optimal performance in downstream tasks.
3.2 Multimodal Learning via Dirichlet Process
In our multimodal learning framework, we formulate the multimodal feature distribution as Gaussian mixture model whose weights are allocated by DP. The DP allocation can ensure that prominent features receive increasing attention. An overview of the proposed DPMM is provided by Figure 2 and the detailed computation steps given in Algorithm [T
Steck-bre Process for Mixture Weights. We define the Dirichlet process multimodal learning framework mixture of M K components, where each mixture is assumed to multivariate Gaussian distribution. We adopt the popular stick-breaking process to formulate the prior distribution of the weights as follows,
Bmn Beta(1,n).
Tink = Bmnk
II (1 (1- (1-Bg) [I BBx). ic[M]jckk
(1)
where Tink is the probability assigned to the k-th mixture component of the m-th modality. Figure provides a graphical illustration of the stick- stick-breaking process under the multimodal context.
By adopting a stick-breaking process to select the mixture weights, the DP can automatically select the most prominent features among the modalities by its -ehhger property.
Gaussian Mixture as Marginal Distribution. To enable more flexible assumption of the feature representation at high dimensions, assume the feature distribution of the m-th modality follows

---

K K-mixture of multivariate Gaussian mixture model (GMM).
(=) Σ (=/ Emk), m 1, 1,..., M,
where Tmk is the mixture weight, Pank is the mean vector, and Emk is the covariance matrix of the k-thh mixture component of the m-th modality. The GMM is a common technique in modelling high-dimensional distributions [29 55 g. Let μ {Mmmk m e [M].k k [K]] and E {E
m [M], [K]]- We set μ and Σ as learnable parameters to allow them to be updated by
gradient back- propagation. We further let 0 {ne, Σ). For computational efficiency, we adopt the parameterization trick [8), which assumes each mm as diagonal matrix.
With the density function for the m-th modality fm defined by (2), the joint distribution is
K M
F(z)=
Σ
Tmn Fm Fm (zi Mmnki Emn),
where the mixture weights given in Eq. [ and (2: Hmnks mmn) is the distribution function of the k-th mixture of the m-th modality. Our probabilistic model in the plate notation is presented in Figure 4
Wat PI1
Fin
Te Pn(11 P1s)
M31 Psa(1
(1)) fzx)
Pas
1 Pax
Amnx
Was Psa(1 - Mss) (2))l1 - Bar)
Piz Pa
Pao
Lamppa (1 Pv) (1- Pi))
fm
TimE
Pink
11
Figure 3: Illustration of the stick-breaking pro- cess under the multimodal context. We perform splitting by modality first and then by mixture components. This allows the DP to allocate weights to the most prominent features from each modality.
Kemnk
Xm
Figure 4: Plate notation of our proposed DPMM model.
Discussion. By modelling the multimodal feature distribution as DP mixture model, the DP can allocate increasing attention to useful feature components mkk. This highlights the most significant contribution of each modality m. On the other hand, the Gaussian components' covariances capture anisotropy (Mahalanobis geometry), aiding gross-modas structure learning. Hence our method can inherently optimize the tradeoff between highlighting prominent intra-modality features and effective cross-modal alignment.
Expressiveness Under Truncation. A DP can be truncated at K without losing its expressiveness. Ishwaran and Zarepour (23 propose represent a DP model with finite mixture models.
Theorem 1. (Ishwaran and Zarepour (23)) Given the base distribution of Dirichlet process for every measurable function f integrable with respect to G, as K co, we have
f f(0)dG" (0) D, f f()))G(().
where is approximated by an infinite mixture and GK is the DP mixture model with finite K mixtures.
One major insight from this theorem the guaranteed convergence the posterior distribution the representation under the DP mixture. Even if we truncate the mixture at K, the learned distribution still preserves the property as an infinite number of mixtures is used. The DP can best estimate the behaviour of embeddings in each modality and generate expressive features.
5

---

Algorithm Sampling algorithm of our proposed DPMM framework.
Input: Multimodal encoder (-) with parameter set θ concentration parameter 17
1: Means and covariances of Gaussian distributions {Mmm, Emk), m 1..... M, h 1, ......K]
(1) 2: Training Drr { {=( ..... M (0) Hi=1
Output: Trained fe
3: Initialize phnk and Link
4: for (ar) ,...,. . in Drr do
(1)
5: Sample Tink with Eq. (
6: g(O)= = (x) with x0 (2)
> Forward propagation
M
7: Compute task-specific loss Cati with g( and y()
8: Compute the KL(q pp) and hence the ELBO
9:
Bropaproe the ELBO to update ② pr, end for Return: Trained fe
11:
3.3 Stochastic Variational Inference
To render our method to a large scale, we adopt stochastic variational inference to estimate the parameters the DP mixture. The variational family specified as
(B3 g00
II (mmk) m= k=
II
I 1o
where q()) is the variational posterior (an approximating distribution) chosen to be close to the intractable true posterior. The evidence lower bound (ELBO) is given by
ELBO (K(((()))p(() (L((( (p((())))))))))))))))))))))))))) Σ (L(
(3)
where (L(((y(0) (((())))))))))) can be interpreted as the task-specific loss for the i-th sample. By adopting stochastic variational inference, we can treat the likelihood from the assumed probabilistic model as the surrogate loss for backpropagation, and hence the DP mixture model becomes feasible for multimodal learning in high dimensions.
Variational updates of the posterior mixture weights. We detail the closed-form updates for the variational posteriors associated with the stick-breaking weights. Let Ym,m, Y2.m IR* respec- tively denote the shape parameters of the Beta distributions the variational factors (B3mk) Beta( (m,k,, 72.mk) for modality m. Minimizing the KL terms yields
MK Σ
mimk 1 Σ Di.nkr
72.mk 11
dir
1-1 1=1r-mbe-
where 7/ is the concentration rate, Φ RR collects the posterior responsibilitiess (i.e., posterior unnormalized weights) over all M K components for sample i. The rog-rssponss
log dinnk E. log Tin]
E [o HHHH)))+HHHH H(qgma (- ( 2; (m, k), x (²)) const,
(5)
followed by normalization -M 6i, = for each i. Here, Tmk are the mixture weights implied by the variational Beta factors, HH is the entropy function, is the indicator function of a mixture component, (1) (ar) ..... x a ) the observed input, and dv.... denotes the component-condiiional variational distribution with parameters v/m
3.4 Missing Modality Imputation
Our DPMM can inherently impute the missing modalities probabilistic framework. While adopting the common missing at random (MAR) assumption, we obtain the marginal distributions F,,..., Far from the joint distribution F. For each x with missing observation the m-th modality, sample from the Gaussian mixture distribution, .x(()) F.. In this way, the sampled embeddings contain both cross cross-modal information and prominent intra-moda information.

---

Experiments
4.1 Datasets and Experimental Settings
Datasets and Preprocessing- We evaluate the performance of our DPMM on two multimodal large-scale clinical
and MIMIC-IV, following the previous work (18]. As CXR images are not available in MIMIC-III, we replace them with clinical notes serving as the second modality. We extracted 25,071 ICU stays with EHR records from MIMIC-IV, 5,931 of which are matched to CXR images and reports. Similarly, we extracted 21,139 ICU stays with EHR records from MIMIC-III, with 5,273 stays matched clinical notes. To evaluate the performance of DPMM on cross-modal alignment, we conduct experiments on to- tally matched bi-modal and tri-modal settings. We also evaluate the performance on partially matched datasets to demonstrate the robustness of DPMM in the presence of missing modalities.
Moreover, to show the general- ization of our framework, we also conduct experiments on
Table I: Summary of the real datasets.
Dataset
No. Train
No. Valid
No. Test
No. Total
CMU-MOSI and POM datasets following the implementations of
Complete Datasets
MIMIC-II III
14,681
3,222
3,236
21.139
Liu al. (32). which are general multimodal datasets encompass- ing video, audio, and language provides an overview of the datasets used in our experiments. Further details on the datasets can be found the supplementary materials,
MIMIC-II NOTE
3,652
815
806
5,273
MIMIC-IV MIMIC-CXR CMU-MOSI
18.064
2,035
4,972
25,071
344,529
9,497
23.069
377,095
1,283
229
686
2,198
modalities. Table [
POM
600
100
203
903
Matched Datasets
HIMIC-II&NOTE MIMIC-IV &CXR
3.652
5,273
4,287
465
1,179
5,931
MIMIC-IV&CXX&&REORR
4,287
465
1.179
5.931
Task & Evaluation Metrics.
Following the common practice in clinical prediction tasks 18 59
50
49
focus two
clinical prediction tasks: (1) In-Hospital Mortality (IHM) prediction, which predicts whether a patient passes away during their hospital stay; and (2) Readmission (READM) prediction, which aims to predict whether a patient is readmitted within 30 days after discharge.
For CMU-MOSI and POM datasets, we follow the settings in 32 and adopt respectively the movie sentiment analysis, and movie traits prediction tasks. Detailed evaluation metrics of each task introduced in the supplementary materials.
Table 2: Results in AUROC, AUPR and F1 with 95% confidence intervals on the MIMIC-II and MIMIC-IV datasets with totally matched modalities. The best results are highlighted in bold. Our proposed DPMM outperforms the baselines in all cases.
IIM
READM AUPR (t)
Model
AUROC (t)
AUPR (1)
FI (t)
AUROC (1)
FI (f)
MIMIC-II
MMIM 26
(.805, 8.47)
0.394 0.990
0.386 0.456;
0.744 0.88)
0.397 0.479;
0.418 6.74
DAFT
0.807 ja n.89;
19y
0.414
0-68;;
m.701 0346)
0.364 8440)
0,364 400
Unified 17
(.
0.437 a.547
0.454
0.52;;
0.729
0.405 0.04
0.410 6477)
MedlFuse 18
0.830 9.870)
0.439
6.400
j.410 0.559)
0.719 0.719 0.764)
0,382 a.45s
0. 0.382
DrFuse
7.79 0.875)
yours 0.67;
jo. 0.531)
0. (2.77 0.77))
0.418 0.503)
0.437 0.507
DPHK
0.854 (990))
0.532 6.rs,
0,490 9.99%)
0.754 0.74
0.460,, a se3
0.461 825,
MIMIC-IV
MMTM 26
j.799 8.82))
0.42 10.356 0.505:
0.456 0.506)
0. 0.705 0.540
0.423 p0.302 a.45)
0.463 6.513;
DAFT 38
0,815 0.80))
0.443 0.277;
0. 0.462g0.00. 0359;
0.73 (a.0
0.423, 0.423 0.49y
0.486 0335;
Unified
(78 nAw)
0.440
los a 6.99)
0.469 0.1;;
0.719 a754)
0.42] f.42)o..e 1sa 4.49)
0.472
MedFuse 18
0,809 0.44)
0.434 6.512)
0.466 0.566)
0.717 0.254)
0.424 4.90)
0.456 5.60)
Driiuse [54
0.8 (,816 8.848)
0.446
0.457 0310:
0.727 (0694 0.700
0.414
0.47 1,.71/0.22... 0331;
DPHN
0.826 29266896
0.482
0.526 0.534
0.7 6.
0.455
0.488 6.524)
7

---

Table 3: Results in AUROC, AUPR, and F1 with 95% confidence intervals on MIMIC-III III and MIMIC-IV datasets with partially matched modalities (i.e., missing modalities). The best results are highlighted in boldface. Our proposed method DPMM outperforms the baselines in all cases.
IHM
READM (1)
Model
(1)
AUPR (1)
Fl (
AUROC
Fl (1)
MIMIC-III
MMTM 26
(.847 0.86;
0,441
6.502;
0,476,0.43, 0.518;
1.757 0.76)
0. 0,446 0.9r;;
0,441 10.007 6473;
DAFT 38
(.854 0.82)
0.482 6.37%
0.496 0337)
(.753, 0775)
0.445 0.000
0.430 453;
Unified
(.84 n.w9;
0.48 g. 0:500)
g.754 739 0.751
a.448 now;
0,435 (o-0 awy
MedFuse
18
(.850 0.809)
0,480
6,524;
0,482 0.523)
0.754 0.75)
0,444 a.ss
0.439 6473)
DrFfuse
[54
(a8777 0.800)
0.474/o.22 6,529;
0 0.482 0.520)
0. 0.745 (.766)
jous 0.722;
0 0.446 6.47%;
DPHN
a.86 0,876)
0.511 (o.422 0.356.
0.498 0.498 0.567
0. 0.756 (0780)
0.445 jo.s 0.525
0.460
MIMIC-IV
MMIM 26
(.85 (.867)
g.000 0,551;
0.483 0.318
0.763 0.780)
0.470 0.504
(, 0.49,
DAFT 38
0.858 0.872)
0.5 a 1544,
0.495
0.22;;
0.761 07w
4.464 a 0.000
0.469 aws
Unified 177
0,852
g0500 6.542)
0.497
0.527)
0. 0.755
0. 064 0.50g
0. 0.460 64877
MedFuuse
(,855 0.870)
0050077777 0.546;
0,486,0.454.00 0.317;
(.753 0.771)
0.45%, a 0.493)
0,456 0.478;
DrFuse
[4
0.854 (.8s))
0.5 10,77 6.557;
0,496,0.465 0.528)
0. 0.765 746, 0,291)
0.478 0.516)
0.466 6.492)
DPHN
a. 0.77)
0.529 0.553,
0,518 0,518 0.597
q.77 a200
0.486 a 0.508
0.472 0-497,
Table 4: Results in AUROC, AUPR on the MIMIC-IV Readmission task with different combinations of modalities.
Modality
MMTM
26
DAFT [X
Unified
17
MedFuuse
[8]
DrFuse AUROC
[5]]
DPMM
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
AUPR
AUROC
AUPR
T+1
0,705
0.423
0.731
0.423
0.719
0.421
0.717
0.424
0.727
0.419
0.736
0.455
T+N
0,703
0.415
0.696
0.391
0,708
0.422
0.707
0.417
0.721
0.426
0.728
0.432
741+N
0,708
0.417
0.703
0.399
0,710
0.422
0.712
0.421
0.724
0.434
0.737
0.455
T only: 0.703 (AUROC), 0.420 (AUPR):
only 0.626 (AUROC), 0.296 (AUPR):
only: 0.538 (AUROC), 0.232 (AUPR)
Backbone Encoders. We utilize ResNet34 [9] as the backbone encoder for CXR image data, following [18). A two-layer stacked LSTM network [14] is employed for time-series data, including lab values and vital signs. For clinical notes and radiology reports, TinyBERT (25 serves as the encoder. Additionally, a projection layer maps all modality embeddings into the same latent space. For experiments on CMU-MOSI and POM datasets, we use an MIP as an encoder for each modality following (32)
4.2 Compared Methods
We compare DPMM on the clinical benchmark with five baselines: (1) DrFuse (54) adopts a transformer- type design which leverages disentangled representation learning to create a shared representation between the EHR and image modalities, even when one modality is missing. (2) MMTM (6 is a flexible plug-in module that facilitates information exchange between modalities. Since the model assumes availability of all modalities, we compensate for missing CXR and clinical notes during training and testing by filling in all zeros for missing values. (3) DAFT (38) is module designed to exchange information between tabular data and image modalities when integrated into CNN models. Similarly, we replace missing CXR and clinical notes with zero matrices during training and testing. (4) Unified [17] is a dynamic approach for integrating auxiliary data modalities, learning modality-specific representations, and combining them via unified classifier. It handles missing data inherently and leverages all available modality-specific information. (5) MedF [8] employs LSTM-based fusion to combine features from image or language encoders with EHR encoders. It handles missing modalities by learning a global representation for absent CXR or clinical notes.
In addition to adapting the aforementioned baselines, we also incorporate two general multimodal baselines for CUM-MOSI and POM: Low-rank Multimodal Fusion (LMF) 32 adopts low- rank tensor to improve the efficiency of multimodal fusion; Tensor Fusion Network (TFN) [57] formulates the multimodal problem as modelling the intra-modal and inter-modal dynamics, and learns these two dynamics with learnable tensors.
8

---

4.3 Experimental Results
Quantitative Results on MIMIC Datasets. Tables 2 and 3 present the performance of DPMM compared existing state-of-the-ard (SOTA) methods on the MIMIC-III and MIMIC-IV datasets. The results demonstrate that our method can overall outperform the SOTA methods satisfactorily. Notably, for the IHM task, DPMM exceeds the best baseline by 2.3% in AUROC MIMIC-III and 8% in AUPR on MIMIC-IV. Moreover, we discover that our method also performs satisfactorily partially matched modalities, indicating the capability of the DP assumption to generate representations for missing modalities. We also perform experiments on tri-modalities with M = 3. Table presents the results compared to SOTA methods. We observe in general an improved performance when more modalities are involved as more information can be gained. However, since the alignment complexity increases drastically with the increase in the number of modalities, the learning performance may not constantly outperform the one using fewer modalities.
Quantitative Results on CMU-MOSI and POM.
We further perform experiments CMU-MOSI and POM, where fol-
Table 5: Performance (%) comparison two general multi- modal datasets CMU-MOSI, and POM.
low the implementations by Liu et al. Table | the results of sentiment analysis and movie trait pre- diction tasks where the modalities are video, audio, and text. We adapted the implementations of the clinical base- lines to these datasets. Our method not only achieves competitive perfor- mance against the clinical baselines,
32
CMU-MOSI
Model
MAE (1)
Accuracy (1)
FL)
MAE (1)
Cor (1)
Accuracy
Unified 28
1.21
0.656
0.657
0.862
0.213
0.353
MedFuuse
[8
1.11
0,700
0.696
0.861
0.262
0.331
DeFuse
1.13
0.856
TPN
1.18
0.682
0.682
0.858
0.263
0.358
DEMN
1.09
0.701
0,702
0.847
0,273
0.359
also some general multimodal learning baselines. This indicates that our model can also obtain a satisfactory performance natural multimodal datasets in addition to the clinical context.
Qualitative Results. We visualize the capability of DPMM selecting the most prominent features for every modality. Figure presents the marginal distributions of the first mixture component of the m-th modality. We observe that the first components of each modality highlight the most significant features (with higher log-lik ge-likkliihodd,, while the second components are allocated less weights their features are less prominent. This validates mixing different mixture components across modalities allows DP to select the most prominent feature distributions via its richer- ticher-gets-riche gets-richer property.
0.740
0.798
0.736
0.734
0.726
Figure Histogram of the log- log-density values of the top 2 mixtures of each of the M modal- ities. We observe that can rank the feature components of each modality according to its prominence.
Figure 6: Performance in AUROC under the readmission task from MIMIC-IV with respect to different values of K and 77. Each line represents a distinct value
4.4 Ablation Analysis
Effect of Concentration Rate n and Maximum Number of Mixture Components K. The concentration rate y determines the greediness of the DP in exploring new mixture components. The

---

Table 6: Ablation study on different components (e.g., seadient-preservving sampling (GPS) and fusion module) of our proposed method. All results are reported in AUROC, AUPR, and F1 with 95% confidence intervals on the MIMIC-IV dataset.
Model
Matched
IHM AUPR (1)
READM
AUROC (1)
FI (1)
AUROC (1)
AUPR (t)
FI (t)
w/o alignment
0.855 a,50
0. 0.50 0.590
0.486u 0.532)
0,753 6.221
0.459 eoss
0. 4.456 4418
wio GPS
x
0.8 0.855rose
0 0.489s ad50y
j.764 6.758
0.466,84422 6.502
0. 0.46 1.467,0.44 0437)
wio fusion
0.857 a.22y
0.527 nuas, au500
0.488 0.22)
0. 0.760, 6.72%
0,460 a d.aus
0 4.46,..2 aews
DPNH
0.860 0.877
0.529 plas9, 0.530
0.5 18 18iass a.5hh
0.771 (0.522 6.788)
0.486 6.518)
0. 20.445 0437)
wio alignment
0.434 0a32)
0,466 a.se)
0.717 Ta a.2s
0.424 a-ww:
0. 0,456, am 2:00
wio fasion
O. 0.800 (0/744
0.436
0. 0,45. a53n
0.7 yassi 6.907)
0.4 0.1 4a.255
0. 0.447 c496)
DPHM
0.826 (222 232.0.80)
0.482, (.2RA.0)))
0.526 0.524
0.7.6a..04 6.753)
0.455 6.529
0,488,a.20 200 0.53
Table 7: Ablation study on alignment losses. Results are reported in AUROC, AUPR, and FI with 95% confidence intervals on the MIMIC-IV dataset.
Alignment
IHM AUPR (t)
READM AUPR (1)
Loss
AUROC (t)
(1)
AUROC (t)
(1)
Cosine loss
0.8 08816,7788 783, 0.549)
0.476,0.405, 0.500)
0.4 6.527)
0.719 4.756
0.427 0.933)
0,478, 0.529)
KI. loss
0.820 73.38, 0,830)
4.464 0,543)
0.499 0.553)
0.724 (0,65s 0,760)
0.436 0.500)
0 0.458 0.500)
DPMM
0.826g..72.0880
0.
0.5 1.526 0.3.9)
0.736 (773)
0,455 0.329)
0. 0,488 0.534
larger value is 17. the faster the model concentrates to the significant features, while the parameter space would be less explored. On the other hand, employing DP mixture model can inherently reduce the reliance on the number of mixtures as the DP can select the optimal number of mixtures by its stick-breaking process. Figure 6 demonstrates the performances in AUROC with respect to different combinations of and K. We discover that the performance is generally robust to different values of the two parameters, with a generally improved performance observed when the number of components is truncated at a high level (i.e., larger K).
Effect of Proposed Components. We analyze the contribution of each proposed component. Table 6 presents the performance with and without the fusion modules, and the gradient-preserving preserving sampling module, respectively. We observe that both the partially matched and completely matched datasets, there are performance improvements after applying these modules, validating their effectiveness.
Effect of DP Alignment. We also study the effects of the DP aligmment loss, which is presented in Table 7 We observe that compared to commonly alignment losses (such as cosine similarity or empirical KI. divergences), method achieves satisfactory performance.
Conclusion
We propose deep DP-based framework for multimodal representation learning. We leverage the richer-gets-richer property of DP to amplify the features in each modality, while at the same time aligning their marginal distributions effectively. To deploy the DP model in high dimensions, we adopt variational inference to estimate the posterior distribution stochastically. Empirical analysis demonstrates the satisfactory performance of DPMM compared to SOTA methods, on both bimodal and trimodal datasets. Ablation analysis demonstrates the effectiveness of the DP mixture model in cross-modal alignment, and the robustness of the framework to changes in key hyperparameters.
Limitations and Future Work. There are other ways to apply Dirichlet process on multimodal learning, such as hierarchical DP which models the mixture components and the marginal distribu- tions through a hierarchy of Dirichlet processes. These potential variants will be explored future works. More advanced assumptions, such as extending the diagonal covariance to full covariance and placing Wishart prior to the covariance matrix, also warrant further exploration.
Broad Impact. DPMM offers a novel perspective for multimodal learning by dynamically allocating feature contributions and emphasizing prominent ones via its richer-gets-ricte property, thereby enhancing feature fusion.
10

---

Acknowledgements
This work was supported in part by the Research Grants Council of Hong Kong (27206123, 17200125, C5055-24G, and 145-401/22-N), the Hong Kong Innovation and Technology Fund (GHP/318/2GD), the National Natural Science Foundation of China (No. 62201483), and Guangdong Natural Science Fund (No. 2024A1515011875)
References
[1] D. M. Blei and M. I. Jordan. Modeling annotated data. In Proceedings of the 26th annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval, pages 127-134, 2003.
[2] D. M. Blei and M. I. Jordan. Variational inference for Dirichlet process mixtures. Bayesian Analysis, 1(1):121-144, 2006.
[3] D. M. Blei, A. Ng. and M. I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993-1022, 2003.
[4] Y. Cao, L. Cui, L. Zhang, F. Yu, Z. Li, and Y. Xu. MMTN: Multi-modal memory transformer network for image-repor consistent medical report generation. Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 277-285, 2023.
[5] Chen, Zhang, D. Lian, J. Yang. Z. Zeng, and Shi. iQuery: Instruments as queries for audio-visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 14675-14686, 2023.
[6] S. Deldari, D. Spathis, M. Malekzadeh, F. Kawsar, Salim, and A. Mathur. Latent masking for multimodal self- superviddd learning in health timeseries. arXiv preprint arXiv:2307.16847, 2023.
[7] B. Dufumier, J. Navarro, D. Tuia, and J.-P. Thiran. What to align in multimodal contrastive learning? In The Thirteenth International Conference on Learning Representations, 2025,
[8] A. Echraibi, J. Flocon-Cholet, S. Gosselin, and S. Vaton. On the variational posterior of Dirichlet process deep latent Gaussian mixture models. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2020.
[9] M. El Banani, K. Desai, and J. Johnson. Learning visual representations via language-guides sampling. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 19208-19220, 2023.
[10] Z. Fang, X. Zhu, C. Yang, Z. Han, J. Qin, and X.-C. Yin. Learning aligned cross-modal representation for generalized zero-shot classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6605-6613, 2022.
[11] Z. Fu, Z. Mao, Y. Song, and Y. Zhang. Learning semantic relationship among instances for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15159 15159-15168, 11,6,6666688 2023.
[12] T. Gan, Q Wang, X. Dong, X. Ren, L. Nie, and Q. Guo. CNVid-3.5M: Build, filter, and pre-train the large-scale scale public chinese video-text dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[13] R. Girdhar, A. El-Nouby, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Omnimae: Single model masked pretraining on images and videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10406-10417, 2023.
[14] A. Graves. Long short-term memory. Supervised Sequence Labelling with Recurrent Neural Networks, pages 37 45, 2012.
[15] H. Harutyunyan, H. Khachatrian, Kale, G. Ver Steeg, and A. Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1):96, 2019.

---

[16] M. Haußmann, F. A. Hamprecht, and M. Kandemir. Variational Bayesian multiple instance learning with Gaussian processes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6570-6579, 2017.
[17] N. Hayat, K. J. Geras, and F. E. Shamout. Towards dynamic multi-modal phenotyping using chest radiographs and physiological data. arXiv preprint arXiv:211 10.11...22 1.02710, 10, 2021.
[18] N. Hayat, K. J. Geras, and F. E. Shamout. Medfuse: Multi- fusion with clinical time-series data and chest X-ray images. In Machine Learning for Healthcare Conference, pages 479-503, 2022.
[19] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[20] C. Huang. Y. Tian, A. Kumar, and C. Xu. Egocentric audio-visual object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22910-20 1222222, 2023.
[21] S.-C. Huang, A. Pareek, S. Seyyedi, I. Banerjee, and M. P. Lungren. Fusion of medical imaging and electronic health records using deep learning: systematic review and implementation guidelines. NPJ Digital Medicine, 3(1):136, 2020.
[22] Y.-H. Hubert Tsai, L..K. Huang, and R. Salakhutdinov. Learning robust visual-semantic embeddings. In Proceedings of the IEEE International Conference on Computer Vision, pages 3571-3580, 2017.
[23] H. Ishwaran and M. Zarepour. Exact and approximate sum representations for the Dirichlet process. Canadian Journal of Statistics, 30(2):269-25) 2002.
[24] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 4916, 2021.
[25] X. Jiao, Y. Yin, Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163 41744,4,4441 2020.
[26] H. R. V. Joze, A. Shaban, M. L. luzzolino, and K. Koishida. MMTM: Multimodal transfer module for CNN fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13289-13299, 2020.
[27] K. Kandasamy, W. Neiswanger, J. Schneider, B. Poezos, and E. Xing. Neural architecture search with Bayesian optimisation and optimal transport. Advances in Neural Information Processing Systems, 31, 2018.
[28] F. Khader, G. Mtiller-Franzer T. Wang. T. Han, S. Tayebi Arasteh, C. Haarburger, J. Stegmaier, K. Bressem, C. Kuhl, Nebelung, N. Kather, and D. Truhn. Multimodal deep learning for integrating chest radiographs and clinical parameters: A case for transformers. Radiology, 309(1):e230006 2023.
[29] D. Kim, J. Park, H. C. Chung, and S. Jeong. Unsupervised outlier detection using random subspace and subsampling ensembles of Dirichlet process mixtures. Pattern Recognition, 156: 10846, 2024.
[30] Z. Li, T. Xi, Deng, G. Zhang, Wen, and R. He. GP-NAS: Gaussian process based neural architecture search. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 1993-119422 2020.
[31] D. Linsley, P. Feng, T. Boissin, A. K. Ashok, T. Fel, S. Olaiya, and T. Serre. Adversarial alignment: Breaking the trade-off between the strength of an attack and its relevance to human perception. arXiv preprint arXiv:2306.03229, 2023.
12

---

[32] Z. Liu, Y. Shen, V. B. Lakshminarasimhan. P. Liang, A. B. Zadeh, and L.-P. Morency. Efficient low-rank multimodal fusion with speeiit factors. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics, 2018
[33] M. Ma, J. Ren, L. Zhao, S. Tulyakov, C. Wu, and X. Peng. SMIL: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2302-2310, 2021.
[34] R. D. Mauldin, W. D. Sudderth, and C. Williams. Pólya trees and random distributions. The Annals of Statistics, pages 1203-12211 1992.
[35] S. Mo and Y. Tian. Audio-visual grouping network for sound localization from mixtures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10565-10574, 2023.
[36] S. M. Paddock, Ruggeri, M. Lavine, and M. West. Randomized Pólya tree models for nonparametric Bayesian inference. Statistica Sinica, pages 443-460, 2003.
[37] S. Park, H. S. Shim, M. Chatterjee, K. Sagae, and L.-P. Morency. Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach. In Proceedings of the 16th International Conference on Multimodal Interaction, pages 50-57, 2014.
[38] S. Pölsterl, N. Wolf, and C. Wachinger. Combining 3D image and tabular data via the dynamic affine feature map transform. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 688-698. Springer, 2021.
[39] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748- 8763, 2021.
[40] C. E. Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning, pages 63-71. Springer, 2003.
[41] Snock, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 25, 2012.
[42] X. Song, X. Zhang, J. Ji, Y. Liu, and P. Wei. Cross-modal contrastive attention model for medical report generation. Proceedings of the 29th International Conference on Computational Linguistics, pages 2388 2397, 2022.
[43] Taniguchi, Yoshino, and Takano. Multimodal hierarchical Dirichlet process-based active perception by a robot. Frontiers in Neurorobotics, 12:22, 2018
[44] T. Theodoridis, T. Chatzis, V. Solachidis, K. Dimitropoulos, and P. Daras. Cross-modal variational alignment of latent spaces. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition Workshops, pages 960-961, 2020.
[45] B. Theodorou, L. Glass, C. Xiao, and J. Sun. Framm: Fair ranking with missing modalities for clinical trial site selection. Patterns, 5(3), 2024.
[46] V. H. Trong, Y. Gwang-hyun, T. Vu. and K. Jin-young. Late fusion of multimodal deep neural networks for weeds classification. Computers and Electronics in Agriculture, 175:105506 2020.
[47] F. Wang, Y. Zhou, S. Wang. V. Vardhanabhuti, and L. Yu. Multi-granularity cross- mrosl align- ment for generalized medical visual representation learning. Advances in Neural Information Processing Systems, 35:33536 33:33536-33549 2022.
[48] L. Wang, C. Zhang. H. Xu, Y. Xu, X. Xu, and S. Wang, Cross-modal contrastive learning for multimodal fake news detection. Proceedings of the 31st ACM International Conference on Multimedia, pages 5696-5704, 2023.
13

---

[49] Y. Wang, M. Pillai, Y. Zhao, C. M. Curtin, and T. Hernandez-Boure Boussard. FairEHR- Towards Fairness-aware clinical predictions with contrastive learning in multimodal electronic health records. In Machine Learning for Healthcare Conference, 2024.
[50] Z. Wu, A. Dadu, N. Tustison, B. Avants, M. Nalls, J. Sun, and F. Faghri. Multimodal patient representation learning with missing modalities and labels. In The Twelfth International Conference Learning Representations, 2024.
[51] J. Xi, J. Osea, Z. Xu, and J. S. Hartford. Propensity score alignment of unpaired multimodal data. Advances in Neural Information Processing Systems, 37:141 11441103-111222 2024.
[52] Y. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, and B. Schiele. Latent embeddings for zero-shot classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 69-77, 2016.
[53] O. Yakhnenko and V. Honavar. Multi-modal hierarchical Dirichlet process model for predicting image annotation and image-object label correspondence. In Proceedings of the 2009 SIAM International Conference on Data Mining, pages 283-293, SIAM, 2009.
[54] W. Yao, K. Yin, W. K. Cheung, J. Liu, and J. Qin. Drfuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 16416-16424, 2024.
[55] D. H. Yi, D. W. Kim, and C. S. Park. Prior selection method using likelihood confidence region and Dirichlet process Gaussian mixture model for Bayesian inference of building energy models. Energy and Buildings, 224: 10293, 2020.
[56] Y. Yoo, L. Y. Tang, D. K. Li, L. Metz, S. Kolind, A. Traboulsee, and R. C. Tam. Deep learning of brain lesion patterns and user-defined clinical and MRI features for predicting conversion to multiple sclerosis from elinically isolated syndrome. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 7(3):250-259, 2019.
[57] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency. Tensor fusion network for multimodal sentiment amalysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 103-11144 14. 2017.
[58] A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency. MOSI: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv: 1ee..6606606666 2016.
[59] C. Zhang, X. Chu, L. Ma, Zhu, Y. Wang. J. Wang, and J. Zhao. M3care: Learning with missing modalities in multimodal healtheare data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2418 221882228,22 2022.
[60] C. Zhang and G. Yin. Bayesian nonparametric analysis of restricted mean survival time. Biometrics, 79(2):13331199) 2023.
[61] X. Zhang, S. Li. Z. Chen, X. Yan, and L. R. Petzold. Improving medical predictions by irregular multimodal electronic health records modeling. In International Conference on Machine Learning, pages 41300
2023.
[62] R. Zhao, L. Zhang, B. Fu, C. Hu, J. Su, and Y. Chen. Conditional variational autoencoder for sign language translation with cross-modal alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19643-19651, 2024.
[63] W. Zhao, X. Wu, and Luo. Cross-domain image captioning via cross-modal retrieval and model adaptation. IEEE Transactions Image Processing, 30:1180-1011 11188011922 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state and accurately reflect our novel contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
We include the discussion of the potential limitations of our work in the conclusions section.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: All theoretical results are accompanied by clearly stated assumptions and complete proofs in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of data processing and bootstrap- based confidence interval computations to enable full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All code and data are provided the anomonymized repository, with clear instructions for reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides/ Saaaad/ee/ooeeSuuumassiioooiioi
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cotee//ddeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify data splits, hyperparameters, optimizers, hardware details, in the Implementation Details section.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We reported confidence interval instead of error bars
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We detail GPU types, memory for all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwwpgp..ccppaaaaa/accaaa
Answer: [Yes]
Justification: We confirm that our research adheres fully to the NeurIPS Code of Ethics guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both positive applications in medical fairness and potential misuse, along with mitigation strategies.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We determine that our work poses no high-risk assets requiring special release safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We credit all reused code and datasets with explicit citations.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
The code of our model is documented with usage details and licensing infor- mation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable as our work does not involve human subject research or crowdsourced data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Not applicable since our paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Not applicable as we did not use LLMs in the core method development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

Summary
In this supplementary material, we first present detailed information on the datasets in A.1 and tasks used in the experiments in A.2 Next, we provide more details on the implementation and hyperparameters used in the experiments in C.1 along with the settings of baseline methods in We present additional experimental results in D
A
Additional Information on Datasets and Tasks
A.1 Datasets
Table A.1 provides summary of the datasets used in our experiments.
MIMIC-II
This dataset includes 46,520 ICU stays, each containing 17 clinical variables. Following the methodology in [15. the dataset is divided into training, validation, and test sets using 70%-
15% 15% split.
MIMIC-IV This dataset comprises 21, ICU stays and includes 17 clinical variables. Following 18]. the data are divided into 70% for training, 10% for validation, and 20% for testing. For both the MIMIC-III and MIMIC-IV datasets, we extract 17 clinical variables that are routinely monitored in the ICU, consisting of 5 categorical and 12 continuous variables. In line with (18), the data is sampled every two hours during the first 48 hours of ICU admission for both tasks. This process results in vector representation of size 76 at each time step in the clinical time-series data.
MIMIC-CXR This dataset comprises 377, chest X-ray images, 5,931 which are linked to MIMIC-IV ICU stays. The data are divided into 4,287 training samples, 465 validation samples, and 1,179 test samples. line with [T8], we select the most recent chest X-ray and apply transformations to the images, resizing them to 224 224 pixels. Additionally, this dataset includes radiology reports, which consist of unstructured text data. We utilize the radiology reports from the MIMIC-CXR dataset when evaluating the performance of DPMM on the tri- modal setting. Since these reports do not contain mortality information, they help mitigate potential overfitting and shorteut learning. The unstructured radiology reports are divided into four sections: Impression, Findings, Last Paragraph, and Comparison.
Posttrrro (PA) projection
MIMIC-III NOTE This dataset contains 5,273 clinical notes linked MIMIC-III ICU stays. The data are split into 3,652 samples for training, 815 for validation, and 806 for testing. Following the approach in [61]. we use the last five clinical notes prior to the prediction time. If fewer than five notes are available for a given ICU stay, those notes are treated as missing. The initial number of matched ICU stays is approximately 15,000. To create the training, validation, and test sets, randomly sample one-third of these ICU stays, ensuring that the scale of the clinical notes remains comparable to the CXRs in the MIMIC-IV dataset. Both the radiology report sections and clinical notes are limited to a maximum length of 512 words. These notes are tokenized into words and embedded into 3 122ddimeniional vectors using the pre-trained TinyBERT model [25
CMU-MOSI 588 This dataset comprises 93 opinion vidcos sourced from YouTube movie reviews. Each video divided into multiple opinion segments, with each segment annotated for sentiment on a scale from -3 to 3, where -3 represents highly negative sentiment and 3 represents highly positive sentiment. Following the approach in (32). we partitioned the dataset into 1,283 samples for training. 229 for validation, and 686 for testing.
This dataset consists of 903 videos containing movie reviews. Each video is labeled with speaker attributes, including: confident, passionate, voice, pleasant, dominant, credible, vivid, expertise, entertaining, reserved, trusting, relaxed, outgoing. thorough, nervous, persuasive, and humorous. In accordance with the method outlined in [32), we divided the dataset into 600 samples for training, 100 for validation, and 203 for testing.
POM 37
_L__112
22

---

Table A.1: Summary of real datasets used in the experiments.
Dataset
Tasks
No. Train
No. Valid
No. Test
No. Pos.
Total
Complete Datasets
MIMIC-III
THM READM
14681
3222
3236
2795
21139
MIMIC-III
14681
3222
3236
3987
21139
MIMIC-III NOTE
3652
815
806
5,273
MIMIC-IV
IHM READM
18064
2035
4972
3153
25071
MIMIC-IV
18064
2035
4972
4603
25071
MIMIC-CXI
344529
9497
23069
377,095
CMU-MOSI
1,283
229
686
2,198
POM
600
100
203
903
Matched Datasets
MIMIC-III & NOTE
THM
3652
815
806
736
5273
MIMIC-II & NOTE
READM
3652
806
998
5273
MIMIC-IV & CXR
THM
4287
465
1179
890
5931
MIMIC-IV & CXR
READM
4287
465
1179
1262
5931
MIMIC-IV & CXR & REPORT
IHM
4287
465
1179
890
5931
MIMIC-IV & CXR & REPORT
READM
4287
465
1179
1262
5931
A.2 Tasks
In Hospital Mortality (IHM) Prediction. The In Hospital Mortality (IHM) prediction task focuses on predicting whether a patient will pass away during his/her hospital stay. As summarized in Table A. the MIMIC-III dataset contains total of 2,795 positive samples, of which 736 matched with clinical notes. Similarly, the MIMIC-IV dataset includes 3,153 positive samples, with 890 matched to CXR.
Readmission (READM) Prediction. The Readmission (READM) prediction task aims to forecast whether patient will be readmitted within 30 days of discharge. In this task, both patients who readmitted and those who pass away in hospital are considered positive samples. As shown in Table A.I. the MIMIC-III dataset contains 3,987 positive samples, with 998 matched clinical notes. In the MIMIC-IV dataset, there are 4,603 positive samples, with 1,262 matched to CXRs.
Sentiment Analysis & Movie Traits Prediction. We follow the implementations in 32 for the sentiment analysis and movie traits prediction tasks. We report the mean absolute error (MAE), accuracy, and F] score for both tasks.
B Multivariate Gaussian Distribution
The multivariate Gaussian distribution
p( E)
exp
(2 1) - -1))
(2T)
Σ
where μ e IR" is p-dimensional mean vector and _ the covariance matrix.
The KL divergence of two multivariate normal distributions N(μ,, Ei) and N N E2) is
Σn p
(M(N(,,,)))N Ez))
log Σ
( ( p1) p1''E, (12 m1)
C More on Baseline Methods and Implementation Details
C.1 Implementation Details and Hyperparameters
We train all models for 100 epochs using the training set, selecting the best-performing model based on validation AUROC. The final evaluation conducted on the test set. The Adam optimizer
is utilized for optimization, and early stopping is employed if the validation AUROC shows
23

---

improvement over 15 consecutive epochs to mitigate overfitting. All experiments are performed on single RTX-3090 GPU. The batch size configured as 32 for the MIMIC-IV & CXR datasets and 16 for the MIMIC-III & NOTE. datasets, except for DrFuse, which is trained with batch size of 8. Hyperparameters are tuned using grid search on the validation set, and the test set results are based on the best configuration. The search space includes:
Dropout ratio: (0,0.1,0.2,0.3)
Learning rate: {1 x 1001 xx ,5x 10-0, , 1x 10-0}
Concentration rate 7): {0.1, 1,
Number of mixture components K: (2. 4,5)
Temperature: {0.001, 0.005, 0.01, 0.05, 0.08)
Regularization parameter ApP (adjusting the strength of DP assumption): {1 x 10-5,5 55x 10- 1 10-
DPMM is developed using Python 3.11 1 and PyTorch 1.9. In line with MedFuse [18] ResNet34 19] is employed as the backbone encoder for CXR images, a two-layer LSTM [14] is utilized for encoding time-series data, and pre-trained TinyBERT (25 is adopted for clinical notes. A projection layer aligns modality embeddings within the same latent space, followed by a two-layer LSTM as the fusion module integrate embeddings. Finally, multilayer perceptron (MLP) with a linear layer and sigmoid activation function is used for classification.
C.2 Additional Settings of Baseline Methods
We compare DPMM with the following baseline methods.
MMTM 26
facilitates cross-modal information exchange through plugin module. Since it assumes all modalities are present, missing modalities (CXR and clinical notes) are filled with zeros during training and testing. For clinical notes, the ResNet34 encoder replaced with TinyBERT for embedding.
DAFT 38 integrates tabular and image data within CNN models. Similar to MMTM, missing modalities are replaced with zero matrices, and TinyBERT used to embed clinical notes.
Unified 17 is dynamic method for integrating auxiliary data modalities by learning modelity-specificc representations and combining them through a unified classifier. in- herently manages missing modalities and leverages all available modality-specific data. TinyBERT is employed to embed clinical notes.
MedF use 18 is an LSTM-based fusion approach that combines features from image and EHR encoders. Missing modalities (CXR clinical notes) handled via global representations. Encoders for time-series data, clinical notes, and CXR are initialized randomly.
DrFuse (54) employs disentangled representation learning create a shared representation between EHR and image modalities, even with missing modalities. DrFuse uses ResNet50 the image encoder and Transformer as the EHR encoder. For clinical notes, ResNet50 is replaced with TinyBERT.
Low-rank Multimodal Fusion (LMF) [32 adopts low-rank tensor to improve the efficiency of multimodal fusion. We use an MLP as an encoder for each modality following [32]
Tensor Fusion Network (TFN) 57 formulates the multimodal problem as modelling the intra-modal and inter-modal dynamics, and learns these two dynamics with learnable tensors. We use an MLP as an encoder for each modality following [32].
The implementation of DrFuse follows the original paper 54
and use the same hyperparameters as the original paper. We directly adopt the implementations of MMTM, DAFT, Unified, and MedFuse
24

---

Table D.1: Results on using learnable weights for GMM on the MIMIC-IV dataset (partially totally matched datasets).
IHM AUPR
READM AUPR
Model
AUROC
F1
AUROC
F1
learnable weights
0.855 0.821
0.521 00.46
0.493
0.759/ 0.719
0.472/ 0.434
0.461 00466
weights
0.860/ 0.826
0.529 00.422
0.518 0.526
0.771 0.736
0.486 0.455
0.472 /0.488
provided by 18 and all hyperparameters are set to the default values provided by [ Moreover, the implementation of LMF and TFN follows [32 and we use the same hyperparameters in the paper. We adapt the implementations of MMTM, DAFT, Unified, MedFuse and DrFuse to tri-modal setting, including EHR time-series data, CXR images, and radiology reports.
D More Experimental Results
Table D.2: Results under different missing ratios of 10%, 40% and 70% on the READM task, MIMIC-IV dataset for different modalities (time series missing / image missing).
Matched
10%
40%
70%
Model
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
MMTM
0.705
0.423
0.685 0.704
0,417 / 0.422
01658 / 0.700
0.356 0.420
0.589 0.699
0.297 / 10,400
DAFT
0.731
0.423
0.728 0.731
0.421 /0.420
0.658 / 00700
0.343/0.416
0.635 / 0.694
0.299 / 0.407
Unified
0.719
0.421
0.717/0.715
0.419/0.420
0,671 /0.712
0.360 /
0.640 / 707
0.323 / 00.455
MedFuse
0.717
0.424
0.715 0.716
0.421 / /0.418
0.639 / 0.707
0.334 0.422
0,550 / 0.705
0.244 / 0.416
DrFuse
0.727
0.419
0.716
0.400 0.419
0.640 0.720
0,327 0.416
0.560 / 0,712
0.259 / 0,413
DPMM
0.736
0.455
0.735
0.442 0.449
0.675 /07716
0.367 0.426
0.649
0.331 / 0.423
D.1 DP Weights against Learnable Weights
While learning weights through neural networks is viable, the selection of weights is observational. Therefore, we adopt DP as probabilistic model to adjust for observational bias and leverage its ficheregets-richer property to select important features. We compare the performance of DPMM with learnable weights against DP weights on the MIMIC-IV dataset Table The results show that using learnable weights might be biased and suboptimal, the ablation study (Tables 6-7 in the main text) also demonstrates the effectiveness of DP weights allocation.
D.2 Different Missing Ratios
We define missing modality as the observations in each modality that is missing, leading to partially matched datasets. We adopt the common assumption that the missing mechanism missing at random (MAR), the most commonly used one in statistics. We further validate that our model performs satisfactorily under different ratios missingness. Table D.2 shows the results for different modalities with missing ratios of 10%, 40%, and 70% the MIMIC-IY V, READM tasks, where with more missingness, the performance deteriorates as expected. Moreover, the performance drop is more severe when the missingness is in the EHR time series modality, which indicates that the EHR is more informative for READM tasks.
D.3 Comparison with More Imputation Methods
We compare our method with more recent and advanced imputation methods, such as PSA (Propensity Score Alignment) [5I] and LSMT 28 on MIMIC-IV Table D.3 Our method outperforms PSA and LSMT in both matched and partially matched settings.
25

---

Table D.3: Performance comparison with PSA and LSMT.
IHM
READM
Match
Model
AUROC
AUPR
AUROC
AUPR
PSA [5
0.744
0.346
0.692
0.370
Totally Matched
LSMT 28
0.803
0.444
0.701
0.421
DPMM
0.826
0.482
0.736
0.455
PSA 51
0.792
0.386
0.720
0.391
Partially Matched
I.MMT 28
0.854
0.508
0.764
0.473
DPMM
0.860
0.529
0.771
0.486
D.4 Comparison with Contrastive Multimodal Models
We include comparisons with three representative explicit alignment models: CLIP [39], ALIGN 24 and CoMM (7). all evaluated under totally matched setting on MIMIC4. shown Table D.4 proposed method (DPMM) consistently outperforms these strong baselines across all metrics, demonstrating the effectiveness and competitiveness of our implicit alignment strategy in multimodal contrastive learning.
Table D.4: Performance comparison with contrastive multimodal models.
IHM
READM
Model
AUROC
AUPR
AUROC
AUPR
CLIP 39
0.816
0.476
0.719
0.427
ALIGN 24
0.819
0.469
0.725
0.428
CoMM 7
0.823
0.465
0.726
0.432
DPMM
0.826
0.482
0.736
0,455
D.5 Ablation on the Amplification and Alignment of DP
To disentangle the effects of DP-driven amplification and feature alignment, conducted controlled ablation study Table D.5 To isolate the alignment effects, we train DP Gaussian mixture model for each modality instead of mixing the mixture components together. The results show that both amplification and alignment contribute to performance gains, with the combination achieving the best results under both totally and partially matched settings. This demonstrates that DPMMbenefits from the dominant feature amplification as well as its implicit alignment across modalities.
D.6 Analysis of the Computational Cost
We report the computational statistics (including the parameter size and training time) for all baselines and our DPMM model Table D.6 DPMM incurs only slightly higher training time than other methods (e.g., 26.57s 18 22s per epoch), which we consider acceptable given its modeling advantages. These results suggest that the stick-breate breaking process presents a reasonable computational bottleneck in standard multimodal learning scenarios.
26

---

Table D.5: Performance on the amplification and alignment of DP.
IHM
READM
Match
Model
AUROC
AUPR
AUROC
AUPR
w/o DP Amplification only Amplification Alignment
0,809
0.434
0.717
0,424
Totally Matched
0.816
0.462
0.724
0.429
0.826
0.482
0.736
0.455
w/o DP
0.855
0.506
0.753
0.459
Partially Matched
Amplification only
0.856
0.525
0.752
0.468
Amplification + Alignment
0.860
0.529
0.771
0.486
Table D.6: Computational cost.
Training time per epoch (totally partially matched)
Model
Batch size
No. Params
MMTM 26
32
89.62 MB
22.39s 79.24s
DAFT
38
32
85.16 MB
18.49% / 40.74s
Unitied
17
32
87.72 MB
18.97s 41.58s
McdFuse
18
32
91.04 MB
17.87s 40.00s
DrFuse
32
200.92 MB
18.73s 61.43s
DPMM
32
93.68 MB
26.57s 70.80s
27


Reviews:
[{"review_id": "CJdm2OdFll", "reviewer": "Reviewer_7M2V", "summary": "This paper proposes Dirichlet-Process Mixture Multimodal Learning (DPMM) to avoid representation degradation caused by excessive inter-modality alignment in multimodal learning. It treats the latent representation distribution of each modality as a Gaussian-mixture model (K components) and generates the mixing ratios via a Dirichlet process with a stick-breaking construction, leveraging its rich-get-richer property to emphasize important features.\nFor missing modalities, gradient-preserving sampling (GPS) performs probabilistic imputation while keeping gradients intact.\nThe proposed method outperforms state-of-the-art approaches on four datasets, including clinical (MIMIC-III/IV) and general-domain (CMU-MOSI, POM) benchmarks.", "questions": "- I would like the authors to address the weaknesses I pointed out.\n- Appendix E shows that replacing the Dirichlet process with learnable but fixed mixture weights yields only modest performance gains. This raises the question of how the results would fare on other datasets.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is very well structured and easy to read.\n- The method makes effective use of the rich-get-richer characteristic of the Dirichlet process, and the proposed GPS mechanism for handling missing modalities is markedly more effective than simple imputation, representing a substantial contribution to the community.\n- Experiments show consistently strong performance compared with the latest multimodal integration methods, validating the effectiveness of the proposal.\n\nWeaknesses:\n- As the authors themselves note, constraining each mixture component’s covariance matrix to be diagonal may limit expressiveness for modalities with strong cross-feature correlations. The authors should clarify the range of multimodal data for which this assumption remains valid.\n- Although the method claims to scale via the Dirichlet process and stochastic variational inference (SVI), there is no discussion (or quantitative evidence) of potential computational bottlenecks introduced by the stick-breaking process. Including such discussions or comparing training times with existing methods would strengthen the practical impact.", "was_revised": true, "final_justification": "My remaining concerns have been largely resolved. Since I have already given an Accept rating, I intend to maintain it.", "rebuttal": "Thank you for your appreciation on the novelty of our work and the strength of experimental results. We would address your main concerns in the response below.\n\n> **Q1** As the authors themselves note, constraining each mixture component’s covariance matrix to be diagonal may limit expressiveness for modalities with strong cross-feature correlations. The authors should clarify the range of multimodal data for which this assumption remains valid.\n\nWe adopt a diagonal covariance for each Gaussian component -- a choice that is common in high‐dimensional latent models such as VAEs (Kingma \\& Welling, 2014 [1]) and diffusion models (Ho et al., 2020 [2], Song et al. [3]), \nbecause (1) encoder networks typically produce decorrelated features, (2) estimating full covariances in large dimensions is computationally prohibitive, and (3) diagonal GMMs have been shown to capture the dominant variance directions in practice.  In our experiments on clinical multimodal benchmarks, this assumption sufficed to model complex interplays with no loss of accuracy.  For modalities where raw features exhibit pronounced cross‐feature correlations (e.g., unprocessed sensor arrays), one could readily extend our DP framework to low‐rank or full covariances, but in most modern pipelines—where embeddings are learned and approximately whitened—the diagonal assumption remains both tractable and valid. Together with recent advances in LoRA (low-rank adaptation), we would explore potential extension of the current assumptions beyond diagonal in future development of the manuscript.\n\n[1] Kingma, Diederik P., and Max Welling. \"Auto-encoding variational Bayes.\" 20 Dec. 2013, International Conference on Learning Representations.\n\n[2] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. \"Denoising diffusion probabilistic models.\" Advances in Neural Information Processing Systems 33 (2020): 6840-6851.\n\n[3] Song, Andrew H., et al. \"Morphological prototyping for unsupervised slide representation learning in computational pathology.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n> **Q2** Although the method claims to scale via the Dirichlet process and stochastic variational inference (SVI), there is no discussion (or quantitative evidence) of potential computational bottlenecks introduced by the stick-breaking process. Including such discussions or comparing training times with existing methods would strengthen the practical impact.\n\nTo address the concern, we have reported the computational statistics (including parameter size and training time) for all baselines and our DPMM model in the table below. As shown from Table 1, DPMM incurs only slightly higher training time than other methods (e.g., 26.57s vs. 18–22s per epoch), which we consider acceptable given its modeling advantages. \nThese results suggest that the stick-breaking process presents a reasonable computational bottleneck in standard multimodal learning scenarios.\n\n### Table 1: Computational Cost\n\n| Model        | Batch Size | Total Params | Training Time per Epoch (Totally / Partially Matched) |\n|--------------|------------|---------------|--------------------------------------------------------|\n| MMTM         | 32         | 89.62 MB      | 22.39 s / 79.24 s                                      |\n| DAFT         | 32         | 85.16 MB      | 18.49 s / 40.74 s                                      |\n| Unified      | 32         | 87.72 MB      | 18.97 s / 41.58 s                                      |\n| MedFuse      | 32         | 91.04 MB      | 17.87 s / 40.00 s                                      |\n| DrFuse       | 32         | 200.92 MB     | 18.73 s / 61.43 s                                      |\n| **DPMM**     | 32         | 93.68 MB      | 26.57 s / 70.80 s  \n\n> **Q3:** Appendix E shows that replacing the Dirichlet process with learnable but fixed mixture weights yields only modest performance gains. This raises the question of how the results would fare on other datasets.\n\nAppendix E compares our Dirichlet process (DP) with a variant that replaces the DP prior with directly learnable mixture weights. Our dataset is large and the backbone is fixed, making it inherently difficult to realize large gains from changes in mixture modeling alone. Hence the observed performance gap is modest. Crucially, this experiment demonstrates that without the DP’s \"richer-gets-richer'' regularization, the learned weights tend to overfit to noisy or less informative features, leading to suboptimal cluster emphasis. The DP prior adaptively amplifies dominant, semantically useful components while downweighting noise, which is essential for robust feature representation—especially under noisy and missing-modality settings (Secs. 4.2–4.3). Thus, using the DP for weight allocation is not only statistically grounded but necessary for ensuring stable and meaningful feature amplification across modalities, beyond what direct backpropagation allows."}, {"review_id": "mqrua5dxW7", "reviewer": "Reviewer_yFYG", "summary": "The paper proposes a novel Dirichlet Process Mixture Model-based multimodal learning framework that addresses the balance between intra-modal feature expressiveness and cross-modal alignment. By leveraging the richer-gets-richer property of the Dirichlet Process, the model is designed to amplify the most prominent features across modalities while still enabling effective modality alignment. The framework is validated on multiple multimodal datasets, including large-scale clinical datasets and general multimodal benchmarks.", "questions": "The questions below are related to what I mentioned in the strengths and weaknesses section.\n\n1. Can you provide a more rigorous ablation to disentangle the effects of alignment and amplification?\n2. Can you compare DPMM to more competitive missing modality baselines?\n3. Can you provide more interpretable qualitative examples?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- I think the use of the Dirichlet Process to dynamically allocate feature importance across modalities is an innovative angle that contrasts with typical cross-modal alignment methods that often suppress intra-modal feature strength. The probabilistic formulation with gradient-preserving sampling provides a natural way to handle missing modalities without relying on simple imputations or discarding data.\n- I appreciate that the model is validated across four multimodal datasets, including both healthcare and general-purpose benchmarks, providing a convincing demonstration of generalizability.\n- Through ablation, the paper explores the role of key components (alignment losses, fusion modules, concentration parameters), strengthening confidence in the design choices.\n\n### Weaknesses\nDespite the compelling contributions, several aspects of the paper need further attention to fully validate its claims.\n- The disentanglement between the effects of alignment and feature amplification is not sufficiently isolated, leaving it unclear whether performance gains stem primarily from the DP-driven amplification or from improved alignment.\n- The baseline selection for missing modality comparisons is limited, with some relying on simplistic zero-imputation, which underrepresents more competitive recent methods like generative imputers or retrieval-based completion.\n- While the theoretical background on the Dirichlet Process is sound, the paper does not rigorously justify why the richer-gets-richer property is always optimal for multimodal tasks, especially when task-relevant signals may not necessarily be statistically dominant. \n- It seems to me that the qualitative analysis remains superficial. A deeper, more interpretable feature analysis is needed to verify that the features amplified by the DP process are semantically meaningful and not just artifacts of the probabilistic modeling.", "was_revised": true, "final_justification": "Thank you for the detailed response and additional findings. They sufficiently address my questions and concerns. I have raised my score.", "rebuttal": "We thank the reviewer for appreciating the innovativeness of our work, the compelling contribution,  the soundness of empirical evaluation, and the solid ablation analysis.\nWe would address your concerns below.\n\n> **Q1:** The disentanglement between the effects of alignment and feature amplification is not sufficiently isolated, leaving it unclear whether performance gains stem primarily from the DP-driven amplification or from improved alignment.\n\n> Can you provide a more rigorous ablation to disentangle the effects of alignment and amplification?\n\nWe appreciate the reviewer’s comment. To **disentangle the effects of DP-driven amplification and feature alignment**, we conducted a controlled ablation study in the table below. To isolate the alignment effects, we train a DP Gaussian mixture model for each modality instead of mixing the mixture components together. The results (Table 2) show that both amplification and alignment contribute to performance gains, with the combination achieving the best results under both totally and partially matched settings. This demonstrates that DPMM benefits from the dominant feature amplification as well as its implicit alignment across modalities.\n\n### Table 2: Performance on the Amplification and Alignment of DP\n\n| Match              | Model                      | IHM AUROC | IHM AUPR | READM AUROC | READM AUPR |\n|--------------------|----------------------------|-----------|----------|-------------|-------------|\n| **Totally Matched**  | w/o DP                     | 0.809     | 0.434    | 0.717       | 0.424       |\n|                    | Amplification only         | 0.816     | 0.462    | 0.724       | 0.429       |\n|                    | **Amplification + Alignment** | **0.826** | **0.482** | **0.736**   | **0.455**   |\n| **Partially Matched** | w/o DP                   | 0.855     | 0.506    | 0.753       | 0.459       |\n|                    | Amplification only         | 0.856     | 0.525    | 0.752       | 0.468       |\n|                    | **Amplification + Alignment** | **0.860** | **0.529** | **0.771**   | **0.486**   |\n\n> **Q2:** The baseline selection for missing modality comparisons is limited, with some relying on simplistic zero-imputation, which underrepresents more competitive recent methods like generative imputers or retrieval-based completion.\n\n> Can you compare DPMM to more competitive missing modality baselines?\n\nWe compare our method with **more recent and advanced imputation metods**, such as PSA [1] (Propensity Score Alignment of Unpaired Multimodal Data and Unpaired Multi-Domain Causal Representation Learning.) and LSMT [2] (Multimodal deep learning for integrating chest radiographs and clinical parameters: a case for transformers) on MIMIC4 in the table below (Table 3). Our method outperforms PSA and LSMT in both matched and partially matched settings.\n\n[1] Xi, Johnny, et al. \"Propensity score alignment of unpaired multimodal data.\" Advances in Neural Information Processing Systems 37 (2024): 141103-141128.\n\n[2] Khader, Firas, et al. \"Multimodal deep learning for integrating chest radiographs and clinical parameters: a case for transformers.\" Radiology 309.1 (2023): e230806.\n\n### Table3: Performance comparison with PSA and LSMT for In-Hospital Mortality (IHM) and Readmission (READM) prediction\n\n| Match              | Model     | IHM AUROC | IHM AUPR | READM AUROC | READM AUPR |\n|--------------------|-----------|-----------|----------|--------------|-------------|\n| **Totally Matched**  | PSA       | 0.744     | 0.346    | 0.692        | 0.370       |\n|                    | LSMT      | 0.803     | 0.444    | 0.701        | 0.421       |\n|                    | **DPMM**  | **0.826** | **0.482**| **0.736**    | **0.455**   |\n| **Partially Matched** | PSA     | 0.792     | 0.386    | 0.720        | 0.391       |\n|                    | LSMT      | 0.854     | 0.508    | 0.764        | 0.473       |\n|                    | **DPMM**  | **0.860** | **0.529**| **0.771**    | **0.486**   |\n\n> **Q3:** While the theoretical background on the Dirichlet Process is sound, the paper does not rigorously justify why the richer-gets-richer property is always optimal for multimodal tasks, especially when task-relevant signals may not necessarily be statistically dominant.\n\nThank you for this important question.  We believe that the DP's \"rich‑get‑richer\" mechanism is mostly optimal—if not universally so—for multimodal learning, because it **adaptively amplifies clusters with consistent, task‐relevant signals while shrinking spurious noise**.  Even when those signals are weak or not statistically dominant, their repeated **co‐occurrence across modalities** allows the DP to distinguish and boost them above random artifacts.  Theoretical results (Lee \\& Sang 2022 [3]) demonstrate that DP mixtures achieve optimal bias–variance trade‐offs and asymptotic consistency as component counts grow.  Empirically (Table 2), adding DP‐driven amplification alone raises IHM AUROC from 0.809 to 0.816 and READM AUROC from 0.717 to 0.724 under full matching (and yields similar AUPR gains), confirming that DP regularization is essential **to filter noise and extract distinguishable features** beyond what direct backpropagation of fixed weights can deliver.  \n\n[3] Lee and Sang. Why the Rich Get Richer?\nOn the Balancedness of Random Partition Models. ICML 2022.\n\n> **Q4:** It seems to me that the qualitative analysis remains superficial. A deeper, more interpretable feature analysis is needed to verify that the features amplified by the DP process are semantically meaningful and not just artifacts of the probabilistic modeling.\n\n> Can you provide more interpretable qualitative examples?\n\nThank you for your comments, we agree that more thorough qualitative analysis is needed to understand the feature amplification process. \nDue to rebuttal formatting constraints, we cannot include additional figures here.\n\nWe would perform a more thorough qualitative evaluations in future versions of manuscript to directly illustrate DP‐driven feature amplification by (1) retrieving the top‑K raw inputs for each highest‑weight DP component and comparing them to the same inputs under a fixed‑weight mixture to show increased thematic coherence, (2) projecting embeddings via UMAP/t‑SNE and overlaying per‑point DP responsibilities to highlight “hot spots” of amplified features, (3) visualizing the change in mixture weights—i.e.$\\Delta \\pi_{mk}$—from a learnable‑weights baseline to our DP model to quantify amplification. We have performed **preliminary analysis** on the changes in the mixture weights $\\pi_{mk}$ comparing the ones learned by DP and the ones learned from gradient backpropagation (Table 4). Preliminary results demonstrate that DP enables sharp concentration the semantically meaningful centroids (faster and more stable weights increase). At the same time, the original Gaussian mixture model assumption ensures **semantically meaningful clustering**. (4) presenting cross‑modal exemplar pairs for shared components to verify that amplified clusters align semantically across modalities, and (5) plotting component weight distributions stratified by ground‑truth labels and comparing against the baseline to confirm that DP amplification enhances task‑relevant features rather than noise.\n\n### Table 4: Comparison of mixture weights $\\pi_{mk}$ across training epochs  \n(Showing three mixture components with the highest mixture weights)\n\n| Method               | Epoch 0               | Epoch 10              | Epoch 20              | Epoch 30              |\n|----------------------|------------------------|------------------------|------------------------|------------------------|\n| Neural Net (Learned) | [0.17, 0.17, 0.17, ...] | [0.20, 0.17, 0.15, ...] | [0.25, 0.15, 0.12, ...] | [0.30, 0.20, 0.10, ...] |\n| DP (Stick-breaking)  | [0.17, 0.17, 0.17, ...] | [0.50, 0.22, 0.14, ...] | [0.75, 0.20, 0.03, ...] | [0.85, 0.10, 0.02, ...] |"}, {"review_id": "CaUO3Woq6w", "reviewer": "Reviewer_qSYf", "summary": "This paper proposes a Dirichlet Process-based multimodal learning framework (DPMM) that adaptively balances intra-modal representation and cross-modal alignment. By modeling each modality as a Gaussian mixture and using the richer-gets-richer property of the DP to weight features, the method emphasizes the most informative components. Experiments show DPMM outperforms existing approaches, with ablation studies confirming the robustness and effectiveness of the DP mechanism.", "questions": "1. Could you clarify how \"implicit alignment\" is achieved in your framework without a shared latent space or explicit alignment objective? As currently presented, the alignment appears to be an incidental byproduct of training rather than a mechanism that is theoretically or structurally grounded.\n2. Would you consider adding experiments to directly validate that the DP structure facilitates cross-modal alignment, rather than simply enabling modality fusion? Clear empirical evidence is needed to distinguish alignment from general feature integration.\n3. Please consider including comparisons with recent state-of-the-art explicit alignment models, including those published after 2024. This would provide a more convincing evaluation of how the proposed implicit alignment method performs relative to current best practices in cross-modal learning.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe proposed approach is conceptually novel, leveraging a shared Dirichlet Process prior to implicitly align multimodal representations.\nWeaknesses:\n1. Unclear Exposition: The lack of detailed mathematical formulations significantly hinders the reader’s understanding of the model design and underlying assumptions.\n2. Unclear Theoretical Justification of “Implicit Alignment”: The claim that the DP structure can \"automatically achieve an optimal balance between intra-modal representation and cross-modal alignment\" lacks theoretical support. While the DP prior emphasizes frequent components within each modality, it does not inherently ensure alignment across modalities. Without a shared latent space or explicit alignment objective, the observed alignment appears incidental rather than structurally guaranteed, and may not generalize well across datasets or domains.\n3. Missing Comparison with Explicit Alignment Models:The experiments do not include comparisons with state-of-the-art alignment methods such as contrastive multimodal models (e.g., CLIP, ALIGN, or CMML), which include direct semantic matching between modalities. This makes it difficult to assess whether the proposed implicit mechanism is truly competitive.", "was_revised": true, "final_justification": "The response just provides some related work about the effectiveness DP prior, but no theoretical justification is provided for the proposed method itself. Therefore, I will keep my rating.", "rebuttal": "Thank you for appreciating the novelty of our method. We would like to address your concerns below.\n\n> **Q1** Unclear Exposition: The lack of detailed mathematical formulations significantly hinders the reader’s understanding of the model design and underlying assumptions.\n\nWe apologize for the terse exposition and will expand the detailed notation and generative model in the revised manuscript; below we provide a formulation of **the model designs and key assumptions**.\n\nLet each sample $x_{im}$ from modality $m$ be mapped to a latent $z_{im}=g_m(x_{im})$.  We define a shared stick‐breaking process over the $M\\times K$ components:\n$$\n\\beta_{mk}\\sim\\mathrm{Beta}(1,\\eta),\\qquad\n\\pi_{mk}\n=\\beta_{mk}\\prod_{\\substack{i\\in[M],\\,j<k}}(1-\\beta_{ij})\n\\prod_{l<m}(1-\\beta_{lk}),\n$$\nso that each modality’s marginal is\n$$\np(z_{im}\\mid\\{\\pi_{mk},\\mu_{mk},\\sigma^2_{mk}\\})\n=\\sum_{k=1}^K\\pi_{mk}\\,\\mathcal{N}\\bigl(z_{im}\\mid\\mu_{mk},\\mathrm{diag}(\\sigma^2_{mk})\\bigr).\n$$\nWe **assume** diagonal covariances for tractability in high dimensions and truncate at $K$ for efficiency.  Variational posteriors over $\\beta_{mk},\\mu_{mk},\\sigma^2_{mk}$ are parameterized by neural networks, and we maximize the ELBO via stochastic gradients. \n\n> **Q2:** Unclear Theoretical Justification of “Implicit Alignment”: The claim that the DP structure can \"automatically achieve an optimal balance between intra-modal representation and cross-modal alignment\" lacks theoretical support. While the DP prior emphasizes frequent components within each modality, it does not inherently ensure alignment across modalities. Without a shared latent space or explicit alignment objective, the observed alignment appears incidental rather than structurally guaranteed, and may not generalize well across datasets or domains.\n\n>Could you clarify how \"implicit alignment\" is achieved in your framework without a shared latent space or\nexplicit alignment objective? As currently presented, the alignment appears to be an incidental byproduct of\ntraining rather than a mechanism that is theoretically or structurally grounded.\n\nWe appreciate the reviewer’s comment and clarify that implicit alignment in our framework is structurally and theoretically grounded. All modality encoders project inputs into a shared latent space, over which we place a single Dirichlet Process (DP) prior. The global stick-breaking process ensures that all modalities share the same pool of mixture components (atoms), and thus cluster around common latent structures. This mechanism is formally supported by the theory in Teh et al. (2006, Thm. 2), which proves that groups drawing from a shared DP base measure will naturally align via shared atoms. Further, consistency results from Ishwaran \\& James (2001) [2] show that DP mixture weights stabilize with sufficient data, reinforcing **this alignment as non-incidental**. While we do not impose an explicit alignment loss, this structure encourages data from different modalities to concentrate on common, high-probability components—achieving alignment through shared probabilistic structure, as also evidenced in our empirical results.\n\n[1] Teh, Yee Whye, et al. \"Hierarchical dirichlet processes.\" Journal of the american statistical association 101.476 (2006): 1566-1581.\n[2] Ishwaran, Hemant, and Lancelot F. James. \"Gibbs sampling methods for stick-breaking priors.\" Journal of the American statistical Association 96.453 (2001): 161-173.\n\n> **Q3: Missing Comparison with Explicit Alignment Models**: The experiments do not include comparisons with state-of-the-art alignment methods such as contrastive multimodal models (e.g., CLIP, ALIGN, or CMML), which include direct semantic matching between modalities. This makes it difficult to assess whether the proposed implicit mechanism is truly competitive.\n\n>Please consider including comparisons with recent state-of-the-art explicit alignment models, including those published after 2024. This would provide a more convincing evaluation of how the proposed implicit alignment method performs relative to current best practices in cross-modal learning.\n\nWe thank the reviewer for the suggestion. To address this, we include comparisons with **three representative explicit alignment models**: CLIP [3], ALIGN [4], and CoMM [5], all evaluated under the totally matched setting on MIMIC4. As shown in Table 1, our proposed method (DPMM) consistently outperforms these strong baselines across all metrics, demonstrating the effectiveness and competitiveness of **our implicit alignment strategy** in multimodal contrastive learning.\n\n[3] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning. PMLR, 2021: 8748-8763.\n\n[4] Jia C, Yang Y, Xia Y, et al. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning. PMLR, 2021: 4904-4916.\n\n[5] Dufumier B, Navarro J C, Tuia D, et al. What to align in multimodal contrastive learning? The Thirteenth International Conference on Learning Representations. 2025. \n\n### Table 1: Performance comparison with contrastive multimodal models for In-Hospital Mortality (IHM) and Readmission (READM) prediction\n\n| Model     | IHM AUROC | IHM AUPR | READM AUROC | READM AUPR |\n|-----------|-----------|----------|--------------|-------------|\n| CLIP      | 0.816     | 0.476    | 0.719        | 0.427       |\n| ALIGN     | 0.819     | 0.469    | 0.725        | 0.428       |\n| CoMM      | 0.823     | 0.465    | 0.726        | 0.432       |\n| **DPMM**  | **0.826** | **0.482**| **0.736**    | **0.455**   |\n\n> **Q4:** Would you consider adding experiments to directly validate that the DP structure facilitates cross-modal alignment, rather than simply enabling modality fusion? Clear empirical evidence is needed to distinguish alignment from general feature integration.\n\nThank you for your suggestions. Empirically, we validate this in the following Table 2: our ablation (\"Amplification+Alignment\") outperforms both \"Amplification only\" and \"w/o DP\" under fully and partially matched settings—demonstrating that **the DP structure delivers alignment beyond mere feature fusion**.  This empirical result confirms that shared DP components yield stronger, generalizable alignment across modalities.\n\n### Table 2: Performance on the Amplification and Alignment of DP\n\n| Match              | Model                      | IHM AUROC | IHM AUPR | READM AUROC | READM AUPR |\n|--------------------|----------------------------|-----------|----------|-------------|-------------|\n| **Totally Matched**  | w/o DP                     | 0.809     | 0.434    | 0.717       | 0.424       |\n|                    | Amplification only         | 0.816     | 0.462    | 0.724       | 0.429       |\n|                    | **Amplification + Alignment** | **0.826** | **0.482** | **0.736**   | **0.455**   |\n| **Partially Matched** | w/o DP                   | 0.855     | 0.506    | 0.753       | 0.459       |\n|                    | Amplification only         | 0.856     | 0.525    | 0.752       | 0.468       |\n|                    | **Amplification + Alignment** | **0.860** | **0.529** | **0.771**   | **0.486**   |"}]

Author rebuttal:

Meta-review: This work proposes Dirichlet-Process Mixture Multimodal Learning (DPMM) to avoid representation degradation caused by excessive inter-modality alignment in multimodal learning. It treats the latent representation distribution of each modality as a Gaussian-mixture model (K components) and generates the mixing ratios via a Dirichlet process with a stick-breaking construction, leveraging its rich-get-richer property to emphasize important features. For missing modalities, gradient-preserving sampling (GPS) performs probabilistic imputation while keeping gradients intact. The proposed method outperforms state-of-the-art approaches on four datasets, including clinical (MIMIC-III/IV) and general-domain (CMU-MOSI, POM) benchmarks.

There were some concerns about the theoretical justification and the comparison to explicit alignment models left. The authors should include those experiments as well as the points raised during rebuttal in the final version of the document.
Decision: Accept (poster)