Cross-Modal Alignment via Variational Copula Modelling
Authors: Feng Wu
Tsai Hor Chan
Fuying Wang
Guosheng Yin
Lequan Yu
Venue: ICML 2025 poster (2025)
Keywords: Copula
Multimodal learning
Missing modality
Healthcare
Primary area: applications->health_medicine
TL;DR: We develop a copula variational inference framework for cross-model alignment

Abstract: Various data modalities are common in real-world applications. (e.g., EHR, medical images and clinical notes in healthcare). Thus, it is essential to develop multimodal learning methods to aggregate information from multiple modalities. The main challenge is appropriately aligning and fusing the representations of different modalities into a joint distribution. Existing methods mainly rely on concatenation or the Kronecker product, oversimplifying interactions structure between modalities and indicating a need to model more complex interactions. Additionally, the joint distribution of latent representations with higher-order interactions is underexplored. Copula is a powerful statistical structure in modelling the interactions between variables, as it bridges the joint distribution and marginal distributions of multiple variables. In this paper, we propose a novel copula modelling-driven multimodal learning framework, which focuses on learning the joint distribution of various modalities to capture the complex interaction among them. The key idea is interpreting the copula model as a tool to align the marginal distributions of the modalities efficiently. By assuming a Gaussian mixture distribution for each modality and a copula model on the joint distribution, our model can also generate accurate representations for missing modalities. Extensive experiments on public MIMIC datasets demonstrate the superior performance of our model over other competitors. The code is anonymously available at https://github.com/HKU-MedAI/CMCM.

Cross-Modal Alignment via Variational Copula Modelling
Feng Wu Tsai Hor Chan Fuying Wang Guosheng Yin Lequan Yu'
Abstract
1. Introduction
Multimodal learning aims to aggregate information from multiple modalities to generate meaningful representations for downstream tasks. It has been widely explored in the context of vision-language models (Fu et al., 2023; El Ba- nani et al., 2023), audio-visual applications (Chen et al., 2023; Mo & Tian, 2023; Huang et al., 2023), image-video models (Girdhar et al., 2023; Gan et al., 2023) and health- care applications (Wu et al., 2024a; Hayat et al., 2022). For example, multimodal learning has been applied to various healthcare tasks such as clinical prediction tasks (Zhang et al., 2023; Wu al., 2024a), report generation (Song et al., 2022; Cao et al., 2023), and clinical trial site selec- tion (Theodorou et al., 2024). The existing fusion strategies can be divided into early, joint, or late fusion (Huang et al., 2020), where the joint fusion paradigm is the most popu- lar strategy and its core idea is model the interactions between the representations of the input modalities (Hayat et al., 2022). The resulting fused embedding encodes the structural interaction between the modalities, enabling accu- rate prediction for each modality.
Various data modalities are common in real-world applications (e.g.. electronic health records, med- ical images and clinical notes in healthcare). It is essential to develop multimodal learning meth- ods to aggregate various information from mul- tiple modalities. The main challenge how to appropriately align and fuse the representations of different modalities into joint distribution. Existing methods mainly rely concatenation or the Kronecker product, oversimplifying the interaction structure between modalities and in- dicating a need to model more complex interac- tions. Additionally, the joint distribution of latent representations with higher-order interactions is underexplored. Copula is powerful statistical structure for modelling the interactions among variables, as it naturally bridges the joint distribu- tion and marginal distributions of multiple vari- ables. We propose a novel copula-driven mul- timodal learning framework, which focuses on learning the joint distribution of various modali- ties to capture the complex interactions among them. The key idea is interpret the copula model as a tool to align the marginal distribu- tions of the modalities efficiently. By assuming a Gaussian mixture distribution for each modal- ity and a copula model on the joint distribu- tion, our model can generate accurate represen- tations for missing modalities. Extensive exper- iments public MIMIC datasets demonstrate the superior performance of our model over other competitors. The code is available at https: / //github. cwm HKUHHKKUMMMM -ww.MMMMdMMMMCCC
However, due to the heterogeneity of different modalities (c.g., electronic health records: EHRs, medical images, med- ical reports). properly aligning their distributions remains challenge. The existing alignment strategies mainly rely concatenation or Kronecker products which oversim- plify the interactions among different modalities. A recent work (Salzmann et al., 2022) emphasizes simple probabilis- tic assumptions on the marginals and neglects to explore statistical assumptions about the joint distributions of the modalities. This approach may result in biased fused repre- sentations, limiting the performance of downstream tasks and the generalizability and robustness of the resulting mul- timodal models. Therefore, there is still need for ap- proach that can more appropriately align the distributions of modalities and model the potentially complex interactions among them.
Copula models have shown great success in modelling the interactions of variables as they construct bridge between the joint distribution and their marginals (Cherubini, 2004). However, copula models are less explored in deep learning field as most existing approaches heavily rely on sampling- based methods (e.g., MCMC (Silva & Gramacy, 2009)),
'Equal contribution 'School of Computing and Data Science, University Hong Kong, Hong Kong, China. Correspondence to: Lequan Yu aqyuuhhku.hks
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Cross-Modal Alignment via Variational Copula Modelling
which are relatively slow and difficult to scale to modern deep learning settings (Smith & Loaiza-Maya, 2023). AI- though some recent works have attempted to introduce cop- ula to deep learning models through stochastic variational inference (Smith & Loaiza-Maya, 2023), the potential of copula in multimodal learning is still underexplored.
tural interactions of the modalities need be encoded in the fused representation for more effective multimodal learn- ing. Recently modelling the interaction between modalities has received increasing attention. Liang et al. (2024) pro- posed an information decomposition framework to define and quantify different types of interactions between modal- ities. Transformer-basee methods have greatly facilitated the progress by modelling the cross- cross-model tokens (Zhang et al., 2023; Theodorou et al., 2024). However, matching the correspondence with transformers introduces high computa- tional complexity, which prompts a more efficient approach for representation alignment.
Moreover, existing multimodal learning methods mostly assume the existence of all modalities. In reality, some modalities may be missing for some observations due to various reasons (e.g., missing medical images or reports for some patients due to clinical and administrative factors in healthcare), which may pose a major challenge in multi- modal learning. The existing solutions either discard these observations or impute simple values (e.g., zeros or means from other observations) to address the missing modality problem. However, these approaches ignore the marginal distributions of the modality and often mislead the learning of the joint distribution. Therefore, properly learning the marginal distributions is also necessary to generate unbi- ased latent representations for the observations with missing modalities.
Copula Deep Learning. Copula is a promising tool in modelling the interactions or correlations between variables and it constructs a bridge between the joint distribution and marginal distributions. Copula has been widely applied in financial risk management (Hofert, 2021; Rodriguez, 2007), signal processing, and healthcare (Zeng& & Wang. 2022) due to its capability in modelling complex interactions. Tradi- tional copula models rely on closed-form solutions of the likelihood and estimate the copula parameter with sampling- hased approaches (e.g., MCMC (Silva & Gramacy, 2009)). However, these algorithms suffer from high time complex- ity, making them less applicable to high-dimensional data. Recently, with the emergence of deep learning. there have been works integrating copula models into deep learning frameworks (Tagasovska et al., 2019; Smith et al., 2020). To tackle the inherent high dimensionality, variational inference is adopted to solve copula models in high dimensions (Tran et al., 2015; Smith & Loaiza- Mayaa 2023). For example, Tagasovska et al. (2019) introduced copula to variational autoencoders to create deep generative models. However, the potential of copula in multimodal learning is still under- explored.
In light of the aforementioned challenges, we propose novel copula-driven multimodal learning framework, namely CM² (Cross-Modal alignment via variational Copula Modelling), to tackle the joint fusion paradigm from prob- abilistic perspective. Our contributions can be summarized as: (1) We for the first time introduce copula modelling into multimodal learning, where we interpret copula as an effec- tive tool of distribution aligmment, guaranteed by Sklar's theorem. (2) We employ a Gaussian mixture model the marginal distribution of each modality to enable more flexi- ble modelling of the eigh-dimensional feature distribution of different modalities. (3) We adopt stochastic variational inference to optimize the copula model, which enables the scalability of our model to large-scale datasets. (4) We adopt the learned marginal distribution as the data generator to accurately impute the missing observations. (5) Empirical results on real multimodal MIMIC datasets demonstrate the good performance of our method and ablation analysis cor- roborates the effectiveness of copula in modality alignments and robustness to potential variations.
Learning with Missing Data. Traditional multimodal learn- ing assumes all modalities are available, but in reality, some observations may be missing, like medical images or reports in clinical data. Late fusion common strategy to ad- dress missing modalities by aggregating predictions (Yoo et al., 2019) or latent space representations (Theodorou et al., 2024) from the available modalities. While effective, it treats each modality independently and lacks interactions among them. Some research focuses on extracting shared information across modalities for downstream tasks (Del- dari et al., 2023; Yao et al., 2024), which can be challenging, particularly with heterogeneous modalities like EHRs and CXRs. Under the missing random (MAR) assumption, imputation methods have become a popular approach for handling missing data. Some approaches assume that the missing modality follows a certain distribution, imputing the missing values using the mean or mode of that distribu- tion (Ma et al., 2021). Others impute missing modalities'
2. Related Works
Multimodal Representation Learning. Multimodal repre- sentation learning aims to effectively integrate information from different modalities for accurate predictions on the downstream tasks. Early works (Hayat et al., 2022; Ding et al., 2022; Trong et al., 2020) focus on late fusion that merges unimodal representations via, for instance, concate- nation or the Kronecker product. However, such approaches oversimplify the interactions of the modalities and mostly lead to biased fused representations. Therefore, the struc-
2

---

Cross-Modal Alignment via Variational Copula Modelling
Modality:l
Modality:m
Modallty:hi
Emcoder
Encoder
Encoder
Mixture
Mixture Weight
Weight
Comcatenate
Multivariate Gaussaan Distribation
Multivoriate Gauesian Distribution
Marginal Disrribution
Marginal Distriburion
Fusion Module
im
Gradient Preserve Sampling
MI.P
Negativee Log-I lkelihood
Capola Distribution
Copula Distribution Function
ELBO
Multimodal Copula Learning
Figure 1. Overview of proposed CM² framework. For dataset with M modalities, extract modality-specifee embeddings Zm. Encoder,, and compute its Gaussian mixture model (GMM). We then model the marginal distrihutions and estimate the joint distribution using copula family C. We sample ê. from its GMM modality a 2-layer LSTM fusion module and MLP classifier to predict g. The ELBO for backpropagation can be obtained by aggregating the specifc loss (c.g., cross-entropy loss) and the negative log- likelihood from the joint distribution.
missing. The concatenated emhedding then passes through
representations in the latent feature space via deep learning models, attempting to preserve model performance by mod- eling relationships (Zhang al., 2022; Wu et al., 2024b) or generating global representations for the missing data (Hayat al., 2022). Despite their successes, these dis- tributional assumptions or the learned relationships may be inaccurate, potentially introducing bias into the model. Therefore a probabilistic assumption is needed to guarantee the unbiasedness of learned marginal distributions.
where g(,, eū a.) = (- (olog 1)) + ( (og v). The effects of different copula families are discussed the ablation analysis. Details of different copula families and their corre- sponding distribution and density functions are provided in Appendix C.
Multimodal Learning. Given the multimodal training dataset D.. = {(x ,..., Mr ) where x is the i-th observation of the m-th modality and ( the cor- responding label, the goal is train multimodal model Ma(-) with parameter θ such that the model achieve optimal performance in downstream tasks.
3. Methodology
3.1. Preliminaries
3.2. Copula Multimodal Learning
Copula. An M-variate function Clu CMI), where Um E [0, 1] for all m, is copula if and only if C defines valid joint cumulative distribution function (CDF) of the random vector (U1,.. UM) with each U. distributed as uniform on the unit interval. Taking the bivariate Gumbel copula as an example, given the CDF values of the first and second modalities and v. the bivariate distribution is
The overview of the proposed copula-driven multimodal learning framework is shown in Figure 1. Given multimodal data, we extract each mspelit embedding and com- pute its Gaussian mixture model (GMM). We then model the marginal densities and estimate the joint distribution using copula family C. If modality m is missing, we generate feature embeddings from its GMM. The concatenated em- beddings z passed through fusion module and MLP classifier for prediction. The evidence lower bound (ELBO) combines the copula log-likelihood and task- loss.
Clu, exp(- [og u)a + (- log v) =
and its copula density is
1
Gaussian Mixture Assumption. The GMM is common technique in machine learning to model the behavior of distributions in high dimensions (Song et al., 2024; Bai
c(u, u;a:) =
(- (-log (oog))--((-111)) 11
ig(u, a))]
201 w)
(a [0(u,
3

---

Cross-Modal Alignment via Variational Copula Modelling
Algorithm 1 Sampling algorithm of our proposed method.
parameter a as trainable. Algorithm presents the overall
1: Input:
workflow of our method.
2: Multimodal model M-(-) with parameter +
Variational Family. We use variational posterior q to approximate the true posterior of the joint distribution. The variational family of the copula model that we optimize during training is given by
3: The copula parameter a
4: Means and covariances of GMM: {Hemk, Emn) Im
1,..., N...,,,K. k = 1...., K}
5: Training set Die = {(x) (2)
(1)
M
y
6: Output: Trained fe
M
7: for (ar
Wi y (1) in Der do
(8)
((2) = (m
((Q((2))...,Qu(tx.))
Qu
8: g(t) = Ma
Lm=1
9: Compute task-specific loss Lojj with /( and y(
where 4m (Zm) is the density of the variational posterior of the GMM of the m-th modality, and Qm (Zm) is the corresponding CDF.
10: Compute the (ll(gl(T) ) and hence the ELBB
11: Backpropagate the ELBO to update θ, a
end for Return: Trained Me
The Evidence Lower Bound (ELBO). The joint objective function can be written as the negation of the negative log- likelihood,
13:
et al., 2022; Ni et al., 2021). To generate more flexible feature distribution, we assume the feature distribution of the m-th modality follows a K K-mixture of multivariate GMM,
log loge(Q1(211) (1.. (M(..))(MMM))))
ELBO
Xoop
t=1
Σ log fm (z/ + Lobj.
fm (zm)
Σ Tim Emk):
(1)
where fm is the marginal density of modality m, (z()) is the copula density, Acop is the regularization parameter of the copula, and Layj the task-specific loss (c.g., cross-entropy loss). We compute the gradient based on the ELBO and backpropagate it to μ and Σ to learn the marginal distributions of each modality, with the copula parameter to learn the interactions among these modalities and the multimodal model parameter θ to learn the embedding, fusion, and classification layers.
where Tink is the mixture weight, Mmnk is the mean vector, and Emn is the covariance matrix of the k-th mixture of the mnthh modality. Let μ {mmk m E [M], k [K]} and Σ [Em : m E [M].k E [K]]. Without loss of generality, we predict Tik with a multilayer perceptron (MLP) with a softmax output layer and adopt the reparame- terization trick (Nalisnick, 2018; Tran et al., 2022), which assumes Emk is diagonal. We further set μ and Σ be trainable by gradient backpropagation. We compute the mulative distribution function of the multivariate Gaussian distributions using the approximation provided in Marmin et al. (2015). By employing a mixture model, we can model a wider range of distributions of each modality and improve the flexibility and robustness.
a(
3.4. Handling Missing Modality
Owing to the probabilistic design of our method, our frame- work can also generate pseudo representations for missing modalities. Without loss of generality, we assume that the missing modalities are missing at random (MAR) and, fol- lowing prior works (Tran et al., 2017; Ma et al., 2021; Zhang et al., 2022; Wang et al., 2023), we impute the features of these missing modalities in the latent space. We consider missing modalities with complete labels where only the ob- servations are missing. The learned GMM for each modality can be treated as a data generation model, and we can gen- erate feature embeddings through sampling from the GMM of each modality (i.e., z/ ~ Fn.. Then the generated feature embeddings can be treated as the feature input to the classification layer and predictions can be obtained.
Multivariate Copula. Using the multivariate copula, the joint distribution function of the modalities is given by
Ear (z) = C(F((z)).......... FM((2M))
where (Z)),.)... Far(2M) )) is the M-ddimensional cop- ula distribution function, and (zm) is the marginal cu- mulative distribution function of the m-th modality which is the CDF of the GMM model defined in Eq. (I).
3.3. Stochastic Variational Inference
By learning the copula parameter a, the marginal distri- bution of each modality contains information from other modalities and information of the interactions. The gener- (x) ated feature representation 2m can thus better reflect the
To tackle the scalability of CM² to modern deep learning settings, we adopt the stochastic variational inference to optimize the proposed copula model and treat the copula

---

Cross-Modal Alignment via Variational Copula Modelling
characteristics of the joint distribution, which, as result, can improve the quality of the representation and the down- stream task performance.
Table 1. Numbers of samples training/validarionnestinng sets
Datasets
Truin
Valid
Test
Total
Complete
Datasets
MIMIC-III
14.681
3,222
3.236
21.139
MIMIC-III NOTE MIMIC-IV
3,652
815
806
5,273
3.5. Theoretical Guarantee with Sklar's Theorem.
18,064
2,035
4,972
25,071
MIMIC--XR
344,529
9,497
23,069
377,095
We make use of Sklar's theorem to demonstrate the unique- ness of the joint distribution as follows.
Mutched
Datasets
MIMIC-III NOTE
3,652
815
806
5,273
MIMIC-IV I CXR MIMIC-IV CXR REPORT
4,287
465
1,179
5,931
Theorem 3.1. (Sklar's theorem) (Sklar, 1959) Let F(z1,..., be M-variate CDF for (X1...., Xas) with the marginal CDF for the m-th variable given by Fm (I(())) m = L....,,.
4,287
465
1.179
5,931
provides an overview of the real datasets and the train- ing/validation/testing split sets. We extract 25,071 ICU stays with EHR records from MIMIC-IV, 5,931 of which are matched to CXR images and reports. Similarly, we extract 21, ICU stays with EHR records from MIMIC- III, with 5,273 stays matched to clinical notes. To evaluate the performance of CM2 on cross-mode moddl alignment, we con- duct experiments on totally matched bi-modal and tri-modal settings. We also evaluate partially matched datasets to demonstrate the robustness of CM² in the presence of miss- ing modalities. Further details on the datasets can be found in Appendix A.I.
1. There exists an M M-dimensional copula such that
(( = F(t1..... (2)
for all F. E R.
2. Conversely, given any copula C and univariate CDFs F1....,Fxs C is a valid CDF for (X1....,X, XM). If F is continuous, then C in Eq. (2) is unique.
The above theorem lays the foundation to construct joint distributions with the same marginals but different depen- dence structures, or conversely by fixing the dependence structure and varying the behaviour in individual modali- ties (Tagasovska et al., 2019). This allows us to update the marginal distributions and the copula parameter separately. Furthermore, since we assume a GMM for each modality and they are continuous by definition, the uniqueness of the copula C' can be guaranteed and the identifiability of the model can be enhanced.
Task and Evaluation Metrics. Following the common prac- tice in clinical prediction tasks (Hayat et al., 2022; Zhang et al., 2022; Wu et al., 2024b; Wang et al., 2024), we focus on two clinical prediction tasks: (1) In-Hospital Mortality (IHM) prediction, which predicts whether patient will pass away during the hospital stay; and (2) Readmission (READM) prediction, which aims to predict whether pa- tient will be readmitted within 30 days after discharge. To assess model performance, we compute the area under the precision-recall curve (AUPR) and the area under the re- ceiver operating characteristic curve (AUROC). Results are reported with the corresponding 95% confidence intervals based on 1,000 bootstrap iterations.
4. Experiments
4.1. Datasets and Experimental Setting
Backbone Encoders. Following Hayat et al. (2022), we uti- lize ResNet34 (He et al., 2016) as the backbone encoder for the CXR image data. For time-series data, we employ a two- layer stacked LSTM network (Graves & Graves, 2012). For clinical notes and radiology reports, we use the TinyBERT encoder (Jiao et al., 2019). projection layer is applied to map the modality embeddings into the same latent space.
Datasets. We evaluate the performance of CM2 using large- scale, real-world EHR datasets: MIMIC-III (Johnson et al., 2016), MIMIC-IV (Johnson et al., 2023), and MIMIC-CXR (Johnson et al., 2019). MIMIC-III and MIMIC-IV are pub- licly available datasets containing real-world EHR data from patients admitted to the intensive care units (ICUs) or emer- gency departments of Beth Israel Deaconess Medical Center (BIDMC), comprising numerical time series and clinical notes. MIMIC-CXR is a dataset of Chest X-ray (CXR) im- ages along with radiology reports collected from BIDMC, with subset of patients matched to those in MIMIC-IV.
4.2. Compared Methods
We compare CM2 against the following basclines: (1) MMTM (Joze al., 2020) is a flexible plugin mod- ule that facilitates information exchange between modali- ties. We address missing CXR and clinical notes during training and testing by filling in the missing data with ze- ros. (2) DAFT (Pölsterl et al., 2021) is a module designed to exchange information between tabular data and image modalities when integrated into CNN models. Similarly, we
Following Hayat et al. (2022), we utilize the MIMIC- IV and MIMIC-CXR datasets for our multimodal exper- iments. Additionally, we extend our experiments to the MIMIC-III dataset. As CXR images are not available in MIMIC-III, we replace them with clinical notes. Table
5

---

Cross-Modal Alignment via Variational Copula Modelling
Table Results of AUROC and AUPR with 95% confidence intervals on MIMIC-III and MIMIC-IV datasets with totally matched modalities. The best results are highlighted in boldface.
IHM
READM
Datasets
Models
AUROC (1)
AUPR (t)
AUROC (1)
AUPR (1)
MMTM (Joze et al., 2020)
728.0..8)))) (..7660.728,0.819)
0.347 (0.268,0.447)
0.716 (0.670.0.762)
0.341 (0.277,0.419))
MIMIC-I1
DAFT (Pölsterl et al., 2021)
0.792 746,0.839)
0 (0.299,0.484))
0.701 65360.76)
0.325 (0.262,0.406
Unified (Hayat et al., 2021)
0.827 (0. 782.0.868)
0.466 (..6660.371.0) (0.31110000
0.714 ((. (0.602,00 0.,779))
0.423 (0.344,0.00))
MedFuse (Hayat et al., 2022)
0.826 781, ,0.866)
0,430 (0.40000003
0.725 676.0.774)
0.414 (0.338,0.00)) 0.552)
DrFuse (Yao et al., 2024)
0.835 (0.5(0..988) (0..9.70.877
0.5 (0.417,0.607)
0.749 (0. 699.0.99.79. 095)
0,441 (0.356.0.527)
CM2
0.854 (0.820.0.80)
0.513 (0.16,00000
0.754 (0. 0.77)
0.445 (0.403.0.487)
MMTM (Joze et al., 2020)
0.802 (0.77,00 635.0.335
0.429 (0.362,0.513)
0.713 (0.677.0.755
0.420 (0.362,0.488)
MIMIC-IV
DAFT (Pölsterl et al., 2021)
0.815 (0.782,0.88)))
0.454 (0.387777
0.729 (0.692,0.76) 666)
0.433 (0.378,0.499)
Unified (Hayat et al., 2021)
0.808 (0.77800.810)
0.429 (0.367,0.512)
0.719 (0. 686)
0.450 0.450 (0.390,0.513
MedFuse (Hayat al., 2022)
(0.77.0.7 84.84)
0,448 0..28)
0.725 (0.690,0.768
0.438 (0.379,0.508)
DrFuse (Yao et al., 2024)
0.818 (0.7884444
0,460 (0.391 0.540)
0.726 (0. 2............ 760)
0.430 (0.370,0.44 0..55)
CM2
0.827 (0.790.0.859)
0.492 (0.423,0.566)
0.737 (0.704.0.773)
0.466 (0.404.0.525
Table 3. Results of AUROC and AUPR with 95% confidence intervals on MIMIC-III and MIMIC-IV datasets with partially matched modalities (i.e., missing modalities). The best results are highlighted in boldface.
IHM
READM
Datasets
Models
AUROC (1)
AUPR (t)
AUROC (1)
AUPR (t)
MMTM (Joze et al., 2020)
0.846 825,0.865)
0.450 (..39900000
(.
3.413 (0.371.0.455)
MIMIC-III
DAFT (Pölsterl et al., 2021)
0.854 (0.54(0.836.0877) 036.0.877)
0.495 (0.4400.5522)
0.748 (0.724,072)
0.429 (0.386,0.473)
Unified (Hayat et al., 2021)
0.849 (0.829.0.868
0.491 (0.436.0.542)
0.751 (0.728,0.772)
0.427 (0.383,0.467)
MedFuse (Hayat et al., 2022)
0.850 (0.60.0.0
0,480 (0.426,0.533
0.753 0.773)
0.437 (0.396,0.480)
DrFuse (Yao et al., 2024)
0.839 (0.817.0.86)
0.474 (0.42,20000
0.749 0.727.0.770
0.411 (0.371,0.455)
CM2
(0.33.0..33008777
0.510 (0.463,0.506)
0.754 (O. 0.7.99)955
0.445 10.358,0.522
MMTM (Joze et al., 2020)
0.855 (0.8000,0.0
0.519 (0.17700.6))
0.765 (0.717, 7.7.3.783)
0.465 (0.430,0.500
MIMIC-IV
DAFT (Pölsterl et al., 2021)
0.857 (1..8111,.87000
0.526 487,0.563)
0.765 (1.,77,0.77))
0,476 (0.76.0.442,0551) 441,0.1000
Unified (Hayat et al., 2021)
0.854 (0. 839,0.870
0.505 (0.463,0.5-4
0.759 (0. 776.0.776)
0.470 (0.466, (0.436,0.508 0.503
MedFuse (Hayat et al., 2022)
0.855 (0.80.00 (0.840.0.)7)
0.500 (0.4510..441)
0.762 (0. 7.7.77877
0.465 (0.430,0.501
DrFuse (Yao et al., 2024)
0.857 (0.811.0.872)
0.518 0.479.0.562)
0.768 (0.719.0.788)
0.485 (0.451,0.520)
CM2
0.858 (0.81110.882))
0.527 (0.490.0.568)
0.771 (0. (0..55.0778))
0.486 (0.452,0.518))
replace missing CXR and clinical notes with zero matrices during training and testing. (3) Unified (Hayat et al., 2021) a dynamic approach for integrating auxiliary data modali- ties and combining all representations via a unified classifier. It handles missing data inherently and leverages all available moodllity-specifee information. (4) MedFUSE (Hayat al., 2022) employs LSSTMbbased fusion to combine features from image or language encoders with EHR encoders. It handles missing modalities by learning a global represen- tation for absent CXR or clinical notes. (5) DrFuse (Yao et al., 2024) leverages disentangled representation learn- ing create a shared representation between the EHR and image modalities, even when one modality is missing.
Table 4. Ablation study on different alignment loss functions with AUROC and AUPR on MIMIC-IV.
Alignment loss
JHM
READM
AUROC (?)
AUPR (1)
AUROC (1)
AUPR (1)
Cosine KL.
0.820
0.470
0.726
0.445
0.826
0.489
0.731
0.446
Copula
0.827
0.492
0.737
0,466
4.3. Experimental Results
Quantitative Results. Table 2 presents results the MIMIC-III and MIMIC-IV datasets with totally matched modalities. CM² outperforms all the five baselines in all cases. Notably, for the IHM task, CM2 exceeds the best base- line by 1.9% in AUROC on MIMIC-III and 3.2% in AUPR on MIMIC-IV. These results demonstrate the effectiveness
6

---

Cross-Modal Alignment via Variational Copula Modelling
copula is more focused the positive dependence between the modalities while the Gaussian copula has lower weight modelling tail dependencies. On the other hand, the Frank copula is sail-symmetric and capable of modelling both positive and negative dependencies. Hence, it can cover more dependency structures, indicating that it may be a more flexible choice for modelling complex interactions. We further demonstrate how CM² learns the interactions through density plots at different epochs. The detailed discussion can be found in Appendix D. We also study how CM² learns the correlation over epochs. Figure 3 presents the change in the estimated a and its corresponding correlation 2-1 over training epochs. We discover that the model learns a positive correlation over the epochs, and the correlation converges at around 0.601. This implies that by backpropagating the gradient the copula parameter a, the model can learn the interactions between the modalities during training.
Table Ablation study on the influence of different components (e.g., copula alignment, pressering sampling (GPS), and fusion module) of our proposed method on MIMIC-IV.
Models
Matched
IHM
READM
AUROC (1)
AUPR (1)
AUROC (1)
AUPR (1)
w/o copula
x
0.855
0.506
0.753
0.459
wlo GPS
X
0.858
0,521
0.763
0.473
w/ fusion
0.860
0.531
0.762
0.476
X
0.858
0.527
0.771
0.486
wro copula
Y
0,809
0.434
0.717
0.424
w/o fusion
0.811
0.446
0.720
0.424
CN2
0.827
0.492
0.737
0.466
Table 6. Results on different copula families and the influence of the missing modality on MIMIC-IV.
IHM
READM
Matched
Copula
AUROC (?)
(1)
AUROC (t)
AUPR (t)
Gumbel
0,858
0.527
0.772
0.485
Gumbel
0.825
0.488
0.735
0.463
Frank
0.858
0.527
0.771
0.486
4.4. Ablation Analysis
Frank
0.827
0.492
0.737
0.466
Gaussian
0,859
0.527
0.771
0.485
Effectiveness of Copula Alignment. We study the effects of the alignment loss, as presented in Table 4. The cop- ula alignment loss achieves the best performance, outper- forming the popular cosine similarity alignment and KL divergence alignment.
Gaussian
0.827
0.488
0.736
0.458
of CM2 in capturing the interactions between modalities and enhancing the performance of multimodal learning tasks in clinical prediction.
Ablation on Contribution of the Designed Modules. To further evaluate the performance of cM2, we conduct an ab- lation study by removing the copula alignment, the gradient- preserving sampling (GPS), and fusion modules, respec- tively. As shown in Table 5, the performance of CM² sig- nificantly declines without copula alignment, underscoring the importance of modeling the copula joint distribution before fusing modality features. Additionally, in most cases, removing the fusion module leads to a notable drop in per- formance, emphasizing its critical role in capturing modality interactions. Furthermore, we observe slight decline when the GPS is removed, indicating its effectiveness in generat- ing unbiased representations for observations with missing modalities.
Table 3 reports results on the MIMIC-III and MIMIC-IV datasets with partially matched modalities (e.g.. missing modality). CM2 outperforms the basclines in all cases, with the best performance on the MIMIC-II dataset, where it outperforms the best baseline by 1.5% in AUPR for the IHM task and 0.8% in AUPR for the READM task. This indi- cates that CM2 effectively learns the joint distribution of the modalities, generating robust and unbiased representations in the presence of missing modalities.
Moreover, our results reveal that the performance on the partially matched datasets is superior to that on the matched datasets. This can be attributed to the larger number of observations in the partially matched datasets, underscoring the importance of multimodal learning in the presence of missing modalities. Lastly, we observe that the performance on MIMIC-IV is better than that on MIMIC-III under the partially matched setting, likely due to the larger number of observations in MIMIC-IV. Additionally, the heterogeneity between modalities in MIMIC-IV may be greater than that in MIMIC-III, contributing to the difference in performance between the two datasets under the totally matched setting.
Ablation on Different Families of Copula. We also com- pare the performance of CM2 under different settings for missing modalities and copula families. The accuracy relies heavily on the assumed copula family (Zeng & Wang, 2022). We examine the performance of our method over an array of commonly used copula families. Table 6 presents the results of CM² on the MIMIC-IV dataset. We discover that while our method is generally robust to the choice of copula family, the perrforfff copula varies across different tasks. This indicates that different tasks highlight different characteristics (e.g., extreme values for mortality) that can be captured when a proper copula family chosen.
Qualitative Analysis. We visualize the densities of different families of copula and see how the interactions between modalities are captured. Figure 2 presents the visualization of learned densities of the Gumbel, Gaussian, and Frank copula families, respectively. We observe that the Gumbel
Extension to More Modalities. We further investigate the
7

---

Cross-Modal Alignment via Variational Copula Modelling
Table Results of AUROC and AUPR with 95% confidence intervals using three modalities (EHR time series, CXR images, and CXR reports) MIMIC-IV.
IHM
READM
Models
AUROC (1)
AUPR (1)
AUROC (1)
AUPR (1)
MMTM (Joze et al., 2020)
0.777 (0.73,0.813)
(o.70(0.312.0.))
0.689 (0.650,0.725)
(o.34 0.436)
DAFT (Pölsterl et al., 2021)
0.788 (0.75,.0.)),
0.397 (0.331,0.171)
0.706 (0.6000000
0.403 (0.366 (0.316666
Unified (Hayat et al., 2021)
0.795 (0.76,0.
0.420 05.351,,.499
0.715 (0.679.0.719)
(o.30 (0.3766,.99)) 0.45)
MedFuse (Hayat et al., 2022)
0,801 (0.76700000))) 0.836)
0.427 (0.307 0.611)
0.713 (0.675,0. 0.79)
0.419 (o.35) (...66664877
DrFuse (Yao et al., 2024)
(0.08,0.773..888))
0.451 (0.376,0.624)
0.728 601.611..0 00.761)
0.433 (0.37000700 0..37000.444
CM2
0.824 (0. 0.795.0.856
0.471 0.554
0.730 (0. 00.094.0.6644
0.444 0.385,0.509)
2506
3579
2.00
2283
3124
270
04-
04-
04-
E570
L359
E822
6647
6500
1324
5.41
E33
00
0.2
0.0
02
36
3.0
00
0.2
Figure 2. Plots the fitted copula density to demonstrate the interrelationship captured by the copula model (Left: Gumbel, middle: Gaussian, right: Frank).
5. Conclusion
ED: 2.5444 lista
We introduce copula modelling into multimodal representa- tion learning. Using a copula can effectively model the inter- actions among different modalities, and impute the missing modalities through sampling from learned marginals. Em- pirical evaluation validates the predictive performance on the multimodal learning tasks, on both the fully and partially matched datasets. Ablation studies show that the proposed copula model can serve as a promising modality alignment tool due to the consistently satisfactory performance over different copula families. Our idea can be potentially ex- tended works that require effective fusion or distribution alignment, including domain adaptation, multi-feature and multi-view learning.
1.5
0.5
Step
20
40
60
80
Figure 3. Plots correlation,Cors (a 1)/a learned by the Gumbel cop- ula model.
comparing the value of a and the
impact of incorporating more auxiliary modalities. We adapt all baselines into the tri-modal setting. Table 7 presents the results for CM2 and the baselines on the MIMIC-IV dataset under the tri-modal setting: EHR time series, CXR images, and radiology reports. Across both tasks, CM2 consistently outperforms the baselines, achieving the best performance. Notably, the baseline models show decline in performance compared to the bi-modal setting, suggesting that incorpo- rating additional modalities becomes more challenging as the alignment complexity increases. Despite this, CM² main- tains strong performance, demonstrating its robustness and effectiveness in aligning multiple modalities.
Limitations and Future Works. Using neural network to learn the copula parameter a may be insufficient (since the joint log-lik ligeiiil may not be convex). Hence, an alterna- tive updating algorithm (e.g., partial likelihood) needed in future development copula multimodal learning to ensure that each loss convex to apply gradient descent. While we select healthcare datasets to demonstrate the effectiveness of our model, our method be extended to other types of multimodal datasets.
A Akknowledgmment. chairs, and reviewers for many constructive suggestions
We thank the program chairs, area
8

---

Cross-Modal Alignment via Variational Copula Modelling
that have significantly improved the paper. This work was supported in part by the Research Grants Council of Hong Kong (17308321, 27206123, C5055-24G, and T45-401/22-N), Patrick SC Poon endowment fund, Hong Kong Innovation and Technology Fund (ITS/273/22 and ITS/274/22), National Natural Science Foundation China (No. 62201483), and Guangdong Natural Science Fund (No. 2024A1515011875),
Fu, Z., Mao, Z., Song, Y., and Zhang. Y. Learning semantic relationship among instances for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15159-15168, 2023,
Gan, T., Wang, Q., Dong, X., Ren, X., Nie, L., and Guo, Q. Chvid-3.5 m: Build, filter, and pre-train the large-scale public chinese video-text dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14815-14824, 2023.
Impact Statement
Girdhar, R., El-Nouby, A., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Omnimae: Single model masked pre- training on images and videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10406-10417, 2023.
The goal of this work is to advance the field of machine learning. Our work can be potentially extended to areas that require effective fusion or distribution alignment, including domain adaptation, multi-feature learning, and multi-vien learning.
Graves, A. and Graves, A. Long short-term memory. Super- vised sequence labelling with recurrent neural networks, pp. 37-45, 2012.
References
Bai, J., Kong, S., and Gomes, C. P. Gaussian mixture vari- ational autoencoder with contrastive learning for multi- label classification. In international conference on ma- chine learning, pp. 1383-1398. PMLR, 2022.
Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., and Galstyan, A. Multitask learning and benchmark- ing with clinical time series data. Scientific data, 6(1 6(1):96, 2019.
Cao, Y. Cui, L., Zhang, L., Yu, F., Li, Z., and Xu, Y. Mmtn: multi-modal memory transformer network for image-report consistent medical report generation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pp. 277-285, 2023.
Hayat, N., Geras, K. J., and Shamout, E. Towards dy- namic multi-modal phenotyping using chest radiographs and physiological data. arXiv preprint arXiv:2111.02710. 2021.
Hayat, N., Geras, K. J., and Shamout, F. Medfuse: Multi- modal fusion with clinical time-series data and chest x-ray images. In Machine Learning for Healthcare Conference, pp. 479-503, PMLR, 2022.
Chen, J., Zhang, R., Lian, D., Yang. J.. Zeng. Z., and Shi, J. iquery: Instruments queries for audio-visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14675- 14686, 2023.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEFE conference computer vision and pattern recognition, pp. 770-778, 2016.
Cherubini, U. Copula methods in finance. John Wiley & Sons google schola, 2:949-956, 2004.
Deldari, S., Spathis, D., Malekzadeh, M., Kawsar, F., Salim, F., and Mathur, A. Latent masking for multimodal self- supervised learning in health timeseries. arXiv preprint MxXiv:2307.16847 2023.
Hofert, M. Right-truncated archimedean and related copu- las. Insurance: Mathematics and Economics, 99:79-91, 2021.
Huang, C., Tian, Y., Kumar, A., and Xu. C. Egocentric audio-visual object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22910-22921, 2023.
Ding, N., Tian, S.-w., and Yu, L. A multimodal fusion method for sarcasm detection based on late fusion. Multi- media Tools and Applications, 81(6):8597-8666, 2022.
Dosovitskiy, An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., and Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine, (1):136. 2020.
El Banani, M., Desai, K., and Johnson, J. Learning visual sampling. In Pro-
representations via language- ceedings of the ieee/cyf conference on computer vision and pattern recognition, pp. 19208-17 -9200, 2023.
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F,, and Liu, Q. Tinybert: Distilling bert

---

Cross-Modal Alignment via Variational Copula Modelling
for natural language understanding. arXiv preprint arXiv: /909. 19359999999. 2019.
Ni, J., Cheng, W., Chen, Z., Asakura, T., Soma, T., Kato, S., and Chen, H. Superclasscondditoone gaussian mixture model for learning fine-grained embeddings. In Interna- tional Conference on Learning Representations, 2021.
Johnson, A. E., Pollard, T. J.. Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., An- thony Celi, I.., and Mark, R. G. Mimic-iii, freely ac- cessible critical care database. Scientific data, 3()):199, 2016.
Pölsterl, S., Wolf, T. N., and Wachinger, C. Combining 3d image and tabular data via the dynamic affine feature map transform. In Medical Image Computing and Computer Assisted Intervention MICCAA 2021: 24th International Conference, Strasbourg. France, September 27 27-October 1. 2021, Proceedings, Part 24, pp. 688 988-698. Springer, 2021.
Johnson, A. E., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-y., Mark, R. G., and Horng, S. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Sci- entific data, 6(1):317, 2019.
Rodriguez, J. C. Measuring financial contagion: A copula approach. Journal of empirical finance, 14(3):401-4233 2007.
Johnson, A. E., Bulgarelli, L., Shen, L., Gayles, A., Sham- mout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
Salzmann, T., Pavone, M., and Ryll, M. Motron: Multi- modal probabilistic human motion forecasting. In Pro- ceedings of the IEEF/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 6457-6466, 2022.
Joze, H. R. V., Shaban, A., Iuzzolino, M. L., and Koishida, K. Mmtmm Multimodal transfer module for enn fusion. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp.
Silva, R. and Gramacy, R. Meme methods for bayesian mix- tures of copulas. In Artificial Intelligence and Statistics, pp. 512-519, PMLR, 2009.
2020. Khader, F. Müller-Franzes, G., Wang, T., Han, T., Taycbi Arastch, S., Haarburger, C., Stegmaier, J., Bressem, K., Kuhl, C., Nebelung, S., et al. Multimodal deep learning for integrating chest radiographs and clini- cal parameters: case for transformers. Radiology, 309 (1):e230806, 2023.
Sklar, M. Fonctions de répartition à n dimensions et leurs marges. In Annales de I'ISUP, volume 8, pp. 229-231, 1959.
Smith, M. and Loaiza-Maya, R. Implicit copula varia- tional inference. Journal of Computational and Graphical Statisties, 32(3):769-78) 2023.
Liang, P. P., Cheng, Y., Fan, X., Ling, C. K., Nie, S.,, Chen, R., Deng, Z., Allen, N., Auerbach, R., Mahmood, F,, et al. Quantifying & modeling multimodal interactions: An information decomposition framework. Advances in Neural Information Processing Systems, 36, 2024.
Smith, M. S., Loaiza-Maya, R., and Nott, D. J. High- dimensional copula variational approximation through transformation. Journal of Computational and Graphical Statistics, 29(4):729-743, 2020.
Song, A. H., Chen, R. J., Ding, T., Williamson, D. F., Jaume, G., and Mahmood, F. Morphological prototyping for un- supervised slide representation learning in computational pathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11566- 11578, 2024.
Ma, M., Ren, J., Zhao. L., Tulyakov, S., Wu, C., and Peng, X. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2302-2310, 2021.
Marmin, S., Chevalier, C., and Ginsbourger, D. Differenti- ating the multipoint expected improvement for optimal batch design. In International workshop on machine learning, optimization and big data, pp. 37-48. Springer, 2015.
Song. X., Zhang, X., Ji, Liu, Y., and Wei, P. Cross-modal contrastive attention model for medical report generation. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 1388-2397,, 2022.
Tagasovska, N., Ackerer, D., and Vatter, T. Copulas as high-dimensional generative models: Vine copula au- toencoders. Advances in neural information processing systems, 32, 2019.
Mo, S. and Tian, Y. Audio-visual grouping network for sound localization from mixtures. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10565-10574, 2023.
Theodorou, B., Glass, L., Xiao, C., and Sun, J. Framm: Fair ranking with missing modalities for clinical trial site selection. Patterns, 5(3), 2024.
Nalisnick, E. T. On priors for Bayesian neural networks. University of California, Irvine, 2018
10

---

Cross-Modal Alignment via Variational Copula Modelling
Tran, B.-H., Rossi, S., Milios, D., and Filippone, M. All you need is a good functional prior for bayesian deep learning. Journal of Machine Learning Research, 23(74): 1-56, 2022.
and Biomedical Engineering: Imaging & Visualization, 7 (3):250-259, 2019.
Zeng, Z. and Wang. T. Neural copula: A unified framework for estimating generie high-dimensional copula functions. arXiv preprint arXiv:2205. 15031, 2022.
Tran, D., Blei, D., and Airoldi, E. M. Copula variational inference. Advances in neural information processing systems, 28, 2015.
Zhang, C., Chu, X., Ma, L., Zhu, Y., Wang, Y., Wang, J., and Zhao, J. M3care: Learning with missing modalities in multimodal healthcare data. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2418-2428, 2022.
Tran, L.,, Liu, X., Zhou, and Jin, R. Missing modalities imputation via cascaded residual autoencoder. In Pro- ceedings of the IEEE conference computer vision and pattern recognition, pp. 1405-1414, 2017.
Zhang, X., Li, S., Chen, Z., Yan, X., and Petzold, L. R. Improving medical predictions by irregular multimodal electronic health records modeling. In International Con- ference on Machine Learning, pp. 41300-41313. PMLR, 2023.
Trong, V. H., Gwang-hyun Y.. Vu, D. T., and Jin-young, K. Late fusion of multimodal deep neural networks for weeds classification. Computers and Electronics in Agri- culture, 175:105506, 2020.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., and Carneiro, G. Multi-modal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15878-15887, 2023.
Wang, Y., Pillai, M., Zhao, Y., Curtin, C., and Hernandez- Boussard, T. Fairehr-clpp Towards cainess-awarr clinical predictions with contrastive learning in multimodal elec- tronic health records. arXiv preprint arXiv:2402.00955, 2024.
Wu, Z., Dadu, A., Tustison, N., Avants, B., Nalls, M., Sun, J., and Faghri, F. Multimodal patient representation learn- ing with missing modalities and labels. In The Twelfth International Conference on Learning Representations, 2024a.
Wu, Z., Dadu, A., Tustison, N., Avants, B., Nalls, M., Sun, J., and Faghri, F. Multimodal patient representation learn- ing with missing modalities and labels. In The Twelfth International Conference Learning Representations, 2024b.
Yao, W. Yin, K., Cheung, W. K., Liu, J., and Qin, Dr- fuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 38, pp. 16416-16424, 2024.
Yoo, Y., Tang, I.. Y., Li, D. K. Metz, I., Kolind, S., Tra- boulsee, A. L., and Tam, R. C. Deep learning of brain le- sion patterns and user-defined clinical and mri features for predicting conversion to multiple sclerosis from clinically isolated syndrome. Computer Methods in Biomechanics

---

Cross-Modal Alignment via Variational Copula Modelling
Summary
In this Appendix, we first present detailed information on the datasets in A. and tasks used in the experiments in A.2. Next, we introduce the multivariate Gaussian distribution in B and some common copula families in C. Then in D, we discuss the implications of how the copula model learns interactions over the epochs. Finally, we provide more details on the implementation and hyperparameters used in the experiments in E.1 along with the settings of baseline methods in E.2.
A. Additional Information on Datasets and Tasks
A.1. Datasets
Table 8 provides a summary of the datasets used in our experiments.
MIMIC-III dataset This dataset contains 46,520 ICU stays, each with 17 clinical variables. We split the dataset into training, validation, and test sets in the ratio of 70 15 15, following the procedure in Harutyunyan et al. (2019).
MIMIC-IV dataset This dataset includes 21,139 ICU stays, also with 17 clinical variables. The dataset split into training, validation, and test sets in the ratio of 70: : 10 20, following Hayat et al. (2022).
For both MIMIC-III and MIMIC-IV datasets, we extract 17 clinical variables commonly monitored in the ICU, including 5 categorical and 12 continuous variables. Data are sampled every two hours during the first 48 hours of ICU admission for both tasks, in accordance with Hayat et al. (2022). This results in vector representation of size 76 at each time step of the clinical seresseres data.
MIMIC-CXR dataset This dataset contains 377,1 10 chest X-ray images, of which 5,931 are associated with MIMIC-IV ICU stays. We split the data into 4,287 training samples, 465 validation samples, and 1,179 test samples. Following Hayat al. (2022), we retrieve the last Anterior-Postert projection chest X-ray and apply transformations to the images, resizing them to 224 x 224 pixels.
This dataset also includes radiology reports, which are unstructured text data. We choose the radiology reports of the MIMIC- dataset as an auxiliary modality to investigate the effectiveness of CM2 on more modalities aligmment since the radiology reports do not contain death information and can avoid possible overfitting and shortcuts. We divide the unstructured radiology reports into 4 sections, including Impression, Findings, Last paragraph, and Comparison.
MIMIC-III NOTE dataset This dataset consists 5,273 clinical notes associated with MIMIC- III ICU stays. The dataset divided into 3,652 training samples, 815 validation samples, and 806 test samples. In line with Zhang al. (2023), we select the last five clinical notes before the prediction time. If fewer than five notes are available, we treat the notes for that ICU stay missing. The original number of matched ICU stays is around 15,000. We randomly sample one-third of the matched ICU stays to form the training, validation, and test sets, keeping the scale of the notes nearly the same as the CXRs in the MIMIC-IV dataset.
Both radiology reports sections and clinical notes are capped at maximum length of 512 words, tokenized into words, and embedded into 312-dimensional vectors using the pre-trained TinyBERT model (Jiao et al., 2
A.2. Tasks
In-Hospital Mortality (IHM) Prediction. The In-Hospital Mortality (IHM) prediction task focuses on predicting whether a patient will pass away during their hospital stay. As summarized Table 8, the MIMIC-III dataset contains a total of 2,795 positive samples, of which 736 are matched with clinical notes. Similarly, the MIMIC-IV dataset includes 3,153 positive samples, with 890 matched CXR.
The Readmission (READM) prediction task aims forecast whether a patient will be readmitted within 30 days of discharge. In this task, both patients who readmitted and those who pass away in hospital considered positive samples. As shown in Table 8, the MIMIC-III dataset contains 3,987 positive samples, with 998 matched to clinical notes. In the MIMIC-IV dataset, there are 4,603 positive samples, with 1, 1,262 matched to CXRs.
Readmission (READM) Prediction.
General ____212D
12

---

Cross-Modal Alignment via Variational Copula Modelling
Table 8. Numbers of samples trainingyaaldddaioonessinng sets
Datasets
Tasks
Train
Valid
Test
Pos.
Total
Complete
Datasers
MIMIC-III
IHM READM
14681
3222
3236
2795
21 1339
MIMIC-III
14681
3222
3236
3987
21139
MIMIC-III NOTE
3652
815
806
5,273
MIMIC-IV
IHM READM
18064
2035
4972
3153
25071
MIMIC-IV
18064
2035
4972
4603
25071
MIMIC-CXR
344529
9497
23069
377,095
Matched Datasets
MIMIC-III NOTE
IHM
3652
815
806
736
5273
MIMIC-III NOTE
READM
3652
815
806
998
5273
MIMIC-IV CXR
IHM
4287
465
1179
890
5931
MIMIC-IV CXR
READM
4287
465
1179
1262
5931
MIMIC-IV I CXR REPORT
ПHM
4287
465
1179
890
5931
MIMIC-IV CXR REPORT
READM
4287
465
1179
1262
5931
B. Multivariate Gaussian Distribution
The multivariate Gaussian distribution is defined as
p ue, Σ) (2T 2 1
exp - (z 1)
"))T Σ 1 m
where ja E RP is p-dimensional mean vector and
E E RPxP is the covariance matrix.
The KL divergence two multivariate normal distributions N N(n, 21) and (uz,, 2;2) is
(, E2))
log
Σ2
+ E + ( m1) 11) 1 (12 -mi)
2
Σ
C. Common Copula Families.
We specify the copula distributions and density functions of common copula families with necessary derivations. Without loss of generality, we consider bivariate copula families.
Archimedean Copula. A subclass of copulas can be easily constructed by a generator function : [0, + [0, oo]. which strictly decreasing and convex so that (0) = oo and (0) ) = 0. Then, a copula C can be constructed as follows,
C(u1, ........ ua) p(11| Σ p(())
The Archimedean copula can generate copula densities when more than one modality exist in the dataset.
C.1. Copula Distribution Functions
Clayton copula
v( = [max[u (u - -
- -1,03] 1//
Frank copula
c c
((uv;a)==1 = a
log
a
where (O).).
13

---

Cross-Modal Alignment via Variational Copula Modelling
Gumbel copula
Clu, 0;)) = exp(- [[- log u)a (-log v) 19)²).
Gaussian copula
p) $b (u), $ (v);pp
where the CDF of the standard Gaussian distribution, and 22 is the bivariate Gaussian distribution.
Student's copula
C(u,u;p,v) = T2, TT <> 0:10 <1,
where T,-1 is the inverse of the CDF of Student's I-distribtuion with degrees of freedom v, and T2, is the bivariate t-distribtuion with degrees of freedom
C.2. Copula Density Functions Clayton copula
du, = (1 + (wv) 1-a(-1111 + u- a + v aaa
2 2-1/a
where a E (-1,
Frank copula
-ae alato) -
c(u,v) (--a e-au av + ac((to))
where a E (-00, a # 0.
14

---

Cross-Modal Alignment via Variational Copula Modelling
Gumbel copula
δ 0
c(u, v)
C(u, v)
du 0e
0 0
expl - [[-log + (- log (-
du dr
0 0
exp( [g(u, v; a)) 2/)
Du de
0
exp lg(u, 2g a)] //0
_ gglu, a)] -0"0
du o(u
a
Ou
0
expl - lo(u, v: /
Lo(u, 2;a)
a( ( 1og
Ou
(- log logg)) (2 [o(u, )
0
expl lo(u, lo
du
i 1 Ou gu, a;a)
+ expl - [o ux),
(- log v) )0-1 Ig(u, v - expl lo(u, a)) ( (o(( θ
Ou (u,,)
+ exp( 1opp-1a ig(u, 0; Kog 1-a
3z Lo(u, uy a)) vaal
Du gl v;a)
0
a
1 a2 lo(u, 201-0
v (- log 0-
Ou g(u, 0) expl- lou,
--1 a²
2a I9(u, Ui aaa
(- log 0)0- (- log u) 0' (olog))-Cuuu, v) [oo 1) [o(u, v;a)] 1=20 + +[9(u,;;)) -01
log v)-- log -- C(u, v) La(u, on a]] 20-0)
[ [a-11) lo(u, (oc,,) a0)]
The flsed-fform density of the trivariate Gumbel copula is computed by
θ a 0
cu,, u. w)
Clu, w)
Ou dv aw
0 0 Ø
expf
+ (- l)g + (- log w) 1/7
du de ow
expl (h(u, wi a))
Ou do Ow
1
(-
(bbgl))) C(,,
now
( (k(u,,, 30-3 (a -1)a w,; 20² 1) (h(u, ()))
(a-2)) 22 1))a w)))-a)
The identity can be generated by the Archimedean copula for M 3, which is less common in multimodal learning,
c(u) = (1(u)) II ( Y(uy),
where a():a)) (log t)at for the Gumbel copula, and (4-1) the first derivative of the inverse of yp
15

---

Cross-Modal Alignment via Variational Copula Modelling
10
1300
12700
2.48
1432
-75
C5094
00007
02
0122
02
C200
06025
624:
6997
10
0.8
00066
0.0
0.7
0.4
D6
00
022
64
106
1156
Figure 4. Plots of the copula densities of the Gumbel family at epochs 5, 50, and 100, respectively.
Gaussian copula The bivariate case is given by
(a² + b2) 2abp 2(1 - p²)
du, v; -T-P exp
where a = 12erf-1(w - 1), and b = 2eel (2u 1).
The multivariate case is given by the following matrix form,
cxp(-<-=~1</2
c(u; E) 1//2 Σ
-1/2 1/2
m=1 (2T)-
where Σ the covariance matrix and Ç= (0-1(u)) (..)),....... 1 x- (um))) T k , SM].
Student's / copula
T (022112-00000((00+)0000000000000
to t -
where v is the degree of freedom, I' is the gamma function, and
+ (1/2)d) (x) = I VVTI (2)
D. How Copula Learns Interactions.
We demonstrate how the copula model learns the interactions over the epochs and further discuss the implications.
Figure 4 presents the copula densities at epochs epochs 5, 50, and 100, respectively. We the Gumbel family as illustrative example. We observe that the copula density is evolving to positive correlation pattern, while the negative correlation scenarios (c.g., u 0.5, < 0.5. or < 0.5, v 0.5) are still considered but the weights allocated are decreasing.
E. More on Baseline Methods and Implementation Details
E.1. Implementation Details and Hyperparameters
We train all models for 100 epochs on the training and select the best-performing model based on the validation set, using the AUROC as the monitoring metric. The final results reported the test set. We optimize the models using the Adam optimizer and apply early stopping if the validation AUROC does not improve for 15 consecutive epochs to prevent overfitting. All experiments are conducted on single RTX-3090 GPU. The batch size is set to 32 for models trained on the MIMIC-IV & CXR datasets, and 16 for models trained on the MIMIC-III & NOTE datasets, except for DrFuse, which trained with batch size of We employ grid search to tune hyperparameters using the validation set and report the best results on the test set. The hyperparameter search space includes:
16

---

Cross-Modal Alignment via Variational Copula Modelling
Dropout ratio: (0,0.1,0.2,0.3)
Learning rate: (1 1,- 10-5, 10-3)
Number of Gaussian mixtures K: {1,2,3, 4,5,6)
Temperature: {0.001, 0.005, 0.01, 0.05, 0.08}
Regularization parameter Acop: {1 x 10-9,5 ,,x 10-6, x 10-9}
CM2 implemented in Python 3.11 using PyTorch 1.9, Following MedFuse (Hayat et al., 2022), we use ResNet34 (He et al., 2016) as the backbone encoder for CXR, a two-layer LSTM (Graves & Graves, 2012) as the encoder for time-series data, and pre-trained TinyBERT (Jiao al., 2019)² as the encoder for clinical notes. We include projection layer map modality embeddings into the same latent space. A two-layer LSTM is used as the fusion module combine modality embeddings, and multilayer perceptron (MLP) with one linear layer and a sigmoid activation function serves the classifier.
E.2. Additional Settings of Baseline Methods
We compare CM² with the following baseline methods.
MMTM (Joze et al., 2020) is a module that can leverage the information between modalities with flexible plugin architectures. Since the model assumes full modality, we compensate for the missing modality CXR and clinical notes with all zeros during training and testing. For clinical notes, we replace the ResNet34 encoder with TinyBERT to embed the clinical notes.
DAFT (Pölsterl et al., 2021) is module that can be plugged into CNN models exchange information between tabular data and image modality. Similarly, replace the input of CXR and clinical notes with matrices of all zeros during training and testing and use TinyBERT to embed the clinical notes.
Unified (Hayat et al., 2021) is a dynamic approach towards integrating auxiliary data modalities, learning the data representations for the individual modalities, and integrating the representations via a unified classifier. It inherently handles missingness and leverages all of the available modality-speciff data. Also, we use TinyBERT to embed the clinical notes.
MedFuse (Hayat al., 2022) uses an LSTM-based fusion combine features from the image encoder (or language encoder) and EHR encoder. Missing modality is handled by learning global representation for the missing CXR or clinical notes. We randomly initialized encoders for the time-series data, clinical notes, and CXR images.
DrFuse (Yao et al., 2024) uses disentangled representation learning to learn shared representation between the EHR and image modality even when one modality is missing. Drfuse uses ResNet50 as the image encoder and Transformer as the EHR encoder. We replace the ResNet50 encoder with TiRyBERT to embed the clinical notes.
The Implementation of DrFuse follows the original paper( Yao et al., 2024) and we use the same hyperparameters as the original paper. We directly adopt the implementations of MMTM, DAFT, Unified, and MedFuse provided by (Hayat et al., 2022)4, and all hyperparameters are set to the default values provided by Hayat et al. (2022). We adapt the implementations of MMTM, DAFT, Unified, MedFuse and DrFuse to tri-modal setting, including EHR time-series data, CXR images, and radiology reports.
F. Additional Experiment Results
Additional Baselines. We compare CM2 two additional healthcare baselines: L.MT (Khader al., 2023) and Interleaved (Zhang al., 2023). The results are shown in Table 9.
httpps:
htt.eeiiiiooattttttaaaeeeneeeeee hhtpps: :/ttis://giie thub. com com//dorothy-yy//rfuuss hhttps: thub. cwt.cco//nnn wtttb/ccoonnaaaccaaaaaaaa
17

---

Cross-Modal Alignment via Variational Copula Modelling
Table Results of additional baselines on the MIMIC-IV datasct. All results are reported in AUROC and AUPR with 95% confidence intervals. The best results are highlighted in boldface.
IHM
READM
Models
AUROC (1)
AUPR (1)
AUROC (t)
AUPR (1)
Totally Matched
LSMT (Khader et al., 2023)
0.803 (0.779900880000 0.837)
0.444 (0.370.0...... 5.99)
0.701 (0.602.0.737)
0.421 (o. (0..356,0.499
Interleaved (Zhang ul., 2023)
0.800 0.800 (0.764,0.888
0.440 (0. (0.374.0.523
(0.02(0.661.0.1
0.42 (0.360,0..0 4877)
CM2
0.827 (0.790.0.859)
0.492 (o. 423,0.560
0.737 (0.704.0.773)
0.466 (04,00..2
Partially Matched
LSMT (Khader et al., 2023) Interleaved (Zhang et al., 2023)
0.854 (0.83880.00000 0.800.870)
0.508 (0.466 0.651)
0. 0.764
0.473 (3.4033306000
(o. 0.0.8.711)
0.508 o. 0.550)
(..58.0.)
0.473 (0.3)o..1)))) 506)
CM2
0.858 (..2)).0.87)
0.527 (o. 190.0.568
0.771 (0.752.0.788)
0.486 (0.1,0000 152.0.518)
Paired_HHM
Partiel_IHM
- Paired__EeADM READM
Partiel_Ree READM
-0- Paired IHM
Partial IHM
Paired READM
Partial READM
0.59
0.9
0.57
0.86
0.53
0.51
AUROC
0.82
0.49
0.78
0.45
0.78
0.13
041
0.7
2
K
Figure 5. Results (left: AUROC; right: AUPR) of CM² MIMIC-IV, where the model reduces multivariate Gaussian disdtribution
when K- = 1.
LSMT (Khader et al., 2023) is a transformer-based model designed for the multimodal medical context.
Interleaved (Zhang al., 2023) is multimodal approach that addresses the irregularity of medical multimodal data and fuses representations from different modalities using cross-modal attention.
Effect of Backbone Encoders. Moreover, we explore the effectiveness of backbone encoders for both time-series data and CXR image data. We conduct additional experiments to evaluate the impact of different encoder architectures for each modality. Specifically, we use the Transformer (Vaswani, 2017) and ViT (Dosovitskiy. 2020) as alternative backbone encoders for the time-series and CXR image data, respectively. The results are shown in Table 10. We observe that method consistently outperforms competitive baselines across various backbone encoders, highlighting its robustness and effectiveness. Furthermore, our method demonstrates greater stability across different backbones, suggesting it is less sensitive to their selection. Besides, the Transformer backbone generally outperforms the LSTM backbone, particularly for MMTM, LSMT, and Interleaved. While the ResNet backbone slightly outperforms the ViT backbone, the performance difference not substantial, suggesting time-series data's greater impact on backbone encoder choice.
Effect of Number of Mixtures K. As a convention in statistical modelling, K is set be small to avoid wer-specification. The popular choice of K is 2 to 3 such that the learned mixture distribution can achieve an optimal degree of flexibility while preventing over-specification. We evaluate how the performance of CM² varies with different values of K, as shown in Figure 5. We observe that the performance is quite robust.
Statistical Tests The p-values of two-sample bootstrapped t-tests of the AUROC and AUPR of CM2 compared to bascline methods are shown in Table 11. We observe that the improvements over the competitive baselines are overall statistically significant under the 5% significance level, validating the effectiveness of our method.
18

---

Cross-Modal Alignment via Variational Copula Modelling
Table 10. Results of different backbone encoders and additional baselines on MIMIC-IV with totally matched modalities. All results are reported in AUROC and AUPR with 95% confidence intervals. The best results are highlighted in boldface.
Backbone TS
IHM
READM
Models
IMG
AUROC (t)
AUPR (T)
AUROC (1)
AUPR (1)
MMTM (Joze et al., 2020)
(0.32,2.2.50
(0.677..07
0.420
DAFT (Pölsterl et al., 2021)
0.815
(0454(0.387.0...
0.729 (0.69220. 766)
0.433 (o.33 0..499)
Unified (Hayat et al., 2021)
(0.08
0.429(0.367,00) (0.387,00051
0.719 (0.680,0.753
0.450 (0.390,0.54
LSTM
ResNer
MedFuse (Hayat al., 2022)
0.813 (0.77,0.777.00444
0.448 0.448 (0.380,0.528)
0.725 (0.800,0.762 762)
0.438 (,438 (03379.0000
DrFuse (Yao et al., 2024)
0.814 (0.70.0000)
0.450 0.450 (0.38..
0.723 (0.687.0 687,0.788 (.66))
0.422 (0.367 08466)
I.STT (Khader et al., 2023)
(0.03
0.444
0.701 (0.662.0. 737)
0.42 (0.356,0.44)
Interleaved (Zhang et al., 2023)
(0.06.0.76400.oo
0.440 (..70....0
0.702 (0.664,0.74 0.741)
0,42 (0. 0.4877
CM2
0.827 (0.790.0.850)
0.492 (0.42330.560)
0.737 (0.704.0.778)
0.466 (0.404000 0.5.0.
MMTM (Joze et al., 2020)
0.805 10..78,0.537
0,446 (..4660.377,00 (0.37)
0.7 (.712 (0.676.0.745
0.4 0.422 (0.3600000 0.44))
DAFT (Pölsterl et al., 2021)
0.808(0..75.0. (0.777.0..4)
0.438 (0438(0.265.0..5) (.1366.0.22)
0.7 0.714(0.678.)) (D.678.0. 753)
0.423 (0.3300030000 (0.36900.90)
Unified (Hayat et al., 2021)
0.803 0..03(0.768. (0.76,0. (0.76,)..83)
0.431 (0.365.0.515)
0.707 (0.667.0.7) 0.713)
0.4 16 16 (0.360,0.482
MedFuse (Hayat et al., 2022)
LSTM
VIT
0.805 (0.05(0.771..8888
0.439 (.331,,0544))
0.715 677,777777
O. 0.424 (.424(0.377.00444 0.449))
DrFuse (Yao et al., 2024)
(o.
0.446(0.379.00
(7.1667.77..))) (7777,
0,421 0,421 (0.364,00460)
LSMT (Khader et al., 2023)
0.801 (0.787,0.88) 0.3.66)))3
0.441 (0.374,0.527)
0.703 (.703(o..822 (0.682,0.78)
0.4 (4.3000558,0
Interlcaved (Zhang et al., 2023)
0.802 (0.766,0.833
0. 0.434 (0.364.0.509
0.710 0.67330.747
0.435 0,435 (0.372.0.50)
CM2
0.826 (..50000856)
0.490 (1.9000441..00 (0.421.0.663)
0.736 0.607,0.771
0.452 (0.394 0.339.0..522
MMTM (Joze et al., 2020)
0.813 (..13(0.780.. (0.7800..810))
0.452 0.452 (0.33.0.000
0.735 0.735 (0.699.0.77))
0.448 (0.388,0.515)
DAFT (Pölsterl et al., 2021)
0.814 (0.782,0.845)
0.437 (0.373.0.522)
0.730 (0.80(0 (0.604.0.700
0. 0.430 (0.3722..)
Transformer
Unified (Hayat al., 2021)
0.812 (0.776...))
0.453 (0.385.00000.. (0..5330.533
0.719 (0.681,0.2))) 754)
0.426 (442660.360 00.365,00 0.480.488)
ResNet
MedFuse (Hayat al., 2022)
0.815 (0.782.0.8440
0.44 (0.373,0.520)
0.728 (0.892,0..0))) 0.72)
0.442 (o.4880 (0.381,, 1,0005)))
DrFuse (Yao ct al., 2024)
0.818 (0.784..8500)
0,460 (0.39110.544)
0.726 (D.680 0..88.800 R,,,0.T.. 760)
0. 0.430 0.465
LSMT (Khader et al., 2023)
0.817 (0.76..88888
(0452.00.386.0.56)) (..33)
0.722 (0.688,0. 758)
0.431 (0.376.0.494)
Interleaved (Zhang et al., 2023)
0.821 (0.791.0.851)
0.459
0.389.00.539
0.72 0..88.00.757
0.429 (0.367,0.497
CK2
0.823 (0.788,0.855)
0.488
(0.42100.600)
0.740
(0.609...7))
0.470 (0.182.0.510)
MMTM (Joze et al., 2020)
3.813 (0. 0.8-16)
0.462 (0.326...66
0.723 (0.686.0 O. 0.761)
0.435 (0.380 (0.38000.00))
DAFT (Pölsterl et al., 2021)
(..0330..6800...
0.432 0.432 (0.3630..51)
0.719 0.719(0.682..) (0.682.0.78
0.42 l (0.367,0.45)
Unified (Hayat al., 2021)
Transformer
0.812 0.812 (0.778 (4.86)
0.463 (0.396.0.5540)
0.719 (0.680,0.0F))) 753)
0.41 (0.3200.333
MedFuse (Hayat et al., 2022)
VIT
0.818 (0.78660..))))
0.461 (0.393,0.542)
0.721 (0.684.0.755 (59)
0.43 (0.371,, 0..49)
DrFuse (Yao et al., 2024)
0.814 0.814 (0.780.0.845))
0.436 (..69.0..56)
0.717 (0.680.0.755)
0.4 ( (0.359.0.480)
LSMT (Khader et al., 2023)
0.815 (0.784.0.847)
0.453 (0.99.0. 58350))35)
0.714 0.714(0.676.0) (0.875.07551 7.11)
0.424, (4.24,0.0550.44)
Interleaved (Zhang et al., 2023)
0.818 10.786.0.8/9
0.453 0.453 (0.38000.0000 380.0.531)
0.71 (0.679.0. 753)
(o.33 374.0.498)
CM2
0.826 7.0.950.855)
0.489 (0.422.0.560)
0.737 (o. (0.700.0.772)
0.465 (0.394.0.517)
Table 1.1 P-values of two-sample bootstrapped t-tests of the AUROC and AUPR of CM2 compared to baseline methods. Most of the tests are significant under the 5% significance level.
IHM
READM
Models
AUROC (t)
AUPR (t)
AUROC (1)
AUPR (1)
MMTM (Joze et al., 2020)
2.02e-06
3.55e-180
4.40e-100
5.36e-291
DAFT (Pölsterl al., 2021)
0.1122
1.53e-132
9.37c-78
2.95c-240
Unified (Hayal et al., 2021)
4.55e-08
5.71e-240
4.80e-73
2.81e-139
MedFuse (Hayat et al., 2022)
7.73e-07
5.66e-129
1.1
3.69e-177 173
DrFuse (Yao al., 2024)
0.1447
4.28c-99
6.05c-67
6,25e-250
19


Reviews:
[{"review_id": "wXfkrwbSv9", "reviewer": "Reviewer_NgEi", "summary": "The paper proposed a copula modeling method for multi-modal representation learning, which could model the interactions between modalities and impute the missing modalities through sampling from learned marginals. The method was empirically evaluated on healthcare benchmarks MIMIC-III and MIMIC-IV datasets for two classification tasks. Code is provided in anonymous github.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank for your valuable feedback and constructive comments. We take great care in responding to several intriguing discussions raised by you as follows:\n> 1.Results on other types of multi-modal datasets\n\nThank you for the suggestion. To evaluate generalizability beyond healthcare, we conducted additional experiments on CMU-MOSI and POM. CM² achieves the best performance across all metrics (see table below) compared to other methods, demonstrating its broad applicability. We will include these results in the final version.\n|Model|**CMU-MOSI**|**CMU-MOSI**|**CMU-MOSI**|**POM**|**POM**|**POM**|\n|-|-|-|-|-|-|-|\n||MAE|Accuracy|F1|MAE|Corr|Accuracy|\n|Unified|1.21|0.656|0.657|0.862|0.213|0.353|\n|MedFuse|1.11|0.700|0.696|0.861|0.262|0.334|\n|DrFuse|1.12|0.700|0.700|0.869|0.243|0.338|\n|LMF|1.13|0.697|0.698|0.856|0.266|0.343|\n|TFN|1.18|0.682|0.682|0.858|0.263|0.358|\n|**CM²**|**1.08**|**0.710**|**0.708**|**0.840**|**0.281**|**0.365**|\n> 2.Significance of the result\n\nWe thank the reviewer for the thoughtful suggestions. We performed two-sample bootstrapped t-tests to compare CM² against **SOTA baselines** and **ablations**. Most comparisons yielded significant p-values, as shown in table below. We will revise Tables 2 and 3 by marking results with significant differences (e.g., using *) for better clarity. Due to space limits, we were unable to include the ROC curves now but will include them in the final version for improved visual interpretation.\n|Model|**IHM AUROC**|**IHM AUPR**|**READM AUROC**|**READM AUPR**|\n|-|-|-|-|-|\n|MIMIC-III Paired|3.95e-46|0.321|0.002|0.166|\n|MIMIC-IV Paired|1.47e-09|4.17e-19|1.34e-11|4.73e-33|\n|MIMIC-III Partial|4.02e-11|8.93e-32|0.290|0.003|\n|MIMIC-IV Partial|0.1447|4.28e-99|6.05e-67|6.25e-250|\n\n|Model|Matched|**IHM AUROC**|**IHM AUPR**|**READM AUROC**|**READM AUPR**|\n|-|-|-|-|-|-|\n|w/o Copula alignment|✗|0.001|1.73e-30|2.00e-93|3.47e-62|\n|w/o GPS|✗|0.398|4.83e-4|1.02e-22|7.07e-17|\n|w/o fusion module|✗|0.003|0.029|4.15e-28|7.09e-11|\n|w/o Copula alignment|✓|5.33e-34|9.48e-68|6.86e-35|4.79e-48|\n|w/o fusion module|✓|5.46e-26|1.64e-46|4.95e-26|4.02e-50|\n> 3.Choice of encoders\n\nWe appreciate the reviewer’s concern. While we followed encoder settings from original papers (e.g., MedFuse, DrFuse), we additionally tested each method with different encoders in **Table 10 in Appendix**. CM² consistently outperforms all baselines across these settings, confirming its robustness to encoder choices. We will clarify this experimental detail in the final version of the paper.\n> 4.In-depth analysis with regard to the healthcare context\n\nWe appreciate this insightful suggestion. Our method is designed to optimize global performance metrics through robust copula-based multimodal alignment, and it inherently relies on probabilistic inference and sampling, which introduce variability at the individual case level. In our framework, each prediction is generated as an expectation over a learned latent distribution that is optimized to maximize overall AUROC and AUPR, rather than to provide deterministic case-specific outputs. This stochastic nature—especially under missing modality scenarios—limits the reliability of direct one-to-one comparisons of individual case predictions across baselines. \n\n> 5.Deal with categorical features\n\nThank you for the question. Our GMM assumption applies to latent representations, not raw inputs. Categorical features are embedded and transformed before modelling; thus, the Gaussian assumption holds in the latent space.\n\n> 6.Impact by potential conflicts among different modalities\n\nWe appreciate the reviewer’s point. Our probabilistic framework regularizes modality interactions via distributional assumptions. This helps mitigate the impact of noisy or conflicting modalities by reducing sensitivity to outliers in the joint space.\n\n> 7.Typos\n\nThank you for catching this. The reference to Table 11 in Section 4.4 should be corrected to Table 6. We will fix this and another rendering error in the final version.", "claims_and_evidence": "Yes. Most claims are well supported by literature and/or experiments results, though some evidence (e.g. significance test) could only be found in the supplementary.", "methods_and_evaluation_criteria": "Yes. The proposed methods are evaluated on two classification tasks from MIMIC datasets. \n- From the results table (e.g. Table 2 and Table 3), it seems the AUROC performances are close to each other (e.g. $CM^2$ and DrFuse). It might be intuitive to compare the ROC curves for better interpretation. \n- As mentioned in the limitations and future works, other types of multi-modal datasets are needed to prove the utility of the proposed method beyond healthcare datasets.", "theoretical_claims": "No. The Sklar's Theorem in part 3.5 is well established.", "experimental_designs_or_analyses": "Yes.\n- Minor issues on experimental designs, especially on the choice of encoders for baseline methods. It seems some baseline methods and/or tasks are sensitive to the choice of encoders. Should the experiment design allow optimizing choice of encoders given a baseline method? Instead of using the same encoders for all baselines.", "relation_to_broader_scientific_literature": "The paper might provider an inspiring idea on how to align multi-modal distributions through copula modeling.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength: \n- The paper shows decent technical soundness (if considering some experiment results in the supplementary materials). Results, though not cross-validated, are tested with bootstrapping experiments. Confidence intervals are reported for most performance metrics, except table 5 ablation study and table 6 different copula families. \n\nWeakness: \n- Some of the important experiment design/results should be included in the main text, e.g. significance tests in table 11.  Performances with significant p-values could be marked with underline, or star, etc, in table 2 and 3. Otherwise the CIs seem to be non-significant at the first glance.  \n- The discussion of experiment results lack the in-depth analysis with regard to the healthcare context. Though the overall AUROC of the proposed method outperforms baselines, the predictions of individual cases might vary across baselines. It'll be better to analyze how much agreement/disagreement of individual cases predictions from different baselines, and what are the potential interpretation and pros/cons of the proposed method.", "other_comments_or_suggestions": "- wrong reference of tables in \"ablation on different families of Copula\" in section 4.4 on page 7. \"Table 11\" might be Table 6?\n- Are those measures in Table 5 and 6 statistically significant?", "questions_for_authors": "- How will the proposed method deal with categorical features, where Gaussian distribution cannot be assumed?\n- How will the proposed method be impacted by potential conflicts among different modalities, either due to data errors or by nature?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fymu175Mmh", "reviewer": "Reviewer_kwzw", "summary": "This work primarily focuses on the problem of multimodal supervised learning, where some modalities may be missing. The authors model the joint latent distribution of all modalities using a copula model with finite Gaussian mixture marginals. In the presence of missing modalities, they impute the missing latents by generating samples from the copula model conditioned on the available modalities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank for your valuable feedback and constructive comments. We take great care in responding to several intriguing discussions raised by you as follows:\n> 1.Modality Alignment Tasks & Alignment Losses\n\nIn our framework, modality alignment is achieved through the copula loss, which explicitly models and optimizes the dependency structure among modalities in a shared latent space. While we do not define separate alignment tasks per se, we evaluate the effectiveness of alignment by measuring cross-modal prediction performance—i.e., how well one modality can predict or reconstruct another. This serves as an implicit test of alignment quality. The copula loss, computed as the negative log-likelihood of the joint copula distribution, encourages aligned representations by capturing inter-modal dependencies beyond marginal distributions. We will clarify this connection more explicitly in the main text.\n\n> 2.Title of the Work\n\nThank you for your suggestions. We would revised the title in future versions of the manuscript to better reflect the contents of our work.\n\n> 3.Tail dependence and Copola Family Analysis\n\nTail dependence is critical in our application because it captures the co-occurrence of extreme events across modalities—events that are often indicative of critical outcomes in healthcare, such as severe physiological deterioration or acute anomalies in imaging. We specifically chose the Gumbel copula because its ability to model strong upper tail dependence aligns with our goal of highlighting these \"strongest signals\" present in the extreme ends of each modality's distribution; these signals, which correspond to rare yet significant observations, are essential for accurate risk prediction and decision-making. In contrast, a Gaussian copula would underrepresent these tail dependencies, potentially diluting the impact of extreme but clinically meaningful observations.\n\n> 4.How imputation for missing latents is performed\n\nYes, your assumption is correct—thank you for the accurate summary.\n> 5.Specify the dependence structure dealing with more than two modalities\n\nIn our current framework, we specify the dependence structure using symmetric copulas rather than adopting a vine copula structure. Specifically, we assume a fully connected density model where the multivariate Gumbel copula is given by\n$$C(u_1, \\dots, u_M) = \\exp\\{-[(-\\log u_1)^\\theta + \\cdots + (-\\log u_M)^\\theta ]^{1/\\theta}\\}$$\nand a general Archimedean copula is defined as\n$$C(u_1, \\dots, u_M) = \\phi^{-1}\\left( \\phi(u_1) + \\cdots + \\phi(u_M) \\right),$$\nwhere $\\phi$ is the generator function and $\\theta$ controls the dependence strength. This symmetric, feature-level assumption makes our approach task-agnostic and computationally tractable, avoiding the quadratic complexity that a vine copula structure would introduce when imputing missing modalities. We plan to investigate more complex dependence models, such as vine copulas with tailored structure selection, in future work.\n\n> 6.Discuss on Recent Multimodal Matching Work\n\nWe both adopt probabilistic assumptions on latent features—while Propensity Score Alignment (PSA) models intra-modal distributions from a causal perspective, our method leverages copula theory to model inter-modal dependencies, particularly capturing complex interactions such as tail dependence. We compare our method with PSA on MIMIC4 in the table below. Our method outperforms PSA in both matched and partially matched settings.\n\n**Totally Matched**\n|Model|**IHM AUROC**|**IHM AUPR**|**READM AUROC**|**READM AUPR**|\n|-|-|-|-|-|\n|PSA|0.744|0.346|0.692|0.370|\n|**CM²**|**0.827**|**0.492**|**0.737**|**0.466**|\n\n**Partially Matched**\n|Model|**IHM AUROC**|**IHM AUPR**|**READM AUROC**|**READM AUPR**|\n|-|-|-|-|-|\n|PSA|0.792|0.386|0.720|0.391|\n|**CM²**|**0.858**|**0.527**|**0.771**|**0.486**|\n\n> 7.Calling of ELBO\n\nWe use the term \"ELBO\" in a generalized sense to reflect a variational inference framework that combines both supervised and unsupervised objectives. Specifically, our method models the joint latent distribution via a copula and optimizes a variational lower bound that includes the copula log-likelihood (unsupervised) and a task-specific loss. For classification tasks, this supervised term corresponds to the KL divergence between the predictive label distribution and the true label (i.e., a variational form of cross-entropy), which aligns with the evidence lower bound on $\\log p(y)$ under a probabilistic decoder. For unsupervised tasks like clustering, this component can instead represent a metric-based loss (e.g., intra-cluster distances). Thus, our objective remains variational in nature, enabling principled integration of both labeled and unlabeled information, and we will clarify this probabilistic interpretation of $p(y|x_1, \\dots, x_M)$ more explicitly for mathematical rigor.\n\n> 8.Typos\n\nThanks for pointing out the rendering error. We will fix it in the final version.", "claims_and_evidence": "I'm not aware of unsupported claims, though I do have questions regarding the expt setting and methodologies.", "methods_and_evaluation_criteria": "- It might be due to my lack of knowledge, but it is unclear which specific modality alignment tasks are reported. Additionally, how are these alignment losses computed? Perhaps I missed it, but I could not find a precise description in the main text.\n\n- Most experiments in this work focus on performing supervised inference with missing modalities rather than analyzing the dependence structure across different modalities. Given this, I feel the title of the work could potentially be misleading.\n\n- Selecting an appropriate copula family is crucial, as different families exhibit distinct dependence properties. For example, the Gaussian copula lacks extreme tail dependence, whereas the Gumbel copula does. Could you elaborate on why tail dependence is relevant to your application and why you specifically chose the Gumbel copula? Additionally, how does tail dependence correspond to \"the strongest signals\" in each modality? Could you clarify this statement with precise reasoning?\n\n- Could you describe in detail how imputation for missing latents is performed? I assume the process involves first generating a uniform random variable from the copula function conditioned on the available modalities, followed by applying the inverse CDF of the Gaussian mixture to recover  z. Is this correct?\n\n- How do you specify the dependence structure in your multivariate copula function when dealing with more than two modalities? Do you adopt a vine copula structure? If so, how do you determine the vine structure for each problem? Alternatively, if you assume a fully connected density model between all modalities, this would introduce quadratic complexity in imputing missing modalities, which may become impractical with a large number of modalities. Could you clarify your approach?", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "see other comments.", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "Several recent works on multimodal matching and alignment are not discussed, such as *Propensity Score Alignment of Unpaired Multimodal Data* and *Unpaired Multi-Domain Causal Representation Learning*. I would appreciate a discussion on whether these methods could be applied to the modality alignment tasks presented in this work, as well as additional experiments benchmarking against these approaches.  \n\nAdditionally, it is worth noting that *Propensity Score Alignment of Unpaired Multimodal Data* also addresses the task of missing modality imputation, which should be mentioned and compared where relevant.", "other_strengths_and_weaknesses": "see other comments.", "other_comments_or_suggestions": "A few minor points:\n\n-  I find calling the objective an ELBO somewhat misleading, as it is unclear how it serves as a lower bound for the evidence $\\log p(x_1, .., x_m)$. This is not a VAE objective (which is typically unsupervised); rather, it appears to be a linear combination of the log-likelihood of the latent distribution and a supervised machine learning objective. Alternatively, if you are referring to the evidence $ \\log p(y)$, for mathematical rigor, it would be beneficial to explicitly write out the probabilistic model for $y|x_1, ..., x_M$ for clarification.\n\n\n- Latex rendering error in line 344.", "questions_for_authors": "see other comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7FikvLeTQ", "reviewer": "Reviewer_g4J9", "summary": "The paper presents a multimodal learning framework based on Copula theory. The modalities are modeled using a Gaussian mixture distribution, and a joint copula model is applied to the joint distribution. The proposed method is validated on a healthcare dataset, considering both cases where modalities are missing and where all modalities are present.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank for your valuable feedback and constructive comments. We take great care in responding to several intriguing discussions raised by you as follows:\n> 1.Impact of initial modeling of the marginals via GMM\n\nThank you for the insightful comment. Our initial marginal modeling, which assumes a feature-level representation, is inherently task-agnostic and thus universally applicable across different downstream tasks. By modeling each modality with a Gaussian mixture model—the most flexible and expressive assumption available in latent space - the framework can robustly capture complex, multimodal distributions. Although any marginal estimation could introduce potential biases, our joint optimization via the ELBO ensures that even minor discrepancies are corrected during copula estimation, allowing the dependency structure between modalities to be accurately aligned. \n> 2.Validation on datasets from other domains\n\nTo evaluate generalizability beyond healthcare, we conducted additional experiments on CMU-MOSI and POM. CM² achieves the best performance across all metrics (see table below) compared to other methods, demonstrating its broad applicability. We will include these results in the final version.\n|Model|**CMU-MOSI**|**CMU-MOSI**|**CMU-MOSI**|**POM**|**POM**|**POM**|\n|-|-|-|-|-|-|-|\n||MAE|Accuracy|F1|MAE|Corr|Accuracy|\n|Unified|1.21|0.656|0.657|0.862|0.213|0.353|\n|MedFuse|1.11|0.700|0.696|0.861|0.262|0.334|\n|DrFuse|1.12|0.700|0.700|0.869|0.243|0.338|\n|LMF|1.13|0.697|0.698|0.856|0.266|0.343|\n|TFN|1.18|0.682|0.682|0.858|0.263|0.358|\n|**CM²**|**1.08**|**0.710**|**0.708**|**0.840**|**0.281**|**0.365**|\n> 3.Theoretical guarantees\n\nWe acknowledge that our current theoretical guarantee, primarily rooted in classical results such as Sklar’s theorem, may appear limited in scope. However, our primary focus in this work is methodological and applied, targeting the complex challenges of healthcare multimodal data. Our main contribution lies in demonstrating the practical efficacy of integrating copula-based alignment with flexible Gaussian mixture modeling in real-world settings, which is supported by strong empirical results. We view our current theoretical framework as a solid foundation and plan to actively explore deeper theoretical insights—such as more rigorous guarantees on the learned dependency structure—in future work.\n> 4.The scalability to higher numbers of modalities\n\nWe assume a fully connected density model where the multivariate Gumbel copula can be obtained by the Archimedean copula\n$$c(\\mathbf{u}) = \\psi^{(d)}(t(\\mathbf{u})) \\prod_{j=1}^d (\\psi^{-1})'(u_j)$$\nwhere $\\varphi(t;\\alpha) = (\\log t)^\\alpha$ for the Gumbel copula Hence the higher number of modalities (i.e., when $M > 3$) can be handled in this case\n> 5.Compare to unsupervised approaches\n\nWe compared CM² against unsupervised multimodal VAEs including MoPoE-VAE and MVTCAE on MIMIC4. While these methods excel in synthetic and vision datasets, they are not tailored for downstream predictive tasks. In contrast, CM² integrates copula-based alignment with task supervision, capturing fine-grained dependencies (e.g., tail risks) that are critical in healthcare. As shown in tables below, CM² consistently outperforms these methods on both fully and partially matched IHM/READM settings. We attribute this to its stronger alignment of modality-specific representations under distributional assumptions that go beyond standard VAE objectives.\n\n**Totally Matched**\n|Model|**IHM AUROC**|**IHM AUPR**|**READM AUROC**|**READM AUPR**|\n|-|-|-|-|-|\n|MVTCAE|0.736|0.341|0.678|0.362|\n|MoPoE-VAE|0.730|0.338|0.671|0.357|\n|**CM²**|**0.827**|**0.492**|**0.737**|**0.466**|\n\n**Partially Matched**\n|Model|**IHM AUROC (↑)**|**IHM AUPR (↑)**|**READM AUROC (↑)**|**READM AUPR (↑)**|\n|-|-|-|-|-|\n|MVTCAE|0.767|0.347|0.701|0.366|\n|MoPoE-VAE|0.778|0.368|0.709|0.379|\n|**CM²**|**0.858**|**0.527**|**0.771**|**0.486**|\n\n> 6.Typos\n\nThanks for pointing out the rendering error. We will fix it in the final version.", "claims_and_evidence": "The claims are supported by evidence however the paper could benifit from additional discussions (see questions).", "methods_and_evaluation_criteria": "The experiments are well-structured and make sense.", "theoretical_claims": "The author uses Sklar’s theorem to demonstrate the uniqueness of the copula joint distribution. While this is valid in this context, it should be stated more clearly how the initial modeling of the marginals via GMM could impact this claim.", "experimental_designs_or_analyses": "The experimental design would benefit from validation on datasets from other domains beyond healthcare.", "relation_to_broader_scientific_literature": "The paper offers a new perspective on multimodal learning using copulas, which is interesting, especially given that multimodal learning is a broad and versatile domain.", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper is well-written and easy to read.\n- The method introduces copula theory to multimodal learning, which is an interesting approach.\n- The experimental validation demonstrates the method's performance compared to state-of-the-art techniques.\n\nWeaknesses:\n\n- The theoretical guarantees are not clearly stated.\n- The paper would benefit from validation on datasets beyond healthcare.", "other_comments_or_suggestions": "Broken reference in line 344.", "questions_for_authors": "1-  Can you discuss how initial marginal modeling (e.g., via GMM) might introduce biases or errors that impact the joint copula estimation.\n\n2- Can the method be generalized beyond healthcare and performs well in other domains.\n\n3 - Can you discuss the scalability to higher numbers of modalities (beyond bimodal/trimodal). \n\n4 - How does this method compare to unsupervised approaches like multimodal VAEs (see some references below) when labels are not available? The advantage of these methods is that they do not require labeled datasets to learn meaningful representations.\n\nSutter, T. M., Daunhawer, I., & Vogt, J. E. (2021). Generalized multimodal ELBO. arXiv preprint arXiv:2105.02470.\nHwang, H., Kim, G. H., Hong, S., & Kim, K. E. (2021). Multi-view representation learning via total correlation objective. Advances in Neural Information Processing Systems, 34, 12194-12207.\n\n\n**Post-rebuttal comment**\n\nThe rebuttal has addressed my questions, and I have decided to maintain my initial positive score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PVt59d5UYK", "reviewer": "Reviewer_WuxM", "summary": "This paper discusses a multi-modal learning algorithm utilizing copula to \"couple\" the marginal distributions in each modality. It employs an standard encoder for learning the latent representation for each modality, and model each latent representation as Gaussian mixtures. It then use a copula (selected from a parametric family) to model the joint distribution the modalities, and use mean-field variational inference to optimise the likelihood of the data. Experiments are conducted on single-modal datasets using several single MIMIC modality, and multi-modal datasets using different modalities from MIMIC.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank for your valuable feedback and constructive comments. We take great care in responding to several intriguing discussions raised by you as follows:\n> 1.Applied to generic tasks\n\nThank you for the suggestion. To support the generality implied by our title, we added results on CMU-MOSI and POM (**see Response 1 to Reviewer NgEi**). CM² consistently outperforms baselines, confirming its effectiveness beyond healthcare.\n> 2.The use of Sklar’s theorem\n\nThank you for the insightful comment. We use Sklar’s theorem to motivate the decomposition of multimodal joint distributions into marginals and a dependency structure, not as a guarantee of recovering the true copula. While the learned copula is parametric, it enables modeling diverse interaction types (e.g., tail dependencies). To address identifiability, we introduce priors and gradient-preserving sampling, leading to a unique (up to permutation) MAP solution. This theoretical foundation supports robust estimation in complex multimodal settings and connects classical copula theory with scalable deep learning. We will revise the claim to clarify its scope and avoid overstatement.\n> 3.Discuss with copula-based machine learning method\n\nThank you for the valuable pointers. Our approach leverages parametric families (e.g., Gumbel) to capture domain-relevant properties such as tail dependence. While neural copulas offer flexibility, they introduce additional complexity and may obscure interpretability—key considerations in healthcare. We will include these works in the related discussion.\n> 4.Significance of the result\n\nThank you for the comment. We performed two-sample bootstrapped t-tests comparing CM² with baselines; most results show statistically significant gains. Please see **Response 2 to Reviewer NgEi** for p-values supporting the improvements.\n> 5.The connection of this theorem to the pape\n\nThank you for the comment. While Sklar’s theorem is classical, our contribution lies in operationalizing it within a scalable deep multimodal framework. We leverage its decomposition to decouple marginals (modeled via GMMs) from dependencies (via parametric copulas), enabling feature-level alignment under missing modalities. Our framework also addresses challenges like marginal non-identifiability and tail dependence through variational inference and gradient-preserving sampling. This structured integration of copula theory into end-to-end multimodal learning constitutes a novel and practical contribution beyond simply restating the theorem.\n> 6.Tail distribution of diffirent copula family\n\nWe appreciate the reviewer’s point. The qualitative discussion of copula families is relevant because it highlights how different dependency structures—especially tail behaviors—affect the joint modeling process. In domains like healthcare, modeling extreme events is critical for risk-sensitive tasks. This analysis also guides practitioners in selecting appropriate copula families based on data characteristics, enhancing the interpretability and adaptability of our method.\n> 7.How to ensure that the marginal distributions of the modalities are uniform on the unit interval\n\nWe ensure uniform marginals via the probability integral transform. In practice, for each modality we first model the latent feature distribution (typically using a flexible Gaussian mixture model) and then compute its cumulative distribution function (CDF). By transforming the latent variables through their respective CDFs, we obtain variables that are uniformly distributed over [0, 1], which is guaranteed by the properties of the CDF. This transformation is a core component of our framework, aligning the modality-specific representations to a common uniform scale and facilitating the subsequent copula-based joint modeling.\n> 8.How much different would there be if a different copula family is used\n\nDue to the limited number of observations, the performance across copula families tends to be more variable in the matched subset . On the other hand, the observations are more sufficient in the partially matched datasets, leading to relatively stable performance across families. This demonstrates the importance of choosing a correct copula family since the tail risks is more evident as the number of observations decreases.\n> 9.Explicit form of $ L_{obj}$ \n\nThe overall objective loss in our framework is defined as $L_{obj} = L_{task} + \\lambda_{cop} \\cdot L_{\\text{copula}}$, where $L_{task}$ is the task-specific loss (such as cross-entropy for classification tasks) and $L_{\\text{copula}}$ is the negative log-likelihood of the joint copula model, with $\\lambda_{cop}$ balancing the two terms. We would proofread to correct the notational errors in future versions of the manuscript.\n>10.Typos\n\nThanks for pointing out the rendering error. We will fix this in the final version.", "claims_and_evidence": "The main claims in this paper is that using copula for the joint distribution of modalities improves performance. This has been supported by experiments on healthcare dataset, namely MIMIC. However, the title of this paper \"Cross-Modal Alignment via Variational Copula Modelling\" gives the impression that the method proposed is designed for generic tasks. For this title, experiments should be conducted on more broader areas of multi-model learning.", "methods_and_evaluation_criteria": "The evaluation criteria is area undre the ROC curve and area under precision-recall curve, which are approriate for the problem at hand.", "theoretical_claims": "There is a theoretical claim regarding the \"uniqueness of the joint distribution\". Unfortunately, the paper simply states the well-known (at least in the field of copula) Sklar's theorem. Citing this theorem here offers almost no insight into the actual multi-modal learning algorithm proposed in the paper. More specifically, the theorem says there exists a unique copula corresponding to the joint distribution; this has no implication on whether the copula learned in this paper is the true underlying copula. In fact, as the copula learning procedure only learns the parameter of a manually selected parametric copula family, it would be unlikely that this learned copula corresponds to the true one. \n\nIt would be better if this theoretical claim is removed from the paper.", "experimental_designs_or_analyses": "The experimental design is reasonable.", "relation_to_broader_scientific_literature": "This paper may contribute to the broader scientific literature with its improved performance of multi-modal learning algorithms.", "essential_references_not_discussed": "In the field of copula-based machine learning methods, there are many recent work which use neural network to represent and learn the copulas. This eliminates the step of manually choose a family of copula and offers better data likelihood. These referenes are entirely missing from the discussion. For example: \n1. Ling et al., Deep Archimedean Copulas. NeurIPS 2020.\n2. Zhang et al., Deep Copula-Based Survival Analysis for Dependent Censoring with Identifiability Guarantees. AAAI 2024", "other_strengths_and_weaknesses": "Strength: The main strength of this paper is that there are some improvements over the compared methods. However, I am not familiar with the latest literature in multi-modal classification and thus cannot judge if these are significant enough for ICML.\n\nWeakness: The contribution in the aspect of copula learning is rather limited, if any. The theoretical claim in this paper is a reiteration of a well-known (or rather, the most well-known) result in copula theory. The connection of this theorem to the paper is limited. \n\nThe qualitative discussion section mainly discusses the characteristics of different families of copulas, e.g., which copula family captures the tail distribution. These are of little relevance to the proposed algorithm's contribution.", "other_comments_or_suggestions": "The paper should be more carefully proofread to avoid minor mistakes. For example: \n1. On page 7, line 343, the reference number is not correctly formatted.\n2. The $L_{obj}$ in page 4, line 183 should be explicitly provided, at least for the tasks considered in the experiments.", "questions_for_authors": "1. How to ensure that the marginal distributions of the modalities are uniform on the unit interval? \n2. For the results reported in the tables, how much different would there be if a different copula family is used?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This manuscript presents CM², a copula-based framework for multimodal representation learning that models interactions between modalities and handles missing modality imputation through learned marginal sampling. The work demonstrates strong empirical performance on healthcare benchmarks MIMIC-III/IV and additional validation on CMU-MOSI and POM datasets. The reviewers highlighted several strengths, including the paper's technical soundness supported by comprehensive statistical testing, the broad applicability of the copula-based approach beyond healthcare domains, and the practical utility of capturing tail dependencies for risk-sensitive tasks. The rebuttal successfully addressed initial concerns about generalizability by providing additional experimental results on non-healthcare datasets. Some limitations noted by reviewers included the need for more theoretical guarantees beyond Sklar's theorem and deeper analysis of individual case predictions in healthcare contexts. The authors' thorough response clarified the methodological choices, particularly regarding copula family selection and marginal distribution modeling. After the rebuttal, reviewers maintained or improved their initial positive assessments, with scores ranging from weak accept to accept, appreciating the additional experiments and clarifications provided. The consensus highlighted the work's methodological contribution in integrating copula theory with deep multimodal learning, while acknowledging areas for future theoretical development.
Decision: Accept (poster)