Disentangled Cross-Modal Representation Learning with Enhanced Mutual Supervision
Authors: Lu Gao
Wenlan Chen
Daoyuan Wang
Fei Guo
Cheng Liang
Venue: NeurIPS 2025 poster ()
Keywords: Variational Autoencoder; Disentangled Representation Learning; Mutual Supervision
Primary area: deep_learning

Abstract: Cross-modal representation learning aims to extract semantically aligned representations from heterogeneous modalities such as images and text. Existing multimodal VAE-based models often suffer from limited capability to align heterogeneous modalities or lack sufficient structural constraints to clearly separate the modality-specific and shared factors. In this work, we propose a novel framework, termed **D**isentangled **C**ross-**M**odal Representation Learning with **E**nhanced **M**utual Supervision (DCMEM). Specifically, our model disentangles the common and distinct information across modalities and regularizes the shared representation learned from each modality in a mutually supervised manner. Moreover, we incorporate the information bottleneck principle into our model to ensure that the shared and modality-specific factors encode exclusive yet complementary information. Notably, our model is designed to be trainable on both complete and partial multimodal datasets with a valid Evidence Lower Bound. Extensive experimental results demonstrate significant improvements of our model over existing methods on various tasks including cross-modal generation, clustering, and classification.

Disentangled Cross-Modal Representation Learning with Enhanced Mutual Supervision
Lu Gao"
Wenlan Chen
School of Computer Science and Engineering Central South University
School of Computer Science and Engineering Central South University
Changsha, China 944711035@csu.edu
Changsha, China 2447010418csu, edu. cn
Daoyuan Wang
Fei Guo
School of Computer Science and Engineering
School of Computer Science and Engineering Central South University
Central South University
Changsha, China 2447010400csu.edu. en
Changsha, China guofei@csu. edu,
Cheng Liang?
School of Information Science and Engineering
Shandong Normal University
Jinan, China al clcc4100sdnu.cm edw. cn
Abstract
Cross-modal representation learning aims to extract semantically aligned rep- resentations from heterogeneous modalities such images and text. Existing multimodal VAE-based models often suffer from limited capability to align het- erogeneous modalities lack sufficient structural constraints to clearly separate the modality-ppccific and shared factors, In this work, we propose novel frame- work, termed Disentangled Moosll Representation Learning with Enhanced Mutual Supervision (DCMEM). Specifically, our model disentangles the common and distinct information across modalities and regularizes the shared representa- tion learned from each modality in mutually supervised manner. Moreover, we incorporate the information bottleneck principle into our model to ensure that the shared and modality-specine tion. Notably, our model is designed be trainable on both complete and partial multimodal datasets with valid Evidence Lower Bound. Extensive experimental results demonstrate significant improvements our model over existing methods various tasks including cross-modal generation, clustering and classification.
factors encode exclusive yet complementary informa-
Introduction
Cross-modal representation learning aims bridge the semantic gap between heterogeneous data from different modalities, such as images, text, audio and video [1, 2, 3]. The key challenge of the task lies in capturing both the modality- speccifc features and the shared semantic structure across modalities. despite their inherent differences in format and statistical properties [4]. Disentangled representation
Both authors contributed equally to this work and are joint first authors.
t "ooresponniig authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Self Retf-eeconttttioonn
Self- Reffoetoostrinnnn
p(w)
Conv
QUW,
Reg.
Reg.
Disentuglemen lemeng
Cony
3
(4, W.
gle,/a)
max (.,
gtx,ir
Z
Cross-modal generation
Downstream tasks
Mutual Supervision
(2 w)
p(w)
Clussification
Clustering
Comr
Figure I: The architecture DCMEM. For each modality, the encoder extracts shared (Zz, x1) and modality-specific (w,: w.; latent variables. relftreconttructionn and KI. regularization (denoted as "Reg.") ensure disentanglement, where w follows a standard Gaussian prior and shared variables serve priors for each other. Shared representations lie in common latent space Z, enabling mutual supervision via cross-modal paths s - Z + and - Z > s. We further align 2, and z; by maximizing their mutual information. DCMEM supports cross-modal generation, classification and clustering.
learning has thus emerged as a promising approach to address this challenge by explicitly separating modality-invarie (shared) factors from modality-specific (private) ones, enabling more interpretable and robust cross-modal representations [5]. Variational Autoencoders (VAEs) have been a prevailing framework for disentanglement with their probabilistic formulation and ability to learn structured latent representations, In particular, multimodal VAEs have gained significant attention for their ability to jointly encode and decode information from multiple modalities in unified probabilistic space [6]. From a methodological perspective, multimodal VAEs can be broadly divided into two categories: models that focus learning shared latent variables and those that incorporate modality-specific private variables. The former (e.g.. MVAE [7], MMVAE [8], MoPoE [9]. MEME [10]) aim to capture the common semantic structure across modalities, while the latter (e.g., MMVAE+ [11], CMVAE [12], IAA [13], Multi- [14]) explicitly disentangle shared and modality-speciff information by introducing separate latent spaces. However, these methods either suffer from limited capability to align heterogeneous modalities or lack sufficient structural constraints between modalities, which hinders the clear separation of informative shared and modality-specific factors. In this work, propose novel multimodal framework, termed Disentangled Cross-Moda representation learning with Enhanced Mutual supervision (DCMEM), to address these challenges. As illustrated Figure our model disentangles the common and distinct information across modalities by extracting shared and modality -speeiff latent representations using multiple VAEs. Specifically, we regularize the shared representation learned from one modality using that from the other in mutually supervised manner. Moreover, we incorporate the information bottleneck principle into our model ensure that the shared and modality-specife factors encode exclusive yet complementary information for reconstructing each modality. Finally, we apply an alignment constraint the shared latent space to promote consistent semantic and inter-modality coherence. Notably, our model is designed to be trainable simultancously on both complete and partial multimodal datasets with a valid Evidence Lower Bound (ELBO). The main contributions of this paper summarized follows:
We propose DCMEM, a novel multimodal framework that disentangles effectively shared and modality-sspeiif factors. Our model leverages enhanced mutual supervision to improve cross- modal alignment and semantic consistency, while simultancously enforcing structured representa- tion learning through the information bottleneck principle.
Our model is inherently designed to be trainable on both complete and partially missing modalities simultancously. This improves the robustness of our model and broadens its applicability to real-world scenarios where incomplete modalities are prevalent.
Extensive experiments on three diverse datasets demonstrate that our method produces more coherent and informative embeddings compared to existing multimodal VAE-based approaches

---

in terms of various evaluation metrics, including generation coherence, clustering accuracy and classification accuracy.
2 Related Work
2.1 Multimodal VAEs
Multimodal generation tasks have gained significant attention, driving research various generative models. Among them, multimodal VAEs have become particularly popular due to their impressive performance. Early multimodal VAEs [15, 16, 17] typically employ joint encoders
concatenated inputs, often requiring auxiliary components or processes to support cross-modal generation. Wu al. [7] propose scalable method based the Product of Experts (PoE) address this problem. However, subsequent research [8. 9, 18, 19] identifies several issues with PoE, such as calibration errors. To address these limitations, Shi al. propose MMVAE [8], using Mixture of Experts (MoE) to model the joint posterior as a mixture of unimodal posteriors. The MoPoΣ model [9] combines PoE and MoE to balance semantic coherence with effective joint distribution learning. Several studies [11, 20] highlight a trade-off between generation quality and coherence in multimodal VAEs. PoE-based models typically achieve lower coherence, while MoE- models suffer from lower generation quality [11, 21]. To address these issues, various models incorporate different regularizers to improve performance. For instance, MVTCAE [22] uses mutual information theory for regularization, mmJSD [23] employs a dynamic prior to combine modality information, MEME [10] enhances performance through mutual supervision and MMVM [24) regularizes learned posterior approximations with a data- data-dependent prior. Moreover, Palumbo et al. introduce MMVAE+ [11], which incorporates modality-specific subspaces to improve cross-modal likelihood estimation [23, 25]. Building MMVAE+, CMVAE [12] introduces clustering variables and mixture distribution for model categories [26, 27], extending multimodal generation to include clustering capabilities. Recently, Gao al. introduce MVP [28] which proposes an informational prior based on cyclic permutations, enabling both generative and clustering tasks.
2.2 Information Bottleneck
Information bottleneck plays crucial role in multimodal clustering and representation learning, enabling models to extract robust and interpretable latent representations by capturing shared and complementary information across different modalities. Wang al. [29] propose supervised method that maximizes mutual information between joint representations and labels while filtering out irrelevant data from original views. Federici et al. introduce MIB [30], which applies the information bottleneck to identify shared and view-specific information between views. Hwang el al. [13] develop cross-domain generative model enable image-to-image translation, while Lin el al. propose COMPLETER [31], which maximizes shared mutual information and minimizes conditional entropy recover missing views. In addition, CMIB-Nets [32] balances the consistency and complementarity of multimodal views by extracting shared and view-specific information. Wang er al. [33] focus on self-supervised learning to regularize mutual information and improve clustering performance. Hu al. [34] introduce a propagation information bottleneck to facilitate the transition from representation learning to clustering structure learning. Huang et al. [35] identify key requirements for effective multi-view learning and propose a model that integrates representation learning with clustering, Mao et al. [36] enhance alignment in clustering by distinguishing between consistency and redundancy through mutual information maximization, while Yan el al. [37] introduce differentiable information bottleneck for deterministic multi-view clustering.
3 Methods
Problem Statement. We consider a cross-modal learning scenario where the data consists of two modalities, s and t, with some observations potentially missing one of the modalities. In this context, we represent the data containing only modality as D.,, the data containing only modality & as D, and the data containing both modalities paired as D.,t- Our goal is to learn meaningful latent representations from the combined datasets D DD U.D U'D,, for various downstream tasks such as clustering and classification, while maintaining high-quality cross- generation and coherence.
3

---

3.1 Disentangled Mutual Supervision
Given the paired data setting (s,t), the information flow s ++ + is often employed in cross- modal representation learning, aiming to extract latent representation x that encodes the infor- mation shared between both modalities [10]. However, such a design lacks a separate space for modality-seecific information, which causes the shared latent representation to inevitably con- tain entangled noise through mutual supervision under both information flows. Therefore, we propose to learn modality specific factor w along with the shared latent to decompose the exclusive and common features of each modality. Specifically, we present the main concept by considering the information flow from 8 to t and the derivation for the opposite direction is sim- ilar. According to our assumption, we define the generative process of our model as follows: p0.v. (t,8, 8, as wx) = p (t) Pv: (z () p (w,) Pa (s Z., w,), where w, captures modality-specife content from 8. z, represents the shared information necessary for generating t. The prior of w, is assumed to follow standard normal distribution, i.e., p (w,) N' I). Based on the genera- tive model, the marginal log-likelihood pe,s, (t, 8) requires integration over the latent variables 2, and Zwo which is generally intractable. We then seek variational posterior I0 (2s, ws (8, t) to overcome this intractability and derive the ELBO via Jensen's inequality:
pa. (t, s) = log f po.v= (t, 8, 2s. w₂) dz 2 Eo (1.1), log po.u. (t, 8, ns. w.) 90.0 (Ze, w. 8, t)
In the course of variational inference, we derive a factorized form of the posterior distribution To.s (Za, W.,& & 8) under the assumptions that (i) the latent variables 2s and w, are indepen- dent, and (ii) the observations s and t are conditionally independent given 2s. Thus, the infer- ence process is represented a5: (Z,, Wa, | 8) |
qé. (z. thw (w, 8) qu (t z.). Once q (Ze, W,, t s) is computed, we can apply Bayes' rule to obtain qo.. (Ze. Wa 8, t) according (*,,
to the following decomposition: 9s.. (Zs, W. s, t)
d0s (#_1a)go., (w,1a)g_(t)x.) ge a)
(t)a)
where 4 (t 8) ʃ 9s. (=, 189g 9. (t =,) dzy. And Φ = {Φ., )w denotes the full set of pa- rameters governing the variational distribution of 25 and w., each modeled as a Gaussian distribution.
By substituting inference process into Eq. (1), we obtain:
2 (t 2.)
log Pv. (z+ t) (wvx) po (8 2x, w., To. (Z, qp (w, 8) q: (t 23)
log po.v. (t, 2 EE (gg(z..
1o..p (t) (1)
+log go (t|s) + log pogp(( (t).
(t serves as an importance weight that adjusts for the mismatch be- tween the sampled latent-induced distribution and the actual conditional distribution. This ad- justment improves the estimation accuracy and prevents the loss of pivotal information regard- ing t during sampling. Moreover, the log term measures the agreement between the genera- tive process (through pa= (z, t)p (w (s z,, w,)) and the variational inference (through Io. (z, 8) q (w, 8) g (t] zg)). Maximizing this term encourages consistency between the latent structure inferred from data and the one induced by the generative process. The additional term log qs... (t|s) acts as a regularization term to encourage its alignment with the true distribution of and balance the other components of the ELBO. Since log p(t) is constant with respect to the model parameters, it does not affect the optimization and can be ignored in the objective function.
In Eq. (2), the ratio (x(t(x.)
Structured Representation Learning. The maximization of the ELBO alone presented in Eq. (2) is insufficient to ensure complete disentanglement of the latent variables w, and z,, as any arbitrary mutually exclusive factorization may be equally favored in the absence of mechanisms that explicitly encourage information retention in the shared factor. Considering this, we add a mutual information maximization term / (z,; t; 8) to enforce 2, containing all relevant information across modailties. We further incorporate another mutual information penalty (zg: wx to encourage the decomposition of W. and zg. The two information constraints can be unified in the following form:
max I (z,; t; 8) - I [z.; w.) = I (s; ž., w.) I (s; w.) - I (z,;
The detailed derivations for Eq. (3) are provided in the Appendix A. Since direct optimization of mutual information is generally intractable, approximation methods such as variational inference or Monte Carlo sampling are often used to estimate these terms [30, 33, 13). We subsequently derive computationally feasible approximations for each mutual information component as follows.

---

The first term I (s; 2s. w,) quantifies the amount of information about the input modality that captured by the latent variables 2s and ws. Since its calculation involves q(s w,, o(w..x.la) ()) po(s) where (s) appears both as the empirical data distribution in the expectation ((2,,,,)
and as an integral term in the denominator, making the computation intractable. We instead obtain variational lower bound based on the generative distribution po (s 2s, w,):
I (Ze, wx; 8) = (0, log (s
Zz. w.,
PD (8)
= H (s) + E ws))po log po (s z W.)
+ wp.) [DL (g Ze. w2) lipe (s n., Wz))
2 H (8) + I. 1F (a)pp(a) log Po (s 22. uns)
where the entropy term H (s) is treated as constant. The remaining expectation term corresponds to a reconstruction loss under Gaussian distribution assumption and indicates that maximizing I (zs, (x; s) facilitates w, and z, to jointly contain all relevant information to modality s. The second term / (s; wx) can be omitted as a standalone constraint. A detailed explanation provided in the Appendix A.3. The conditional mutual information term - (z,: t) is minimized to suppress view-specific redundancy. The latent representation 2g is regularized by the counterpart modality t, encouraging to capture only the information accessible from both views. Specifically, by defining 2a and 2, over a shared latent space 2, I (z+; 8 t) can be variationally approximated as:
-1 (zzi s t) 2 (x..) [D&L (go. (z+ = 8) 1l (Zz z [t)]]
(5)
complete derivation of Eq. (5) is provided in Appendix A.2. Consequently, by integrating these vari- ationally tractable surrogate objectives, namely the reconstruction-bast surrogate for / (z., W..; 8) in Eq. (4), the KL divergence bound for (s; wx) and the moos- regularization for (zz; s t)
in Eq. (5), the original ELBO in Eq. (2) becomes the following tractable optimization objective:
9 (t 2.)
log Pv. (z. (t (w.) Pa (s
Za, w..
C ((,.v) = Egg (z,,,.1))
Ao.. (+| Is. (Zs 8) I (w, 8) qup 23)
(6)
+ log N (t|x)= + Eg (o)po(a) log (s w))
- Epp(e.t) DDr.. (1o. (z, = = Igw. (2: = = [))]
where + = (0, and v {1b, 0} correspond to the encoder and decoder parameters, respectively.
Mutual Supervision. Mutual supervision leverages reciprocal guidance between modalities to learn a semantically consistent shared latent space. Unlike explicit integration methods like PoE or MoE, it offers greater flexibility and robustness without requiring strict alignment or direct fusion. Building on the notion that each modality both informs and constrains the other, we formulate the objective in the case where t represents the source data and s the target data, i.e. 2t > 8, as follows:
(v,,))(8 = Ey (t) qo (8 Ze)
Pe. (Zr 8) (w:) p&p (t 2,, w.)
log
20 (s E)
Iv. (Z+ t) 9%h.. (we, t) go (s ze)
+ log ga.v (s|t) + E (z, (()p(()) log Pup 2,, w:)
(p.(). [DEL (gv. (ze = 1t) 11]1 (z.
8))]
This formulation mirrors the standard direction Lio, v. (s. t) and captures the reverse information flow. To instantiate this symmetric structure, we implement the model by swapping the roles of the generative and inference networks, where we exchange the parameter sets & and V. To integrate both directions of information flow, we combine the contributions from both L( (s, (s, and Ci *) (s, t), yielding the objective function:
LBi (s, = (L(,) + C (s.
8
Shared Representations Alignment. We leverage [(z,, 2,) to directly align the modality-invarian latent variables, preventing them from diverging encoding discrepant semantics for the same content. This ensures that mutual supervision is not only reflected the generative reconstruction paths, but also enforced through semantic alignment at the latent representation level. We incorporate
5

---

[[z,, z1) as regularization component in addition to Eq. (8) and derive the following training objective in the paired-data scenario:
LBi (s, =
(C (
+ L (s, ()) + aT (2s; ze).
where a serves regularization coefficient that balances the contribution of the mutual information term 7 (z,; zt). Empirically, we estimate (zz;2z) using contrastive learning, which has been confirmed as an effective way to solve for the mutual information maximization. Specifically, we align paired latent features while distinguishing unpaired ones, computing pairwise cosine similarities within each batch and applying cross-entropy without extra projection layers.
3.2 Partial Observations Scenario
Our model naturally extends to single-view scenarios due to its autoregressive cross-modal generative structure. Given an observed modality, the shared latent representation can be inferred and used to approximate the missing modality via learned generative paths. First considering that the s-mode data is available, we can derive a variational approximation for log P0.v (s) by marginalizing over the unobserved modality t, resulting in the following lower bound:
log Pe. (8) log / pt) Po. (z+ t) (wr)) pe (s Z.: w.) dtddz,dw
(10)
2 Be w. log po (s 2s. w. (()) Put (2.)
qá. (zs 8) 9o... (w
where (2.) = (t( Pe. (z, t) dt. Notably, even in the absence of paired t-observations, the model can still regularize the latent representation z, through a shared prior derived from the distribution of t. Inspired by VampPrior [38], we define batch-dependent prior over B representative
anchors {ut)e sampled from the t-modality, yielding: P., (z,) D Pé. (zo (sslu(), where
dynamic resampling ensures the prior adapts to the evolving latent structure, improving stability and expressiveness. Eq. (10) can then be rewritten
L. (8) = Eo (x.l,,,,,) (e)e) log po (s
ns, w.) p (wa)
FB ii=1 Pvs
(Ze u
qs. (zs 8) Iow.. (w,
= (0,1(z.. log Po (8 nzy w.) - DEL
1o. (z.
8)
1
Σ
i=1 Po.
(z,
ut)
(11)
B
-KK (go.. (w,
|pp (w.)).
With this design, we can still leverage the information from the t-modality effectively constrain and guide the model. In a comparable manner, when the s-modality missing and t-modality is available, the objective is defined as:
C (t) = Eng(znang log Po (t zt. wi) DEL (gu. (ze ()) E Po. (ze u3)
(12)
- DKL (go. (we(t) ppp (urr))
The final objective function integrates the target loss functions for both paired and unpaired cases, encompassing three scenarios: paired samples, samples with only s-modality, and samples with only t-modality. It is expressed follows and the training procedure is detailed in Appendix A.7.
C (D) = Σ
LBi (8, + Σ L. (8) + Σ La (t). SED,
(13)
B.
IED.
4 Experiments
4.1 Experiments Setup
Datasets.
Two widely used datasets, i.e. MNIST-SVHN and CUBICC, are adopted in our experi-
ments. The MNIST-SVHN dataset consists of the MNIST and Street View House Numbers (SVHN) datasets, where the samples share digit labels (10 classes) but have different digit styles [8]. The

---

SVINSTMNISTT
MNIST=SVHN
DCMEM
MVP
MVP
R?7
CMVAE
CMVAE
Calemeece
066
MMVAE-
MMVAEI
0.5
MUME
MEME
MePoE
McPoE
(14
MMVAA
9.4
MMVAE
0.3
MVAE
MVAE
02-
02-
81
120
40
300
Figure 2: The cross-modal generation performance of DCMEM and existing multimodal VAEs on the MNIST-SVHN dataset. Each model was run independently three times. The best performance is located in the top-right corner of each figure.
CUB Image-Captionns for Clustering (CUBICC) dataset, a variant of the CUB image caption dataset constructed by Palumbo et al. [12), consists of two modalities: bird images and their corresponding descriptive captions. The dataset is divided into 8 categories based on bird species. In addition, the Human Breast Cancer spatial transcriptomics dataset provides high-resolution spatial measurements across multiple modalities, including gene expression, spatial coordinates, and tissue morphological features [39]- The dataset comprises 20 distinct spatial label categories corresponding to different tissue or structural regions. These datasets selected representative benchmarks to evaluate the model's ' ability to disentangle and align multimodal latent spaces rather than to cover all possible modality combinations.
Baselines. To comprehensively evaluate the performance of the proposed method, we compare with eight existing multimodal VAEs, including MVAE [7], MMVAE [8), MoPoE [9], MEME [10], MMVAE+ [11]. CMVAE [12], MMVM [24] and MVP [28]. For all comparison methods, we adopt the model architectures proposed in their respective papers and use their default optimal parameters. To assess the model's capability in handling partially observed datasets, construct a set of incomplete bimodal datasets by randomly removing one modality at missing rates of 7 {0.25, 0.5, 0.75) and then train MVAE, MoPoE, MEME, MVP as well as method on these modified datasets. Each method is run three times to ensure the reliability of the results. We provide the implementation details of our method in Appendix B.2.
Evaluation. For the generative task, we primarily evaluate generation coherence and generation quality as in previous work [11, 12, 24]. To evaluate generation coherence, we use pretrained classifier to classify the generated samples and evaluate the generation coherence in terms of the classification accuracy. The generation quality is assessed using the FID metric [40]. Moreover, we investigate the effectiveness of the learned latent representations based on classification and clustering analyses. For classification, we follow previous work by training linear classifier on the latent space and report the accuracy. For clustering, we apply K K-means and evaluate the results using ACC, NMI and ARI. For models with both shared and modalittyspeeiine latent variables, all evaluations are conducted on the shared latent space, which captures modality-invarian semantics.
4.2 Comparison of Generation Performance
In this section, we compare the performance of the proposed DCMEM model with that of the aforementioned competitive multimodal VAEs in terms of cross-modal generation. We first conduct the cross-modal generation experiment on the fully paired MNIST-SVHN dataset. As shown in Figure 2, our model demonstrates superior performance, achieving both high generative coherence and quality. In Figures and 7, we present the cross-modal generation results for each model. It is evident from these results that our model effectively captures the underlying digit labels and generates accurate cross-modal samples. In contrast, other models tend to misidentify similar digits, struggling with certain digit pairs. This discrepancy highlights the advantages of our model in handling the complexity of cross modal generation tasks, maintaining both high fidelity and label consistency.
7

---

10
0
2
2
2
2
2 S 4
8
3
S
3
4
4
4
4
&
7
7
DCMEM
MVP
MMVM
CMVAE
MMVAEI
MЕME
MoPoE
MMVAE
MVAE
Figure 3: Qualitative results of cross-modal generation on the MNIST-SVHN dataset for each model.
The left side shows the input samples, while the right side displays the cross- cross-modal generated samples.
LIG
IG
O
4
5
$
9
3
8
5
3
5
5
6
C
4
1
5
T
3
0
3
L
3
4
5
8
9
C
/
6
S
5
4
4
3
6
CMVAR
Figure 4: Five SVHN-to-MNIST samples generated by varying only the modality-specifie latent variables on the MNIST-SVHN dataset.
the bird has black belly that turns into orange breust with black wings.
the small bird is belly brown with a bcak small orange head and black cycs.
bright bird has mostly scarlet plumage with black eyes, wings, tail feathers, and feet.
red bird, short beak with black eyes.
(a) Caption--lmage
(b) Image->Caption
Figure 5: Qualitative results of cross-modal generation by DCMEM on the CUBICC dataset. The samples on the left side of the arrows represent the input samples, while the samples the right side show the generated cross-modal samples.
Among the models, MMVAE+. CMVAE, MVP and our DCMEM all incorporate both shared and modality-specific latent variables. To further analyze the generation coherence and quality, we fix the shared latent variables and randomly sample five different modality-speeine latent variables for cross-modal generation. This allows us to evaluate how well each model can generate diverse samples while maintaining consistency across modalities. The experimental results are illustrated in Figures& 4 and 8. Our model demonstrates the ability to generate images that maintain the same digit class, while introducing variations in digit shape and color as the podality-speciinn variables are modified. In contrast, both MMVAE+ and CMVAE exhibit entanglement between class information and modality-speeific variables, leading to inconsistencies MVP fails to introduce sufficient variation through its modality-speciff latent variables, resulting in limited sample diversity. These comparisons further highlight the strength of our approach in producing diverse and high-quality cross-modal generations without sacrificing semantic consistency or class identity.
the generated samples. Meanwhile,
To further demonstrate the performance of our model in more complex scenarios, we conduct a cross- modal generation experiment on the CUBICC dataset and Figure 5 presents the generation results. As expected, our model consistently achieves high-qualit, cross-mode modal generation performance across different modalities. The generated samples align well with the corresponding modalities, maintaining high level of coherence between the image and text representations. This showcases model's ability to effectively handle multimodal data more intricate and diverse settings, demonstrating its robustness in real-world applications. Morcover, the model achieves robust cross-

---

Table 1: Quantitative comparison of clustering performance for each model's latent representations on the CUBICC dataset. The best and second-best results are highlighted bold and underlined, respectively.
Image
Representation
Caption Representation
Joint Representation
Methods
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
MVAE
26.2
12.4
7.5
18.1
2.4
0.9
38.7
26.8
18.0
MMVAE
23.1
12.1
6.1
14.5
1.3
0.1
15.8
1.5
0.2
MoPoE
33.4
17.6
11.5
43.5
27.1
19.9
40.8
30.4
20.2
MEME
44.8
43.4
28.4
36.3
29.5
18.6
19.8
4.8
2.1
MMVAE+
27.7
11.9
7.1
48.7
36.4
26.8
64.4
52.6
44.1
CMVAE
67.7
58.3
47.4
65.1
53.3
42.7
73.7
67.4
57.2
MMVM
58.9
56.9
44.5
23.9
9.4
5.4
66.8
67.0
55.5
MVP
64.1
53.8
41.8
48.5
34.4
26.1
61.1
55.6
44.0
DCMEM
86.9
77.4
72.4
69.7
52.2
44.2
86.3
76.8
71.5
SVIN Latent Representation ACC
MNIST Latent Regresentation ACC
1.00
08
DOMENL
SAY SVIN
0.7
565 SECST
0.90
Actuaay
0.5
0.85
SEVUE SHIN
0.4
SEVAR MNNT
0.80
SECTH
OMVAE
0.2
0.75
0.1
fraction
Observation
Figure 6: Classification accuracy under different missing rates the MNIST-SVHN dataset. Shaded areas represent the standard deviation across multiple runs. The subscripts in method names indicate the observed modality. For example, DCMEM SVHN (Observation fraction 0.25) denotes that the training data consists of 25% paired samples and 75% unimodal SVHN samples.
modal generation under varying missing rates, effectively balancing generative quality and class-level semantic consistency (see Appendix C.1).
4.3 Latent Representation Analysis
In this section, we focus on evaluating the effectiveness of the latent representations Iearned by the model. All models are trained on the fully paired MNIST-SVHN and CUBICC datasets and the latent representations for the test are extracted using the trained models. We also evaluate the models under partially paired (incomplete) scenarios, as detailed in Appendix C.2. We first perform clustering on the latent representations of each modality to assess their individual clustering performance, followed by clustering on the joint representations. For models such as MVAE and MoPoE, which produce joint latent representations, we directly use the joint representations for K-means clustering. CMVAE directly yields the clustering labels as it learns clustering variable. For other models that provide neither the joint representations nor clustering factors, we concatenate the latent representations from each modality and then apply K-means clustering to the resulting joint representation. The clustering results are summarized in Tables and
Quantitative metrics clearly demonstrate that our model outperforms others in terms of clustering performance. The T-SNE [41] plots of our model's latent representations are presented in Figure From these visualizations, it is evident that the latent representations for both modalities naturally form distinct clusters. Moreover, the latent representations of the same category across modalities align into a single cohesive cluster, indicating that our model effectively captures the shared information between modalities. In contrast, the latent representations of other models either fail to form well- defined clusters exhibit separation between modalities, as shown in Figures 10 and 11. These results further emphasize the superiority of our model learning coherent and meaningful latent representations, making it highly effective for clustering tasks involving multimodal data.

---

For the classification task, we present the results based solely on the classification accuracy metric, evaluating model performance under various missing data conditions. The classification results are shown in Figures 6 and 12. Our model consistently outperforms the other alternatives and demonstrates robust classification performance across varying missing rates. In contrast, other baseline models exhibit significant performance degradation at certain missing rates, indicating the limited capability in exploiting incomplete data. For example, MMVAE+ underperforms on the MNIST-SVHN dataset. This suboptimal performance is likely due to its inability to effectively capture glass-discimminative features within the shared latent space. Moreover, although models such as MEME and MVP are designed to handle missing modalities, their performance degrades substantially as the missing rate increases. This suggests a limited robustness to incomplete data, highlighting the advantage of our method in maintaining high classification accuracy under varying degrees of missing information. Analyses in Appendix C.3 and Appendix C.4 further show that our model consistently preserves semantic and class-level alignment across modalities. Appendices C.5-C.7 provide detailed descriptions of the ablation study, parameter analysis, and computational resources.
4.4 Application to Human Breast Cancer Dataset
To evaluate the applicability of our model in other fields, we conduct an additional experiment on spatial transcriptomics dataset of human breast cancer [39]. The dataset presents significant challenges for accurate spatial domain identification due to technical noise and inherent biological variability. We compare our model with eight multimodal VAE baselines and five spatial domain clustering methods, including Scanpy [42], STAGATE [43], GraphST [44), SiGra [45], and xSiGra [46]. For the VAF-based baselines lacking explicit spatial relationships modeling, we adopt the CoordConv strategy [47], encoding spatial (x. y) coordinates as two additional image channels. In contrast, spatial clustering methods are inherently designed to capture spatial dependencies through graph-based or attention mechanisms, As illustrated in Figure 13, DCMEM achieves the highest clustering performance, with an ARI of 55.1 and NMI of 69.7, substantially outperforming both VAE-based and spatially-awared approaches. Among multimodal VAEs, MMVAE achieves the best results, yet it still falls short compared Scanpy, GraphST and our model. Methods such as MVAE and MMVM struggle to resolve ane-grained cell population structures. Although spatial methods like STAGATE and GraphST generate more coherent partitions, DCMEM produces sharper cluster boundaries and exhibits better alignment with ground truth labels. These results demonstrate the effectiveness of our method in integrating heterogeneous modalities and capturing informative spatial and molecular patterns, underscoring its strong potential for real-world clustering tasks.
5 Broader Impact & Limitations
This work focuses on learning disentangled cross-modal representations and enhancing generation quality and coherence by leveraging mutual supervision along with the information bottleneck princi- ple. Our approach is applicable to variety of tasks in scientific and engineering domains and holds the potential for positive societal impact. In the biomedical field, for example, our model can detect fine fine-grained cell populations with precise boundaries and identify potential biomarkers associated with tumor heterogeneity for subsequent clinical validation, by analyzing spatial transcriptomics data at the single-cell level. At the bulk cancer omics level, it can also assist in identifying novel disease subtypes and predicting the survival outcomes to inform better treatment strategy design. Although our model yields better representations and more coherent outputs, it is specifically designed for bimodal data scenarios and may require non non-trivial effort to achieve competitve performance on general multimodal datasets.
6 Conclusion
In this work, we propose DCMEM, variational framework that integrates disentanglement and mutual supervision to learn structured cross-modal representations. It separates shared and modality- specific information via dedicated latent spaces and promotes semantic alignment by maximizing mutual information between shared latent variables across modalities, achieving strong performance across diverse tasks and settings.
10

---

Acknowledgements
This work was supported by the National Natural Science Foundation of China (62372279, 62322215, 62532017), and the Natural Science Foundation of Shandong Province (2R2025QB62. ZR2023MF119) This study was also supported in part by the High-Performance Computing Center of Central South University.
References
Vinitra Swamy, Malika Satayeva, Jibril Frej. Thierry Bossy, Thijs Vogels, Martin Jaggi, Tanja Kaser, and Mary-Anne Hartley. Multimoin--ullinddaa multi-task, interpretable modular networks. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 28115 28138. Curran Associates, Inc., 2023.
[2] Divyam Madaan, Taro Makino, Sumit Chopra, and Kyunghyun Cho. Jointly modeling inter- &amp; intra-modality dependencies for multi-modal learning. In A. Globerson, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 116084-116105 Curran Associates, Inc., 2024.
[3] Zichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang, Mu Li, Anshumali Shrivastava, and Andrew Gor- don Wilson. Learning multimodal data augmentation in feature space. In The Eleventh International Conference on Learning Representations, 2023.
[4] Zhou Lu. A theory of multimodal learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances Neural Information Processing Systems, volume 36, pages 57244-57255. Curran Associates, Inc., 2023.
[5] Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Learning disentangled representation by exploiting pretrained generative models: contrastive learning view. International Conference on Learning Representations, 2022.
[6] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learning. In S. Bengio, H. Wallach, H. Larochelle, Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[8] Yuge Shi, N. Siddharth. Brooks Paige, and Philip H. S. Torr. Variational mixture-of -e-eeetsss autoencoders for multi-modal deep generative models. 2019.
Thomas Sutter, Imant Daunhawer, and Julia E Vogt. Generalized multimodal elbo. In International Conference on Learning Representations, 2021.
[10] Tom Joy, Yuge Shi, Philip Torr, Tom Rainforth, Sebastian M Schmon, and Siddharth N. Learning multimodal VAEs through mutual supervision. In International Conference on Learning Representations, 2022.
[11] Emanuele Palumbo, Imant Daunhawer, and Julia E Vogt. MMVAF+: Enhancing the generative quality of multimodal VAFs without compromises. In The Eleventh International Conference on Learning Representations, 2023.
[12] Emanuele Palumbo, Laura Manduchi, Sonia Laguna, Daphné Chopard, and Julia E Vogt. Deep generative clustering with multimodal diffusion variational autoencoders. In The Twelfth International Conference on Learning Representations, 2024.
[13] HycongJoo Hwang, Geon Eeoon- Kim, Scunghoon Hong, and Sce-Eung Kim. Variational interaction information maximization for cross-domain disentanglement. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 22479-22491. Curran Associates, Inc., 2020.
[14] Jie Xu, Yazhou Ren, Huayi Tang. Xiaorong Pu, Xiaofeng Zhu, Ming Zeng, and Lifang He. Multi-vae: Learning disentangled view-common and pieulicc visual representations for multi-view clustering. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV,, pages 9214-9223, 2021.
[15] Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint www.vv:1611.0eeee 2016.
[16] Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. In International Conference Learning Representations, 2018.
[17] Xun Huang, Ming Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised mmage-to-image translation. In Proceedings of the European conference on computer vision (ECCV), pages 172-189, 2018.
11

---

[18] Mike Wu and Noah Goodman. Multimodal generative models for compositional representation learning. arXiv preprint arXiv:1912.0507,, 2019.
[19] Richard Kurle, Stephan Günnemann, and Patrick Van der Smagt. Multi-source neural variational inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4114-4121 4121, 2019.
[20] Imant Daunhawer, Thomas M Sutter, Kieran Chin-Cheong, Emanuele Palumbo, and Julia E Vogt. On the limitations of multimodal vaes. In The Temth International Conference on Learning Representations (ICLR
2022). 2022.
[21] Jannik Wolff, Tassilo Klein, Moin Nabi, Rahul G Krishman, and Shinichi Nakajima. Mixture-of-cxpers vacs can disregard variation in surjective multimodal data, arXiv preprint arXiv:2204.05229 2022.
[22] HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, and Kee-Eung Kim. Multi-view representation learning via total correlation objective. Advances in Neural Information Processing Systems, 34:12194- 12207, 2021.
[23] Thomas Sutter, Imant Daunhawer, and Julia Vogt. Multimodal generative learning utilizing jensen shasnon- divergence. Advances in neural information processing systems, 33:6100-611 6110, 2020.
[24] Thomas Sutter, Yang Meng, Andrea Agostini, Daphné Chopard, Norhert Fortin, Julia Vogt, Babak Shahbaba, and Stephan Mandt. Unity by diversity: Improved representation learning for multimodal vaes. Advances in Neural Information Processing Systems. 11774222--4297, 2024.
[25] Mihee Lee and Vladimir Pavlovic. Private-shared disentangled multimodal vac for learning of latent representations. In Proceedings of the ieeelevf conference on computer vision and pattern recognition, pages 1692 1700, 2021.
[26] Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulku- maran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.0266 2016.
[27] Zhuxi Jiang. Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised generative approach to clustering. In 26th International Joint Conference on Artificial Intelligence, IJCAI 2017, pages 1965-1972. International Joint Conferences on Artificial Intelligence, 2017.
[28] Xin Gao and Jian Pu. Deep incomplete multi-view learning via cyclic permutation of vaes. In The Thirteenth International Conference on Learning Representations, 2025.
[29] Qi Wang, Claire Boudreau, Qixing Luo, Pang-Ning Tan, and Jiayu Zhou. Deep multi-view information bottleneck. In Proceedings of the 2019 SIAM International Conference on Data Mining, pages 37-45. SIAM, 2019.
[30] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust represen- tations via multi-view information bottleneck. In International Conference on Learning Representations. 2020.
[31] Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng. Completer: Incomplete multi-view clustering via contrastive prediction. In Proceedings of the IEEF/CVF conference on computer vision and puttern recognition, pages 11 11174-11 1183, 2021.
[32] Zhibin Wan, Changqing Zhang, Pengfei Zhu and Qinghua Hu. Multi-vien iosommttion-botlennce representation learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10085- 2021.
[33] Shiye Wang, Changsheng Li, Yanming Li, Ye Yuan, and Guoren Wang. Self-supervised information bottleneck for deep multi-view subspace clustering, IEEE Transactions on Image Processing, 32:1 1567, 2023.
[34] Shizhe Hu, Zenglin Shi, Xiaoqiang Yan. Zhengzheng Lou, and Yangdong Ye. Multiview clustering with propagating information bottlencck. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[35] Weitian Huang, Sirui Yang, and Hongmin Cai. Generalized information-theoreii multi-view clustering. Advances in neural information processing systems, 36:58752-58764, 2023.
[36] Yiqiao Mao, Xiaoqiang Yan, Jiaming Liu, and Yangdong Ye. Congme: Consistency- guided multimodal clustering via mutual information maximin. IEEE Transactions on Multimedia, 26:5131-5144 2023.
[37] Xiaoqiang Yan, Zhixiang Jin, Fengshou Han, and Yangdong Ye. Differentiable information bottleneck for deterministic multi-view clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27435-27444, 2024.
[38] Jakub Tomezak and Max Welling. Vae with a vampprior. In Amos Storkey and Fernando Perez- editors, Proceedings of the Twenty-Firs International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 1214-1223, PMLR, 09 11 Apr 2018
12

---

[39] Sunny Z Wu, Ghamdan Al-Eryani, Daniel Lee Roden. Simon Junankar. Kate Harvey, Alma Andersson, Aatish Thennavan, Chenfei Wang. James R Torpy, Nenad Bartonicek, al. single-cell and spatially resolved atlas human breast cancers. Nature genetics, 53(9)11334-1347 2021.
[40] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[41] Laurens Van der Maaten and Geoffrey Hinton, Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[42] F Alexander Wolf, Philipp Angerer, and Fabian Theiss Scanpy; large-scale single-cell gene expression data analysis. Genome biology, 19:1-5, 2018.
[43] Kangning Dong and Shihua Zhang. Deciphering spatial domains from spatially resolved transcriptomics with an adaptive graph attention auto-encoder. Nature communications, 13(1):1739, 2022.
[44] Yahui Long, Kok Siong Ang. Mengwei Li, Kian Long Kelvin Chong, Raman Sethi, Chengwei Zhong, Hang Xu, Zhiwei Ong, Karishma Sachaphibulki,, Ao Chen, et al. Spatially informed clustering, integration. and deconvolution of spatial transcriptomies with graphst. Nature Communicutions, 14(1):1 2023.
[45] Ziyang Tang. Zuotian Li, Tieying Hou, Tonglin Zhang, Baijian Yang. Jing Su, and Qianqian Song- Sigra: single-cell spatial elucidation through image-augmented graph transformer. Nature Communications, 14(1):5618, 2023.
[46] Aishwarya Budhkar, Ziyang Tang, Xiang Liu. Xuhong Zhang. Jing Su, and Qianqian Song. xsigra: explainable model for single-cell spatial data clucidation. Briefings in Bioinformatics, 25(5) 38b 2024.
[47] Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosin- ski. An intriguing failing of convolutional neural networks and the coordconv solution. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, page 9628-9639, Red Hook, NY, USA, 2018. Curran Associates Inc.
[48] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03/7
2018
[49] Tom Joy, Sebastian M Schmon, Philip HS Torr, N Siddharth, and Tom Rainforth. Capturing label characteristics in vaes. In International Conference Learning Representations, 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the paper, we present novel multimodal approach that outperforms existing multimodal methods on multiple datasets. The contributions are clearly stated in the abstract and introduction, and align with the theoretical and empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the paper in the Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: All theoretical results include clear assumptions and complete proofs, provided in the main text appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe all the experiments in full detail in the appendix (see Appendix B) such that all the results on all datasets can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the complete source code along with instructions, including README files and environment settings in the supplementary material. All datasets used in our experiments are publicly available, and we include clear instructions for accessing and preprocessing the data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ public/guicn /ttp:/gudideeeeeeeuiiiieeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ btciss.ccppublee hwtpppugwuuils.////pppppppppeeel httgggudds///// httpe//guiddeeeoeeeuubbmiiii Pttps//SSooSaaassssssooooo.oo.. ssioioolcyy for details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the necessary information either in the main text and appendix (see Appendix B).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the mean and standard deviation of the results obtained from experiments conducted using three different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
We have provided details on the compute resources used, including type of workers, memory, and execution time, well the total compute required for the experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eurrrp..eeee/eee httpe:///urrpps.ccppbbltlt
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and confirm that our research complies with all of its guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential broader impact the proposed work alongside its limitations in Section 5.
17

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymmly of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does involve the release of models or datasets with high risks of misuse, such as pretrained language models, image generators, or scraped datasets. Therefore, there are no specific safeguards discussed in the paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and code used in work are publicly available and properly cited in the paper. We have explicitly acknowledged the original sources and included license information where applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
18

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included the code as supplementary material with our submission to ensure reproducibility during the review process. Upon acceptance, we will make the code publicly available with comprehensive documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https httt:p///eerrrpe.c///oooferrre// cc/Coofferecccs/2255/M for what should or should not be described.
20

---

A Supplementary Technical Details
A.1 Mutual Information Decomposition for Structured Representation Learning
Define X, Y and be random variables. The the chain rule for mutual information is:
I(;;;: 2) I(X:Y) I(X:; 2| = [Xx; 2) - [X;; Z|Y) I(Y; 2) - (14) As defined in the main text, the modality- invariiat variable 2g and the modality-speciin variable w, have already been introduced. [(z,; w.) and / (s; t; xx) can be computed as follows:
I [zz: wx, = / (z.; 8) - / (z.; ; 8 wx.) + I (w,; 2x
(15)
I (8; t; =2) = (z.:
(z.; t)
Under the assumption that the modality-invariant w, are conditionally independent given the input 8, we have: q (w,, 8) = q (ww, 8, zz). Thus, the conditional mutual information / (wyi &s 8) simplifies to / (w,i 8) H (w, 88
variable 2, and the modality-speciffi variable
H (w, 8, 2.) = O. As a result, (wg: zx) can be further decomposed as follows: [(z W.) I 82);8) I (2.; w.) = I (z s) + I (s; w.) I (s; z.. w.). In conclusion, the mutual infor- mation decomposition for disentangled representation learning is given as follows:
(zgi t; I (zzi w.)
= [ - I (zz; 8 t) + I 2s; w.) I ws; T(zsia)
(16)
I (8; Ze, w.) I (s; w.) I (zyi
A.2 Evidence Lower Bound on the cross-modal regularization
Assuming that both 2, and n, lie in the same latent space Z:
log 4d. (z, = 8)
-I (Zzz t) Epple.t) Byo (#xla)
Pv. (2x 2 (t)
log q (z,
8) qo (zt t)
EEp (2)1x)
Tv. (Zt = 2 t) Pv. (Z. = z t)
(17)
-Eplltt) [DKL (16. (Z, = 8) 11a0. (ze
t))]
+ Ep)( [DKL (Pys (Zs t) 1190. (Zt
t))]
(.)))) [DKL (40, (Z. x 8) 1190. (zt t))]
Similarly, -T(2,; t 8) can be computed as:
- -1 & 8) 2 Epp(..t) [Drr. (1v. (z, = = t) llgo. (z. = 2 |8)]]-
(18)
A.3 Derivation of the Mutual Information Term [(s; w.)
The second term -1 (s; also poses computational challenges due to its depen- dence on the marginal distribution (D()), where q(w.) = -s(ww,
(s)pss)ds.
In a similar vein, this term is approximated using its variational lower bound - Epp(s) [DKL (go... (w, s))p((w)))] This KL divergence term naturally appears in the ELBO
objective (Eq. (2)) due to Egg( (s) [Dxz (go... s) ))pppw (w.))
(p(t) log p(w,,
Eqg. (w, 8) log go (w.]ss]
p(wn.)
(t(s)
gow [=),l=)
Hence, it is implicitly optimized through the ELBO and can
be omitted standalone constraint.
A.4 Mutual Information Approximation
The mutual information / (=,,,z) between the shared latent variables 2, and 2, is approximated using a contrastive loss that encourages alignment between representations of paired inputs while distinguishing those from non-paired ones [48]. Given batch of B paired latent features (h,, h2) sampled from 2, and 21. we concatenate them into set of 2B vectors. Pairwise cosine similarities are computed and scaled by a fixed temperature - = 0.5 to construct the contrastive logits. Positive pairs are defined between the i-th feature in h, and the i-th feature in h,, as they correspond to the paired input instances. All other pairs in the batch are treated as negatives. We apply a cross-entropy loss to encourage the model to assign higher similarity to positives than to negatives.
21

---

A.5 Cross-Modal Reconstruction Mechanism
In our framework, 9p x,) serves a key role in the mutual supervision mechanism. Specifically, 2, is the shared latent representation inferred from modality and 9p (t z,) models the reconstruction of modality t based solely on this shared information. This setup intentionally designed to exclude the private latent variable since the goal is to assess what information is common and transferable across modalities. In practice, we implement Up (t 2s) by setting ur and passing the concatenation (z,, 0J into the decoder of modality t. This design offers two key advantages: (1) avoids introducing an additional decoder by reusing Pe (t 2x, w) with w set to zero, keeping the architecture compact; and (2) it promotes effective disentanglement, reconstructing t from alone forces 2g to capture modality-invariant, shared information.
A.6 Importance Sampling Stability
The importance sampling weight (t(t),,) in Equation plays a critical role in our training objective, but its direct computation can introduce numerical instability due to high variance in gradient estimates. This arises because both the numerator and denominator are parameterized distributions that are learned during training, and their stochastic nature may result in noisy, unreliable updates when used in Monte Carlo estimation of the ELBO, To address this issue, we adopt a stop-grade gradient strategy stabilize training without compromising the objective, following rationale similar to that in prior work [49, 10]. Concretely, we prevent gradients from flowing through the (=(t)z)) treating
(el.)(((a))
T0s.p (t(a)
it fixed scalar during backpropagation: stop_grat gro_gggdient [_((t)x,) ) This modification ensures (
that the parameters are optimized based on more stable signals, as it avoids amplifying gradient noise through the ratio. Importantly, this treatment does not alter the forward computation of the objective but improves the robustness and reliability of the training dynamics. In summary, this design provides a practical and effective solution aariance-induced instability in training, while still aligning with the theoretical intent of the original ELBO formulation.
A.7 Training procedure for DCMEM
Algorithm 1: Optimization Procedure of DCMEM
Input: Multimodal dataset: D D, U U'D., Training epochs number: M; Hyperparameters: o; Model parameters (
Output: Latent representation z, and 2, Randomly initialize model parameters { }) for epoch < M do
for each sample in paired data D., do
Compute [( v) (s, t) and ((v,o) (os (s,t) t) via Eqs. (6) and (7); Estimate enhanced mutual-supervised information regularization by Eq. (8);
Compute the bidirectional lower bound (B,(s, , t) by Eq. (9);
7
for each sample in modality-speciff data DØ do L Compute C, (s) via Eq. (11);
for each sample in modality-spccin data DD do Compute L. (t) via Eq. (12);
10
11
Update parameters (Φ, D by maximizing the overall objective in Eq. (13); Compute latent representations z, fo. (s) and 2, fv. (t) using the optimized parameters; return Latent representationas ze and zt
13
B Dataset and Implementation Details
B.1 Dataset Licences
MNIST MNIST_SVHN: originally published in [8]. downloaded the data from http://yann. lecun. wwwp://wwwccoc/e/////iiistts http://ufidl.stanfort htw.aa/ddtttaaoooo.cduoouseeenne cemruseeeussssseerrrrrs and the code from https://gihubb.co//fffffmmmmm licensed under GPL 3.0.
22

---

0
3
0
7
00
6
8
2
2
3
3
4
+/
1
9
7
9
8
DCMEM
MVP
MMVM
CMVAE
MMVAEF
MЕME
MoPoE
MMVAE
MVAE
Figure 7: Supplementary qualitative results of cross- generation on the MNIST-SVHN dataset for each model.
Z
2
0
2
C
2
do
0
2
8
2
0
5
9
DCMEM
MVP
CMVAE
MMVAE+
Figure 8: Five MNIST-to-SVHIN samples are generated by varying only the modality-specific latent variables on the MNIST-SVHN dataset.
CUBICC:
originally
published in [12]. downloaded the data from published under the
MIT license.
Cancer: originally published in [39]. downloaded the data from https://www. https://www.1xxeeomiiii www.l/xeeessss..oomm (oxgenomiss.commmattsettt
Human Breast
1-0-0, published under the CC BY 4.0 license.
B.2
Implementation Details
To ensure a fair comparison, all models GPU, 64 GB of RAM running Ubuntu 18.04. For our model, we use a ResNet encoder and decoder for image data, and convolutional encoders and decoders for text data. The parameter is set to 1. For the MNIST-SVHN dataset, the dimensions of the shared and specific latent spaces are set to 32. We use the Adam optimizer with learning rate of 5e-4, batch size of 64 and train the model for 100 epochs. For the CUBICC dataset, the dimensions of the shared and specific latent spaces are set to 48 and 16, respectively. The Adam optimizer is used with learning rate of le-4, batch size of 16 and training is conducted for 200 epochs. For the spatial transcriptomics dataset, we preprocess the gene expression data by selecting the top 3000 highly variable genes, followed standard normalization and log-transformation Both the encoder and decoder for this modality are implemented as fully connected neural networks. For the tissue morphology modality, input images resized to 128x 128 pixels. To incorporate spatial context, we adopt the CoordConv [47] technique by appending the 2D spatial coordinates (x, y) as additional input channels, resulting in 5-channel input. This modality is processed using convolutional neural networks for both encoding and decoding. Both the shared and specific latent dimensions are set 32. Optimization is performed using Adam with learning rate of 5e-4, a batch size of 64 and 100 training epochs.
local server with NVIDIA GeForce RTX 2080
C Additional Experimental Results
C.1 Additional Results on Cross-Modal Generation
To further evaluate the generative capability of our model, we conduct additional experiments under varying missing rates, focusing both generative coherence and quality. The quantitative results
23

---

1087
40
beg.
bugu
Oekeut
Optn3
-20
-40
-40
SIBL 9
20
(a) MNIST-SVHN
(b) CUBICC
Figure 9: T-SNE plot of the latent representations obtained by DCMEM on the MNIST-SVHN and CUBICC datasets. Here, MNIST_O 0 represents the data from the MNIST modality with the digit label
O, and similarly for other labels.
MAAT
MMNAE
McNEE
MEME
MMVVE+
Figure 10: T-SNE plot of the latent representations obtained by baseline models on the MNIST-SVHN dataset.
MVAL
MMVAE
Matol:
MEME
Figure 11: T-SNE plot of the latent representations obtained by baseline models on the CUBICC dataset.
on the MNIST-SVHN dataset are shown in Figures 14 and 15, while the qualitative results are presented in Figures 16-22. shown the figures, our model consistently achieves robust cross- modal generation under different levels of missing data. Although slight degradation in generative consistency observed as the missing rate increases, the model still maintains strong performance and demonstrates competitive stability compared other baselines.
24

---

Table 2: Quantitative comparison of clustering performance for each model's 's latent representations the MNIST-SVHN dataset. The best and second- -est results are highlighted in bold and underlined,
respectively.
SVHN
Representation
MNIST
Representation
Joint Representation
Methods
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
MVAH
27.9
16.0
13.1
79.2
65.5
62.6
42.7
35.3
24.5
MMVAE
22.0
10.4
10.1
21.8
10.3
10.1
22.6
10.7
10.1
MoPoE
37.9
27.2
18.5
50.5
45.6
33.0
64.1
60.5
50.7
MEME
21.9
10.3
10.0
36.5
32.1
20.4
22.4
10.6
10.1
MMVAE+
23.9
11.4
11.1
21.3
10.4
10.0
22.9
11.9
10.8
CMVAE
42.2
36.3
25.4
28.1
15.9
14.5
32.3
19.5
13.4
MMVM
42.2
27.1
20.7
88.1
82.1
80.4
77.5
72.2
67.5
MVP
53.6
38.7
30.1
81.4
79.6
73.6
84.8
76.4
70.6
DCMEM
91.5
80.6
82.0
99.1
97.3
98.0
99.5
98.4
98.9
Image Lancet Begresaristion ACC
Caption Laters Repremntaiion ACC
25
3.8
23
MeAE_
41
MNNM
CAYAAE
02
MANAD
SAEVAE
41-
Obunralie 0.55
0.35
18
Obsoveten Bociien
Classification accuracy under different missing rates on the CUBICC dataset. Shaded represent the standard deviation across multiple runs. The subscripts in method names indicate the observed modality. For example, DCMEM Caption (Observation fraction = 0.25) denotes that the training data consists of 25% paired samples and 75% unimodal Caption samples.
Figure 12:
Similarly, cross-modal generation results on the CUBICC dataset are shown in Figures 23 and 24. Our model achieves the highest generative coherence across all missing conditions, indicating its effectiveness preserving class-level semantics despite incomplete input. However, we observe moderate decline in generation quality. This trade-off is primarily due to the inherent tension between the generation and clustering objectives: while generation benefits from latent representations that retain fine-grained podalltt-speciffc details, clustering prefers representations that focus global class-level features. As result, our model strategically balances these two objectives rather than optimizing solely for one, which inevitably limits performance in either direction when pursued independently.
C.2 Additional Results on Clustering
We evaluate the clustering performance of MVAE, MoPoE, MEME, MVP and DCMEM under varying missing scenarios on both the MNIST-SVHN and CUBICC datasets. As shown in Tables 4, 5 and 6, we consider two settings for each dataset: one where the first modality partially missing (e.g., MNIST or Image), and one where the second modality is partially missing (e.g.. SVHN or Caption). For each setting, clustering performed based on the latent representations learned from individual modalities as well as their joint embedding. DCMEM consistently achieves the best performance across all settings and representation types, demonstrating strong robustness to incomplete data. Even at low paired data fractions (e.g.. 25%), DCMEM maintains high clustering accuracy, while baseline models such as MVAE, MoPoE, MEME and MVP exhibit significant performance degradation. This is especially evident in the Caption modality of the CUBICC dataset, where several baselines struggle to learn meaningful representations under high missing-viee -iie rates. Overall, these results highlight the effectiveness of DCMEM in learning coherent and discriminative
25

---

ARE: 45.9 NMI: 64.0
ARI: 53.2 NMI: 68.7
ARI: 46.1 NMI: 67.5
ARE: 42.6 NMI: 64.7
MVAE
MMVAE ARI: 38.1 NME:
MoPoB
MEME
ARL: 39.9 NME: 63.7
NM1:598
ARI: 44.3 NMI: 65.2
М ARI: 53.4 NMI: 64.0
CMVAE
MMVM ARI: 35.7 NMI: 68.8
MVP
ARI: NMI: 66.1
ARI: 48.1 NMI: 57.6
Scanpy
STAGATE ARI: 55.1
GraphST
SiGra
ARI: 46.6 NMI: $9.7
NMI: 69.7
IDC.S
IDC.7
DCTSSLCTS
IDC
Elccity
Tanoe elge
Elcilty
Tanne, 2aa.2
IDC
Tamee sope
TDC 2
Tanne wlg
IDC
Tanne vlx_
IDC.4
Taaoe.
xSiGra
DCMEM
Ground Truth
Figure 13: Visualization of clustering results the human breast cancer dataset. Each subplot shows the clustering output of different method, with colors indicating predicted clusters. Each method is three times, and the mean ARI and NMI scores are reported above each plot.
26

---

SVAN-
MNISTASVAN
0.8
0.7
0.6
MINEE SWINN
Colenenee
Cohemmne
MIDGE MOIST
0.5
MaNul SVIN
Motul SOUST
HVOZ SVIN
HVaL MINST
MNVV
CHWAA
0.2
MNCAD:
0.1
0.25
0.5
0.75 1.0
0.25
0.5
0.75 1.0
Figure 14: Classification accuracy of cross-modal generations under different missing rates on the MNIST-SVHN dataset. Shaded areas represent the standard deviation across multiple runs. The subseripts in method names indicate the observed modality. For example, DCMEM SVHN (Observation fraction = 0.25) denotes that the training data consists of 25% paired samples and 75% unimodal SVHN samples.
SVINN-MMIST
MNIST=SVINN
DOMEN SVIS
DOMEN SRGST
4-
158
NVZ JYHN
MVZ MNNS
384
MINE SVIN
MINE MISIT
0
250
MUQUE SYIIN
HUVE SOIST
122
140
MSAR MINST
MVAE SVIR
300
NDHVM
CHNAE
MONINE:
MYAWNE
025
100
0.25
075
Obaruake Padion
Obervedds
Figure 15: FID score of cross-modal generations under different missing rates on the MNIST-SVHN dataset. Shaded areas represent the standard deviation across multiple runs. The subscripts in method names indicate the observed modality. For example, DCMEM SVHN (Observation fraction 0.25) denotes that the training data consists of 25% paired samples and 75% unimodal SVHN samples.
latent spaces across different datasets and under various levels of modality incompleteness. Its ability to leverage both paired and unimodal data allows it to maintain superior clustering performance, setting it apart from existing multimodal VAE approaches.
C.3 Semantic Relatedness in the Latent Space
Semantic relatedness refers to the notion that semantically aligned multimodal inputs should yield more similar latent distributions than unrelated pairs. To investigate whether our models as well as the baselines exhibit this behavior, we adopt the 2- Wasserstein distance as a measure of semantic similarity between latent distributions. This metric is well-suited for comparing Gaussian distributions due to its closed-form expression in such cases. In our experiment, we compute pairwise 2- Wasserstein distances between all combinations of latent distributions within a mini-batch. We then visualize the resulting distances using histograms, color-coded to distinguish paired samples from unpaired samples. A clear separation between the two groups in the histogram indicates that the model captures meaningful semantic alignment across modalities. Figure 25 illustrates the relatedness histograms produced by our model the MNIST-SVHN and CUBICC datasets. Figures 26 and 27 show the results for the baseline models.
As shown in Figure 25, our model exhibits consistently lower 2- 2-Wasserstein distances for paired sam- ples, while unpaired samples yield significantly higher distances. This clear separation demonstrates that our model effectively captures semantic alignment across modalities. In contrast, bascline models
27

---

3
2
3
b
60
5
6
6
7
6
1
3
3
5
a
5.
5
O
0
/
9
3
9
2
77
2
19
7
2
4
9
D
4
6
4
9
5
84
6
6
6
0
0
5
5
5
6
7
7
10
O
0
4
7
/
5
7
6
3
6
L 2 2 3 9 5
7
&
8
9
H
2
1
13
C
3
3
3
3
6
6
9999
0
9
0
6
6
6
7
7
5
12
5
15
5
/
3
2
2
S
5
44
4
8
4
28
8 6
5
7
Figure 16: MNIST= (Left) and SVHN->MNIST (Right), for the fully observed case.
6
866
9 9
6
6
6
8 6
8
7
7
7
18
77
M8
27
7
7 7
88
8
9
8
7
8
6
1
8
P
60 6
6
6
16
16
C
6 6
7
3
9
2
3
6
4
18
33
2
4
1
2
3
3
3
9
3
O
50
/
7
7177
0
O
/
7
7
7
0
6
A
4
O
F
8
8
6
6
6
2
2
8
8
3
L
#
L
21
8
8
8
6
6
4
4
Z
2
2
B3
5
ONIO
3
5
O
6
D
6
6 6
4
2
9
0
2
9 9
97
7
D
21
4
2
7
7
2
2
4
4
4
4
4
Figure 17: MNIST->SVHN (Left) and SVHN->MNIST (Right), when MNIST observed 75% of the time.
4
4
2
/
2
9
36
3
4
2 A D
Z
8
R
D
10
2
0
8 8
9 4
0
131
8888
8
3
88
8
8
8
6
1 9
6
6
1
8
6
6
6
66
6 6
9
2
6
6
G
16
2
2 2
6
6
B
18
71
8
8
8
8
3
4
505
14
4
4
5
5
6
6
2
3
6
14
/
2
2
5
4
4
4
7
7
2
91
6
#
9
6
6
4
4
3
2T
/
3
3
7
7
7
/
16
7
6
7
7
7
7
7
6
6
8
7
6
5
M
8
8
3
9
9
3
9
4
5
2
7
Figure 18: MNIST->SVHN (Left) and SVHN->MNIST (Right), when SVHN is observed 75% of the time.
28

---

3
X
8
3
5
3
8
8
6
6
6
2
6
6
6
2
8
7
3
8
2
61
8 8
8
7
6
5
5
6
2
9
3
O
5 S
S
5
5
3
8
//
7
50
0
2
27
2
2
12
8
O
6
6
2
2
8
8
O
0
6
6
3
4
9
9
9
9
3
2
4
3
8
R
9
8
6
&
9
4
0
0
7"
6
8
8
7
7
1
?
7
8
8
18
b
4
5
9
q
6
6
6
9
5
S 5
9
4
q
3
Z
9
Figure 19: MNIST->SVHN (Left) and SVHN->MNIST (Right), when MNIST is observed 50% of the time.
19
9
9
4
3
4
4
6
16
0
0
161G
6
O
0
6
6
0
0
6
6
0
O
0
0
0
0
0
7
9
9
0
4
O
144
6
16
16
0
0
4 4 0 7 7
4
6
4
4
0
O
0
0
717
T
0
O
O
17
2
772
22
7
12
2
7
2
2
G
51
2
2
2 90
G
0
/
5
5
1999
5
5
5
7 7
19
9
4
58
5
5
5
5
5
7
7
07
7
7
7
5
5
19
9
/
18
1801
9
J
D
3
2
18
8
O
2
2
0
8
1
V
5
5
4
9
2
7
7
1
0 O
0
3 3
/
3
0
4 4
4
4
9
4
Figure 20: MNIST=>SVHN (Left) and SVHN->MNIST (Right), when SVHN is observed 50% of the time.
91
8
&
5
Q
8
8
88
8
9
8
8
8
8
8
99
9 9
101
19
7
9
&
16
$ 5
0
0
275
7
7
81
L
?
J
&
18
4 9 9
15
1
8
8
1
8
3
0
8
4
O
S
/
O
6
4
4
4
4
4
B
88
O
8
8
9
9
8
TOLO
5
5
8
O
0
4
3
5 3
1
7
V
4
3
0
2
5
5 6
5
G
0
P
O
OF
4 9 0
9
4
4
4
4
0
0
0
15
2
3
15
5
3
9
O
0
8
4
5 5
4
4
3
/
3
0
Figure 21: MNNST->SVHN (Left) and SVHN->MNIST (Right), when MNIST observed 25% of the time.
29

---

6
30
6
3
/
6
6
9
9 4
9
0
O
441
7
9
4
4
4 4
4
P
81
4
4
4
10
0
O
6
8
9
9
1
4
2
2
O
0
O
0 /
O
/
7
4
6I
1000
0
6
0
7
0
10
0
0
3
O
5
15
5
/
5
5
6
6
5
S
S
II
/
2
5
5
S
5
6
6
C
6
88
6
7
7
5
5
88
8
8 1 3
1
76
7
3
I
5
5
5
9
9
3
1
2
3
3
3
3
2
7
9
9
9
2
6
GIGG
3
2
2
6
31
84
8
8
3
60
O
O
3
3
60
7
4
4
4
3
3
6
6
7
611
06
2
7
4
3
6
Figure 22: MNIST=>SVHN (Left) and SVHN->MNIST (Right), when SVHN is observed 25% of the time.
Coptio--rfrage
DA JUMEALLE
05
175
MVP Traps
MVP Cagene
DOMIN
206
MIME Cipees
MIME Teags
3.1
MANEE Cuzza
MATVE Inug
SPAA Cipice
SIAA JeaK
275
SREVM
ONIVAC
02
316
UNVVAD
MNVVE
01
0.25
075
0.2
03
18
Obaarako
Bocken
Figure 23: Classification accuracy and FID score of cross-modal generations under different missing rates on the CUBICC dataset. Shaded areas represent the standard deviation across multiple runs. The subscripts method names indicate the observed modality. For example, DCMEM _Caption (Observation fraction 0.25) denotes that the training data consists of 25% paired samples and 75% unimodal Caption samples.
such as MVAE and MMVAE display similar distance distributions for both paired and unpaired data, suggesting that their latent representations fail to encode meaningful semantic information. Although MoPoE and MEME capture certain degree of semantic relatedness, evidenced by the relatively small gap between paired and unpaired distributions, they achieve weaker semantic alignment compared
model. MMVAE+, CMVAE and MMVM are only able to capture seman- tic differences on a single dataset with limited generalization capability. Although MVP shows a noticeable separation, the contrast between paired and unpaired distances is less pronounced than in our model. These comparisons further highlight the superior ability of our approach to learn semantically structured and modality-aligned latent representations.
C.4 Class-Contesee Relatedness in the Latent Space
To evaluate whether our model captures class-level semantic alignment across modalities, we conduct a dlasscconexxtua relatedness analysis on both the MNIST-SVHN and CUBICC datasets. Following the methodology proposed in MEME [10], we compute plass-conditionel distance matrix K e CCC, where C' the number classes in the dataset. Each entry Ki, represents the average 2- Wasserstein distance between the latent distributions of class from one modality and class from the other. Ideally, the model successfully aligns class-level semantics across modalities, we expect the matrix to exhibit low distances along the diagonal (representing matched classes) and higher
30

---

Images observed 75% of the time.
medium sized bird has bright yellow and black, und is white beak curve shaped the feet.
this bird has un orange bill and white body with white wings that have black edges.
bird is black and orange with some white, is little but has a pointy bill.
bird red brcast and belly and small bill.
Captions observed 75% of the time.
this bird with yellow helly and black back with white on its wing.
this colorful bird has a white belly and orange crown with blue wings and black gray tarsus pointed brown.
white body, black crown, a bright orange beak with black tip features of this bird.
small yellow bird but under, black bands, and black throat brown.
Images observed 50% of the time.
bird grey white body facial a grey wings.
totally red bird with a long tail.
this bird has wings that are grey and has white belly.
this yellow bird dark breast yellow patch and its tip to with brown bottom.
Captions observed 50% of time.
this bird has flat pointed bill with brown head, and white putch the rectrices of upper its patch check tail light.
this colorful bird yellow bottom half' with an impressive upper half.
bird, black secondaries and small yellow gray tarsus and
this bird white hody long gray and long white..
Images observed 25% the time.
an all dark black bird with sharp thick beak.
this small bird has light brown wings and the beak sits and smaller of that tip.
this small bird is black and blue with purple naple, small white und slightly curved beak.
this bird has black orange body long spots orange.
Captions observed 25% of the time.
small brown and white bird with long pink tarsus and long brown beak.
this small brown bird is colored of leaf sexcs dark stripes.
this bird has white and grey breast with yellow hill.
bird has black and the wings the flank and orange spot.
Figure 24: Cross-modal generations on the CUBICC dataset by DCMEM.
31

---

400
Pained
Folred
300
230
150
100
50
distance
(a) MNIST-SVHN
(b) CUBICC
Figure 25: Histograms of 2- Wasserstein distances between latent distributions for paired and unpaired multimodal samples obtained by DCMEM on the MNIST-SVHN and CUBICC datasets.
MVAE
MMVAE
MEME
MMVAE+
CMVAE
MMVM
MVP
Figure 26: Histograms of 2- Wasserstein distances between latent distributions for paired and unpaired multimodal samples obtained by baseline models on the MNIST-SVHN dataset.
MVAE
MMVAE
MEME
MMVVE+
CMVAE
MMVM
MVP
Figure 27: Histograms of 2- Wasserstein distances between latent distributions for paired and unpaired multimodal samples obtained by baseline models on the CUBICC dataset.
32

---

0.4
010
0.12
0.10
0.08
0.66
0.2
1004
SV--
(a) MNIST-SVHN
(b) CUBICC
Figure 28: Heatmaps of class-conditioned 2- Wasserstein distances between latent distributions obtained by DCMEM on the MNIST-SVHN and CUBICC datasets.
413000
MMVAE+
MMVM
MVP
Figure 29: Heatmaps of class-conditioned 2- Wassersteii distances between latent distributions obtained by baseline models on the MNIST-SVHN dataset.
S000
SUM
MMVVE+
CMVAE
MMVM
MVP
Figure 30: Heatmaps of class-conditioned 2- Wasserstein distances between latent distributions obtained by baseline models on the CUBICC dataset.
33

---

Table 3: Quantitative comparison of clustering performance based latent representations under different missing rates in the MNIST modality on the MNIST-SVHN dataset. Fraction indicates
the proportion of paired samples relative the full training set. Each model is trained dataset
consisting of paired data at a proportion of Fraction and unimodal SVHN data at a proportion of
1-Fraction, and evaluated the complete
SVHN Representation
MNIST Representation
Joint Representation
Fraction
Methods
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
MVAE
18.0
6.0
3.1
77.5
63.4
59.8
43.3
41.9
28.1
MoPoE
27.0
13.4
8.3
50.3
41.4
29.0
61.8
52.4
42.7
0.75
MЕME
11.8
0.3
0.1
41.6
35.9
24.2
11.7
0.4
0.1
MVP
56.4
54.6
39.9
61.3
60.3
45.7
37.8
37.6
22.1
DCMEM
87.4
73.8
74.0
98.7
96.3
97.2
99.5
98.4
98.9
MVAE
18.0
6.0
3.2
70.6
56.0
52.1
43.3
41.8
27.7
MoPoE
22.5
10.0
5.8
52.4
45.8
34.9
52.8
45.4
34.3
0.5
MEME
11.7
0.3
0.1
41.8
35.8
24.6
11.6
0.3
0.1
MVP
29.0
16.0
10.2
50.6
48.8
33.2
42.9
39.0
26.7
DCMEM
87.5
76.1
75.8
98.3
95.2
96.2
98.4
96.6
97.5
MVAE
17.9
5.9
68.3
56.5
50.4
39.3
33.7
22.4
MoPoE
18.6
6.1
3.3
54.3
43.5
31.8
42.5
35.9
23.9
0.25
MЕME
11.6
0.1
45.8
36.9
27.0
11.7
0.3
0.1
MVP
12.3
0.6
0.1
39.1
42.0
24.8
12.3
0.6
0.1
DCMEM
62.0
47.2
38.4
92.3
83.4
90.3
92.3
83.4
84.0
Table 4: Quantitative comparison of clustering performance based latent representations under different missing rates the SVHN modality on the MNIST-SVHN dataset. Fraction indicates the proportion of paired samples relative to the full training set. Each model is trained on dataset
consisting of paired data at proportion of Fraction and unimodal MNIST data at a proportion of
1-Fraction, and evaluated the complete test
SVHN Representation
MNIST Representation
Joint Representation
Fraction
Methods
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
MVAE
17.2
4.8
2.6
82.4
68.0
65.7
40.3
34.6
23.9
MoPoE
32.1
21.6
13.4
53.8
47.2
35.5
61.1
52.4
42.9
0.75
MEME
13.0
3.8
2.2
39.3
36.8
24.1
13.4
6.6
3.3
MVP
49.0
31.0
23.9
78.1
71.0
64.9
75.9
70.9
64.0
DCMEM
89.0
76.2
77.2
99.1
97.3
98.0
99.7
99.1
99.4
MVAE
16.4
5.5
2.4
79.6
65.4
62.5
38.6
38.4
24.7
MoPoE
31.1
18.5
11.6
54.8
47.2
36.3
59.4
49.6
41.9
0.5
МЕME
14.5
3.3
1.6
39.6
33.5
22.1
12.3
0.6
0.2
MVP
49.7
33.1
24.5
83.3
72.9
69.9
77.4
73.5
67.3
DCMEM
85.5
72.4
72.7
98.7
96.3
97.1
99.6
98.8
99.1
MVAE
17.1
5.4
2.7
76.9
63.7
60.6
38.3
31.1
20.7
MoPoE
25.7
13.0
7.4
65.3
53.0
46.3
54.6
44.0
34.3
0.25
МЕME
15.8
4.9
2.6
51.1
43.0
34.4
20.5
13.0
7.7
MVP
47.6
49.5
33.5
85.7
71.3
68.4
87.2
76.9
76.6
DCMEM
82.6
65.6
65.4
95.2
88.4
89.6
98.5
96.0
96.8
values off-diagonal (mismatched classes). To visualize this, we present the resulting matrices as heatmaps in Figure 28, where darker colors indicate smaller distances. The baseline results shown in Figures 29 and 30.
As shown in Figure 28, our model produces a clear diagonal structure in the class-conditionss distance matrices, indicating that effectively aligns semantically corresponding classes across modalities. This pattern is consistently observed on both the MNIST-SVHN and CUBICC datasets, suggesting robust class-level semantic alignment
learned latent space. comparison, baseline models such as MVAE, MMVAF+ and CMVAE fail to exhibit clear diagonal on at least one of the datasets, revealing their limited ability to consistently model class-level correspondence, Other models, including MEME and MVP, either produce diagonals with less pronounced contrast show
34

---

Table 5: Quantitative comparison of clustering performance based latent representations under different missing rates in the Image modality on the CUBICC dataset. Fraction indicates the proportion of paired samples relative to the full training set. Each model is trained on dataset consisting of paired data at proportion of Fraction and unimodal Caption data at proportion of
1-Fraction, and evaluated the complete test
Image Representation
Caption Representation
Joint Representation
Fraction
Methods
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
MVAE
27.2
11.8
7.0
17.8
2.1
0.7
36.9
27.7
18.7
MoPoE
38.1
28.4
17.4
50.8
37.0
27.5
58.5
46.6
35.2
0.75
MEME
45.8
42.1
25.0
27.6
16.3
8.5
45.0
37.1
24.1
MVP
53.1
46.6
34.8
44.3
30.2
21.6
72.7
61.7
51.1
DCMEM
83.6
72.4
66.3
62.6
43.9
35.0
84.1
73.5
67.0
MVAH
27.0
14.3
7.7
17.7
2.4
0.9
36.4
22.1
13.6
MoPoE
32.6
20.2
12.4
46.2
26.6
17.0
42.1
32.6
23.4
0.5
MEME
20.4
4.8
2.4
16.9
1.4
0.2
19.8
4.2
1.8
MVP
58.7
45.4
36.4
31.0
18.5
13.0
52.4
44.2
36.4
DCMEM
84.3
75.7
68.0
51.3
37.2
28.0
79.9
68.5
60.3
MVAE
28.2
13.1
7.9
27.0
12.0
7.3
30.5
17.4
10.5
MoPoE
19.4
5.1
2.4
25.4
11.6
6.1
26.9
10.2
5.8
0.25
MЕME
16.9
1.7
16.1
1.0
17.2
1.8
0.3
MVP
41.7
30.1
20.4
20.6
6.5
4.2
31.4
20.9
17.7
DCMEM
73.6
62.0
51.3
53.1
39.8
32.6
82.7
70.9
64.3
Table 6: Quantitative comparison of clustering performance based latent representations under different missing rates in the Caption modality on the CUBICC dataset. Fraction indicates the proportion of paired samples relative to the full training set. Each model is trained on dataset consisting of paired data at a proportion of Fraction and unimodal Image data at a proportion of
1-Fraction, and evaluated the complete test set.
Image Representation
Caption Representation
Joint Representation
Fraction Methods
ACC
NMI
ART
ACC
NMI
ART
ACC
NMI
ART
MVAE
29.0
12.1
7.5
22.6
11.3
3.8
33.7
26.9
15.1
MoPoF
27.9
15.8
38.2
23.0
15.2
42.0
27.9
17.0
0.75
MEME
24.5
9.1
4.7
24.8
15.7
6.5
24.5
16.2
7.9
MVP
58.8
48.2
37.2
42.9
30.1
21.6
52.3
50.5
36.7
DCMEM
82.0
70.5
63.2
60.8
44.7
33.9
83.9
74.5
67.2
MVAE
28.0
11.3
6.4
20.4
6.3
3.1
46.2
33.1
23.0
MoPoE
27.8
13.1
8.0
33.0
19.3
12.0
31.0
18.9
11.6
0.5
MEME
17.0
1.3
0.3
16.7
1.4
0.2
17.1
1.6
0.3
MVP
50.0
36.0
27.1
45.8
27.4
19.0
64.1
50.9
41.8
DCMEM
81.0
68.4
61.9
57.2
38.8
29.5
82.0
70.1
63.4
MVAE
29.5
17.3
21.2
2.9
35.9
24.6
15.7
MoPoF
24.5
7.6
4.5
25.0
10.3
5.4
27.6
11.7
7.0
0.25
MEME
15.9
0.8
0.1
16.4
1.1
0.1
16.3
1.0
0.1
MVP
51.1
33.4
25.8
40.1
24.1
14.7
54.5
41.8
31.1
DCMEM
65.6
55.2
42.2
54.7
40.4
29.3
67.4
62.9
50.4
undesirably low distances in off-diagonal entries, which implies confusion between unrelated classes. Notably, only our model consistently achieves a strong diagonal with low intra-class distances and high inter-class distances across both datasets, highlighting its superior capability capturing and preserving cross-modal semantic structure.
C.5 Ablation Study
The explicit mathematical definitions of the objective functions used in the ablation study are as follows:
35

---

SVIN--NNIST
MNIS SVINN
Doptiom--Image
DOMEM
DCMBM
6.5
DCMEM
0.3
0.4
0.3
0.2
142
1.20
320
28.0
240
200
Figure 31: Generation performance with different modules ablated. Lstru represents the structured representation learning module and Lshar represents the shared representations alignment module.
Table 7: Clustering performance of joint latent representations with different modules ablated. Cstru represents the structured representation learning module and Lshar represents the shared representations alignment module.
Datasets
UNIST_SVHI
CUBICC
CELBO
Sstru
Shaa
ACC
NMI
ARI
ACC
NMI
ARI
50.6+3.1
32.4+2.1
27.442.2
26.4+0.7
13.7+0.2
00.3+0.1
91.5+2.4
80.6#1.8
82.0+1.2
84.6#1.8
74.1
68.6+2.1
/
72.7+1.9
63.5+1.3
57.7+1.2
19.3+1.3
50.7+1.9
35.0+0.9
/
V
99.5+0.1
98,4+0.2
98.9+0.1
86.3+1.8
76,8+2.8
71.523.1
(1) LELBO denotes the variational lower bound derived in Section 3.1. Under our mutual supervision setup, it includes both - 2 -> and + z -> directions. For the + - direction, the ELBO
4s, (#.]a)) (w]]e] + log Lo 8) + log p(). symmetric term is used for the 2 + direction. Together, they form the total CELBO used in training.
term is given by: LELBO = Ees log Pv- ptp.:lpppee.peee.....
(2) Ssm corresponds to the Structured Representation Learning term introduced in Section 3.1. It also includes bidirectional modeling. For example, the + z -> direction includes a reconstruction term and latent distribution alignment term: S Ty
(*., (*.. (s) log po (s 2z, w.,
(s,() [DLL (4o, (z, = 8) l19.. (z, t))] and vice versa for the t -
direction.
(3) LShr corresponds to the Shared Representations Alignment term introduced Section 3.1. captures the mutual information between z, and æt, defined as: L al zt), where ((5)) is estimated via contrastive learning.
To evaluate the contribution of each component in our model, we perform an ablation study by selectively removing the structured representation learning (Lstra) and the shared representations alignment (Crnar). The results are summarized in Figure 31 and Table 7. As shown in Figure 31, we evaluate the contribution of each module to generation performance using FID and coherence scores across three tasks. The full model consistently achieves the lowest FID and highest coherence scores, indicating superior visual fidelity and semantic consistency. Removing either Lstra or Lshar leads to a clear decline in performance, confirming the necessity of both components. Table 7 reports the clustering performance of different ablation settings on MNIST-SVHN and CUBICC datasets. We observe that omitting Lshar results in modest performance drop as the model loses the alignment constraint for shared latent features, Icading to suboptimal cross- modal representation. In contrast, removing Cstru causes significant decline in all metrics. This suggests that without proper disentanglement of shared and modality-spent sspecifcc information, the model fails to preserve meaningful semantic structure in the shared space. Overall, the ablation results demonstrate that both structured representation learning and shared representations alignment are indispensable for achieving strong performance in both generation and clustering tasks.
To further isolate our architectural contribution, we conduct experiments on the CUBICC dataset by enhancing MVP with the VampPrior mechanism. MVP is selected as the strongest non- SonVVamprrior baseline terms of cross-modal generation and its competitive performance in clustering and classification under various pairing rates (Figures 12, 23; Tables 5, 6). The resulting variant, MVP_ VP,
36

---

Table 8: Generation performance of MVP_ VP (The subscript of each metric indicates the observed modality. For example, Coherence Image (Fraction 0.25) denotes that the training data consists of 25% paired samples and 75% unimodal Image samples).
Fraction
Methods
Coherence_ Image
IiD_Image
Coherence_
FID_ "ID_Caption
MVP
0.242
253.243
0.239
250.773
0.75
MVP VP
0.241
265.464
0.234
251.384
DCMEM
0.497
203.981
0,515
207.708
MVP
0.237
256.383
0.224
255.310
0.5
MVP_ VP
0.215
260.604
0,223
259.447
DCMEM
0.517
204.286
0.471
212.218
MVP
0.231
259.064
0.195
256.498
0.25
MVP VP
0.217
250.562
0.187
271.769
DCMEM
0.313
214.203
0.294
221.351
Table 9: Classification accuracy of MVP_ VP (The subscript of each metric indicates the observed modality. For example, Image Representation Image (Fraction 0.25) denotes that the training data consists of 25% paired samples and 75% unimodal Image samples).
Image Repres entation entation_Image
Caption Repres entation Image
Image Repres
Caption Repres
Fraction
Methods
entation entation_Caption
entation_ Cattoonn_atiionnnn
MVP
0.864
0.561
0.877
0.566
0.75
MVP VP
0.824
0,512
0,804
0.532
DCMEM
0.866
0.631
0.873
0.645
MVP
0.865
0.557
0.825
0.537
0.5
MVP_ VP
0.754
0.520
0.755
0.485
DCMEM
0.871
0.618
0,879
0.620
MVP
0.702
0.518
0.676
0.458
0.25
MVP VP
0.635
0,453
0.647
0.386
DCMEM
0.712
0.526
0.717
0.576
uses pseudo-points from the missing modality construct a Gaussian mixture prior, which guides latent learning from the observed modality. As shown in the Tables 8, 9 and 10, MVP VP does not yield consistent improvements over the original MVP baseline. On the contrary, often leads degradation in generation metrics, classification accuracy and clustering performance, particularly at lower pairing rates. We hypothesize that this is due to a mismatch in modeling assumptions: MVP relies on cycle-consistency alignment, which degenerates to trivial alignment (i.e., with itself) when only one modality is present, yielding zero loss for such cases. The introduction of VampPrior forces these unpaired samples to align with a prior constructed from the missing modality, introducing non-trivial loss term that may disrupt the overall optimization, especially since the alignment does not follow the same cyclic mechanism as MVP's original design. Apart from these results, it is worth noting that two baseline models, MMVM and MEME which use a similar VampPrior strategy, also underperform compared to model. This further indicates that our performance gains stem not only from the use of VampPrior, but from the integration of disentangled representation learning and mutual information alignment within unified mutual supervision framework, which ensures consistent robustness under both paired and missing data scenarios.
C.6 Parameter Analysis
To evaluate the impact of the shared representations alignment component, we conduct a parameter analysis on its weighting factor a. As illustrated in Figure 32, we plot the FID and Coherence scores under different values of across three cross-modal generation tasks. The results reveal that the model achieves optimal performance when a is set to 0.5 or 1. In this range, the alignment module effectively bridges modality gaps by aligning latent representations, thereby preserving semantic consistency and enhancing generation quality. In contrast, when a is too small, the alignment term contributes minimally and results suboptimal cross-mos modal coherence. On the other hand, excessively large a values may lead to overfitting or over-alignment which adversely affects performance. In addition, Table reports the clustering performance of joint latent representations under different a values. We observe that the model maintains strong performance when ranges from 0.1 to 1. However,
37

---

WP Table 10: Clustering accuracy of MVP_ VP (The subscript of each metric indicates the observed modality. For example, Image Representation, Image (Fraction 0.25) denotes that the training data consists of 25% paired samples and 75% unimodal Image samples).
Image Repres
Caption Repres
Joint Repres
Fraction
Methods
entation Image
entation Image
entation Image
MVP
58.8
42.9
52.3
0.75
MVP VP
29.4
32.2
34.5
DCMEM
82.0
60.8
83.9
MVP
50.0
45.8
64.1
0.5
MVP VP
40.2
36.7
44.2
DCMEM
81.0
57.2
82.0
MVP
51.1
40.1
54.5
0.25
MVP VP
25.0
25.4
27.8
DCMEM
65.6
54.7
67.4
Image Repres entation entation_Caption Caption
Caption Repres entation C_pt_Cnaatnn
Joint Repres entation entation_Caption
Fraction
Methods
MVP
53.1
44.3
72.7
0.75
MVP_W VP
41.9
32.3
39.6
DCMEM
83.6
62.6
84.1
MVP
58.7
31.0
52.4
0.5
MVP VP
37.8
34.0
42.9
DCMEM
84.3
51.3
79.9
MVP
41.7
20.6
31.4
0.25
MVP VP
24.8
19.9
22.7
DCMEM
73.6
53.1
82.7
SVIN--MNIST
MNIST-SVIEN
Capton-Inagg
0.44
0.34
0.92
066
0.53
0.88
10
0.80
m:100
0.54
0.74
260
Figure 32: Generation performance under different values.
Table 11: Clustering performance of joint latent representations under different values.
Datasets
MNIST SVHN
CUBICC
a
ACC
NMI
ARI
ACC
NMI
ARI
0.01
94.4+1.2
90.0+1.0
91.4+1.5
84.5+2.1
74.1+1.7
68.6+1.7
0.1
97.9+0.3
95.8+0.2
96.7+0.3
86.7+1.1
75.7+1.4
69.8+1.9
0.5
99.1+0.2
98.6+0.4
97.9+0.3
85.641.6
76.4+1.3
71.2+1.5
99.5+0.1
98.440.2
98.940.1
86.3+1.8
76.8+2.8
71.5+3.1
5
96.5+1.1
91.1+1.3
92.4+1.2
83.4+1.4
72.320.8
66.621.6
10
92.3+1.9
82.1+1.4
83.8+1.7
75.0+1.3
69.7+0.7
59.340.9
100
85.9+1.7
72.6+1.4
71.8+1.8
67.0+1.1
49.3+1.4
40,4+1.
when a exceeds this range, the clustering performance degrade significantly, further confirming the importance a well-halanced alignment strength. Based on both generation and clustering results, recommend setting between 0.5 and I in practice for robust and consistent performance.
38

---

C.7 Computational Resources
All experiments are conducted on machine equipped with an NVIDIA GeForce RTX 2080 Ti GPU and 64 GB of RAM. For the MNIST-SVEH dataset, each run uses 4 CPU workers and approximately 10 GB of GPU memory, with an average training time of around 8 hours per run. We evaluate different methods, each with 3 random seeds, resulting in total compute time of approximately 216 GPU hours (8 x 3). For the CUBICC dataset, each run uses CPU workers and approximately 9 GB of GPU memory. Each training run takes about 16 hours on average. Evaluating 9 methods over seeds results in total compute time of roughly 432 GPU hours (16
3). For the Human Breast Cancer dataset, each run uses 2 CPU workers and around 4 GB of GPU memory. The average runtime approximately hours. With 14 methods and 3 seeds, the total compute time amounts to about 84 GPU hours (2 14 x 3). In total, the experiments require approximately 732 GPU hours. Additional GPU time is used during model development and hyperparameter tuning, which is not included in the above calculation.
39


Reviews:
[{"review_id": "4KiR1SKook", "reviewer": "Reviewer_iAZE", "summary": "This paper presents DCMEM, a VAE-based framework for cross-modal representation learning. The model disentangles latent factors into shared and private components, using mutual supervision and the information bottleneck principle for robust alignment. A key feature is its ability to train on both complete and partial data, showing strong performance across generation, clustering, and classification tasks.", "questions": "1. The model uses both KL divergence and a contrastive MI estimator to align shared latents. Why are both mechanisms necessary, and what unique aspects of alignment does the contrastive term capture?\n2. To isolate model's architectural contributions, has the author considered applying the same VampPrior mechanism to the strongest baseline of the missing data experiment?\n3. Regarding the empirical prior for missing data, are the N anchors fixed pseudo-inputs, or are they sampled dynamically during training?\n4. What is the intuition for why the model, with a simple coordinate encoding, outperforms specialized graph neural network models on the spatial transcriptomics task?", "limitations": "Yes, the authors have adequately addressed the limitations of their work in Appendix A.4. They explicitly state that the model is designed for bimodal data scenarios, which is a fair acknowledgment of the current scope.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The comprehensive experimental validation across multiple tasks, diverse datasets, and various data missingness rates provides strong empirical evidence for the method's effectiveness. This thoroughness makes a compelling case for the model's performance.\n- The model's native ability to handle partially observed data is a significant practical advantage, supported by a valid ELBO derivation. This feature enhances its real-world applicability where complete data pairs are often unavailable.\n\nWeaknesses:\n- The method for handling missing data relies on an empirical prior over N anchors, but the paper fails to specify how these anchors are selected or updated.\n- The comparison on incomplete data may not be entirely fair, as the proposed model leverages a VampPrior-style mechanism that some baselines lack. The observed performance gains might stem more from this specific prior than from the core architectural innovations of DCMEM.\n- The model claims to disentangle shared and private variables, but the shared latent z is inferred from a single modality before being regularized. This indirect enforcement of \"sharedness\" may be less effective at purging modality-specific attributes than methods that infer the shared latent from a joint posterior.", "was_revised": true, "final_justification": "The new experimental results and clarifications have successfully addressed my concerns. Consequently, I believe the paper is now a clear 'Accept' and I have updated my score accordingly.", "rebuttal": "We thank the reviewer for the encouraging feedback and helpful suggestions.\n\n>**Q1: The model uses both KL divergence and a contrastive MI estimator to align shared latents. Why are both mechanisms necessary and what unique aspects of alignment does the contrastive term capture?**\n\n**A1:** While both the KL divergence and the contrastive mutual information (MI) estimator align the shared latent representations, they serve complementary purposes. The KL divergence term $D_{KL}(q_{\\phi_z}(z_s = z \\mid s) \\|\\| q_{\\psi_z}(z_t = z \\mid t))$ arises from a variational upper bound on $I(z_s; s \\mid t)$, which measures the view-specific information retained in the shared latent variable $z_s$ beyond what is predictable from the other modality $t$. Minimizing this term encourages $z_s$ to discard modality-specific redundancy and retain only information accessible to both modalities. In this way, the KL divergence provides a global, distribution-level regularization that promotes the extraction of shared, modality-invariant features. However, KL divergence alone cannot guarantee that the shared latents from both modalities are semantically aligned for individual samples. The contrastive MI estimator directly maximizes $I(z_s, z_t)$, explicitly pulling together corresponding samples (positives) across modalities and pushing apart mismatched pairs (negatives). This provides instance-level alignment and enforces semantic consistency. Using both terms ensures that the shared latent variables are aligned globally in distribution and locally in semantics. Removing either term would weaken the disentanglement and cross-modal consistency, as each addresses a distinct aspect of the alignment problem.\n\n>**Q2, W2: To isolate model's architectural contributions, has the author considered applying the same VampPrior mechanism to the strongest baseline of the missing data experiment?**\n\n**A2:** To isolate our architectural contribution, we conduct experiments on the CUBICC dataset by enhancing MVP with the VampPrior mechanism. MVP (ICLR, 2025) is selected as the strongest non-VampPrior baseline in terms of cross-modal generation and its competitive performance in clustering and classification under various pairing rates **(Figures 12, 23; Tables 5, 6)**. The resulting variant, MVP_VP, uses pseudo-points from the missing modality to construct a Gaussian mixture prior, which guides latent learning from the observed modality.\n\nAs shown in the tables below, MVP_VP does not yield consistent improvements over the original MVP baseline. On the contrary, it often leads to a degradation in generation metrics, classification accuracy and clustering performance, particularly at lower pairing rates. We hypothesize that this is due to a mismatch in modeling assumptions: MVP relies on cycle-consistency alignment, which degenerates to a trivial alignment (i.e., with itself) when only one modality is present, yielding zero loss for such cases. The introduction of VampPrior forces these unpaired samples to align with a prior constructed from the missing modality, introducing a non-trivial loss term that may disrupt the overall optimization, especially since the alignment does not follow the same cyclic mechanism as MVP's original design.\n\nGeneration Performance (The subscript of each metric indicates the observed modality. For example, Coherence_Image (Fraction=0.25) denotes that the training data consists of 25% paired samples and 75% unimodal Image samples.):\n|Fraction|Methods|Coherence_Image↑|FID_Image↓|Coherence_Caption↑|FID_Caption↓|\n|-|-|-|-|-|-|\n|0.75|MVP|0.242|253.243|0.239|250.773|\n||MVP_VP|0.241|265.464|0.234|251.384|\n||DCMEM|**0.497**|**203.981**|**0.515**|**207.708**|\n|0.5|MVP|0.237|256.383|0.224|255.310|\n||MVP_VP|0.215|260.6037|0.223|259.447|\n||DCMEM|**0.517**|**204.286**|**0.471**|**212.218**|\n|0.25|MVP|0.231|259.064|0.195|256.498|\n||MVP_VP|0.217|250.562|0.187|271.769|\n||DCMEM|**0.313**|**214.203**|**0.294**|**221.351**|\n\nClassification Accuracy:\n|Fraction |Methods |Image Representation_Image |Caption Representation_Image |Image Representation_Caption |Caption Representation_Caption |\n|-|-|-|-|-|-|\n|0.75|MVP|0.864|0.561|**0.877**|0.566|\n||MVP_VP|0.824|0.512|0.804|0.532|\n||DCMEM|**0.866**|**0.631**|0.873|**0.645**|\n|0.5|MVP|0.865|0.557|0.825|0.537|\n||MVP_VP|0.754|0.520|0.755|0.485|\n||DCMEM|**0.871**|**0.618**|**0.879**|**0.620**|\n|0.25|MVP|0.702|0.518|0.676|0.458|\n||MVP_VP|0.635|0.453|0.647|0.386|\n||DCMEM|**0.712**|**0.526**|**0.717**|**0.576**|\n\nClustering Accuracy:\n|Fraction |Methods |Image Representation_Image |Caption Representation_Image |Joint Representation_Image |Image Representation_Caption |Caption Representation_Caption |Joint Representation_Caption |\n|-|-|-|-|-|-|-|-|\n|0.75|MVP|58.8|42.9|52.3|53.1|44.3|72.7|\n||MVP_VP|29.4|32.2 |34.5|41.9 |32.3 |39.6 |\n||DCMEM|**82.0**|**60.8**|**83.9**|**83.6**|**62.6**|**84.1**|\n|0.5|MVP|50.0|45.8 |64.1|58.7|31.0 |52.4|\n||MVP_VP|40.2|36.7 |44.2|37.8 |34.0 |42.9 |\n||DCMEM|**81.0**|**57.2**|**82.0**|**84.3**|**51.3**|**79.9**|\n|0.25|MVP|51.1|40.1 |54.5|41.7 |20.6 |31.4 |\n||MVP_VP|25.0|25.4 |27.8|24.8 |19.9 |22.7|\n||DCMEM|**65.6**|**54.7**|**67.4**|**73.6**|**53.1**|**82.7**|\n\nApart from the above results, it is worth noting that two baseline models, MMVM (NeurIPS, 2024) and MEME (ICLR, 2022), which use a similar VampPrior strategy, also underperform compared to our model. This further indicates that our performance gains stem not only from the use of VampPrior, but from the integration of disentangled representation learning and mutual information alignment within a unified mutual supervision framework, which ensures consistent robustness under both paired and missing data scenarios.\n\n>**Q3, W1: Regarding the empirical prior for missing data, are the N anchors fixed pseudo-inputs, or are they sampled dynamically during training?**\n\n**A3:** The prior anchor set ${u^t}$ is dynamically sampled on a per-batch basis rather than being fixed throughout training. For each batch, we randomly select a set of samples from the $t$ modality, with the number of selected anchors matching the batch size. These samples are passed through the encoder to obtain a set of latent Gaussian distributions, which serve as components of a Gaussian Mixture Model, with uniform weights assigned to all components. Since the anchor set is resampled at each training step, the prior remains adaptive and batch-dependent. This dynamic design enables the model to better capture the evolving latent structure during training, improving both stability and expressiveness compared to a fixed prior.\n\n>**Q4: What is the intuition for why the model, with a simple coordinate encoding, outperforms specialized graph neural network models on the spatial transcriptomics task?**\n\n**A4:** While specialized graph neural networks (e.g., STAGATE (NC, 2022), GraphST (NC, 2023), xSiGra (BIB, 2024)) are explicitly designed to model spatial dependencies through neighborhood graphs or attention mechanisms, they often rely on predefined graph structures or hand-crafted connectivity priors. These assumptions may not fully adapt to the irregular, noisy and highly variable nature of spatial transcriptomics data, especially at single-cell resolution. In contrast, our model leverages a simple yet effective coordinate encoding strategy (CoordConv (NeurIPS, 2018)), treating spatial coordinates as part of the input feature space. This allows the model to jointly learn spatial and transcriptomic representations in a flexible and data-driven manner, without being constrained by fixed graph topology or local smoothness assumptions. The shared latent space learns to capture global structure and local spatial patterns directly from the data, which can be more robust to technical noise and biological heterogeneity. Moreover, the strong performance of our model can be attributed to its mutually supervised framework, which unifies gene expression, spatial coordinates and morphological features. By promoting structured disentanglement and cross-modal alignment, it enables spatial structures to emerge naturally without relying on explicitly constructed spatial graphs.\n\n>**W3: The model claims to disentangle shared and private variables, but the shared latent z is inferred from a single modality before being regularized. This indirect enforcement of \"sharedness\" may be less effective at purging modality-specific attributes than methods that infer the shared latent from a joint posterior.**\n\n**A5:** As a matter of fact, this indirect enforcement of sharedness provides a flexible regularization which allows us to leverage mutual supervision together with mutual information-based alignment to ensure that $z$ captures cross-modal, modality-invariant information. Specifically, methods that infer the shared latent from a joint posterior can be grouped into two categories based on the adopted regularization term. For methods such as MoPoE (ICLR, 2021) and MMVAE+ (ICLR, 2023), they use the normal distribution as the regularizers for the learned joint posterior. However, this can be problematic as strong regularization would lead to less informative shared information. Moreover, in the extreme case with a disentanglement design, the modality-specific encoding with enough capacity would contain all information for the given modality and thus the shared subspace is ignored by the decoders. Alternatively, methods use VampPrior strategies such as MMVM (NeurIPS, 2024) and MEME (ICLR, 2022) fail to disentangle the shared and private latents, which inevitably results in entangled representations where modality-specific attributes may leak into the shared space, degrading the quality of the shared latent and ultimately hindering performance on downstream tasks. These limitations are also validated by our experiments: as shown in **Figures 2-3, 6-11, 25-30 and Tables 1-2**, our approach consistently outperforms these baseline methods on both the cross-modal generation and clustering tasks."}, {"review_id": "XaqQn8lZe8", "reviewer": "Reviewer_Qzph", "summary": "This paper proposes a variational autoencoder-based framework for cross-modal representation learning that explicitly disentangles shared and modality-specific factors using enhanced mutual supervision and information bottleneck constraints. The method ensures semantic coherence across modalities by enforcing mutual supervision between their shared latent representations, while also promoting clear disentanglement between shared and specific latent variables.", "questions": "1. The datasets used in this paper (such as MNIST-SVHN and CUBICC) are standard but relatively simple benchmarks in cross-modal research. Is the method proposed in the article applicable to complex datasets such as video-text and audio-visual?\n2. The core contributions appear primarily as combinations or incremental refinements of existing methods. Could the authors clarify explicitly why the proposed combination of existing techniques is theoretically superior or unique compared to closely related existing methods? Can additional theoretical analyses or experiments be provided to demonstrate these unique advantages?\n3. While the method claims to be robust to partial modality availability, it's unclear how performance scales under severe missingness or when one modality is significantly more informative than the other. How does the model perform when one modality is systematically less informative, noisy, or entirely missing?", "limitations": "Yes, the authors have acknowledged key limitations in Appendix A.4", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The proposed model is technically sound, incorporating well-established theoretical concepts like mutual information regularization, disentanglement, and variational inference frameworks.\n2. The paper is generally well-structured, with a logical flow from motivation through methodology to evaluation.\n3. The proposed method addresses important challenges in multimodal representation learning, notably modality disentanglement and handling incomplete multimodal data\n\nWeakness\n1. The experimental settings rely on commonly used yet simplified datasets. The lack of evaluations on more challenging, realistic multimodal tasks (such as video-audio, complex textual descriptions, or noisy sensor data) reduces the paper’s broader practical significance.\n2. The core technical contributions largely consist of recombinations and incremental refinements of existing techniques from prior literature. Although presented clearly, the paper does not introduce significantly new theoretical insights, nor does it thoroughly articulate why this particular combination of existing methods provides a critical advantage.\n3. The mathematical exposition, especially concerning mutual information approximations and ELBO derivations, is densely presented and may impede broader understanding. Providing more intuitive explanations, intermediate interpretations, or summarizing key equations clearly would significantly enhance readability.", "was_revised": true, "final_justification": "Thanks for the authors' response, most of my concerns have been addressed. Thus, I will increase my rating to borderline accept.", "rebuttal": "Thank you for your valuable and insightful comments.\n\n>**Q1, W1: The datasets used in this paper (such as MNIST-SVHN and CUBICC) are standard but relatively simple benchmarks in cross-modal research. Is the method proposed in the article applicable to complex datasets such as video-text and audio-visual?**\n\n**A1:** First, we would like to clarify that the primary goal of our work is not to propose a universal solution applicable to all cross-modal scenarios (e.g., video-text, audio-visual), but rather to address critical and long-standing limitations in existing VAE-based cross-modal frameworks -- insufficient alignment of the shared latent variables across modalities and inadequate disentanglement between shared and private latents.\n\nTo address this, we design a novel model architecture with explicit shared latent variables $z$ and modality-specific variables $w$ and leverages mutual information terms to regulate their contents. By maximizing the mutual information between $z$ and observed multimodal data while minimizing the redundancy between $z$ and $w$, our model achieves principled disentanglement. Furthermore, mutual-supervision-based learning scheme and enhanced alignment mechanisms promote better cross-modal consistency in the shared latent space, which directly benefits downstream tasks such as cross-modal generation, classification and clustering.\n\nWe emphasize that experiments on standard benchmarks such as MNIST-SVHN and CUBICC are well-aligned with our technical objectives and previous studies in this field: these datasets offer clear structural contrasts between modalities (e.g., digits-house numbers, images-captions) and provide a controlled setting to rigorously test the ability of VAE-based models to disentangle and align multimodal latent spaces. Rather than covering all modality types, our focus is on enhancing the VAE-based framework and validating it within the well-established image-text setting. While extending our method to more complex modalities such as video or audio is in principle possible, it represents a distinct research direction. These modalities typically require specialized backbone architectures and feature extractors to capture temporal dependencies, which go beyond the scope of the current study.\n\n>**Q2, W2: The core contributions appear primarily as combinations or incremental refinements of existing methods. Could the authors clarify explicitly why the proposed combination of existing techniques is theoretically superior or unique compared to closely related existing methods? Can additional theoretical analyses or experiments be provided to demonstrate these unique advantages?**\n\n**A2:** Our model is not just a simple combination of existing components, but a principled and structured innovation built around a unified evidence lower bound (ELBO) that serves as the core training objective. This ELBO is specifically designed to address two core limitations in current VAE-based cross-modal models: the insufficient disentanglement between shared and modality-specific information, and insufficient alignment of the shared latent variables, both of which critically undermine cross-modal generation and representation quality.\n\nThe motivation for addressing the insufficient disentanglement between shared and modality-specific information in our work arises from the observation that both explicit (e.g., MMVAE+ (ICLR, 2023), CMVAE (ICLR, 2024)) and implicit (e.g., MoPoE (ICLR, 2021), MMVM (NeurIPS, 2024)) disentanglement strategies in existing VAE-based methods often fail to prevent information leakage between shared and private representations **(Figures 29-30)**. For instance, MMVAE+ and CMVAE introduce a modality-specific latent space $w$ to encourage disentanglement but lack sufficient constraints. As **Section 4.2 (Figures 6-7)**, class-relevant shared information is still encoded in the private variables, leading to significant representation redundancy and degradation in generation and clustering performance.\n\nTo overcome this, we explicitly define both shared ($z$) and private ($w$) latent variables, and introduce a structural mutual information objective to enforce effective disentanglement: (1) By maximizing the mutual information between the shared latent $z$ and observed modalities, we encourage $z$ to encode information that is commonly present across modalities. (2) By minimizing the mutual information between $z$ and $w$, we reduce redundancy and prevent leakage of shared information into the modality-specific space. This structured objective enables us to learn high-quality, disentangled latent representations, as evidenced by our ablation study **(Section B.7, Figure 31, Table 7)**. Removing the structural mutual information objective (w/o \\$\\mathcal{L}\\_{\\text{Stru}}\\$) leads to a marked drop in generation and clustering performance, as $z$ fails to capture shared information while $w$ dominates the representation. This demonstrates that our improvements stem not from simple component aggregation, but from a theoretically grounded disentanglement mechanism that effectively separates shared and private factors.\n\nMoreover, we observe that in previous mutual supervision settings (e.g., MEME (ICLR, 2022)), shared representations inferred from different modalities are often not well-aligned in latent space **(Figures 10-11, 26-27)**. Our model directly aligns the shared representations via mutual information maximization and  encourages consistent shared representation inference across modalities at the sample level, which further enhances cross-modal generation and representation learning. As shown in **Section 4.3 (Figures 9-11)**, our model achieves superior modality alignment, while other VAE-based models (e.g., MVAE (NeurIPS, 2018), MoPoE, MMVM) fail to do so. Furthermore, our ablation study **(Section B.7, Table 7)** confirms the critical role of this alignment mechanism: removing the objective (w/o $\\mathcal{L}_{\\text{Shar}}$) leads to a noticeable performance drop in joint clustering tasks due to insufficient alignment of the shared latent representations across modalities.\n\nFinally, our model introduces a novel disentanglement structure (shared $z$, private $w$) combined with mutual information-based objectives, leading to substantial improvements in downstream tasks including cross-modal generation, classification, and clustering **(Sections 4.2-4.4, B.3-B.6)**. Taken together, these contributions demonstrate that our method is not a mere recombination of prior techniques, but a theoretically motivated and empirically validated framework that effectively addresses long-standing challenges in cross-modal VAE-based learning. We hope this clarification helps convey the novelty and significance of our work.\n\n>**Q3: While the method claims to be robust to partial modality availability, it's unclear how performance scales under severe missingness or when one modality is significantly more informative than the other. How does the model perform when one modality is systematically less informative, noisy, or entirely missing?**\n\n**A3:** In the original manuscript, we have reported the results under varying missing rates, including experiments with 75% modality missingness, which is considered a severe level of incompleteness. As reported in **Sections 4.3, B.3-B.4** of our paper **(Figures 5,12-15, 21-24, and Tables 3-6)**, our evaluations span cross-modal generation, classification and clustering tasks, consistently demonstrating the model's robustness under varying levels of modality missingness.\n\nTo further address the reviewer's concern, we conduct additional experiments on the CUBICC dataset simulating an even more challenging scenario: 90% of the data contains only the Caption modality, and only 10% is paired data, meaning the Image modality is almost entirely missing during training. This setup effectively mimics a situation where modalities are both highly imbalanced and one modality is significantly less available or informative. As shown below, DCMEM still outperforms other baselines across generation, classification, and clustering tasks.\n\nCross-Modal Generation:\n|Methods|Coherence↑|FID↓|\n|-|-|-|\n|MVAE|0.130|353.2|\n|MoPoE|0.140|291.8|\n|MEME|0.133|309.4|\n|MVP|0.189|276.9|\n|DCMEM|**0.290**|**239.6**|\n\nClassification Accuracy:\n|Methods|Image Representation|Caption Representation|\n|-|-|-|\n|MVAE|0.513|0.198|\n|MoPoE|0.384|0.313|\n|MEME|0.135|0.145|\n|MVP|0.674|0.353|\n|DCMEM|**0.706**|**0.559**|\n\nJoint Representation Clustering Performance:\n|Methods|ACC|NMI|ARI|\n|-|-|-|-|\n|MVAE|28.2|10.3|6.7|\n|MoPoE|18.1|2.3|0.9|\n|MEME|16.3|1.5|0.3|\n|MVP|30.8|15.4|10.9|\n|DCMEM|**66.8**|**55.5**|**44.8**|\n\nThese results provide strong evidence that our model remains robust even when one modality is noisy, uninformative, or mostly missing. This robustness stems from DCMEM's ability to effectively leverage limited supervision and unimodal data, enabled by its disentangled latent structure, mutual information-guided alignment and mutual supervision mechanism. As discussed in the **Limitations section**, our framework assumes access to partially observed modalities; if one modality is entirely missing, the problem reduces to a unimodal setting, which falls outside the intended scope of our cross-modal integration design.\n\n>**W3: The mathematical exposition, especially concerning mutual information approximations and ELBO derivations, is densely presented and may impede broader understanding. Providing more intuitive explanations, intermediate interpretations, or summarizing key equations clearly would significantly enhance readability.**\n\n**A4:** To balance the rigor of the paper with space constraints, we have placed the detailed intermediate derivations in the appendix for readers who wish to explore them in depth. Meanwhile, we will add more intuitive explanations and summaries of key equations in the main text to improve the overall readability and accessibility of the paper."}, {"review_id": "Ww2jjVbhya", "reviewer": "Reviewer_RUfA", "summary": "This paper proposes DCMEM, a novel variational framework for cross-modal representation learning that jointly disentangles shared and modality-specific latent variables using an information-theoretic objective. The model introduces a mutual supervision mechanism that aligns shared representations across modalities through cross-modal inference paths, avoiding restrictive assumptions used in prior work like PoE or MoE. An information bottleneck constraint is applied to promote semantic disentanglement, while mutual information regularization ensures consistency between the shared latent codes of different modalities. Notably, the model is designed to support training on partially observed multimodal data via an empirical prior approximation. Extensive experiments on MNIST-SVHN, CUBICC, and a spatial transcriptomics dataset demonstrate strong performance in generation, clustering, and classification tasks, particularly under high missing-view rates, validating both the flexibility and effectiveness of the proposed method.", "questions": "1. How is $I(z_s; z_t)$ approximated? What contrastive loss and projection architecture are used? What temperature or batch strategies are applied?\n\n2. In Table 1, DCMEM underperforms CMVAE on the caption modality. Can you offer an explanation? Is this due to weaker textual representations, or modality imbalance?\n\n3. How is the prior anchor set $\\{u_i^t\\}$ selected? Is it fixed across epochs or sampled per batch?\n\n4. The initialization of K-means has not been clearly explained, such as whether random seed settings, running times, etc. have an impact on the results?", "limitations": "The authors explicitly state in the appendix that the model is specifically designed for bimodal data scenarios and may require non-trivial modifications to generalize to datasets with more than two modalities. This is a reasonable and appropriate limitation to acknowledge. However, this information currently appears only in the appendix, and would be more visible and useful if summarized briefly in the main paper, for instance in the conclusion. Additionally, while the authors mention potential positive societal impact in biomedical domains, the discussion would be strengthened by also addressing data privacy, fairness or ethical considerations related to patient-derived spatial transcriptomics data.\n\nSuggestions:\n\n1. Move or repeat the bimodal-design limitation in the main body of the paper.\n\n2. Add a short statement discussing privacy protection and ethical compliance when working with biomedical data.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1. The proposed DCMEM framework effectively combines bi-directional mutual supervision ($ s \\rightarrow z \\rightarrow t $ and $ t \\rightarrow z \\rightarrow s $), shared-private latent disentanglement, and mutual information regularization into a coherent and principled architecture.\n\n2. The model is capable of learning from partially paired multimodal data using an empirical prior approximation for missing modalities.\n\n3. The method is extensively evaluated on three representative datasets spanning different domains (image-image, image-text, biomedical), and across multiple tasks including cross-modal generation, clustering, and classification.\n\n4. DCMEM achieves state-of-the-art or highly competitive results compared to recent multimodal VAEs, particularly in clustering and generation tasks.\n\nWeaknesses\n\n1. On certain tasks such as the caption modality clustering in CUBICC, DCMEM underperforms CMVAE (Table 1), but no explanation is provided.\n\n2. The paper mentions contrastive learning to estimate $I(z_s; z_t)$ but gives no details about the architecture or training strategy used for the contrastive term.\n\n3. Key implementation choices are insufficiently specified. For example, it is unclear whether the prior anchor set ${u_i^t}$ is fixed or sampled per batch, and how K-means initialization (e.g., random seeds, number of runs) might affect the results.\n\n4. While technically sound, some mathematical derivations (e.g., Eq. 2, 6) are presented with dense notation and limited intuition, reducing accessibility to a broader audience.", "was_revised": true, "final_justification": "Thank the authors for their efforts in clarifying these points. As a result, the main issues have been addressed, and I will maintain my rating.", "rebuttal": "We thank the reviewer for the encouraging feedback and valuable comments.\n\n>**Q1, W2: How is $I(z_s;z_t)$ approximated? What contrastive loss and projection architecture are used? What temperature or batch strategies are applied?**\n\n**A1:** The mutual information $I(z_s;z_t)$ between the shared latent variables $z_s$ and $z_t$ is approximated using a contrastive loss that encourages alignment between representations of paired inputs while distinguishing those from non-paired ones [1]. Given a batch of $B$ paired latent features $(h_s, h_t)$ sampled from $z_s$ and $z_t$, we concatenate them into a set of $2B$ vectors. Pairwise cosine similarities are computed and scaled by a fixed temperature $\\tau = 0.5$ to construct the contrastive logits. Positive pairs are defined between the $i$-th feature in $h_s$ and the $i$-th feature in $h_t$, as they correspond to the paired input instances. All other pairs in the batch are treated as negatives. We apply a cross-entropy loss to encourage the model to assign higher similarity to positives than to negatives. We do not use any additional projection architecture. The batch size matches that of the main training process to ensure sufficient negative diversity.\n\n>**Q2, W1: In Table 1, DCMEM underperforms CMVAE on the caption modality. Can you offer an explanation? Is this due to weaker textual representations, or modality imbalance?**\n\n**A2:** The caption modality in the CUBICC dataset presents inherent challenges for clustering, as the captions are typically short and offer only high-level visual descriptions (e.g., “the bird has a black belly that turns into an orange breast with black wings”), which contain limited discriminative information for species-level categorization. Given that the ground-truth labels are defined based on image content, the caption modality is relatively weak in clustering signal. While this makes it difficult for any method to achieve high performance, our slightly lower NMI compared to CMVAE (ICLR, 2024) may reflect the limitations of our current textual representation learning. Nonetheless, DCMEM still achieves better ACC and ARI scores, suggesting it captures substantial structure from this modality despite the challenge.\n\n>**Q3, W3: How is the prior anchor set selected? Is it fixed across epochs or sampled per batch?**\n\n**A3:** The prior anchor set ${u^t}$ is dynamically sampled on a per-batch basis rather than being fixed throughout training. For each batch, we randomly select a set of samples from the $t$ modality, with the number of selected anchors matching the batch size. These samples are passed through the encoder to obtain a set of latent Gaussian distributions, which serve as components of a Gaussian Mixture Model (GMM), with uniform weights assigned to all components. Since the anchor set is resampled at each training step, the prior remains adaptive and batch-dependent. This dynamic design enables the model to better capture the evolving latent structure during training, improving both stability and expressiveness compared to a fixed prior.\n\n>**Q4: The initialization of K-means has not been clearly explained, such as whether random seed settings, running times, etc. have an impact on the results?**\n\n**A4:** For all methods, including ours and the baselines, we use the standard KMeans implementation from the scikit-learn library with the random seed fixed to 0 via the random_state parameter. This ensures consistent initialization across runs. As a result, the clustering outcomes are solely influenced by the learned latent representations rather than stochastic variations from KMeans initialization or multiple runs.\n\n>**W4: While technically sound, some mathematical derivations (e.g., Eq. 2, 6) are presented with dense notation and limited intuition, reducing accessibility to a broader audience.**\n\n**A5:** We will revise the manuscript to simplify the notation and provide more intuitive explanations for the key mathematical derivations (e.g., Eqs. 2 and 6) to improve clarity and accessibility for a broader audience.\n\n[1] van den Oord A, Li Y, Vinyals O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018."}, {"review_id": "hbDKxC0tHM", "reviewer": "Reviewer_GUru", "summary": "This paper proposes a disentangled mutual supervision framework for cross-modal representation learning with missing modalities. It introduces shared and modality-specific latent variables, guided by mutual information regularization, to learn semantically aligned and disentangled representations. Experiments on MNIST-SVHN, CUBICC, and a spatial transcriptomics dataset show that the method outperforms eight multimodal VAEs in generation quality, semantic consistency, and robustness, demonstrating its effectiveness across both vision and biomedical domains.", "questions": "1. The authors should provide more details on the network architecture and parameter settings used for the spatial transcriptomics dataset.\n2. The implementation of $q_{\\phi}(t | z_s)$ is unclear because it does not involve the variable$ w$; this raises questions about how this distribution is modeled in practice. \n3. The authors are encouraged to discuss the numerical stability and computational method of the importance sampling weight $\\frac{q(t|z_s)}{q(t|s)}$ in Equation (2). Additional explanation would help readers understand its validity and implementation.\n4. The authors should provide the explicit mathematical formulations of the loss functions used in the ablation experiments would improve clarity and help readers better understand the individual contributions of each component.\n5. Minor point: The number of classes for the MNIST-SVHN dataset is not clearly stated.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The method introduces a unique mutual supervision mechanism between modality-specific and shared latent variables, promoting semantic alignment while preserving modality-specific features. This improves interpretability and robustness compared to adversarial or alignment-based methods.\n2. The framework naturally supports partially observed inputs without requiring separate strategies for missing data, making it practical for real-world multimodal scenarios.\n3. The model’s ability to reveal spatial patterns in real biological data demonstrates its practical utility and potential for biomedical discovery.\n\nWeaknesses\n1. The paper provides insufficient details on the network architecture and parameter settings used for the spatial transcriptomics dataset, which limits reproducibility and understanding of model design.\n2. The modeling of $q_{\\phi}(t | z_s)$ remains unclear due to the absence of the latent variable $w$, leaving ambiguity in how this conditional distribution is practically implemented.\n3. The importance sampling weight $\\frac{q(t|z_s)}{q(t|s)}$ introduced in Equation (2) lacks discussion regarding its numerical stability and computational method, which raises concerns about its practical validity and implementation.\n4. The ablation study refers to loss functions without explicitly linking them to corresponding equations in the main text, reducing clarity and making it harder to interpret the individual contributions of each component.", "was_revised": true, "final_justification": "Thank the author for their rebuttal. All my concerns have been resolved, and I will maintain the score.", "rebuttal": "We thank the reviewer for the encouraging feedback and incisive questions.\n\n>**Q1, W1: The authors should provide more details on the network architecture and parameter settings used for the spatial transcriptomics dataset.**\n\n**A1**: For the spatial transcriptomics dataset, we preprocess the gene expression data by selecting the top 3000 highly variable genes, followed by standard normalization and log-transformation. Both the encoder and decoder for this modality are implemented as fully connected neural networks. For the tissue morphology modality, input images are resized to 128×128 pixels. To incorporate spatial context, we adopt the CoordConv (NeurIPS, 2018) technique by appending the 2D spatial coordinates (x, y) as two additional input channels, resulting in a 5-channel input. This modality is processed using convolutional neural networks for both encoding and decoding. Model hyperparameters are set as follows: the weighting coefficient $\\alpha$ is fixed at 1 and both the shared and modality-specific latent dimensions are set to 32. Optimization is performed using Adam with a learning rate of 5e-4, a batch size of 64 and 100 training epochs. The detailed architectures are summarized below:\n\nTranscriptomics modality:\n|Encoder_z|Encoder_w|Decoder|\n|-|-|-|\n|$\\text{Input} \\in {\\mathbb{R}^{3000}}$|$\\text{Input} \\in {\\mathbb{R}^{3000}}$|$\\text{Input} \\in {\\mathbb{R}^{64}}$|\n|FC. 512 & ReLU|FC. 512 & ReLU|FC. 512 & ReLU|\n|FC. 512 & ReLU|FC. 512 & ReLU|FC. 512 & ReLU|\n|FC. 32, FC. 32 & Softplus|FC. 32, FC. 32 & Softplus|FC. 3000|\n\nMorphology modality with spatial coordinates:\n|Encoder_z|Encoder_w|Decoder|\n|-|-|-|\n|$\\text{Input} \\in {\\mathbb{R}^{5 \\times 128 \\times 128}}$|$\\text{Input} \\in {\\mathbb{R}^{5 \\times 128 \\times 128}}$|$\\text{Input} \\in {\\mathbb{R}^{64}}$|\n|$4 \\times 4$ conv. 64 stride 2 pad 1 & ReLU|$4 \\times 4$ conv. 64 stride 2 pad 1 & ReLU|$4 \\times 4$ upconv. 512 stride 1 pad 0 & ReLU|\n|$4 \\times 4$ conv. 128 stride 2 pad 1 & ReLU|$4 \\times 4$ conv. 128 stride 2 pad 1 & ReLU|$4 \\times 4$ upconv. 512 stride 2 pad 1 & ReLU|\n|$4 \\times 4$ conv. 256 stride 2 pad 1 & ReLU|$4 \\times 4$ conv. 256 stride 2 pad 1 & ReLU|$4 \\times 4$ upconv. 256 stride 2 pad 1 & ReLU|\n|$4 \\times 4$ conv. 512 stride 2 pad 1 & ReLU|$4 \\times 4$ conv. 512 stride 2 pad 1 & ReLU|$4 \\times 4$ upconv. 128 stride 2 pad 1 & ReLU|\n|$4 \\times 4$ conv. 512 stride 2 pad 1 & ReLU|$4 \\times 4$ conv. 512 stride 2 pad 1 & ReLU|$4 \\times 4$ upconv. 64 stride 2 pad 1 & ReLU|\n|$4 \\times 4$ conv. 32 stride 1 pad 0, $4 \\times 4$ conv. 32 stride 1 pad 0 & Softplus|$4 \\times 4$ conv. 32 stride 1 pad 0, $4 \\times 4$ conv. 32 stride 1 pad 0 & Softplus|$4 \\times 4$ upconv. 5 stride 2 pad 1 & Sigmoid|\n\n>**Q2, W2: The implementation of $q_\\phi(t|z_s)$ is unclear because it does not involve the variable $w$; this raises questions about how this distribution is modeled in practice.**\n\n**A2**: In our framework, $q_\\phi(t \\mid z_s)$ serves a key role in the mutual supervision mechanism. Specifically, $z_s$ is the shared latent representation inferred from modality $s$ and $q_\\phi(t \\mid z_s)$ models the reconstruction of modality $t$ based solely on this shared information. This setup is intentionally designed to exclude the private latent variable $w$, since the goal is to assess what information is common and transferable across modalities.\nIn practice, we implement $q_\\phi(t \\mid z_s)$ by setting $w = 0$ and passing the concatenation $[z_s, 0]$ into the decoder of modality $t$. \nThis design offers two key advantages: (1) it avoids introducing an additional decoder by reusing $p_\\phi(t \\mid z_s, w)$ with $w$ set to zero, keeping the architecture compact; and (2) it promotes effective disentanglement, as reconstructing $t$ from $z_s$ alone forces $z_s$ to capture modality-invariant, shared information. \n\n>**Q3, W3: The authors are encouraged to discuss the numerical stability and computational method of the importance sampling weight $\\frac{q_\\varphi(t \\mid z_s)}{q_{\\phi_z, \\varphi}(t \\mid s)}$ in Equation (2). Additional explanation would help readers understand its validity and implementation.**\n\n**A3**: The importance sampling weight $\\frac{q_\\varphi(t \\mid z_s)}{q_{\\phi_z, \\varphi}(t \\mid s)}$ in Equation (2) plays a critical role in our training objective, but we acknowledge that its direct computation can introduce numerical instability due to high variance in gradient estimates. This arises because both the numerator and denominator are parameterized distributions that are learned during training, and their stochastic nature may result in noisy, unreliable updates when used in Monte Carlo estimation of the ELBO.\nTo address this issue, we adopt a stop-gradient strategy to stabilize training without compromising the objective, following a rationale similar to that in prior work [1, 2]. Concretely, we prevent gradients from flowing through the  $\\frac{q_\\varphi(t \\mid z_s)}{q_{\\phi_z, \\varphi}(t \\mid s)}$, treating it as a fixed scalar during backpropagation:\nstop_gradient$ \\left(\\frac{q_\\varphi(t \\mid z_s)}{q_{\\phi_z, \\varphi}(t \\mid s)}\\right)$.\nThis modification ensures that the parameters are optimized based on more stable signals, as it avoids amplifying gradient noise through the ratio. Importantly, this treatment does not alter the forward computation of the objective but improves the robustness and reliability of the training dynamics.\nIn summary, this design provides a practical and effective solution to variance-induced instability in training, while still aligning with the theoretical intent of the original ELBO formulation.\n\n>**Q4, W4: The authors should provide the explicit mathematical formulations of the loss functions used in the ablation experiments would improve clarity and help readers better understand the individual contributions of each component.**\n\n**A4**: The explicit mathematical definitions of the objective functions used in the ablation study are as follows：\n\n (1) $\\mathcal{L}\\_{\\text{ELBO}}$ denotes the variational lower bound derived in **Section 3.1**. Under our mutual supervision setup, it includes both $s \\rightarrow z \\rightarrow t$ and $t \\rightarrow z \\rightarrow s$ directions. For the $s \\rightarrow z \\rightarrow t$ direction, the ELBO term is given by: $\\mathcal{L}\\_{\\text{ELBO}}^{s \\rightarrow t} = \\mathbb{E}\\_{q_{\\phi}(z_s, w_s \\mid s)} \\left[\\log \\frac{p_{\\psi_z}(z_s \\mid t) p(w_s) p_\\theta(s \\mid z_s, w_s)}{\nq_{\\phi_z}(z_s \\mid s) q_{\\phi_w}(w_s \\mid s)} \\right]+ \\log q_{\\phi_z,\\phi}(t \\mid s) + \\log p(t)$. A symmetric term is used for the $t \\rightarrow z \\rightarrow s$ direction. Together, they form the total $\\mathcal{L}\\_{\\text{ELBO}}$ used in training. \n\n(2) $\\mathcal{L}\\_{\\text{Stru}}$ corresponds to the Structured Representation Learning term introduced in **Section 3.1**. It also includes bidirectional modeling. For example, the $s \\rightarrow z \\rightarrow t$ direction includes a reconstruction term and a latent distribution alignment term: $\\mathcal{L}\\_{\\text{Stru}}^{s \\rightarrow t} = \\mathbb{E}\\_{q_\\phi(z_s, w_s \\mid s)} \\left[\\log p_\\theta(s \\mid z_s, w_s)\\right] - D_{\\text{KL}}\\left( q_{\\phi_z}(z_s \\mid s) \\|\\| q_{\\psi_z}(z_t \\mid t) \\right)$ and vice versa for the $t \\rightarrow z \\rightarrow s$ direction. \n\n(3) $\\mathcal{L}\\_{\\text{Shar}}$ corresponds to the Shared Representations Alignment term introduced in **Section 3.1**. It captures the mutual information between $z_s$ and $z_t$, defined as: $\\mathcal{L}\\_{\\text{Shar}} = \\alpha I(z_s; z_t),$ where $I(\\cdot;\\cdot)$ is estimated via contrastive learning. \nFor clarity, we will explicitly include the mathematical definitions of these objective terms in the appendix and cross-reference them in the ablation discussion.\n\n>**Q5: Minor point: The number of classes for the MNIST-SVHN dataset is not clearly stated.**\n\n**A5**: We clarify that the MNIST-SVHN dataset used in our experiments consists of 10 classes, corresponding to the digits 0 through 9. We will explicitly state this in the revised version to avoid any ambiguity.\n\n>**Paper Formatting Concerns: The notation contains formatting issues such as an incomplete term I ()(e.g., line 462), which should be corrected for clarity and correctness; There are instances where words are not properly separated by spaces (e.g., line 517); Some equations contain extraneous symbols, such as in Equation (18).**\n\n**A6**: We have carefully revised the manuscript to correct the issues mentioned: (1) The incomplete mutual information term $I()$  on line 462 has been fixed to correct mathematical expression $I(s; t; z_s)$. (2) All instances of missing spaces (e.g., line 517) have been reviewed and corrected to ensure proper word separation. (3) Extraneous or unintended symbols in Equation (18) and other equations have been removed or corrected for clarity and consistency.\n\n[1] Joy T, Schmon S M, Torr P H S, et al. Capturing label characteristics in {vae}s. In International Conference on Learning Representations, 2021.\n\n[2] Joy T, Shi Y, Torr P, et al. Learning Multimodal VAEs through Mutual Supervision. In International Conference on Learning Representations, 2022."}]

Author rebuttal:

Meta-review: This paper presents DCMEM, a VAE-based framework for cross-modal representation learning that explicitly disentangles shared and modality-specific latent variables using mutual information regularization. The work received positive feedback from four reviewers with final ratings of 5, 4, 5, and 5, demonstrating strong agreement between reviewers. The method addresses limitations in existing cross-modal VAE approaches and shows consistent improvements across diverse tasks and challenging missing data scenarios. While some reviewers initially raised concerns about incremental novelty and technical clarity, the authors' responses and additional experiments effectively addressed these issues. The paper makes solid contributions to multimodal representation learning with clear practical applications, particularly in biomedical domains.

__Strengths__:
- Extensive evaluation across multiple datasets and tasks with particularly strong performance under severe missing data conditions 
- Well-motivated combination of mutual information disentanglement, dual-level shared latent alignment, and unified mutual supervision
- Native ability to train on partially observed data without requiring separate strategies for missing modalities

__Weaknessses__:
- Core contributions largely consist of principled combinations of existing techniques rather than fundamentally new theoretical insights
- Framework specifically designed for bimodal scenarios and would require non-trivial modifications for more than two modalities
- Underperforms some baselines on certain metrics without clear explanations in the original submission
Decision: Accept (poster)