Title: Disentangled Cross-Modal Representation Learning with Enhanced Mutual Supervision
Venue: NeurIPS 2025
Accepted: yes
Keywords: Variational Autoencoder; Disentangled Representation Learning; Mutual Supervision
Abstract: Cross-modal representation learning aims to extract semantically aligned representations from heterogeneous modalities such as images and text. Existing multimodal VAE-based models often suffer from limited capability to align heterogeneous modalities or lack sufficient structural constraints to clearly separate the modality-specific and shared factors. In this work, we propose a novel framework, termed **D**isentangled **C**ross-**M**odal Representation Learning with **E**nhanced **M**utual Supervision (DCMEM). Specifically, our model disentangles the common and distinct information across modalities and regularizes the shared representation learned from each modality in a mutually supervised manner. Moreover, we incorporate the information bottleneck principle into our model to ensure that the shared and modality-specific factors encode exclusive yet complementary information. Notably, our model is designed to be trainable on both complete and partial multimodal datasets with a valid Evidence Lower Bound. Extensive experimental results demonstrate significant improvements of our model over existing methods on various tasks including cross-modal generation, clustering, and classification.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a disentangled mutual supervision framework for cross-modal representation learning with missing modalities. It introduces shared and modality-specific latent variables, guided by mutual information regularization, to learn semantically aligned and disentangled representations. Experiments on MNIST-SVHN, CUBICC, and a spatial transcriptomics dataset show that the method outperforms eight multimodal VAEs in generation quality, semantic consistency, and robustness, demonstrating its effectiveness across both vision and biomedical domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The method introduces a unique mutual supervision mechanism between modality-specific and shared latent variables, promoting semantic alignment while preserving modality-specific features. This improves interpretability and robustness compared to adversarial or alignment-based methods.\n2. The framework naturally supports partially observed inputs without requiring separate strategies for missing data, making it practical for real-world multimodal scenarios.\n3. The model’s ability to reveal spatial patterns in real biological data demonstrates its practical utility and potential for biomedical discovery.\n\nWeaknesses\n1. The paper provides insufficient details on the network architecture and parameter settings used for the spatial transcriptomics dataset, which limits reproducibility and understanding of model design.\n2. The modeling of $q_{\\phi}(t | z_s)$ remains unclear due to the absence of the latent variable $w$, leaving ambiguity in how this conditional distribution is practically implemented.\n3. The importance sampling weight $\\frac{q(t|z_s)}{q(t|s)}$ introduced in Equation (2) lacks discussion regarding its numerical stability and computational method, which raises concerns about its practical validity and implementation.\n4. The ablation study refers to loss functions without explicitly linking them to corresponding equations in the main text, reducing clarity and making it harder to interpret the individual contributions of each component.\n\nQUESTIONS:\n1. The authors should provide more details on the network architecture and parameter settings used for the spatial transcriptomics dataset.\n2. The implementation of $q_{\\phi}(t | z_s)$ is unclear because it does not involve the variable$ w$; this raises questions about how this distribution is modeled in practice. \n3. The authors are encouraged to discuss the numerical stability and computational method of the importance sampling weight $\\frac{q(t|z_s)}{q(t|s)}$ in Equation (2). Additional explanation would help readers understand its validity and implementation.\n4. The authors should provide the explicit mathematical formulations of the loss functions used in the ablation experiments would improve clarity and help readers better understand the individual contributions of each component.\n5. Minor point: The number of classes for the MNIST-SVHN dataset is not clearly stated.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DCMEM, a novel variational framework for cross-modal representation learning that jointly disentangles shared and modality-specific latent variables using an information-theoretic objective. The model introduces a mutual supervision mechanism that aligns shared representations across modalities through cross-modal inference paths, avoiding restrictive assumptions used in prior work like PoE or MoE. An information bottleneck constraint is applied to promote semantic disentanglement, while mutual information regularization ensures consistency between the shared latent codes of different modalities. Notably, the model is designed to support training on partially observed multimodal data via an empirical prior approximation. Extensive experiments on MNIST-SVHN, CUBICC, and a spatial transcriptomics dataset demonstrate strong performance in generation, clustering, and classification tasks, particularly under high missing-view rates, validating both the flexibility and effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The proposed DCMEM framework effectively combines bi-directional mutual supervision ($ s \\rightarrow z \\rightarrow t $ and $ t \\rightarrow z \\rightarrow s $), shared-private latent disentanglement, and mutual information regularization into a coherent and principled architecture.\n\n2. The model is capable of learning from partially paired multimodal data using an empirical prior approximation for missing modalities.\n\n3. The method is extensively evaluated on three representative datasets spanning different domains (image-image, image-text, biomedical), and across multiple tasks including cross-modal generation, clustering, and classification.\n\n4. DCMEM achieves state-of-the-art or highly competitive results compared to recent multimodal VAEs, particularly in clustering and generation tasks.\n\nWeaknesses\n\n1. On certain tasks such as the caption modality clustering in CUBICC, DCMEM underperforms CMVAE (Table 1), but no explanation is provided.\n\n2. The paper mentions contrastive learning to estimate $I(z_s; z_t)$ but gives no details about the architecture or training strategy used for the contrastive term.\n\n3. Key implementation choices are insufficiently specified. For example, it is unclear whether the prior anchor set ${u_i^t}$ is fixed or sampled per batch, and how K-means initialization (e.g., random seeds, number of runs) might affect the results.\n\n4. While technically sound, some mathematical derivations (e.g., Eq. 2, 6) are presented with dense notation and limited intuition, reducing accessibility to a broader audience.\n\nQUESTIONS:\n1. How is $I(z_s; z_t)$ approximated? What contrastive loss and projection architecture are used? What temperature or batch strategies are applied?\n\n2. In Table 1, DCMEM underperforms CMVAE on the caption modality. Can you offer an explanation? Is this due to weaker textual representations, or modality imbalance?\n\n3. How is the prior anchor set $\\{u_i^t\\}$ selected? Is it fixed across epochs or sampled per batch?\n\n4. The initialization of K-means has not been clearly explained, such as whether random seed settings, running times, etc. have an impact on the results?\n\nLIMITATIONS:\nThe authors explicitly state in the appendix that the model is specifically designed for bimodal data scenarios and may require non-trivial modifications to generalize to datasets with more than two modalities. This is a reasonable and appropriate limitation to acknowledge. However, this information currently appears only in the appendix, and would be more visible and useful if summarized briefly in the main paper, for instance in the conclusion. Additionally, while the authors mention potential positive societal impact in biomedical domains, the discussion would be strengthened by also addressing data privacy, fairness or ethical considerations related to patient-derived spatial transcriptomics data.\n\nSuggestions:\n\n1. Move or repeat the bimodal-design limitation in the main body of the paper.\n\n2. Add a short statement discussing privacy protection and ethical compliance when working with biomedical data.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a variational autoencoder-based framework for cross-modal representation learning that explicitly disentangles shared and modality-specific factors using enhanced mutual supervision and information bottleneck constraints. The method ensures semantic coherence across modalities by enforcing mutual supervision between their shared latent representations, while also promoting clear disentanglement between shared and specific latent variables.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The proposed model is technically sound, incorporating well-established theoretical concepts like mutual information regularization, disentanglement, and variational inference frameworks.\n2. The paper is generally well-structured, with a logical flow from motivation through methodology to evaluation.\n3. The proposed method addresses important challenges in multimodal representation learning, notably modality disentanglement and handling incomplete multimodal data\n\nWeakness\n1. The experimental settings rely on commonly used yet simplified datasets. The lack of evaluations on more challenging, realistic multimodal tasks (such as video-audio, complex textual descriptions, or noisy sensor data) reduces the paper’s broader practical significance.\n2. The core technical contributions largely consist of recombinations and incremental refinements of existing techniques from prior literature. Although presented clearly, the paper does not introduce significantly new theoretical insights, nor does it thoroughly articulate why this particular combination of existing methods provides a critical advantage.\n3. The mathematical exposition, especially concerning mutual information approximations and ELBO derivations, is densely presented and may impede broader understanding. Providing more intuitive explanations, intermediate interpretations, or summarizing key equations clearly would significantly enhance readability.\n\nQUESTIONS:\n1. The datasets used in this paper (such as MNIST-SVHN and CUBICC) are standard but relatively simple benchmarks in cross-modal research. Is the method proposed in the article applicable to complex datasets such as video-text and audio-visual?\n2. The core contributions appear primarily as combinations or incremental refinements of existing methods. Could the authors clarify explicitly why the proposed combination of existing techniques is theoretically superior or unique compared to closely related existing methods? Can additional theoretical analyses or experiments be provided to demonstrate these unique advantages?\n3. While the method claims to be robust to partial modality availability, it's unclear how performance scales under severe missingness or when one modality is significantly more informative than the other. How does the model perform when one modality is systematically less informative, noisy, or entirely missing?\n\nLIMITATIONS:\nYes, the authors have acknowledged key limitations in Appendix A.4", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents DCMEM, a VAE-based framework for cross-modal representation learning. The model disentangles latent factors into shared and private components, using mutual supervision and the information bottleneck principle for robust alignment. A key feature is its ability to train on both complete and partial data, showing strong performance across generation, clustering, and classification tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The comprehensive experimental validation across multiple tasks, diverse datasets, and various data missingness rates provides strong empirical evidence for the method's effectiveness. This thoroughness makes a compelling case for the model's performance.\n- The model's native ability to handle partially observed data is a significant practical advantage, supported by a valid ELBO derivation. This feature enhances its real-world applicability where complete data pairs are often unavailable.\n\nWeaknesses:\n- The method for handling missing data relies on an empirical prior over N anchors, but the paper fails to specify how these anchors are selected or updated.\n- The comparison on incomplete data may not be entirely fair, as the proposed model leverages a VampPrior-style mechanism that some baselines lack. The observed performance gains might stem more from this specific prior than from the core architectural innovations of DCMEM.\n- The model claims to disentangle shared and private variables, but the shared latent z is inferred from a single modality before being regularized. This indirect enforcement of \"sharedness\" may be less effective at purging modality-specific attributes than methods that infer the shared latent from a joint posterior.\n\nQUESTIONS:\n1. The model uses both KL divergence and a contrastive MI estimator to align shared latents. Why are both mechanisms necessary, and what unique aspects of alignment does the contrastive term capture?\n2. To isolate model's architectural contributions, has the author considered applying the same VampPrior mechanism to the strongest baseline of the missing data experiment?\n3. Regarding the empirical prior for missing data, are the N anchors fixed pseudo-inputs, or are they sampled dynamically during training?\n4. What is the intuition for why the model, with a simple coordinate encoding, outperforms specialized graph neural network models on the spatial transcriptomics task?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations of their work in Appendix A.4. They explicitly state that the model is designed for bimodal data scenarios, which is a fair acknowledgment of the current scope.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}