Title: Learning Shared Representations from Unpaired Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Learning

Unpaired data

Spectral Embedding

universal embedding
Abstract: Learning shared representations is a primary area of multimodal representation learning. The current approaches to achieve a shared embedding space rely heavily on paired samples from each modality, which are significantly harder to obtain than unpaired ones. In this work, we demonstrate that shared representations can be learned almost exclusively from unpaired data. Our arguments are grounded in the spectral embeddings of the random walk matrices constructed independently from each unimodal representation. Empirical results in computer vision and natural language processing domains support its potential, revealing the effectiveness of unpaired data in capturing meaningful cross-modal relations, demonstrating high capabilities in retrieval tasks, generation, arithmetics, zero-shot, and cross-domain classification. This work, to the best of our knowledge, is the first to demonstrate these capabilities almost exclusively from unpaired samples, giving rise to a cross-modal embedding that could be viewed as universal, i.e., independent of the specific modalities of the data. Our project page: https://shaham-lab.github.io/SUE_page.


Peer reviews:
comments: SUMMARY:
The authors demonstrate that shared representations can be learned almost exclusively from unpaired data. First, they use pre-trained unimodal encoders to extract feature embeddings from each modality. Then, they use parametric spectral embeddings to reveal universal structures. Finally, they align the modalities linearly via canonical correlation analysis and correct any misalignment using maximum mean discrepancy.

STRENGTHS AND WEAKNESSES:
**Strengths**
* The authors address an important problem in multimodal learning, supporting their claims with compelling evidence.
* The theory is clearly explained and well-motivated, which helps readers better understand the proposed method.
* The ablation study complements the main experiments and connects the theoretical and empirical parts of the paper.

**Weakness**
* The experiments were conducted using only image and text modalities. Ideally, more pairs should be included.
* The scaling experiments in Figure 5 should be extended to see if and when the performance plateaus, as in (c).
* The authors use a SOTA vision encoder (DINOv2), but the text encoder (Mini-LM) seems small and outdated.

QUESTIONS:
* The paper would greatly benefit from additional experiments concerning scaling and additional modalities.
* The text encoder seems weak and outdated compared to the vision model, which should be changed.
* The results for other downstream tasks do not seem as convincing and should be better contextualized.
* It is worrying that (Fig. 5c) the method does not scale with additional labels; if the scaling from unpaired data also plateaus, then the proposed approach has reached its limits, while others like CLIP would still scale.

LIMITATIONS:
Limitations stated in the paper.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on a cross-modal representation learning problem with limited paired multimodal data instances and a relatively larger amount of unpaired unimodal data. The proposed method combines CCA and MMD, two existing methods. CCA first alignment between the representations from unimodal encoders, and MMD further refines the alignment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tThe paper provides very intuitive explanations to justify why to separate the methodology into 3 stages, especially to justify the random walk stage and the CCA stage.\n2.\tThe paper visualizes the results for the embedding space for a better understanding of the problem and method.\n3.\tThe authors ran an excessive number of experiments on various domains to demonstrate the capability of the proposed method.\nWeaknesses:\n1.\tI believe that the authors missed comparisons and reference to previous works on the same problem: “**when large amounts of unpaired data are available alongside only a very small number of paired samples**”, which contradicts the authors’ claim: “**To the best of our knowledge, this work is the first to learn shared representations from almost exclusively unpaired data.**”\nThe authors should compare their method with at least the most recent work cited below. [3] also leverages CCA to find a multimodal representation space with limited paired data, which I think is the same setting and similar methodology as this paper. \n\nReference:\n[1]  Moschella, Luca, et al. \"Relative representations enable zero-shot latent space communication.\" International Conference on Learning Representations (2023)\n[2] Norelli, Antonio, et al. \"Asif: Coupled data turns unimodal models to multimodal without training.\" Advances in Neural Information Processing Systems 36 (2023).\n[3] Li, Po-han, et al. \"CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features\". International Conference on Learning Representations (2025)\n\n2.\tThere is no ablation study on how changing the unimodal encoders affects the performance on the various tasks and the intuition of when to use which encoder.\nMinor Weaknesses:\n1.\tThe caption of Fig. 1 does not indicate that the retrieval result is by SUE.\n2.\t“he Marqo-FashionSigLIP image encoder” in Section D.\n3.\tThe figures are not sorted in the given numbers in the appendix (Fig. 14 comes earlier than Fig. 12 and Fig. 13).\n\nQUESTIONS:\n1.\tWeakness 1: I think a more comprehensive comparison to the referred works is needed, as this paper does not really compare the proposed method to any baselines. \n2.\tSee Weakness 2.\n3.\tI cannot find the number of parameters of the MMD network. I am skeptical of how the authors trained an MMD network using such a small amount of paired data, especially given that the contrastive baseline does not work.\nOverall, this is a strong paper. However, I have concerns about the lack of baseline comparisons and the unresolved issue in Q3. If these are addressed, I will consider updating my score.\n\nLIMITATIONS:\nNo. \n\nLimitations:\nThe authors mentioned that the limitations are listed in Section 6. However, the section mainly focuses on future works and conclusion, where the discussion on limitations is sparse. I suggest that the authors separate the limitations to another section for clarity.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to learn multi-modal representations from unpaired data. Current methods are trained on large amounts of paired data, which is often difficult to obtain. This work learns shared embeddings largely from unpaired data using spectral embeddings of random walk matrices built separately for each modality. The method is validated on vision-language and vision-vision datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written and addresses an important topic. The motivation behind the problem is explained clearly. \n2. Provided ablation studies and visualization for validation on vision-language and vision-vision datasets. \n\nWeaknesses: \n1. The approach assumes that semantic similarity graphs (for example as shown in Fig 2), derived from unimodal foundation models are sufficiently aligned across different modalities. In practice, text and images can emphasize different semantic aspects. Also, the text caption is subjective to the annotator. This crucial assumption may not always hold.\n2. On a similar note, in lines 148-150, the approach assumes the existence of a single unified latent semantic manifold shared across modalities. Realistically, the semantic structure captured by different modalities (e.g., images vs. text) may differ significantly, violating this assumption.\n3. The argument that similar random walks imply aligned eigenfunctions is presented without rigorous guarantees. Stronger theoretical analysis is required to validate this critical assumption.\n4. The empirical evidence for the approach is drawn largely from vision-language data and some vision-vision data with curated and strongly paired datasets. It is unclear how well the method would generalize to other modality pairs with weaker or noisier semantic alignment. Further, the method is used for vision-language datasets which already have large amounts of paired data. What would be interesting is if this could be validated in domains where such aligned pairs are hard to obtain such as medical imaging, etc. \n5. My major concern is the lack of extensive evaluation. The method is only compared against Contrastive loss and no other state-of-the art methods or losses. \n6. An important ablation is needed. How much % of unpaired data is needed to roughly get similar performance as compared to fully paired data. Performance plotted across % of unpaired data would give us better insights about how well this method works.\n\nQUESTIONS:\n1. The method is validated primarily on vision-language tasks with datasets that are carefully annotated paired data. It is unclear whether the approach would generalize to other modality pairs with weaker or noisier alignment. Further, the method is used for vision-language datasets which already have large amounts of paired data. What would be interesting is if this could be validated in domains where such aligned pairs are hard to obtain such as medical imaging, etc. \n2. The final MMD-based alignment step assumes that the distributions are already well-aligned after CCA. If this initial alignment is poor, MMD alone may not be sufficient to close the gap between modalities without paired supervision. How would this be mitigated? \n3. More details about how these unpaired datasets were constructed are needed. \n4. For retrieval, mAP should also be reported which considers the ranking of the retrieved examples. \n5. The paper claims ‘An order of magnitude more pairs are required to achieve similar results with contrastive learning’. Was any example mining for contrastive loss done? Would the result still hold if informative sampling is done?\n\nLIMITATIONS:\nSeveral important limitations as highlighted in detail in Weaknesses and Questions sections. \n1. Mainly related to crucial assumptions that the method is based on \n2. Need more extensive evaluation, comparison with the state-of-the-art and ablation studies", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for learning a shared embedding for multiple modalities, such as text and image, using primarily unpaired data. The core idea is to first learn unimodal representations for each modality separately using spectral embeddings, which are argued to effectively preserve the global manifold structure. Then these independently learned manifolds are aligned into a shared space using MMD and a small set of paired samples via CCA. The primary contribution is a framework that significantly reduces dependency on large-scale paired data.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n**Originality and Significance:** The paper tackles the highly significant problem of reducing the need for large-scale paired data in multimodal representation learning.\n\n**Methodological Soundness:** The design of the method is well-motivated, and explained clearly. The ablation studies presented in the appendix are a particular strength, clearly demonstrating how each component helps. Extensive experiments across multiple datasets and tasks provide strong empirical support for the method's effectiveness.\n\n### Weaknesses\n\n**Insufficient Baseline Comparisons:** The most critical weakness is the lack of adequate baseline comparisons. The comparison only comparison (in the main paper), is a naive contrastive baseline, making it impossible to judge the practical utility and trade-offs of the method. While the authors argue no direct baselines exist for this setting, several semi- or weakly-supervised methods (mentioned in the related work), would help contextualize the proposed method. Some discussion of existing methods for aligning representations such as https://openreview.net/forum?id=eimAJqoIWt&noteId=eimAJqoIWt or https://openreview.net/forum?id=l2izo0z7gu\nOn a related note, the choice of the autoencoder as a non-spectral baseline in the ablation study is not well-justified - why not a contrastive method (e.g. SimCLR)?\n\n**Coarse-Grained Alignment:** The qualitative results, particularly in the concept arithmetic experiments (Figure 4c), suggest that the alignment between modalities is rather coarse, failing to preserve fine-grained details like facial identity. This may be an inherent trade-off of the chosen approach; by prioritizing global manifold structure via spectral embeddings, the method may naturally lose local, fine-grained information. This trade-off should be investigated more thoroughly.\n\n### Minor Issues\n\n- l. 69: the abbreviation CCA is never defined, unlike MMD and SE.\n- l. 129: \"kenrnel\" -> \"kernel\"\n\nQUESTIONS:\n1.  **Impact of Pre-trained Encoders:** Could you please clarify exactly how features were extracted from the pre-trained backbones (e.g., `[CLS]` token vs. averaged features)? This introduces a hidden confounder that may \"pre-homogenize\" the features between modalities. E.g. filter out features from the backgrounds in images (which also tend not appear in the captions). Please discuss the potential confounding effect of using representations that were part of a pre-training objective. \nI would expect representations of images/videos to have a higher information density than text (especially captions), in which case the global manifold structure across modalities may not be as alignable - does this undercut the proposed method?\n2.  **Baseline Comparisons:** Could you provide comparisons against established semi-supervised or weakly-supervised multimodal learning methods? This is critical for assessing the paper's contribution, and my score would increase significantly if SUE proves competitive.\n3.  **Robustness of \"No Modality Gap\" Claim:** The claim that SUE has no modality gap (Appendix B) is quite interesting. Could it be highlighted more in the main paper? Is this observation robust across different datasets and embedding dimensions, or is it specific to the UMAP visualization shown in Figure 14? Generally, there should be more (of the extensive) results in the main paper, rather than relegated to the appendix.\n\nLIMITATIONS:\nNo. The limitations should be discussed more thoroughly - e.g. inherent potential trade-offs between preserving global structure and fidelity.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}