Title: Scaling Language-centric Omnimodal Representation Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: representation learning

multimodal

embedding
Abstract: Recent multimodal embedding approaches leveraging multimodal large language models (MLLMs) fine-tuned with contrastive learning (CL) have shown promising results, yet the underlying reasons behind their superiority remain underexplored. This work argues that a crucial advantage of MLLM-based approaches stems from implicit cross-modal alignment achieved during generative pretraining, where the language decoder learns to exploit multimodal signals within a shared representation space for generating unimodal outputs. Through analysis of anisotropy and kernel similarity structure, we empirically confirm that latent alignment emerges within MLLM representations, allowing CL to serve as a lightweight refinement stage. Leveraging this insight, we propose a Language-Centric Omnimodal Embedding framework, termed LCO-Embed. Extensive experiments across diverse backbones and benchmarks demonstrate its effectiveness, achieving state-of-the-art performance across modalities. Furthermore, we identify a Generation-Representation Scaling Law (GRSL), showing that the representational capabilities gained through contrastive refinement scale positively with the MLLM's generative capabilities. This suggests that improving generative abilities evolves as an effective paradigm for enhancing representation quality. We provide a theoretical explanation of GRSL, which formally links the MLLM's generative quality to the upper bound on its representation performance, and validate it on a challenging, low-resource visual-document retrieval task, showing that continual generative pretraining before CL can further enhance the potential of a model's embedding capabilities. Codes, models, and resources are available at https://github.com/LCO-Embedding/LCO-Embedding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper empirically demonstrates that latent alignment emerges within MLLM representations, allowing contrasive learning to serve as a lightweight refinement stage. The authors first propose two ways to identify that text-only contrasive learning is beneficial for the multimodal ability of MLLMs, and then propose Language-centric Omnimodal Representation Learning to improve the multimodel performance of MLLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is well-written and easy to follow, with a logical flow and clear explanations of the proposed methodology. The overall readability makes it accessible not only to experts in the field but also to researchers and practitioners who may be less familiar with the specific domain.\n2. The experimental results demonstrate that the proposed method performs well across a variety of benchmark datasets. This consistency suggests that the approach is robust and effective in different settings, which enhances its credibility and practical value.\n3. The proposed method introduces a novel strategy that can be applied to a wide range of Multi-Modal Large Language Models (MLLMs). Its adaptability across different architectures and tasks indicates that it has the potential to become a valuable tool for future research and real-world applications in multi-modal learning.\n\nWeaknesses:\n1. In Section 2.1, the authors mention conducting text-only contrastive learning, but the process is not clearly explained. Specifically, it would be helpful to clarify how the contrastive loss is formulated in the absence of visual inputs, what kind of data is used, and how this component contributes to the overall training objective. Providing more details would improve reproducibility and deepen the reader’s understanding of the method.\n2. The experiments appear to rely on LoRA (Low-Rank Adaptation) for training the language decoder. However, it is unclear how the performance would change if the model were trained without LoRA—i.e., through direct fine-tuning of the full decoder. A comparison between these two approaches would help assess whether the use of LoRA affects performance or generalization capabilities.\n3. Section 4.2 presumably presents an ablation study or some form of analysis; however, the specific results are not clearly reported in the current version. Including concrete numerical results or visualizations would allow readers to better understand the findings and evaluate the effectiveness of the design choices made by the authors.\n\nMinor:\n1. Line 97 has no citations.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates why multimodal large language models (MLLMs) fine-tuned with contrastive learning (CL) outperform traditional CLIP-style models. It argues that MLLMs benefit from implicit cross-modal alignment gained during generative pretraining. Text-only CL can then serve as a lightweight refinement step that enhances this latent alignment across modalities, eliminating the need for large-scale multimodal pairings during fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1- The paper addresses an important topic, i.e., understanding why MLLM-based approaches are outperforming CLIP-like models.\n\n2- Providing analysis in Section 2 through the use of geometric metrics, such as anisotropy degree and kernel-level similarity.\n\n\n**Weaknesses:**\n\n1. The paper claims that \"a crucial advantage of MLLM-based approaches stems from implicit cross-modal alignment achieved during generative pretraining.\" However, the analyses and experiments presented seem to indicate that text-only contrastive learning plays a significant role in boosting alignment between modalities and enhancing the textual and non-textual representations. This appears to be different from the core claim.\n2. It’s unclear whether the experiments and analyses support the conclusion that text-only contrastive learning enhances both textual and non-textual representations. From the presented results, improvements seem largely attributable to better textual representations. This may, in part, be due to addressing issues such as modality forgetting in MLLMs. Clarifying whether non-textual features also improve, and providing supporting evidence, would strengthen the paper's claims.\n\n3. Ambiguities in Section 2.1 – Anisotropy Analysis:\n- The experimental setup for the anisotropy analysis is not fully detailed. For instance, how many samples were used to compute the anisotropy? Without this, we don’t know that the approximation is fairly accurate for the degree of anisotropy.\nWhat exactly does the layer index in Figure 1 refer to, and why does similarity decrease with increasing layer depth?\n\n\n- The Qwen2.5-Omni model shows a high degree of anisotropy collapse, yet still performs well on benchmarks. How is this explained?\n\n\n- One of the datasets used, PixmoCap, includes long textual descriptions. Could this influence similarity measurements (e.g., shared phrases inflating similarity)? And what is the effect on the fine-tuning with contrastive learning?\n\n\n4. Ambiguities in Section 2.2 – Kernel-level Similarity:\n- In Figure 2, the 3B model appears to have more layers than the 7B model. Why is this the case?\n\n\n- The difference between the original and contrastively fine-tuned models is relatively small in the figure. This raises the question: are improvements mainly due to better textual representations rather than improved multimodal alignment?\n\n\nPrior work has shown that MLLMs can forget text tasks post-pretraining. Could the improvements observed here simply reflect restored text capabilities, rather than enhancements to multi-modal representations?\n\n\n5. Figure 4 lacks baseline performance metrics. If Qwen2.5-VL and Qwen2.5-Omni were fine-tuned, it's essential to show their original (pre-fine-tuning) performance for comparison. This would help quantify the effectiveness of the fine-tuning process and the contribution of the proposed method.\n\n**Minor Weaknesses:**\n\nThe citation for llama at line 97 is missing\n\nQUESTIONS:\nPlease answer the questions mentioned in the Weaknesses section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a language-centric omni-modal representation learning (LC-OMRL) approach that treats a multimodal LLM’s language decoder as the sole trainable component. After freezing all modality encoders and the projector, the decoder is fine-tuned on text-only contrastive triplets (AllNLI or Scale-1M) using LoRA adapters. The authors argue that generative pre-training of the decoder has already induced a latent cross-modal alignment; the lightweight contrastive stage simply “unlocks” this alignment for similarity matching.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n* Updating only the decoder with text contrastive loss is computationally cheap (2 epochs; LoRA rank 64) while yielding SOTA embeddings on a broad benchmark suite.\n* Improvements on image, audio, and video retrieval without seeing any non-text signals during fine-tuning support the latent-alignment hypothesis.\n\n# Weaknesses\n* The lack of comparisons with native Qwen-VL/Omni embeddings and well-known cross-modal alignment methods (e.g., CLIP-style or cross-modal contrastive learning). Without those references, the superiority of LC-OMRL is less convincing.\n* Existing work typically reports cross-modal retrieval (R@k) before/after alignment to quantify improvements. While the paper includes multilingual image retrieval, it does not contrast pre- vs post-training performance, nor does it evaluate retrieval on other modalities (audio, video) explicitly for alignment.\n* The “collapsed vs isotropic” embedding spaces are merely schematic; the axes are not defined, and no real data projection (e.g., t-SNE) is given. Readers cannot verify that clusters truly spread out after training.\n* The text states two advantages of LoRA, but does not compare with classic alternatives such as adding a shallow projection head or fully fine-tuning the decoder; quantitative ablations are confined to LoRA hyper-parameters.\n\nQUESTIONS:\n* Could you add native Qwen-VL/Omni embeddings and other alignment approaches (e.g., CLIP-style contrastive learning) to Figure 4 so that readers can judge absolute and relative gains?\n* Have you measured pre- vs post-contrastive cross-modal retrieval (image→text, text→audio, etc.) to directly quantify alignment improvement? If so, please share those numbers; if not, why was this omitted?\n* Please provide the t-SNE visualization results for the embedding space.\n* What empirical evidence supports choosing LoRA over adding a lightweight projection head or full-decoder tuning?\n\nLIMITATIONS:\nPlease refer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper aims to explore the latent alignment mechanism of MLLM and proposes a GRSL to address this issue. The extensive results demonstrate that the performance surpasses that of existing Open SOTA and API-based methods.

STRENGTHS AND WEAKNESSES:
Strengths:

1. From the identification of the problem, to the design of the empirical experiments, and finally to the proposal of the solution, the entire process is coherent and flows very well.
2. The theoretical analysis in the Appendix provides a solid foundation for the paper.

Weaknesses:

1. The terms "Open SOTA" and "API" are a bit confusing. Could you replace them with specific works? This would make the comparison clearer and more appropriate.
2. Despite the extensive analysis of the problem, Figure 3 still does not make it clear to me what core issue this paper actually solves.
3. The paper lacks case studies beyond quantitative metrics that would help in understanding the benefits of the proposed method. Could you please add some of these details?

QUESTIONS:
I have some high-level concerns, and I would be willing to raise my score if the authors can address them.

LIMITATIONS:
Please refer to the weakness.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05