Scaling Language-centric Omnimodal Representation Learning
Authors: Chenghao Xiao
Hou Pong Chan
Hao Zhang
Weiwen Xu
Mahani Aljunied
Yu Rong
Venue: NeurIPS 2025 poster ()
Keywords: representation learning
multimodal
embedding
Primary area: deep_learning
TL;DR: We systematically study the behavior and mechanism of a language-centric learning method to learn omnimodal representation, and propose the generation-representation scaling law.

Abstract: Recent multimodal embedding approaches leveraging multimodal large language models (MLLMs) fine-tuned with contrastive learning (CL) have shown promising results, yet the underlying reasons behind their superiority remain underexplored. This work argues that a crucial advantage of MLLM-based approaches stems from implicit cross-modal alignment achieved during generative pretraining, where the language decoder learns to exploit multimodal signals within a shared representation space for generating unimodal outputs. Through analysis of anisotropy and kernel similarity structure, we empirically confirm that latent alignment emerges within MLLM representations, allowing CL to serve as a lightweight refinement stage. Leveraging this insight, we propose a Language-Centric Omnimodal Embedding framework, termed LCO-Embed. Extensive experiments across diverse backbones and benchmarks demonstrate its effectiveness, achieving state-of-the-art performance across modalities. Furthermore, we identify a Generation-Representation Scaling Law (GRSL), showing that the representational capabilities gained through contrastive refinement scale positively with the MLLM's generative capabilities. This suggests that improving generative abilities evolves as an effective paradigm for enhancing representation quality. We provide a theoretical explanation of GRSL, which formally links the MLLM's generative quality to the upper bound on its representation performance, and validate it on a challenging, low-resource visual-document retrieval task, showing that continual generative pretraining before CL can further enhance the potential of a model's embedding capabilities. Codes, models, and resources are available at https://github.com/LCO-Embedding/LCO-Embedding.

Scaling Language-Centrie Omnimodal Representation Learning
Chenghao Xiao
Hou Pong Chan lif
Hao Zhang l.t Yu Rong1.2.
Weiwen Xu
Mahani Aljunied
DAMO Academy, Alibaba Group
HHupan Lab
Abstract
Recent multimodal embedding approaches leveraging multimodal large language models (MLLMs) fine-tuned with contrastive learning (CL) have shown promising results, yet the underlying reasons behind their superiority remain underexplored. This work argues that a crucial advantage of MLLM MLLM-based approaches stems from implicit moosl alignment achieved during generative pretraining, where the language decoder learns to exploit multimodal signals within a shared representa- tion space for generating unimodal outputs. Through analysis of anisotropy and ker- nel similarity structure, we empirically confirm that latent alignment emerges within MILM representations, allowing CL to serve lightweight refinement stage. Sanguuage-Centrre Omnimodal Embedding
Leveraging this insight, we propose framework, termed LCO-EMB. Extensive experiments across diverse backbones and benchmarks demonstrate its effectiveness, achieving state-of-the-ar perfor- mance across modalities. Furthermore, we identify a Generation Generation-Reprrsentatiio Scaling Law (GRSL), showing that the representational capabilities gained through contrastive refinement scale positively with the MLLM's generative capabilities. This suggests that improving generative abilities evolves as effective paradigm for enhancing representation quality. We provide a theoretical explanation GRSL, which formally links the MI.LM's generative quality to the upper bound on its representation performance, and validate it on challenging, low-resource retrieval task, showing that continual generative pretraining before CL can further enhance the potential of a model's 's embedding capabilities'
Introduction
Cross-moda representation alignment, such as vision-langun -liigage alignment, has traditionally relied on massive-scale contrastive learning (CL) over paired cross-modal data, as seen CLiP-style models [37] 50 83 Prior work primarily focuses on scaling model size, dataset volume, and batch
size during training and zero-shot classification [50 83 performance tends to plateau complex
58
83.. While these strategies demonstrate benefits tasks like linear
probing 9 25
50
tasks requiring deeper cross-modal modal comprehension, e.g., multilingual image retrieval [57 60] visual
text representations
19 74), and tasks involving interleaved multimodal encodings 69
Recent approaches utilize autoregressive multimodal large language models (MLLMs) as the back- bone models, followed by CL fine-tuning, to enhance representational capabilities, leading to im- proved performance on these complicated tasks g 3g 84 However, the underlying reasons for the performance advantages of MLLM- -aseed embedding approaches over traditional CLIP-based ones
Corresponning authors, kenchanhp@gmail.c.m com, hzhang266out1oook.com
Project head.
Codes, models, and resources available att/ttp:://gtthuu.coo//// L0tt://LLC000mmocddd/gLL0000 Bnb.dddinnnnninnnggggggng
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

remain underexplored. This represents critical research gap in understanding the limitations of CLIP-style models and the specific strengths MLLMs bring to these challenging scenarios.
To address this research gap, we conduct a systematic study of MILL--based embedding models across modalities. First, empirically investigate the embedding space patterns of MILLM repre- sentations before and after lightweight CL fine-tuning using only textual data, via anisotropy and kernel-level similarity. Our results show that text-only fine-tuning not only improves the discrim- inability of text embeddings but also generalizes to enhance the discriminability of embeddings in non- non-textual modalities. This finding reveals that MLLMs achieve implicit poss-modal alignment during generative pretraining, such that representation activation for one modality generalizes to others. We posit that the generative objective of MLLMs enables them to leverage multimodal information in the same semantic space by learning to generate textual outputs during pretraining. Thus, we argue that the know ledge foundation and intrinsic multimodal alignment established during generative pretraining grant
embedding models the fundamental advantages.
Building on the observations, we propose Sanguuage-Centriee Omnimodal Embedding framework, termed LCO-EMB, that employs language-centric paired data for efficient CL refinement. We highlight that CL can function as lightweight, post-hoc refinement step for mapping pre-aligned generative embeddings into similarity-matching space in MLLMs, which differs sharply from the computationally intensive CL required by CLIPs for alignment. Accordingly. this emerging paradigm shifts emphasis towards preserving the cross- cross-modal alignment structure established during MLLM pretraining. In line with recent work [24 86), LCO-EMB adopts LoRA (27] for representation activation of MILM,, aiming to enhance its representation capability with minimal disruption to the pretrained generative capabilities and latent multimodal alignment.
Extensive experiments across diverse backbones and benchmarks show that LCO-EMB outperforms state-of-the-art multimodal embedding models trained with much larger multimodal training sets, with text-only training sets. Combining minimal additional multimodal paired data in diverse formats further calibrates the embedding space of LCO-EMB for downstream tasks, setting new state-of- the-art on MIEB [76). while also providing competitive performance on audio and videos. Further analysis reveals that LoRA with language- contrastive learning yields superior results compared to alternative fine-tuning strategies, suggesting the importance of preserving the latent alignment structure during CL. through minimal modification to the MILM's pretrained knowledge. CI. acts lightweight activation mechanism, less as a means of introducing new knowledge and more similanity-matchnng subspace.
serving primarily to project the embedding space into a similarity matching
As LCO-EMB relies on the inherent multimodal alignment capability of MLLMs, we further inves- tigate the relationship between potentials of representation quality and the underlying generative ability of MLLMs. Through experiments with backbones of various sizes and generation strengths, we identify a Gennrattion-Represnttaiii
Scaling Law (GRSL), indicating that multimodal rep- resentational capabilities gained through contrastive refinement scales positively with the MLLM'S generative capabilities before CL. GRSL suggests that improving the MLLM's initial generative capability-via continued generative pretraining or supervised fine-tuning- is an effective strategy for enhancing its potential in multimodal representations. We offer a theoretical explanation for GRSL through PAC-Bayesian generalization bound, showing that an MLLM's generative capability determines an upper bound for its representational potential. To empirically validate this, we introduce SeaDoc, the most difficult visual document retrieval task to date in low-resource Southeast Asian languages. Through continual OCR-intense pretraining in low-resource languages, we show that retrieval performance enhances after the same amount of text-only contrastive learning.
Our contributions are threefold: (I) We propose language-centric omnimodal representation learning framework, achieving promising performance across various MLLM backbones and embedding benchmarks. (2) We identify a Generation Represstttnnn Scaling Law (GRSL), that representational capabilities after CL scales positively with the MILM's generative capabilities. (3) We provide theoretical justification for GRSL, followed by comprehensive empirical studies, demonstrating that generative capability sets fundamental upper bound on representational quality in MLLMs.
Latent Cross-Modal Alignment in MLLMs
In this section, we conduct an in-depth empirical analysis of multimodal large language models (MLLMs) to investigate whether their internal representations exhibit latent cross-modal alignment
2

---

Pixmo Cap
AudioCaps
MSR-VTT
Text before OL
AntoEtop Essmutes
Text bofore DL
Audlo before
Text before
Image Ce
8:7
Image before CT
Audio bedure CL
Audoo after
Video before CL
Text before
Test before CL.
Text befor: CL
Tost
Audie after
Text ofler CL
Text after CL
Testatter CL
Test after CL
Text after CL
10
15
30 35
Laper
Layer Index
Layor
Figure 1: The anisotropy estimates of Sween..5-mmii-3 embeddings actoss text, image, audio, and video modalities. The vanilla model exhibits typical representation degeneration (anisotropy) for all modalities. After applying text-only contrastive learning, embeddings across modalities become more isotropic, indicating latent language-centric cross-modal alignment within the model.
through two geometric properties, iee, degree of anisotropy 2 and kernel-level similarity 29 Specifically, starting with an MILM, we directly take out its text decoder 30), i.e. the LLM,, and fine-tune using text-only contrastive learning with LoRA on anchor-entailment text pairs from NLJ datasets. Then, we merge the trained LoRA weights into the LLM and re-plug into the original MLLM architecture. The detailed experimental settings are summarized Section 4.1
2.1 Analysis of Anisotropy Degrees
Language models trained on sule-supe objectives are known to suffer from anisotropy 21,, an embedding degeneration issue characterized by hidden representations collapsing into confined region of representation space, resulting in high expected cosine similarity between random inputs. Contrastive learning is known to have the uniformity promise [68] 73 through enhancing discriminability across random negative pairs. Here, we employ contrastive learning to fine-tune multimodal large language models (MLLMs) exclusively with paired text data. We then compare the behaviors of models before and after fine-tuning to assess whether text-only training can effectively mitigate anisotropy for non-textual modalities, even in the absence of explicit multimodal training. The successful transfer of improvements across modalities would provide empirical evidence that MLLMs inherently preserve geometrically aligned latent spaces among different modalities.
17
We follow Ethayarajh 17 and Xiao et al. [73 to approximate the degree of anisotropy using the expected mean of cosine similarity between random data points. Let h;, hy ~ D be the embedding vectors sampled independently and identically distributed (i.id.) from the empirical distribution D of the representation space. Then, the degree of anisotropy is calculated as:
h by
Anisotropy Ey,, [cos(O.g)) }] Eyg
[|b by
In practice, we approximate it empirically using finite sample of N embeddings (h
by3 as:
É [cos(()]
2
Σ
h b
N(N ISiCJEN
h
Specifically, we use Owen2.5-0mmi-- 777 the backbone model and fine-tune it with text-only contrastive learning. To ensure objective and fair semantic comparison between text and other modalities, utilize paired datasets, i.e., Pixmo Cap [13] for image-text, AudioCaps [32 for audio- text, and MSR- VTT [79] for video-text, for anisotropy comparison. The changes in the embedding spaces of different modalities after the text-only contrastive learning are depicted in Figure [ As anticipated, the embedding space produced by Qwen2.5-0mmi-33 initially exhibits a collapsed structure and poorly separated distribution across modalities. After text-only contrastive learning, embedding spaces of non-text modalities surprisingly generalize to become more isotropic, dispersing more uniformly across the respective subspaces. The generalized reduction in anisotropy for image, audio, and video embeddings reflects an underlying latent semantic alignment with textual representations within the base model.
2.2 Analysis of Kernel-level Similarity
Building on the latent cross-modal alignment in MILMss we further employ kernel-level similarity to analyze the improvement in similarity structure alignment across modalities after fine-tuning.
3

---

Given a function f: X' - IR" that maps inputs to high- dithemsiensinnl representations, the associated kernel K X X + R characterizes the induced similarity structure via inner product K (z;, z,) (f(zz,, f(zy)) where I;, 2. E X and K E K. Then, kernel alignment metric m. K. x KC + IR is adopted to quantify the similarity between two kernels, i.e., the "similarity of similarity structures", by assessing how closely the distance metric induced by one representation space aligns with that of another. Prior work [29] examines these structures across independently trained models and finds convergence in their representations. For instance, despite being trained separately, LLaMA [64] and DINOv2 [49] exhibit comparable similarity perception of captions and images from paired datasets.
Similar to Huh et al. [29], we adopt mutual NNN to quantify the overlap in the top-k nearest neighbors of each data point shared across the similarity structures induced by two models, f and g. The data samples (o,, yr) are drawn in mini- batches of size b from a distribution X. Taking the image-text alignment as an example, each (ii pair, i.., an image and its corresponding caption, is assumed to share the same semantic content, denoted as x; f y/- These paired samples serve as semantic anchors for evaluating the representations across modalities. Given the models f and ght corresponding embeddings of each paired sample are attained as di - [(x)) and s/; = -(v)). For a mini-batch of b data samples, we can derive the feature sets o [(o1.... Φo and v ( 16). For each feature o, E & (and similarly 24 E V)), the kNN set S(Ø,) (or S(v;) comprises the indices of its nearest neighbors within its feature collection, excluding itself, which is determined by dkna as:
S(Ø;) = denn (0:, 4b \. {o:
S(p)) = dRmn (4),, V {v%e))
(3)
The kernel-level similarity score miNN for a specific feature pair (øi, v/;) is the normalized cardinality of the intersection of their INN sets:
(oN vb) - ((s)))
which indicates the proportion of shared nearest neighbors, where is denotes the number of nearest neighbors. The overall maN metric is computed as the average of the individual mNN (Øt, thr) scores across the mini-batch.
Different from Huh et al. (29 we utilize ker- nel alignment metrics to inspect cross-modal alignment within the same model. Specifically, we attain hidden representations at all layers from the 3B and 7B variants of Qwen2.5-Vi VL-- Instruct, and assess the self-similarity between the vision and language kernels using Equa- tion The text-only contrastive learning is applicd to the language decoder, and alignment scores are compared before and after fine-tuning: As illustrated in Figure 2 two notable findings emerge: (1) cross-modal kernel alignment im- proves after text-only contrastive learning, in- dicating the presence of inherent latent align- ment across modalities; and (2) the 7B variant exhibits consistently stronger cross-modke ker- nel alignment than the 3B variant, both before and after fine-tuning. This advantage may be at- tributed to the expanded parameter space of the larger model, yielding better expressivity and superior ability to capture latent crosll relationships during pre-training Collectively, these findings suggest that inherent cross-modal binding enables the optimization of representation in one modality to induce corresponding improvements in other modalities.
Vision-Language Kernel Alignment across Layers
95
Owen-VL-3B
Qwen-Vi-77--o
Qwen-VL. VL.-
44
Owen-VI.-7P
Qwen-VI.-7B..l
Kornel Alignmeat
0.3
Qwen-VL-7B
02
01
Owen-Vi-BB
Layer Index
Figure 2: Layer-wise vinion-language kernel align- ment before and after text-only contrastive learn- ing, evaluated on Qwen-VI. models with 7B (28 layers) and 3B (36 layers) parameters. Note the 3B model has more layers than the 7B model.
3 Langucge Omnimodal Representation Learning
The preliminary experiments reveal that MLLMs implicitly acquire cross-modal alignment dur- ing pretraining. Although initial embeddings are suboptimal for similarity matching, latent align-
In this example, use the same model to encode the image z; and its caption yı. i.e. f g.

---

Before training:
During training:
After training:
Generalize;
Text-only Cl.
Collapsed embedding space
Isotropic embedding space
LLM Backbone
Vision last token hidden
Audio last loken hidden
Vision
Audio
Summarize the above in one word:
Text last token hidden
Encoder
Encoder
The power of language-centric omnimodal representation learning: Before text-only contrastive learning (CL), representations across modalities in multimodal large language models (MLLMs) exhibit anisotropy. collapsing into a confined subspace. Text-only CL disperses textual representations by increasing their separation, effectively reducing anisotropy. Notably, this process generalizes to alleviate anisotropy in non-textual modalities, despite the absence of direct supervision.
Figure 3:
ment emerges in intermediate layers. This inherent alignment can be efficiently unlocked through lightweight text-only contrastive fine-tuning, enhancing representation quality across both textual and non-textual modalities. Building on this insight, we introduce Language-Cenn Centric Omnimodal repre- sentation learning (LCO-EMB), framework that leverages language-cent centric data and lightweight contrastive learning to boost MILM representation capabilities across modalities.
The contemporary architectures of MLLM are composed of modality-specifet encoders, a projector, and language decoder (i.e., an L.MM), with the projector aligning modality-specifn representations to the decoder's embedding space 2 40 59 (7]. For text-only variants of LCO-EMB, we isolate and fine-tune only the language decoder via text-only contrastive learning, while freezing the parameters of modality encoders and the projector. After training, the updated decoder is re- ce-plugged into the original model. We further incorporate minimal multimodal paired data to calibrate the embedding space for downstream tasks, resulting in multimodal variants of ICO-EMB
Central our method is the preservation of the latent cross- modsl aligmment established during generative pretraining. This alignment, wherein multimodal embeddings are integrated into shared latent subspace by the language decoder, fundamental to the model's multimodal representation capability. We employ LoRA [27]. which introduces low-rank trainable parameters into select layers while freezing the original model. While LoRA widely recognized for enabling parameter-effictee line-tuning, its primary advantage in context is its ability to minimally perturb the original model. This approach yields two critical benefits: (1) preserves the model's 'ss generative capabilities through minimal weight modifications 44; and (2) it maintains the latent cross-modal alignment, especially in the language decoder's 's embedding layer, which is unaffeectd by the adaptation.
Experiments
4.1 Experimental Settings
Backbones and hyperparameters. We use LLaVA-Next [41, Qwen2.5-VL. (2), and Qwen2.5- Omni [77 as backbones, all conforming to the standard architecture of modality-specific encoders, projector, and a language decoder. LLaVA-Nex: and Qwen2.5-VL. focus on image/video-text, while Qwen2.5-Omni supports omnimodal inputs, covering text, image, video, and audio. We utilize the 8B variant of LLa VA-Next, the 3B and 7B variants for both Qwen2.5-VL and Qwen2.5-Omni. We adopt AdamW optimizer with cosine learning rate schedule, a peak learning rate of 4 10 4 and batch size of 768 to train the model for 2 epochs. The LoRA rank (r) and a are set as 64 and 16 for text-only variants and 64 and 128 for multimodal variants, respectively. For multimodal variants of Qwen2.5-0mmi-778 we use reduced learning rate of 3 x 10 due to the loss spike.
Datasets. (1) Text-only Setting. We consider data settings: alI-NLI and Scale-1M. The all-NLI combines MNLI (70] and SNLI [5], both frequently used for sentence representation learning. Each
For multimodal variants with additional image-text and interleaved data, we scale batch size by the ratio of total text-only data size. Thus, for our 370k data (276k text-only), the batch size is 1, 052 (1.37 768).
5

---

Table 1: MIEB-Lite (51 tasks) results broken down by task categories. We provide averages of both English and multilingual tasks. Models are ranked by the Mean (m) column. Shortcuts are x= "CCossliiggull" m= m="Multiliggual" "nnnlssssh", and task categories from MIEB [6].. We refer to the latest MIEB leaderboard to obtain scores for the compared baselines.
MIEB-Lite (51 Tasks)
Model Name (1)
Datasct Size
Rirr:
Clus.
78.
L.P.
Cmp.
VC:
Dox.
STS
(en)
Rirv. (m)
(xkm)
+STS
Mean (en)
Mean
(m)
(11)
(2)
(7)
(8)
(6)
(5)
(2)
(2(44))
(299
(47)
(51)
Eneoder basellars
CLIP-V/T-bigG
34
2B
34,2
S0.8
72.4
77.8
35.0
43.0
35.5
73.4
26.2
34,5
56.5
51.3
SigLIP 50400m
33
75.9
73.8
78.8
32.8
48.0
69.6
35.4
41.4
57.3
53.5
MLLM baselines
VLMPVge-LoRA. LoRA
662k
21.0
66.1
32.1
64.8
29.4
65.3
42.7
70.9
24.8
12.2
49.1
46.0
ES-V 100
46.3
51.9
Wiyage Multimodal
65
76.6
48.6
69.3
84.3
70.4
mmE5 (118)
2.1M
34.2
77.D
59.8
71.1
27.8
59.2
53.9
78.8
66.6
54.6
57.7
61.8
GME (7B) 84
8.0M
37.9
69.6
35.5
68.7
52.2
55.4
86.1
81.8
62.4
75.4
63.4
64.5
Our Text-owly Varients
LCO- (38B
276k
32.6
61.8
45.0
67.4
38.5
$1.7
62.1
86.1
52.4
76.5
56.4
$8.0
LCO- ELMB-VL. (7B)
276k
31.8
52.7
49.1
68.5
40.4
63.1
66.0
88.4
59.8
84.3
57.5
60.4
Multimadal Varinats
LCO- ExcB (.
350k
34.0
71.6
58.1
68.3
46.1
57.8
73.0
83.8
$4.6
76.1
61.6
62.3
LCO- (7BB
370k
36.4
76.0
66.8
72.5
$1.0
65.2
75.6
86.6
63.
83.3
66.2
67.6
I. LCO-FMM-DDM FarF--n--nni (3R)
3706
34.5
78.1
66.1
72.8
48.2
59.2
73.4
85.7
54.5
80.4
64.8
65.3
LCD--EMn-Oaan
82.4
68.8
instance includes premise with three hypotheses (entailment, neutral, contradiction). We use ~-66 triplets from alI-NLI with entailments positives and contradictions as hard negatives. We further construct Scale-1M, curated collection of 1M sentence pairs sampled from 20M multilingual parallel corpora, including Global Voice [48] MUSE [53], News Commentary (62. Tatoeba (Н) Talks [52]. WikiMatrix [55] and other Sentence Transformers sources (51]. This design leverages diverse descriptive text simulate image captions, activating image representations without direct image supervision, and integrates multilingual pairs to enhance cross-lingual alignment, which may in turn enhance multimodal alignment across languages. (2) Multimodal Setting. Building on all-NLI, we further add 944 synthetic multimodal pairs (Appendix [B) to enhance alignment in the downstream task format space, yielding a final dataset of <370 triplets.
Benchmarks. For image-text embedding tasks, we adopt MIEB-Lite (51 tasks), the official lightweight version of MIEB (130 tasks; (66)). which covers eight categories (Appendix including Linear Probing, Retrieval (English and Multilingual), Zero-shot Classification, Compositionality Evaluation, Vision-centric QA, Document Understanding. Clustering, and Visual STS (English and Cross-lingual). For rapid iteration and ablation, we employ a compact subset of 18 overlapping tasks, termed MIEB-Sub18 (Appendix D). For audio-text embedding tasks, we evaluate on AudioCaps [32] and Clotho [16 datasets. For video-text embedding tasks, we utilize MSR-VTT 779 and Activi- tyNet (26] datasets. The performance on these tasks provides complementary evidence supporting the universality and effectiveness of LCO-EMB. extending beyond the vision and language modalities. For both audio-text and video-text embedding tasks, we utilize the Recall@) as the evaluation metric.
4.2 Performance Comparison on the MIEB Benchmark
To better understand the results, we brief the goal and metric of each MIEB-Lite category: (1) Visual STS reformulates semantic textual similarity as vision task by rendering text as images to test models' semantic understanding, evaluated by Spearman correlation; (2) Document Understand- ing/Visual Document Retrieval measures model's ability to capture layout-aware textual semantics in visual documents and image-text alignment, evaluated by nDCG@ 5; (3) Image Linear Probing assesses the discriminative and transferable quality of frozen visual representations using accuracy; (4) Compositionality Evaluation tests fine-grained image-text alignment with accuracy; (5) Vision- centric QA evaluates visual reasoning and understanding through accuracy; (6) Retrieval measures -odality-specifie and joint encoding performance with nDCG @ 10; (7) Zero-shot Classification evaluates similarity- based classification using accuracy: and
(8) Clustering examines the structural coherence of embeddings using the NMI metric. Detailed task descriptions in Appendix C
We evaluate LCO-EMB on the tasks of the MIEB-Lite benchmark. As reported Table
LCO-EM --MBB consistently outperforms strong baselines, including E5-V [30], VLM2Vec 31. Voyage-

---

Table 2: Performance and efficiency comparisons of different training strategies using 3B and 7B
variants of Qwen2.5-VLL backbones. GPU hours are benchmarked by hours x number of H20 GPUs.
Training Time
Multiling. Img. Rir
V-STS
V-STS (cross)
V-STS (multi)
Doc. Und.
Linear Probe
Training Strategy
(GPU Hours)
(Eng.)
Average
Qwen2.5- -L-33B
n/a
31.73
73.82
59.03
68.57
28.82
46.96
51.49
w/ Ctyl-style CL. (multimodal)
~5500 Hours
25.15
72.51
67.45
65.22
48.91
41.05
53,38
w/ Linear Proj. (lext-only)
4.5 Hours
31.31
75.25
62.95
69.32
12
49.19
$2.69
w/ w//Ful-Finetuue (lext-only)
8.5 Hours
44.61
81.65
68.67
77.75
49.71
50.21
62.10
w/ LoRA (text-only)
-.77 Hours
51.61
81.88
74.97
78.30
57.90
53.05
66.28
Qwen2.5.Vi V
n/a
40.31
73.82
59.03
68.56
28.82
46.96
52.92
w/ CLIP-style CL (multimodal)
55000 Hours
18.24
73.92
68.70
65.41
44.89
38.93
50.02
w/ Linear Proj. (text-only)
8.8 Hours
40.29
72.05
65.46
70.88
35.69
52.96
56.22
w/ Full-Finctune (text-only)
17.3 Hours
44.05
83.15
79.09
81.28
58.02
53.34
66.49
w/ LoRA (text-only)
9.3 Hours
56.64
85.05
85,30
83.48
67.49
53.91
71.98
Multimodal-3 65 mmF5 [ and GME Remarkably, despite using only ~0.37M training pairs-about 21x less data than GME (~8M) our multimodal variants set a new state-of-the-art on emhedding models excel at tasks leveraging MLLM backbones' reasoning and cross-modal understanding abilities, such as multilingual alignment, compositionality and document understanding. Beyond these strengths, LCO-EMB also attains competitive results on clustering, linear probing, and zero-shot classification areas where MLLM-based representations typically lag behind CLIP-style models. Notably, even our text-only variants, trained with minimal text-only contrastive data, surpass advanced proprietary model Voyage- Multimodal-3, Incorporating only ~994 additional multimodal samples (image- texteetexttteee text and interleaved data; Appendix B further calibrates the representation space for downstream task formats, resulting in compact yet highly effective dataset ~370k triplets.
MIEB. Consistent with findings from Xiao et al. (76),
4.3 Analysis of Representational Capability of LCO-EMB
To better analyze the representational capability of LCO-EMB, we use the text-only variants, i.e., without utilizing the synthetic multimodal pairs, as evaluation targets and conduct extensive validation and ablation studies on MIEB-Sub18 1u benchmark.
Main results. We assess text-only variants of LCO-EMB against the advanced embedding methods on the MIEB- 18. As illustrated in Figure 6 (Appendix D). (CO-EMB, trained the 3B and 7B variants of Qwwe2.5-VLL-Instrue leading embedding models across a variety of multimodal downstream tasks. On average across all evaluation categories, the text-only variants of LCO-EMB have outperformed E5-V (30) and Doyage-Multimodal-13 [65] by 21.69 and 13.00 points, where E5-V and Voyage-M3 are advanced open-soure and proprietary MLLM embedding models. respectively. Notably, LCO- -MBB delivers significant improvements on the Linear Probing, Cross-lingual Visual STS, and Multilingual Image Retrieval tasks, outperforming prior advanced methods by margins of 21.02, 10.26, and 15.35 points, respectively. The results highlight the effectiveness and generalizability of LCO-EMB It is also noteworthy that while Voyage-M3 is commercial model explicitly optimized on PDF-text pairs for document understanding tasks, LCO-EMB, trained solely on textual data, still achieve comparable results.
(VL) and Qwen2.5- 5mmin (Omni), consistently outperforms the
Comparison of different training strategies. We apply LoRA to enhance representational capacity while preserving latent cross- alignment. To assess this design, we experiment with Qwen2.5- VL 3B and 7B backbones, comparing LoRA against three baselines: (1) standard CLIP-style contrastive fine-tuning on 800K PixmoCaps image-caption pairs, (2) full fine-tuning and (3) a shallow projection that adds a linear layer after the output. Reported in Table [ the CLIP-style baseline underperforms text-only LoRA, requires 50x more training time, and the shallow projection increases parameters but does not effectively leverage pretrained cross mooal structure, yielding only marginal gains over native embeddings. Full fine-tuning achieves reasonable results but remains notably inferior to LoRA. We attribute this gap to an objective mismatch: contrastive loss deviates from the model's pretraining objective, and full fine-tuning consequently induces larger perturbations to the pretrained parameters, which are more likely to disrupt the established cross-modal alignment. El Detailed analysis of LoRA hyperparameters"" is presented in Appendix E We further analyze the
7

---

VL-78
243
OCR: celuux Einboddiig Pefommans
Ouzzi 7B
12T Recaltol)
Autles Retrival AZT Rccollal
242
72-
Orai 13
Domi-101
24.07
10
VI.18
B
Villosto
Ammri-31
VL-38
Perfomaace
Figure 4: Scaling relationship between generation benchmark performance (X-axis) and representa- tion benchmark performance after language-centric contrastive learning (Y-axis).
impact of different training datasets and model merging technique in Appendix
5 Generation-Reprrenttaiii Scaling Law
The superior performance LCO-EMS EMB is primarily attributed to the intrinsic multimodal alignment capabilities of the backbone MLLMs, which we activate through lightweight contrastive fine-tuning This observation prompts a fundamental question: What the relationship between the inherent mul- timodal generative ability of MLLMs and their representation potential? Through empirical analysis, we reveal a positive scaling correlation between these two aspects. Furthermore, we substantiate our empirical findings with theoretical analysis with PAC- PAC-Bayesian generalization bound, linking models' generative capabilities with the upper bound of their representation performance.
5.1 The Relationship between Generative and Representational Capabilities
We conduct an empirical analysis to investigate the relationship between improving multimodal representation capabilities via text-only contrastive learning and the intrinsic generative capacity of MLLMs. Our analysis spans three different types of modality pairs, i.e., OCR-based image-texl tasks, video-text tasks, and audio-text tasks. The experimental setup is detailed below:
We evaluate DCR-dependent capabilities through paired repre- sentation and generative tasks. For representation tasks, we average scores from Visual Semantic Textual Similarity H-STS-Ennglshh and Document Understanding. Generative performance is sured by averaging results from TextVQA [56), Doc (45), OCRBench [4). and ChartQA 44).
OCR-Based Image-Text Tasks:
Video-Text Tasks: Representation capabilities are assessed using video-text Recall @ scores averaged across MSR-VTT (78) and ActivityNet [6]. Generative performance combines results
from Video-MME sub 20 and MVBench 38
Audio-Text Tasks: We compute Recall @ scores using Clotho 16 and AudioCaps For gen-
erative evaluation, we average performance on MMAU S4 and VoiceBench (10, comprehensive
benchmarks encompassing multiple sub-tasks.
Results. As depicted in Figure we observe consistently positive correlation between baseline generative performance before CL and the post-CL representational performance across different MLLM backbones on all task categories. This observation leads to the discovery of Generation- Representation Scaling Law (GRSL), where the representational abilities of MLLMs, enhunced through contrastive refinement, scale positively with the model's 's original generative capability This insight suggests an alternative pathway for advancing multimodal models by harnessing the scaling effects of generative capacity. Next, provide theoretical analysis of GRSL, which formally links the MLLM's generative quality to the upper bound on its final embedding performance.
'There is a slight difference in the evaluation resolution between Table 4 Table due to different codebase versions used for the experiments.

---

5.2 Theoretical Analysis of Senneration-Repreeennii Scaling Law
We aim to prove that stronger generative prior of MLLMs leads better representations after contrastive fine-tuning. We formalize this intuition using the PAC-Bayesian framework. We begin by defining our central hypothesis, deferring formal definitions of population/empirical risk (C Lemp) and the generative quality of the prior (Xr(X; (X; Y)) to Appendix
5.2.1 Central Hypothesis
The core our argument that a good generative prior provides a "warm start" for contrastive fine-tuning. We formalize this follows.
Hypothesis 1 (Generative Warm Start). Let P be a generative prior and Q the posterior after contrastive fininnuning The expected empirical loss under Q is bounded by:
Be-Q [ ((mm)()) - log N - Ip(X;Y) CP,
(1)
where Ep 2 O captures the gap between the ceermatiin-theoreee optimum and the loss achieved after finite-stpp contrastive fune-tuning. A better prior (higher Ip) leads smaller ep.
A detailed justification for this hypothesis, linking Ip(X; to the pre-existing alignment of repre- sentations, is provided in Appendix G.2
5.2.2 Main Theoretical Result
Theorem 1 Cicnerative-Contraaii PAC-Bayes Bound). Let P be generative prior and Q the posterior after contrustive fine-tuning on n samples. Under Hypothesis 7 1 8, the expected population contrastive risk is bounded by:
with probability at least
KT,(QI))) + log
Eagq [0) S N
EP
(2)
2n
Generative Bottleneck
Inefficiency Gap
PAC-Bayes Complexity Penulty
The proof is provided in Appendix H
Corollary 1 (Generative Performance Governs Representation Bound), Using the approximation
IY) HY) (P) (see Appendix G). the bound is directly governed by the prior's
generative loss:
KTQ(Q) P) (T((Q)
(10(
Eg (Cpop(0)) (P) (log H(Y)) +cp
(3)
2n
This result formalizes our central claim: lower generative loss Cp(P) directly tightens the upper bound on the final contrastive performance.
Interpretation of the Bound. The theorem
38
and its corollary reveal three distinct factors that govern the final representation quality:
36
Baseline (high-resolution)
1. The Generative Bottleneck (log NV Ip(X;Y)): The final quality is limited by the mutual information, (X; Y'), captured by the generative prior. A stronger generative model (higher Tp) lowers this performance floor.
34
n-DCG@10
32
30
Continul Generative Training
28
Baseline
26
Baseline (high-resolution)
2. The Optimization Inefficiency (e) A bet- ter prior creates a more favorable optimiza- tion landscape, resulting in smaller "ineffi- ciency gap" 6p-
24
Baseline
SeaDac
SeaDoc (II)
SeaDacIPC
SeaDoe PCC (I))
3. The Fine-tuning Cost (y...): The PAC- Bayes complexity penalty justifies using parameter-efficient methods (like LoRA A) to keep (T[(Q) P) small, thus retaining the ben- efits of the strong generative prior.
Figure 5: Retrieval performance Qwen2.5-VL VL- 3B fine-tuned various continual generative fine- tuning strategies before CL on SeaDoc benchmark. PC denotes PixmoCaps. H denotes training with high resolution.

---

We further empirically investigate whether improving generative capabilities before contrastive learn- ing raises the representational upper bound, which the key aspect of GRSL. We construct a difficult Visual Document Retrieval (VDR) task: SeaDoc, VDR task in low-resource SouthEast Asian lan- guages. Tigure@ssummriies the results for Qwen2.5-VL-331 with four continual pre-training settings: SeaDoc-train, SeaDoc-train + PixmoCaps (PC) to preserve general image perception capabilities, and training both settings at high resolution (H), The trend confirms that representation performance after CL improves as generative capability strengthens. Full details of SeaDoc construction, experimental settings, and analyses are provided in Appendix
6 Related Work
Omnimodal Representation Learning. Existing approaches omnimodal representation learn- typically rely on large-scale cross-modal pairs to train modality-speciffe encoders. Recent highlights the potential of MLLMs for image- text alignment. However, the effec- tiveness of exploiting the latent aligmment inherent in MLLMs' generative capabilities for omnimodal representation learning- and its underlying theoretical basis remisins unexplored.
ing [22 67
progress
39 84
-edality-centric Representation Learning. Prior work explores representation learning for single modality. For instance, ImageBind [22 leverages the image modality the anchor for contrastive learning to align with all other modalities. Web-SSL. [18] explores language-fres (thus "vision-centric") visual representation learning, which scales data volume to be on par with CLIPs to train DINOv2. By scaling up data volume, the vision-centric self-supervised learning can achieve OCR performance on par with CLIP, which is typically thought to attain through textual supervi- sion 63 E5-V [30] leverages text-only learning to generalize to images and composed retrieval tasks. We extensively study the language-centric view to train omnimodal representation models.
Representation Capabilities. Through investigating 50 models across 130 tasks in 39 languages, Xiao et al. (76) report that CLIP's performance gains from scaling data, batch size, and model size have largely plateaued on advanced representation benchmarks, including interleaved encodings [69 851 compositionality [61], textual visual representations [19 (74,, and mmaae-multilinguil text alignment [57]. They further highlight MLLM-based embedding models as a promising alternative, motivating our exploration of the relationship between representational and generative capabilities in MLLMs. Prior work has explored this connection: Cambrian-1 63 combines a shared language decoder with various vision encoders for training MLLMs and demonstrates that the downstream performance of MLLMs scales with the representation capabilities of the vision encoders, while Yang et al. (8)) formalizes the law between visual representation and MLLM generative capabilities. In contrast, we explore fundamentally different concept: the law" between generation and representation capabilities of the MILMM itself. We see above as the "Representation Gennnateenernnnnn Scaling Law" where the MLLM's generation performance scales with the strength of modality codaality-spccffi encoders. In this work, we explore a fundamentally different concept: the "GeeneaationRReeeeeetatinio Scaling law" where the MLLM's representation abilities scale with its own generation capabilities. Our findings align closely with Xiao et al. (75), who demonstrate that LLM-based embeddings excel at instruction following and reasoning-oriented retrieval.
-epreetnReeeesentaationn Scaling
7 Conclusion
In this work, we reveal that the superior performance of MLLM-based embedding approaches originates from implicit cross-modal alignment established during generative pretraining, wherein the language decoder learns to integrate multimodal information within a unified representation space. Leveraging this insight, we develop LCO-EMB, a language-centric omnimodal embedding framework that treats contrastive learning as a lightweight refinement stage, enhancing representation quality while preserving the model's generative capabilities. Building on this formulation, we introduce the -eeraaiiio-Reeeeetattiionn Scaling Law (GRSL), which establishes positive correlation between a model's 's generative capacity and the representation upper bound. Our theoretical analysis, through a PAC- -CC-Bayeiann generalization bound, together with extensive empirical validation on diverse and challenging benchmarks, confirms both the efficacy of LCO-EMB and the generality of GRSL. Collectively, these findings re-conceptualize the role of contrastive learning and position generative pretraining- not merely the expansion of cross-modal data- as the central driver of scalable, efficient, and robust multimodal representation learning. We discuss the limitations of this work in Appendix
10

---

References
[1] Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610, 2019. URL BRL wttes///claanhhoogy..oom
[2]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song. Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.
ArXiv, abs/2502.13323, 2025. URL htttp://aarat arxiv..ivvv.. org/ abs/ 2502. 13923
[3] Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L. Alexander, David W. Jacobs, and Peter N. Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 2019-2026, 2014. URL
10. 1109/CVPR. 2014. 259
[4] Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA learns less and forgets less. Transactions on Machine Learning Research, 2024. URL httptttt/ppeereeiiiiii
net pet/forruniddaaoErzuu
[5] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal, September 2015, Association for Computational Linguistics. URL https: (0r//D55-17550
[6] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Ac- tivitynet: A large-scale video benchmark for human activity understanding. In Proceed- ings of the IEEE conference on Computer Vision and Pattern Recognition, pages 961- 970, 2015. URL URL //wwww fowddonatiionn.ommmmmmm org/ wwwpaaacccecccoteeeeee wewpeetttcvpprpeeeee cvpr_ 2015/ pdf
[7] Guizhen Chen, Weiwen Xu, Hao Zhang, Hou Pong Chan, Deli Zhao, Anh Tuan Luu, and Yu Rong. Geopqa: Bridging the visual perception gap in milmms for geometric reasoning, ArXiv, abs/2509. 17437, 2025. URI. petttps:/aartee /ttp:://aaivvvv.. org/ hase//a///// abs/2509 17437
[8] Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. mmE5: Improving multimodal multilingual embeddings via high-quality synthetic data. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8254- 8275, Vienna, Austria, July 2025. Association for Computational Linguistics. U https
/aaclanthology-oo. org/ 2025.022222222 findings-ace 433/
[9]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020. httpttt::///oceeeteeeeee chen20j html
mlr. press/v119/
[10] Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. Voicebench: Benchmarking Ilm-based voice assistants. ArXiv, abs/2410.17196, 2024. URL
https://arxiv org/ abs/ bbs/240 2410. 17196
[11] Yew Ken Chia, Liying Cheng, Hou Pong Chan, Chaoqun Liu, Maojia Song, Sharifah Mahani Aljunied. Soujanya Poria, and Lidong Bing. M-longdoc: A benchmark for multimodal super- long document understanding and a retrieval-aware tuning framework. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025. Association for Computational Linguistics, 2025. URL httth:://///rtt org abs//bb//244/ (6176)
[12] Google DeepMind. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. ArXiv, abs/2507.06261 2025. URI. https: / //axxiv...... org/ abs/2507 2tt.//557000666111

---

[13] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mo- hammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Witllf,, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state- saaie-off-hhe-eee the- vision- language models. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 91-104, June 2025. URL https: / //openaccess.. thecvf. com/ Open_Weights, laaiss_nndd___p Wwt_e_peenaaaaaaaae htta__pppp.aaaaa.ooo_aaaaeeeeeee
Molmo and _ind__ixoo___ 2025 Batps__ppper__e sooper.pdff........
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. URL 10. 1 1111//1PR....// 2009. 5206848
[15] Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai- Chang. Open- vlthinker: An early exploration complex lisiun-language reasoning via iterative self- improvement. ArXiv, abs/2503. 17352, 2025. URL httlhttps://axxtttt //tpss:////xvii/ org/ abs///a/..///.////////////// 17352.
[16] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning 2019. URL https::t
dataset. ArXiv,
http//abs//11..../0 09387
[17] Kawin Ethayarajh. How contextual are contextualized word representations? Compar- ing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55-65, Hong Kong, China, November 2019, Association for Computational Linguistics. URL sttpp://aclanthooogg.oorrg
[18] David Fan, Shengbang Tong. Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language-free visual representa- tion learning. ArXiv, abs/2504.01017,, 2025. URL httts::////a//:ti:
01017
[19] Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Celine Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https htwpar/wwwweeeetetfooorrr/aeeeee
[20]
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal 1lms in video analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pages 24108 241 18, June 2025. URL
thecvf Video MME_ TThe_Fire Ever_ Comprehensive.
com/ httm//conentt//eemm http:///222222////////////////// of Mwlti modal LLMs Din_iinnCC__ CVPR, 2025 2alll_2222r___2222_ .ddf
[21] Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu, Representation degeneration problem in training natural language generation models. In International Conference Learning Representations, 2019. httt/hht////eeeeeeeeee
teee/forummidd=SEEYYY Rqqm
[22] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 15180-15190 2023. URL
thecvf. com/ cot//////ttt////22///////// Sapers/Girdhar Taaer_mmmeeBddddd One Lame___meedding..... Space To_ Bind Them A11_CVPR.
2023 2023_paper: _pppppprrr.. .df.
12

---

[23] Nathan Godey, Eric Clergerie, and Benoît Sagot. Anisotropy is inherent to self-attention in transformers. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics Vollue I: Long Pupers), pages 35-48, St. Julian's, s. Malta, org/
March 2024, Association for Computational Linguistics. 2024. eacl- 3//
[24] Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, and Han Xiao. jina-embeddings-ve Universal embeddings for multimodal multilingual retrieval. ArXiv, abs/2506. 18902, 2025. URL httpt:t//axxiiii org/ /a////bb////.//////////// 18902
[25] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), June 2020. URL https //openaccess thecvf cww/ content CVpptt__________ CVPR__CCRR____0__ WWPPR_2000/appers/BBe haw.r//reeeeeemmmmmuuuuumo___ Momentum Coinenttu_CCotrast... for Jor_Unsupervise__Lsuuu Represssttnpreesttttitooo.ooo www.errnnnnnnnn..a___ee 2002_ Sattlopppppppp__
[26] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Ac- tivitynet: A large-scale video benchmark for human activity understanding. In 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 961-970, June 2015. URL http::tt////att cv www.::/atooooooiiiio..ooo..oooo.
http:///peeaccccc/cotttttttttttt papers/ Sopers/Heilbue HwwHHHeeehoooooeee hwt.::/w.....ctiiaeeoeoaaaaaaaaa
pdf
[27] Edward Hu. yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL
[28] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. ArXiv, abs/2503.067499 2025. Uttp:ttts:///arii org/ abs/2503 250.00663330909999
[29] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Muchine Learning Research, pages 20617-20642 PMLR, 21-27 Jul 2024. htttttttt///:ooceeeeee
mmr www.pppwss///////2 hwh24ahhh.aa html
[30] Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang. and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. ArXiv, abs/2407.12580, 2024. URL https://arxiv.. org/ 2b4/
12580
[31] Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. VLM2vec: Training vision-language models for massive multimodal embedding tasks. In The Thirteenth International Conference on Learning Representations, 2025. URL httph::t///// //op:r/v//eeeeee
[32] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generat- ing captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 119-132, Minneapolis, Minnesota, June 2019. Asso- ciation for Computational Linguistics. URL
-tt.//ogyy..gg/1---111111 1011/
[33] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In 20/3 IEEE International Conference on Computer Vision Workshops, pages Clipriitbbiggg111-ainn2---99 2024.
2013. URL 10. www1/11199...... 2013.77
[34] laion.
http::/hnugiinffccecc//aa
CLIP- www..--bgcc-----11ooo-----------
[35] Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vasion-language models: insights and future directions. ArXiv, abs/2408. 12637,
2024. TRRLhttps::// wwwpaawwwww./////////...
org/ httgg//bs/240.... 12637
13

---

[36] Sicong Leng, Jing Wang, Jiaxi Li, Hao Zhang, Zhiqiang Hu, Boqiang Zhang, Yuming Jiang, Hang Zhang, Xin Li, Lidong Bing. Deli Zhao, Wei Lu, Yu Rong. Aixin Sun, and Shijian Lu. Mmrl: Enhancing multimodal reasoning with variance-aware sampling and open resources. ArXiv, abs/2509.2 21268, 2025. URL.ttttss httph:tttt:/aart
werg/aba//20 /509 21268
[37] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-
image pre-training for unified lsioonlanguage understanding and generation. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12980. PMLR, 17-23 Jul 2022. URL https //proceedings mlr. prwpppeeesse.... http//vv22/111222 htmll
[38] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pages 22195-22206, June 2024. URL hnttps://openaccess: thecvf. httpc/tcc...oooeeee////////////p Comprehensive_Mulltee Multi modal_ Video_ www.aoowwwwoooeeeaaaaaaaaaaaaaaa pdf
A_
[39] Sheng-Chie Chieh Lin, Chankyu Lee. Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS. In The Thirteenth International Conference on Learning Representations, 2025. URL fotrrummddiii0e https: /ttpprs///penrvvee... httee/oorrmmddd145002222..
[40] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In
Advances in Neural Information Processing Systems, volume 36, pages 34892-34916. Curran Associates, Inc., 2023, URL //ppoe/ee/o nwwrips.e.ppp..pe.. cc/paperpppppeeeeeeeeeeeeeeee faaeeerfffllepppppeeeeeeee pdf
[41] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae
Lee. Llava-next: Improved reasoning, oer, and world knowledge, January 2024. URL https
/ '1lavaavl.gither http.//ooo///000000-------------
[42] Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12), 2024. URL https://arrs
org/ abs/ 07895
[43] Feipeng Ma, Hongwei Xue, Guangting Wang, Yizhou Zhou, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, and Xiaoyan Sun. Multi-modal generative embedding model. ArXiv, abs/2405. 19333, 2024. URL https://///x org/ / abs 2405. 19333
[44] Ahmed Masry, Xuan Long Do. Jia Qing Tan, Shafiq Joty, and Enamul Hoque ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263-2223777 Dublin, Ireland, May 2022. Association for Computational Linguistics. URL
org/
2022. 1092.findings-ac acl.innngsaa...7. 177
[45] Minesh Mathew, Dimosthenis Karatzas, and C.V. Jawahar. Docvqa: A dataset for vqa document images. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision (WACV), pages 2200-2209, January 2021. URL htttps:// htttps://openaccesss thecrf
DocVQA D_aaaat_tttt.tttaaaa for VQA_ on Se__D_Dooetttttmeee Images WACV
[46] Niklas Muennighoff, Hongjin SU, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In The Thirteenth International Conference on Learning Representations, 2025. URL /ttpss//opperrevees het/forum??ddBBAIIVSS3
[47] Niklas Muennighoff, Nouamane Tazi, Loie Magne, and Nils Reimers. MTEB leaderboard. 2025.
14

---

[48] Khanh Nguyen and Hal Daumé III. Global Voices: Crossing borders in automatic news summarization. In Proceedings of the 2nd Workshop on New Frontiers Summarization, pages 90-97, Hong Kong, China, November 2019, Association for Computational Linguistics. URL attpss//aclammthoooy..org// D
[49]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin Hl-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. URL attps://openreevess.o wwwp:/ewwwwwwiieeeeoeeeeeeeeeeee
[50]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pages 8748-8763. PMLR, 18-24 Jul 2021. URL hetpps//proceedingg..u mlr /ttpe/ediii...l..pppee//////////
[51] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China, November 2019. Association for Computational Linguistics. URL httpp::/acclattboooog..oo ortp:///oooooooo1/-1-11191 1410/
[52] Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512 4525, Online, November 2020. Association for Computational Linguistics. URL http:tt://aaaattoolol.o//
366/
[53] Vin Sachidananda, Ziyi Yang, and Chenguang Zhu. Filtered inner product projection for crosslingual embedding alignment. In International Conference on Learning Representations, 2021. URL htt.::ttpp//// ftttap:///penrevieee net aet//forummtddAADNNoooooo
[54] S Sakshi, Utkarsh Tyagi. Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. MMAU: massive multi-task audio understanding and reasoning benchmark. In The Thirteenth International Conference on Learning Representations, 2025. URL httpss//openrevieet net met/foounme ht/ffformmmdd Pd-
[55] Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. Wiki- Matrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1351-1361, Online, April 2021. Association for Computational Linguistics. URL https::/acllathhll...
.ow.org///000 2021 ww1.2eeacc-maii... 115/
[56] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326.
2019. URL wttpt::t///// htttts::///peeaccese wwwraaacccssss..eeeeeeeee twwctfteeecceeeeeee com wwwctotetnnccccceeeeeoe Towards_ Sntardd_VQA_laee VQA_Models___a_eeoooooooaageeese Tha__hht__too Laa__aa______e___________ 1ttpo1pppppprp..... .ddf
[57] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia-basse based image text dataset for multimodal multilingual machine learning. In Pro- ceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 2443-2449, New York, NY, USA, 2021. Association for Computing Machinery. URL. htttps:///d/t org/
/trg//1..o 10. 10111114444444 04445/4443355.. 3463257
[58] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. ArXiv, 2303.15388 2023. URL. 15389
15

---

[59] Kimi Team, Kimi-vl technical report. ArXiv, abs/2504.074911 2025. URL https://arxx org/abs/2504 07499
[60] Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. Proceedings the 2022 Conference on Empirical Methods in Natural Language Processing, pages 715-729, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https
/ //aclanthology . org/ 2022. emnlp- maip. 45/
[61] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 5238 5248, June 2022. URL LrrbbinngViiionnnneeeieeeeeioooe and Language 2a1pl22pppperppppppp
thecvf com/
Winogrounddoooooeeeeoo Models_ Lirels_for__vaee httpapass.aaaaaannaaeetoooooooaa
[62] Jörg Tiedemann. News commentary v16. https://opus nlp1. comess-Commntaryy
corpus/versionn/eess- Commentarrm 2012. Accessed: May 16. 2024.
[63]
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multi- modal llms. Advances Neural Information Processing Systems, 37:87310-87356 cwwpaper.cpppppppepppp/ep.
2024. URL. https://prrocedd www.eeedddinnnnnnnnneeeeee.eeeee neurips. cc/ paper_ fil.fillsppaa pttle///ppprr/22///ieee @eea6f@ccceaboaaa1ccc111111eeeee orneraaeeerrrrtiieeeeeeooeooronr
Conference .df
[64] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient founda- tion language models. ArXiv, abs/2302.19711, 2023. URL httthtt/////xxiit www..iii.g/............oooo hatg/ab/2200 2302 13971
[65] Voyage AI. voyage-multimodall:: hamp:/addd-.tlll-l---ooeeeeeeeeo embbediin model for interleaved text, images, and screenshots. 2024.
Nov
[66] Haozhe Wang, Chao Qu, Zuming Huang. Wei Chu, Fangzhen Lin, and Wenhu Chen. VI- rethinker: Incentivizing self-reflection of iision-language models with reinforcement learning. 08837
ArXiv, hiss/2504.08837, 2025. URL htttps://aaaat //tp//axvvvvvv.////v//e///a.aavv
[67] Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. One-peace: Exploring one general representation model toward unlimited modalities. ArXiv, abs/2305. 2023. URL htttt:t//axiii org www.ooggg//////eoooooooo abs52305
11172
[68] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Con- ference Machine Learning, volume 119 of Proceedings of Machine Learning Research. pages 9929-9939. PMLR, 13-18 Jul 2020. URL httttt::/ppoceeeeeeeeee wang20k. html
mlr. hat.ppeess//1/
[69] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, pages 387-404. Springer, 2024. URL https 7/www.ecwl http://ppereeeeccooo////ppp.o..p
[70] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 1112-1122, New Orleans, Louisiana, June 2018 Association for Computational Linguisties.
.rgg 1101/
16

---

[71] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23965-23998, PMLR, 17-23 Jul 2022. URL htttps:/prooceedingg mlr. p.e..pressvvv.). wortsman22a.c 1
[72] Chenghao Xiao, Yizhi Li, G Hudson, Chenghua Lin, and Noura Al Moubayed. Length is a curse and blessing for document-level semantics. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1396- Singapore, December 2023. Association for Computational Linguistics. URL paattp:://allanthoooog)) hwwpaaatttooooooooo/22222.2
emn malllp-minn 86/
[73] Chenghao Xiao, Yang Long, and Noura AI Moubayed. On isotropy, contextualizatton and learning dynamics of contrastive- -bati sentence representation learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12266-12883. Toronto, Canada, org/
July 2023. Association for Computational Linguistics. URL
2023. f findings-acl. 7788/
[74] Chenghao Xiao, Zhuoxu Huang, Danlu Chen, G Thomas Hudson, Yizhi Li, Haoran Duan, Chenghua Lin, Jie Fu. Jungong Han, and Noura Al Moubayed. Pixel sentence representation learning. ArXiv, abs/2402.08 183, 2024. http:ttt///aiii...rrrtrer www..rrggaa/////aaa 08183
[75] Chenghao Xiao, G Thomas Hudson, and Noura Al Moubayed. Rar-b: Reasoning as retrieval benchmark. ArXiv, abs/2404.06347, 2024. URL https://arxiv: org/ /rwg//////b/444433333
[76] Chenghao Xiao, Isaac Chung, Imene Kerboua, Jamie Stirling, Xin Zhang, Márton Kardos, Roman Solomatin, Noura Al Moubayed, Kenneth Enevoldsen, and Niklas Muennighoff, Mieb: Massive image embedding benchmark. ArXiv, (in//5004.14411, 2025. URL https://arxiv 10471
[77] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5- omni technical report. ArXiv, 001225533.20215 2025. GRRpsttts:/// /ttp////xxvv.v/// wrrxiv.org/oe 20215
www.oogg///////ooooo.oooooe hts///b/////// 2503
[78] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages content_cvpp _ttp__cvrrr________ee https//appr////// MSR VTT VTV_A_TT_L__LLLL wwtpgeeecv___211111
2016. URL httptt://tpppaeeeeeeeeee
thecvf com/ pdf
[79] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 5288-5296, June 2016. URL 2016__aper_.__________ MSR- Lwa.VVTTAALLLL_LPLPL CVPR_ com/content_ccvp httppp/006//aappaaauuuaa
thecvf
[80] Shijia Yang, Bohan Zhai, Quanzeng You, Jianbo Yuan, Hongxia Yang, and Chenfeng Xu. Law of vision representation in milms. ArXiv, abs/2408.16357, 2024. URL htttttttt/////at org/ abs/2408. 16357
[81] Ruifeng Yuan, Chenghao Xiao, Sicong Leng, Jianyu Wang, Long Li, Weiwen Xu, Hou Pong Chan, Deli Zhao, Tingyang Xu, Zhongyu Wei, et al. Vl-cogito: Progressive curriculum reinforcement learning for advanced multimodal reasoning. ArXiv, abs/2507.226610 2025. URL https://arxiv.. .rg/ owg/abs////oooooooooooo..oo.e. 2507. 2550/ 22607
[82] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023. httphttsss//ppereeeeeee
17

---

[83] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-traiming.
Proceedings of the IEEE/CVF Interna- lional Conference on Computer Vision (ICCV), pages 11975-11986, October 2023. Siigiiididddd_
URL htttps://ppeacccess. thecff. comm///onttat//////0222///pppp/p Loss_ford fa___urrr_aaaaeee_.____________ Ima___ Pww.iniieenrrnnn... wtt.a/iiinnncce.22221aa..1.
pdf
[84] Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkum Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal llms. ArXiv, abs/2412.16855, 2024. URL Utthtpp///arxi org/ 2212. 16855
[85] Xin Zhang, Ziqi Dai, Yongqi Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, Jun Yu, Wenjie Li, and Min Zhang. Towards text-image interleaved retrieval. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 4254 4269, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 179-8-899176-2514 doi: 10. 10.188655/11/2222..cl URL https:://acllanhhology org 2025. aclalllllnnn..e .ong.214/
[86) Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang. Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. ArXiv, abs/2506.05176. 2025. BRL https: //axxiv org/ abs/2506.e. 211.//556600006666
[87] Yaowei Zheng. Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400-410, Bangkok, Thailand, August 2024. Association for Computational Linguistics. acl 38/
URL htl..ttpp://allaatiooooooooo////
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Yes, the abstract and introduction clearly state claims and conclusion made in our paper. Specifically, propose language-centrie omnimodal representation learning framework, discover the Generation- theoretical justification for the GRSL.
Justification:
Scaling Law (GRSL), and provide
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the limitations of the work in the Limitation section of the Ap- pendix.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provide the full set of assumptions and complete proof in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose all the information needed to reproduce the main experimen- tal results in the experiment settings sections.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We provide source code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //ips.cc/ppblic. wwwpppppc/gguueieecceceeeo.ocepe /ttpg//dddd//oeSuumiiiieeeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We disclose all the detail that is necessary to appreciate the results in the experiment settings sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Results of embedding evaluation are not affected by random seed. And due the large scale of trained models and compute requirements for MLLM training and evaluation, we were not able to perform each of our 20+ training configurations multiple times and evaluate their checkpoints' performance across random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide the experiments compute resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The paper conform the NeurIPS Code of Ethics in every repsect. We preserve anonymity and the datasets that we used are open-sourced datasets that do not have privacy issues.
The paper conform the NeurIPS Code of Ethies every repsect. We preserve anonymity and the datasets that we used are open-sourced datasets that do not have privacy issues.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive socictal impacts of our work in the Appendix.
22

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This method proposed in this paper is embedding model. It will not generate harmful contents.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited the original paper that produced the code package dataset, as well as indicating the dataset version we used.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
23

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the balaaets.ppaerssttt paperssii thcode. cow con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We describe our dataset construction methods Section 3.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Yes, the usage of LLMs in our method is descrihed in Section 3.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
25

---

A Limitations
In this work, we have studied the scaling law between generative capabilities of pretrained MLLMs. their latent multimodal alignment, and their representational capabilities after contrastive learning. We use MLLMs that have gone through generative pretraining and those that have attained different levels of generative capabilities, and let them go through lightweight contrastive learning. During contrastive learning, model weights are minimally adjusted, through low-rank adaptation, to project the original knowledge space into an embedding space suitable for similarity matching. However, we do note that one can also jointly train generative loss and contrastive loss [43 46] to maintain a model's 'ss knowledge (through continual generative training), and enhance its representational power (through continual contrastive learning). Due to the high computational cost of this approach, we leave it as promising direction for future work in the context of omnimodal representation learning.
B Details of Additional Multimodal Data
On top of the main text-only setting, the alI-NLI training corpus-which plays a crucial role in unlocking the model's representational capacity we further construct approximately 94k multi- modal training samples to align the embedding space with the downstream multimodal task space. Specifically, we include: (1) Visual Document. Unlike most prior studies, intentionally construct only about 23k triplets from Colpali [19] and Docmatix (35), rather than performing exhaustive data exposure. We found that large-scale visual document data, when not balanced with text and other task datasets, can degrade overall task generalization. (2) Retrieval and Compositionality: We include only 3k triplets from MS-COCO, aiming to introduce basic image- text alignment. To enhance robustness to varying input lengths, we apply augmentation techniques from LA(SER)* [72). Interest- ingly, this not only improves length robustness but also enhances the model's spatial perception and image-text text compositional reasoning. (3) Multilingua/Diversee Text Data. To enhance linguistic and contextual diversity, we sample several thousand examples from our Scale- 1M dataset introduced in the main paper. (4) General Synthetic Data. We further construct around 60k synthetic samples in diverse formats to maintain and reinforce the model's instruction-follovinn and interleaved alignment capabilities wwich is important for tasks like VQA under the Reasoning-ass -estriiaal paradigm. The diverse synthetic data also benefits classification tasks, improving both probing and zero-shot performance.
C Details of MIEB-Lite Benchmark
The MIEB-Lit benchmark comprises 51 tasks in 8 categories, where the details of each category are summarized as follows:
Visual STS [74]: It conceptualizes traditional semantic textual similarity (STS) vision task by rendering text as images and evaluating the semantic understanding of visual encoders. Similarity scores computed from the embeddings of image-text pairs and compared against human annotations using Spearman correlation. This task comprises three subcategories: English (STS 13 and STS 15), cross-lingual (STS-17, with image pairs in different languages, e.g., Frabic-English), and multilingual (STS-b, with pairs in the same language, e.g., Italian- Visual STS naturally assesses model's interleaved encoding ability to capture semantic meaning from text in image form, with Spearman correlation as the primary evaluation metric.
Document Understanding/Viiet Visual Document Retrieval: MIEB-lite selects 6 tasks from the Vidore benchmark 19), which is to retrieve visual documents that contain information to solve the problem in the query. This task assesses model's 's ability to understand the complex layouts and textual information in visual documents, and the interleaved image-text alignment. Here we use nDCG @5 as the evaluation metric.
Image Linear Probing: MIEB-lite selects 8 challenging linear-probing datasets, including Coun- try211, DTD, EurOSAT, GTSRB, OxfordPets, PatchCamelyon, RESISC45, and SUN397, which MLLMs typically struggle compared to CLIP-style models, as indicated by the MIEB benchmark leaderboard. We follow Xiao et al. (76) to adopt 16-shot linear probing, which closely preserves ranking compared full-dataset probing, and report accuracy as the metric.
Compositionality Evaluation: evaluates fine-ge fine-grained alignment of image-text features, requiring retrieving the groundtruth texts corresponding to the correct composition of all elements, e.g., an
26

---

accurate fine-grained caption of an image, and vice versa for images given texts. This category includes BRO-Benchmark 82 and Winoground [6]]. Here we use accuracy as the evaluation metric.
Given interleaved input composed of question conditioned the task requires models to retrieve the correct answer under the reasoning-ase as as-retrieval paradigm 771. This task category is mostly made of tasks assessing vision-centric capabilities [63]. such as spatial relation perception, depth estimation, and relative distance. Here we use accuracy as the evaluation metric.
Vision-centric QA:
image,
MIEB-Lite adopts 11 retrieval tasks, consisting of image-only retrieval, image-text retrieval, and interleaved retrieval, providing comprehensive assessment of models' modality- specific and composed encoding capabiliities In addition, it also selects WIT datasets [57] and XM3600 [60], totally covering image retrieval tasks across 38 different languages, i.e. multilingual image retrieval, to assess a model's alignment capability between image and multilingual text embeddings, using nDCG@ @ 10 as the primary metric.
Retrieval:
Zero-shot Classification: Zero-shot Classification assesses classification in mmmilarity-matching fashion. We use text prompts like "an image of a (label)" following common practices Radford al. (50) and Xiao al. (76). MIEB-lite selects challenging fine-grained zero-shot classification tasks, including CIFAR 100, Country21 1, FER2013. 3, FGVCAircraft, Food101, OxfordPets, and StanfordCars. Here we use accuracy as the evaluation metric.
Clustering: Clustering provides an extra lens to inspect the clustered structure of embeddings. MIEB-lite adopts two clustering tasks, including fine-graing tasks such as Imagenet-Dogl5 [14] which MLLM-based embedding models typically fail compared to CLIP-style models [76). The Normalized Mutual Information (NMI) is utilized as the main evaluation metric.
D Details of MIEB-Sub18 Benchmark
We further select a smaller-scale subset than MIEB-lite, including 18 tasks from MIEB Xiao et al. (76 as MIEB-Sub18, which comprises 47 subtasks that are considered most challenging to the image-text embedding models, particularly in evaluating the capabilities of visual text representation, multilingual understanding, and interleaved encodings. Specifically, we focus Visual STS (74), multilingual image retrieval [57]. and document understanding from Vidore [19]. Additionally, we assess three image linear probing tasks where MLLM embeddings underperform relative CLIP and self-supervised vision models, as reported on the MIEB leaderboard 47] All evaluations are conducted using the official MIEB codebase 766).
Visual STS [44 It conceptualizes traditional semantic textual similarity (STS) vision task by rendering text as images and evaluating the semantic understanding of visual encoders. Similarity scores are computed from the embeddings of image-text pairs and compared against human annotations using Spearman correlation. This task comprises three subcategories: English (STS- 12~~16,, cross- minss-lin (STS-17, with image pairs in different languages, e.g., Arabic-English), and multilingual (STS-b, with pairs in the same language, e.g., Italian Italian) Visual STS naturally assesses model's interleaved encoding ability to capture semantic meaning from text in image form, with Spearman correlation as the primary evaluation metric.
Multilingual Image Retrieval: We utilize the WIT datasets (57] and select its image retrieval subtasks across 11 different languages. This task accesses model's '''s aligmment capability between image and multilingual text embeddings with nDCG@ 10 the main metric.
Document Understanding: We select tasks from the Vidore benchmark [9), which is to retrieve visual documents that contain information to solve the problem in the query. This task assesses a model's ability handle the complex layouts in visual documents and the interleaved image-text alignment. Here we use nDCG@* @ as the evaluation metric.
Image Linear Probing: We evaluate three linear probing Locks-Stanford Cars [3]. BirdSnap [3] and Country211 [50] which MLLMs struggle the most, as indicated by the MIEB benchmark leaderboard. We follow Xiao et al. [76] to adopt 16-shot linear probing, which closely preserves ranking compared to full-dataset probing, and report accuracy as the metric.
27

---

90
11.00
VinelSTS (Mallllged))
Dvcunear Underiindia
Maltiagul Iragge Bariial
Spet Cowsless
4DCO
Aah
CC0
520
Figure 6: Comparison between the text-only variants of LCO-EMB with advanced open-source (E5-V 30 and proprietary (Voyage Multimodal 3 655 embedding models on MIEB-Sub18.
Table 3: Comparison of different LoRA ranks and alpha values using Qwen2.5 VL-7B as the backbone. *y 256, 512 setting experiences unrecoverable loss spikes in training.
Rank (r)
Alpha (a)
Comp.
VC-
Multiling. Img. Rtr
V-STS
V-STS (cross)
V-STS (multi)
Doc. Und.
Linear Probe
Average
QA
(eng.)
16
48.35
58.08
56.64
85.05
85,30
83.48
67.49
53,91
67.29
64
16
55.64
60.62
55.62
84.60
85.16
83.40
65.76
52.44
67.90
64
128
43.40
51.86
58.93
84.98
84.44
83.39
67.66
57.24
66.49
256
16
52.29
57.30
57.49
84.88
85.68
83.61
66.95
53.36
67.70
256
128
43.07
57.56
56.32
85.89
84,82
83.98
67.33
55.51
66.81
256
512
85.52*
39.24
0,70*
5.90*
12,90
7,80*
0,90*
1,50*
19.31*
E Impact of LoRA Hyperparameters for LCO-EMB
Our approach, LCO-EMB, employs LoRA fine-tuning for lightweight contrastive learning, which aims to refine MILM representations while minimally perturbing the model's 's intrinsic knowledge and abilities, thereby effectively preserving its inherent cross-modal alignment capability. We encapsulate this benefit as the "learn less, forget less" characteristic of LoRA. Here, we further analyze the impact of two critical LoRA hyperparameters- rank (r) and a-on the performance of LCO-EMB.
In LoRA, rank (r) and alpha (a) jointly control the capacity for new knowledge integration and the extent to which it modulates existing knowledge. The rank defines the dimensionality of the trainable weight matrices used to approximate the original model's weight updates; a higher rank thus increases the capacity for injecting new knowledge. Conversely, alpha scales the contribution of these matrices the overall model weights, meaning a larger alpha amplifies the extent to which this new knowledge is infused into the model.
Table presents the results of LCO-EMB under different values of rank and alpha. We observe distinct patterns for different task categories, and there doesn't exist global optimal setting of LoRA hyperparameters. For instance, LoRA hyperparameters bring minimal variations to tasks optimized by the training (e.g.. V-STS, whose textual counterpart STS is deemed directly optimized by All-NLI in text embedding literature, invariant LoRA hyperparameters) The optimal performance for multilingual retrieval, document understanding, and image linear probe generally occurs when alpha scaled up appropriately to rank r, such as r 8, a 16 and r 64, a 128. However, we notice that for tasks whose capabilities assessed largely differ from those which the training set optimizes, e.g., compositionality and vision-centric QA, larger alpha a generally brings significant performance degradation, showing the importance of the preservation of the base model's knowledge for generalization OOD tasks. We observe that with rank 256 and alpha 512, models experience unrecoverable loss spikes in training.
28

---

Table 4: Exploring the impact of training dataset utilization and model ensemble on LCO-EMB, where LCO- BMMB-En denotes the ensemble model produced by applying the model soup [ technique to L.OO-MBB variants fine-tuned on alI-NI.I and Scale- M..
v-STS
v-STS (cross)
v-STS (multi)
Doc. Und.
Multi. Img. Rtr.
Model
Data Source
Linear Prob.
Avg.
(Eng.)
LCO-EMB
all-NLI
51.86 58.61
84.69
85,23
83,48
65,36
56.37
71.17
LCO-EMB
Scale-IM IM
81.27
81.42
78.42
62.12
64.81
71.11
LCO-EMB-- EMB-EMBBEE
55.69
83.79
84.88
82.82
63.15
62.67
72.17
We acknowledge that optimal rank and alpha likely exist for models of each size, striking balance between introducing new knowledge and the extent to which it modifies pretrained model weights. We leave more comprehensive empirical analysis and theoretical study quantify this relationship for future work.
Impact of Different Training Datasets and Model Merging for LCO-EMB
Recognizing that language models fine-tuned on different datasets often demonstrate distinct strengths, we independently fine-tune ACO-EMB, using Drren-255-VL-Instie as the backbone model, on all- NLI and Scale-IM IM via contrastive learning. then assess the performance of each variant in isolation. Subsequently, we investigate the effect of model ensembling by applying the model soup [71] technique, which merges the parameters of multiple separately fine-tuned models by averaging their weights. The results, presented in Table provide the following three key insights:
Performance of alI-NLI fine-tuned variant. LCO-EMB trained by alI-NLI excels in Visual STS and Document Understanding, indicating that NLI supervision sharpens not only textual similarity perception but also generalizes to improve their ability to preserve vision-text semantic similarity.
Performance of Scale-1M fine-tuned variant. LCO-EMB adapted by Scale- 1M leads Linear Probing and Multilingual Image Retrieval tasks. Since Scale-1M supplies semantically rich descrip- tions of real-world scenes, LCO-EMB fine-tuned on this corpus appears to emulate image-caption pre-training, thereby activating image representations without explicit visual data.
Performance of model ensemble. The LCO-EMB-Ein through merging the LCO-EMB variants trained by all-NLI and Scale- 1M, achieves the best overall performance, demonstrating that the model ensemble strategy effectively integrates the complementary strengths of each checkpoint.
G Preliminaries of Theoretical Analysis
G.1 Definitions
Definition 1 (Population and Empirical Risk), Let D be the data distribution. The population contrastive risk for model 0 is its true expected InfoNCE loss:
CP (0) = Ex (xx.))>D/(intovE//X (XX Y;
(4)
Given a training set S = ((x,, 1 of size n, the empirical contrastive risk defined as:
Cemp (0)
ListoNCE Y):
(5)
n
Definition 2 (Generative Quality of the Prior). Let P be the prior distribution over the parameters of a pre-trained autoregressive generative model. In the common case where P = 68, is a point mass of the model parameters at pretrained checkpoint dg, define its generative quality via the mutual information captured by fb:
Ip(X:Y) = Is, (X;Y).
(6)
Under the standard approximation that the generative cross cros--enroopy loss (P) estimates the condi- tional entropy (ref. Appendix!! D. we have the following approximation:
PP HYY Ip(X;Y)
Ip(X;Y) H(Y) CP)
(7)
29

---

Here, HY) is the Shannon entropy of the target data distribution, which quantifies the inherent diversity and complexity the target modality (e.g., text). Thus, for a fixed dataset, a lower generative loss Lg P) corresponds to higher mutual information and therefore a higher generative quality.
G.2 Justification for Hypothesis !
A high [p(X; Y) implies that the representation of the generative model before contrastive finetuning. fr(X), , already predictive of Y. This pre-existing alignment means positive pairs are closer in representation space, enabling contrastive optimization to reach a lower empirical loss. The term log - (xX;; Y) is the theoretical lower bound on InfoNCE loss under ideal conditions. This hypothesis aligns with our empirical findings in Section 5.1 which show that stronger pretrained generative models yield better representations for downstream tasks, and is consistent with recent literature decoder-basee embedding models (e.g.. (66).
H Proof of Theorem
Proof. We begin with the standard PAC-Bayesian generalization bound, which holds with probability at least 1 - 6 for any posterior Q:
C Comp (0) + KL(Q + log(1/8)
Eg Ex [CPOP Ea-Q
(8)
2n
According to Hypothesis
1 the empirical risk is bounded as:
(+) S log N Ip(X:Y) + cp.
E
(9)
Substituting 9 into
yields Theorem
I Relationship Between Generative Loss and Conditional Entropy
Definition (Generative Quality of the Prior). Let P be the prior distribution over the parameters of a pre-trained autoregressive generative model, centered at 0g. We define its generative quality via the mutual information [PX; == La (X;Y) that its representations capture between the input X and the target output Y.
The mutual information is defined as Ip(X;Y)= = H(Y) - HY| While the true conditional entropy H XY is unknown, it can be estimated by the model's generative cross-entropy loss, (P). The formal relationship is:
(P H( + (Paaaa (YJ)) || pa, (Y)X)),
(10)
where Pa, is the model's predictive distribution. For well-trained MLLM, the goal of minimiz- ing generative loss to minimize this KL divergence. Thus, for a strong prior, we can use the approximation H(Y z (P). Substituting this into the definition of mutual information yields:
1p(X;Y) (YY) (P)
(11)
Here, H() is the entropy of the target data, which constant for a given dataset. Therefore, lower generative loss (P) directly corresponds to higher mutual information and thus a higher generative quality of the prior.
Improving Representation Bounds via Enhancing Generative Capability
To further investigate the hypothesis that enhancing an MLLM's generative ability improves its representations, a key aspect of the EmmerationnRepreeentaiio Scaling Law, we introduce a challeng- ing cross-lingual multimodal document retrieval task, SeaDoc. This task enables comprehensive evaluation of MI.M's representational capacity. In this task, an English query used retrieve corresponding multimodal document page in low-resource target language.
30

---

J.1
Data Curation
SeaDoc is cross-lingual visual document retrieval benchmark specifically designed for low-resource SouthEast Asian (SEA) languages. While building upon foundational concepts from existing visual document understanding benchmarks like ViDoRe [9], SeaDoc uniquely challenges MLLMs' visual document understanding capabilities on non-English languages at unprecedented scale.
Specifically, we utilize Gemini-2.5-Flash
to annotate each PDF page by sequentially applying OCR, translating the content into English, and generating an English query answerable exclusively from that specific page. This results in 5, 055 annotated (OCR, English translation, English query) triplets. To construct a high-quality query pool for the retrieval dataset in SeaDoc, we implement stree-stage quality control process:
Qwen2.5--B-Instruct first used to filter out functional pages (e.g., title pages, author pages, tables of contents), which reduces the dataset to 4, 491 content page annotations.
2. The same model then scores these annotations for Quality and Groundedness Only questions with a quality score of at least 9 and a groundedness score of 10 are retained. Note that Quality measures the informativeness of the content and relevance of the query, and Groundedness measures the exclusivity of the answer to the page.
10-point scale.
3. Our in-house linguists conduct final review of the remaining triplets to ensure their quality.
As a result, we derive 1, 001 high-quality queries to be used for retrieval tasks within the 5, 055 page corpus.
For conducting additional OCR-intensive generative training, we construct a training set leveraging images that do not correspond retrieval test set queries, resulting in 4k seed images. We construct SFT tasks per image: 1) OCR the image. 2) OCR the image, then generate a question from the image. 3) Provide the English translation given the OCR'd dd text. Provide the English translation directly from the image. 5) Provide the answer to the generated query. Note that compared to the SeaDoc test set, the training set is separately generated and includes additional "provide answer to the generated question" part in the seed prompt. This process leads us to an around 20k training set to enhance targeted generative capability on low-resource visual documents, which also explore combining with the PixmoCap dataset (7 (710k) for general capability preservation in the main experiments.
J.2 Experimental Settings
We use DWw-2.5-VL-3B as the backbone and establish a baseline with lightweight contrastive learning. To assess whether enhanced generative ability benefits embedding quality, we further train variant with additional generative pre- training before lightweight contrastive learning.
38
Baseline (high-resotution)
R-DCG@10
32-
We apply supervised fine-tuning to enhance the model's image-to-text generative capabil- ity. This stage utilizes a mixture of image-to- text training data, comprising OCR data in SEA languages (derived from the training split of SeaDoc) and generul-domain image caption- ing data, i.e., PixmoCaps [13]. The OCR data strengthens its capability in generating SEA lan- guages from visual documents, while the inclu- sion of general image captioning data helps pre- serve its semantic alignment between image and text modalities in the general domain.
30-
Continull Gonerative Training
28
Baseline
26
Baseline (high-resolution)
24
Baseline
SeaDac
SeaDoc (11)
SeaDaciPC SeaDoc ICC (CH
Figure 7: Retrieval performance of Qwen2.5-Vi 3B fine-tuned on various continual generative fine- tuning strategies before CL on SeaDoe benchmark, where "PC" represents PixmoCaps and "H" de- notes high-resolution. The results suggest that en- hancing the generative ability of MLLMs before CL can enhance their embedding capability.
Given that text in multimodal documents can be small, requiring higher image resolution for MIL.Ms to accurately read textual content, we further employ two settings-hig high- and low-resolution-tus assess the impact of image resolution. For the low-resolution setting, we follow standard practice by using maximum of 262, 144 pixels [87].
31

---

For the high-resolution setting, we use a 10x larger maximum 2f 621, 440 pixels. Here, we adopt nDCG@ @ 10 the primary metric.
J.3 Experimental Results
Figure 7 summarizes the retrieval performance of the same backbone model fine-tuned using different continual SFT strategies before the same CL tuning process on SeaDoc benchmark We draw the following key observations:
(1) When SFT training is conducted exclusively
-CRiintensive data, i.e., SeaDoc-train at lower resolution, the model experiences a significant capability collapse compared to the baseline (Qwen2.5-VL.:3) after lightweight CL... This SFT-induced degradation aligns with observations in recent multimodal reasoning research [7 [5] 28
666, 81. Since foundation models have 36
already undergone extensive SFT and RL, continual SFT can lead overfitting and degrade models' generalization capability.
(2) Training on SeaDoc with higher resolution partly mitigates this collapse. This is because the text visual documents is typically small; training with higher resolution allows for better grounding of the generated output in the visual text of the source image, opposed to overfitting to example-level visual cues.
(3) Incorporating PixmoCaps captions into the training set further boosts visual document retrieval performance post-CL.. This because general-domain image captioning data helps preserve the latent image-text alignment learned by MLLMs during pre-training. This preserved alignment is crucial for its effective exploitation by the subsequent text-only contrastive finetuning process.
Unless otherwise specified, evaluate model performance at the maximum resolution used during training.
32


Reviews:
[{"review_id": "SBsuBKroqg", "reviewer": "Reviewer_iKcM", "summary": "This paper aims to explore the latent alignment mechanism of MLLM and proposes a GRSL to address this issue. The extensive results demonstrate that the performance surpasses that of existing Open SOTA and API-based methods.", "questions": "I have some high-level concerns, and I would be willing to raise my score if the authors can address them.", "limitations": "Please refer to the weakness.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. From the identification of the problem, to the design of the empirical experiments, and finally to the proposal of the solution, the entire process is coherent and flows very well.\n2. The theoretical analysis in the Appendix provides a solid foundation for the paper.\n\nWeaknesses:\n\n1. The terms \"Open SOTA\" and \"API\" are a bit confusing. Could you replace them with specific works? This would make the comparison clearer and more appropriate.\n2. Despite the extensive analysis of the problem, Figure 3 still does not make it clear to me what core issue this paper actually solves.\n3. The paper lacks case studies beyond quantitative metrics that would help in understanding the benefits of the proposed method. Could you please add some of these details?", "was_revised": true, "final_justification": "This paper explores how text-only contrastive learning enhances the quality of omni-modal representations and gives solid experimental results. After reading the rebuttal of other reviewers, I tend to accept.", "rebuttal": "Thank you for your constructive comments. We address your high-level concerns as follows:\n\n**Open SOTA and API SOTA**: Thanks for the suggestion. \"Open SOTA\" and \"API SOTA\" refer to E5-V and Voyage-multimodal-3, respectively, as cited in line 222 and the caption of Figure 4. To improve clarity, we will also update the legend in Figure 4 to display the actual model names.\n\n**What Figure 3 expresses**: Figure 3 illustrates our proposed method. This figure is intended to outline how text-only contrastive learning enhances the quality of omni-modal representations. Specifically, text-only contrastive learning can effectively mitigate anisotropy in non-textual modalities, even in the absence of explicit multimodal training. We conduct empirical studies in Section 2.1 to demonstrate this effect. As shown in Figure 1, text-only contrastive learning reduces the expected cosine similarity of random text-image, text-audio, and text-video pairs from over 0.8 to around 0.5. This indicates that anisotropy is reduced and the embedding space becomes more discriminative.\n\n**Case studies**:\nWe find that conducting text-only contrastive learning significantly improves the model’s fine-grained cross-modal alignment, enabling it to better distinguish between hard-negative texts. For instance, the models perform much better on compositionality evaluation tasks—such as examples from the ARO benchmark—where the goal is to match the correct, fine-grained caption to each image rather than its hard negative.\nAs an example, consider a picture of a man sitting to the right of a door, with the following two fine-grained descriptions:\n- \"The door is to the left of the shirt\" (correct)\n- \"The shirt is to the left of the door\" (incorrect)\n\nPreviously, the model would assign a higher cosine similarity to the incorrect caption. After applying text-only contrastive learning, however, it correctly assigns higher similarity to the accurate caption.\nWe appreciate your suggestion and will include such case studies in the paper!"}, {"review_id": "uOohYoXEKX", "reviewer": "Reviewer_oULY", "summary": "The paper introduces a language-centric omni-modal representation learning (LC-OMRL) approach that treats a multimodal LLM’s language decoder as the sole trainable component. After freezing all modality encoders and the projector, the decoder is fine-tuned on text-only contrastive triplets (AllNLI or Scale-1M) using LoRA adapters. The authors argue that generative pre-training of the decoder has already induced a latent cross-modal alignment; the lightweight contrastive stage simply “unlocks” this alignment for similarity matching.", "questions": "* Could you add native Qwen-VL/Omni embeddings and other alignment approaches (e.g., CLIP-style contrastive learning) to Figure 4 so that readers can judge absolute and relative gains?\n* Have you measured pre- vs post-contrastive cross-modal retrieval (image→text, text→audio, etc.) to directly quantify alignment improvement? If so, please share those numbers; if not, why was this omitted?\n* Please provide the t-SNE visualization results for the embedding space.\n* What empirical evidence supports choosing LoRA over adding a lightweight projection head or full-decoder tuning?", "limitations": "Please refer to the weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n* Updating only the decoder with text contrastive loss is computationally cheap (2 epochs; LoRA rank 64) while yielding SOTA embeddings on a broad benchmark suite.\n* Improvements on image, audio, and video retrieval without seeing any non-text signals during fine-tuning support the latent-alignment hypothesis.\n\n# Weaknesses\n* The lack of comparisons with native Qwen-VL/Omni embeddings and well-known cross-modal alignment methods (e.g., CLIP-style or cross-modal contrastive learning). Without those references, the superiority of LC-OMRL is less convincing.\n* Existing work typically reports cross-modal retrieval (R@k) before/after alignment to quantify improvements. While the paper includes multilingual image retrieval, it does not contrast pre- vs post-training performance, nor does it evaluate retrieval on other modalities (audio, video) explicitly for alignment.\n* The “collapsed vs isotropic” embedding spaces are merely schematic; the axes are not defined, and no real data projection (e.g., t-SNE) is given. Readers cannot verify that clusters truly spread out after training.\n* The text states two advantages of LoRA, but does not compare with classic alternatives such as adding a shallow projection head or fully fine-tuning the decoder; quantitative ablations are confined to LoRA hyper-parameters.", "was_revised": true, "final_justification": "The authors have addressed my questions during the rebuttal, and I changed my evaluation to positive.", "rebuttal": "Thank you for your valuable feedback. We address your comments point by point as follows:\n* ### **Comparison with native Qwen-VL/Omni embeddings, and well-known cross-modal alignment methods**:\n\nWe present a comparison of results between the native Qwen2.5-VL embeddings, a CLIP-style training baseline, and our text-only CL method on vision-and-language embedding tasks in the Table below. For the CLIP-style baseline, we fine-tune the native Qwen2.5-VL-3b model using 800k image-caption pairs from the PixmoCaps dataset with contrastive learning. The results of our text-only contrastive learning method, which is trained on the all-NLI dataset (270k pairs), are also reported.\nExperiment results show that our text-only method consistently improves the performance of the native Qwen embeddings across all tasks. Moreover, our method outperforms the CLIP-style baseline, while requiring only about 1/50th of the training time.\n\n| Size | Model Type | Training Time | Multilingual Retrieval | Visual STS (En) | Visual STS (Cross) | Visual STS (Multi) | Document | Linear Probe | Average |\n|------|--------------------------------|---------------|------------------------|-----------------|--------------------|--------------------|----------|--------------|---------|\n| 3b | Qwen-VL-3b (vanilla) | - | 31.73 | 73.82 | 59.03 | 68.57 | 28.82 | 46.96 | 51.49 |\n| 3b | Qwen-VL-3b (after CLIP-style CL) | ~50 hours | _25.15_ | _72.51_ | _67.45_ | _65.22_ | _48.91_ | _41.05_ | _53.38_ |\n| **3b** |**Qwen-VL-3b (after text-only CL)** |**~1 hour** |**51.61** |**81.88** |**74.97** |**78.30** |**57.90** |**53.05** |**66.28** |\n\n\n\nThe following table also presents the comparison results of Qwen2.5-Omni embedding model before and after finetuned by our text-only CL method on vision-and-language embedding tasks. We will include the results of the CLIP-style baseline in the final version. \n| Model | Multilingual Retrieval | Visual STS (En) | Visual STS (Cross) | Visual STS (Multi) | Document Understanding | Linear Probe | Average |\n|------|------------------------|-----------------|--------------------|--------------------|----------|--------------|---------|\n| Qwen-Omni-3b (vanilla)       | 42.08                  | 71.98           | 68.35              | 71.01              | 20.33    | 50.27        | 54.00   |\n| **Qwen-Omni-3b (after text-only CL)** |  **51.99**                  | **83.90**           | **78.59**              | **80.97**              | **54.82**    | **55.21**        | **67.58**   |\n\nThe following tables present the performance of Qwen-VL and Qwen-Omni before and after our text-only CL method on video retrieval and audio retrieval tasks. \nFor the results of video retrieval, we report the average of the t2v and v2t recall@5 scores on the msrvtt and ActivityNet datasets. For the results of audio retrieval, we report the average of the t2a and a2t recall@5 scores on the Clotho and AudioCaps datasets.\n\n| Model                        | Video Retrieval |\n|------------------------------|-----------------|\n| Qwen-VL-3b (vanilla)         | 47.83           |\n| **Qwen-VL-3b (after text-only CL)**  | **66.77**           |\n| Qwen-VL-7b (vanilla)         | 50.76           |\n| **Qwen-VL-7b (after text-only CL)**  | **66.77**          |\n\n| Model                        | Audio Retrieval |\n|------------------------------|-----------------|\n| Qwen-Omni-3b (vanilla)       | 35.19           |\n| **Qwen-Omni-3b (after text-only CL)** | **47.40**           |\n| Qwen-Omni-7b (vanilla)       | 39.84           |\n| **Qwen-Omni-7b (after text-only CL)** | **49.63**           |\n\n\n* ### **Visualizing collapsed vs isotropic space** \nWe have created visualizations using t-SNE and UMAP data projections. Unfortunately, PDF uploads and link sharing are not permitted in the rebuttal. We will include these visualizations in the camera-ready version given the chance. \n\nWe would also like to highlight that we have provided quantitative results on the degree of anisotropy in Section 2.1. Measuring the expected cosine similarity between random samples is a validated method for approximating anisotropy degree, with lower values indicating higher isotropy [1]. Our results demonstrate that our text-only contrastive learning method effectively mitigates anisotropy, thereby enhancing the isotropy of the embedding space.\n\n* ### **LoRA training vs other methods (e.g., Projection Layer or Full-Finetuning)**\nWe compare LoRA with a full-finetuning method and a shallow projection baseline that only adds a linear projection layer after the output layer. The results of Qwen2.5-VL-3b are presented in the following table. The shallow projection baseline introduces additional parameters to the model, but does not fully leverage the pretrained latent cross-modal alignment, resulting in only a modest average improvement over native embeddings. While full fine-tuning yields reasonable performance, it still lags significantly behind LoRA training. We posit that this is because the contrastive loss used during fine-tuning differs substantially from the pretraining loss (next word prediction) of MLLM, causing full fine-tuning to introduce large perturbations to the model’s pretrained parameters. These perturbations are more likely to disrupt the model’s pre-established cross-modal alignment.\n\n| Size | Model Type | Multilingual Retrieval | Visual STS (En) | Visual STS (Cross) | Visual STS (Multi) | Document | Linear Probe | Average |\n|------|--------------------------------|---------------|------------------------|-----------------|--------------------|--------------------|----------|--------------|\n| 3b | Vanilla | 31.73 | 73.82 | 59.03 | 68.57 | 28.82 | 46.96 | 51.49 |\n| 3b | Text-only (Linear Projection) |  31.31 | 75.25 | 62.95 | 69.32 | 28.12 | 49.19 | 52.69 |\n| 3b |Text-only (Full-Finetuning) |  _44.61_ | _81.65_ | _68.67_  | _77.75_ | _49.71_ | _50.21_ | _62.10_ |\n| **3b** |**Text-only (Lora)** | **51.61** |**81.88** |**74.97** |**78.30** |**57.90** |**53.05** |**66.28** |\n\n\nFurthermore, full-finetuning is much more compute-intensive for contrastive-learning given the quadratic GPU RAM requirements of contrastive learning loss over batch size. e.g., it would take 16 GPUs of 96G RAM to perform the same full-finetuning experiment on 7B models.\n\n[1] Ethayarajh 2019, How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings"}, {"review_id": "zE5D6bfqkz", "reviewer": "Reviewer_GWW4", "summary": "The paper investigates why multimodal large language models (MLLMs) fine-tuned with contrastive learning (CL) outperform traditional CLIP-style models. It argues that MLLMs benefit from implicit cross-modal alignment gained during generative pretraining. Text-only CL can then serve as a lightweight refinement step that enhances this latent alignment across modalities, eliminating the need for large-scale multimodal pairings during fine-tuning.", "questions": "Please answer the questions mentioned in the Weaknesses section.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1- The paper addresses an important topic, i.e., understanding why MLLM-based approaches are outperforming CLIP-like models.\n\n2- Providing analysis in Section 2 through the use of geometric metrics, such as anisotropy degree and kernel-level similarity.\n\n\n**Weaknesses:**\n\n1. The paper claims that \"a crucial advantage of MLLM-based approaches stems from implicit cross-modal alignment achieved during generative pretraining.\" However, the analyses and experiments presented seem to indicate that text-only contrastive learning plays a significant role in boosting alignment between modalities and enhancing the textual and non-textual representations. This appears to be different from the core claim.\n2. It’s unclear whether the experiments and analyses support the conclusion that text-only contrastive learning enhances both textual and non-textual representations. From the presented results, improvements seem largely attributable to better textual representations. This may, in part, be due to addressing issues such as modality forgetting in MLLMs. Clarifying whether non-textual features also improve, and providing supporting evidence, would strengthen the paper's claims.\n\n3. Ambiguities in Section 2.1 – Anisotropy Analysis:\n- The experimental setup for the anisotropy analysis is not fully detailed. For instance, how many samples were used to compute the anisotropy? Without this, we don’t know that the approximation is fairly accurate for the degree of anisotropy.\nWhat exactly does the layer index in Figure 1 refer to, and why does similarity decrease with increasing layer depth?\n\n\n- The Qwen2.5-Omni model shows a high degree of anisotropy collapse, yet still performs well on benchmarks. How is this explained?\n\n\n- One of the datasets used, PixmoCap, includes long textual descriptions. Could this influence similarity measurements (e.g., shared phrases inflating similarity)? And what is the effect on the fine-tuning with contrastive learning?\n\n\n4. Ambiguities in Section 2.2 – Kernel-level Similarity:\n- In Figure 2, the 3B model appears to have more layers than the 7B model. Why is this the case?\n\n\n- The difference between the original and contrastively fine-tuned models is relatively small in the figure. This raises the question: are improvements mainly due to better textual representations rather than improved multimodal alignment?\n\n\nPrior work has shown that MLLMs can forget text tasks post-pretraining. Could the improvements observed here simply reflect restored text capabilities, rather than enhancements to multi-modal representations?\n\n\n5. Figure 4 lacks baseline performance metrics. If Qwen2.5-VL and Qwen2.5-Omni were fine-tuned, it's essential to show their original (pre-fine-tuning) performance for comparison. This would help quantify the effectiveness of the fine-tuning process and the contribution of the proposed method.\n\n**Minor Weaknesses:**\n\nThe citation for llama at line 97 is missing", "was_revised": true, "final_justification": "Since my concerns have been addressed, I updated my score accordingly.", "rebuttal": "Thanks for your detailed comments. We aim to address your concerns as follows:\n### **1. Clarifying Core Claim** \nWe clarify the logic between our experiment findings and the core claim as follows:\n\nOur experiments demonstrate that applying text-only contrastive learning to MLLMs not only improves the isotropy of the embedding space in the text modality, but **also generalizes to enhance isotropy in non-textual modalities—even without finetuning on multi-modal data**. The successful transfer of these improvements across modalities provides empirical evidence that MLLMs inherently maintain geometrically aligned latent spaces among different modalities. We will emphasize this finding in the revised version.\n\nWe have provided quantitative results on the degree of anisotropy in Section 2.1. The degree of anisotropy is a standard metric for approximating isotropy, with lower values indicating higher isotropy. In lines 72 to 75, we mentioned that \"we examine whether this text-specific fine-tuning effectively mitigates anisotropy in non-textual modalities without requiring explicit multimodal training. Successful transfer of improvements across modalities would provide empirical evidence that MLLMs inherently maintain geometrically aligned latent spaces across different modalities.\"  \n\n### **2. Whether other modality features improve**\nWe provide extensive results on the performance of our method in embedding tasks across different modalities in Figure 4, 5 and Table 1, including both uni-modal tasks (**image-only**: linear probing and visual STS) and cross-modal tasks (**image-text**: document understanding, multilingual image retrieval; **audio-text**: audio retrieval; **video-text**: video retrieval). In the table below, we present the embedding performance of the MLLM before and after text-only contrastive finetuning to enable a direct comparison.\n\n| Model                        | Multilingual Retrieval | Visual STS (En) | Visual STS (Cross) | Visual STS (Multi) | Document | Linear Probe | Average |\n|------------------------------|------------------------|-----------------|--------------------|--------------------|----------|--------------|---------|\n| Qwen-VL-3b (vanilla)         | 31.73                  | 73.82           | 59.03              | 68.57              | 28.82    | 46.96        | 51.49   |\n| **Qwen-VL-3b (after text-only CL)**  | **51.61**                  | **81.88**           | **74.97**              | **78.30**              | **57.90**    | **53.05**        | **66.28**   |\n| Qwen-VL-7b (vanilla)         | 40.31                  | 72.77           | 65.25              | 70.96              | 29.01    | 51.77        | 55.01   |\n| **Qwen-VL-7b (after text-only CL)**  | **56.37**                  | **84.69**           | **85.23**              | **83.48**              | **65.36**   | **51.86**        | **71.17**   |\n| Qwen-Omni-3b (vanilla)       | 42.08                  | 71.98           | 68.35              | 71.01              | 20.33    | 50.27        | 54.00   |\n| **Qwen-Omni-3b (after text-only CL)** |  **51.99**                  | **83.90**           | **78.59**              | **80.97**              | **54.82**    | **55.21**        | **67.58**   |\n| Qwen-Omni-7b (vanilla)       | 42.62                  | 71.58           | 68.28              | 69.90              | 25.36    | 51.71        | 54.91   |\n| **Qwen-Omni-7b (after text-only CL)** | **53.36**                  | **84.55**           | **84.30**              | **83.13**              | **56.42**    | **52.71**        | **69.08**   |\n\n| Model                        | Video Retrieval |\n|------------------------------|-----------------|\n| Qwen-VL-3b (vanilla)         | 47.83           |\n| **Qwen-VL-3b (after text-only CL)**  | **66.77**           |\n| Qwen-VL-7b (vanilla)         | 50.76           |\n| **Qwen-VL-7b (after text-only CL)**  | **66.77**          |\n\n| Model                        | Audio Retrieval |\n|------------------------------|-----------------|\n| Qwen-Omni-3b (vanilla)       | 35.19           |\n| **Qwen-Omni-3b (after text-only CL)** | **47.40**           |\n| Qwen-Omni-7b (vanilla)       | 39.84           |\n| **Qwen-Omni-7b (after text-only CL)** | **49.63**           |\n\nWe would like to highlight that, cross-modal retrieval performance would have degraded if non-textual modality spaces do not follow the enhanced textual representation space geometrically.\n\n### **3. Questions for Section 2.1 Anisotropy Analysis** \nFirst, we want to highlight that **the lower is the better** for anisotropy estimate value, as we are measuring the expected cosine similarity of **random** examples, instead of paired examples.\n\n‒ **Setting.** We use 1000 examples for each modality (1000 texts & images, 1000 texts & audios, 1000 texts & videos ) for the 3 settings, respectively, leading to 499,500 cosine similarity values (lower triangular of the similarity matrix) for each setting, and we take the mean for it. This is typically deemed sufficient for approximating anisotropy [1].\n\n‒ **Why cosine similarity decreases every layer.** Note that as we are measuring cosine similarity expectation of **random**  pairs, and it is a good property for random pairs to have lower similarities (more isotropic, less anisotropic). As the layer progresses, the model contextualizes each input more, leading to more different embeddings for each input, and thus, their cosine similarity with other inputs would become lower.\n\n‒ **\"Qwen-omni is anisotropic but why good on benchmarks?\"** After text-only contrastive learning, the expected cosine similarity goes down (less anisotropic) for all modalities, making the embedding space more isotropic. We show in the paper that this leads to state-of-the-art performance on embedding tasks.\n\n### **4. Questions for Section 2.2 Kernel Analysis** \n\n‒ **Why Qwen-VL 3b has more layers than Qwen-VL 7b**: According to Table 1 of the Qwen2.5-VL technical report [2], Qwen-VL 3B has more layers in the decoder LLM than Qwen-VL 7B (36 layers vs. 28 layers). The report does not provide an explicit rationale for this architectural decision. We suspect that this configuration may lead to better overall performance.\n\n‒ **Improvement is small** This section measures cross-modal alignment, instead of the actual benchmark performance. We intend to show that, through text-only contrastive learning, cross-modal alignment actually improves, as opposed to decreasing.\n\n### **5. Performance gain over base models.**\n\nIn point 2 above, we provided extensive results comparing base models.\n\n### **6. Missing citations** \n\nThanks for spotting the typo. We will add the citation to LLaMA in Line 97 in the revised version.\n \n\n\n[1] Ethayarajh 2019, How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings\n\n[2] Qwen2.5-VL Technical Report. Qwen Team et al. 2025."}, {"review_id": "AgDJi8KIgC", "reviewer": "Reviewer_FpEU", "summary": "This paper empirically demonstrates that latent alignment emerges within MLLM representations, allowing contrasive learning to serve as a lightweight refinement stage. The authors first propose two ways to identify that text-only contrasive learning is beneficial for the multimodal ability of MLLMs, and then propose Language-centric Omnimodal Representation Learning to improve the multimodel performance of MLLMs.", "questions": "See above", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The paper is well-written and easy to follow, with a logical flow and clear explanations of the proposed methodology. The overall readability makes it accessible not only to experts in the field but also to researchers and practitioners who may be less familiar with the specific domain.\n2. The experimental results demonstrate that the proposed method performs well across a variety of benchmark datasets. This consistency suggests that the approach is robust and effective in different settings, which enhances its credibility and practical value.\n3. The proposed method introduces a novel strategy that can be applied to a wide range of Multi-Modal Large Language Models (MLLMs). Its adaptability across different architectures and tasks indicates that it has the potential to become a valuable tool for future research and real-world applications in multi-modal learning.\n\nWeaknesses:\n1. In Section 2.1, the authors mention conducting text-only contrastive learning, but the process is not clearly explained. Specifically, it would be helpful to clarify how the contrastive loss is formulated in the absence of visual inputs, what kind of data is used, and how this component contributes to the overall training objective. Providing more details would improve reproducibility and deepen the reader’s understanding of the method.\n2. The experiments appear to rely on LoRA (Low-Rank Adaptation) for training the language decoder. However, it is unclear how the performance would change if the model were trained without LoRA—i.e., through direct fine-tuning of the full decoder. A comparison between these two approaches would help assess whether the use of LoRA affects performance or generalization capabilities.\n3. Section 4.2 presumably presents an ablation study or some form of analysis; however, the specific results are not clearly reported in the current version. Including concrete numerical results or visualizations would allow readers to better understand the findings and evaluate the effectiveness of the design choices made by the authors.\n\nMinor:\n1. Line 97 has no citations.", "was_revised": true, "final_justification": "Considering the rebuttal, i think this paper is novel with sufficient experiemental results. I will maintain my positive rating and lean towards accepting this paper.", "rebuttal": "We appreciate your positive comments and insightful suggestions.\n\n**Details of text-only CL in Section 2.1** The key process is similar to that described in Section 3.1. We first decouple the LLM from the MLLM and train it using a standard contrastive loss (InfoNCE loss), where each positive pair consists of two texts; we then plug the LLM back to the MLLM after training. For the experiments in Section 2.1, we use the all-NLI dataset, in which each positive pair is composed of an anchor and an entailment from a natural language inference dataset; for example, Anchor: \"Children smiling and waving at the camera\"; Entailment: \"There are children present.\" All other texts within the same batch are treated as in-batch negatives. In contrast, for the main experiment in Section 3.1, we employ an additional dataset we constructed, Scale1M, which incorporates a diverse range of multilingual data sources.\nWe appreciate your suggestion and will enrich the descriptions in Section 2.1 accordingly!\n\n\n**Performance of Full-Finetuning vs. LoRA** We provide full-finetuning results as follows, which provides decent performance but lags behind LoRA training, when trained with the same Qwen2.5-VL 3b backbone.\n| Size | Model Type | Training Time | Multilingual Retrieval | Visual STS (En) | Visual STS (Cross) | Visual STS (Multi) | Document | Linear Probe | Average |\n|------|--------------------------------|---------------|------------------------|-----------------|--------------------|--------------------|----------|--------------|---------|\n| 3b | Base Model | - | 31.73 | 73.82 | 59.03 | 68.57 | 28.82 | 46.96 | 51.49 |\n| 3b |Text-only (Full-Finetuning) |~2 hours | _44.61_ | _81.65_ | _68.67_  | _77.75_ | _49.71_ | _50.21_ | _62.10_ |\n| **3b** |**Text-only (Lora)** |**~1 hour** |**51.61** |**81.88** |**74.97** |**78.30** |**57.90** |**53.05** |**66.28** |\n\n\nWe hypothesize that full-finetuning introduces more perturbation to the pretrained parameter space, which stores latent cross-modal alignment. In contrast, LoRA beter preserves this alignment through minimal adaptation of model parameters.\n\n**The experiment in Section 4.2** \nThe experimental results discussed in Section 4.2 are visualized in Figure 5. After establishing the theoretical relationship between the generative and representational capabilities of MLLMs, we compare the base models' performance on generative benchmarks with their post-contrastive learning performance on embedding benchmarks, showing a positive scaling correlation between generation and representation capability, supporting our theoretical analysis.\n\n**Missing citations** Thanks for spotting the typo. We will add the citation to LLaMA in Line 97 in the revised version."}]

Author rebuttal:

Meta-review: The submission addresses how multimodal large language models that were fine-tuned with contrastive learning perform and argues that the MLLMs acquire an implicit alignment across modalities during generative pre-training. They use this insight to propose GRSL. The experimental analyses and comparisons demonstrate the benefit. 

All reviewers acknowledge the merit of the submission and recommend acceptance, though mainly with a borderline stance. The AC agrees and recommends acceptance.
Decision: Accept (poster)