VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
Authors: Rui Meng
Ziyan Jiang
Ye Liu
Mingyi Su
Xinyi Yang
Yuepeng Fu
Can Qin
Raghuveer Thirukovalluru
Xuan Zhang
Zeyuan Chen
Ran Xu
Caiming Xiong
Yingbo Zhou
Wenhu Chen
Semih Yavuz
Venue: Accepted by TMLR ()

Abstract: Multimodal embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering over different modalities. However, existing multimodal embeddings like VLM2Vec, E5-V, GME are predominantly focused on natural images, with limited support for other visual forms such as videos and visual documents. This restricts their applicability in real-world scenarios, including AI agents, retrieval-augmented generation (RAG) systems, and recommendation. To close this gap, we propose VLM2Vec-V2, a unified framework for learning embeddings across diverse visual forms. First, we introduce MMEB-V2, a comprehensive benchmark that extends MMEB with five new task types: visual document retrieval, video retrieval, temporal grounding, video classification and video question answering -- spanning text, image, video, and visual document inputs. Next, we train VLM2Vec-V2, a general-purpose embedding model that supports text, image, video, and visual document inputs. Extensive experiments show that VLM2Vec-V2 achieves strong performance not only on the newly introduced video and document retrieval tasks, but also improves over prior baselines on the original image benchmarks. Through extensive evaluation, our study offers insights into the generalizability of various multimodal embedding models and highlights effective strategies for unified embedding learning, laying the groundwork for more scalable and adaptable representation learning in both research and real-world settings.

Published in Transactions on Machine Learning Research (02/2026)
VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
Rui Meng 1+f
Ziyan Jiang Ye Liu'
Mingyi Su Xuan Zhang Zeyuan Chen Ran Xu Wenhu Chen
Xinyi Yang! Yuepeng Fu
Can Qin
Raghuveer Thirukovallrru
Caiming Xiong*
Yingbo Zhou
Semih Yavuz!
*Salesforce Research
UC Santa Barbara
Universitty of Waterloo National University of Singapore
*Tsinghua University
Duke University
Reviewed on OpenReview:
https://tiger-ai-BB htt.:gtttub...//LMWee.....
Abstract
Multimodal embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering over different modalities. However, existing multimodal embeddings like VLM2VEC, E5-V, GME are predominantly focused on natural images, with limited support for other visual forms such as videos and visual documents. This limits their applicability in settings that require broader visual understanding, including Al agents, recommendation. In this work, we propose VLM2VEC-V2, a unified framework for learning embeddings across diverse visual forms. To support comprehensive evaluation, introduce MMEB-V2, an expanded benchmark that builds upon MMEB by adding five new task types: visual document. retrieval, video retrieval, temporal grounding, video classification and video question answering. We then train VLM2VEC-V2, a unified embedding model capable of handling text, images, videos, and visual documents. The results demonstrate that VLM2VEc-V2 not only achieves strong performance on the newly introdueed video and visual document tasks but also improves upon prior baselines on established image benchmarks. Our findings provide deeper insights into effective training strategies for unified embedding learning across diverse visual modalities.
-uggmette generation (RAG) systems, and
1 Introduction
Embedding models play crucial role in connecting data across various modalities. By encoding helerogeneous multimodal data into shared dense representation space, they enable many down-stream applications like classification, clustering, retrieval. In recent years, we have witnessed significant advances in embedding models, largely driven by the progress of large foundation models. For instance, recent breakthroughs text embedding (Su al., 2023; Wang et al., 2024a; Meng al., 2024; BehnamGhader et al.) have been achieved by integrating pretrained large language models with multi-task instruction embedding tuning, Similarly, Jiang et al.; Zhang et al. (2024); Chen et al. (2025) demonstrated strong performance across multiple text-image tasks by instruction-tumn tuniinntttinngg multimodal large language models (MLLMs) into effective embedding models.
Existing ing iultil embedding models are trained on datasets like MMEB (Jiang el al.) and M-BEIR (Wei et al., 2024), which are focused predominantly on natural images or photographs, sourced from MSCOCO (Lin et al., 2014), Flickr (Plummer al., 2015) and ImageNet (Deng et al., 2009) datascts. These datascts fail cover broader forms of visual information, such documents, PDFs, websites, and videos.
"Contributed equally. Now Google.

---

Published in Transactions on Machine Learning Research (02/2026)
This limited coverage makes existing embedding models overly domain-specific and prevents them from generalixing well to realistic tasks such as document retrieval and YouTube video search. We empirically validate this claim in our experiments (Table 2). A model trained exclusively on visual documents, such as Colpali (Faysse et al.), achieves strong performance on document tasks (71.0%) but suffers a severe performance drop on image (34.9%) and video lasks (28.2%). Conversely, a model trained solely on image data, such as VLM2VEC (2B) (Jiang et al.), excels on image tasks (59.7%) but performs substantially worse on document (41.6%) and video tasks (29.0%).
To address these limitations, introduce MMEB-V2, advanced multimodal embedding dataset designed to train and evaluate embedding models across three key visual modalities: images, videos, and visual documents. Expanding on the original MMEB (Jiang et al.) framework, MMEB-V2 broadens the evaluation scope to encompass five new tasks, including four video-based
Retrieval, Moment Retrieval, Video Classification, and Video Question Answering as well as one task centered on visual documents: Visual Document Retrieval. This comprehensive suite of tasks allows for robust assessment of multimodal embedding models across static, temporal, and structured visual data settings.
Built top of MMEB-V2, we propose VLM2Vec-V2, framework that adapts oftteatf the-art multimodal large language models, such as Qwen2-VI. VL (Wang et al., 2024b), into a unified embedding model. VLM2VEC-V2 is trained using a mixture of instruction-folloiing tasks spanning multiple task categories, enabling it. to learn unified representations across a wide range of visual modalities.
In this paper, we adopt Qwen2-VL-2B the base model for all experiments, balancing computational cost with the inclusion of video training data. Our framework is general and can be readily scaled to larger MLLMs.
The contributions of this work are threefold.
We propose MMEB-V2, a comprehensive dataset for systematically evaluating embedding models on diverse tasks involving vidcos, images, and visual documents.
We develop VLM2VEC-V2, a unified multimodal embodding framework that supports instruction following produce general-purpose embeddings for wide range of downstream tasks.
Our experiments demonstrate that the 2B-parameter version of VLM2VEC-V2 outperforms prior baselines across 78 datasets. Through detailed ablation studies, we identify effective training strategies for learning multimodal embedding models.
2 MMEB-V2: Expanding Embedding Benchmarks Beyond Image- Text
We introduce MMEB-V2, a comprehensive dataset designed evaluate model performance on multimodal embedding tasks involving various combinations of text, image, video, and visual document modalities. In addition to the five task categorics that involve natural images and text, MMEB-V2 includes four video understanding tasks and one visual document understanding task. Figure presents an overview of MMEB-V2.
Each task presents the model with a query and a corresponding set of candidate responses, where the goal is to select the correct target. The query consists of a combination of an instruction, a textual component, and a video or document. For based tasks, videos are represented as sequences of frames sampled uniform intervals from the raw footage to ensure consistent temporal coverage. Instructions serve to specify the model's objective (e.g., "Recognize the category of the video contents."), and query lexts can be questions, descriptions, commands specific to the video (e.g., "How many red socks are above the fireplace at the end of this video?", "Select the clips of videos that contain a dolphin."). Each task is associated with a specific target type, which varies depending on the nature of the task. For example, in video classification, the model must identify the activity or object class label (e.g., "Yoga" or "Car").
To make the datasct practical and accessible for future rescarch, we selectively downsample certain datascts to ensure that the full benchmark can be run within a reasonable amount of time. Our datasets span diverse domains, including sports, object recognition, daily-life activities, and movie TV show clips. These
2

---

Published in Transactions on Machine Learning Research (02/2026)
Irage Clssaliaaten
10
MMEB-V2
Asseertty
Vouaa Questin
10
Vec2007
BUN397
OKVVA
A-OKYOX
DocVGA
Objecthiei
Counlhy211
PlecoSe5
Massive Multimodal Embedding Benchmark
InioVDA
ChariDA
VoewiWWW
Imagaliet-1K
Honet /'derne
ScinnoaCA
GBA
Multi-task
ImseeelalA
Inscudius
Text/OA
Meta tasks/78 Taask
Video Question
Video Clacoliation
Multimodal
Answenig
VideosMMME
MyVBercc's
UCF101
HVD0851
Tweilmage' Sr DoDuummmenn
NEXT-QA
Eax8charma
Kinetcs-700
Brrckkiet
AdiilyyaaOA
Sumneiing-Somedingg V2
Voual Decumontt
Meciwall
24
Viteoiemi fisiivaa
ViDRRo (10)
ViDoRR V2
Isage/ivell feeeriaa
12
(6)
(2)
MTR.VTT
MSVO
MSCOCO.PP
MSCOCO_1T T21
MWLonnSench Doo (2)
DID:Mo
Viccaal
YouCook2
VisuaNows__g
NIGHTS
WebGA
Visua Grousting
EDIS
OVEN
MSCOCO
ReCOCO
GhWilgghihts
Charades-5
WKK-88NG
Fastionio
VauuePW-PPaattng
MismentSonier
Figure 1: An overview of MMEB-V2, which includes 9 meta-tasks and 78 tasks in total. In addition to the original MMEB benchmark, MMEB-V2 introduces five new meta-tasks focused video and visual documents. Tasks from MMEB are indicated with blue borders, while newly introduced tasks in MMEB-V2 are marked with red borders.
samples are drawn from varied sources such as YouTube, professional productions, and crowd-sourced content, ensuring both diversity and real-world relevance. Summary statisties for each task are presented in Table 1. Details about constructing each dataset are provided in Appendix A.1.
Video Retrieval (V-RET) The query consists of an instruction, descriptive text related to the video content, and sequence of video frames. The model must retrieve the correct corresponding video from a pool of thousands of video candidates.
Moment Retrieval (M-RET) The query consists of an instruction, a textual description, and optionally a full video, with the goal of retrieving the temporal segments that best matches the description. The model must select the ground-truth clip from approximately 10 candidate segments within the full video. Video Classification (V-CLS) Given an instruction and sequence of video frames, the model is taskod with predicting the correct class Aebe-eelated to the scene or action depicted from a set of possible classes
Video QA (V-QA) The input consists of an instruction, a textual question, and vidco. The model must select the correct answer from several options, including one correct choice and many distractors. Visual Document Retrieval (VisDoc) This task category evaluates the model's ability to retrieve structured visual documents such as multi-page PDF's and slide decks in response to natural language queries. We include five datasets in this benchmark. ViDoRe V1 & V2 (Faysse et al.; Macé et al., 2025) and VisRAG (Yu et al., 2024) are composed of multiple document QA datasets and cover broad range of document types and use cases (c.g., charts and slides), though they were not originally designed for retrieval. To complement them, we include ViDoSeek (Wang el al., 2025) and MMLongBench--Doc (Ma et al., 2024), which provide fine-graine -reiieraidd, page-level annotations suitable for retrieval evaluation. Besides, we reformat the two datasets to support both document-level and page-level evaluation.
3 Unified Embedding Model for Video, Image, and Visual Document
Built upon the VLM2VEC framework, VLM2VEC-V2 extends embedding learning a broader range of modalities and tasks, which introduces additional challenges due to their structural and semantic characteristies. This
3

---

Published in Transactions on Machine Learning Research (02/2026)
Task
Query MOD
Target MOD
Domain
#Query
#Candidates
Video Hetrieval (5 Tasks)
DIDeMo
Open
1,001
1,001
MSR-VTT
Open
1,000
1,000
MSVD
V
Open
670
VATEX
V
Open
4,478
4,478
YouCook!
Cooking
3,179
3,179
Moment Retrieval (3 Tasks)
QVilighlights
T
Vlog/News
1,083
Charades-S1D STA
Activity
727
Seeket
1+V
Open
1,800 (1,002 deduped)
10
Video Classiflcation (5 Tasks)
Kineties-700
T
Open
1,000
700
88v2
T
Human-Obie Object Interaction
1,000
174
HMDB51
Open
UCF101
T
Open
1,000
101
Breakfast
T
Cooking
433
10
Video QA (5 Tasks)
MVBench
V+T
T
Rempaal/Temprrl
4,000
~
Vidco-MME
V
Real-world
2,700
NExT-QA
V+T
Daily activity
8,564
EgeSchena
V+T
T
Egucentric
500
ActivityNeeQA
V+T
Activity
1000
Visual Document Retrieval (24 Tasks)
ViDoRe (10)
Documents
1,646
- 909
ViDoRe-V2
Documents
640
452 1,538
VisRAG (6)
D>
Documents
63 816
500 9.590
ViDoSeek (2)
D
Documents
1,142
5,349
Doc (2)
6,492
Table 1: The statistics of MMEB-V2, which includes lasks across five mela-lask categories addition the original MMEB, are summarized below. Here, we list only the additional datasets introduced beyond those in MMEB. We consider four modalities (MOD): T (Text), I (Image), V (Video), and D (Visual Document).
section presents our methodology, beginning with a brief overview of the VLM2Vec framework (Section 3.1) and the foundation model selection (Section 3.2). Sections 3.4-3.4 then describe the key extensions beyond VLM2VEC, including our multimodal data mixing strategy (Section 3.3), and batching strategies to balance data sources during training (Section 3.4).
3.1 Preliminary: VLM2Vec Framework
As one of the earliest works to adapt MLLMs for universal multimodal embedding learning, VLM2VEC (Jiang et al.) formulates multimodal embodding learning as instruction-condiioone conditionod embedding alignment across diverse tasks and modalities. In VLM2VEG, each training example consists of query target pair (g, (q,t*), where both g and f+ may comprise text, images, videos (extended in VLM2VEc-V2), with predefined task-specific instruction. Both g and &+ are independently passed through the same MLLM, and the final embeddings hy and h are obtained from the last token's hidden state. Training is performed using a standard InfoNCE objective with in-batch negatives and optional hard negatives to align query and target reprosentations, defined as:
combinations thereof, together
o(hg h,t) C = log s(,, h++) Ei-eN()) @(b,, h--)' o(h, h.) = exp cos(h,, hi))
(1)
where cos(:.) denotes the cosine similarity function and 7 is the temperature hyperparameter. Here, L denotes a negative target sampled from the in in-batch candidates or mined hard negatives associated with the query q. GradCache (Gao et al., 2021), a gradient caching technique, used increase the effective batch size for contrastive learning by decoupling backpropagation between the contrastive objective and the encoder, thereby removing encoder backward-pass dependencies along the batch dimension.
4

---

Published in Transactions on Machine Learning Research (02/2026)
3.2 Foundation Model Selection
While VLM2VEC established an effective paradigm for transforming multimodal large language models into image-text embeddings, its design was primarily tailored to static visual inputs. In contrast, VLM2VEC-V2 extends this paradigm to support interleaved sequences of text, images, and videos, as well as long-form inputs such as multi-page visual documents and full-length videos.
Our goal is to learn unified embedding space that generalizes across diverse visual modalities and tasks. This necessitates backbone model that can flexibly encode interleaved multimodal sequences while scaling to long-context inputs. Based on these requirements, we adopt Qwen2-VI. as the backbone of VLM2VEC-V2.
Qwen2-VL is particularly well suited to the challenges posed by VLM2VEC-V2, offering (i) Naive Dynamic Resolution for efficient processing of inputs with varying spatial resolutions, (ii) Multimodal Rotary Position Embedding (M-RoPE) to capture spatial and temporal structure in videos and visual documents, and (iii) a unified architecture that integrates 2D and 3D convolutions for consistent image and video understanding. Together, these capabilities enable VLM2VEC-V2 to effectively handle diverse and complex multimodal data.
3.3 Training Data and Multimodal Data Mixing
To train VLM2VEC-V2 effectively across diverse multimodal tasks, curate training dataset comprising three main sources: video-language instruction data, visual document retrieval, and image-based vision tasks,
First, we utilize training data from LLaVA-Hound (Zhang et al., 2025), which includes synthetic video-caption pairs and video QA examples generated by ChatGPT. Specifically, we use 300k video-caption pairs and 240k video QA pairs. For the caption data, we adopt two formats: using the caption as the query and video as the target in a video retrieval setup, or using the video as the query to retrieve the most relevant textual description from candidate captions. We denote these three video training datasets as Vid2Cap, Cap2Vid, and VideoQA in the following discussion.
Second, for visual document retrieval tasks, we incorporate datasets from ViDoRe (Faysse et al.) and VisRAG (Yu et al., 2024), including colpali train set (118k), VisRAG synthetic (239k), and VisRAG in-domain (123k), which provide training examples for image-based document understanding and retrieval. We denote these three video training datasets as ColPali, VisRAG and ViaRAGayn in the following discussion.
Finally, we include image-text datasets from MMEB-train (Jiang al.) support generalization across a wide range of visual understanding tasks, including question answering, classification, retrieval, and visual grounding. These datasets help improve the robustness of the learned embeddings across multiple tasks.
We perform on-the-fly batch mixing guided by a pre-defined sampling weight table. This table specifies the relative probabilities of sampling from each dataset, enabling controlled exposure to different task types. By dynamically drawing samples during training, we ensure balanced coverage and prevent overfitting any single modality or domain. We find that jointly mixing datasets from diverse sources and modalities leads to the most effective model training. Detailed ablation studies are presented in Section 4.5.1.
3.4 Batching Strategies
To support effective multi-task training over heterogeneous data sources, we introduce an homogeneous sub-batching strategy to enhance the hardness and stability of contrastive learning. Specifically, each full batch (e.g,, size 1024) is divided into smaller sub- bat-batches (e.g., 8 bat-batches of size 128), where each sub-batch is sampled independently from single data source only. Compared to per-sample independent sampling, grouping similar samples inlo sub-batches increases intra-sub-batch homogeneity, which raises the difficulty of contrastive discrimination. At the same time, interleaving multiple such sub-batcher preserves cross-tad tross-task diversity within the full batch, avoiding the instability commonly observed in completely homogencous batches that originate from a single source. This strategy strikes a balance between sample diversity and structural consistency, fostering more stable and robust. optimization dynamics.
5

---

Published in Transactions on Machine Learning Research (02/2026)
4 Experiments
4.1
Experiment Setting
4.1.1 Training Setting
We train VLM2VEC-V2 using Qwenn-VT--B-Instee
(Wang al., 2024b) backbone, batch size of 1,024 for 2K steps and homogeneous sub- batch size of 64, with the loss temperature set to 0.02. To support scalable training, we use GradCache (Gao et al., 2021) to enable large global batch sizes and run all experiments on 8 H100 GPUs. For parameter-efficien training, we apply LoRA luning with a rank of 16 and scaling factor a = 64 using the PEFT framework (Mangrulkar et al., 2022). We use 8 uniformly sampled frames to represent each video during both training and evaluation.
Training VLM2VEC-V2 with 2B backbone 8 NVIDIA H100 GPUs takes approximately 15 hours per Ik steps. Considering the training cost, all analyses in this work conducted using the 2B backbone. Nevertheless, our framework is general and can be readily adapted to larger MLLMs, which may yield stronger embedding models.
4.1.2 Baselines
We compare against several VLM embedding models, including GME (Zhang et al., 2024), VLM2Vec (Jiang et al.), and LamRA (Liu et al., 2025), most of which are primarily trained on image-text pairs. While these models are not explicitly designed for video tasks, many can be adapted to handle video inputs by encoding multiple frames as sequential images. For video evaluation, GME and LamRA use a single middle frame, while the remaining models use 8 uniformly sampled frames.
In addition, to provide fair comparison across modalities, we evaluate VLM2 V2 against representative models specialized for each modality. Specifically, we include ColPali (Faysse et al.) (v1.3), model tailored for document retrieval using late interaction matching mechanism.
4.2 Evaluation Metrics
We use @il@l as the primary evaluation metric for all video and image lasks, measuring the proportion of queries where the correct target is ranked at the top. For visual document tasks, we report NDCG@5 to remain consistent with prior work in this domain.
4.3
Main Results
Image
Video
VisDoc
Model
All
RET
QA
MRET
VDRv1
VDRv2
Overall
Datasets
10
12
5
24
79
Baseline Models
ColPali v1.3 (31)
40.3
11.5
48.1
40.3
34.9
26.7
37.8
21.6
25.5
28.2
83.6
52.0
81.1
43.1
71.0
46.0
GME (2B)
54.4
20.9
66.9
55.5
51.9
34.9
62.0
25.6
32.4
33.9
86.1
54.0
62.5
43.1
72.7
$4.1
GME (7B)
57.7
31.7
71.2
50.3
56.0
37.4
50.4
28.4
39.2
38.6
89.4
55.6
85.0
44.4
75.2
57.8
LamRA-Quenn (7B)
20.2
26.5
70.0
62.7
54.1
30.3
42.6
24.3
34.6
35.2
22.0
11.5
37.4
21.0
23.9
40.4
LamflA Qwe22.5 (7B)
51.7
31.1
66.9
50.7
52.4
32.9
42.6
23.2
37.8
33.7
56.3
33.3
58.2
40.1
50.2
47.4
S.e2Voo-wwes2VL.... (20)
41.6
VweelVe--weee2LL (78)
16.4
52.3
VLM (2B)
62.9
50.3
69.5
77.3
61.9
39.3
31.3
28.8
38.5
31.6
73.5
41.9
79.4
39.4
65.4
58.0
Table 2: Performance comparison between bascline models and our VLM2VEC-V2 across image, vidco, and visual document lasks. CLS: classification, QA: question answering, RET: retrieval, GD: grounding, MRET: moment retrieval, VDR: ViDoRe, VR: VisRAG, OOD: out-of-domain.
http:://hunggigiicccccc///eee/// ww//uene/qun/./...e Instruct

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2 presents comprehensive comparison between VLM2VEC-V2 and diverse set of bascline models across 78 datasets covering image, video, and visual document tasks. Full results are detailed in Appendix A.3. VLM2VEC-V2 achieves the highest overall average score (58.0), outperforming multiple strong baselines, including GME, LamRA and VLM2VEC, which were built on the same Qwen2-VI. VL backbone. This highlights the effectiveness of our unified training approach in delivering strong and balanced performance across different. modalities and tasks. On image tasks, VLM2VEC-V2 shows strong results, outperforming most baselines by a large margin and achieving performance comparable to VLM2VEC-7B despite being only 2B in size. For video tasks, it achieves competitive performance despite being trained on relatively small amount of video data. In visual document tasks, VLM2VEC-V2 outperforms all VLM2VEC variants, while still trailing behind ColPali, which is specifically optimized for VisDoc tasks.
4.4 Qualitative Analysis
To more comprehensively demonstrate improvements over baseline models, we perform qualitative comparisons between VLM2VEc-V2 and VLM2VEC on representative examples spanning both video and document tasks, detailed in Section A.6.
Figures 3 and 4 show that VLM2VEC mainly relies on surface-level visual or keyword similarity rather than the details of the whole vidco and the given text. In Figure 3, although the query asks about the vase that gets smashed, VLM2VEC only detects the existence of "a vase" and retrieves visually similar vases from unrelated scenes, without understanding which vase is actually involved. In Figure 4, ignores the contextual and temporal cues in the video, retrieving frames that simply include the mentioned characters and door, even though they are irrelevant to the described situation. In contrast, VLM2VEC-V2 leverages the visual context from the in video, allowing it to locate the correct moment that matches the intended meaning of the query. This indicates that VLM2VEC-V2 learns a much stronger and more comprehensive video representation.
For document understanding, Figures 5 and 6 show that VLM2VEC mainly relics on keyword matching rather than reading and understanding the content of the whole document. In Figure 5, the incorrectly retrieved document contains the word "child", which appears in the query, but its actual content does not answer the question. Similarly, in Figure 6, VLM2VEC retrieves a document that mentions "social media", while ignoring the key details needed to provide the correct answer. In contrast, VLM2VEC-V2 leverages the broader semantic information in the document and identifies the correct article that contains the detailed content required by the query. This demonstrates that VLM2Vec-V2 offers a stronger capability in document comprehension beyond superficial keyword overlap.
4.5
Ablation Analysis
4.5.1
Generalization Across Modalities
To investigate how multimodal training configurations affect cross-modal generalization, we begin by examining the single-modality training results in Table 3. In the first column block, each model is trained a single modality under a matched budget of 100K samples for a fair comparison. Under this controlled setting, models trained on image-only data consistently achieve the strongest cross-modal performance, demonstrating that high-quality image supervision serves as a crucial foundation for adapting MLLMs into unified multimodal embedding models. In contrast, models trained exclusively on VisDoc or Video data show substantial performance degradation across most evaluation modalities, especially on imago-basod benchmarks, indicating limited transferability when image supervision is absent.
Table 3 further analyzes the impact of data source quality within the second and third column blocks, where the modality and training scale remain fixed (100K samples) but the datasct source varies. Within VisDoc, ColPali-based training significantly outperforms both VisRAGIna and VisRAGayn across all modalities, suggesting that curated VisDoc datasets provide more effective supervision than weakly constructed or synthetic sources. A similar trend holds for video training: Vid2Cap yields clear advantages over Cap2Vid and VideoQA. These findings demonstrate that, even when the training modality is fixed, the dataset source critically dictates generalization capability.
7

---

Published in Transactions on Machine Learning Research (02/2026)
Eval. Modality
Training Modality
VisDoc Training Source
Video Training Source
Image
VisDoc
Video
ColPali
VisRAGsyn VisRAGmad
Vid2Cap
Cap2Vid
VideoQA
Image
58.45
39.87
30.64
32.29
4.14
3.53
20.48
3.52
4.12
VisDoc
58.73
61.72
41.55
53.66
1.26
0.79
34.23
0.80
0.56
Video
33.58
23.62
26.30
25.40
10.86
10.76
23.89
10.43
10.23
AVG
52.80
39.87
30.64
37.27
4.80
4.36
25.50
4.28
4.43
Table 3: Performance comparison across evaluation modalities (rows) under different training data settings (columns).Roo .Rows indicate the evaluation modality of the test datasets (Image, VisDoc, or Video). The first column block compares models trained with single modality to analyze modality-level training effects. The second block varies VisDoc training data sources to study the quality of various VisDoc data sources. The third block varies Video training data sources to analyze the impact of video data sources. For fair comparison, each training configuration uses 100K training samples.
Training Data Configuration
Eval. Modality
Image 100K
Image 100K
Image 100K
All Sources 200K
Image 200K
ColPali 50K
ColPali 100K
VideoCap 100K
VideoCap 50K
Image
60.75
59.01
58.23
59.20
58.62
VisDoc 56.69
64.45
56.33
63.40
66.87
Video
34.02
34.00
33.13
35.11
33.19
AVG
53.33
54.91
51.85
54.93
55.29
Table 4: Modality mixing analysis under different training data configurations. Rows correspond to evaluation modality, while columns represent different training data configurations formed by mixing modalities. For lair comparison, each configuration uses a fixed budget of 200K training samples.
We then analyze multimodal training using Table 4, which fixes a larger budget of 200K samples. Image-only training already provides a strong baseline across evaluation modalit due to the richness and transferability of image supervision. Building on this foundation, incorporating additional modalities can further improve performance, but the gains depend hcavily modality compatibility and data quality. Notably, mixing image data with low-quality video data may degrade performance, particularly for image- and Aideo-based evaluations. In contrast, combining image supervision with high-quality VisDoc data consistently improves results across all modalities. Finally, the two full mixtures of Image + VisDoc + Video achieve the best overall average performance, demonstrating that broad modality coverage is beneficial for learning universal multimodal providd that image supervision remains the anchor and that additional modalities are carcfully curated. To rigorously assess the generalization impact of multimodal training configurations, we further conduct lask-level paired I-test to evaluate statistical significance (Section A.5).
4.5.2 Ablation Study Data Sampling Strategies
As part of our ablation study, we investigate the impact of homogeneous sub-batching on model performance across the three modalitics. When the sub-batch size (IB) set to O, all samples in the batch are randomly drawn from different sources without grouping. A value of 64 indicates that batch of size 1,024 divided into sub-batches of size 64, resulting in data from 16 distinct sources per batch. At the other extreme, IB of 1024 means the entire batch comes from a single source, effectively disabling interleaving. This setup allows us to analyze how different levels of source mixing influence training dynamics and cross-modal generalization.
As shown in Table 5, increasing the sub-batch size consistently improves performance for both VisDoc and Vidco. Conversely, the best performance on the Image modality achieved with sub-batch size of 64, exhibiting an inverted U-shaped trend-meen monotooiillllll increasing from 0 64, followed by a monotonic decline from 64 to 1024.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Modality IISB-0
IISB-32
IISB-64
IISB-128
IISD-1024
Image
61.2
62.3
63.2
62.0
G0.7
VisDoc
48.6
51.0
52.1
53.9
54.3
Video
34.6
33.2
33.5
34.5
35.4
Table 5: Performance comparison across different homogeneous sub-batch size for different modalities.
4.5.3 Ablation Study on Model Settings
We investigate the impact of different LoRA ranks (8, 16, 32) on model performance across modalities to understand how the capacity of parameter-effiient tuning affects generalization. As shown in the left part Figure 2, a LoRA rank of 16 yields the best overall performance across image, video, and visual document tasks. This suggests that a moderate number of tunable parameters beneficial for handling diverse modalities, while further increasing the rank to 32 does not lead to additional gains.
We also examine performance across training steps to understand how each modality benefits from continued training. As shown in the right part of Figure 2, all three modalitics exhibit improved performance with increased training steps. Notably, there is no clear sign of saturation by 5K steps, particularly for VisDoc and Video, suggesting that further gains may be achievable with extended training. We leave a more in-depth exploration of long-horizon training and convergence behavior to future work.
70
NNrformence urder Diffsennt LoRA Barkss
Image
VisDoc
Video
$23532
LoBA 32
64
35,0
Perfomsinc
Perfrmance
Performance
Perforance
34.5
62
60
40
33.5
424 3p
33.0
Image
VaDoc Modallty
Video
1000
2000
3000 4000 5000 Stop
1000 2000 Stop
3000
4000
5000
1000
2000
3000
4000 5000
Figure 2: The left figure shows performance across LoRA ranks for different modalities, while the right figure illustrates performance trends across training steps.
5
Related Works
5.1
Multimodal Embedding Benchmarks
Numerous benchmarks have been proposed to evaluate multimodal models, with most carly efforts focusing on static image-texl pairs. Datasets such as MSCOCO (Lin al., 2014), Flickr30K (Plummer el al., 2015), and Conceptual Captions (Sharma et al., 2018) enabled progress in tasks like image captioning and retrieval. Linear probing is also common evaluation sctting that trains lincar layer for image classification (Radford et al., 2021) to investigate the generalization of representation vectors. More recent benchmarks like M-BEIR (Wei et al., 2024) and MMEB (Jiang et al.) introduced multi-task evaluations for multimodal embedding models, covering tasks such as retrieval and QA. However, these benchmarks remain limited to static images and short contexts. Video-based benchmarks such as MSR-VTT (Xu et al., 2016), QVHighlights (Lei et al., 2021), and ActivityNet Captions (Krishna et al., 2017) target retrieval and captioning tasks, yet lack unified evaluation frameworks for embeddings. Our MMEB-V2 addresses these gaps by providing a comprchensive embedding benchmark for diverse modalities.
5.2 Video Representation Learning
Video representation learning has evolved significantly, progressing from carly convolutional approaches to sophisticated transformer-basel architectures. Traditional vision-language models like CLIP (Radford
9

---

Published in Transactions on Machine Learning Research (02/2026)
et al., 2021) and BLIP (Li et al., 2022), while effective for image-text tasks, often struggle to capture the temporal dynamics inherent. in video data. To address this, recent models have been developed to better handle the complexities of video understanding. VideoCLII (Xu et al., 2021), VideoCoCa (Yan et al., 2022) integrates contrastive learning with captioning objectives to enhance vidco-text representation alignment. InternVideo2 (Wang et al., 2024c) employs progressive training approach that unifies masked video modeling, cross-modal contrastive learming, and next-token prediction, resulting in superior performance on over 60 video and audio tasks. Recent models like LLaVE (Lan et al., 2025) and LamRA (Liu et al., 2025), though trained exclusively on image-text data, have demonstrated the ability to generalize to text-video retrieval tasks in zero-shot manner. These advancements highlight the ongoing efforts to develop models capable of effectively understanding and representing the complex temporal and semantic information in video data.
5.3 Visual Document Representation Learning
Visual document representation learning has become increasingly vital for tasks such as document retrieval and retrieval-augmenned generation. Traditional text-based models often struggle to capture the rich visual and structural information present in documents, necessitating approaches that integrate both visual and textual modalities. One notable advancement is ColPali (Faysse el al.), which leverages vision-language models to enhance document retrieval efficiency by effectively capturing both textual and visual features. In the realm of retrieval-augmented generation, VisRAG (Yu et al., 2024) establishes a vision-based RAG pipeline that directly embeds documents as images using vision-language models, thereby preserving the original document information and outperforming traditional text-based RAG systems. Similarly, ViDoRAG (Wang et al., 2025) introduces a multi-agent framework tailored for complex reasoning across visual documents, employing dynamic iterative reasoning process to enhance retrieval and generation tasks. Furthermore, benchmarks like MMT.ongBncc--Do (Ma et al., 2024) have been developed to assess long-context document. understanding with visualizations, providing comprehensive evaluation framework for multimodal models.
5.4 Unified Modality Representation Learning
Unified modality representation learning aims to build a single model capable of processing information across diverse data types, such as text, images, audio, and video. Some methods, like Uni-Retrieval (Jia et al., 2025), leverage multimodal large language models (MLLMs) and prompt-tuning to accommodate a varioty of querics, achieving strong performance on universal benchmarks. While other recent studies have achieved remarkable results on diverse text and image tasks (Zhang et al., 2024; Thirukovallrru et al., 2025; Xiao et al., 2025; Jiang et al.), these models were not designed to unify image, video, and visual document retrieval within a single framework. Our latest work, VLM2VEC-V2, is the first to address this specific challenge. Following its (-Fmbeeding Gheed-Fmmeddinng-tee
release, subsequent efforts such as TTE (Cui et al., 2025) and have proposed advanced techniques and demonstrated strong performance on our MMEB-V2 benchmark.
6 Conclusion
We introduced MMEB-V2. V2, a comprehensive benchmark for evaluating multimodal embedding models across text, image, video, and visual document modalities. Alongside it,, we proposed VLM2VEC-V2, a strong bascline trained via contrastive learning across a diverse range of tasks and modality combinations. Our extensive experiments demonstrate the effectiveness of VLM2VEC-V2 and the diagnostic value of MMEB-V2.
7 Broader Impact and Limitations
Multimodal embedding models are incrcasingly used in real-world systems such as document retricval, multi- modal search, and content recommendation, where unified representations across heterogeneous modalities enable more accurate and efficient retrieval of relevant content. While our framework advances multimodal em- bedding lcarning, the MMEB-V2 benchmark primarily constructed from English-language data and therefore may not adequately evaluate multilingual or cross-lingual capabilities. In addition, MMEB-V2 focuses on visual modalities and does not cover other important modalities such as audio, limiting its ability to assess truly
10

---

Published in Transactions on Machine Learning Research (02/2026)
universal multimodal representations. We view MMEB-V2 as first step toward comprehensive multimodal evaluation and encourage future work to extend its coverage to additional languages and modalities.
References
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pp. 5803 5812, 2017.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau; Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. In First Conference Language Modeling.
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset, 2022. URL https://arxiv wrg
Boqi Chen, Anuj Khare, Gaurav Kumar, Arjun Akula, and Pradyumna Narayana. Seeing beyond: Enhancing visual question answering with multi-modal retrieval. In Proceedings of the 31st International Conference on Compulalional LÅnguislics: Industry Track, pp. 410 421, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL atttps//accanntholo0e
org/2025. cowing industry. 35/.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguisties; human language technologies, pp. 190-200, 2011.
Xuanming Cui, Jianpeng Cheng, Hong-you Chen, Satya Narayan Shukla, Abhijeet Awasthi, Xichen Pan, Chaitanya Ahuja, Shlok Kumar Mishra, Qi Guo, Scr-Nam Lim, et al. Think then embed: Generative context improves multimodal embedding. Xir preprinl arxiv:2510.0000 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: A large-scale hicrarchical image database. 2009 IEFE conference on computer vision and poltern recognition, pp. 248 255. Teee, 2009.
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mie: The first-ever comprehensive evaluation benchmark molti-modal ilms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14108-24111 2025.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language
qucry. In Proceedings of the IEEE international conference on computer vision, pp. 5267-5275, 2017.
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. Scaling deep contrastive learning batch size under memory limited setup. Xiv preprint arXi::200106698, 2021.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyíska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller- Muelter-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The 'something something' video database for learning and evaluating visual common sense, 2017. URL /ttps://arxiv.
abbs/1706.042.
Yanhao Jia, Xinyi Wu, Hao Li, Qinglin Zhang, Yuxiao Hu, Shuai Zhao, and Wenqi Fan. Uni-retrieval: A multi-style retricval framework for stem's oducation. ACL, 2025.
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. V1m2voc: Training vision- language models for massive multimodal embedding tasks. In The Thirteenth International Conference on Learning Representations.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Denso-captioning events in videos. In Proceedings of the JEEE international conference on computer vision, pp. 706-715, 2017.
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: A large video database for human motion recognition. In 2011 International Conference on Computer Vision, pp. 2556 2563, 2011. doi: 10.1109/ICCV. 201 1.612663
Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the synlax and semantics of goal-directed human activities. In 2014 IEEE Conference Computer Vision and Pottern Recognition, pp. 780-787, 2014. doi: (0.109/CCVP..0011
Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. Llave: Large language and vision embedding models with wordness-weighten contrastive learning. arXiv preprint arXiv:2503.04811 2025.
Jie Lei, Tamara Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:11846 11868, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- -magg pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888 12900. PMLR, 2022.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, al. Mvbench: comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CK /CFF Conference on Computer Vision and Pattern Recogrition, pp. 22195 22206, 2024.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 440- Springer, 2014.
Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint arXio:1907.1844 2019.
Yikun Liu, Yajie Zhang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4015-4025, 2025.
Huaishao Luo, Lci Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXie preprint ur Xiv:22104 08860, 2021.
Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-doc:: Benchmarking long-context document understanding with visualizations. Advances in Neural Information Processing Systems, 37:95963 96010, 2024.
Quentin Macé, António Loison, and Manuel Faysse. Vidore benchmark v2: Raising the bar for visual retrieval. arXiv preprint ar Xrv:22505 17166, 2025.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: diagnostic benchmark for very long-fore form-formmao video language understanding. Advances in Neural Information Processing Systems, 36: 46212-46244, 2023.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of- the-arrtth--tttt parameter-effiiien fine-tuning methods. https://githu: com/hhugiingfacc/ppftt 2022.
Rui Meng, Ye Liu, Shafiq Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embedding22: Ad- vanced text embedding with multi-stage training, 2024. URL https: sttps://huggingfaace co/Salesforce/ SFR-Embedding-227
12

---

Published in Transactions on Machine Learning Research (02/2026)
Antoine Miech, Dimitri Zhukov, Jean- lean-Baptiste Alayrae, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 2630 2640, 2019.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer Imaeeto-ssentend models. In Proceedings of the IEEE international conference on computer vision, pp. 2641 2641-2649, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8748-8763. PMLR, 2021.
Seed- E-eeeddmn--eeeaaat team. Geeddl..emmeddiing 17.
github io/. Accessed: 2025-06-
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsoureing data collection for activity understanding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 11-14, 2016, Proceedings, Port I 14, pp- 510 526. Springer, 2016.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012. URL attps://arxie org/abs/////ooooooo.ooo.eee/ar// hatss//22...a 12122.002...
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: finettteo text embeddings. In Findings of the Association for Computational Lènguistics: ACL 2023, pp. 1102 1121, 2023.
Raghuveer Thirukovalluru, Rui Meng, Ye Liu, Mingyi Su, Ping Nie, Semih Yavuz, Yingbo Zhou, Wenhu Chen, Bhuwan Dhingra, et al. Breaking the batch barrier (b3) of contrastive learning via smart batch mining. arXiv preprint Xrx:::250 1 11203, 2025.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint arxiv:2401.0 20000000,, 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vasion-language model's 's perception of the world at any resolution. Xrx preprint 110:::209.1219,,, 2024b.
Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengium Xie, and Feng Zhao. ViDoRAG: Visual document retrieval-angmented generation via dynamic iterative reasoning agents. In Christos Christodoullpoulles, Tanmoy Chakraborly, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 9124 9145, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-4 2025. 464. URL htctp:://aclanthooogyy org/2025.
doi: 10.18633/v1/ wttp:///////.......mm...... emnlp-main.......ee 4644//
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for Ande-and--languee research. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 4581-4591, 2019.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Chenting Wang, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. ar preprint 1oXi::4408.15977 2024c.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In ECCV (87), 2024.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IFEE/CVF conference on compuler vision and pattern recognition, pp. 9777-9786, 2021.
Zilin Xiao, Qi Ma, Mengting Gu, Chun-cheng Jason Chen, Xintao Chen, Vicente Ordonez, and Vijai Mohan. Metaembed: Scaling multimodal retrieval at test-time with ficxible late interaction. arXiv preprint Xrxiv:2509. (0t:2509.18095, 2025.
Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. EMNLP, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-52966 2016.
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Videococa: Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXtw:2112.0497 2022.
Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-angmented generation on multi-modality documents. arviv preprint 2rXiv:2410.10500 2024.
Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In Proceedings of the European conference on computer vision (ECCV), pp. 471-487, 2018.
Huaying Yuan, Jian Ni, Yueze Wang, Junjie Zhou, Zhengyang Liang, Zheng Liu, Zhao Cao, Zhicheng Dou, and Ji-Rong Wen. Momentseeker: A comprehensive benchmark and strong baseline for moment. retrieval within long videos. arXiv preprint Xrv::22002 12558, 2025.
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of vidco large multimodal models from language model reward. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Humon Language Technologies (Volume 1: Long Papers), pp. 694-717, 2025.
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang, Gme: Improving universal multinodal retrieval by multimodal 1lms, arXiv preprint ar XrXie:2412. 16855, 2024.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the A AT conference on artificial intelligence, volume 32, 2018
14

---

Published in Transactions on Machine Learning Research (02/2026)
A Appendix
A.1 Details of Baseline Models
VLM2 (Jiang et al.) converts vision-language models (VLMs) into the embedding models capable of handling diverse tasks. It reformulates all tasks as eetruuctoo-fooloving ranking problems. Using contrastive learning and task-specific instructions, VLM2VEC learns to produce fixed-dimensional embeddings aligned across modalities.
ColPali (Faysse et al.) leverages a vision-lagguage model trained to generate high-quality multi-vector embeddings from document page images. Combined with late interaction matching mechanism, it achieves strong performance on visual document retrieval tasks.
LamRA (Liu et al., 2025) explores the use of large multimodal models (LMMs) for retrieval, unifying diverse retrieval lasks under single framework without lask-specific fine-tuning. It achieves this by employing two-stage training- language-only pretraining followed by multimodal instruction tuning to enhance retrieval effectiveness.
GME (Zhang el al., 2024) is a unified multimodal embedding model finetuned from Qwen2 VL. It supports retrieval across single-modal, cross-modal, and fused-modal settings. GME is trained via contrastive learning using a diverse set of multimodal pairs including text, images, and image-text combinations.
A.2 Details of Benchmark Construction
A.2.1 Video Retrieval
MSR- VTT (Xu et al., 2016) is a dataset composed of 10K open-domain videos, each video clip ranging from 10 to 32 seconds in length and accompanied by a total of 200K captions. Following JSFusion (Yu et al., 2018), we sampled 1K clip-text pairs to incorporate into our benchmark. The query side contains both the instruction and the video caption, while the candidates consist of all 1K videos.
DiDeMo (Anne Hendricks et al., 2017) consists of 10K videos collected from Flickr, each trimmed to a maximum of 30 seconds. Each video inchudes approximately 3 to annotated pairs of descriptions and their corresponding distinct moments. Following previous work (Liu et al., 2019; Luo et al., 2021), we concatenate these descriptions and perform "paragraph-too-vdeer retrieval on this benchmark. The official test split, which contains 1,004 paragraph-video pairs, is used.
MSVD (Chen & Dolan, 2011) contains 80K English descriptions for 1,970 YouTube videos, each ranging from 1 to 62 seconds in length. Each video is annotated with approximately 40 sentences. We use the official test split, which includes 670 videos, and select one sentence per video to construct 670 test cases.
YouCook2 (Zhou et al., 2018) consists of 14K video clips sourced from 2K instructional cooking videos on YouTube. Each vidco contains multiple actions performed by the chef, accompanied by corresponding textual descriptions and temporal annotations. Each video clip is extracted and annotated with single sentence. We follow the common practice (Miech et al., 2019) of using the validation split and removing videos that also appear in How To100M. Different papers may report slightly varying numbers of test cases, typically ranging from 3.1K to 3.3K. Our benchmark includes 3,179 clip-text pairs from YouCook2.
VATEX (Wang et al., 2019) contains 41,250 video clips sourced from Kinetics-600 dataset and 825K sentence- level descriptions. The public test set originally contained 6K videos. However, since many of them have been removed or to private and are no longer accessible online, we use only subset of 4,468 available videos. For each video, we select one description to include in our benchmark.
A.2.2 Moment Retrieval
QVHighlights (Lei et al., 2021) is a dataset comprising 10K videos collected from YouTube, covering a diverse range of topics. Each video annotated with high-quality labels for both query-based video moment retrieval and highlight detection. In our embedding benchmark, we adopt the standard practice of ranking
15

---

Published in Transactions on Machine Learning Research (02/2026)
candidate clips and evaluating performance using Recall@1. In contrast, the QVHighlights paper and some other Vision-Tanguage Models like InternVideo2 (Wang et al., 2024c) evaluate models using Recall@0.5 and Recall@0.7 with Intersection over Union (IoU) as threshold, a metric that is not well-suited for embedding-bascd approaches.
Charades-STA (Gao et al., 2017), derived from the Charades (Sigurdsson et al., 2016) dataset, includes sentence-level temporal annotations for approximately 10K videos. Unlike its predecessor, Charades, Charades- STA replaces annotated action types with temporal sentences that describe actions. To minimize ambiguity in candidate clips, we created filtered subset of the Charades-STr. test set by applying condition that selects videos where the relevant segment occupics less than one-third of the total vidoo length.
MomentSeeker (Yuan et al., 2025) is a dataset designed to benchmark multimodal relrievers on long video moment retrieval tasks. Containing 1.6K queries, MomentSeeker consists of 4 subtasks with various query-side modalities. Additionally, MomentSocker spans diverse range of topics, including egocentric vidcos, cartoons, sports, and movies. For each query, we uniformly sampled nine negalive clips and included all the ground truth clips as positive examples.
A.2.3 Video Classification
Sinetics-700-2021 (Carreira al., 2022) is made up of approximately 648K Youtube video clips, covering wide range of human actions, around 700 labels in total, such as cooking, driving, and drawing. Each video clip lasts 3 seconds average. We sampled 1K video answer pairs from the validation set into benchmark. The candidate texts are the list of all the labels. The raw video data are retrieved from CVD Foundation Github.
Something Something v2 (Goyal et al., 2017) is the updated version of the Something Something v1 dataset. Tt consists of 220K crowd-source videos focusing on the physical interactions between humans and objects, with an average length of 4.03 seconds and a total of 174 action classes. We randomly sampled 1000 videos-text pairs from the validation split into our benchmark. The candidate texts are the list of all action classes. The raw video data are retrieved from Qualcomm.
HMDB51 (Kuehne et al., 2011) is composed of 6K video clips, including both movies and web videos, with 51 action labels, such as catch, drink, and kick. We sampled 1K frames-texi pairs from the test splits into our benchmark. The candidate texts are all labels. The raw video data are retrieved from the official website.
Breakfast (Kuchne et al., 2014) contains around 1.9K crowdsource video clips in the wild, more than 70 hours of total length, which about preparing for 10 different types of breakfast, such as cereal, milk, pancakes, and fried eggs. There are 6 different camera viewpoints, and we only selected the clips filmed with camera 01, ignoring those filmed by other cameras. We used all the vidco clips of camera 01, around 433 samples in total. The candidate texts are the 10 types of breakfast. The raw video data are retrieved from the official website.
UCF101 (Soomro et al., 2012) is an open domain video data set consisting of approximately 13K videos with 101 action categories, such as applying makeup, sports and playing instruments. We sampled 1K clip-text pairs from test splits into our benchmarks, and the candidate texts are all the action categories. The raw video data are retrieved from the official website.
A.2.4 Video QA
While embedding models are not primarily designed for open-ended visual question answering, QA tasks offer valuable way to assess whether a model can effectively understand visual inputs for different downstream purposes. They also enable fair comparison between embedding- based and generation-based approaches. To this end, we select multi-choice QA benchmarks that span a wide range of task types and are relatively less dependent on knowledge or reasoning abilities. We retain the original dataset configurations to ensure compatibility with prior work and facilitate direct comparison with existing models.
MVBench (Li al., 2024) is a comprehensive benchmark designed to evaluate multi-modal large language models on video understanding, with a particular focus temporal understanding. It defines 20 video tasks
16

---

Published in Transactions on Machine Learning Research (02/2026)
covering wide spectrum of temporal abilities from perception to cognition by transforming static tasks into dynamic, multiple-choice QA formats.
Video-MME (Fu et al., 2025) is a full-spectrum benchmark for evaluating Multi-modal Large Language Models (MLLMs) on video understanding. It consists 2,700 manually annotated QA pairs based on 900 videos (totaling 254 hours). It ensures broad scenario coverage and captures diverse temporal dynamics by including variety of video types and durations.
NExT-QA (Xiao et al., 2021) is video question answering benchmark focused on causal and temporal reasoning in untrimmed daily activity videos. It supports both multiple-choice (what we use) and open-ended QA formats. In our study, we utilize only the multiple-choice portion to evaluate models' ability to reason about complex action dynamics and object interactions.
EgoSchema (Mangalam et al., 2023) a diagnostic benchmark for long-form video understanding, constructed from Ego4D and comprising over 5,000 multiple-choice QA pairs spanning more than 250 hours of egocentric video. Each question is grounded in 3-minute clip and largets long-range temporal reasoning. In our study, use subset of 500 questions for which answer annotations are publicly available.
A.2.5 Visual Document Retrieval
ViDoRe (Faysse et al.; Macé et al., 2025) benchmark designed evaluate document retrieval systems. The first version (v1) includes 10 subtasks. The dataset originates from two sources: (1) for academic tasks, it repurposes widely used visual question-answering (VQA) benchmarks, trcating cach question as query and the corresponding page the gold document; (2) for practical tasks, publicly accessible PDF documents are collected, and queries relevant to document pages are generated using Claude-3 Sonnet.
To address the saturation of the original ViDoRe benchmark, ViDoRe-v2 introduces more realistic and challenging retrieval lasks, including four new diverse and multilingual datasets.
VisRAG (Yu al., 2024) serves the test set for the VisRAG pipeline, which assesses multimodal retrievers in document retricval. This benchmark consists of six subtasks adapted from VQA datascts, with filtering process applied to exclude context-dependent queries unsuitable for retrieval.
ViDoSeek (Wang et al., 2025) is a large-scale document collection qnestion-answering dataset originally designed to evaluate relrieval-auurmenteed generation (RAG) performance requiring complex reasoning. We adapt it for retrieval by using questions as queries and reference pages as gold images, with each query linked to relevant images from a collection of approximately 5,000 images. The datasct covers diverse content types, including text, charts, tables, and structured layouts.
MMLongBench-Dos (Ma et al., 2024) is long-context multimodal VQA benchmark containing 1,082 expert-annotated questions. Unlike previous VQA datascts, it is built on 135 lengthy PDF documents, averaging 47.5 pages each. To ensure comprehensive evaluation, questions require evidence from multiple sources (text, images, charts, tables, and layout structures) and locations (e.g., specific page numbers). We repurpose this dataset for retrieval, treating questions as queries and evidence pages as gold images, with each query linked to relevant images from collection of approximately 6,000 images.
A.3 Detailed Scores
17

---

Published in Transactions on Machine Learning Research (02/2026)
CoiPali v1.3
GME28
GME-B
LamRA-Qwer2
LamRA-QuesiZ.
VLMIN60-28
VIMM2
Aug. (78 taske)
46.0
54.1
07.8
40.4
47.0
62.3
Aug. Image (36 tiaks, HitR))
34.0
64.1
50.7
65.6
64.9
Ang Video (18 tasks. Hits1)
28.2
351.6
38.1
35.0
35.6
28.6
33.7
$1.8
Ang Vindoc (24 taslos, NDCGB5)
75.8
72.7
75.2
23.9
50.2
31.6
J8.4
05.4
CLS (10)
60.3
51..1
57.1
59.2
51.7
58.7
02.7
62.9
IQQ (10)
11.5
20.9
311.7
26.3
34.1
19.3
56.1
56.3
LRET (12)
48.1
08.9
71.2
70.0
06.9
65.0
631 4
029.3
LVG (d)
60.3
36.5
50.33
62.7
50.7
72.9
81.22
77.3
V-CLS (5)
20.7
34.9
37.4
39.3
32.9
15.4
30.
38.33
V-QA (6)
37.8
42.0
50.4
42.6
42.6
30.5
30.0
$4.3
V-RET-(G) (G)
21.5
25.16
28.4
24.3
28.2
20.6
29..0
225.8
V-MIt (5)
25.5
31.1
37.0
32.8
37.2
30.7
38.0
96.5
VI-Viiore-V1 [10)
83.6
26.1
80.1
22.0
56.3
40.5
56.0
76.5
Vi-Vidor--V V2 (4)
52.0
54.0
55.6
11.5
33.3
13.5
9.4
44.9
VD VieBLAG (6)
$1.1
62.5
85.0
37.4
56.2
51.8
50. 1
79.4
VD-00D (4)
45.1
13.11
41.1
21.0
10.1
33.5
36.1
39.4
Ies-g-Nes-1K
42.4
08.3
64.6
72.3
68.9
12-5
80.1
80.8
Neds
25.5
50.1
50.5
61.3
29.8
13.7
71.
72.9
HaeefuiMMemes
50.8
52.5
53.6
19.0
61.3
58.3
68.1 7
56.3
09.8
70.9
90.1
78.7
74.3
H01
NU77
27.5
35.8
55.3
$5.9
ImageNett
16.9
25.8
47.0
50.9
58.1
47.4
ImageNNe
61.8
78.6
80.5
68.5
77.0
84.7
73.9
89.5
Oet
45.5
70L6
1880
66.4
08.
37.1
40.1
05.2
Country211
6.0
26.5
21.
28.3
21.7
21.5
20.6
25.2
OK VIJA
0.6
20.5
33.2
87.8
30.9
88.5
56.6
51.5
A-OKVQA
6.6
18.6
21.0
27.0
34.1
39.5
47.
43.6
11.5
78
41.4
22.3
37.
82.5
80.7
00.
InfogrephiosSQA
5.0
11.6
90-8
16.5
23.7
47.7
60.0
08.8
CharoQA
5.7
11.7
12.9
Vi
23.3
ScienerQA
16.3
27.3
26.3
20.7
38.2
VizWis
27.6
37.0
39.0
52.0
32.0
38.0
30.0
65.5
GQ3A
8.5
76.1
76.6
38.5
67.4
68.5
66.1
61.9
TextQA
15.8
40.8
8000
41.1
153.3
71.
722
Vist
41.2
48.1
60.8
61.3
62.6
74.3
81.0
82.7
CIRR
8.2
44.2
54.5
61.7
44.7
46.8
61.1
57.5
VieaalNews
50.1
71.7
70.7
70.4
70.1
73.1
80.5
74.5
ViaudNNows
47.6
7X-8
63.6
83.9
74.2
73.7
75.2
59.2
G3-11
71-2
72
65.7
73.6
77.1
75.3
MSCOCO 12x
63.1
S7.
73.7
71
NIGHTS
65.5
06.5
WebQA
53.8
58.5
91.
81.0
85.7
85.0
88.3
90.6
Mushionl))
3.9
32.9
37.8
42.0
333-4
14.0
17.1
19.5
WIKLSS-NQ
80.5
70..
78.2
69.7
67.0
14.02
62.3
06.9
OVEN
50.0
72.3
75.1
82.0
64..8
08.3
60.5
61.5
EDIS
64.7
91.8
96.0
85.9
78.7
81.2
85.7
84.1
XSSOCC
38..7
25.16
31.4
44.8
36.00
196.5
77.77
67.1
RefO000
64.5
35.6
60.0
62.8
67:1
80.0
87.6
87
RolfttoCO0-Maaeeinnn
3.0
723-3
78.4
76.7
75.7
84.6
85.8
VisualTW Potnting
58.1
06.5
67.3
51.2
08.3
81.0
K700
25.4
51.4
35
58.0
Sutth-mmthV2
25.1
25.9
30.66
96.3
20.9
312
42.8
HMDDB11
26.8
47.9
40.5
25.8
13.8
42.0
40.9
ICF110
49.4
12.4
54.7
60.4
63.0
67.5
61.8
00.0
Brosisfett
10.9
11.66
113
16.9
20.1
13.4
20.6
14.5
MVBBeach
55.7
37.5
46.6
37.2
37.6
30.5
28.5
55.7
Vidco-MME
30.8
24.3
202-2
34.1
25.1
26.30
27.8
2017
NESTQQA
35.2
30.5
60.6
45.7
44.9
20.5
20.3
20.9
Fgy Sichemm
35.4
40.8
46.8
44.8
47.0
25.4
21.8
$4.0
Ae
51.3
58.2
39.6
DiDeMo
22.8
26.4
22.8
MSR-VTT TT
27.3
22.1
25.2
34.6
25.5
MSVD
65.4
47.6
40.7
16.1
11.9
38.2
46.7
48.1
VATEX
16.7
23.0
24.9
10.1
18.7
16.2
25.5
26.5
YouCook22
5.3
7.8
1.1
92.2
7.5
4.1
0.0
10.6
QVHighlightt
19.0
43.6
50.5
63.8
60.9
46.2
67.7
49-4
CharacessS TA
29.0
14.9
14.0
10.9
18.8
15.6
19.8
20-2
MomeesSSeker
29.3
37.1
39.3
35.9
33.3
35.4
41.0
42.9
ViDoRe arricg
81.7
80.8
86.0
10.8
80.6
doergg
58.8
SAL1
57.5
19.1
27.0
ViDoRe tabdqiad
86.9
00.3
02.6
ViDolte tatdg
70.9
60.9
71.1
11.4
25.9
19.5
27.6
ViDolte shiftprojett
75.1
80.5
96.5
12.0
27.3
11.8
36.6
00.8
ViDoRe actifiiall intellgennee
95.7
97.6
1986
10.3
72.0
56.01
67.7
88.5
ViDolte energy
96.7
91.9
98.3
24.8
65.2
19.1
60.4
86.5
ViDeRt government reporta
93.6
94.6
98.8
16.4
72.2
67.1
61.8
85.0
ViDolte bealtisate Indiustry
95.9
96.7
98.3
25.9
80.8
59.6
68.9
92.2
ViDofte esg. reports. Jimman Jabeled v2
51.5
61.0
63.1
7.6
35.0
12.6
6.8
45.6
biomedical lestures v2 indtiingudd
54.7
5410
45.5
13.4
7.4
ViDolte exxmomiie reports anildingual
ViDolte osg_ reporte v2_ inllliingg
52.0
55.4
5.9
ViaRAG ArsivQAA
80.9
82.0
2.0
41.8
VisRAG ChariQA
78.2
57.9
711.2
$4.4
ViRRAG MP-DoeVVQA
86.8
84.4
80.2
33.4
64.5
43.2
02.6
718
ViRRAG SEdAVQA
95.0
90.1
91.5
56.5
76.5
76.0
72.6
91.5
ViRRAG IntoVQA
85.7
91.1
90.5
56.3
73.
70.7
72.0
85.7
ViRRAG PlaqA
61.3
G4.1
63.4
34.6
41.2
28.4
04.4
66.
ViDoSeek--agr
22.2
21.6
23.3
11.3
23.1
17.7
22.
21.9
ViDeSeniedie
85.7
83.6
83.9
37.1
80.3
76.3
80.2
MeuLongeendh page
9.61
11.8
MMMoomDBenhh doc
52.
27.6
32.6
40.5
43.7
Table 6: Performance comparison of various models on the full MMEB-V2 benchmark, covering 78 tasks across image, vidco, and visual document modalities. Numbers in parentheses represent the task count for cach category.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Rank
Model
Size (B)
Overall
Image
Video
VisDoc
seed1 Soedllleeeeeddin-PPee. 1215
75.59
77.99
67.74
77.88
WeMMM-Fabedding-S11
8.77
71.42
78.09
63.21
77.30
3
IFM-TTE-7B
8.29
74.07
77.90
59.19
79.48
WeMM--Emeedding22222
2.13
71.67
76.08
58.67
74.81
5
RzenEmbed-v2277 7B
8.29
71.61
75.92
55.73
77.06
Beedl-.--mmbedding
71.27
77.78
55.34
73.44
7
RrenEmbeed-v1-BB
8.29
68.88
73.60
48.87
76,80
Ops-MMM-enbberddnn--11777
8.29
67.61
72.72
53.76
70.34
9
UME-R1-77
8.29
64.50
71.25
47.50
67.13
RxenFmbee-x1122B
2.21
64.36
68.53
42.62
74.41
11
Ops-UMM-emmboddin-----35
2.21
63.44
69.03
47.56
66.96
12
ImtrrsddMM-URR-CARR--77
8.03
60.63
67.56
42.40
63.92
13
UME-R1-2B
2.21
60.11
66.56
42.23
63.86
14
JuiMBB-VLLLLaVA--Oeeevi---
8.03
59.56
71.77
39.01
56.68
VLM2Vee- V2..- 22.teee-22..-00e2222-----
2.21
58.02
64.85
34.58
65.36
Table 7: Leaderboard results on the MMEB-V2 V2 benchmark as of December 2025.
Scores are reported for overall performance as well as for modality-speciffe subsets, including Image, Video, and VisDoc.
A.4 Updated Results on the MMEB-V2 Benchmark
Since the release of the MMEB series benchmarks, they have become widely adopted as standard evaluations for MILM enboddings. As of December 2025, more than models have been evaluated on MMEB-V V1 and over 30 models on MMEB-V2. In this appendix, we include additional results from several recent state-of-the-art multimodal embedding models evaluated the MMEB-V V2 benchmark.
19

---

Published in Transactions on Machine Learning Research (02/2026)
Training Data Configuration
Image A
Video A
VisDoc A
Global A
Image 200K (Baseline)
Image 100K ColPali 100K
-.774
-0.02
+7.76
+1.88
Image 100K VideoCap 100K
-2.5
-0.89
-0.36
-1.48
***
Image 100K + ColPali 50K + VideoCap 50K
-1.55
**
+1.09
-6.69
s*8
+1.59
**
All Sources 200K
-2113
-0.4
+10.19
+1.96
Table 8: Task-level significance evaluation of different modality mixing strategies compared to the Image 200K baseline. Listed values indicate the mean performance change across 78 MMEB-V2 tasks, with posilive values reflecting consistent gains. Statistical reliability is assessed using paired t-tests: < 0.05, << 0.01, *** p < 0.001.
A.5 Significance Evaluation of Modality Mixing Strategies
To rigorously assess the generalization impact of multimodal training configurations (Section 4.5.1; Table 4), conduct task-level paired t-test to evaluate statistical significance. Instead of treating individual queries independent observations- which would artificially inflate significance due large sample counts treat each dataset (Lask) one independent sample (N 78 lasks). For each ablation model M, compule a paired difference vector D = [ScoreM,1 Score ScooeM,,N Score comparing its performance to that of the Imagc-only baseline on a per-task basis. We then tost the null hypothesis that the mean of D is zero, where the sign of the mean difference reflects improvement or degradation, and the p-value indicates whether the effect is statistically reliable across the full task distribution.
The significance results in Table 8 confirm the key observations from Section 4.5.1. In particular, effective modality mixing requires both high-quality data and modality compatibility: adding curated VisDoc supervision (e.g., ColPali) leads to statistically significant gains, whereas incorporating noisier video supervision (VideoCap) can negatively affect performance. Furthermore, the two full Image + VisDoc + Video mixtures deliver the strongest global improvements, highlighting that broader modality coverage is most beneficial when additional modalities are carefully selected and curated.
20

---

Published in Transactions on Machine Learning Research (02/2026)
A.6 Example Cases
What happened to the vase afterward?
VLM2VeC-V1
VLM2VeC-V2
Figure 3: Example of improved video understanding in VLM2VEC-V2 over VLM2VEC.
whose door are the cartoon sponge and the cartoon
starfish knocking in the picture?
VLM2Vec-V1
VLM2VeC-V2
Figure Example of improved video understanding VLM2VEC-V2 over VLM2VEC.
21

---

Published in Transactions on Machine Learning Research (02/2026)
what percentage of parents want to receive text messages from there child teacher?
Shepherds
Social Media In School
Friendly
121 say not boing able accessa social problem.
43%
55% mmessages from their child teacher school
57%
46% %
teacheral have pintted idea
increase from 2010 proference!
P Pinterest
PARENTS GUIDE
TEACHING CHILDREN ABOUT MONEY
of administrators say they socall communicate with parents and Twitter effective tool for those communicsations.
51 %
(grades 121
77%
Tech Tools in Dream School
sard social media tools are uttimato dedimition
they can connect and callabbral nate others.
teaching safety good manners, but with money few know where start.
parents
Only in parents and teachers agree.
Trowarch
serntt and helpbry on timn unddrrtendeeeeee movey on cely aer horm veroge
Nowyor rppii-t
Crariing rengere tonraio andi supporte anneadiv congueann Rockiand Wraad Deseniiee
and Schiadic Labrs about slanch Geaings fron
to
X
VLM2VeC-V1
VLM2Vec-V2
Figure 5: Example of improved document understanding in VLM2VEc-V2 VLM2VEC. The images are cropped for clearer presentation.
Which the second most popular social media job title according to the pie chart?
SOCIAL MEDIA &Or RECRÜITMENT
SOCIAL MEDIA JOB TITLES Breakdown
16
Misson
STRATBOIST
27 %
14
SOCIAL MEDIA fOr HIRING
ALL TIME HIGH
Switter
MARKETING
MANAGER
facebook
7%
30 %
Linked
5%
df
Pintered
MENAGER
COMPANIES SOCLIS SECRES RECRUTTING
LIN ICS
MANAGER
WHITER
SOCIAL MEDIA
X
VLM2Vec-V1
VVM2Vec-V2
Figure Example of improved document understanding VLM2VEC-V2 over VLM2VEC. The images are cropped for clearer presentation.
22


Reviews:
[{"review_id": "kLMmZXVuGN", "reviewer": "Reviewer_CKXA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new framework for learning embeddings for a variety of different visual forms and introduces an extended benchmark that adds five new task types to an existing benchmark. The paper reports the results of experiments on both the new tasks and the original benchmark tasks. The results demonstrate that the proposed approach outperforms (or performs competitively with) existing methods and performs well on the new task types.", "claims_and_evidence_yn": "No", "claims_explanation": "It is claimed in the abstract and introduction that the paper:\n(1)\tproposes a framework (VLM2Vec-V2) for learning embeddings across diverse visual forms.\n(2)\tintroduces an extended benchmark that adds five new task types to an existing benchmark.\n(3)\tdemonstrates through extensive experiments that VLM2Vec-V2 achieves strong performance on the newly introduced video and document retrieval tasks and improves over prior baselines on the original image benchmarks. \n(4)\toffers insights into the generalizability of various multimodal embedding models.\n(5)\thighlights effective strategies for unified embedding learning.\n(6)\tlays the groundwork for more scalable and adaptable representation learning in both research and real-world settings.\n\nClaim 1 is established through the introduction of a framework for fine-tuning a VLM. Claim 2 is established through the extension of the benchmark. Claim 3 is supported by extensive experiments, although the analysis and discussion of the experimental results could be much more thorough. \n\nIt is less clear that the paper satisfactorily achieves claims 4-6 and the claims should probably be moderated, phrased carefully, or made precise. The discussion of the main results is very brief and it is difficult to extract from the paper the “insights into the generalizability of … models”. Most of the commentary is dedicated to observing that the introduced model is competitive or outperforms other methods. \nThe paper provides an ablation analysis that studies performance on various modalities when the model is trained on different (combinations of) modalities, but there is no in-depth analysis of these results either. Claim 6 seems unsupported since there is no evidence that the results extend to “real-world” settings.", "audience_yn": "Yes", "audience_explanation": "The paper addresses an important area and makes an interesting contribution, contributing both an expanded benchmark with new tasks as well as the results of a new framework.", "requested_changes": "(1)\tThe title indicates that this is V2 of VLM2VEC. With this in mind, it is important to clarify what the main changes are compared to VLMVEC. The related work glosses over VLMVEC, mentioning it only in passing. While the paper doesn’t need to dwell on this too much, it is important for a reader to understand what have emerged as critical changes with the incorporation of videos and related tasks. The changes to the contrastive learning and data formatting appear minor, but more than half of the text in Section 3 is dedicated to them.   \n\n(2)\tThe discussion of the main results (Section 4.2) is very brief and does not amount to much beyond “our model performs well”. The paper would be strengthened by a considerably more detailed analysis. Insights into how some of the baseline models performed on particular tasks would be useful (when and why do they fail; when do they do well). Are there tasks where the introduced model performed slightly less well? If so, is there an understanding as to why this might be the case? Within the tasks themselves, do models tend to have the same failure cases (e.g., in MRET are >55 percent of the tasks failed by all models) or do the successes vary between models? There should be many ways to delve into the experimental results. \n\n(3)\tThe ablation analysis results are interesting, but the experiment results are again collapsed to a single table without any nuanced or in-depth analysis. Counterintuitive results are not explained or discussed (why is performance on video the worst when the training source is video?). There are no confidence intervals provided, nor are is there any statistical significance testing, so it is not clear that the reported differences are meaningful. There are concerns that the results are not genuinely a reflection of the modality of the data but perhaps more the amount, quality, and variety of the data. It’s also unclear why Video training data would be so useful for VisDoc (considerably better than VisDoc itself).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6tlj5iJykz", "reviewer": "Reviewer_5ybX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors provide two contributions to the development of multimodal models.\n1. They builds a new datasets named MMEB-V2, aiming to provide an extensive benchmark for a variety of multimodal tasks. This new dataset is not obtained from the creation of new instances, but by extending the MMEB benchmark with tasks borrowed from other datasets in the literature. Still, a consequent work has been done to unify all datasets.\n2. They provide a new multimodal model, named  VLM2Vec-V2, that provides a unified treatment for learning embedding intended to be used to solve downstream multimodal tasks (including videos, images and documents). They compared the accuracy of this new model to several baselines on the proposed MMEB-V2 benchmark, resulting in an extensive and convincing study.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I would like first to state that I don't feel qualified to conduct this review, as I am not aware of the current state of Large Multimodal Models. \n\nThat said, I acknowledge that the empirical evaluation of VLM2Vec-V2 is extensive, and the results are convincing. I appreciate the presence of a proper ablation study.\n\nOne caveat is that the introduction announces more fundamental answers than the paper provides:\n> Through VLM2Vec-V2 and MMEB-V2, we aim to investigate the following research questions: How well can a multimodal embedding generalize across diverse visual modalities? What are the key ingredients for training robust and versatile multimodal embedding models? What are the key challenges in representing temporal information in videos and structured information in visual documents?\n\nThe last two questions are not explicitly answered in the manuscript.", "audience_yn": "Yes", "audience_explanation": "I am not certain of my assessment here, but I guess researchers working on Large Models and Multimodalities would benefit from the proposed extended dataset (MMEB-V2) and a baseline for comparing future approaches (VLN2Vec-V2)", "requested_changes": "I consider all the following adjustments important:\n1. It isn't easy to know what the difference is between the VLM2Vec-V2 model and previous models, especially the changes made to the first version (VLM2Vec). This information should be clearly stated. Related to this, I would like to assess the contributions at a more fine-grained level: which components of the model or training strategies are new?\n2. The result tables provide \"performance\" metrics, but I could not find the precise metrics for each task.\n3. Section 5.4 suggests that the proposed models and benchmarks have been released before submitting the paper to TMLR, and other models have been tested in the meantime (TTE and Seed1.6-Embedding). These results should be included in the manuscript.\n4. More information on the resources needed to train the VLM2Vec-V2 model. Section 4.1.2 states that experiments were performed on 8 NVIDIA H100 GPUs. How long does it take to train VLM2Vec-V2? How does it compare to baseline models? \n5. The bibliography needs to be proofread to include the references to published works (accepted after a peer-reviewed process) in place of their arXiv counterparts.  \n     - BehnamGhader et al. (2024) was accepted at COLM 2024 \n     - Faysse et al (2024) was accepted at ICLR 2025\n     - Jia et al. (2025) was accepted at ACL 2025\n     - and so on", "broader_impact_concerns": "The manuscript does not contain any broader impact statement. As multimodal models are intended for deployment across a variety of real-world contexts, I think the work deserves a statement on their societal impact (positive and negative). A discussion of the potential shortcomings of the MMEB-V2 dataset's content would also be of interest.", "additional_comments": "If I understand correctly, the datasets included in the MMEB-v2 benchmark are completely distinct from the ones used to train the embedding model VLM2Vec-V2. How did the authors decide which dataset to include in the benchmark and which ones to include in the training dataset? Are we confident that the baseline approaches have not been trained on datasets included in the benchmark?", "extra_scores": {}, "extra_text": {}}, {"review_id": "bTgYlDv25g", "reviewer": "Reviewer_Lus6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new dataset (MMEB-V2) and model (VLM2VEC-V2) for multimodal embeddings. MMEB-V2 builds upon prior work by incorporating new datasets spanning five new tasks focusing on video and visual document understanding. The proposed model outperforms other relevant multimodal embedding baselines (of a similar size) across a wide variety of 78 tasks.\n\n**Strengths**:\n- The scope of datasets, tasks, modalities, etc. included is very impressive. This is an all-encompassing effort with diverse downstream applications.\n- Experiments are very thorough, spanning many tasks of all varieties. Competitive, relevant baselines are chosen as well.\n- The style of writing is generally straightforward and clear. \n\n**Weaknesses**:\n- This reads more like a technical report than a research paper necessarily. There are often instances of informal language and brief/cursory discussions of concepts.\n- Unless I am missing this, there is no mention of code or data access. For this type of paper, details about code/data release will be very important to determine its utility.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Experiments demonstrate the utility of the proposed model against relevant baseline multimodal embedding models across a wide variety of tasks.", "audience_yn": "Yes", "audience_explanation": "This paper will surely be of interest to the TMLR readership, particularly if it is accompanied by the open-source release of code, data, and/or model weights. The idea of an openly accessible multimodal embedding model spanning so many different tasks and modalities will be highly appealing to certain readers.", "requested_changes": "**Main feedback**\n- Ensure that each paragraph has a dedicated purpose and is fully fleshed out. If claims are made, they should always be substantiated by relevant citations when possible. E.g., is there a reference that can support a claim like the following? \"The lack of coverage causes the existing embedding models to fall behind on many realistic tasks like article searching, website searching, Youtube video search, etc.\"\n- Comment on the availability of code, data, and model weights.\n- How do these general multimodal embedding models compare, e.g., to “specialized” models in individual tasks and domains? This would be helpful with regard to understand the utility of such a general approach.\n- What is the unique value proposition of VLM2Vec-V2? I see that it outperforms other 2B-parameter models, but is on par with GME 7B, for example. Practically speaking, what is the difference from an end user’s perspective between using a 2B and 7B model? Make this value proposition very clear upfront, even though it is stated across several different places in the paper.\n\n**Minor feedback**\n- Abstract: “These datasets fail to cover broader forms of visual information like documents, pdf, websites, videos, slides, etc. The lack of coverage causes the existing embedding models to fall behind on many realistic tasks like article searching, website searching, Youtube video search, etc.” I would avoid repeating “etc.” multiple times; e.g., could write “tasks such as article searching…”\n- Sec 3.2: “where tau is a temperature”. I would add “hyperparameter” to the end of this\n- Sec 3.2: Example of informal tone: “we feed query and target into it”", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)