VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
Authors: Ziyan Jiang
Rui Meng
Xinyi Yang
Semih Yavuz
Yingbo Zhou
Wenhu Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Vision Language Model
Representation Learning
Multimodal Embeddings
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We transform vision-language models into strong multimodal embedders, achieving 10–20% gains on MMEB, a new benchmark covering 36 datasets.

Abstract: Embedding models play a crucial role in a variety of downstream tasks, including semantic similarity, information retrieval, and clustering. While there has been a surge of interest in developing universal text embedding models that generalize across tasks (e.g., MTEB), progress in learning universal multimodal embedding models has been comparatively slow, despite their importance and practical applications. In this work, we explore the potential of building universal multimodal embeddings capable of handling a broad range of downstream tasks. 

Our contributions are twofold: (1) we propose MMEB (Massive Multimodal Embedding Benchmark), which covers four meta-tasks (classification, visual question answering, multimodal retrieval, and visual grounding) and 36 datasets, including 20 training datasets and 16 evaluation datasets spanning both in-distribution and out-of-distribution tasks, and (2) VLM2Vec (Vision-Language Model → Vector), a contrastive training framework that transforms any vision-language model into an embedding model through contrastive training on MMEB. 

Unlike previous models such as CLIP and BLIP, which encode text and images independently without task-specific guidance, VLM2Vec can process any combination of images and text while incorporating task instructions to generate a fixed-dimensional vector. We develop a series of VLM2Vec models based on state-of-the-art VLMs, including Phi-3.5-V, LLaVA-1.6, and Qwen2-VL, and evaluate them on MMEB’s benchmark. With LoRA tuning, VLM2Vec achieves a 10% to 20% improvement over existing multimodal embedding models on MMEB’s evaluation sets. Our findings reveal that VLMs are surprisingly strong embedding models.

Published as a conference paper at ICLR 2025
VLM2VEC: TRAINING VISION-LANGUAGE MODELS FOR MASSIVE MULTIMODAL EMBEDDING TASKS
Ziyan Jiang' Rui ,eng Langg,Semile Yavuz?, Yingbo Zhou?, Wenhu Chen' University of Waterloo, "Salesforce Research
ziyanjiang528 @ gmail. com, ruimeng @ salesforce.com, wenhuchen@uwaterioon
https://tiger lab. io
ABSTRACT
Embedding models play a crucial role in a variety of downstream tasks, including semantic similarity, information retrieval, and clustering. While there has been surge of interest in developing universal text embedding models that generalize across tasks (e.g., MTEB), progress in learning universal multimodal embedding models has been comparatively slow, despite their importance and practical appli- cations. In this work, we explore the potential of building universal multimodal embeddings capable of handling broad range of downstream tasks. Our contri- butions are two fold: (1) we propose MMEB (Massive Multimodal Embedding Benchmark), which covers 4 meta-tasks (i.e. classification, visual question an- swering, multimodal retrieval, and visual grounding) and 36 datasets, including 20 training datasets and 16 evaluation datasets covering both in-distribution and cattof-ddisrrbbution tasks, and (2) VLM2VEC VVisionLLanguage Model - Vector), contrastive training framework that converts any vision- vision-language model into an embedding model via contrastive training on MMEB. Unlike previous mod- els such CLIP and BLIP, which encode text and images independently without task-specific guidance, VLM2VEC can process any combination of images and text while incorporating task instructions to generate develop a series of VL VLM2VSc models based on statc-of-the-arr VLMs, including Phi-3.5-V, LLaVA-1166, and Qwen2-VL, VL. and evaluate them on MMEB's bench- mark. With LoRA tuning, VLM2VEC achieves 10% to 20% improvement over existing multimodal embedding models on MMEB's evaluation sets. Our findings reveal that VLMs are secretly strong embedding models.
sieedddimessional vector. We
INTRODUCTION
Embeddings, or distributed representations, encode inputs (whether text or images) as Exed-dimensiona. vectors, enabling range of downstream tasks. Since the advent of Word2Vec (Mikolov] (013) and GloVe (Pennington et al. 2014), substantial research efforts have focused on learning textual embeddings (Kiros et al.} [015: Conneau al. (217)) and image em- beddings (Radford et al. 2021 Li et al. [202]] Jia et al. 1 Yu et al. 2022). These embeddings facilitate variety of applications, including textual and visual semantic similarity (Agirre et al.
2012
Marelli et al., 2014 Chechik et al.
2010 Cer et al. 2017). information retrieval (Mitra et al.,
2017
Karpukhin et al.] 2020; Lin al.
2014), automatic evaluation (Zhang et al. 2020 Sellam
et al.
2020), prompt retrieval for in-context learning (Liu et al. 2022 Rubin et al. 2022 Hongjnn
al.
2022). and retrieval-augmented generation (Lewis et al.) 2020 Guu et al.) 2020; [zacard &
Grave
2020h. A recent shift in research has focused on developing universal embeddings that can
generalize across a wide range of tasks. For instance, Mucnnighoff et al. (003)) introduced MTEB (Massive Text Embedding Benchmark) to comprehensively assess text embeddings across tasks such as classification and clustering. MTEB has become the standard for evaluating universal text em- beddings. Recent works (Wang et al. 2022a (Su et al.,, 2023: Wang et al.) 2024a SSrinnger al.) 2024 BBehnamGhader et al.) (024) have demonstrated promising results on the MTEB benchmark. However, progress in multimodal embeddings has been relatively slower. Despite advancements
WWork done during internship at University of Waterloo in collaboration with Salesforce Research. Cor- responding authors are Ziyan Jiang, Rui Meng and Wenhu Chen

---

Published as a conference paper at ICLR 2025
Reproued the glve mage The menu po mest, above lamb.
agumert glve image
hal under the stanci the eft
Classification
Vaual Quesiion Answering
dlons porlon
irage
provils exdence
quiitinn and
ryll sandwich, hall
What the namn of this placel
Voual Grounding
Rebrieual
VLM2Vec
MMEB
The Titsee is lake
Württenbenng
Foods
rapkin
We develop universal multimodal embedding benchmark, MMEB, along with VLM2VEC, an embedding model adapted from vision-language models (VLMs). VLM2VEC capa- ble of following instructions and performing various multimodal embedding tasks, accommodating any combination of image and text modalities.
Figure 1:
in text embeddings, the lack of both benchmarks and methodologies in the multimodal embedding domain remains challenge.
Current research in multimodal embeddings faces two primary limitations: (1) existing studies typ-
ically evaluate visual embeddings on isolated tasks, such as ImageNet classification (Deng et al,
2009 Hendrycks et al. 2021a b) MSCOCO/Flickr retrieval (Lin et al. 2014 Plummer et al.
(2) most existing models, such as CLIP (Radford et al. (021). BLIP (Li et al. 2022), and SigLiP (Zhai al. 2023), either process text and images separately or perform shallow fusion of visual and textual information (Wei et al., 2023), limiting their ability to fully capture the relation- ships between text and image modalities. Furthermore, these models exhibit limited reasoning and generalization capabilities, particularly in zero-shot scenarios for complex reasoning tasks.
2015:
In this paper, we attempt to build an universal multimodal embedding framework to pave road for the future research, which consists of two efforts:
We introduce novel benchmark, MMEB (Massive Multimodal Embedding Benchmark), which includes 36 datasets spanning four meta-task categories: classification, visual question an- swering, retrieval, and visual grounding. MMEB provides a comprehensive framework for training and evaluating embedding models across various combinations of text and image modalities. All tasks are reformulated as ranking tasks, where the model follows instructions, processes a query, and selects the correct target from set of candidates. The query and target can be image, text, or a combination of both. MMEB is divided into 20 in-distribution datasets, which can used for training, and 16 out-of-distribution datasets, reserved for evaluation.
-MEE::
VLM2VEC: We adopt the pre-trained vision-language models like Phi-3.5-V (Abdin et 2024) and LL. (Li et al. 2024) as the backbone for VLM2VSC. In contrast to other multimodal embedding models like UniRR (Wei al. (203) and MagicLens (Zhang et al. 2024), which rely on late fusion of CLIP (Radford et al.) (021) features, our approach leverages the deep integration of vision and language features within a transformer architecture. There are several advantages to this approach: (1) VLMs are trained on massive multimodal datasets and can handle any combi- nation of images and text, as well as high-resolution images and long text inputs; (2) vision and language features are deeply fused in the transformer model, improving the model's ability to cap- ture cross-modal relationships; and (3) these models are well-suited for generalizing across diverse capabilities. These factors make VIM2VE an ideal choice for task generalization. We trained VLM2VEC on the 20 MMEB training datasets using contrastive learning and compared its performance with various baselines.
tasks, particularly those requiring instruction-folloviin
Following extensive contrastive training, VLM2VEC can handle any combination images and text, producing -imessiimasionnl vectors. We evaluate V1.M2VRC against a wide array of mul-
2

---

Published as a conference paper at ICLR 2025
timodal embedding models, including CLIP (Radford et al. 2021 BLIP2 Li SigLIP (Zhai et al. (023). MagicLens (Zhang et al. 2024), UniR (Wei et al. 2023) and ES- Jiang et al. 2024), demonstrating consistent improvements across all task categonies. Notably, compared to the best baseline model without fine-tuning, our model achieves a 21.1 point improvement (from 44.7 to 65.8) across all 36 MMEB datasets and 16. 1-point increase (from 41.7 to 57.8) 16 out-of- distribution datasets for zero-shot evaluation. Compared to the best baseline model with fine-tuning, our model achieves 18.6 point improvement (from 47.2 to 65.8) across all 36 MMEB datasets and a 14.7-point increase (from 43.1 to 57.8) on 16 out-of-distribution datasets for zero-shot evaluation. Moreover, as a general multimodal representation model, VLM2VIC с can still achieve competitive zero-shot T2I (Text-to-lmage) and 12T (Image-to-Texx) performance on Flickr30K compared to ex- isting CLIP-like models, as presented in Table
al. 2023a
2 МMEB: A BENCHMARK FOR MULTIMODAL EMBEDDINGS
2.1 DATASET OVERVIEW
We present MMEB (Massive Multimodal Embedding Benchmark). comprehensive benchmark designed to evaluate multimodal embeddings across a diverse set of tasks. MMEB consists of 36 datasets organized into four meta-tasks: classification, visual question answering, retrieval, and visual grounding. Each task is reformulated as ranking problem, where the model is provided with an instruction and query (which may consist of text, images, or both) and is tasked with selecting the correct answer from set of candidates. These candidates could be text, images, or additional instructions. The datasets are divided into two categories: 20 in-distribution datasets for training and 16 cut-of-distribution datasets for evaluation. We report performance metrics across all 36 tasks. An overview MMEB provided Figure 2 and the dataset statistics are provided in Table T
The embedding models are supposed to compress the query side into vector and the target can- didates into a set of vectors. The candidate with the highest dot-product will be selected as the prediction for evaluation. We measure the Precision @ I to reflect the percentage of top candidate matching the groundtruth. For the number of target candidates, a higher count could increase eval- uation costs and hinder rapid model iteration, while lower count might make the benchmark too simple and prone to saturation. To strike a balance between these extremes, we have chosen 1,000 candidates. Further details about this decision can be found in Section[...
MMEB offers a wide range of tasks from various domains, such as common, news, Wikipedia, web, and fashion. The benchmark incorporates diverse combinations of modalities for both queries and targets, including text, images, and text-image pairs. Additionally, tasks are designed to follow different types of instructions. For instance, tasks may involve object recognition (e.g., "Identify the object shown in the image."), retrieval (e.g., "Find an image that matches the given caption.") "), or visual grounding (e.g., "Select the portion of the image that answers the question." "). Examples for The diversity in MMEB makes it an
each dataset in MMEB are provided in Tables 7 8 and 10
ideal testbed for universal embeddings.
2.2 META-TASE -AAS AND DATASET DESIGN
MMEB is organized into four primary meta-task categories:
Classification The query consists of an instruction, an image, optionally accompanied by related text, while the target is the class label. The number of candidates equals the number of classes.
Visual Question Answering The query consists of an instruction, an image, and a piece of text as the question, while the target is the answer. Each query has 1 ground truth and 999 distractors as candidates.
Information Retrieval Both the query and target sides involve combination of text, images, and instructions. Each query has ground truth and 999 distractors as candidates.
Visual Grounding The category is adapted from object detection tasks. The query combines an instruction (e.g., "Select the portion of the image that isolates the object of the given label: red apple") with the full image. This instruction guides the model to focus specific object within the image. Each candidate corresponds to cropped regions (bounding boxes) of the image, including both the object of interest and distractor regions. Each query includes 1,000 candidates: ground
3

---

Published as a conference paper at ICLR 2025
Table 1: The statistics of MMEB: 36 datasets across meta-task categories, with 20 in-distribution datasets used for training and 16 out-of-distributte datasets used exclusively for evaluation.
Mcta-Task
Dataset
Query
Target
OOD?
#Training 100K
#Eval
#Candidares
ImageNet-IK
1000
1000
N24News
1+T
49K
1000
24
HatefolMemes
8K
1000
VOC2007
8K
1000
20
Classification
SUN397
T
20K
1000
397
(10 Tasks)
56ace3665
1000
ImageNet-
T
1000
1000
ImageNer-R
1000
200
ObjectNet
1000
Country-211
T
1000
211
OK-VQA
9K
1000
1000
V-O
17K
Doc
40K
1000
1000
Infographic VQA
24K
1000
1000
VQA
ChartQA
28K
1000
1000
(10 Tasks)
Visual
70K
1000
1000
ScienceQA A
1000
1000
VizWiz
1000
1000
GQA
1000
1000
TexIVQA
1000
1000
VisDial
123K
1000
1000
CTRR
26K
1000
1000
VisualN ews-12i
1OOK
1000
1000
VisuallN ewx 121
TOOK
1000
1000
MSCOCO-22
100K
1000
1000
Retrieval
MSCOCO-12
I13K
1000
1000
(12 Tasks)
NIGHTS
16K
1000
1000
WebQA
17K
1000
1000
OVEN
1000
1000
FashionIQ
I+1
1000
1000
EDIS
1000
1000
Wiki-SS-NQ
1000
1000
MSCOCO
100K
1000
1000
Visual Grounding
VisualWW Pointing
1000
1000
Tasks)
RefCOCO
1000
1000
ReICOCO-Mathing
1000
1000
truth and 999 distractors. These distractors may include hard negatives from the same object class, other objects in the image, or random objects from different images.
Further details on dataset processing can be found in Section A.
3 VLM2VEC: TRANSFORMING LVMS TO EMBEDDERS
3.1 CONTRASTIVE TRAINING
We develop VLM2VEC, contrastive training framework designed convert any state-of-the-art vision-language model into an embedding model, as illustrated in Figure 3 A relevant query-targe pair denoted as (g. t+t) Both q and t could be either single image, text or single image + text. We define q (gt, 9:) and & (17 (
We then apply the instruction to the original query g to generate a new one Tunst
fins [IMAGE_TOKE] Instruct: (task_defiittion 'nn Query: {q}
(1)
is a placeholder for a one-sentence description the embedding task. To enhance the embedding model's generalizability by better understanding instructions, we have
where
crafted spec-fc instructions, as shown in Tables 7 g 9
and
I0
Given pretrained VLM, feed query and target into it to obtain the query and target embeddings (hama h ) by taking the last layer vector representation of the last token. To train the embedding model, we adopt the standard InfoNCE loss L. over the in-batch negatives and hard negatives:
@(h h
min C log Φ (hama h, )
E
@b hr )
t- t-EN
4

---

Published as a conference paper at ICLR 2025
MMEB: Massive Multimodal Embedding Benchmark Classification
VQA
VOC2007
N24News
SUN397
OK-VQA
A-OKVQA
DocVQA
ImageNet-1K
HatefulMeees
InfoVQA
ChartQA
Visual7W
ObjectNet
Country211
ImageNet-A
ImageNet--R
TextVQA
VizWiz
Retrieval
Visual Grounding
MSCOCO -I2t
MSCOCO -t21
VisDial
MSCOCO
RefcOCo
VisuelNews
VisualNews
NIGHTS
WebQA
RefCOCO-Matthing
Visual7W-Pointing
Wi
FashionIQ
OVEN
EDIS
Figure 2: An overview of the tasks and datasets in MMEB. MMEB includes four meta-tasks and 36 datasets: 20 in-distribution datasets used for training and out-of-distribution datasets used exclusively for evaluation.
where denotes the set of all negatives, and o(b, h,) is function that computes the matching score between query q and target In this paper, adopt the temperature-scaled cosine similarity function as <(h, h₂) = exp(= cos(hy, h,) where 7 is temperature hyper-parameter.
h.))
3.2 INCREASING BATCH SIZE THRROUGI GRADCACHE
Since hard negatives often difficult ambiguous collect for most multimodal datasets, using larger batch sizes becomes crucial. This increases the number of in-batch random negatives, which in turn helps improve the performance of the embedding model.
A bottleneck lies the GPU memory that limits us from increasing the batch size and the number of in-batch random negatives during training, as each training instance may include one image (ei- ther from the query or target side) or multiple images (from both query and target sides), resulting in substantial memory consumption. We apply GradCache (Gao et al. 2021a), gradient caching tech- nique that decouples backpropagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension.
Mathematically, supposed we have large batch of queries Q. and we divide it into set of bab-batches, each of which can fit into memory for gradient computation: Q [Q,, Q2.... There are two major steps: "Representation Gradient Computation and Caching" and "Sub-batch Gradient Accumulation". First, gradient tensors within each subbatch is calculated and stored: ac V, = af(t))
Dc u, af(g))
Then gradients are accumulated for encoder parameters across all sub-batches:
7c = Σ Σ
Of(g) u 20
+ ΣΣ 0f(t,) V
20
d0
(3)
QEEQ
TETTEE
5

---

Published as a conference paper at ICLR 2025
LAST
Contrastive
LAST
Projection
Projection
Image Encoder
VLNAVee
Image Encoder
VLM22aes(Targe)
Represent the given news image with the following captoo domain classilication.
Foods
point
Figure 3: VLM2VEC uses a VLM as the backbone to deeply integrate image and text features. trained with contrastive loss between the query and target, following task-specific instructions. The training data consists of diverse combinations of modalities on both the query and target sides, which may include images, text, or image-text pairs.
EXPERIMENTS
In this section, we adopt Phi-3.5-V (Abdin al. 2024). LLaVA-1.6 (Li et al. 2024) and Qwen2- VL-7B-Instruct (Wang et al. 2024b) as the backbone VLMs, with training conducted via either full model fine-tuning or LoRA. The temperature for the loss function is set to 0.02, with a batch size of 1,024, maximum text length of 256 tokens, and 2K training steps. The LoRA variant uses a rank of 8. For VLM2VEC leveraging Phi-3.5-V as the backbone, we configure the number of sub-image crops to 4. For VLM2VEC using VA-166 and Qwen2-VL as the backbone, resize the input images to a uniform resolution, employing two setups: high-resolution configuration of 1344 1344 and low-resolution configuration of 336 336. For the 20 training datasets, randomly select up to 100K data points. When using GradCache, we set butcbatch size accordingly, with the total batch size accumulated 1,024. All experiments were run on H100 GPUs. We report Precision @ for all models in Table 2 It measures the ratio of positive candidates being ranked in the top place for all queries.
4.1 BASELINES
Four groups of baselines are reported this study.
CLIP-family: We utilize vision/language encoders such CLIP (Radford et al. 2021), Open- CLIP (Cherti et al. 2023), SigLIP (Zhai et al. 2023). and BI.IP2 (Li et al. 2023a) as our baseline. queries target text in certain tasks may
Due to the length limitations of the text encoder, be truncated. We apply score-level fusion by combining multimodal features using element-wise addition with equal weights (w) w2 = 1). We do use instructions, as they could potentially degrade performance. For more details, please refer to Section 4.3.4
UnilR WWi et al. 2023) is unified, instruction-guidee multimodal retriever designed to handle eight different retrieval tasks across multiple modalities. The model builds on CLIP and BLIP, employing shallow fusion techniques such as score-level and feature-level fusion to integrate modalities. In this study, we use the CLIP SF and BLIP FF variations as baselines.
UnilR:
MagicLens (Zhang al.) 2024 is self-supervised image retrieval model capable of handling open-ended instructions. It utilizes a dual-encoder architecture with shared parameters, initializing the vision and language encoders with either CoCa or CL.IP. The model uses a multi- head attention pooler to unify multimodal inputs into single embedding. For this study, we report results using the CLiP-Large backbone.
MagicLens:
E5-V: ES- V (Jiang et al. 2024) is a contemporary model that also leverages vision-language models for multimodal embedding tasks. It proposes a single-modality training approach, where the model trained exclusively on text pairs. In contrast, our model is trained on multimodal pairs, which include various combinations of image and text modalities on both the query and target sides.
For all our baselines, we first use their original versions. Additionally, we have fine-tuned both CLIP and OpenCIIP MMEB training datasets. We adopt the same experimental configurations

---

Published as conference paper at ICLR 2025
VLM2VEC to ensure a fair comparison. For the remaining baseline models, UnilR and MagicLens also utilize a shallow fusion approach based on CLIP models, with their primary contribution being the datasets they were trained on. E5-V proposes training exclusively text pairs, making it un- suitable for fine-tuning on our datasets. Therefore, we have not included the fine-tuned versions of these three models in this comparison.
4.2 MAIN RESULT
Table 2:
Results the MMEB benchmark. The scores are averaged per meta-task. For detailed scores per dataset, see T We include baselines with and without fine-tuning on MMEB training datasets and our models with three different backbones using batch size of 1,024.
Per Mela-Task Score
Average Score
Model
Classification
VQA
Retrieval
Grounding
IND
OOD
Overall
datasets
10
12
16
Baseline Models Fine-tuning MMEB Training)
CL.IP (Rdfordd al. 2021
42.8
9.1
53.0
51.8
37.1
38.7
37.8
BLIP2 err K0248
27.0
4.2
33.9
47.0
25.3
25.1
25.2
SigLIP (Zha) el 2026
40.3
31.6
38.0
OpenCLIP (Cherti
2023
47.8
52.3
53.3
39.3
40.2
39.7
UnilR (BLIPFFP) ([LIPPFF)]WWW [Wei
2023
42.1
15.0
60.1
62.2
44.7
40.4
42.8
UnilR (CLIP_SF) (Wer
2125
44.3
16.2
61.8
65.3
47.1
41.7
44.7
B5-VV (Jiang el 20/4
21.8
11.5
Magiclens /Zhang et
2024
38.8
8.3
35.4
26.0
31.0
23.7
27.8
Baseline Models (Fine-tuning MMEB Training)
CLIP OpenCLIP
55.2
19.7
53.2
62.2
47.6
42.8
45.4
56.0
21.9
55.4
64.1
50.5
43.1
47.2
Ours (V W2VFc
Phi-3.5-V, Full-model fine-tuned (#crop=4)
52.8
50.3
$7.8
47.4
Phi-3.5-V, LoRA (#crop=4)
54.8
54.9
62.3
79.5
52.0
60.1
LLaVA-116, LoRA (low-res)
54.7
50.3
56.2
64.0
61.0
47.5
55.0
LLaVA-1.6, LoRA (high-res)
61.2
49.9
67.4
86.1
67.5
57.1
62.9
Qwen2-VI. VI., LoRA (high-res)
62.6
37.8
A Best baseline (No Fine-tuning)
+17.9
+41.6
+8.1
+25.1
+25.1
+16.1
+21.1
Best baseline (Fine-tuning)
+6.6
+35.9
+14.5
+22.0
+21.7
+14.7
+18.6
From Table 2 the best variant of V1.22VRC leverages Qwen2-VI VI., is trained with LoRA, and pro- cesses input images at relatively high resolution of 1344 1344. It achieves an average preci- sion of 65.8% across all 36 datasets from MMEB. Additionally, maintains an average preci- sion @ of 57.8% 16 out-of-distribution tasks in zero-shot evaluation, suggesting strong general- ization ability. These results indicate that when trained on datasets spanning diverse task categories, domains, and modality combinations, VLM2VEC effectively follows instructions to align visual and textual representations while generalizing well to unseen tasks.
Furthermore, V1.22VRC using Phi-3.5-V and LLL also outperforms baseline models. No- tably, LLaVA-1.6 has a transparent pre-training data recipe with minimal overlap with MMEB's OOD datasets, reinforcing that the strong zero-shot performance of VLM2VEC not due prior exposure of its backbone to the OOD datasets. When using the same backbone, the full fine-tuning variant achieves slightly lower scores than the LoRA version. For detailed discussion comparing full fine-tuning and LoRA, please refer to Section 4.3.1
Compared to other baseline models, with or without fine-tuning on MMEB training data, our model demonstrates consistent improvements. Compared to the best baseline model without fine-tuning, our model achieves 21.1 point improvement (from 44.7 to 65.8) across all 36 MMEB datasets and 16.1 16.1-ppint increase (from 41.7 to 57.8) on 16 out-of-distributiin datasets for zero-shot evaluation. Compared to the best baseline model with fine-tuning, our model achieves a 18.6 point improvement (from 47.2 65.8) across all 36 MMEB datasets and 14.7-point increase (from 43.1 to 57.8) on 16 out-of out-of--istribbn distribution datasets for zero-shot evaluation. Additionally, unlike the baseline models, which fail to demonstrate reasonable performance across all different task categories, VIM2VEc achieves strong performance across all four meta-task categories. This highlights its capability to handle wide range of multimodal embedding tasks effectively.

---

Published as a conference paper at ICLR 2025
4.3 RESULT ANALYSIS
To train an effective and generalizable multimodal embedding, various factors need to be considered, ranging from the data the training setup. In this section, we present detailed ablation studies these factors. We will discuss two training setups: Full Fine-Tuning vs. LoRA, along with Training parameters, and two topies related data: Meta-task generalization and Impact of instructions.
4.3.1 FULL FINE- TUNING VS. LORA
When fine-tuning the VLMs, key decision is whether to conduct full fine-tuning, which updates all parameters in the model, or to use a parameter-efficiet method such as LoRA. We compare the performance of fully fine-tuned VLM2VEC with its LoRA variants different ranks. The train- ing and data setups are kept consistent across all models. We observe that LoRA achieves better performance when the rank appropriately configured.
We compare the performance of fully fine-tuned VLM2VEC with its LoRA variants different ranks. LoRA can achieve better performance when the rank appropriately configured. All the models utilize Phi-3.5-V as their backbone.
Table 3:
Meta-Tase Task Average Score
Average Score
Model
Classification
VQA
Retrieval
Grounding
IND
OOD
Overall
# of datasets
10
10
12
4
20
16
36
Full Fine-Tuning (bs=256)
50.4
46.4
52.6
68.6
57.9
44.7
52.0
LoRA r= 4 (bs=256)
52.7
53.6
60.1
80.2
64.9
50.4
58.4
LoRA (bs=256)
52.9
52.5
60.3
80.0
64.2
50.8
58.2
LoRA = 16 (bs=256)
51.1
40.5
52.0
72.5
54,9
45.8
50.8
LoRA (bs=256)
50.6
47.8
53.9
72.5
46.5
53.4
4.3.2 TRAINING PARAMETERS
During our experiments, we identified three key parameters that significantly impact the perfor- mance of VLM2VEC: training batch size, the number of sub-image crops, and the number of training steps. In Figure 4 we observe that the final performance gradually improves as we increase the batch size, training step size, and number of sub-image crops. We particularly want highlight the impact of batch size. Due to the lack of hard negatives, using a large batch size with plenty of random negatives, supported by the GradCache technique, plays crucial role in enhancing the performance of VLM2VEC, discussed in Section 3.2
55
Performance (%)
54
Performance (%))
54
Performance
52
52
52
50
50
50
200
400
600
800
1000
2000
4000
6000
8000
10
15
Batch Size
Step Size
Number Crops
Figure 4: The figures demonstrate the influence of the training setup on VLM2VEd's final perfor- mance. Here, examine the effects of training batch size, the number of sub-image crops, and the number training steps. All the models utilize Phi-3.5-V V as their backbone.
4.3.3 META-TASK GENERALIZATION
We have demonstrated that VLM2VEC has the potential to transfer to out-of--istribution datasets after being trained diverse range of in-distribution datasets, with the instruction-ollooiing set-
8

---

Published as a conference paper at ICLR 2025
tings. An interesting question arises to whether focusing specific meta-task can enhance the model's 's overall generalizability. We have trained three models, each focused solely on one meta-task (classification, visual question answering, and retrieval). Visual grounding the limited number of training datasets. We then evaluated the models' transferability to other meta- tasks. We refer to these three models as V1,2VVR RET, trained on 8 retrieval tasks, VL.22VRC vQA. trained on 6 visual question answering tasks, and VLM2VEC CLS, trained on 5 classification tasks.
not included due to
Figure iillsttrate the generalizability of these three models on unseen meta-tasks. We could ob- serve that VLM2VEC RET has better generalizablilty on other meta-task, compared with other two models, especially on visual grounding categories. The reason is that retrieval tasks involve more diverse combination of text and visual modalities from both the query and target sides, which helps the model generalize better to unseen tasas This observation highlights the benefits of using more diverse tasks in the VLM2VFC C training process.
VLM2VeGvox VLMZVeGRET
VLMZVeGGaA Vs VLMZVeCau
VLM2Vecugr VLM2Vetas
60
VLM2Vettxx
60
VLM2Veevoa
VLMIVEOGTT
VLM2Verier
51.3
VL.M2Vecax
VLM2Vecuus
51.3
50
40
35.9
33.0 35.1
35.1
30
30
20
20
13.0
12.6
10
Classilicatio
Grounding
Retrieval
Grounding
VQA
Grounding
Figure 5: The figures show the generalization ability of models trained on one meta-task other unseen meta-tasks. For example, the first subplot compares the performance of VLM2VEC trained exclusively on VQA tasks with VLM2VEC trained exclusively on retrieval tasks across the other two meta-task categories: classification and visual grounding. Overall, VLM2VEc trained on retrieval tasks demonstrate better generalization ability. All the models utilize Phi-3.5-V as their backbone.
4.3.4 IMPACT OF INSTRUCTIONS
Previous studies have shown the influence of instructions addressing various tasks. VLM2VsC, which leverages a VLM as its backbone and is trained on large-scale datasets with instructions, expected better generalize across tasks and improve performance in multimodal embedding tasks. In this section, we evaluate the performance of both CLIP and VLM2VEC with and without task-specific instructions to quantify the impact of incorporating instructions into the embedding process. As shown in Table 4 incorporating instructions reduces the CLIP model's performance by 29.4 29.4%, while our VLM2VI achieves 49.4% improvement. This highlights our model's ability to follow instructions.
5 RELATED WORK
5.1 TEXT EMBEDDING
Text embeddings have demonstrated significant potential in powering downstream applications such as information retrieval (Karpukhin et al., 2020; Xiong et al.] 2020), text similarity (Gao et al. 2021b), prompt retrieval for in-context learning (Hongjin
2022). and classification (Lo- geswaran & Lee 2018 Reimers & Gurevych] 2019). Early work focused on creating effective embeddings for specific tasks. With the rise of pretrained language models, efforts have shifted to- ward developing universal embedding models capable of handling wide range of embedding tasks. Studies such as GTR (Ni et al. 2022) and (Wang et al. 2022a) leveraged large amounts of noisy paired data to pretrain and fine-tune dense retrievers. More recent works like TART (Asai et al. 2022) and InstructOR (Su et al. 2023) introduced natural language prompts to guide embedding models in producing tesk-relevant embeddings. Building this, models like E5Mistral (5Misttal/Waaggag et al. 2024a), Embedding(MMen) et al. 2024), RepLLaMA et al. 2024b). GTE-Q STTE-Qwen2(Li et al.
2023b), and NV-Embed (Lee al. 2024 have utilized pretrained large language models (LLLMs)

---

Published as a conference paper at ICLR 2025
Table 4: Comparison of CLIP and our VLM2VEC with and without task-specific instructions, Incor- porating instructions could decrease CLIP's performance by 29.4%, whereas our VLM2VEC achieves 49.4% improvement. VLM2VEC utilizes Phi-3.5-V as its backbone.
Meta- Task Average Score
Average Score
Model
Classification
VQA
Retrieval
Grounding
IND
OOD
Overall
# of datasets
10
10
12
20
16
36
CILP
w/o instruction
42.8
9.1
53.0
51.8
37.1
38.7
37.8
w/ instruction
17.4
8.0
41.3
52.9
23.8
30.3
26.7
-59.3%
-12.1%
-22.1%
2.1%
-35.8%
-21.7%
-29.4%
Ours (VLM2VEc)
w/o instruction
36.7
33.5
31.1
44.3
37.3
31.6
34.8
w/ instruction
50.4
46.4
52.6
68.6
57.9
44.7
52.0
37.3%
38.5%
69.1%
54.9%
55.2%
41.5%
49.4%
as their backhone, fine-tuning them with multi-task data and instructions. These models have de- livered significant improvements over earlier approaches that did not use LLMs for initialization or instruction tuning.
5.2
MULTIMODAL EMBEDDINGS
Multimodal embeddings have long been significant research challenge. Early works like CLIP (Radford al. 2021). BLIP (Li et al. 2022; 2023a), Align (Jia al. (021), SigLIP (Zhai et al. 2023), Sim VLM (Wang et al. 2022b) and CoCa Yu et al. 2022) primarily focused on learn- ing universal representations from large-scale, weakly supervised image-text pairs. These models generally encode images and text separately, projecting them into shared space. This approach has laid the groundwork for more recent multimodal models like LLaVA (Liu et al., 2024).
Most research on universal multimodal embeddings involves fine-tuning models like CLIP or BLIP, typically using simple fusion mechanisms to combine visual and language information. For instance, UnilR (Wei al.] 2023) creates multimodal embeddings by simply adding text and visual features, while MagicLens Zhhang et al. 2024) employs shallow self-attention layers to integrate these fea- tures more effectively. The study most similar is E5-V (Jiang et al. 2024), contemporary work that fine-tunes a languae model using only text training data.
5.3 EMBEDDING BENCHMARKS
Significant efforts have been made to develop benchmarks for evaluating retrieval systems. For text retrieval models, MS MARCO (Nguyen et al. 2016) and Natural Questions (Kwiatkowski et al., 2019b) are two of the most widely used benchmarks in general domains. To broaden the evaluation across more diverse domains, BEIR (Thakur et al.) was introduced, incorporating 18 datasets from various fields. Building on this, MTEB (Muennighoff' et al.) 2023) further expands BEIR's scope by adding more tasks, such classification, clustering, and semantic textual similarity (STS).
For multimodal retrieval, several benchmarks have been introduced to evaluate model performance across different modalities. MBEIR (Wei et al. 2023) includes 8 tasks and 16 datasets, designed to test models' ability to retrieve information based on various forms of queries and instructions.
CONCLUSION
In this paper, we aim to build the first large-scale multimodal embedding framework, comprising two main components: MMEB and VIM2VEC. MMEB includes 36 datasets across four meta-task categories, providing comprehensive and diverse framework for training and evaluating embed- ding models. V1.M2VRe leverages VLMMs backbone to deeply fuse visual and textual spaces, enhancing generalization to unseen tasks through instruction following.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Eneko Agine, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agire, SemEval- 2012 task 6: A pilot on semantic textual similarity. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret (eds.), *SEM 2012: The First Joint Conference on Lexical and Com- putational Semantics - Volume I: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop
Semantic Evaluation (SemEval 2012), pp. 385-393, Montréal, Canada, 7-8 June 2012. Association for Computational Linguis- org/ S12-1012550 1051
tics. URL https: btttps://aclaaate http///aaaatttooocoooo
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Han- naneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. arXiv preprint DrXiv:2211.09200 2022.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapa- dos, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:24004.0 009611, 2024.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Ju- rgens (eds.), Proceedings of the I 11th International Workshop on Semantic Evaluation (SemEval- 2017), pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. a anthol www.://-//111120 2001
doi: 1188655/11/1172202
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16495-16504, 2022.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11(3), 2010.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive aagguage-imagee learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670-680, 2017.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2266-35,,2 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Mark Everingham, S. M. Ali Eslami, Lue Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111:98- - 136, 2014. htRl/ttps:://// //api. semanticschol.a.
atte//orrpusDD:227555770
11

---

Published as a conference paper at ICLR 2025
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint 2tt.:23000003334, 2023.
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. Scaling deep contrastive learning batch size under memory limited setup. arXiv preprint arXiv:2101.088 100::2000.06883,, 2021a.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence
embeddings. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, 2021b.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A criti- cal analysis of out-of-distributio generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 15262-15277 2021b.
SU Hongjin, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations, 2022.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-We Chang. Open-domain visual entity recognition: Towards recogniz- ing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12065 12075, 2023.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6700-6709, 2019.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering, 2020. URL wwtp:oogg//aaboeo. /2007 2028
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International conference on machine learning, pp. 4916 PMLR, 2021.
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407. 12580, 2024.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, 2020.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798. 2014.
12

---

Published as a conference paper at ICLR 2025
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ring- shia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech multimodal memes. Advances in neural information processing systems, 13:2611-26222 2020.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. Advances in neural information processing systems, 28, 2015.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019a.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466 2019b.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Ny-embed: Improved techniques for training Ilms as generalist embedding models. arXiv preprint arXhv:2405.1742 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. setrieval-augmenter genera- tion for knowledge- knowledge-intensivy intensive nlp tasks. Advances Neural Information P Processing Systems, 33: 2020.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleavy Tackling multi- tultti-mmage, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07880 477900799, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-ima image pre- training for unified dision-languse understanding and generation. In International conference machine learning, pp. 12888-12904 PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- immge pre pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th Europeun Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Fuxiao Liu, Yinghan Wang. Tianlu Wang, and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning. arXiv preprint arXiv:2010.03744. 2020.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? DeeLlO 2022, pp. 100, 2022.
Siqi Liu, Weixi Feng, Tsu-jui Fu. Wenhu Chen, and William Yang Wang. Edis: Entity-driven image search over multimodal web content. arXiv preprint arXiv:2305. 2023.
Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-tree pre-trained vision-and- Dision-and-language and-language models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2125-2134, 2021.
13

---

Published as a conference paper at ICLR 2025
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence repre-
sentations. In International Conference on Learning Representations, 2018. URL https:
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Xueguang Ma, Sheng- Chichh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. arXiv preprint arXiv:2406. 11511, 2024a.
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2421-2425, 2024b.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Ninth International Conference Language Resources and Evaluation (4), pp. 216- 223, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA), URL http:// /www:///wwwww 1 Lree-conff http://ooo..oooooccceiiiiggoeeoo htt.ig/ssseeee11111 / Paper.pppp.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. Proceedings of the IEEE/cyf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographievga. In Proceedings the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Rui Meng, Ye Liu, Shafiq Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embedding-2: Advanced text embedding with multi-stage training, 2024. URL htttps://uuggingee
Tomas Mikolov, Efficient estimation of word representations in vector space. arXiv:1301.3781, 2013.
arXiv preprint
Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In Proceedings of the 26th international conference on world wide web, pp. 1291-1299, 2017.
Niklas Muennighoff. Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text em- bedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2014-2037, 2023.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. Novem- ber 2016. URL htLpst::::: http://www..cooooooooottttotttot ms mn-maaren rac-
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-We Wei Chang, et al. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9855, 2022.
14

---

Published as a conference paper at ICLR 2025
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting rgioon-to-hrras correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp- 2641-2649, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-BERT Sentence embeddings using Siamese BERT- networks, In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-JJCNLP), pp. 3982-399 3992, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10.18653/v//D19-. 1410. URL httppss//cclaathooooy...oo
D19-1410
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, pp. 2655-267577 2022.
Dustin Schwenk, Apoory Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. Bleurt: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Rep- etition improves language model embeddings. arXiv preprint arXi::2402. 15449, 2024.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: caaruction-finetune text embeddings. In Findings of the Association for Computational Linguisties: ACI. 2023, pp- 1102-1121, 2023.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15388 2023.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by sepely-supervised contrastive pre-training. arXiv preprint arXiv:2212.035533 2022a.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang. Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models. arXiv preprint arXiv:2401.00336 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing aision-lanuue model's perception of the world at any resolution. arXiv preprint 1rXiv:2409.22191 2024b.
15

---

Published as a conference paper at ICLR 2025
Zhen Wang, Xu Shan, Xiangxie Zhang. and Jie Yang. N24news: A new dataset for multimodal news classification. arXiv preprint arXiv:2108. 13327, 2021.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. In International Conference on Learning Reprexentations, 2022b.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. arXiv preprint vrXiv:2311.17136, 2023.
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad AI-Halah, Steven Rennie, Kristen Grauman, and Roge- rio Feris. Fashion iq: A new dataset towards retrieving images by natural language feedback. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp-
11307-11317, 2021.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-34444 3492. IEEE, 2010.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.000 000008 2020.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learn- ing Research, 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 11975-11986, 2023.
Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming- Wei Chang. Magiclens: Self-supervised image retrieval with open-ended instructions. arXiv preprint arXiv:2403. 19651, 2024.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluat- ing text generation with bert. In International Conference on Learning Representations, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answer- ing in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4995-5004, 2016.
16

---

Published as a conference paper at ICLR 2025
A DETAILS OF MMEB
In this section, we provide additional details about our proposed benchmark. MMEB (Massive Mul- timodal Embedding Benchmark). Section A. A.I outlines the specifies of the 36 datasets used in the MMEB benchmark. Section A.2 explains the process for determining the number of candidates in MMEB.
A.1 DATASET DETAILS
A.1.1 CLASSIFICATION
There are total of 10 datasets for classification tasks.
ImageNet-IK (Deng et al. 2009) The dataset large-scale dataset commonly used in image
classification, consisting of over million images across 1K different classes.
ImageNet-A (Hendrycks et al., 2021b) The dataset contains images from a distribution unlike the ImageNet training distribution. ImageNet-A examples belong to ImageNet classes, but the exam- ples are harder and can cause mistakes across various models. They cause consistent classification mistakes due to scene complications encountered in the long tail of scene configurations and by exploiting classifier blind spots.
ImageNet-R (Hendrycks al. 2021ab The dataset contains set of images labeled with ImageNet labels obtained by collecting art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paint- ings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video game renditions of ImageNet classes.
VOC2007 [verrighhm et al. (0114) The dataset focuses on recognizing objects in realistic scenarios and contains 20 object classes.
N24News (Wang al., 2021 The dataset is sourced from the New York Times and consists of 24 categories, with each news article containing both text and image information. The task is to classify the given news image and its accompanying text into one of the 24 categories.
HatefulMemes (Kiela et al. 2020) The dataset proposes a new challenge set for multimodal classi- fication, focusing on detecting hate speech in multimodal memes.
Place365 (Zoou et al. 2017) The dataset is a repository of 10 million scene photographs, labeled with scene semantic categories, comprising large and diverse list of the types of environments encountered in the world.
SUN397 (Xiao et al. 2010) The dataset is a dataset for scene recognition consisting of 397 cate- gories.
ObjectNet (Barbu et al. 2019) The dataset crowd-sourced test set of 50K images featuring objects in unusual poses and cluttered scenes, designed to challenge recognition performance. It includes controls for rotation, background, and viewpoint, and covers 313 object classes.
20u (Radford et al. 2021) The dataset is designed to assess the geolocation capability of visual representations. filters the YFCC100M dataset find 211 countries that have least 300 photos with GPS coordinates.
A.1.2 VISUAL QUESTION ANSWERING (VQA)
There total of 10 datasets for VQA tasks.
OK-YQA MMaiino et al. 2019) The dataset includes questions that require external resources for
answers.
A-OKVQA (Schwenk et al. 2022) The dataset is an augmented successor of OK- requiring broad base of commonsense and world knowledge to answer. The questions generally cannot be answered by simply querying a knowledge base, and instead require some form of commonsense reasoning about the scene depicted in the image.
17

---

Published as a conference paper at ICLR 2025
DocVQA (Mathew al. 2021) ) The dataset contains questions for document analysis and recogni- tion over document images of various types and content.
Infographics VQA (Mathew et al., 2022) The dataset comprises a diverse collection of infographics accompanicd by natural language question and answer annotations. The questions require methods capable of jointly reasoning over the document layout, textual content, graphical elements, and data visualizations.
2022) The dataset is designed for question answering about charts, with focus on visual and logical reasoning applied to real-world charts.
ChartQA (Masry et al.
ScienceQA (Lu et al. 2022) The dataset contains questions with diverse science topics and annota- tions of their answers with corresponding lectures and explanations.
Visual7W-telling (Zhu et al. 2016) The dataset establishes semantic link between textual descrip- tions and image regions through object-level grounding. It has two types of questions: "telling" and "pointing"" It leverages the six W questions (what, where, when, who, why, and how) to sys- tematically examine a model's 's capability for visual understanding through telling questions. Addi- tionally, a seventh "which" question appended for visual answers as pointing questions. We use "Visual7W-telling" in our VQA category and "Visual7W- Disuul/W-ppointee "WW-oontiing" in our visual grounding category.
VizWiz Gurari et al. 2018) The dataset originates from natural visual question answering sce- nario, where blind individuals captured images and recorded spoken questions about them, along with 10 crowdsourced answers for each visual question. For our task, we select only the answerable questions.
Tex (Singh et al. 2019) The dataset is designed to benchmark visual reasoning based on text within images. Models need to read and reason about the text in images to answer related questions.
GQA (Hudson & Manning, 2019) The dataset is designed for real-world visual reasoning and com- positional question answering. It uses real images from the Visual Genome dataset. Each image accompanied by scene graph annotations that describe the classes and attributes of objects in the scene, well their pairwise relationships.
A.1.3 RETRIEVAL
There are a total of 12 datasets for retrieval tasks.
VisDial (Das al. 2017) The dataset features dialogues created by Amazon Mechanical Turk workers. One worker takes the role of the "questioner", who only sees the text description of an image, while the other plays the "answerer", who has access the image. They engage in 10- round Q&A session about the image. We repurpose this dataset as a retrieval task, where the goal is retrieve the image based the given dialogue.
CIRR (Liu et al. 2021) The dataset is designed for the task of composed image retrieval. It consists of pairs of real-life reference and target images, along with modification sentence that describes the changes made between the two images.
FashionIQ (Wu al. (021)) The dataset contains images of fashion products with crowd-sourced descriptions highlighting the differences between these products. Similar to CIRR, FashionIQ can also be used for the task of composed image retrieval, where each test case consists of pair of reference and target images, along with modification sentence that describes the changes between the two images.
VisualNews (Liu et al. 2020) The dataset contains publicly available news image paired with cap- tions. We split this task into two setups: "VisualNews-i2t", which retrieves the caption given the news image and "VisualNews-(2i? which retrieves the news image given the caption.
MSCOCO (Lin et al. 2014) The dataset is a well-known image caption dataset. Similar to Visu- alNews, WE split this task into two setups: "MSCOcO-i2t. which retrieves the caption given the image and "MSCOCO-12". which retrieves the image given the caption.
WebQA (Chang al. 2022) The dataset is multihop. multimodal QA dataset that requires trieving a Wikipedia page to answer a given question. We use the Wikipedia page's 's image and text descriptions as the candidates for retrieval.
18

---

Published as a conference paper at ICLR 2025
NIGHTS (Fu et al. 2023) The dataset contains human similarity judgments image pairs that are alike in various ways. The original dataset consists of triplets: a reference image and two perturbed versions, along with human judgments indicating which version is most similar to the reference. Following M-BEIR (Wei et al. 2023), we refactor this dataset into a retrieval task to match pairwise images, where the reference image serves the query, and the perturbed version that aligns with human judgment is the target.
OVEN (Hu et al. 2023) The dataset contains instances that include an image and visual recog- nition text question. Additionally, each instance provides a related Wikipedia image along with its corresponding text deseription (the Wikipedia title and the first 100 tokens of its summary) as a reference for answering the question, which we treat as the target candidate.
EDIS (Liu et al. )023) The dataset is cross-modal image search in the news domain. This dataset contains entity-rich queries, requiring the model to understand both entities and events from the text queries. The candidate consists of the news image and its accompanying headline.
Wiki-SS-NQ (Ma et al. 2024a) The dataset is another retrieval-based VQA dataset. Unlike the
original Natural Questions dataset (Kwiatkowski et al.) 2019a). which uses a Wikipedia paragraph to answer the question, this dataset leverages Wiki-SS, utilizing Wikipedia page screenshots as the corpus. The screenshot provides more comprehensive information than plain Wikipedia paragraph.
For CIRR, FashionIQ. VisualNews, MSCOCO, WebQA, NIGHTS, OVEN and EDIS, we use the processed versions from M-BEIR (Wei et al. )023).
A.1.4 VISUAL. GROUNDING
There are a total of datasets for visual grounding tasks.
MSCOCO (Lin et al. 2014) The dataset includes an object detection task, which involves recogniz- ing an object from a given class in an image. We have repurposed this task into ranking problem within the MMEB format. The query consists of the image and the object name, while the target the cropped image of the specified object. We gather distractors from other objects in the same image well as from different images. We discard test cases where the object is too small.
RefCOCO (Kazemzadeh et al. 2014) The dataset includes an object detection task that requires more reasoning than MSCOCO. Unlike simply identifying the object class, the RefCOCO dataset uses language expressions to refer to specific objects within an image. In our MMEB, we have two tasks related to RefCOCO: "RefCOCO" and "RefCOCO-Matehing consists of the image and the language expressions referring to specific object, while the target is the cropped image of that object. In "RefCOCO-Matching" both the query and the target contain the image and the language expressions referring to a specific object, where the two objects are identical.
In "RefCOCO", the query
Visual7W- (Zhu et al. 2016) The dataset establishes a semantic link between textual de- scriptions and image regions through object-level grounding. It has two types of questions: "telling" and "pointing". It leverages the six W questions (what, where, when, who, why, and how) to sys- tematically examine a model's 's capability for visual understanding through telling questions. Addi- tionally, seventh "which" question is appended for visual answers as pointing questions. We use "Visual7W Wettlliggg in our VQA category and "Visual7W- Wo-nnnniing" in our visual grounding category.
A.2 SELECTION OF NUMBER OF CANDIDATES
A large number of candidates make the benchmark more challenging and realistic. However, we also considered the computational cost when designing the benchmark. Choosing an excessively large number of candidates could result in very high inference costs, which may hinder rapid model iteration. As shown in Table we compare the performance of VLM2VEC with different numbers of candidates in the MMEB benchmark. The results show that if the number of candidates is too small, the benchmark becomes saturated quickly. To balance evaluation cost with benchmark difficulty, we selected 1,000 as the optimal number of candidates.
19

---

Published as a conference paper at ICLR 2025
Table 5:
We compare the performance of VLM2VEc using different numbers of candidates in MMEB. To balance evaluation cost with benchmark difficulty, selected 1, 1,000 as the optimal number of candidates.
Meta-Tase Average Score
Average Score
#Candidates
Classification
VQA
Retrieval
Grounding
IND
OOD
Overall
# of datasets
10
10
12
4
20
16
36
100
54.8
81.8
86.1
89.6
85.2
65.9
76.6
500
54.8
65.9
72.6
82.8
74.6
57.3
66.9
1000
54.8
54.9
62.3
79.5
66.5
52.0
60.1
2000
54.8
50.1
56.7
71.0
62.2
48.0
55.9
5000
54.8
41.3
46.5
65.3
54.5
43.2
49.5
20

---

Published as a conference paper at ICLR 2025
Table 6: The detailed results of the baselines and our VLM2VEC on MMEB, which includes 20 in- distribution datasets and 16 out-of-distribution datasets. The out- oft-o--iistribution datasets are high- lighted with yellow background the table. VIM2VIC is built upon the LLaVA-1.8 backbone.
CLIP OpenCLII
SigLIP
BLIP2
MagicLens
ES-V
UnilR
VLM2VEC
Classification (10 tasks)
ImageNet-1K
55.8
63.5
45.4
10.3
48.0
9.6
58.3
74.5
N24News
38.6
13.9
33.7
23.4
HatelulMemnes
67.9
VOC2007
50.7
52.4
64.3
52.1
51.6
49.9
66.2
91.5
SUN399
43.4
68.8
39.6
34.5
$7.0
33.1
63.2
75.8
Place365
28.5
37.8
20.0
21.5
31.5
8.6
36.5
44.0
ImageNct-A
25.5
14.2
42.6
3.2
8.0
2.0
9.8
43.6
ImageNet-R
83.0
75.0
39.7
70.9
30.8
66.2
79.8
ObjecINet
32.2
39.6
Country-21 211
16.8
14.2
6.2
3.1
11.3
14.7
All Classification
42.8
47.8
40.3
27.0
38.8
21.8
44.3
61.2
VQA (10 tasks)
OK-VQA
7.5
11.5
2.4
8.7
12.7
8.9
25.4
69.0
A-OKVQA
3.8
3.3
1.5
3.2
2.9
5,9
8.8
$4.4
DocVQA
4.0
5.3
4.2
2.6
3.0
1.7
6.2
$2.0
Infographics VQA
4.6
4.6
2.7
2.0
5.9
2.3
4.6
30.7
ChartQA
1.5
2.4
34.8
Visual7W
2.6
1.2
2.5
5,8
14.5
49.8
ScienceQA
9.4
10.2
7.9
6.8
5.2
3.6
12.8
42.1
VizWiz
8.2
6.6
2.3
4.0
1.7
2.6
24.3
43.0
GQA
41.3
$2.5
$7.5
9.7
43.5
7.8
48.8
61.2
TextVQA
7.0
10.9
1.0
3.3
4.6
8.2
15.1
62.0
All VQA
9.1
10.9
8.3
4.9
16.2
49.9
Retrieval (12 tasks)
VisDial
30.7
25.4
21.5
24.8
9.2
42.2
80.9
CIRR
12.6
15.4
15.1
39.1
$1.3
49.9
VisualNews-12i
78.9
74.0
$1.0
48.1
50.7
13.5
74.3
75.4
VisualNews-i2e
79.6
78.0
52.4
13.5
21.1
8.1
76.8
80.0
MSCOCO-22
59.5
63.6
58.3
53.7
54.1
20.7
68.5
75.7
MSCOCO-i2t
57.7
62.1
55.0
20.3
40.0
14.0
72.1
73.1
NIGHTS
62.9
58.1
4.2
66.2
WebQA
FashionIQ
11.4
13.8
20.1
2.8
40.2
16.2
Wiki-SS-NQ
55.0
44.6
55.1
28.7
18.7
8.6
12.2
60.2
OVEN
41.1
45.0
$6.0
39,5
1.6
5,9
69.4
$6.5
EDIS
81.0
77.5
23.6
54.4
62.6
26.8
79.2
87.8
All Retrieval
52.3
31.6
33.9
61.8
67.4
Visual Grounding (4 tasks)
MSCOCO
34.5
46.4
28.9
22.1
46.6
80.6
RefCOCO
56.9
$4.2
70.8
22.8
67.8
88.7
RefCOCO-mmtching
61.3
68.3
50.8
59.5
35.6
38.9
62.9
84.0
Visual7W-pointing
55.1
56.3
70.1
52.0
23.4
14.3
71.3
90.9
All Visual Grounding
51.8
53.3
59.5
47.0
26.0
19.0
65.3
86.1
Final Score (36 tasks)
All
37.8
39.7
34.8
25.2
27.8
13.3
44.7
62.9
All IND
37.1
39.3
32.3
25.3
31.0
14.9
47.1
67.5
OOD
38.7
40.2
38.0
25.1
23.7
11.5
41.7
57.1
21

---

Published as a conference paper at ICLR 2025
Table 7: Examples of datasets MMEB (Part of 4). Instructions are written italic font style.
Category
Dataset
Query Text
Query Image
Target Text
Turget Image
2009
ImageNet-1K
Deng
for classification
Represent given image
greyhound
ImageNet-A et al. 20211
(Hendrycks
Represent given image for classification.
seu unemonne,
ImageNet-K R et 2021a
Henddrycss
Represent the given image for classification.
player
Represent the image with the following caption for domain classifi-
Wang
2021
Ms. Goodman styled Am- ber Valletta with wings bergh for Harper's 's Bazaar.
Style
Classification
Everingham
Ideutiff object the image.
et al. 2014
bus
SUN397
Xizo al 2010
Identify scene shown in the image.
firing range indoce
Objectter
Barbu et
Identify object shown the image.
2
mug
Country-211 et all. 2021
Radiford
Idensify countr picted the image.
China
Represent the given image for binary clatsifccation to desermine whether stitues hateful speech or not.
HatefulMemes
Kiela al
No
20177
Place365
Zbou
Identify the scone shown in the image.
Airport Terminal
22

---

Published as a conference paper at ICLR 2025
Table 8: Examples of datasets in MMEB (Part 2 of 4). Instructions are written in italic font style.
Category
Dalnet
Query Text
Query Image
Target Text
Target image
Resecsest the image
chibuatse
A-OKVQA Scrweak folionise givve image na sedd buid?
baske
folloxing image What univeratt?
DocVIA Mathew
coliforzia
InlormphiiiVV@4 Mathon
follin guomon
Whilh social platforn has heaay fe-
jinzroo
male audieno?
Brywoorn she
uitk
Chanqq Maasy st.l. 2012
folles ive quction
How nany tood ltets shown in the her yeaph?
Repesont the giice image with the follesing lig guenion
SchmorQA L La Poor
Which of these states is finhoo nonb??
South Caréiin
Rrpossoa the givee iuage with the follosive ive queition. Where the mas sitlig??
VissalTW telline
Zla es al. E016
the compase
Aepeassa she givse image witk the folles lep guemins
VieWiz Gunri et 2013
Can you whan this medicine plase!
right time
Repesostt the givee iaage with the follos leg guemins
004
Morning
The rupkin under
Repossnt the givs image quemios
TecVQa
Sugph 2019
23

---

Published as a conference paper at ICLR 2025
Table 9: Examples of datasets MMEB (Part 3 of 4). Instructions are written italic font style.
Category
Query
Query Image
Target
Target Imange
Repooonnt the graas dislogur cbout
iasge und Doage
people wher the sosls
Vidial of 2017
ing Acahie
dross Qratar
Reprssont the glven
color her tessis macket Abblack Qia she weuiing hat Ala visor Qui she close the nci Ano Quio you see arother player Anoo Qudo you see venris bag Ane
image
ofp-
VieaaNNeaa-11
Li Dall
US pulkeoper Hope Solo makes
Repessent the glven
exeaday
MSCOCO-G I2
Lin dta 2014
muicher dee goenn capmon Man ridng mone bis ditt mad on the commyaddr.
Ravesont the givcn imagee
Repeana the sive Mikgeeii age
image-parayy
Retrivval
Do both Hays Cnurry bosee San Bie Wood House 227 Mitchell Sireet in San Varone Trase hove sis orlomes on their from cntramaz?
County Gaunhous: San Minnos Touss Linaal the National Repaier of Hisocci Places 237 Mitchell, Sat Marcos, Treas the Wood House 227 Mitchell Street in San Marcos,
Sai Marcos
WiKDA Change al N00s
Tiras
Bysoount glve image
possidel
EDIS Lau 2025
Holland his the Spiky sui Captain America Civil War.
Comi: Reftdlon Fovreas reprise loon Man nole for Spader Mar: Homaonning.
Wit SS-
Maciall 2024
Find the decaront screetsbee atanner the given
Reposoos the screntaaat.
docament
Casadian Prine Minizer Harpe slakea hazdb with President Ohamo daring the Nort Anericat Laades Sumnit in Tilncs Mesico in Fehuaay 2014.
VisuaNNows 1Zt
Lie al.
2000
Find a caption for the natr the plan phono.
MISDDCC-22
Un
2010
Find as way cxptioe deveriting tee gove eteryday isage.
runn on bicyck riding reat toa tnin.
24

---

Published as a conference paper at ICLR 2025
Table 10: Examples of datasets in MMEB (Part 4 of 4). Instructions are written italic font style.
Category
Datneet
Query
Query Imago
Target
"larget Image
CERR Lundaal
with dercribed
Reposoent the glvwn
Show ürree bottes of sutt drikk.
auttc Junhion
Fas FaahismO)
Wa clal 2011
style shiny and siher with shorter skaness and and fvre.
Repessent the glven
Retrieval
Find dapanoday that inoks sieller the provided inaggs
NIGHTS
Fil et. 2013
Repesset the glacn image.
Beyoneee Milipodio age
Bodea
Wunlembery. covers 1.3 (kan)) and overage 20 (in deep owes ins romotion to the Feilters placeer the munanes whsch were founeed in the Ples- toome spech and nowalags
Rariec a Wikipedia inage- desvipptio puir that provides exienve for guestios of image
Therral DIAS
place?
los
the Witcch
The water of the Tiisee this drais evennually into the Upper Rliie be tuces Tiangen and Waldihus. Or the zunt shore las the.
portion of the loage onjer labvl labie "sup
MSCOCO 2014
given enppodd
STOP
Solect of Whith behind ling besch?
ManallWW
empyod
Grounding
RaCUCC
(Kanemzadeh
Soles follias
Represeut copped object.
2014
in blak cout
ReCCCCO
Select partivn of she foliess the longaage eapessious kid right back. Moodidh hair
Selevr the pamin ahe image thet filines the langange coveraious right kid
d. 2014
Table 11: Zero-shot text-image retrieval performance Flickr30K. As general multimodal rep- resentation model, VLM2VEC can still achieve competitive T2I (maage)--mmage) and I2T (Image- to-Text) scores when compared to existing CLIP-like models. The baseline numbers are sourced from Sun al. (2023) and Zhang et al. (2024). VLM2VEc is built upon the LLa backbone.
Model
image retrieval
text retrieval
R@1
R@5
R
R@1
R@5
10
OpenAl CLIP-B/16
62.1
85.6
91.8
81.9
96.2
98.8
Open CLIP-B/16
69.8
90.4
94.6
86.3
97.9
99.4
EVA-02-CLIP-B/I1
71.2
91.0
94.7
85.7
96.7
98.9
OpenAl CLIP-L/14
65.2
87.3
92.0
85.2
97.3
99.0
Open CLIP-L/14
75.0
92.5
95.6
88.7
98.4
99.2
EVA-Q2-CLIP-LI14
77.3
93.6
96.8
89.7
98.6
99.2
MagieLens-B
76.2
93.7
96.5
87.9
97.7
99.5
MagicLens-L
79.7
95.0
97.4
89.6
98.7
99.4
VLM2VEC (LLaVA-1.6)
80.3
95.0
97.4
94.6
99.5
99.8
25


Reviews:
[{"review_id": "gjdAlfMhDM", "reviewer": "Reviewer_bv32", "summary": "This paper presents to learn universal multimodal embedding models and introduces a novel benchmark (massive multimodal embedding benchmark, MMEB) to evaluate multimodal embedding.\nThe proposed VLM2VEC framework makes any state-of-the-art vision-language models (VLMs) multimodal embedding models through contrastive learning.\nThe MMEB dataset is collected from 36 public datasets (e.g. ImageNet, MSCOCO), covering 4 meta-tasks (e.g. VQA, retrieval).\nExtensive experiments provide several baselines for multimodal embeddings and validate the effectiveness of the proposed VLM2VEC framework.", "questions": "**[Q1] About [EOS] token in the learning objective.** VLM2VEC applies the InfoNCE loss to [EOS] tokens. I understand that the output [EOS] token reflects information gathered through the model’s attention over all preceding input tokens. Could the [EOS] token be replaced by a class token or learnable token? I expect that there will not be a big difference in performance. I wonder if the authors have a valid reason for taking [EOS] tokens or can provide results regarding output token selection.\n\n**[Q2] Inference procedure.** [EOS] token of the output may be used as a single multimodal embedding. Although the inference procedure for each meta-task is well known, it would be good to explain the inference procedure through multimodal embedding. Specifically, I think that classification, VQA, and retrieval can be performed by measuring the similarity between outputs. However, it is not easy to guess how the inference of visual grounding is done.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**[S1]** This paper is well-motivated and well-written.\n\n**[S2]** I strongly agree with the requirement for a benchmark for evaluating multimodal embedding. The MMEB benchmark covers a wide range of tasks and is therefore sufficient to assess multimodal embeddings' capability appropriately.\n\n**[S3]** This paper presents extensive experimental results with several baselines for multimodal embedding, providing a good benchmark for the following research.", "weaknesses": "**[W1] Novelty of VLM2VEC.**\nThe main concern is the limited novelty of the proposed VLM2VEC framework.\nThe proposed framework is significantly similar to the recent works that ground LLMs to visual data with pretrained image models and large language models, such as [1, 2].\nIn other words, projecting visual representations into LLM's input space and tuning the whole or partial model with text generation or contrastive objectives is a typical approach for the recent multimodal large language models.\nAlthough I acknowledge the contribution to the purpose and necessity of this work, it would be good to emphasize the difference that the proposed framework is exclusive only for multimodal embedding.\n\n```\n[1] Li et al., \"Visual instruction tuning,\" NeurIPS, 2024\n[2] Xue et al., \"xGen-MM (BLIP-3): A family of open large multimodal models,\" arXiv, 2024\n```", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2dns9FZ9Sp", "reviewer": "Reviewer_kzgZ", "summary": "This paper proposes a novel framework called VLM2Vec for training vision-language models into embedding models capable of handling massive multimodal embedding tasks. This paper proposes MMEB, a benchmark with 36 datasets spanning classification, visual question answering, retrieval, and visual grounding. VLM2Vec deeply integrates visual and textual features, enabling it to capture cross-modal relationships more effectively.", "questions": "Refer to weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Extensive experiments are conducted using multiple datasets and baselines, demonstrating the effectiveness of the proposed framework.\n2. The development of the MMEB is original, as it provides a comprehensive framework for training and evaluating embedding models across diverse tasks and modalities.\n3. Detailed ablation studies are performed to analyze the impact of various hyperparameters and training strategies on the model's performance.", "weaknesses": "Here are my constructive insights on how the paper could improve:\n1. Benchmark Design: The number of candidates for each retrieval task in the MMEB benchmark is limited to 1000, which may not accurately reflect real-world scenarios where the number of candidates could be much larger. Increasing the number of candidates would make the benchmark more realistic and challenging.\n2. Instead of randomly selecting 1000 candidates for each task, the standard test sets provided by the original datasets should be used. This would ensure a fairer comparison with other methods that have been evaluated on these datasets.\n3. Fair Comparison: Table 2 compares VLM2Vec with baseline methods that have not been trained on the MMEB training sets. For a more accurate comparison, it would be insightful to retrain the baseline methods on the same training sets and then compare their performance with VLM2Vec. \n4. Baseline Evaluation: The paper mentions UniIR but does not include results for the UniIR-CLIP version, which according to the UniIR paper, performs better than the UniIR-BLIP version. Including these results would provide a more comprehensive comparison.\n5. The MMEB benchmark reformulates classification, VQA, retrieval, and visual grounding tasks as ranking problems. While ranking-based formulations can be useful for classification and retrieval tasks, applying the same approach to VQA and visual grounding might not be ideal. These tasks often require more nuanced understanding and reasoning. Considering alternative formulations, such as direct answer generation for VQA and object localization for visual grounding, might yield more meaningful results and better reflect the nature of these tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8yf8uY9jyr", "reviewer": "Reviewer_hG4f", "summary": "This paper focuses on the development of universal multimodal embedding models. The authors introduce the Massive Multimodal Embedding Benchmark (MMEB), which comprises classification, visual question answering (VQA), retrieval, and grounding tasks. They propose a contrastive training framework, VLM2Vec, to transform VLM into an embedding model. Experimental results demonstrate that VLM2Vec achieves promising performance on the MMEB.", "questions": "1. What do the x-axis and legend in Figure 5 represent? What do the three subplots represent respectively?\n2. Does this training method affect the original capabilities of MLLM? I'm curious about the performance of VLM2Vec on MLLM-related benchmarks, such as MMBench and MME.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The writing is clear and easy to understand. This paper is easy to follow.\n2. The constructed benchmark, MMEB, covers a wide range of tasks and is valuable.", "weaknesses": "1. The experiments presented are insufficient, as the authors limit their evaluation to the proposed MMEB benchmarks using only the precision metric. To provide a more comprehensive comparison with other clip-like models, it is crucial to assess the embedding model on some traditional benchmarks, such as ELEVATER and zero-shot retrieval on COCO and Flickr30k.\n2. The evaluation of out-of-distribution may be unfair. Current MLLMs are usually fine-tuned on various downstream datasets. Most of datasets that construct MMEB are usually used in the fine-tuning stage of MLLMs. Since Phi-3.5-V doesn't provide details of fine-tuning datasets,  it is hard to evaluate the out-of-distribution capabilities of VLM2Vec.\n3. Lack of some experimental details. The authors need to provide details of the training of the comparative models on the MMEB in-domain datasets. Additionally, it is unfair to compare CLIP without instructions. Although the authors claim that based on other related work, introducting instructions to CLIP would not perform well, Table 4 shows that instructions have a significant impact, and the CLIP model should also have experimental results under the same settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oM1MF1RTQB", "reviewer": "Reviewer_z3VP", "summary": "This paper presents a multimodal embedding framework and introduces a comprehensive dataset, MMEB, which includes 36 datasets across four meta-task categories, specifically designed for training and evaluating embedding models. Additionally, the paper proposes VLM2VEC, a contrastive training framework that adapts an existing vision-language model into an embedding model. Experimental results indicate that the proposed model achieves 10%~20% improvement over previous multimodal embedding models, on both in-distribution and out-of-distribution datasets.", "questions": "n/a", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The method introduced in this paper is robust. With the authors' commitment to open- sourcing both the model and data, the proposed approach will to be valuable for a wide range of applications.\n2. The experiments are thorough, effectively demonstrating the model's efficacy.\n3. The paper is well-structured and clearly written.", "weaknesses": "1. The method proposed in this paper is relatively straightforward. Its performance, however, varies significantly across different tasks. Would suggest including more comparative experiments to explore and clarify the reasons behind these performance differences across tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)