VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

## Abstract

Abstract Embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering. Recently, there has been a surge of interest in developing universal text embedding models that can generalize across tasks (e.g., MTEB). However, progress in learning universal multimodal embedding models has been relatively slow despite their importance. In this work, we aim to explore the potential for building universal embeddings capable of handling a wide range of downstream tasks. Our contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark), which covers 4 meta-tasks (i.e. classification, visual question answering, multimodal retrieval, and visual grounding) and 36 datasets, including 20 training and 16 evaluation datasets, and (2) Vlm2Vec (Vision-Language Model → → \rightarrow → Vector), a contrastive training framework that converts any state-of-the-art vision-language model into an embedding model via training on MMEB. Unlike previous models such as CLIP and BLIP, Vlm2Vec can process any combination of images and text to generate a fixed-dimensional vector based on task instructions. We build a series of Vlm2Vec models on Phi-3.5-V and evaluate them on MMEB’s evaluation split. Our results show that Vlm2Vec achieves an absolute average improvement of 10% to 20% over existing multimodal embedding models on both in-distribution and out-of-distribution datasets in MMEB.

## 1 Introduction

Figure: Figure 1: We develop a universal multimodal embedding benchmark, MMEB, along with Vlm2Vec, an embedding model adapted from vision-language models (VLMs). Vlm2Vec is capable of following instructions and performing various multimodal embedding tasks, accommodating any combination of image and text modalities.
Refer to caption: x1.png

Embeddings, or distributed representations, encode inputs (whether text or images) as fixed-dimensional vectors, enabling a range of downstream tasks. Since the advent of Word2Vec (Mikolov, ) and GloVe (Pennington et al., ), substantial research efforts have focused on learning textual embeddings (Kiros et al., ; Conneau et al., ) and image embeddings (Radford et al., ; Li et al., ; Jia et al., ; Yu et al., ). These embeddings facilitate a variety of applications, including textual and visual semantic similarity (Agirre et al., ; Marelli et al., ; Chechik et al., ; Cer et al., ), information retrieval (Mitra et al., ; Karpukhin et al., ; Lin et al., ), automatic evaluation (Zhang et al., ; Sellam et al., ), prompt retrieval for in-context learning (Liu et al., ; Rubin et al., ; Hongjin et al., ), and retrieval-augmented generation (Lewis et al., ; Guu et al., ; Izacard & Grave, ). A recent shift in research has focused on developing universal embeddings that can generalize across a wide range of tasks. For instance, Muennighoff et al. () introduced MTEB (Massive Text Embedding Benchmark) to comprehensively assess text embeddings across tasks such as classification and clustering. MTEB has become the standard for evaluating universal text embeddings. Recent works (Wang et al., ; Su et al., ; Wang et al., ; Springer et al., ; BehnamGhader et al., ) have demonstrated promising results on the MTEB benchmark. However, progress in multimodal embeddings has been relatively slower. Despite advancements in text embeddings, the lack of both benchmarks and methodologies in the multimodal embedding domain remains a challenge.

Current research in multimodal embeddings faces two primary limitations: (1) existing studies typically evaluate visual embeddings on isolated tasks, such as ImageNet classification (Deng et al., ; Hendrycks et al., ; ) or MSCOCO/Flickr retrieval (Lin et al., ; Plummer et al., ); (2) most existing models, such as CLIP (Radford et al., ), BLIP (Li et al., ), and SigLIP (Zhai et al., ), either process text and images separately or perform shallow fusion of visual and textual information (Wei et al., ), limiting their ability to fully capture the relationships between text and image modalities. Furthermore, these models exhibit limited reasoning and generalization capabilities, particularly in zero-shot scenarios for complex reasoning tasks.

In this paper, we attempt to build an universal multimodal embedding framework to pave road for the future research, which consists of two efforts:

- MMEB: We introduce a novel benchmark, MMEB (Massive Multimodal Embedding Benchmark), which includes 36 datasets spanning four meta-task categories: classification, visual question answering, retrieval, and visual grounding. MMEB provides a comprehensive framework for training and evaluating embedding models across various combinations of text and image modalities. All tasks are reformulated as ranking tasks, where the model follows instructions, processes a query, and selects the correct target from a set of candidates. The query and target can be an image, text, or a combination of both. MMEB is divided into 20 in-distribution datasets, which can be used for training, and 16 out-of-distribution datasets, reserved for evaluation.
- Vlm2Vec: We adopt the pre-trained vision-language model Phi-3.5-V (Abdin et al., ) as the backbone for Vlm2Vec. In contrast to other multimodal embedding models like UniIR (Wei et al., ) and MagicLens (Zhang et al., ), which rely on late fusion of CLIP (Radford et al., ) features, our approach leverages the deep integration of vision and language features within a transformer architecture. There are several advantages to this approach: (1) VLMs are trained on massive multimodal datasets and can handle any combination of images and text, as well as high-resolution images and long text inputs; (2) vision and language features are deeply fused in the transformer model, improving the model’s ability to capture cross-modal relationships; and (3) these models are well-suited for generalizing across diverse tasks, particularly those requiring instruction-following capabilities. These factors make Vlm2Vec an ideal choice for task generalization. We trained Vlm2Vec on the 20 MMEB training datasets using contrastive learning and compared its performance with various baselines.

Following extensive contrastive training, Vlm2Vec can handle any combination of images and text, producing fixed-dimensional vectors. We evaluate Vlm2Vec against a wide array of multimodal embedding models, including CLIP (Radford et al., ), BLIP2 (Li et al., ), SigLIP (Zhai et al., ), MagicLens (Zhang et al., ), UniIR (Wei et al., ) and E5-V (Jiang et al., ), demonstrating consistent improvements across all task categories. Notably, Vlm2Vec achieves a 17.3-point improvement (from 42.8 to 60.1) across all 36 MMEB datasets and a 11.6-point increase (from 40.4 to 52.0) on 16 out-of-distribution datasets in zero-shot evaluation, highlighting the effectiveness of our proposed framework. We are committed to releasing all data, training code, and models to facilitate future research in this area.

## 2 MMEB: A Benchmark for Multimodal Embeddings

### 2.1 Dataset Overview

We present MMEB (Massive Multimodal Embedding Benchmark), a comprehensive benchmark designed to evaluate multimodal embeddings across a diverse set of tasks. MMEB consists of 36 datasets organized into four meta-tasks: classification, visual question answering, retrieval, and visual grounding. Each task is reformulated as a ranking problem, where the model is provided with an instruction and a query (which may consist of text, images, or both) and is tasked with selecting the correct answer from a set of candidates. These candidates could be text, images, or additional instructions. The datasets are divided into two categories: 20 in-distribution datasets for training and 16 out-of-distribution datasets for evaluation. We report performance metrics across all 36 tasks. An overview of MMEB is provided in Figure 2 and the dataset statistics are provided in Table 1.

The embedding models are supposed to compress the query side into a vector and the target candidates into a set of vectors. The candidate with the highest dot-product will be selected as the prediction for evaluation. We measure the Precision@1 to reflect the percentage of top candidate matching the groundtruth. To ensure the task difficulty, we introduce a large amount of candidates.

MMEB offers a wide range of tasks from various domains, such as common, news, Wikipedia, web, and fashion. The benchmark incorporates diverse combinations of modalities for both queries and targets, including text, images, and text-image pairs. Additionally, tasks are designed to follow different types of instructions. For instance, tasks may involve object recognition (e.g., “Identify the object shown in the image.”), retrieval (e.g., “Find an image that matches the given caption.”), or visual grounding (e.g., “Select the portion of the image that answers the question.”). Examples for each dataset in MMEB are provided in Tables 6, 7, 8 and 9. The diversity in MMEB makes it an ideal testbed for universal embeddings.

**Table 1: The statistics of MMEB: 36 datasets across 4 meta-task categories, with 20 in-distribution datasets used for training and 16 out-of-distribution datasets used exclusively for evaluation.**
| Meta-Task | Dataset | Query | Target | OOD? | #Training | #Eval | #Candidates |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Classification<br>(10 Tasks) | ImageNet-1K | I | T |  | 100K | 1000 | 1000 |
| N24News | I + T | I |  | 49K | 1000 | 24 |  |
| HatefulMemes | I | T |  | 8K | 1000 | 2 |  |
| VOC2007 | I | T |  | 8K | 1000 | 20 |  |
| SUN397 | I | T |  | 20K | 1000 | 397 |  |
| Place365 | I | T | ✓ | - | 1000 | 365 |  |
| ImageNet-A | I | T | ✓ | - | 1000 | 1000 |  |
| ImageNet-R | I | T | ✓ | - | 1000 | 200 |  |
| ObjectNet | I | T | ✓ | - | 1000 | 313 |  |
| Country-211 | I | T | ✓ | - | 1000 | 211 |  |
| VQA<br>(10 Tasks) | OK-VQA | I + T | T |  | 9K | 1000 | 1000 |
| A-OKVQA | I + T | T |  | 17K | 1000 | 1000 |  |
| DocVQA | I + T | T |  | 40K | 1000 | 1000 |  |
| InfographicVQA | I + T | T |  | 24K | 1000 | 1000 |  |
| ChartQA | I + T | T |  | 28K | 1000 | 1000 |  |
| Visual7W | I + T | T |  | 70K | 1000 | 1000 |  |
| ScienceQA | I + T | T | ✓ | - | 1000 | 1000 |  |
| VizWiz | I + T | T | ✓ | - | 1000 | 1000 |  |
| GQA | I + T | T | ✓ | - | 1000 | 1000 |  |
| TextVQA | I + T | T | ✓ | - | 1000 | 1000 |  |
| Retrieval<br>(12 Tasks) | VisDial | T | I |  | 123K | 1000 | 1000 |
| CIRR | I + T | I |  | 26K | 1000 | 1000 |  |
| VisualNews_t2i | T | I |  | 100K | 1000 | 1000 |  |
| VisualNews_i2t | I | T |  | 100K | 1000 | 1000 |  |
| MSCOCO_t2i | T | I |  | 100K | 1000 | 1000 |  |
| MSCOCO_i2t | I | T |  | 113K | 1000 | 1000 |  |
| NIGHTS | I | I |  | 16K | 1000 | 1000 |  |
| WebQA | T | I + T |  | 17K | 1000 | 1000 |  |
| OVEN | I + T | I + T | ✓ | - | 1000 | 1000 |  |
| FashionIQ | I + T | I | ✓ | - | 1000 | 1000 |  |
| EDIS | T | I + T | ✓ | - | 1000 | 1000 |  |
| Wiki-SS-NQ | T | I | ✓ | - | 1000 | 1000 |  |
| Visual Grounding<br>(4 Tasks) | MSCOCO | I + T | I |  | 100K | 1000 | 1000 |
| Visual7W-Pointing | I + T | I | ✓ | - | 1000 | 1000 |  |
| RefCOCO | I + T | I | ✓ | - | 1000 | 1000 |  |
| RefCOCO-Matching | I + T | I + T | ✓ | - | 1000 | 1000 |  |

### 2.2 Meta-task and Dataset Design

MMEB is organized into four primary meta-task categories:

Classification
This category comprises 5 in-distribution and 5 out-of-distribution datasets. Queries consist of instructions and images, optionally accompanied by related text. Targets are class labels, and the number of class labels corresponds to the number of classes in the dataset.

Visual Question Answering
This category includes 6 in-distribution and 4 out-of-distribution datasets. The query consists of an instruction, an image, and a piece of text as the question, while the target is the answer. Each query has 1,000 target candidates: 1 ground truth and 999 distractors.

Information Retrieval
This category contains 8 in-distribution and 4 out-of-distribution datasets. Both the query and target sides can involve a combination of text, images, and instructions. Similar to the VQA task, each query has 1,000 candidates, with 1 ground truth and 999 distractors.

Visual Grounding
This category includes 1 in-distribution and 3 out-of-distribution datasets, which are adapted from object detection tasks. Queries consist of an instruction, an image, and text referring to a specific region or object within the image. The target may include a cropped image of the object or text describing the same region. Each query includes 1,000 candidates: 1 ground truth and 999 distractors. These distractors may include hard negatives from the same object class, other objects in the image, or random objects from different images.

This task evaluates the model’s ability to recognize and represent the same object or concept across different modalities (image or text) and from varying perspectives. The task involves correctly identifying or referring to specific regions or objects within an image based on the provided instructions or language expressions.

For each evaluation split within the MMEB datasets, 1,000 samples are randomly selected. Further details on dataset processing can be found in Section A.1.

Figure: Figure 2: An overview of the tasks and datasets in MMEB. MMEB includes four meta-tasks and 36 datasets: 20 in-distribution datasets (blue) used for training and 16 out-of-distribution (orange) datasets used exclusively for evaluation.
Refer to caption: x2.png

## 3 Vlm2Vec : Transforming LVMs to Embedders

### 3.1 Contrastive Training

We develop Vlm2Vec, a contrastive training framework designed to convert any state-of-the-art vision-language model into an embedding model, as illustrated in Figure 3. A relevant query-target pair is denoted as ($q,t^{+}$). Both $q$ and $t^{+}$ could be either single image, text or single image + text. We define $q:(q_{t},q_{i})$ and $t^{+}:(t_{t}^{+},t_{i}^{+})$.

We then apply the instruction to the original query $q$ to generate a new one $q_{\text{inst}}$:

$$ $q_{\text{inst}}=\text{{[IMAGE_TOKEN]}}\text{Instruct: {\{task_definition\}}} \ \backslash n\ \text{Query:\ }\{q\}$ (1) $$

where “*{task_definition}*” is a placeholder for a one-sentence description of the embedding task. To enhance the embedding model’s generalizability by better understanding instructions, we have crafted task-specific instructions, as shown in Tables 6, 7, 8 and 9.

Given a pretrained VLM, we feed query and target into it to obtain the query and target embeddings ($\mathbf{h}_{q_{\text{inst}}},\mathbf{h}_{t^{+}}$)
by taking the last layer vector representation of the last token. To train the embedding model,
we adopt the standard InfoNCE loss $\mathcal{L}$ over the in-batch negatives and hard negatives:

$$ $\min\ \ \mathcal{L}=-\log\frac{\phi(\mathbf{h}_{q_{\text{inst}}},\mathbf{h}_{t ^{+}})}{\phi(\mathbf{h}_{q_{\text{inst}}},\mathbf{h}_{t^{+}})+\displaystyle \sum_{t^{-}\in\mathbb{N}}(\phi(\mathbf{h}_{q_{\text{inst}}},\mathbf{h}_{t^{-}} ))}$ (2) $$

where $\mathbb{N}$ denotes the set of all negatives,
and $\phi(\mathbf{h}_{q},\mathbf{h}_{t})$ is a function that computes the matching score between query $q$ and target $t$.
In this paper, we adopt the temperature-scaled cosine similarity function as follows:

$$ $\phi(\mathbf{h}_{q},\mathbf{h}_{t})=\text{exp}(\frac{1}{\tau}\cos(\mathbf{h}_{ q},\mathbf{h}_{t}))$ (3) $$

$\tau$ is a temperature hyper-parameter.

Figure: Figure 3: Vlm2Vec uses a VLM as the backbone to deeply integrate image and text features. It is trained with a contrastive loss between the query and target, following task-specific instructions. The training data consists of diverse combinations of modalities on both the query and target sides, which may include images, text, or image-text pairs.
Refer to caption: x3.png

### 3.2 Increasing Batch Size Through GradCache

Since hard negatives are often difficult or ambiguous to collect for most multimodal datasets, using larger batch sizes becomes crucial. This increases the number of in-batch random negatives, which in turn helps improve the performance of the embedding model.

A bottleneck lies in the GPU memory that limits us from increasing the batch size and the number of in-batch random negatives during training, as each training instance may include one image (either from the query or target side) or multiple images (from both query and target sides), resulting in substantial memory consumption. We apply GradCache (Gao et al., ), a gradient caching technique that decouples backpropagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension.

Mathematically, supposed we have a large batch of queries $\mathcal{Q}$, and we divide it into a set of sub-batches, each of which can fit into memory for gradient computation: $\mathcal{Q}=\{\hat{Q}_{1},\hat{Q}_{2},\dots\}$. There are two major steps: “Representation Gradient Computation and Caching” and “Sub-batch Gradient Accumulation”.
First, gradient tensors within each subbatch is calculated and stored: $\mathbf{u}_{i}=\frac{\partial\mathcal{L}}{\partial f(q_{i})}$.

Then gradients are accumulated for encoder parameters across all sub-batches:

$$ $\displaystyle\begin{split}\frac{\partial\mathcal{L}}{\partial\Theta}=\sum_{ \hat{Q}_{j}\in\mathcal{Q}}\sum_{q_{i}\in\hat{Q}_{j}}\frac{\partial\mathcal{L}} {\partial f(q_{i})}\frac{\partial f(q_{i})}{\partial\Theta}=\sum_{\hat{Q}_{j} \in\mathbb{Q}}\sum_{q_{i}\in\hat{Q}_{j}}\mathbf{u}_{i}\frac{\partial f(q_{i})} {\partial\Theta}\end{split}$ (4) $$

## 4 Experiments

### 4.1 Experiment Settings

In this paper, we adopt Phi-3.5-V (Abdin et al., ) as our backbone VLM, with training conducted via either full model fine-tuning or LoRA. The temperature is set to 0.02. Multiple experimental configurations are explored to assess the impact of key hyperparameters, such as batch size and input length. In the basic setting, we use a batch size of 256, 2,000 training steps, 4 image crops, and a maximum text length of 256 tokens. Detailed ablation studies on these parameters are discussed in Section 4.4.

For the 20 training datasets, if a dataset contains more than 50K samples, we randomly select 50K for consistency, resulting in a total training set of 662K data points. When using GradCache, we set a sub-batch size of 4 to enable full model tuning, with the total batch size accumulated to 1,024. All experiments were run on 8 H100 GPUs.

### 4.2 Baselines

Four groups of baselines are reported in this study.

CLIP-family: We utilize vision/language encoders such as CLIP (Radford et al., ), OpenCLIP (Cherti et al., ), SigLIP (Zhai et al., ), and BLIP2 (Li et al., ) as our baseline. Due to the length limitations of the text encoder, some queries or target text in certain tasks may be truncated. We apply score-level fusion by combining multimodal features using element-wise addition with equal weights ($w_{1}=w_{2}=1$). As found in UniIR (Wei et al., ), we do not use instructions, as this was observed to potentially degrade performance.
UniIR: UniIR (Wei et al., ) is a unified, instruction-guided multimodal retriever designed to handle eight different retrieval tasks across multiple modalities. The model builds on CLIP and BLIP, employing shallow fusion techniques such as score-level and feature-level fusion to integrate modalities. In this study, we use the BLIP_FF variation as a baseline.
MagicLens: MagicLens (Zhang et al., ) is a self-supervised image retrieval model capable of handling open-ended instructions. It utilizes a dual-encoder architecture with shared parameters, initializing the vision and language encoders with either CoCa or CLIP. The model uses a multi-head attention pooler to unify multimodal inputs into a single embedding. For this study, we report results using the CLIP-Large backbone. Since MagicLens requires image inputs, we represent pure text inputs by using the output vector of the text encoder.
E5-V: E5-V (Jiang et al., ) is a contemporary model that also leverages vision-language models for multimodal embedding tasks. It proposes a single-modality training approach, where the model is trained exclusively on text pairs. E5-V demonstrated strong performance on three multimodal retrieval tasks. In contrast, we propose MMEB, extending the evaluation to a larger and more diverse set of tasks. Additionally, our model is trained on multimodal pairs, which include various combinations of image and text modalities on both the query and target sides.

### 4.3 Main Result

**Table 2: Results on the MMEB benchmark. The scores are averaged per meta-task. For detailed scores per dataset, see Table 5.**
| Model | Per Meta-Task Score |  | Average Score |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Classification | VQA | Retrieval | Grounding |  | IND | OOD | Overall |  |
| # of datasets $\rightarrow$ | 10 | 10 | 12 | 4 |  | 20 | 16 | 36 |
| *Baseline Models* |  |  |  |  |  |  |  |  |
| CLIP (Radford et al., ) | 42.8 | 9.1 | 53.0 | 51.8 |  | 37.1 | 38.7 | 37.8 |
| BLIP2 (Li et al., ) | 27.0 | 4.2 | 33.9 | 47.0 |  | 25.3 | 25.1 | 25.2 |
| SigLIP (Zhai et al., ) | 40.3 | 8.4 | 31.6 | 59.5 |  | 32.3 | 38.0 | 34.8 |
| OpenCLIP (Cherti et al., ) | 47.8 | 10.9 | 52.3 | 53.3 |  | 39.3 | 40.2 | 39.7 |
| UniIR (Wei et al., ) | 42.1 | 15.0 | 60.1 | 62.2 |  | 44.7 | 40.4 | 42.8 |
| E5-V (Jiang et al., ) | 21.8 | 4.9 | 11.5 | 19.0 |  | 14.9 | 11.5 | 13.3 |
| Magiclens (Zhang et al., ) | 38.8 | 8.3 | 35.4 | 26.0 |  | 31.0 | 23.7 | 27.8 |
| *Ours (Vlm2Vec)* |  |  |  |  |  |  |  |  |
| LoRA (bs=1024) | 54.8 | 54.9 | 62.3 | 79.5 |  | 66.5 | 52.0 | 60.1 |
| Full Fine-Tuning (bs=1024) | 52.8 | 50.3 | 57.8 | 72.3 |  | 62.8 | 47.4 | 55.9 |
| $\Delta$ - Best baseline | +7.0 | +39.9 | +2.2 | +17.3 |  | +21.8 | +11.6 | +17.3 |

We report Precision@1 for all models in Table 2. It measures the ratio of positive candidates being ranked in the top place for all queries. For Vlm2Vec, both the LoRA and fully fine-tuned variants follow the same setup of 1024 training batch size, 2K training steps and 4 sub-image crops. The LoRA variant uses a rank of 8.

From Table 2, the LoRA version of Vlm2Vec is the best variant, achieving an average precision@1 of 60.1% across all 36 datasets from MMEB. Additionally, it maintains an average precision@1 of 52.0% on 16 out-of-distribution tasks in zero-shot evaluation, suggesting strong generalization ability. This indicates that our model, when well-trained on datasets from diverse task categories, domains, and modality combinations, can effectively follow instructions to align the visual and text spaces and generalize well to unseen tasks. The full fine-tuning variation achieves slightly lower scores than the LoRA version. For a detailed discussion comparing full fine-tuning and LoRA, please refer to Section 4.4.1.

Compared to other baseline models, we observe consistent improvements in our model across all meta-task categories. Notably, our model achieves a 17.3-point improvement (from 42.8 to 60.1) across all 36 MMEB datasets and a 11.6-point increase (from 40.4 to 52.0) on 16 out-of-distribution datasets for zero-shot evaluation. Additionally, unlike the baseline models, which fail to demonstrate reasonable performance across all different task categories, Vlm2Vec achieves relatively strong performance (at least 50%) across all four meta-task categories. This highlights its capability to handle a wide range of multimodal embedding tasks effectively. It’s worth noting that our contemporary work E5-V (Wang et al., ), which is also based on vision-language models, performs much worse than Vlm2Vec. This is due to the fact that E5-V has been trained exclusively on text data without using multimodal data.

### 4.4 Result Analysis

To train an effective and generalizable multimodal embedding, various factors need to be considered, ranging from the data to the training setup. In this section, we present detailed ablation studies on these factors. We will discuss two training setups: Full Fine-Tuning vs. LoRA, along with Training parameters, and two topics related to data: Meta-task generalization and Impact of instructions in this section.

#### 4.4.1 Full Fine-Tuning vs. LoRA

When fine-tuning the VLMs, a key decision is whether to conduct full fine-tuning, which
updates all parameters in the model, or to use a parameter-efficient method such as LoRA. We compare the performance of fully fine-tuned Vlm2Vec with its LoRA variants at different ranks. The training and data setups are kept consistent across all models. We observe that LoRA achieves better performance when the rank is appropriately configured.

**Table 3: We compare the performance of fully fine-tuned Vlm2Vec with its LoRA variants at different ranks. LoRA can achieve better performance when the rank is appropriately configured.**
| Model | Meta-Task Average Score |  | Average Score |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Classification | VQA | Retrieval | Grounding |  | IND | OOD | Overall |  |
| # of datasets $\rightarrow$ | 10 | 10 | 12 | 4 |  | 20 | 16 | 36 |
| Full Fine-Tuning (bs=256) | 50.4 | 46.4 | 52.6 | 68.6 |  | 57.9 | 44.7 | 52.0 |
| LoRA r = 4 (bs=256) | 52.7 | 53.6 | 60.1 | 80.2 |  | 64.9 | 50.4 | 58.4 |
| LoRA r = 8 (bs=256) | 52.9 | 52.5 | 60.3 | 80.0 |  | 64.2 | 50.8 | 58.2 |
| LoRA r = 16 (bs=256) | 51.1 | 40.5 | 52.0 | 72.5 |  | 54.9 | 45.8 | 50.8 |
| LoRA r = 32 (bs=256) | 50.6 | 47.8 | 53.9 | 72.5 |  | 58.9 | 46.5 | 53.4 |

#### 4.4.2 Training parameters

During our experiments, we identified three key parameters that significantly impact the performance of Vlm2Vec: training batch size, the number of sub-image crops, and the number of training steps. In Figure 4, we observe that the final performance gradually improves as we increase the batch size, training step size, and number of sub-image crops. We particularly want to highlight the impact of batch size. Due to the lack of hard negatives, using a large batch size with plenty of random negatives, supported by the GradCache technique, plays a crucial role in enhancing the performance of Vlm2Vec, as discussed in Section 3.2.

Figure: Figure 4: The figures demonstrate the influence of the training setup on Vlm2Vec’s final performance. Here, we examine the effects of training batch size, the number of sub-image crops, and the number of training steps.
Refer to caption: x4.png

#### 4.4.3 Meta-task generalization

We have demonstrated that Vlm2Vec has the potential to transfer to out-of-distribution datasets after being trained on a diverse range of in-distribution datasets, with the instruction-following settings. An interesting question arises as to whether focusing on a specific meta-task can enhance the model’s overall generalizability. We have trained three models, each focused solely on one meta-task (classification, visual question answering, and retrieval). Visual grounding was not included due to the limited number of datasets. We then evaluated the models’ transferability to other meta-tasks. We refer to these three models as $\text{VLM2Vec}_{\text{RET}}$, trained on 8 retrieval tasks, $\text{VLM2Vec}_{\text{VQA}}$, trained on 6 visual question answering tasks, and $\text{VLM2Vec}_{\text{CLS}}$, trained on 5 classification tasks.

Figure 5 illustrates the generalizability of these three models on unseen meta-tasks.
We could observe that $\text{VLM2Vec}_{\text{RET}}$ has better generalizablilty on other meta-task, compared with other two models, especially on visual grounding categories. The reason is that retrieval tasks involve a more diverse combination of text and visual modalities from both the query and target sides, which helps the model generalize better to unseen meta-tasks. This observation highlights the benefits of using more diverse tasks in the Vlm2Vec training process.

Figure: Figure 5: The figures show the generalization ability of models trained on one meta-task to other unseen meta-tasks. Models trained on retrieval tasks demonstrate better generalization ability because retrieval tasks involve a more diverse combination of text and visual modalities from both the query and target sides.
Refer to caption: x5.png

#### 4.4.4 Impact of Instructions

Previous studies have shown the influence of instructions on addressing various tasks. Vlm2Vec, which leverages a VLM as its backbone and is trained on large-scale datasets with instructions, is expected to better generalize across tasks and improve performance in multimodal embedding tasks. In this section, we evaluate the performance of Vlm2Vec with and without task-specific instructions to quantify the impact of incorporating instructions into the embedding process. As shown in Table 4, excluding instructions leads to an average performance drop of around 30%, highlighting the importance of instruction-guided embeddings.

**Table 4: Comparison of Vlm2Vec with and without task-specific instructions. The percentage decrease highlights the significant performance drop when instructions are not incorporated.**
| Model | Meta-Task Average Score |  | Average Score |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Classification | VQA | Retrieval | Grounding |  | IND | OOD | Overall |  |
| # of datasets $\rightarrow$ | 10 | 10 | 12 | 4 |  | 20 | 16 | 36 |
| w/ instruction | 50.4 | 46.4 | 52.6 | 68.6 |  | 57.9 | 44.7 | 52.0 |
| w/o instruction | 36.7 | 33.5 | 31.1 | 44.3 |  | 37.3 | 31.6 | 34.8 |
| $\Delta$ | -27.2% | -27.8% | -40.9% | -35.4% |  | -35.6% | -29.3% | -33.1% |

## 5 Related Work

### 5.1 Text Embedding

Text embeddings have demonstrated significant potential in powering downstream applications such as information retrieval (Karpukhin et al., ; Xiong et al., ), text similarity (Gao et al., ), prompt retrieval for in-context learning (Hongjin et al., ), and classification (Logeswaran & Lee, ; Reimers & Gurevych, ). Early work focused on creating effective embeddings for specific tasks. With the rise of pretrained language models, efforts have shifted toward developing universal embedding models capable of handling a wide range of embedding tasks. Studies such as GTR (Ni et al., ) and E5 (Wang et al., ) leveraged large amounts of noisy paired data to pretrain and fine-tune dense retrievers. More recent works like TART (Asai et al., ) and InstructOR (Su et al., ) introduced natural language prompts to guide embedding models in producing task-relevant embeddings. Building on this, models like E5Mistral(Wang et al., ), SFR-Embedding(Meng et al., ), RepLLaMA(Ma et al., ), GTE-Qwen2(Li et al., ), and NV-Embed (Lee et al., ) have utilized pretrained large language models (LLMs) as their backbone, fine-tuning them with multi-task data and instructions. These models have delivered significant improvements over earlier approaches that did not use LLMs for initialization or instruction tuning. However, these advances come at a cost: both the model size and the dimensionality of output vectors have increased substantially, leading to higher costs.

### 5.2 Multimodal Embeddings

Multimodal embeddings have long been a significant research challenge. Early works like CLIP (Radford et al., ), BLIP (Li et al., ; ), Align (Jia et al., ), SigLIP (Zhai et al., ), SimVLM Wang et al. () and CoCa (Yu et al., ) primarily focused on learning universal representations from large-scale, weakly supervised image-text pairs. These models generally encode images and text separately, projecting them into a shared space. This approach has laid the groundwork for more recent multimodal models like LLaVA (Liu et al., ).

Most research on universal multimodal embeddings involves fine-tuning models like CLIP or BLIP, typically using simple fusion mechanisms to combine visual and language information. For instance, UniIR (Wei et al., ) creates multimodal embeddings by simply adding text and visual features, while MagicLens (Zhang et al., ) employs shallow self-attention layers to integrate these features more effectively. The study most similar to ours is E5-V (Jiang et al., ), which converts a multimodal generative model into an embedding model. We compare our approach with E5-V and find that our models achieve significant improvements in performance.

### 5.3 Embedding Benchmarks

Significant efforts have been made to develop benchmarks for evaluating retrieval systems. For text retrieval models, MS MARCO (Nguyen et al., ) and Natural Questions (Kwiatkowski et al., ) are two of the most widely used benchmarks in general domains. To broaden the evaluation across more diverse domains, BEIR () was introduced, incorporating 18 datasets from various fields. Building on this, MTEB (Muennighoff et al., ) further expands BEIR’s scope by adding more tasks, such as classification, clustering, and semantic textual similarity (STS), to assess the generalization capabilities of embedding models.

For multimodal retrieval, several benchmarks have been introduced to evaluate model performance across different modalities. MBEIR (Wei et al., ) includes 8 tasks and 16 datasets, designed to test models’ ability to retrieve information based on various forms of queries (text, image, or a combination) and instructions that span multiple modalities.

## 6 Conclusion

In this paper, we aim to build the first large-scale multimodal embedding framework, comprising two main components: MMEB and Vlm2Vec. MMEB includes 36 datasets across four meta-task categories, providing a comprehensive and diverse framework for training and evaluating embedding models. Vlm2Vec leverages VLMs as a backbone to deeply fuse visual and textual spaces, enhancing generalization to unseen tasks through instruction following.

## Appendix A Appendix

### A.1 Dataset details

#### A.1.1 Classification

There are a total of 10 datasets for classification tasks.

ImageNet-1K (Deng et al., ) The dataset is s large-scale dataset commonly used in image classification, consisting of over 1 million images across 1K different classes.

ImageNet-A (Hendrycks et al., ) The dataset contains images from a distribution unlike the ImageNet training distribution. ImageNet-A examples belong to ImageNet classes, but the examples are harder and can cause mistakes across various models. They cause consistent classification mistakes due to scene complications encountered in the long tail of scene configurations and by exploiting classifier blind spots.

ImageNet-R (Hendrycks et al., ) The dataset contains set of images labeled with ImageNet labels obtained by collecting art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video game renditions of ImageNet classes.

VOC2007 (Everingham et al., ) The dataset focuses on recognizing objects in realistic scenarios and contains 20 object classes.

N24News (Wang et al., ) The dataset is sourced from the New York Times and consists of 24 categories, with each news article containing both text and image information. The task is to classify the given news image and its accompanying text into one of the 24 categories.

HatefulMemes (Kiela et al., ) The dataset proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes.

Place365 (Zhou et al., ) The dataset is a repository of 10 million scene photographs, labeled with scene semantic categories, comprising a large and diverse list of the types of environments encountered in the world.

SUN397 (Xiao et al., ) The dataset is a dataset for scene recognition consisting of 397 categories.

ObjectNet (Barbu et al., ) The dataset is a crowd-sourced test set of 50K images featuring objects in unusual poses and cluttered scenes, designed to challenge recognition performance. It includes controls for rotation, background, and viewpoint, and covers 313 object classes.

Country-211 (Radford et al., ) The dataset is designed to assess the geolocation capability of visual representations. It filters the YFCC100M dataset to find 211 countries that have at least 300 photos with GPS coordinates.

#### A.1.2 Visual Question Answering (VQA)

There are a total of 10 datasets for VQA tasks.

OK-VQA (Marino et al., ) The dataset includes questions that require external resources for answers.

A-OKVQA (Schwenk et al., ) The dataset is an augmented successor of OK-VQA, requiring a broad base of commonsense and world knowledge to answer. The questions generally cannot be answered by simply querying a knowledge base, and instead require some form of commonsense reasoning about the scene depicted in the image.

DocVQA (Mathew et al., ) The dataset contains questions for document analysis and recognition over document images of various types and content.

InfographicsVQA (Mathew et al., ) The dataset comprises a diverse collection of infographics accompanied by natural language question and answer annotations. The questions require methods capable of jointly reasoning over the document layout, textual content, graphical elements, and data visualizations.

ChartQA (Masry et al., ) The dataset is designed for question answering about charts, with a focus on visual and logical reasoning applied to real-world charts.

ScienceQA (Lu et al., ) The dataset contains questions with diverse science topics and annotations of their answers with corresponding lectures and explanations.

Visual7W-telling (Zhu et al., ) The dataset establishes a semantic link between textual descriptions and image regions through object-level grounding. It has two types of questions: “telling” and “pointing”. It leverages the six W questions (what, where, when, who, why, and how) to systematically examine a model’s capability for visual understanding through telling questions. Additionally, a seventh “which” question is appended for visual answers as pointing questions. We use “Visual7W-telling” in our VQA category and “Visual7W-pointing” in our visual grounding category.

VizWiz (Gurari et al., ) The dataset originates from a natural visual question answering scenario, where blind individuals captured images and recorded spoken questions about them, along with 10 crowdsourced answers for each visual question. For our task, we select only the answerable questions.

TextVQA (Singh et al., ) The dataset is designed to benchmark visual reasoning based on text within images. Models need to read and reason about the text in images to answer related questions.

GQA (Hudson & Manning, ) The dataset is designed for real-world visual reasoning and compositional question answering. It uses real images from the Visual Genome dataset. Each image is accompanied by scene graph annotations that describe the classes and attributes of objects in the scene, as well as their pairwise relationships.

#### A.1.3 Retrieval

There are a total of 12 datasets for retrieval tasks.

VisDial (Das et al., ) The dataset features dialogues created by two Amazon Mechanical Turk workers. One worker takes the role of the “questioner”, who only sees the text description of an image, while the other plays the “answerer”, who has access to the image. They engage in a 10-round Q&A session about the image. We repurpose this dataset as a retrieval task, where the goal is to retrieve the image based on the given dialogue.

CIRR (Liu et al., ) The dataset is designed for the task of composed image retrieval. It consists of pairs of real-life reference and target images, along with a modification sentence that describes the changes made between the two images.

FashionIQ (Wu et al., ) The dataset contains images of fashion products with crowd-sourced descriptions highlighting the differences between these products. Similar to CIRR, FashionIQ can also be used for the task of composed image retrieval, where each test case consists of a pair of reference and target images, along with a modification sentence that describes the changes between the two images.

VisualNews (Liu et al., ) The dataset contains publicly available news image paired with captions. We split this task into two setups: “VisualNews_i2t”, which retrieves the caption given the news image and “VisualNews_t2i”, which retrieves the news image given the caption.

MSCOCO (Lin et al., ) The dataset is a well-known image caption dataset. Similar to VisualNews, WE split this task into two setups: “MSCOCO_i2t’, which retrieves the caption given the image and “MSCOCO_t2i”, which retrieves the image given the caption.

WebQA (Chang et al., ) The dataset is a multihop, multimodal QA dataset that requires retrieving a Wikipedia page to answer a given question. We use the Wikipedia page’s image and text descriptions as the candidates for retrieval.

NIGHTS (Fu et al., ) The dataset contains human similarity judgments on image pairs that are alike in various ways. The original dataset consists of triplets: a reference image and two perturbed versions, along with human judgments indicating which version is most similar to the reference. Following M-BEIR (Wei et al., ), we refactor this dataset into a retrieval task to match pairwise images, where the reference image serves as the query, and the perturbed version that aligns with human judgment is the target.

OVEN (Hu et al., ) The dataset contains instances that include an image and a visual recognition text question. Additionally, each instance provides a related Wikipedia image along with its corresponding text description (the Wikipedia title and the first 100 tokens of its summary) as a reference for answering the question, which we treat as the target candidate.

EDIS (Liu et al., ) The dataset is a cross-modal image search in the news domain. This dataset contains entity-rich queries, requiring the model to understand both entities and events from the text queries. The candidate consists of the news image and its accompanying headline.

Wiki-SS-NQ (Ma et al., ) The dataset is another retrieval-based VQA dataset. Unlike the original Natural Questions dataset (Kwiatkowski et al., ), which uses a Wikipedia paragraph to answer the question, this dataset leverages Wiki-SS, utilizing Wikipedia page screenshots as the corpus. The screenshot provides more comprehensive information than a plain Wikipedia paragraph.

For CIRR, FashionIQ, VisualNews, MSCOCO, WebQA, NIGHTS, OVEN and EDIS, we use the processed versions from M-BEIR (Wei et al., ).

#### A.1.4 Visual Grounding

There are a total of 4 datasets for visual grounding tasks.

MSCOCO (Lin et al., ) The dataset includes an object detection task, which involves recognizing an object from a given class in an image. We have repurposed this task into a ranking problem within the MMEB format. The query consists of the image and the object name, while the target is the cropped image of the specified object. We gather distractors from other objects in the same image as well as from different images. We discard test cases where the object is too small.

RefCOCO (Kazemzadeh et al., ) The dataset includes an object detection task that requires more reasoning than MSCOCO. Unlike simply identifying the object class, the RefCOCO dataset uses language expressions to refer to specific objects within an image. In our MMEB, we have two tasks related to RefCOCO: “RefCOCO” and “RefCOCO-Matching”. In “RefCOCO”, the query consists of the image and the language expressions referring to a specific object, while the target is the cropped image of that object. In “RefCOCO-Matching”, both the query and the target contain the image and the language expressions referring to a specific object, where the two objects are identical.

Visual7W-pointing (Zhu et al., ) The dataset establishes a semantic link between textual descriptions and image regions through object-level grounding. It has two types of questions: “telling” and “pointing”. It leverages the six W questions (what, where, when, who, why, and how) to systematically examine a model’s capability for visual understanding through telling questions. Additionally, a seventh “which” question is appended for visual answers as pointing questions. We use “Visual7W-telling” in our VQA category and “Visual7W-pointing” in our visual grounding category.

**Table 5: The detailed results of the baselines and our Vlm2Vec on MMEB, which includes 20 in-distribution datasets and 16 out-of-distribution datasets. The out-of-distribution datasets are highlighted with a yellow background in the table. We only include the best version of Vlm2Vec in the table, which was trained using LoRA.**
|  | CLIP | OpenCLIP | SigLIP | BLIP2 | MagicLens | E5-V | UniIR | Vlm2Vec |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Classification (10 tasks) |  |  |  |  |  |  |  |  |
| ImageNet-1K | 55.8 | 63.5 | 45.4 | 10.3 | 48.0 | 9.6 | 53.7 | 65.6 |
| N24News | 34.7 | 38.6 | 13.9 | 36.0 | 33.7 | 23.4 | 33.9 | 79.5 |
| HatefulMemes | 51.1 | 51.7 | 47.2 | 49.6 | 49.0 | 49.7 | 51.0 | 67.1 |
| VOC2007 | 50.7 | 52.4 | 64.3 | 52.1 | 51.6 | 49.9 | 62.7 | 88.6 |
| SUN397 | 43.4 | 68.8 | 39.6 | 34.5 | 57.0 | 33.1 | 61.7 | 72.7 |
| Place365 | 28.5 | 37.8 | 20.0 | 21.5 | 31.5 | 8.6 | 38.0 | 42.6 |
| ImageNet-A | 25.5 | 14.2 | 42.6 | 3.2 | 8.0 | 2.0 | 12.9 | 19.3 |
| ImageNet-R | 75.6 | 83.0 | 75.0 | 39.7 | 70.9 | 30.8 | 61.6 | 70.2 |
| ObjectNet | 43.4 | 51.4 | 40.3 | 20.6 | 31.6 | 7.5 | 37.1 | 29.5 |
| Country-211 | 19.2 | 16.8 | 14.2 | 2.5 | 6.2 | 3.1 | 8.8 | 13.0 |
| All Classification | 42.8 | 47.8 | 40.3 | 27.0 | 38.8 | 21.8 | 42.1 | 54.8 |
| VQA (10 tasks) |  |  |  |  |  |  |  |  |
| OK-VQA | 7.5 | 11.5 | 2.4 | 8.7 | 12.7 | 8.9 | 24.5 | 63.2 |
| A-OKVQA | 3.8 | 3.3 | 1.5 | 3.2 | 2.9 | 5.9 | 10.6 | 50.2 |
| DocVQA | 4.0 | 5.3 | 4.2 | 2.6 | 3.0 | 1.7 | 5.6 | 78.4 |
| InfographicsVQA | 4.6 | 4.6 | 2.7 | 2.0 | 5.9 | 2.3 | 5.0 | 40.8 |
| ChartQA | 1.4 | 1.5 | 3.0 | 0.5 | 0.9 | 2.4 | 1.8 | 59.0 |
| Visual7W | 4.0 | 2.6 | 1.2 | 1.3 | 2.5 | 5.8 | 12.3 | 47.7 |
| ScienceQA | 9.4 | 10.2 | 7.9 | 6.8 | 5.2 | 3.6 | 11.6 | 43.4 |
| VizWiz | 8.2 | 6.6 | 2.3 | 4.0 | 1.7 | 2.6 | 19.2 | 39.2 |
| GQA | 41.3 | 52.5 | 57.5 | 9.7 | 43.5 | 7.8 | 49.3 | 60.7 |
| TextVQA | 7.0 | 10.9 | 1.0 | 3.3 | 4.6 | 8.2 | 10.3 | 66.1 |
| All VQA | 9.1 | 10.9 | 8.4 | 4.2 | 8.3 | 4.9 | 15.0 | 54.9 |
| Retrieval (12 tasks) |  |  |  |  |  |  |  |  |
| VisDial | 30.7 | 25.4 | 21.5 | 18.0 | 24.8 | 9.2 | 37.6 | 73.3 |
| CIRR | 12.6 | 15.4 | 15.1 | 9.8 | 39.1 | 6.1 | 53.2 | 47.8 |
| VisualNews_t2i | 78.9 | 74.0 | 51.0 | 48.1 | 50.7 | 13.5 | 63.6 | 67.2 |
| VisualNews_i2t | 79.6 | 78.0 | 52.4 | 13.5 | 21.1 | 8.1 | 68.8 | 70.7 |
| MSCOCO_t2i | 59.5 | 63.6 | 58.3 | 53.7 | 54.1 | 20.7 | 72.0 | 70.6 |
| MSCOCO_i2t | 57.7 | 62.1 | 55.0 | 20.3 | 40.0 | 14.0 | 74.1 | 66.5 |
| NIGHTS | 60.4 | 66.1 | 62.9 | 56.5 | 58.1 | 4.2 | 69.7 | 66.1 |
| WebQA | 67.5 | 62.1 | 58.1 | 55.4 | 43.0 | 17.7 | 86.3 | 88.1 |
| FashionIQ | 11.4 | 13.8 | 20.1 | 9.3 | 11.2 | 2.8 | 39.3 | 12.9 |
| Wiki-SS-NQ | 55.0 | 44.6 | 55.1 | 28.7 | 18.7 | 8.6 | 11.3 | 56.6 |
| OVEN | 41.1 | 45.0 | 56.0 | 39.5 | 1.6 | 5.9 | 66.6 | 47.3 |
| EDIS | 81.0 | 77.5 | 23.6 | 54.4 | 62.6 | 26.8 | 78.2 | 79.9 |
| All Retrieval | 53.0 | 52.3 | 31.6 | 33.9 | 35.4 | 11.5 | 60.1 | 62.3 |
| Visual Grounding (4 tasks) |  |  |  |  |  |  |  |  |
| MSCOCO | 33.8 | 34.5 | 46.4 | 28.9 | 22.1 | 10.8 | 46.6 | 67.3 |
| RefCOCO | 56.9 | 54.2 | 70.8 | 47.4 | 22.8 | 11.9 | 67.8 | 84.7 |
| RefCOCO-matching | 61.3 | 68.3 | 50.8 | 59.5 | 35.6 | 38.9 | 62.9 | 79.2 |
| Visual7W-pointing | 55.1 | 56.3 | 70.1 | 52.0 | 23.4 | 14.3 | 71.3 | 86.8 |
| All Visual Grounding | 51.8 | 53.3 | 59.5 | 47.0 | 26.0 | 19.0 | 62.2 | 79.5 |
| Final Score (36 tasks) |  |  |  |  |  |  |  |  |
| All | 37.8 | 39.7 | 34.8 | 25.2 | 27.8 | 13.3 | 42.8 | 60.1 |
| All IND | 37.1 | 39.3 | 32.3 | 25.3 | 31.0 | 14.9 | 44.7 | 66.5 |
| All OOD | 38.7 | 40.2 | 38.0 | 25.1 | 23.7 | 11.5 | 40.4 | 52.0 |

**Table 6: Examples of datasets in MMEB (Part 1 of 4). Instructions are written in italic font style.**
| Category | Dataset | Query Text | Query Image | Target Text | Target Image |
| --- | --- | --- | --- | --- | --- |
| Classification | ImageNet-1K (Deng et al., ) | Represent the given image for classification |  | Italian greyhound | - |
| ImageNet-A (Hendrycks et al., ) | Represent the given image for classification. |  | sea anemone, anemone | - |  |
| ImageNet-R (Hendrycks et al., ) | Represent the given image for classification. |  | baseball player | - |  |
| N24News (Wang et al., ) | Represent the given news image with the following caption for domain classification.<br>Ms. Goodman styled Amber Valletta with wings for a 1993 shoot by Peter Lindbergh for Harper’s Bazaar. |  | Style | - |  |
| VOC2007 (Everingham et al., ) | Identify the object shown in the image. |  | bus | - |  |
| SUN397 (Xiao et al., ) | Identify the scene shown in the image. |  | firing range indoor | - |  |
| ObjectNet (Barbu et al., ) | Identify the object shown in the image. |  | mug | - |  |
| Country-211 (Radford et al., ) | Identify the country depicted in the image. |  | China | - |  |
| HatefulMemes (Kiela et al., ) | Represent the given image for binary classification to determine whether it constitutes hateful speech or not. |  | No | - |  |
| Place365 (Zhou et al., ) | Identify the scene shown in the image. |  | Airport Terminal | - |  |

**Table 7: Examples of datasets in MMEB (Part 2 of 4). Instructions are written in italic font style.**
| Category | Dataset | Query Text | Query Image | Target Text | Target Image |
| --- | --- | --- | --- | --- | --- |
| VQA | OK-VQA (Marino et al., ) | Represent the given image with the following question.<br>What breed of dog is this? |  | chihuahua | - |
| A-OKVQA (Schwenk et al., ) | Represent the given image with the following question.<br>What is the metal basket near the net used to hold? |  | tennis balls | - |  |
| DocVQA (Mathew et al., ) | Represent the given image with the following question.<br>What is name of university? |  | university of california | - |  |
| InfographicsVQA (Mathew et al., ) | Represent the given image with the following question.<br>Which social platform has heavy female audience? |  | pinterest | - |  |
| ChartQA (Masry et al., ) | Represent the given image with the following question.<br>How many food item is shown in the bar graph? |  | 14 | - |  |
| ScienceQA (Lu et al., ) | Represent the given image with the following question.<br>Which of these states is farthest north? |  | South Carolina | - |  |
| Visual7W-telling (Zhu et al., ) | Represent the given image with the following question.<br>Where is the man sitting? |  | At the computer | - |  |
| VizWiz (Gurari et al., ) | Represent the given image with the following question.<br>Can you tell me what this medicine is please? |  | night time | - |  |
| GQA (Hudson & Manning, ) | Represent the given image with the following question.<br>What is under the utensil on the left? |  | The napkin is under the utensil. | - |  |
| TextVQA (Singh et al., ) | Represent the given image with the following question.<br>What is the brand of this camera? |  | dakota | - |  |

**Table 8: Examples of datasets in MMEB (Part 3 of 4). Instructions are written in italic font style.**
| Category | Dataset | Query Text | Query Image | Target Text | Target Image |
| --- | --- | --- | --- | --- | --- |
| Retrieval | VisDial (Das et al., ) | Represent the given dialogue about an image, which is used for image retrieval.<br>Q:do you see a lot of people A:just 3 Q:what is the tennis player wearing A:white tennis dress Q:what color is her tennis racket A:black Q:is she wearing a hat A:a visor Q:is she close to the net A:no Q:do you see another player A:no Q:do you see a tennis bag A:no | - | Represent the given image. |  |
| VisualNews_t2i (Liu et al., ) | Retrieve an image of this news caption.<br>US goalkeeper Hope Solo makes a save. | - | Represent the given image. |  |  |
| MSCOCO_t2i (Lin et al., ) | Find me an everyday image that matches the given caption.<br>Man riding a motor bike on a dirt road on the countryside. | - | Represent the given image. |  |  |
| WebQA (Chang et al., ) | Find a Wikipedia image-passage pair that answers this question.<br>Do both the Hays County Courthouse in San Marcos, Texas and the Ike Wood House at 227 Mitchell Street in San Marcos, Texas have six columns on their front entrance? | - | Represent the given Wikipedia image with related text information.<br>Hays County Courthouse (2018), San Marcos, TX The Hays County Courthouse in San Marcos, Texas. Listed on the National Register of Historic Places. 227 Mitchell, San Marcos, Texas Ike Wood House at 227 Mitchell Street in San Marcos, Texas. |  |  |
| EDIS (Liu et al., ) | Find a news image that matches the provided caption.<br>Tom Holland makes his debut in the Spidey suit in Captain America Civil War. | - | Represent the given image with related text information.<br>Comic RiffsJon Favreau is set to reprise his Iron Man role for Spider Man: Homecoming. |  |  |
| Wiki-SS-NQ (Ma et al., ) | Find the document screenshot that can answer the given query. | - | Represent the given document screenshot. |  |  |
| VisualNews_i2t (Liu et al., ) | Find a caption for the news in the given photo. |  | Canadian Prime Minister Stephen Harper shakes hands with President Obama during the North American Leaders Summit in Toluca Mexico in February 2014. | - |  |
| MSCOCO_i2t (Lin et al., ) | Find an image caption describing the given everyday image. |  | A man on a bicycle riding next to a train. | - |  |

**Table 9: Examples of datasets in MMEB (Part 4 of 4). Instructions are written in italic font style.**
| Category | Dataset | Query Text | Query Image | Target Text | Target Image |
| --- | --- | --- | --- | --- | --- |
| Retrieval | CIRR (Liu et al., ) | Given an image, find a similar everyday image with the described changes.<br>Show three bottles of soft drink. |  | Represent the given image. |  |
| FashionIQ (Wu et al., ) | Find an image to match the fashion image and style note.<br>Is shiny and silver with shorter sleeves and fit and flare. |  | Represent the given image. |  |  |
| NIGHTS (Fu et al., ) | Find a day-to-day image that looks similar to the provided image. |  | Represent the given image. |  |  |
| OVEN (Hu et al., ) | Retrieve a Wikipedia image-description pair that provides evidence for the question of this image.<br>What is the name of this place? |  | Represent the given Wikipedia image with related text information.<br>Titisee. The Titisee is a lake in the southern Black Forest in Baden-Württemberg. It covers an area of 1.3 (km2) and is an average of 20 (m) deep. It owes its formation to the Feldberg glacier, the moraines of which were formed in the Pleistocene epoch and nowadays form the shores of the lake. The lake’s outflow, at 840 (m) above sea level, is the River Gutach, which merges with the Haslach stream below Kappel to form the Wutach. The waters of the Titisee thus drain eventually into the Upper Rhine between Tiengen and Waldshut. On the north shore lies the. |  |  |
| Grounding | MSCOCO (Lin et al., ) | Select the portion of the image that isolates the object of the given label<br>The lable of the object is “stop sign”. |  | Represent the given cropped image of the object. |  |
| Visual7W-Pointing (Zhu et al., ) | Select the portion of the image that answers the given question.<br>Which door is behind a person sitting on a bench? |  | Represent the given cropped image of the object. |  |  |
| RefCOCO (Kazemzadeh et al., ) | Select the portion of the image that follows the language expressions.<br>man in black coat |  | Represent the given cropped image of the object. |  |  |
| RefCOCO-Matching (Kazemzadeh et al., ) | Select the portion of the image that follows the language expressions.<br>kid on right in back, blondish hair |  | Select the portion of the image that follows the language expressions.<br>top right kid |  |  |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: **[Q1] About [EOS] token in the learning objective.** VLM2VEC applies the InfoNCE loss to [EOS] tokens. I understand that the output [EOS] token reflects information gathered through the model’s attention over all preceding input tokens. Could the [EOS] token be replaced by a class token or learnable token? I expect that there will not be a big difference in performance. I wonder if the authors have a valid reason for taking [EOS] tokens or can provide results regarding output token selection.

**[Q2] Inference procedure.** [EOS] token of the output may be used as a single multimodal embedding. Although the inference procedure for each meta-task is well known, it would be good to explain the inference procedure through multimodal embedding. Specifically, I think that classification, VQA, and retrieval can be performed by measuring the similarity between outputs. However, it is not easy to guess how the inference of visual grounding is done.
soundness: 3
strengths: **[S1]** This paper is well-motivated and well-written.

**[S2]** I strongly agree with the requirement for a benchmark for evaluating multimodal embedding. The MMEB benchmark covers a wide range of tasks and is therefore sufficient to assess multimodal embeddings' capability appropriately.

**[S3]** This paper presents extensive experimental results with several baselines for multimodal embedding, providing a good benchmark for the following research.
summary: This paper presents to learn universal multimodal embedding models and introduces a novel benchmark (massive multimodal embedding benchmark, MMEB) to evaluate multimodal embedding.
The proposed VLM2VEC framework makes any state-of-the-art vision-language models (VLMs) multimodal embedding models through contrastive learning.
The MMEB dataset is collected from 36 public datasets (e.g. ImageNet, MSCOCO), covering 4 meta-tasks (e.g. VQA, retrieval).
Extensive experiments provide several baselines for multimodal embeddings and validate the effectiveness of the proposed VLM2VEC framework.
weaknesses: **[W1] Novelty of VLM2VEC.**
The main concern is the limited novelty of the proposed VLM2VEC framework.
The proposed framework is significantly similar to the recent works that ground LLMs to visual data with pretrained image models and large language models, such as [1, 2].
In other words, projecting visual representations into LLM's input space and tuning the whole or partial model with text generation or contrastive objectives is a typical approach for the recent multimodal large language models.
Although I acknowledge the contribution to the purpose and necessity of this work, it would be good to emphasize the difference that the proposed framework is exclusive only for multimodal embedding.

```
[1] Li et al., "Visual instruction tuning," NeurIPS, 2024
[2] Xue et al., "xGen-MM (BLIP-3): A family of open large multimodal models," arXiv, 2024
```

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Refer to weaknesses.
soundness: 2
strengths: 1. Extensive experiments are conducted using multiple datasets and baselines, demonstrating the effectiveness of the proposed framework.
2. The development of the MMEB is original, as it provides a comprehensive framework for training and evaluating embedding models across diverse tasks and modalities.
3. Detailed ablation studies are performed to analyze the impact of various hyperparameters and training strategies on the model's performance.
summary: This paper proposes a novel framework called VLM2Vec for training vision-language models into embedding models capable of handling massive multimodal embedding tasks. This paper proposes MMEB, a benchmark with 36 datasets spanning classification, visual question answering, retrieval, and visual grounding. VLM2Vec deeply integrates visual and textual features, enabling it to capture cross-modal relationships more effectively.
weaknesses: Here are my constructive insights on how the paper could improve:
1. Benchmark Design: The number of candidates for each retrieval task in the MMEB benchmark is limited to 1000, which may not accurately reflect real-world scenarios where the number of candidates could be much larger. Increasing the number of candidates would make the benchmark more realistic and challenging.
2. Instead of randomly selecting 1000 candidates for each task, the standard test sets provided by the original datasets should be used. This would ensure a fairer comparison with other methods that have been evaluated on these datasets.
3. Fair Comparison: Table 2 compares VLM2Vec with baseline methods that have not been trained on the MMEB training sets. For a more accurate comparison, it would be insightful to retrain the baseline methods on the same training sets and then compare their performance with VLM2Vec. 
4. Baseline Evaluation: The paper mentions UniIR but does not include results for the UniIR-CLIP version, which according to the UniIR paper, performs better than the UniIR-BLIP version. Including these results would provide a more comprehensive comparison.
5. The MMEB benchmark reformulates classification, VQA, retrieval, and visual grounding tasks as ranking problems. While ranking-based formulations can be useful for classification and retrieval tasks, applying the same approach to VQA and visual grounding might not be ideal. These tasks often require more nuanced understanding and reasoning. Considering alternative formulations, such as direct answer generation for VQA and object localization for visual grounding, might yield more meaningful results and better reflect the nature of these tasks.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. What do the x-axis and legend in Figure 5 represent? What do the three subplots represent respectively?
2. Does this training method affect the original capabilities of MLLM? I'm curious about the performance of VLM2Vec on MLLM-related benchmarks, such as MMBench and MME.
soundness: 2
strengths: 1. The writing is clear and easy to understand. This paper is easy to follow.
2. The constructed benchmark, MMEB, covers a wide range of tasks and is valuable.
summary: This paper focuses on the development of universal multimodal embedding models. The authors introduce the Massive Multimodal Embedding Benchmark (MMEB), which comprises classification, visual question answering (VQA), retrieval, and grounding tasks. They propose a contrastive training framework, VLM2Vec, to transform VLM into an embedding model. Experimental results demonstrate that VLM2Vec achieves promising performance on the MMEB.
weaknesses: 1. The experiments presented are insufficient, as the authors limit their evaluation to the proposed MMEB benchmarks using only the precision metric. To provide a more comprehensive comparison with other clip-like models, it is crucial to assess the embedding model on some traditional benchmarks, such as ELEVATER and zero-shot retrieval on COCO and Flickr30k.
2. The evaluation of out-of-distribution may be unfair. Current MLLMs are usually fine-tuned on various downstream datasets. Most of datasets that construct MMEB are usually used in the fine-tuning stage of MLLMs. Since Phi-3.5-V doesn't provide details of fine-tuning datasets,  it is hard to evaluate the out-of-distribution capabilities of VLM2Vec.
3. Lack of some experimental details. The authors need to provide details of the training of the comparative models on the MMEB in-domain datasets. Additionally, it is unfair to compare CLIP without instructions. Although the authors claim that based on other related work, introducting instructions to CLIP would not perform well, Table 4 shows that instructions have a significant impact, and the CLIP model should also have experimental results under the same settings.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: n/a
soundness: 3
strengths: 1. The method introduced in this paper is robust. With the authors' commitment to open- sourcing both the model and data, the proposed approach will to be valuable for a wide range of applications.
2. The experiments are thorough, effectively demonstrating the model's efficacy.
3. The paper is well-structured and clearly written.
summary: This paper presents a multimodal embedding framework and introduces a comprehensive dataset, MMEB, which includes 36 datasets across four meta-task categories, specifically designed for training and evaluating embedding models. Additionally, the paper proposes VLM2VEC, a contrastive training framework that adapts an existing vision-language model into an embedding model. Experimental results indicate that the proposed model achieves 10%~20% improvement over previous multimodal embedding models, on both in-distribution and out-of-distribution datasets.
weaknesses: 1. The method proposed in this paper is relatively straightforward. Its performance, however, varies significantly across different tasks. Would suggest including more comparative experiments to explore and clarify the reasons behind these performance differences across tasks.

Meta-review: This paper studies two tasks: 1. evaluate the multimodel embedding. 2. how to convert the VLM to a embedding model. For the task 1, this paper proposed a massive benchmark: MMEB, which can comprehensively evaluate the embedding model performance. For the task 2, the paper proposed VLM2Vec, which is LoRA finetuned VLM. The model is optimized via contrastive learning. 

Strength:
1. The multimodal embedding evaluation task is a quite important problem, while it has been fully explored compared to the text side.
2. The VLM2Vec is a great idea for converting the VLM to embedding model. The LoRA finetuned version shows great potential.

Weakness:
1. The reviewer argues the fairness (# of parameters) in the comparison b/w CLIP and the VLM
2. The reviewer argues the VLM might train on additional data which might achieve advantage on those benchmark over CLIP.
3. The negative space of the benchmark might not enough.

After reading the paper and the rebuttal, based on my judgement and my experience in multimodal LLM and representation learning, I think it is important to have a massive benchmark can reflect the model performance, as imagenet and coco retrieval is hard to justify the effectiveness of the approach. Furthermore, how to convert the VLM to Vec is an open question in both academia and industry. I think this paper proposed a simple yet effective approach to tackle this open question. 

Given the above consideration, I would recommend Accept.

Decision: Accept (Poster)