VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

Authors: Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, Wenhu Chen
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Vision Language Model, Representation Learning, Multimodal Embeddings

Abstract:
Embedding models play a crucial role in a variety of downstream tasks, including semantic similarity, information retrieval, and clustering. While there has been a surge of interest in developing universal text embedding models that generalize across tasks (e.g., MTEB), progress in learning universal multimodal embedding models has been comparatively slow, despite their importance and practical applications. In this work, we explore the potential of building universal multimodal embeddings capable of handling a broad range of downstream tasks. 

Our contributions are twofold: (1) we propose MMEB (Massive Multimodal Embedding Benchmark), which covers four meta-tasks (classification, visual question answering, multimodal retrieval, and visual grounding) and 36 datasets, including 20 training datasets and 16 evaluation datasets spanning both in-distribution and out-of-distribution tasks, and (2) VLM2Vec (Vision-Language Model → Vector), a contrastive training framework that transforms any vision-language model into an embedding model through contrastive training on MMEB. 

Unlike previous models such as CLIP and BLIP, which encode text and images independently without task-specific guidance, VLM2Vec can process any combination of images and text while incorporating task instructions to generate a fixed-dimensional vector. We develop a series of VLM2Vec models based on state-of-the-art VLMs, including Phi-3.5-V, LLaVA-1.6, and Qwen2-VL, and evaluate them on MMEB’s benchmark. With LoRA tuning, VLM2Vec achieves a 10% to 20% improvement over existing multimodal embedding models on MMEB’s evaluation sets. Our findings reveal that VLMs are surprisingly strong embedding models.

Decision: accept (poster)
Reviewer scores: 8, 5, 3, 8

Reviews and discussion:
<thread>
creation date:
1730596682295

thread_type:
official_review

<thread_object>
creator: reviewer_z3vp

<content>
summary:
This paper presents a multimodal embedding framework and introduces a comprehensive dataset, MMEB, which includes 36 datasets across four meta-task categories, specifically designed for training and evaluating embedding models. Additionally, the paper proposes VLM2VEC, a contrastive training framework that adapts an existing vision-language model into an embedding model. Experimental results indicate that the proposed model achieves 10%~20% improvement over previous multimodal embedding models, on both in-distribution and out-of-distribution datasets.

strengths:
1. The method introduced in this paper is robust. With the authors' commitment to open- sourcing both the model and data, the proposed approach will to be valuable for a wide range of applications.
2. The experiments are thorough, effectively demonstrating the model's efficacy.
3. The paper is well-structured and clearly written.

weaknesses:
1. The method proposed in this paper is relatively straightforward. Its performance, however, varies significantly across different tasks. Would suggest including more comparative experiments to explore and clarify the reasons behind these performance differences across tasks.

questions:
n/a
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622813476

thread_type:
official_review

<thread_object>
creator: reviewer_hg4f

<content>
summary:
This paper focuses on the development of universal multimodal embedding models. The authors introduce the Massive Multimodal Embedding Benchmark (MMEB), which comprises classification, visual question answering (VQA), retrieval, and grounding tasks. They propose a contrastive training framework, VLM2Vec, to transform VLM into an embedding model. Experimental results demonstrate that VLM2Vec achieves promising performance on the MMEB.

strengths:
1. The writing is clear and easy to understand. This paper is easy to follow.
2. The constructed benchmark, MMEB, covers a wide range of tasks and is valuable.

weaknesses:
1. The experiments presented are insufficient, as the authors limit their evaluation to the proposed MMEB benchmarks using only the precision metric. To provide a more comprehensive comparison with other clip-like models, it is crucial to assess the embedding model on some traditional benchmarks, such as ELEVATER and zero-shot retrieval on COCO and Flickr30k.
2. The evaluation of out-of-distribution may be unfair. Current MLLMs are usually fine-tuned on various downstream datasets. Most of datasets that construct MMEB are usually used in the fine-tuning stage of MLLMs. Since Phi-3.5-V doesn't provide details of fine-tuning datasets,  it is hard to evaluate the out-of-distribution capabilities of VLM2Vec.
3. Lack of some experimental details. The authors need to provide details of the training of the comparative models on the MMEB in-domain datasets. Additionally, it is unfair to compare CLIP without instructions. Although the authors claim that based on other related work, introducting instructions to CLIP would not perform well, Table 4 shows that instructions have a significant impact, and the CLIP model should also have experimental results under the same settings.

questions:
1. What do the x-axis and legend in Figure 5 represent? What do the three subplots represent respectively?
2. Does this training method affect the original capabilities of MLLM? I'm curious about the performance of VLM2Vec on MLLM-related benchmarks, such as MMBench and MME.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698181459

thread_type:
official_review

<thread_object>
creator: reviewer_kzgz

<content>
summary:
This paper proposes a novel framework called VLM2Vec for training vision-language models into embedding models capable of handling massive multimodal embedding tasks. This paper proposes MMEB, a benchmark with 36 datasets spanning classification, visual question answering, retrieval, and visual grounding. VLM2Vec deeply integrates visual and textual features, enabling it to capture cross-modal relationships more effectively.

strengths:
1. Extensive experiments are conducted using multiple datasets and baselines, demonstrating the effectiveness of the proposed framework.
2. The development of the MMEB is original, as it provides a comprehensive framework for training and evaluating embedding models across diverse tasks and modalities.
3. Detailed ablation studies are performed to analyze the impact of various hyperparameters and training strategies on the model's performance.

weaknesses:
Here are my constructive insights on how the paper could improve:
1. Benchmark Design: The number of candidates for each retrieval task in the MMEB benchmark is limited to 1000, which may not accurately reflect real-world scenarios where the number of candidates could be much larger. Increasing the number of candidates would make the benchmark more realistic and challenging.
2. Instead of randomly selecting 1000 candidates for each task, the standard test sets provided by the original datasets should be used. This would ensure a fairer comparison with other methods that have been evaluated on these datasets.
3. Fair Comparison: Table 2 compares VLM2Vec with baseline methods that have not been trained on the MMEB training sets. For a more accurate comparison, it would be insightful to retrain the baseline methods on the same training sets and then compare their performance with VLM2Vec. 
4. Baseline Evaluation: The paper mentions UniIR but does not include results for the UniIR-CLIP version, which according to the UniIR paper, performs better than the UniIR-BLIP version. Including these results would provide a more comprehensive comparison.
5. The MMEB benchmark reformulates classification, VQA, retrieval, and visual grounding tasks as ranking problems. While ranking-based formulations can be useful for classification and retrieval tasks, applying the same approach to VQA and visual grounding might not be ideal. These tasks often require more nuanced understanding and reasoning. Considering alternative formulations, such as direct answer generation for VQA and object localization for visual grounding, might yield more meaningful results and better reflect the nature of these tasks.

questions:
Refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730878559484

thread_type:
official_review

<thread_object>
creator: reviewer_bv32

<content>
summary:
This paper presents to learn universal multimodal embedding models and introduces a novel benchmark (massive multimodal embedding benchmark, MMEB) to evaluate multimodal embedding.
The proposed VLM2VEC framework makes any state-of-the-art vision-language models (VLMs) multimodal embedding models through contrastive learning.
The MMEB dataset is collected from 36 public datasets (e.g. ImageNet, MSCOCO), covering 4 meta-tasks (e.g. VQA, retrieval).
Extensive experiments provide several baselines for multimodal embeddings and validate the effectiveness of the proposed VLM2VEC framework.

strengths:
**[S1]** This paper is well-motivated and well-written.

**[S2]** I strongly agree with the requirement for a benchmark for evaluating multimodal embedding. The MMEB benchmark covers a wide range of tasks and is therefore sufficient to assess multimodal embeddings' capability appropriately.

**[S3]** This paper presents extensive experimental results with several baselines for multimodal embedding, providing a good benchmark for the following research.

weaknesses:
**[W1] Novelty of VLM2VEC.**
The main concern is the limited novelty of the proposed VLM2VEC framework.
The proposed framework is significantly similar to the recent works that ground LLMs to visual data with pretrained image models and large language models, such as [1, 2].
In other words, projecting visual representations into LLM's input space and tuning the whole or partial model with text generation or contrastive objectives is a typical approach for the recent multimodal large language models.
Although I acknowledge the contribution to the purpose and necessity of this work, it would be good to emphasize the difference that the proposed framework is exclusive only for multimodal embedding.

```
[1] Li et al., "Visual instruction tuning," NeurIPS, 2024
[2] Xue et al., "xGen-MM (BLIP-3): A family of open large multimodal models," arXiv, 2024
```

questions:
**[Q1] About [EOS] token in the learning objective.** VLM2VEC applies the InfoNCE loss to [EOS] tokens. I understand that the output [EOS] token reflects information gathered through the model’s attention over all preceding input tokens. Could the [EOS] token be replaced by a class token or learnable token? I expect that there will not be a big difference in performance. I wonder if the authors have a valid reason for taking [EOS] tokens or can provide results regarding output token selection.

**[Q2] Inference procedure.** [EOS] token of the output may be used as a single multimodal embedding. Although the inference procedure for each meta-task is well known, it would be good to explain the inference procedure through multimodal embedding. Specifically, I think that classification, VQA, and retrieval can be performed by measuring the similarity between outputs. However, it is not easy to guess how the inference of visual grounding is done.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732638905613

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revision

comment:
We thank the reviewers for the time and expertise they devoted to reviewing our paper. We appreciate that the reviewers found our work to be well-motivated, recognized the value of our proposed MMEB benchmark, and acknowledged that our extensive experiments and ablations demonstrate the effectiveness of the VLM2Vec framework.

And we are grateful for the constructive feedback. To help better address the questions and concerns raised by the reviewers and further improve the paper's quality, we made the following updates to the paper PDF. These contents are also used in our response to the reviewers

* Benchmark design (**Reviewer kzgZ**): We have added **Appendix A.2** and **Table 5** to explain why we chose 1,000 as the number of candidates, striking a balance between evaluation cost and benchmark difficulty.

* Baseline evaluation using the best version of the UniIR model (**Reviewer kzgZ**): We have updated the results of this baseline model in our **Table 2**.

* Baseline CLIP-based model with instruction (**Reviewer hG4f**): We have updated **Section 4.3.4** and **Table 4** to discuss the impact of instructions on the CLIP-based model, explain why we do not use instructions with the CLIP-based model, and highlight VLM2Vec’s instruction-following capability.

* Fine-tune baseline models on MMEB training sets and include the training setup (Reviewer **kzgZ**, Reviewer **hG4f**): We have updated **Section 4.1** and **Table 2** to include two baseline models fine-tuned on our own datasets.

* More clarifications for Figure 5 (**Reviewer hG4f**): We have updated the caption of **Figure 5** for improved clarity.

* On OOD Evaluation (**Reviewer hG4f**): We have added the results of LlaVA-1.6-based VLM2Vec to **Table 2**. Compared to Phi-3.5-v, LlaVA-1.6 offers a transparent training recipe, and the OOD datasets from our MMEB are valid. We have demonstrated that VLM2Vec is an effective framework capable of supporting different VLMs.

* On Evaluating Traditional Benchmarks (**Reviewer hG4f**): We have added zero-shot retrieval performance on Flickr30K in **Table 11**.

* On inference procedure for visual grounding (Reviewer **bv32**): We have updated **Section 2.2** to clarify the description of the visual grounding task.

Once again, we would like to thank all the reviewers for their constructive suggestions, which have helped us improve our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies two tasks: 1. evaluate the multimodel embedding. 2. how to convert the VLM to a embedding model. For the task 1, this paper proposed a massive benchmark: MMEB, which can comprehensively evaluate the embedding model performance. For the task 2, the paper proposed VLM2Vec, which is LoRA finetuned VLM. The model is optimized via contrastive learning. 

Strength:
1. The multimodal embedding evaluation task is a quite important problem, while it has been fully explored compared to the text side.
2. The VLM2Vec is a great idea for converting the VLM to embedding model. The LoRA finetuned version shows great potential.

Weakness:
1. The reviewer argues the fairness (# of parameters) in the comparison b/w CLIP and the VLM
2. The reviewer argues the VLM might train on additional data which might achieve advantage on those benchmark over CLIP.
3. The negative space of the benchmark might not enough.

After reading the paper and the rebuttal, based on my judgement and my experience in multimodal LLM and representation learning, I think it is important to have a massive benchmark can reflect the model performance, as imagenet and coco retrieval is hard to justify the effectiveness of the approach. Furthermore, how to convert the VLM to Vec is an open question in both academia and industry. I think this paper proposed a simple yet effective approach to tackle this open question. 

Given the above consideration, I would recommend Accept.