E5-V: Universal Embeddings with Multimodal Large Language Models

## Abstract

Abstract Multimodal large language models (MLLMs) have shown promising advancements in general visual and language understanding. However, the representation of multimodal information using MLLMs remains largely unexplored. In this work, we introduce a new framework, E5-V, designed to adapt MLLMs for achieving universal multimodal embeddings.
Our findings highlight the significant potential of MLLMs in representing multimodal inputs compared to previous approaches. By leveraging MLLMs with prompts, E5-V effectively bridges the modality gap between different types of inputs, demonstrating strong performance in multimodal embeddings even without fine-tuning.
We propose a single modality training approach for E5-V, where the model is trained exclusively on text pairs. This method demonstrates significant improvements over traditional multimodal training on image-text pairs, while reducing training costs by approximately 95%. Additionally, this approach eliminates the need for costly multimodal training data collection.
Extensive experiments across four types of tasks demonstrate the effectiveness of E5-V. As a universal multimodal model, E5-V not only achieves but often surpasses state-of-the-art performance in each task, despite being trained on a single modality.

## 1 Introduction

With the development of MLLMs, there is an increasing need for embedding models to represent multimodal inputs.
Although CLIP [] shows impressive results in text-image retrieval by aligning visual and language representations with contrastive learning, it struggles to represent interleaved visual and language inputs. Moreover, the text encoder of CLIP demonstrates a low capacity for understanding complicated text []. To achieve universal multimodal representation, some works [, ] continue to train CLIP on interleaved image-text data, while collecting such data can be challenging and may require GPT-4 to synthesize data  [] or mannualy annotated.

Recent works demonstrate [, ] that scaling up the size of text embedding models leads to better performance. However, replicating this scaling approach for universal multimodal embeddings poses significant challenges and expenses, which arises from the unstable for scaling CLIP and the complexity of collecting extensive multimodal datasets [].
Nevertheless, previous works like adapting CLIP to universal multimodal embeddings still has shortcomings, such as poor language understanding, limited real-world knowledge, and shallow fusion of visual and linguistic information.

In this work, we introduce a new framework, called E5-V, to directly adapt MLLMs instead of CLIP like models for achieving universal multimodal embeddings. There are several advantages to representing multimodal information with MLLMs: First, benefiting from interleaved visual and language training, MLLMs can initially learn to represent multimodal information according to their meanings with prompt. Second, MLLMs are capable of representing interleaved visual and language inputs to handle tasks like composed image retrieval. Third, MLLMs have stronger language understanding and reasoning capabilities compared to CLIP.

However, since MLLMs are not initially trained with contrastive learning to represent inputs as embeddings, it can be challenging for them to represent multimodal inputs as well as CLIP, which performs contrastive learning on large-scale text-image pairs.
In this work, we propose a prompt-based representation method to adapt MLLMs for multimodal embeddings inspired by [].
By explicitly instructing MLLMs to represent multimodal inputs into words in Figure 1,
this method initially unifies multimodal embeddings into the same space, which direclty remove the modality gap [] in multimodal embeddings.

By unifying multimodal embeddings into the same space, MLLMs are able to achieve robust multimodal embedding performance through single modality training with only on text inputs.
This eliminates the need for expensive multimodal training data collection. By focusing solely on text data, we can remove other components, such as the visual encoder, in the MLLMs during training and decrease the input size, significantly reducing the training cost.
Compared to multimodal training, we observe training solely on text pairs even help MLLMs better represent multimodal inputs than image-text pairs, and find text pairs can be more effective in contrastive learning than image-text pairs.

To validate the effectiveness of E5-V, we conduct experiments on various tasks: text-image retrieval, composed image retrieval, sentence embeddings, and image-image retrieval.
By comparing E5-V with the strong baselines of each task, we demonstrate the effectiveness of E5-V in representing multimodal information, which achieves competitive performance on all tasks as a universal multimodal embeddings model trained on text pairs only.

Our contributions are as follows:

- •
We study how to achieve universal multimodal embeddings by leveraging MLLMs. By designing prompts to project multimodal inputs into the same embedding spaces, we show that MLLMs can represent multimodal inputs correctly even without fine-tuning.
- •
We introduce a new framework, E5-V, to adapt MLLMs for achieving universal multimodal embeddings. With single modality training on text pairs, E5-v even achieve better multimodal embeddings than image-text pairs.
- •
Extensive experiments on text-image retrieval and composed image retrieval tasks demonstrate the effectiveness of E5-V in representing multimodal information. E5-V successfully transfers single modality representation capabilities to multimodal embeddings by following task-specific prompts that were not included in the training data.

## 2 Related Work

### 2.1 Multimodal Large Language Models

With the success of LLMs, there is a trend to extend LLMs to handle multimodal information, called MLLMs. MLLMs, such as BLIP [], KOSMOS [], LLaMA-Adapter [], and LLaVA [, , ], show promising progress in multimodal information understanding and reasoning.
To achieve this, a typical MLLM is composed of an LLM, a modality encoder, and a projector to connect them. The modality encoder projects raw multimodal inputs into vectors to connect with LLMs [].

One efficient method [, ] is to directly use a pretrained LLM and a pretrained modality encoder, such as CLIP []. To achieve this, LLaVA uses two training stages. The first stage aligns the text and image with image-text pairs by only training the projector between LLM and modality encoder, and the second stage fine-tunes the model on a visual instruction dataset, which ensure it can follow complex instructions like LLM, such as represent the multimodal inputs in our work.

While the impressive performance of MLLMs in understanding multimodal information and instruction following, the representation of multimodal information using MLLMs remains largely unexplored. Although recent studies [, ] have shown advancements in embedding texts with LLMs and scaling up has demonstrated improved performance in text representation, a significant challenge persists: the batch sizes requires to train multimodal embeddings models such as CLIP is signficantly larger than text embeddings models.
For example, CLIP requires a batch size of 32k samples with contrastive learning, while the text embedding models such as E5 [] only requires a batch size of 2k samples.
Limited by the size of MLLMs, it can be very challenging to use the similar batch size like CLIP to train robust multimodal embeddings models.

### 2.2 Multimodal Embeddings

CLIP [], as a pioneering work on multimodal embeddings, has been widely used in subsequent works.
CLIP uses separate encoders for image and text by aligning them with contrastive learning on large-scale image-text pairs.
Despite its strong performance in text-image retrieval, CLIP has several limitations due to its internal framework. The text encoder of CLIP has a low capacity for understanding complicated text because it is pretrained on short image captions, which also limits CLIP’s performance on long text retrieval []. Additionally, due to the use of separate encoders, CLIP struggles to represent interleaved visual and language inputs, such as in composed image retrieval [, ].

To achieve universal multimodal embeddings, there are several works, such as and UNIIR [], fine-tune CLIP on interlevaed dataset with fusion modal to fuse the visual and language information.
There are also some works like VISTA [] or UniVL-DR [] feed text encoder with CLIP outputs to input visual information. However, it can harm the original text-image retrieval performance of CLIP, and is hard to make text encoder understand the visual information with only contrastive learning, which show poor zero-shot performance on composed image retrieval tasks.

To achieve universal multimodal embeddings, several works, such as UNIIR [], fine-tune CLIP with a fusion model to integrate visual and language information. Other works, like VISTA [] or UniVL-DR [], feed the text embedding models with CLIP outputs to incorporate visual information. However, this approach can harm the original text-image retrieval performance of CLIP and makes it difficult for the text embedding models to understand visual information using only contrastive learning. As a result, these methods show poor zero-shot performance on composed image retrieval tasks.
Moreover, these methods require large interleaved training data to achieve universal multimodal embeddings. Collecting such high-quality interleaved pairs for performing contrastive learning is more challenging than gathering image-text pairs or text pairs. This process can require complex annotation and sometimes even synthesizing data from GPT-4 [].

Figure: Figure 2: Single modality training in E5-V. By unifying multimodal representations into the same embedding space with prompts, E5-V improves multimodal embeddings using only contrastive learning on text pairs. During training, we remove the modality encoder and projector in MLLM.
Refer to caption: extracted/5737908/fig2.png

## 3 E5-V

### 3.1 Unifying Multimodal Embeddings

Previous works [] have demonstrated the existence of a modality gap between text and image embeddings in multimodal models like CLIP [], which can negatively impact the performance of multimodal embeddings. Similarly, we observe this phenomenon when using MLLMs to represent multimodal inputs.

We visualize the distribution of multimodal embeddings from MLLM in Figure 3(a) following  []. For implementation, we use the last token embeddings of LLaVA-NeXT-8B [] to represent the images and captions of COCO. The embeddings are obtained directly from MLLM without fine-tuning and visualized with PCA.
Compared to CLIP, although MLLM represents the image and text with the same encoder, the multimodal embeddings from MLLM show a clear modality gap between text and image embeddings.

Figure: (a) w/o our method
Refer to caption: extracted/5737908/dis_2.png

To unify multimodal embeddings, we propose a prompt-based representation method with MLLMs inspired by previous text embedding work [].
The key idea is to explicitly instruct MLLMs to represent the multimodal inputs into words. We can use prompts like <text> \n Summary of the above sentence in one word: to represent the text and <image> \n Summary above image in one word: to represent the image.
We notice these prompts directly remove the modality gap between text and image embeddings, as shown in Figure 3(b).
For the design of the prompts, it has two parts: the first part is about extracting the meaning of the multimodal inputs, and the second part is about compressing the meaning into the next token embeddings and unifying the multimodal embeddings by using in one word:.
Specifically,
the embeddings of image and caption about a plane in Figure 1 will have a close distance to the token embeddings of “Plane”, “Air”, “Flying” and “Above”, which represent multimodal inputs based on the corresponding meaning instead of their modality.
By removing modality gap, it also allows MLLMs to represent interleaved inputs for tasks like composed image retrieval.
We show our method significantly improves the performance of MLLM on multimodal retrieval tasks in Table 6.

### 3.2 Single Modality Training

By unifying multimodal embeddings, we propose single modality training for multimodal embeddings, as shown in Figure 2. Since there is no longer a modality gap in the embeddings, we can transfer the single modality representation capabilities to multimodal embeddings by training on text pairs only. In this way, our method is trained without any visual or interleaved inputs and no longer relies on multimodal training data, which can be difficult to collect.

To achieve it, E5-V trains MLLMs with contrastive learning on text pairs. Since there are no visual inputs during training, we remove the modality encoder and projector and only remain the LLM of MLLM.
For the training data, we simply use sentence pairs from NLI datasets following [], which have no relation to the multimodal tasks.
Each sentence pair $(x_{i},x_{i}^{+},x_{i}^{-})$ has a positive sentence $x_{i}^{+}$ and a negative sentence $x_{i}^{-}$ for the input sentence $x_{i}$.
We use the prompt <text> \n Summary above sentence in one word: to embed the sentence pairs into $(\mathbf{h}_{i},\mathbf{h}_{i}^{+},\mathbf{h}_{i}^{-})$.
The training objective is folowing:

$$ $\mathcal{L}=-\log\frac{e^{\operatorname{cos}\left(\mathbf{h}_{i},\mathbf{h}_{i }^{+}\right)/\tau}}{\sum_{j=1}^{N}\left(e^{\operatorname{cos}\left(\mathbf{h}_ {i},\mathbf{h}_{j}^{+}\right)/\tau}+e^{\operatorname{cos}\left(\mathbf{h}_{i}, \mathbf{h}_{j}^{-}\right)/\tau}\right)}$ (1) $$

where $\tau$ is the temperature hyperparameter and $N$ is the batch size and in contrastive learning.
Compared to multimodal training, we find single modality training achieve better performance on multimodal retrieval tasks, while significantly reducing the training cost in Table 7.

## 4 Experiments

We evaluate E5-V on four tasks: text-image retrieval, composed image retrieval, sentence embeddings, and image-image retrieval to demonstrate the effectiveness of E5-V in representing multimodal information. All tasks are evaluated in a zero-shot setting with the same model without additional fine-tuning on specific datasets.

For the backbone of E5-V, we use LLaVA-NeXT-8B [], which builds on LLaMA-3 8B [], with a frozen CLIP ViT-L as the visual encoder. For the training data, we use NLI sentence pairs from [], with around 273k sentence pairs. We fine-tune the LLM of LLaVA-NeXT-8B with 1000 steps and 768 batch size.
To save the GPU memory, we use QLoRA [] and gradient checkpointing with DeepSpeed ZeRO-2.
For the prompts in training, we use <text> \n Summary above sentence in one word:, where <text> is the placeholder for the input sentence, and use the last token embeddings to represent the embeddings for contrastive learning.
We also report the performance with other MLLMs in Appendix C.

### 4.1 Text-Image Retrieval

We first benchmark E5-V on text-image retrieval with Flickr30K [] and COCO [] to evaluate zero-shot image retrieval and zero-shot text retrieval performance. For the baselines, we select the following text-image retrieval models: CLIP with ViT-B and ViT-L[], BLIP with ViT-L[], and the large CLIP model EVA-02-CLIP with 5B parameters []. All baselines are trained with contrastive learning on large-scale image-text pairs using separate visual and language encoders, while cannot represent interleaved visual and language inputs.
For the prompt used in text-image retrieval tasks, we use the following prompts to represent image and text inputs, respectively:

Text prompt:
<text>
Summary above sentence in on word:
Image prompt:
<image>
Summary above image in one word:

We report Recall@K (R@K) for K=1, 5, 10 with image retrieval and text retrieval in Table 1. Compared to strong baselines, E5-V, as a universal multimodal embeddings model, achieves competitive performance on both the Flickr30K and COCO datasets.

Compared to EVA-02-CLIP, which uses a 4.4B visual encoder with contrastive learning on large-scale image-text pairs [], E5-V
shows better ability on zero-shot image retrieval, while it is only trained on text pairs with contrastive learning.
It is worth noting that E5-V uses the same visual encoder as CLIP ViT-L and keeps it frozen during training.
Although E5-V shares the same visual encoder with CLIP, referring to the same way to encode visual inputs, E5-V demonstrates significantly better performance than CLIP on both the Flickr30K and COCO datasets for image retrieval and text retrieval tasks. Specifically, in image retrieval tasks, E5-V outperforms CLIP ViT-L by 12.2% on Flickr30K and 15.0% on COCO with Recall@1.

E5-V shows a strong ability to transfer single modality representation capabilities to multimodal embeddings by following task-specific prompts that were not included in the training data. It also seamlessly integrates visual and language information into the same embedding space with prompts.
For unseen prompt in training, E5-V can successfully follow it like “Summary the above image in one word:” to represent the image according to its semantics.

Table: Table 1: Zero-shot text-image retrieval performance on Flickr30K and COCO.

### 4.2 Composed Image Retrieval

To understand the effectiveness of E5-V in representing interleaved visual and language inputs, we evaluate it on composed image retrieval tasks with two popular datasets: FashionIQ [] and CIRR [].
This task focuses on retrieving images based on interleaved inputs, which requires the model to retrieve target images based on modified reference images, where the modification is described in the text.
For FashionIQ, it contains three subtypes of fashion products: Dress, Shirt and Toptee. Given a picture of a fashion product and a modification corresponding to the style, the model needs to retrieve the target image that matches the modification.
For CIRR, it extend FashionIQ on real-life images, which has more diverse images and modifications.

We compare E5-V with several zero-shot image-composed baselines: Pic2Word [], Context-I2W [], LinCIR [], the LLM-based method CIReVL [], and the current state-of-the-art method iSEARLE-XL [].
For a fair comparison, we report the results of all baseline models using the large visual encoder CLIP ViT-L, as in E5-V. Note that the E5-V also freezes visual encoder same as other baselines.
These baselines are designed exclusively for zero-shot composed image retrieval tasks and can not apply to other tasks.
Most of the baselines are not end-to-end embedding interleaved inputs, which introduce complex pipelines like textual inversion.
For example, CIReVL requires captioning an image first, generating the target image caption based on LLMs, and then retrieving the target image based on the caption.
However, E5-V can directly represent the interleaved visual and language inputs with prompts without any textual inversion.

To represent the interleaved inputs for E5-V, we use the following prompts for FashionIQ and CIRR.
For FashionIQ, which requires the model to mainly represent the style of the fashion product, we can directly let E5-V represent the style of the corresponding fashion products. Since the evaluation of FashionIQ is split into three subtypes, including Dress, Shirt, and Toptee, we can also provide the subtype information in the prompts.
For CIRR, we can directly let E5-V modify the image based on the modification described in the text and then represent the modified image in one word.
Although these prompts are unseen during training and have a complex format, E5-V can still correctly represent the interleaved inputs, even in specific domains like fashion products.

Composed image prompt for FashionIQ:
<image> change the style of this shirt/dress/toptee to <text>
Desribe this modified shirt/dress/toptee in one word based on its style:
Image prompt for FashionIQ:
<image>
Describe this shirt/dress/toptee in one word based on its style:
Composed image prompt for CIRR:
<image> modify this image with <text>
Desribe modified image in one word:
Image prompt for CIRR:
<image>
Describe this image in one word:

**Table 2: Zero-shot composed image retrieval performance on CIRR.**
|  | Recall@K |  |  |  |
| --- | --- | --- | --- | --- |
| Method | K=1 | K=5 | K=10 | K=50 |
| Pic2Word | 23.90 | 51.70 | 65.30 | 87.80 |
| Context-I2W | 25.60 | 55.10 | 68.50 | 89.80 |
| LinCIR | 25.04 | 53.25 | 66.68 | – |
| CIReVL | 24.55 | 52.31 | 63.92 | 86.34 |
| iSEARLE-XL | 25.40 | 54.05 | 67.47 | 88.92 |
| E5-V | 33.90 | 64.12 | 75.88 | 93.54 |

We report the composed image retrieval performance of CIRR and FashionIQ on Table 2 and 3.
All methods use CLIP ViT-L as the visual encoder. The results of other baselines are directly from their original papers.
Following previous works, we report Recall@K for K=1, 5, 10, and 50 on CIRR test set with their test evaluation server, and report Recall@K for K=10, 50 on three subsets of FashionIQ.
For the settings of E5-V,
we use original E5-V without additional fine-tuning on specific datasets and tricks like textual inversion.
E5-V directly represents the interleaved inputs and image inputs with above prompts and uses the last token embeddings to represent the multimodal embeddings.

Compared to zero-shot composed image retrieval baselines,
E5-V achieves significant improvements on both the CIRR and FashionIQ datasets without using techniques like textual inversion or annotation.
Specifically, E5-V outperforms the current state-of-the-art method iSEARLE-XL by 8.50% on Recall@1 and 10.07% on Recall@5 on CIRR. For FashionIQ, E5-V outperforms by 2.56% on Recall@10 and 4.24% on Recall@50 compared to iSEARLE-XL, which demonstrates the great ability of E5-V understanding the interleaved visual and language inputs and representing them correctly.

Table: Table 3: Zero-shot composed image retrieval performance on FashionIQ.

### 4.3 Image-Image Retrieval

By unifying multimodal representations into the same embedding space with prompts, E5-V demonstrates a strong ability to understand text through visual input and represent it accurately. To validate this, we designed an image-image retrieval task based on Flickr30K and COCO, referred to as I2I-Flickr30K and I2I-COCO. We rendered all textual captions in the datasets as images and used the embeddings of these images as the caption embeddings. The detailed implementation of text rendering can be found in Appendix A. For the prompts of E5-V, we simply used the image prompt in text-image retrieval tasks to represent images.

We report the results of CLIP, BLIP, EVA-02-CLIP, and E5-V in Table 4. Compared to text-image retrieval tasks, we notice that the performance of baselines drops significantly on image-image retrieval tasks, which indicates the difficulty of understanding text through visual input and representing it accurately. Due to separate visual and language encoders, these models struggle to understand the textual information via images by using their visual encoders. However,
E5-V correctly represents text through visual input and shows outstanding results on these two datasets.

Table: Table 4: Zero-shot image-image retrieval performance on I2I-Flickr30K and I2I-COCO.

### 4.4 Sentence Embeddings

Since E5-V is trained on text pairs, it also shows strong performance in representing textual inputs. We evaluate E5-V on the sentence embedding tasks using 7 STS tasks. Compared to other sentence embedding methods, including SimCSE-RoBERTa [], PromptRoBERTa [], and LLM-based methods such as SGPT [], ST5-Enc [], and PromptEOL [], E5-V, as a universal multimodal model, achieves the best performance on the STS tasks in Table 5, demonstrating its strong ability to represent textual inputs according to their semantics.

**Table 5: Sentence embeddings performance on STS tasks.**
| Method | STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SimCSE-RoBERTa | 76.53 | 85.21 | 80.95 | 86.03 | 82.57 | 85.83 | 80.50 | 82.52 |
| PromptRoBERTa | 76.75 | 85.93 | 82.28 | 86.69 | 82.80 | 86.14 | 80.04 | 82.95 |
| SGPT | 74.28 | 85.35 | 79.21 | 85.52 | 82.54 | 85.50 | 79.53 | 81.70 |
| ST5-Enc | 80.10 | 88.75 | 84.70 | 88.86 | 85.17 | 86.77 | 80.39 | 84.96 |
| PromptEOL | 79.16 | 90.22 | 85.40 | 88.99 | 86.25 | 88.37 | 81.51 | 85.70 |
| E5-V | 80.03 | 89.94 | 85.67 | 89.09 | 85.89 | 87.88 | 83.51 | 86.00 |

## 5 Discussion

### 5.1 Effect of the Representation Method

To validate the effectiveness of our prompt representation method, we compare it with two other methods: 1) Last: using the last token embeddings of the input as the multimodal embeddings, and 2) Prompt: using the same prompt as our methods, but removing in one word: in prompt.
We report the performance of these methods with and without fine-tuning in Table 6.
For the fine-tuning, we fine-tune each method with corresponding prompts on sentence pairs with contrastive learning following the same training settings as E5-V.

Table: Table 6: Effect of the representation method on different tasks. We report Recall@50 for FashionIQ, Spearmans correlation for STS tasks and Recall@5 for other tasks. For CIRR, we report the results on the validation set.

Our method shows significant improvements on all tasks compared to the Last and Prompt. For the setting without fine-tuning, we observe that our method can directly leverage the MLLM to represent the multimodal embeddings. However, other methods cannot represent the multimodal inputs properly. We also find that these methods have a large modality gap between image and text embeddings, as shown in Appendix B.
For the setting with fine-tuning,
we also observe the performance gap between our method and other methods. Although Prompt uses same template with our method and just removes in one word: in it, it still shows significant performance drop compared to our method especially on tasks with visual inputs.
One possible reason may be the modality gap limit it to transfer the single modality representation capabilities learned on text inputs to multimodal embeddings.

### 5.2 Effect of Single Modality Training

We also compare single modality training with multimodal training. For multimodal training, we train the MLLM on 558K text-image pairs from CC3M using the same training settings and prompts as single modality training. We report the performance of single modality training and multimodal training on different tasks in Table 7.
We find that MLLM achieves better multimodal embeddings with single modality training. Even on the image-text retrieval tasks, where multimodal training uses similar training data, single modality training still shows better performance.
For other tasks, we notice that multimodal training cannot represent the interleaved inputs in FashionIQ and CIRR, or text inputs in STS well, which leads to a performance drop compared to single modality training.
Moreover, single modality training is more efficient by removing the visual encoder and only uses 32 max tokens for text inputs, significantly reducing the training time compared to multimodal training. Single modality training only takes 1.5 hours on 32 V100 GPUs, while multimodal training takes 34.9 hours under same environment.

Table: Table 7: Effect of single modality training on different tasks. We measure the training time on 32 V100 GPUs.

### 5.3 Zero-shot Instruction Following Ability on Multimodal Embeddings

We find an interesting ability of E5-V to represent inputs based on fully zero-shot instructions. Although E5-V is trained on text inputs with the static prompt, it can correctly represent visual and interleaved inputs based on unseen prompts. These prompts can be more detailed and specific based on the tasks.
For example, in FashionIQ, a specific domain dataset about fashion products, we can design specific prompts to let E5-V embed the image based on their styles. Moreover, the interaction between visual and language inputs in E5-V can also be more detailed, such as change the style of this shirt to. Compared to other methods, which simply fuse the visual and language inputs, E5-V provides a more nuanced and specific approach.

## 6 Conclusion

In this work, we propose E5-V, a MLLM based universal multimodal model that can represent interleaved visual and language inputs accurately. E5-V uses the prompt based representation method to unify multimodal representations into the same embedding space without additional fine-tuning or tricks.
With single modality training, E5-V achieves strong performance on various tasks, including text-image retrieval, composed image retrieval, image-image retrieval, and sentence embeddings. We also conduct extensive ablation studies to validate the effectiveness of our method.

## Appendix A Implementation of Text Rendering in Image-Image Retrieval

We introduce the implementation of text rendering in image-image retrieval tasks. We use the PIL library to render the text as an image by using the Arial font with 40 pixels font size and 800$\times$400 resolution.
To fit the long text, we also automatically break the text into multiple lines to fit the image size.
We provide a example of rendering text as image in Figure 4.

Figure: Figure 4: An example of rendering text as image with text “A man in a black shirt rides an elephant as a man walks near it down a street.”
Refer to caption: extracted/5737908/render_example.png

The python code is shown below:

## Appendix B Distruibution of Image and Text Embeddings with Different Representation Methods

We visualize the distribution of multimodal embeddings from MLLM with three different representation methods: Last, Prompt, and Our. The embeddings are directly from LLaVA-NeXT-8B without fine-tuning on any specific dataset.
Our method removes modality gap between image and text embeddings, which is shown in Figure 5.
The prompt for each method is following:

Text prompt in Last:
<text>
Image prompt in Last:
<image>
Text prompt in Prompt:
<text>
Summary above sentence:
Image prompt in Prompt:
<image>
Summary above image:
Text prompt in Our:
<text>
Summary above sentence in on word:
Image prompt in Our:
<image>
Summary above image in one word:

Figure: (a) Last
Refer to caption: extracted/5737908/dis_2.png

## Appendix C Multimodal Embeddings with Different MLLMs

We also evaluate the performance of E5-V with different MLLMs, including Phi3, LLaVA 1.6, and LLaVA-NeXT in Table 8.
Despite the different LLM in each MLLM, E5-V also show strong performance with same prompts.
Among these MLLMs, LLaVA-NeXT shows the best performance with fine-tuning benfit from the high capacity of LLM.

Table: Table 8: Performance of E5-V with different MLLMs on different tasks.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. It is intriguing to see adding one modality could improve the performance of multiple modalities. I am wondering about the scalability of this method. What will happen if there is more text data included in fine-tuning? (Furthermore, will adding multimodal pairs further improve the performance?)
2. Some claims need supporting citations. For example, line 42-43, "adapting CLIP to universal multimodal embeddings will have shortcomings such as poor language understanding, limited real-world knowledge, .."
3. Regarding the robustness of this method, Would the difference in prompt during inference give substantial performance variations?
soundness: 2
strengths: 1. A new multimodal representation framework leveraging the multimodal understanding capability of MLLMs, with a clear motivation and a simple yet effective method.
2. Strong performance on text-image retrieval and composed image retrieval tasks with text-only training data, eliminating the need for costly multimodal training data and computing.
summary: The paper introduces E5-V, a prompt-based representation designed to achieve universal multimodal embeddings based on Multimodal Large Language Models (MLLMs). E5-V applies a cost-effective single-modality training approach using text-only data, and achieves better multimodal embeddings than image-text pairs across four tasks.
weaknesses: There are no major weaknesses. Just some minor weaknesses and a few questions in need of clarification.
Minor weaknesses:
1. The authors provide a visualization to illustrate the "modality gap". It would be better to provide a quantitative metric ( such as
https: //openreview. net/pdf? id=S7Evzt9uit3 ) to evaluate the gap between image embeddings and text embeddings from MLLMs w/o prompt-based representation.
2. Could you clarify whether the visualization of Figure 3(b) is before or after single modality training? Could you provide a visualization comparison of before and after single modality training?
3. One-word embedding might be not enough. I am wondering whether expanding the length of output words will lead to performance gains. For example, What if in Figure 2, there are multiple objects in the image and the instruction is to modify one object of them, will E5-V get it right?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - I would appreciate the authors' response about my comment in "Weaknesses".
- I recommend placing some samples of query and retrieved image pairs from the composed image retrieval and image-image retrieval tasks in the main body and/or Appendix. Such visualization will provide readers with a clear vision of the differences between the conventional and proposed models.
soundness: 2
strengths: 1. The paper is well written. I could understand their motivation, the proposed method, and the experimental results.
2. The proposed method is simple but effective. If we have a good pretrained MLLM, we can easily fine-tune it for multimodal representations. This method can be applied to other combinations of two or more modalities as well.
3. A trained embedding space is free from the modality gap issue, one of the important issues of contrastive learning-based multimodal models.
4. The authors conducted several experiments to show the proposed training method works well.
summary: This paper presents a method of training a multimodal model that can output multimodal representations. In contrast to conventional multimodal models such as CLIP, which consists of separate encoders for different modalities, the proposed E5-V is based on a single pretrained MLLM (multimodal large language model) such as LLaVA-Next. E5-V is trained on only text data (natural language inference dataset) and requires a lower computational cost than the previous models by leveraging the knowledge of the MLLM. The authors conducted experiments on text-image retrieval, composed image retrieval, image-image retrieval, and sentence embedding tasks. The experiments demonstrated that the proposed training method works well.
weaknesses: - I acknowledge that the proposed training method is efficient if we have a good enough pretrained MLLM and that a trained embedding space is free from the modality gap. These properties are excellent. However, I doubt if the comparative experiments are fair in terms of model size. The model size of E5-V (LLaVA-Next-8B) is 8B, which is larger than CLIP, BLIP, and even EVA-CLIP used for comparison. As shown in the [EVA-CLIP paper](https://arxiv.org/abs/2303.15389), the model size affects its performance. If possible, a smaller size of E5-V should be evaluated/compared as well. I know that LLMs show emergent abilities and smaller models might have no/weak capabilities of encoding images/text very well. Even if so, showing comparisons will be informative to readers.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Since there is no training for modality encoder and projector, how can the modality encoder and projector generalize to the model in the 
inference time?

E5-V is trained on text-only data, yet it shows minimal improvement in text-only tasks (e.g., sentence embeddings: 86.00 vs. previous SOTA 85.70). However, it achieves significant gains in image-related tasks, such as image retrieval. Why does this difference in improvement occur?
soundness: 3
strengths: Using MLLM so that E5-V could deal with interleaved input and using small batch size.

Only trained on text-only data leads to better performance compared to the model trained on multimodal data.

Use “summary XXX in on word” to bridge the modality gap between text and vision effectively
summary: E5-V is a framework that adapts multimodal large language models (MLLMs) for universal multimodal embeddings. By training solely on text pairs, E5-V reduces training costs by 95% and demonstrates high effectiveness across tasks, bridging modality gaps efficiently.
weaknesses: The "summary XXX in one word" prompt enables the model to handle simple image or text inputs. If the input complexity increases, a word cannot illustrate the input text /image well.

For new multi-model task, the new prompt needs to be designed to ensure effective performance.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: In Tab1, why E5-V achieved worse text-retrieval results, consistently,
than EVA-02-CLIP-5B? E5-V was finetuned on NLI datasets, which are
supposed to improve its ability of embedding texts. But this seems not
the case. What are the hypotheses of this observation?
soundness: 3
strengths: - E5-V is simple, intuitive, and can be potentially adopted in various applications.
 
- Training E5-V seems lightweight -- it just requires training on text-only data.
  
- E5-V highlights the potential of extracting representations from LLM-based models. Expanding the scope of the current research and
  real-world applications where CLIP-like models are still extensively used.

- Representing modality-interleaved sequences is beneficial and can unlock many new applications.
summary: This paper focuses on obtaining representations of texts and images
from multimodal LLMs (MLLMs).  Authors approach this goal by proposing
E5-V, which leverages prompts like "summarize above image in one word"
along with an image
to input to the MLLMs to close the modality gap.  E5-V is a Llava-based
MLLM finetuned on text-only data (NLI) to extract multimodal
representations. It also supports for embedding interleaving text and
image sequences, which potentially supports for new applications.

Authors show that E5-V outperforms CLIP-based methods in text-to-image
retrieval benchmarks. Usage of E5-V on composed image retrieval and
image-to-image retrieval is also presented.
weaknesses: - Compared with CLIP-like models, inference with LLM-based E5-V seems
  still heavy (e.g., 8B of E5-V vs 5B of EVA-CLIP).

- More in-depth analyses are desirable/needed to show the inner
  workings of E5-V, e.g., how did it close the modality gap, by just
  using a prompt? Also, some qualitative examples are necessary. E.g.,
  is E5-V better at encoding higher-level semantic information than
  some low-level features like textures?

- Image-to-text retrieval performance (Tab1) on standard benchmarks
  Flickr30K and COCO are not good and consistently worse than
  baselines. This is counter-intuitive, given that E5-V was finetuned
  on the  NLI datasets, which is text data.

So overall I feel the paper is a bit pre-mature, and more rounds would
be beneficial.

Meta-review: This paper presents E5-V, which turns a multimodal large language models to output multimodal representations. The idea of employing single-modality training approach using text-only data achieves better multimodal embeddings is novel.

The reviewers agree the framework is interesting and the idea of training on text-only data leads to a simply, intuitive, and lightweight tuning approach. The major concern from the reviewers is its lack of sufficient analysis and comparison to CLIP like models. Quota as “The idea of the paper is inspirational but the analysis is a bit less comprehensive.” 

After a careful discussion, the reviewers agree this paper is promising on its direction, but still pre-mature.

Decision: Reject