mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data
Authors: Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou
Accepted at: acl25

mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data

Haonan Chen1*, Liang Wang2, Nan Yang2, Yutao Zhu1
Ziliang Zhao1, Furu Wei2, Zhicheng Dou1
1Gaoling School of Artificial Intelligence, Renmin University of China
2Microsoft Corporation
{hnchen,dou}@ruc.edu.cn
{wangliang,nanya,fuwei}@microsoft.com
https://github.com/haon-chen/mmE5

Abstract

Multimodal embedding models have gained significant attention for their ability to map data from different modalities, such as text and images, into a unified representation space. However, the limited labeled multimodal data often hinders embedding performance. Recent approaches have leveraged data synthesis to address this problem, yet the quality of synthetic data remains a critical bottleneck. In this work, we identify three criteria for high-quality synthetic multimodal data. First, broad scope ensures that the generated data covers diverse tasks and modalities, making it applicable to various downstream scenarios. Second, robust cross-modal alignment makes different modalities semantically consistent. Third, high fidelity ensures that the synthetic data maintains realistic details to enhance its reliability. Guided by these principles, we synthesize datasets that: (1) cover a wide range of tasks, modality combinations, and languages, (2) are generated via a deep thinking process within a single pass of a multimodal large language model, and (3) incorporate real-world images with accurate and relevant texts, ensuring fidelity through self-evaluation and refinement. Leveraging these high-quality synthetic and labeled datasets, we train a multimodal multilingual E5 model mmE5. Extensive experiments demonstrate that mmE5 achieves state-of-the-art performance on the MMEB Benchmark and superior multilingual performance on the XTD benchmark. Our codes, datasets, and models are released in https://github.com/haon-chen/mmE5.

1 Introduction

Multimodal embedding models encode multimedia inputs, such as images and text, into latent vector representations. They have demonstrated effectiveness across diverse downstream tasks, including classification (Deng et al., 2009), visual question answering (VQA) (Singh et al., 2019), and cross-modal retrieval (Hu et al., 2023). Prior studies have focused on training multimodal embedding models using simple text-image pre-trained models such as CLIP (Radford et al., 2021). More recently, researchers have turned to multimodal large language models (MLLMs), including LLaVA (Liu et al., 2023a) and Phi (Abdin et al., 2024), to develop universal embedding models.

These vision-language models (VLMs) mostly rely on high-quality human-labeled datasets to achieve robust embedding capabilities. Such datasets suffer from data scarcity because they require high costs of multimodal annotations. To address this, researchers have leveraged the advanced language modeling capabilities of large language models (LLMs) and MLLMs to synthesize datasets for fine-tuning multimodal embedding models (Zhang et al., 2024a; Zhou et al., 2024a; Zhang et al., 2024b). However, existing works lack a comprehensive exploration into the quality of synthetic embedding data. Typically, most data generated by them are limited to specific modality types of English retrieval tasks, harming the generalization capabilities of the embedding models.

After analyzing common application scenarios of multimodal embedding models, we identify

![An illustration of our data synthesis framework. “X→Y” denotes a modality combination, where “X” represents the query side and “Y” denotes the target side. “T” denotes text and “I” denotes image.](page_1016_370_388_246.png)

Figure 1: An illustration of our data synthesis framework. “X→Y” denotes a modality combination, where “X” represents the query side and “Y” denotes the target side. “T” denotes text and “I” denotes image.
three key criteria and introduce a data synthesis framework guided by these principles: (1) Broad scope. Multimodal embedding models are commonly employed in tasks such as classification, visual question answering (VQA), and retrieval, which require understanding various input combinations of text and images. Additionally, multilingual contexts are increasingly popular in daily scenarios. As shown in Figure 1, our framework synthesizes datasets covering three tasks, seven modality combinations, and 93 languages, ensuring that models trained on it generalize effectively across diverse scenarios. (2) Robust cross-modal alignment. In multimodal tasks, models must understand and align information across different modalities to generate meaningful representations. Without accurate cross-modal alignment, embeddings may fail to capture the underlying relationships, leading to poor performance in downstream tasks. To synthesize data of robust cross-modal alignment, our framework incorporates a deep thinking process. Specifically, for each sampled image, we first employ an MLLM to interpret it from four perspectives before generating data: general information, object-level description, contextual background information, and task-specific brainstorming, i.e., how the image relates to the given task. Additionally, the entire data synthesis process is executed within a single pass of an MLLM. By this, the MLLM can “see” the images at the whole time, avoiding potential information loss that might occur due to multiple I/O steps in previous works (Zhou et al., 2024a; Zhang et al., 2024b). (3) High fidelity. The individual quality of each modality (e.g., real images, high-quality instructions, queries, and hard negatives) determines the overall usefulness of the dataset. To enhance fidelity, our framework uses real images sampled from an open-source corpus (LAION-400m (Schuhmann et al., 2021)) as the input images. We also apply a series of quality control measures, such as self-evaluation and refinement, ensuring that the synthetic components accurately reflect real-world distributions and maintain strong cross-modal alignment.

With the synthesized data ready, we train a multimodal multilingual E5 model (mmE5). It achieves state-of-the-art performance on the 36 datasets of MMEB (Jiang et al., 2024b), using 45 times less training data than the previous SOTA model MMRet (Zhou et al., 2024a) (560K compared to 26M) in a zero-shot setting. After incorporating labeled data, mmE5 still demonstrates the best performance. Besides, mmE5 achieves the best results on the multilingual benchmark XTD (Aggarwal and Kale, 2020), demonstrating its superior multilingual capabilities.

In summary, our contributions are as follows:

• Based on our analysis of common scenarios for multimodal embedding models, we identify three key criteria of high-quality synthetic data: broad scope, robust cross-modal alignment, and high fidelity.

• We introduce a data synthesis framework guided by the proposed principles. This framework leverages an MLLM to produce high-quality synthetic datasets that cover a wide range of tasks, modality combinations, and languages. It ensures robust cross-modal alignment through a comprehensive multi-aspect interpretation process and maintains high fidelity by employing self-evaluation and refinement mechanisms.

• Compared to the previous leading model, mmE5 achieves SOTA performance on the MMEB benchmark while using 45x less synthetic data in both zero-shot and supervised settings. mmE5 also demonstrates superior multilingual capabilities on the XTD benchmark.

2 Related Work

Multimodal Embedding Previous studies, such as CLIP (Radford et al., 2021), Align (Jia et al., 2021), BLIP (Li et al., 2022), and CoCa (Yu et al., 2022), have employed large-scale weakly supervised data to learn separate multimodal representations through pre-training. Some works attempt to obtain universal embeddings for texts and images utilizing existing CLIP-like models (Wei et al., 2024; Liu et al., 2023b; Zhou et al., 2024b,c). For instance, UniIR (Wei et al., 2024) integrates separate embeddings from different modalities into unified features. Recent approaches finetune MLLMs to leverage their multimodal reasoning capabilities for obtaining universal representations (Jiang et al., 2024a,b; Zhang et al., 2024b; Zhou et al., 2024a; Lin et al., 2024). For example, VLM2Vec (Jiang et al., 2024b) utilizes instruction tuning to transform MLLMs into embedding models.

Synthetic Data The generation of synthetic data has been extensively explored for text embedding
<table>
  <tr>
    <th>Method</th>
    <th># Languages</th>
    <th>Task</th>
    <th>Modality Combinations</th>
    <th>w/ MLLM</th>
    <th>One Pass</th>
    <th>Self-evaluation</th>
  </tr>
  <tr>
    <td>MagicLens</td>
    <td>1 (English)</td>
    <td>Retrieval</td>
    <td>IT→I</td>
    <td>×</td>
    <td>√</td>
    <td>×</td>
  </tr>
  <tr>
    <td>MegaPairs</td>
    <td>1 (English)</td>
    <td>Retrieval</td>
    <td>IT→I</td>
    <td>√</td>
    <td>×</td>
    <td>×</td>
  </tr>
  <tr>
    <td>GME</td>
    <td>1 (English)</td>
    <td>Retrieval</td>
    <td>T→IT, IT→IT</td>
    <td>×</td>
    <td>×</td>
    <td>×</td>
  </tr>
  <tr>
    <td>mmE5 (Ours)</td>
    <td>93 (English, Spanish, etc.)</td>
    <td>Classification, VQA, Retrieval</td>
    <td>IT→I, T→IT, IT→IT, I→I, I→T, IT→T, T→I</td>
    <td>√</td>
    <td>√</td>
    <td>√</td>
  </tr>
</table>

Table 1: Comparison of the synthetic datasets in our work with those from previous methods. Our synthetic datasets encompass 93 languages, two additional tasks, and a wider range of modality combinations. “IT→T” denotes a modality combination, where “IT” denotes images and texts on the query side and “T” denotes texts on the target side. The entire data synthesis process is executed within a single pass of an MLLM, thereby avoiding potential information loss and ensuring robust cross-modal alignment. We also employ real images and self-evaluation to maintain fidelity.

tasks (Wang et al., 2024a; Chen et al., 2024; Li et al., 2024b). With the recent emergence of MLLMs like Phi-3.5-V (Abdin et al., 2024) and LLaVA (Liu et al., 2023a), along with diffusion models such as Stable Diffusion (Rombach et al., 2022), researchers have been focusing on synthesizing data to address the scarcity of multimodal instruction-tuning datasets. For example, MagicLens (Zhang et al., 2024a) utilizes co-existing images from the same webpage and an LLM to create multimodal data triplets (query image, instruction, relevant image), i.e., IT→I paradigm. MegaPairs (Zhou et al., 2024a) aims to synthesize more diverse data triplets by retrieving relevant images from different perspectives. GME (Zhang et al., 2024b) employs an LLM and a diffusion model to generate a fused modality dataset that includes both T→IT and IT→IT types. Table 1 presents a comparison of the synthesized data in this study with that of previous works.

3 Methodology: mmE5

In this section, we present our method, which synthesizes high-quality multimodal data for the further finetuning of our embedding model mmE5. As shown in Figure 2, our method consists of five stages: (1) Initially, for each data sample to be synthesized, we configure the specifics of the task, modality combination, language, and input images. (2) We employ an MLLM to generate multi-grained descriptions for the input images, ensuring that the synthesized texts are well-aligned with the images. (3) Utilizing this MLLM, we synthesize text data based on both the images and their descriptions. (4) The MLLM then evaluates its synthesized data from multiple perspectives, offering revised data to enhance cross-modal alignment and fidelity. (5) Finally, the synthesized texts and images are used to finetune an MLLM specifically for embedding tasks. To minimize potential information loss, stages (2), (3), and (4) are executed within a single pass of the MLLM.

3.1 Preliminaries

An MLLM can accept text, image, or text-image pairs as input, allowing both the query side q and the document side d to be multimodal. Inspired by existing works on synthetic text embedding data (Wang et al., 2024a; Chen et al., 2024), each data sample we generate is a quadruple of (task instruction, query, positive document, hard negative document), denoted as \((t, q, d^+, d^-)\). For each data piece, we first sample images from the large-scale open-source image corpus LAION-400M (Schuhmann et al., 2021) as the query image, positive image, and hard negative image \((q_i, d_i^+, d_i^-)\). Then, with these three images as input, an MLLM \( \pi_\theta \) can synthesize a multimodal embedding data sample \( y \sim \pi_\theta(y \mid q_i, d_i^+, d_i^-) \), where \( y = (t, q_i, d_i^+, d_i^-) \). As a result, the synthetic data can have a maximum of seven elements: \(\{t, (q_i, q_i), (d_i^+, d_i^+), (d_i^-, d_i^-)\}\). More data examples can be found in Appendix D.

3.2 Data Synthesis Framework

Guided by the principles of high-quality synthetic multimodal data, i.e., broad scope, robust cross-modal alignment, and high fidelity, we introduce a data synthesis framework. This framework is designed to synthesize high-quality data that transforms an MLLM for downstream embedding tasks.
Figure 2: An illustration of our method. We take the generation of an IT→IT retrieval data sample as an example.

3.2.1 Data Configuration
To prepare for the data synthesis process, we configure the input data from three aspects:

Task and Modality Combination We aim to synthesize data with a broad scope by generating beyond simple retrieval data of IT→IT and T→IT types. Our data cover three key multimodal embedding tasks identified by previous work (Jiang et al., 2024b): classification, VQA, and retrieval. After selecting a task for synthesis, we will sample a modality combination with respect to the specific task, such as choosing from seven possible combinations for the retrieval task type. Note that we only synthesize data of modality types that are included in the MMEB benchmark (Jiang et al., 2024b), which can cover most scenarios.

Image Despite the powerful multimodal capabilities of modern MLLMs (e.g., GPT-4o, Llama-3.2 (Meta, 2024), and Llava-1.6), most cannot generate images, and those that can often produce low-fidelity images (Zhou et al., 2024b). Following previous works (Zhang et al., 2024a; Zhou et al., 2024a), we sample real images from the LAION-400M corpus (Schuhmann et al., 2021). First, we will sample a query image from the corpus (\( q_i \in \mathcal{I} \)). Then, for the modality types involving images on the document side (e.g., IT→IT), we use a small embedding model, jina-clip-v2 (Koukounas et al., 2024), to retrieve a similar positive image \( d_i^+ \) and a hard negative image \( d_i^- \) efficiently.

Language Most existing models only focus on high-source languages like English, harming the multilingual ability of embedding models. To synthesize multilingual data, we sample languages from the language list of XLM-R (Conneau et al., 2020) during configuration. In order to facilitate the common usage scenarios, we give high-source languages higher weights. Note that the generated task instruction will always be in English for effective instruction tuning.

3.2.2 One-pass Generation with MLLM
With the data configuration ready, we introduce a deep thinking process that involves interpreting input images, generating data, and performing self-evaluation. To ensure that the MLLM always takes the image context into account, we execute this entire process in a single pass.

Multi-aspect Visual Interpretation To obtain a comprehensive understanding of the images, the MLLM \( \pi_\theta \) first analyzes them from multiple perspectives: (1) the general information, (2) detailed description of the objects present, (3) contextual background information, and (4) potential connections between the image and the text that may be synthesized. The deep understanding of the images enables \( \pi_\theta \) to produce texts that are closely aligned with the visual content, thereby enhancing the cross-modal alignment.

Synthesizing Data Using the images and their descriptions as input, we prompt \( \pi_\theta \) to synthesize texts \( (t, q_i, d_i^+, d_i^-) \). Specifically, the text instruction \( t \) is expected to connect \( q_i \) with \( d_i^+ \). The query and document texts should be relevant to their respective images. Note that the input and output formats for the synthetic data may vary depending on the combination of modalities. For example, for I→IT and T→IT types, there can be no query text and image, respectively.

Self-evaluation To further enhance the quality of the synthetic data, \( \pi_\theta \) evaluates the data it synthesizes from: (1) the relevance of the texts to their corresponding images, (2) the plausibility of hard negatives, (3) the clarity of \( t \), and (4) the diversity

1Because of limited space, full prompts are omitted in this section. The complete prompts can be found in Appendix C.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">Per Meta-Task Score</th>
    <th colspan="3">Average Score</th>
  </tr>
  <tr>
    <th>Class.</th>
    <th>VQA</th>
    <th>Retr.</th>
    <th>Ground.</th>
    <th>IND</th>
    <th>OOD</th>
    <th>Overall</th>
  </tr>
  <tr>
    <th colspan="8"><i>Zero-shot Setting Models</i></th>
  </tr>
  <tr>
    <td>CLIP (Radford et al., 2021)</td>
    <td>42.8</td>
    <td>9.1</td>
    <td>53.0</td>
    <td>51.8</td>
    <td>-</td>
    <td>-</td>
    <td>37.8</td>
  </tr>
  <tr>
    <td>BLIP2 (Li et al., 2023)</td>
    <td>27.0</td>
    <td>4.2</td>
    <td>33.9</td>
    <td>47.0</td>
    <td>-</td>
    <td>-</td>
    <td>25.2</td>
  </tr>
  <tr>
    <td>SigLIP (Zhai et al., 2023)</td>
    <td>40.3</td>
    <td>8.4</td>
    <td>31.6</td>
    <td>59.5</td>
    <td>-</td>
    <td>-</td>
    <td>34.8</td>
  </tr>
  <tr>
    <td>OpenCLIP (Cherti et al., 2023)</td>
    <td>47.8</td>
    <td>10.9</td>
    <td>52.3</td>
    <td>53.3</td>
    <td>-</td>
    <td>-</td>
    <td>39.7</td>
  </tr>
  <tr>
    <td>ES-V (Jiang et al., 2024a)</td>
    <td>21.8</td>
    <td>4.9</td>
    <td>11.5</td>
    <td>19.0</td>
    <td>-</td>
    <td>-</td>
    <td>13.3</td>
  </tr>
  <tr>
    <td>MagicLens (Zhang et al., 2024a)</td>
    <td>38.8</td>
    <td>8.3</td>
    <td>35.4</td>
    <td>26.0</td>
    <td>-</td>
    <td>-</td>
    <td>27.8</td>
  </tr>
  <tr>
    <td>MMRet (w/ 26M synthetic data)</td>
    <td>47.2</td>
    <td>18.4</td>
    <td><b>56.5</b></td>
    <td>62.2</td>
    <td>-</td>
    <td>-</td>
    <td>44.0</td>
  </tr>
  <tr>
    <td><b>mmE5 (w/ 560K synthetic data)</b></td>
    <td><b>60.6</b></td>
    <td><b>55.7</b></td>
    <td><b>54.7</b></td>
    <td><b>72.4</b></td>
    <td>-</td>
    <td>-</td>
    <td><b>58.6</b></td>
  </tr>
  <tr>
    <th colspan="8"><i>Partially Supervised Finetuning Models†</i></th>
  </tr>
  <tr>
    <td>UniIR (Wei et al., 2024)</td>
    <td>42.1</td>
    <td>15.0</td>
    <td>60.1</td>
    <td>62.2</td>
    <td>-</td>
    <td>-</td>
    <td>42.8</td>
  </tr>
  <tr>
    <td>MM-EMBED (Lin et al., 2024)</td>
    <td>48.1</td>
    <td>32.2</td>
    <td>63.8</td>
    <td>57.8</td>
    <td>-</td>
    <td>-</td>
    <td>50.0</td>
  </tr>
  <tr>
    <td>GME (Zhang et al., 2024b)</td>
    <td>56.9</td>
    <td>41.2</td>
    <td>67.8</td>
    <td>53.4</td>
    <td>-</td>
    <td>-</td>
    <td>55.8</td>
  </tr>
  <tr>
    <th colspan="8"><i>Supervised Finetuning Models</i></th>
  </tr>
  <tr>
    <td>CLIP (Radford et al., 2021)</td>
    <td>55.2</td>
    <td>19.7</td>
    <td>53.2</td>
    <td>62.2</td>
    <td>47.6</td>
    <td>42.8</td>
    <td>45.4</td>
  </tr>
  <tr>
    <td>OpenCLIP (Cherti et al., 2023)</td>
    <td>56.0</td>
    <td>21.9</td>
    <td>55.4</td>
    <td>64.1</td>
    <td>50.5</td>
    <td>43.1</td>
    <td>47.2</td>
  </tr>
  <tr>
    <td>VLM2Vec (Jiang et al., 2024b)</td>
    <td>61.2</td>
    <td>49.9</td>
    <td>67.4</td>
    <td>86.1</td>
    <td>67.5</td>
    <td>57.1</td>
    <td>62.9</td>
  </tr>
  <tr>
    <td>MMRet (Zhou et al., 2024a)</td>
    <td>56.0</td>
    <td>57.4</td>
    <td>69.9</td>
    <td>83.6</td>
    <td>68.0</td>
    <td>59.1</td>
    <td>64.1</td>
  </tr>
  <tr>
    <td><b>mmE5 (w/ synthetic data + labeled data)</b></td>
    <td><b>67.6</b></td>
    <td><b>62.7</b></td>
    <td><b>71.0</b></td>
    <td><b>89.7</b></td>
    <td><b>72.4</b></td>
    <td><b>66.6</b></td>
    <td><b>69.8</b></td>
  </tr>
</table>

Table 2: Results on MMEB benchmark, consisting of 36 tasks across four types: classification (Class.), VQA, retrieval (Retr.), and visual grounding (Ground.). † UniIR, MM-EMBED, and GME are not strictly zero-shot models. UniIR and MM-EMBED are trained on the MBEIR dataset (Wei et al., 2024), which includes 10 retrieval datasets included in the MMEB. Similarly, GME is trained on the UMRB dataset (Zhang et al., 2024b), which shares 14 datasets with the MMEB. For VLM2Vec, we use the LLaVA-based version with high-resolution images reported in its original paper. The second-best performances are underlined, and the best performances are in bold.

(creativity) of the synthesized data. Following this evaluation, \( \pi_\theta \) provides suggestions for potential improvements. Finally, a revised version of each data sample is produced and utilized for the subsequent contrastive training phase.

3.3 Finetuning Embedding Model mmE5

Following previous works of instruction-tuned text embedding models (Xiao et al., 2024; Li et al., 2024a) and multimodal embedding models (Jiang et al., 2024b), we apply an instruction template on each query: [IMAGE] {t} \n {q_t} {q_i}, where “[IMAGE]” is the image token that varies from different MLLMs. We then append an “[EOS]” token to each query and document. The representation of each input in an MLLM is derived from the output of the “[EOS]” token from the final layer.

We utilize the InfoNCE loss (van den Oord et al., 2018) to perform the standard contrastive learning objective on our synthetic data \( \mathcal{D} \):

\[
\mathcal{L} = - \log \frac{\phi(\mathbf{q}, \mathbf{d}^+)}{\phi(\mathbf{q}, \mathbf{d}^+) + \sum_{d \in \mathcal{N}} \phi(\mathbf{q}, \mathbf{d}^-)},
\]

where \( \mathbf{q} \) is the encoded multimodal query, \( \mathbf{d} \) represents the encoded document, and \( \mathcal{N} \) denotes the set of negative documents. The function \( \phi(\cdot) = \exp(\cos(\cdot)/\tau) \), where \( \cos(\cdot) \) denotes cosine similarity, and \( \tau \) is a temperature hyperparameter.

4 Experiments

4.1 Experimental Setup

We synthesize a total of 560K multimodal embedding data samples. The MLLM utilized for data synthesis is <i>GPT-4o-2024-08-06</i>. The backbone model for mmE5 is Llama-3.2-11B-Vision<sup>2</sup>. For finetuning mmE5, we employed LoRA (Hu et al., 2022) with a rank of 8. We evaluate the general embedding performance in terms of Precision@1 on the MMEB benchmark (Jiang et al., 2024b). This benchmark comprises 36 multimodal embedding tasks across four categories: classification (10), VQA (10), retrieval (12), and visual grounding (4). Our synthetic dataset is distributed among classification, VQA, and retrieval tasks in a 1:1:2 ratio. We synthesize more retrieval data since this type contains more kinds of modality combinations. We do not synthesize visual grounding data since they are relatively simpler for MLLM based on the MMEB results. To evaluate multilingual multimodal capabilities, we conducted tests using the

2https://huggingface.co/meta-llama/Llama-3.2-11B-Vision
XTD benchmark (Aggarwal and Kale, 2020). Following MURAL (Jain et al., 2021), we conduct experiments on seven languages of XTD and report Recall@10 results. Additional details regarding the synthetic data, prompts, and implementation can be found in Appendix A, B, and C, respectively.

4.2 Results on MMEB

The overall results on the MMEB benchmark are presented in Table 2. mmE5 achieves the best performance on both zero-shot setting (with synthetic data only) and supervised setting (with IND training datasets of MMEB). This demonstrates the quality of our synthetic data and the effectiveness of our multimodal embedding model. Furthermore, we can make the following observations: (1) mmE5 generalizes well on all four kinds of tasks. This demonstrates the broad scope of our synthetic multimodal embedding data in terms of task types. (2) With only 560K synthetic data, mmE5 manages to perform better than MMRet which uses 26M data. This proves the quality of our synthetic data again. (3) Intriguingly, mmE5 underperforms MMRet on retrieval tasks in a zero-shot setting. This is because MMRet is trained on 26M pure retrieval data, which makes it perform well on retrieval tasks, but generalizes poorly on other task types.

4.3 Multilingual Performance on XTD

We synthesize a multilingual multimodal dataset that consists of 93 languages, in order to train our embedding model mmE5 to generalize across more languages. The language distribution of our dataset is presented in Figure 3. Notably, the dataset primarily consists of English data samples, facilitating common usage scenarios. For the 75 low-resource languages, we evenly synthesize data samples to obtain a balanced multilingual dataset that supports comprehensive cross-linguistic generalization.

To evaluate the multilingual capability of mmE5, we conduct experiments across seven languages on a text-to-image retrieval benchmark XTD. As presented in Table 3, mmE5 outperforms other models in terms of overall performance on all languages, demonstrating its superior multilingual multimodal embedding capability. The following observations can be made: (1) The multilingual performance of multimodal embedding models is largely dependent on their foundational models. For example, jina-clip-v2 and M-CLIP outperform VLM2Vec-LLaVA, despite VLM2Vec’s strong performance on MMEB. GME exhibits robust performance on XTD, which can be attributed to the powerful multilingual MLLM, Qwen2-VL (Wang et al., 2024b). (2) The performance of mmE5 declines when labeled data is omitted, indicating that general multimodal capabilities remain essential for multilingual retrieval tasks. (3) In a zero-shot setting, mmE5 trained on multilingual synthetic data (mmE5 w/ synthetic data only) outperforms mmE5 with the same amount of English synthetic data (mmE5 w/ english synthetic data). This suggests that the extensive language coverage provided by our synthetic data enhances the multilingual capabilities of embedding models.

<table>
  <tr>
    <th>Model</th>
    <th>it</th>
    <th>es</th>
    <th>ru</th>
    <th>zh</th>
    <th>pl</th>
    <th>tr</th>
    <th>ko</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>ALIGN (Jia et al., 2021)</td>
    <td>87.9</td>
    <td>88.8</td>
    <td>82.3</td>
    <td>86.5</td>
    <td>79.8</td>
    <td>73.5</td>
    <td>76.6</td>
    <td>82.2</td>
  </tr>
  <tr>
    <td>MURAL (Jain et al., 2021)</td>
    <td>91.8</td>
    <td>92.9</td>
    <td>87.2</td>
    <td>89.7</td>
    <td>91.0</td>
    <td>89.5</td>
    <td>88.1</td>
    <td>90.0</td>
  </tr>
  <tr>
    <td>VLM2Vec (Jiang et al., 2024b)</td>
    <td>83.7</td>
    <td>87.1</td>
    <td>86.7</td>
    <td>92.8</td>
    <td>76.1</td>
    <td>37.2</td>
    <td>63.9</td>
    <td>75.4</td>
  </tr>
  <tr>
    <td>jina (Koukounas et al., 2024)</td>
    <td>93.6</td>
    <td>94.1</td>
    <td>89.8</td>
    <td>91.8</td>
    <td>94.3</td>
    <td>92.7</td>
    <td>90.1</td>
    <td>92.3</td>
  </tr>
  <tr>
    <td>M-CLIP (Carlsson et al., 2022)</td>
    <td>93.1</td>
    <td>93.6</td>
    <td>90.0</td>
    <td>94.0</td>
    <td>94.3</td>
    <td>93.1</td>
    <td>89.0</td>
    <td>92.4</td>
  </tr>
  <tr>
    <td>GME (Zhang et al., 2024b)</td>
    <td>95.1</td>
    <td>96.4</td>
    <td>92.3</td>
    <td>96.4</td>
    <td>94.9</td>
    <td>89.8</td>
    <td>93.6</td>
    <td>94.1</td>
  </tr>
  <tr>
    <td>mmE5 (full)</td>
    <td>96.1</td>
    <td>96.2</td>
    <td>93.3</td>
    <td>96.3</td>
    <td>95.4</td>
    <td>93.6</td>
    <td>96.0</td>
    <td><b>95.3</b></td>
  </tr>
  <tr>
    <td>w/ synthetic data only</td>
    <td>90.9</td>
    <td>89.6</td>
    <td>86.3</td>
    <td>90.2</td>
    <td>90.3</td>
    <td>87.2</td>
    <td>86.7</td>
    <td>88.7</td>
  </tr>
  <tr>
    <td>w/ english synthetic data</td>
    <td>86.3</td>
    <td>86.3</td>
    <td>84.2</td>
    <td>88.8</td>
    <td>84.9</td>
    <td>81.0</td>
    <td>84.4</td>
    <td>85.1</td>
  </tr>
</table>

Table 3: Results on XTD benchmark, a text-to-image retrieval task covering seven languages.

![Pie chart showing the distribution of languages in the synthetic data](page_120_120_600_400.png)

Figure 3: Distribution of languages in the synthetic data.
<table>
  <tr>
    <th>Base MLLM</th>
    <th>Avg. on MMEB</th>
  </tr>
  <tr>
    <td>Phi-3.5-V (Abdin et al., 2024)</td>
    <td>61.0</td>
  </tr>
  <tr>
    <td>LLaVA-1.6 (Liu et al., 2023a)</td>
    <td>65.8</td>
  </tr>
  <tr>
    <td>LLaMA-3-Vision (Meta, 2024) (Ours)</td>
    <td><b>69.8</b></td>
  </tr>
  <tr>
    <th colspan="2">Baselines (For Reference)</th>
  </tr>
  <tr>
    <td>VLM2Vec (Phi-3.5-V)</td>
    <td>60.1</td>
  </tr>
  <tr>
    <td>VLM2Vec (LLaVA-1.6)</td>
    <td>62.9</td>
  </tr>
  <tr>
    <td>MMRet (LLaVA-1.6)</td>
    <td>64.1</td>
  </tr>
  <tr>
    <td>VLM2Vec (LLaMA-3.2)</td>
    <td>64.8</td>
  </tr>
</table>

Table 4: Performances of mmE5 with different MLLMs.

3.5-V\textsuperscript{4}. The performances of mmE5 with different foundation MLLMs are presented in Table 4. The results show that models trained using our method consistently outperform baseline models built on the same foundational MLLMs. This indicates that our synthetic data can effectively enhance the capability of MLLMs to embed multimodal inputs.

4.5 Discussions of Data Synthesis Process

In this section, we will further investigate the data synthesis process via zero-shot experiments.

4.5.1 Ablation Studies

To evaluate each component of our data synthesis framework, we conduct ablation studies of mmE5:

Deep Thinking Process To synthesize high-quality data, we introduce a deep thinking process to boost data synthesis. As presented in Table 5, the performance of mmE5 declines when the Visual Interpretation and Self-evaluation components are excluded. For example, mmE5 performs worse when utilizing the original data compared to the revised data. This indicates that the self-evaluation mechanism can enhance data fidelity, facilitating the training of a more robust embedding model.

Embedding Task Types In order to expand the scope of data, we synthesize data across three task types: classification, VQA, and retrieval. The performance of mmE5 decreases after each type of multimodal embedding data is omitted, demonstrating that our diverse synthetic data can facilitate model generalization. Intriguingly, the performance drops the least after removing the retrieval data, which is inconsistent with previous research (Jiang et al., 2024b). One possible explanation is that our backbone, Llama-3.2 Vision, inherently exhibits more robust retrieval capabilities than Phi-3.5-V.

<table>
  <tr>
    <th>Model</th>
    <th>Avg. on MMEB</th>
  </tr>
  <tr>
    <td>mmE5 (280K synthetic data only)</td>
    <td><b>57.4</b></td>
  </tr>
  <tr>
    <td>w/o. Visual Interpertation</td>
    <td>57.2</td>
  </tr>
  <tr>
    <td>w/o. Self-evaluation</td>
    <td>56.0</td>
  </tr>
  <tr>
    <td>w/o. Classification Data</td>
    <td>52.5</td>
  </tr>
  <tr>
    <td>w/o. VQA Data</td>
    <td>55.1</td>
  </tr>
  <tr>
    <td>w/o. Retrieval Data</td>
    <td>56.5</td>
  </tr>
  <tr>
    <td>w/ IT2I only (MagicLens & MegaPairs)</td>
    <td>30.1</td>
  </tr>
  <tr>
    <td>w/ IT2IT & T2IT only (GME)</td>
    <td>28.6</td>
  </tr>
  <tr>
    <td>w/o. Hard Negative</td>
    <td>56.2</td>
  </tr>
  <tr>
    <td>w/ English Data only (280K)</td>
    <td>57.6</td>
  </tr>
  <tr>
    <td>w/o. English Data (280K)</td>
    <td>56.9</td>
  </tr>
</table>

Table 5: Performances of ablated models on MMEB. For efficient test, we conduct zero-shot experiments on 280K synthetic data, which has the same tasks, modality types and languages as the full synthetic data.

Modality Combinations Most prior works focus on one or two modality types, such as “IT2I” (\emph{e.g.}, MagicLens (Zhang et al., 2024a) and MegaPairs (Zhou et al., 2024a)) or “IT2IT & T2IT” (\emph{e.g.}, GME (Zhang et al., 2024b)). We propose to synthesize data across various modality combinations to enhance the diversity of our synthetic dataset, \emph{i.e.}, the scope of our synthetic multimodal data. To evaluate the impact of these additional modality combinations, we train mmE5 with the same amount of datasets that contain types “IT2I” or “IT2IT & T2IT” only. The performance of mmE5 significantly decreased when limited to these combinations from previous works, which indicates that the additional modalities enable our embedding model to generalize more effectively across different combinations and task types.

Hard Negative Each sample in our synthetic dataset incorporates a hard negative document to help mmE5 learn subtle differences. After excluding the hard negatives, the model’s performance drops significantly, which demonstrates the importance of this technique for contrastive learning.

Language To investigate the impact of linguistic diversity on model performance on English benchmarks, we conducted experiments using synthetic data in two configurations: English-only and non-English languages only. Our model, mmE5, demonstrated a slight performance advantage with English-only synthetic data, although the difference was minimal. Nonetheless, mmE5 achieved satisfactory results with 280K data samples from languages other than English. This suggests that
our multilingual dataset enhances the embedding model’s ability to generalize effectively in both multilingual and English-only contexts.

4.5.2 Scaling Effect
The scaling effect is an important aspect of synthetic data generation for multimodal embedding models (Zhang et al., 2024b; Zhou et al., 2024a). It explores how the performance of the model varies with the size of synthetic datasets. Besides, the data synthesis and training processes demand significant computational resources and time. Therefore, studying the scaling effect allows us to identify the point of diminishing returns, ensuring that resources are utilized efficiently without overproducing redundant data.

In this section, we further investigate the performance of mmE5 using synthetic datasets of varying sizes. Specifically, we conduct zero-shot experiments on MMEB to analyze the scaling effect. As illustrated in Figure 4, mmE5 consistently achieves better performance with increased training data, demonstrating the high quality of our synthetic data again. This paradigm also indicates a linear-log relationship between the model performance and data size, consistent with previous works of text embedding (Chen et al., 2024) and dense retrieval (Fang et al., 2024). This finding facilitates the balancing of the cost and the multimodal embedding model performance for future works.

4.6 Hyperparameter Analysis
In order to analyze the training process of our multimodal embedding model, we perform experiments with mmE5 using various training settings. For efficiency, we report zero-shot results for mmE5 trained with 280K synthetic data. Note that we tune these hyperparameters on evaluation datasets comprising 1K samples from each training set. However, for consistency with previous experiments, we present results on the MMEB test sets.

LoRA Rank denotes the rank of the additional low-rank matrices in LoRA. This parameter influences the number of parameters added to the original model, balancing the model’s capacity and computational efficiency. As shown in the left part of Figure 5, the performance of mmE5 initially improves then drops. This demonstrates a trade-off: a lower rank reduces memory and computation but may lead to underfitting if r is too small, whereas a higher rank risks harming the pre-trained multimodal reasoning capabilities of MLLM.

Training Batch Size In contrastive learning, batch size plays a critical role because it directly affects the number of negative samples available for training. As presented in the middle part of Figure 5, the performance of mmE5 consistently increases with larger batch size. However, large batches demand significantly more GPU memory, i.e., more computational resources.

Temperature The temperature parameter \( \tau \) in the InfoNCE loss (Equation 1) influences the separation between positive and negative samples in the embedding space. We can observe that mmE5’s performance first improves then declines with larger temperature. This pattern suggests a trade-off: a low \( \tau \) forces the model to strongly penalize near-positive negatives, which can lead to overfitting, while a high \( \tau \) leads to a more uniform distribution of embeddings, which may hinder the effective separation of positive and negative samples.

Figure 4: The impact of synthetic data size on multi-modal embedding performance on MMEB.

Figure 5: The zero-shot performances of mmE5 with different training settings on MMEB (280K synthetic data for efficient test).

5 Conclusion
In this work, we synthesize high-quality multimodal multilingual data to train the model mmE5. We first define high-quality multimodal synthetic data based on three criteria: broad scope, robust cross-modal alignment, and high fidelity. Then, we develop a data synthesis framework guided by these principles. Finally, we train a multimodal multi-
lingual embedding model using the high-quality synthetic data. mmE5 achieves SOTA performances on both the general benchmark MMEB and the multilingual benchmark XTD.

Limitations

Our work has several limitations that we intend to resolve in future research:

1. Our model currently relies on the proprietary MLLM GPT-4o for synthesizing multimodal data. Future work should explore aligning smaller MLLMs with the knowledge from GPT-like models to achieve more efficient data synthesis.

2. mmE5 focus on text and image modalities. Future models should aim to extend coverage to additional modalities, such as audio and video.

3. Due to the cost limitation and the observed scaling effect, we limited the amount of data produced for model training. Future research may consider increasing data size while preserving diversity to optimize model performance.

Acknowledgments

This work was supported by Beijing Natural Science Foundation No. L233008, Beijing Municipal Science and Technology Project No. ZZ31100010323009, National Natural Science Foundation of China No. 62272467. The work was partially done at the Beijing Key Laboratory of Research on Large Models and Intelligent Governance and Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE.

References

Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojgan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Ma-houd Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sam-budha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. 2024. Phι-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404.14219.

Pranav Aggarwal and Ajinkya Kale. 2020. Towards zero-shot cross-lingual image retrieval. CoRR, abs/2012.05107.

Fredrik Carlsson, Philipp Eisen, Faton Rekhati, and Magnus Sahlgren. 2022. Cross-lingual and multilingual CLIP. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022, pages 6848–6854. European Language Resources Association.

Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. 2024. Little giants: Synthesizing high-quality embedding data at scale. CoRR, abs/2410.18634.

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Itsev. 2023. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 2818–2829. IEEE.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8440–8451. Association for Computational Linguistics.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248–255. IEEE Computer Society.

Yan Fang, Jingtao Zhan, Qingyao Ai, Jiaxin Mao, Weihang Su, Jia Chen, and Yiqun Liu. 2024. Scaling laws for dense retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research
and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 1339–1349. ACM.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. 2023. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 12031–12041. IEEE.

Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge. 2021. MURAL: multimodal, multitask retrieval across languages. CoRR, abs/2109.05125.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4904–4916. PMLR.

Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. 2024a. E5-V: universal embeddings with multimodal large language models. CoRR, abs/2407.12580.

Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhui Chen. 2024b. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. arXiv preprint arXiv:2410.05160.

Andreas Koukounas, Georgios Mastrapas, Bo Wang, Mohammad Kalim Akram, Sedigheh Eslami, Michael Günther, Isabelle Mohr, Saba Sturua, Scott Martens, Nan Wang, and Han Xiao. 2024. jina-clip-v2: Multilingual multimodal embeddings for text and images.

Chaofan Li, Zheng Liu, Shitao Xiao, Yingxia Shao, and Defu Lian. 2024a. Llama2vec: Unsupervised adaptation of large language models for dense retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 3490–3500. Association for Computational Linguistics.

Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohua Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. 2024b. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. CoRR, abs/2402.13064.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 19730–19742. PMLR.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 12888–12900. PMLR.

Sheng-chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. 2024. Mm-embed: Universal multimodal retrieval with multimodal llms.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning.

Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv, Zhiyuan Liu, and Ge Yu. 2023b. Universal vision-language dense retrieval: Learning A unified representation space for multi-modal retrieval. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10674–10685. IEEE.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. LAION-400M: open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/2111.02114.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA models that can read. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8317–8326. Computer Vision Foundation / IEEE.

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024a. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 11897–11916. Association for Computational Linguistics.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. CoRR, abs/2409.12191.

Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhui Chen. 2024. Unir: Training and benchmarking universal multimodal information retrievers. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXXVII, volume 15145 of Lecture Notes in Computer Science, pages 387–404. Springer.

Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muenninghoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 641–649. ACM.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text foundation models. Trans. Mach. Learn. Res., 2022.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11941–11952. IEEE.

Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhui Chen, Yu Su, and Ming-Wei Chang. 2024a. Magiclens: Self-supervised image retrieval with open-ended instructions. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. 2024b. Gme: Improving universal multimodal retrieval by multimodal llms.

Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. 2024a. Megapairs: Massive data synthesis for universal multimodal retrieval. arXiv preprint arXiv:2412.14475.

Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. 2024b. VISTA: visualized text embedding for universal multi-modal retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 3185–3200. Association for Computational Linguistics.

Tianshuo Zhou, Sen Mei, Xinze Li, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, Yu Gu, and Ge Yu. 2024c. MARVEL: unlocking the multi-modal capability of dense retrieval via visual module plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 14608–14624. Association for Computational Linguistics.
Appendix

A  Details about Synthetic Data

<table>
  <tr>
    <th>Task</th>
    <th>Modality combination</th>
    <th># Samples</th>
  </tr>
  <tr>
    <td rowspan="2">Classification</td>
    <td>image-to-text</td>
    <td>126,177</td>
  </tr>
  <tr>
    <td>(image,text)-to-text</td>
    <td>13,823</td>
  </tr>
  <tr>
    <td rowspan="5">Retrieval</td>
    <td>image-to-text</td>
    <td>98,040</td>
  </tr>
  <tr>
    <td>(image,text)-to-text</td>
    <td>41,960</td>
  </tr>
  <tr>
    <td>(image,text)-to-image</td>
    <td>56,185</td>
  </tr>
  <tr>
    <td>image-to-image</td>
    <td>27,988</td>
  </tr>
  <tr>
    <td>(image,text)-to-(image,text)</td>
    <td>27,656</td>
  </tr>
  <tr>
    <td>text-to-image</td>
    <td>14,090</td>
  </tr>
  <tr>
    <td>text-to-(image,text)</td>
    <td>14,081</td>
  </tr>
  <tr>
    <td>VQA</td>
    <td>(image,text)-to-text</td>
    <td>140,000</td>
  </tr>
</table>

Table 6: Statistics of the multimodal synthetic data used for training mmE5.

In this study, we introduce a synthetic multi-modal multilingual embedding dataset designed to facilitate model learning. This section delves into the details of our synthetic dataset. The dataset is comprised of three distinct tasks and seven modality combinations, totaling 560K data samples. Table 6 provides a detailed statistical overview of our synthetic data, categorized by tasks and modalities.

B  Implementation Details

B.1  Data Synthesis

For the data synthesis process, we employ the MLLM GPT-4o-2024-08-06 model to generate data samples. Both the temperature and top-p parameters are set to 1.0 to ensure diverse and coherent outputs. Our image corpus is sourced from LAION-400m (Schuhmann et al., 2021), from which we exclude images that are either corrupted or have inaccessible URLs. Each synthetic data sample incorporates one image sampled from this corpus as the query image. For modality combinations that include images on the document side, we utilize the jina-clip-v2\footnote{https://huggingface.co/jinaai/jina-clip-v2} model to retrieve a similar image, along with a hard negative image, to serve as additional inputs.

B.2  Finetuning Embedding Model

We train mmE5 using the open-source MLLM, Llama-3.2-11B-Vision\footnote{https://huggingface.co/meta-llama/Llama-3.2-11B-Vision}. The training is conducted on 64 NVIDIA A100 GPUs, each equipped with 40GB of memory. To optimize GPU memory usage, we employ gradient checkpointing and set the gradient accumulation steps to 4. The model is trained with a learning rate of 2e-5 for one epoch, utilizing both synthetic and labeled data. LoRA (Hu et al., 2022) is applied to the MLLM with a rank of 8. Each training sample incorporates one hard negative document. Hard negatives are mined for each subset of MMEB using VLM2Vec-LoRA\footnote{https://huggingface.co/TIGER-Lab/VLM2Vec-LoRA}, with the 70th position in the ranking list selected as the hard negative sample.

C  Prompts

We use different prompts of data synthesis for different tasks. For retrieval task, we design two prompts for modality combinations that involve images on the document side or not. Let us take the prompt of generating classification data for an example to illustrate the prompt design.

First, we sample a modality combination from {image-to-text, (image,text)-to-text}. If the query side does not include texts, the “input_text” of the classification data sample will be an empty string. Similarly, for modalities of retrieval task that do not include document texts, the “positive_document” and “hard_negative_document” will be empty. Following previous works of synthesizing text embedding data (Wang et al., 2024a; Chen et al., 2024), we will randomly select a clarity and difficulty setting to enhance diversity.

Then, for the multi-aspect visual description process, we ask the MLLM to explicitly \textbf{include four perspectives of description}. Besides, for the data synthesis process, we also ask the MLLM to \textbf{follow some specific guidelines}. Furthermore, the MLLM will \textbf{evaluate the initially generated data from several aspects} and provide “possible\_improvements”. Finally, the revised version of data will be used as the output data sample. Note that there are no task instructions generated for the VQA task, since they are all fixed as “Represent the given image with the following question:”.

D  Data Examples

In this section, we present the examples of the synthetic multimodal embedding data for Retrieval (Figure 6 and Figure 7), Classification (Figure 8), and VQA (Figure 9) tasks.
Prompt: Synthesizing Classification Data

Your mission is to first produce a detailed visual description of the image (within 300 words), identifying all potential aspects for generating high-quality data for a {image-to-text, (image,text)-to-text} classification task.

Based on the description, brainstorm a potentially useful task.

Here are a few examples for your reference: {example tasks}

Then, you should write one multi-modal classification example for this task in JSON format. The JSON object must contain the following keys:

- "description": a string, your detailed visual description, listing all required elements.
- "task_instruction": a string, describing the classification task.
- "input_text": {"an empty string", "a string the input text specified by the classification task"}.
- "label": a string, the correct label of the image and input_text (if not empty) based on the task instruction.
- "misleading_label": a string, an incorrect label that is related to the task.
- "evaluation": a string, a brief summary of the evaluation of data quality.
- "possible_improvements": a string, suggestions for improving the data based on the guidelines.
- "revised_task_instruction": the revised task instruction.
- "revised_input_text": the revised input text, {"an empty string", "a string the input text specified by the classification task"}.
- "revised_label": the revised label.
- "revised_misleading_label": the revised misleading label.

For the description, please include the following elements:
- General Description: Provide an overall summary of the image, including the primary objects, scene, and notable features.
- Object-Level Details: Identify the individual objects in the image, their attributes (e.g., color, size, position), and their relationships to one another.
- Contextual Features: Describe the scene or environment, including background details, lighting, and any actions taking place.
- Task-specific Brainstorming: Analyze explore how this image could relate to text (e.g., captions, contextual descriptions).

Please adhere to the following guidelines:
- Task should be suitable for the given image.
- Avoid generate task similar to classification of sentiment / subject / study field / genre / main topic / spam / urgency / language.
- The "input_text" should be {"less than 10", "at least 10", "at least 50", "at least 100", "at least 200"} words and diverse in expression (if not empty).
- The "misleading_label" must be a valid label for the given task, but not as appropriate as the "label" for the image.
- The text of "task_instruction" should be in English and others fields should be in {language}.
- Avoid including the values of the "label" and "misleading_label" fields in the "input_text" (if not empty), that would make the task too easy.
- The "input text" (if not empty) is {"clear", "understandable with some effort", "ambiguous"} and requires {"high school", "college", "PhD"} level education to comprehend.
- When generating the data, please evaluate the following aspects:
  1. Relevance: Are the generated input texts and labels (if not empty) tightly connected to their corresponding image and task objectives? Does the task instruction effectively link the query image with the positive label?
  2. Plausibility: Are misleading labels sufficiently relevant to the image or labels while remaining definitively incorrect? Could they mislead the model?
  3. Clarity: Is the generated task clear and unambiguous, providing sufficient instruction to connect the query image with the label, without being overly specific or abstract?
  4. Diversity: Does the generated data introduce variation in task instructions, texts (if not empty), and labels to avoid repetitive patterns in the dataset?
- Provide a detailed evaluation of the data based on the above criteria. For each criterion, explain specific flaws or strengths.
- Suggest specific revisions to address any identified weaknesses, ensuring the revised data better aligns with the guidelines and task objectives.
- Avoid revisions that overly simplify the task instruction, text (if not empty), or labels, as this may reduce their utility for training.
- Ensure that revised data maintains consistency with the corresponding image content and classification task requirements.

Your output must always be a JSON object only. Do not explain yourself or output anything else. Be creative!
Prompt: Synthesizing VQA Data

Your mission is to first produce detailed visual descriptions of the image (within 300 words), identifying all potential aspects for generating high-quality data for a visual QA task.

Based on the description, write one visual QA example based on the given image in JSON format. The JSON object must contain the following keys:

- "description": a string, your detailed visual description, listing all required elements.
- "question": a string, specifying the question based on the image content.
- "positive_answer": a string, the correct answer for the question based on the image content.
- "hard_negative_answer": a string, an incorrect answer that appears plausible but is ultimately wrong.
- "evaluation": a string, a brief summary of the evaluation of data quality.
- "possible_improvements": a string, suggestions for improving the data based on the guidelines.
- "revised_question": the revised question.
- "revised_positive_answer": the revised positive answer.
- "revised_hard_negative_answer": the revised hard negative answer.

For the description, please include the following elements:
- General Description: Provide an overall summary of the image, including the primary objects, scene, and notable features.
- Object-Level Details: Identify the individual objects in the image, their attributes (e.g., color, size, position), and their relationships to one another.
- Contextual Features: Describe the scene or environment, including background details, lighting, and any actions taking place.
- Task-specific Brainstorming: Analyze explore how this image could relate to text (e.g., captions, contextual descriptions).

Please adhere to the following guidelines:
- The "question" should be { "less than 10", "at least 10", "at least 50", "at least 100", "at least 200" } words and diverse in expression.
- The "hard_negative_answer" must be plausible but less appropriate than the "positive_answer".
- The values for all fields should be in {language}.
- Avoid including explicit hints in the question that make the answer too obvious.
- The "question" (if not empty) is {"clear", "understandable with some effort", "ambiguous"} and requires {"high school", "college", "PhD"} level education to comprehend.
- When generating the data, please evaluate the following aspects:
  1. Relevance: Are the generated question and answers tightly linked to the image content and consistent with the task requirements?
  2. Plausibility: Does the "hard_negative_answer" closely resemble the "positive_answer" while remaining definitively incorrect? Could it mislead the model?
  3. Diversity: Does the generated data introduce variation in questions, and answers to avoid repetitive patterns in the dataset?
- Provide a detailed evaluation of the data based on the above criteria. For each criterion, explain specific flaws or strengths.
- Suggest specific revisions to address any identified weaknesses, ensuring the revised data better aligns with the guidelines and task objectives.
- Avoid revisions that overly simplify or trivialize the "question".
- Ensure revised data maintain consistency with the image content and task-specific requirements.

Your output must always be a JSON object only. Do not explain yourself or output anything else. Be creative!
Prompt: Synthesizing Retrieval Data (Only Query Image)

Your mission is to first produce a detailed visual description of the image (within 300 words), identifying all potential aspects for generating high-quality data for a {image-to-text, (image,text)-to-text} retrieval task.

Based on the description, brainstorm a potentially useful task.

Here are a few examples for your reference: {example tasks}

Then, you should write one retrieval example for this task in JSON format. The JSON object must contain the following keys:

- "description": a string, your detailed visual description, listing all required elements.
- "task_instruction": a string, describing the retrieval task.
- "query": {"an empty string", "a random user search query specified by the retrieval task and the query image."}
- "positive_document": a string, the relevant document for the query image content.
- "hard_negative_document": a string, a hard negative document that only appears relevant to the query image content.
- "evaluation": a string, a brief summary of the evaluation of data quality.
- "possible_improvements": a string, suggestions for improving the data based on the guidelines.
- "revised_task_instruction": the revised task instruction.
- "revised_query": the revised query, {"an empty string", "a random user search query specified by the retrieval task and the query image."}.
- "revised_positive_document": the revised positive document, a string, the relevant document for the query image content.
- "revised_hard_negative_document": the revised hard negative document, a string, a hard negative document that only appears relevant to the query image content.

For the description, please include the following elements:
- General Description: Provide an overall summary of the image, including the primary objects, scene, and notable features.
- Object-Level Details: Identify the individual objects in the image, their attributes (e.g., color, size, position), and their relationships to one another.
- Contextual Features: Describe the scene or environment, including background details, lighting, and any actions taking place.
- Task-specific Brainstorming: Analyze explore how this image could relate to text (e.g., captions, contextual descriptions).

Please adhere to the following guidelines:
- The task should involve both query and documents (positive and hard negative, if not empty). It must directly indicate the relation without being overly detailed or abstract.
- The query (if not empty) should be {"extremely long-tail", "long-tail", "common"}, {"less than 5 words", "5 to 15 words", "at least 10 words"}, {"clear", "understandable with some effort", "ambiguous"}, and diverse in topic.
- All documents (if not empty) must be created independent of the query. Avoid copying the query verbatim. It’s acceptable if some parts of the "positive_document" are not topically related to the query.
- All documents (if not empty) should be at least ["10", "30", "200", "300"] words long.
- The "hard_negative_document" (if not empty) contains some useful information, but it should be less useful or comprehensive compared to the "positive_document".
- The text of "task_instruction" should be in English and others fields should be in {language}.
- Do not provide any explanation in any document (if not empty) on why it is relevant or not relevant to the query.
- Do not use the word "query" or "document" in the generated content.
- Both the query and documents (if not empty) require {"high school", "college", "PhD"} level education to understand.

When generating the data, please evaluate the following aspects:
1. Relevance: Are the generated query and documents (if not empty) tightly connected to their corresponding image and task objectives? Does the task instruction effectively link the query image with the positive text?
2. Plausibility: Are hard negatives sufficiently similar to the query or positive examples while remaining definitively incorrect? Could they mislead the model?
3. Clarity: Is the generated task clear and unambiguous, providing sufficient instruction to connect the query image with the positive document, without being overly specific or abstract?
4. Diversity: Does the generated data introduce variation in task instructions, queries, and documents to avoid repetitive patterns in the dataset?
- Provide a detailed evaluation of the data based on the above criteria. For each criterion, explain specific flaws or strengths.
- Suggest specific revisions to address any identified weaknesses, ensuring the revised data better aligns with the guidelines and task objectives.
- Avoid revisions that overly simplify the task instruction, query, or documents, as this may reduce their utility for training.
- Ensure that revised data maintains consistency with the corresponding image content and retrieval task requirements.

Your output must always be a JSON object only. Do not explain yourself or output anything else. Be creative!
Prompt: Synthesizing Retrieval Data (With Document Images)

Your mission is to first produce detailed visual descriptions of the images (within 600 words), identifying all potential aspects for generating high-quality data for a {(image,text)-to-image, image-to-image, (image,text)-to-(image,text), text-to-image, text-to-(image,text)} retrieval task that involves both query and document images.

Based on the description, brainstorm a potentially useful task.

Here are a few examples for your reference: {example tasks}

Then, you should write one retrieval example for this task in JSON format. The JSON object must contain the following keys:

- "description": a string, your detailed visual description, listing all required elements.
- "task_instruction": a string, describing the retrieval task.
- "query": {"an empty string", "a random user search query specified by the retrieval task and the query image."}
- "positive_document": {"an empty string", "a string, the relevant document for the query based on the query text and image content"}
- "hard_negative_document": {"an empty string", "a string, a hard negative document that only appears relevant to the query and the query image content."}
- "evaluation": a string, a brief summary of the evaluation of data quality.
- "possible_improvements": a string, suggestions for improving the data based on the guidelines.
- "revised_task_instruction": the revised task instruction.
- "revised_query": the revised query, {"an empty string", "a random user search query specified by the retrieval task and the query image."}.
- "revised_positive_document": the revised positive document, a string, {"an empty string", "a string, the relevant document for the query based on the query text and image content"}
- "revised_hard_negative_document": the revised hard negative document, {"an empty string", "a string, a hard negative document that only appears relevant to the query and the query image content."})

For the description, please include the following elements:
- General Description: Provide an overall summary of the image, including the primary objects, scene, and notable features.
- Object-Level Details: Identify the individual objects in the image, their attributes (e.g., color, size, position), and their relationships to one another.
- Contextual Features: Describe the scene or environment, including background details, lighting, and any actions taking place.
- Task-specific Brainstorming: Analyze explore how this image could relate to text (e.g., captions, contextual descriptions).

Please adhere to the following guidelines:
- The task must connect the query image and positive image through their content. It must directly indicate the relation without being overly detailed or abstract.
- The query (if not empty) should be {"extremely long-tail", "long-tail", "common"}, {"less than 5 words", "5 to 15 words", "at least 10 words"}, {"clear", "understandable with some effort", "ambiguous"}, and diverse in topic.
- The query (if not empty) should effectively associate the query image with the positive image.
- All documents (if not empty) must be created independent of the query. Avoid copying the query verbatim. It’s acceptable if some parts of the "positive_document" are not topically related to the query.
- All documents (if not empty) should be at least ["10", "30", "200", "300"] words long.
- The "hard_negative_document" (if not empty) contains some useful information, but it should be less useful or comprehensive compared to the "positive_document".
- The text of "task_instruction" should be in English and others fields should be in {language}.
- Do not provide any explanation in any document (if not empty) on why it is relevant or not relevant to the query.
- Do not use the word "query" or "document" in the generated content.
- Both the query and documents (if not empty) require {"high school", "college", "PhD"} level education to understand.

When generating the data, please evaluate the following aspects:
1. Relevance: Are the generated query and documents (if present) tightly linked to their corresponding images? Does the task instruction effectively connect the query image to the positive image?
2. Plausibility: Are the negative examples, including hard negatives, realistic and similar enough to the positive image to challenge the model, while still being definitively incorrect?
3. Clarity: Is the generated task clear and unambiguous, providing sufficient instruction to connect the query image with the positive image, without being overly specific or abstract?
4. Diversity: Does the generated data introduce variation in task instructions, queries, and documents to avoid repetitive patterns in the dataset?
- Provide a detailed evaluation of the data based on the above criteria. For each criterion, explain specific flaws or strengths.
- Suggest specific revisions to address any identified weaknesses, ensuring the revised data better aligns with the guidelines and task objectives.
- Avoid revisions that overly simplify the task or create unrealistic connections between the query and positive image.
- Ensure that revised data maintains consistency with the corresponding image content and retrieval task requirements.

Your output must always be a JSON object only. Do not explain yourself or output anything else. Be creative!
An Example of Synthetic Retrieval IT2IT Data (Description & Initial Data)

<table>
  <tr>
    <th>Images (Query; Positive; Hard Negative)</th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td rowspan="3">Description</td>
    <td><b>The first image</b> portrays a vibrant outdoor scene of a powerful waterfall cascading down a ledge surrounded by lush foliage. The waterfall is centrally located, with a stream flowing in the foreground over several rocks and branches. The image captures a vivid display of sunlight filtering through dense greenery, with bright shades of green and the mist from the waterfall adding to the natural beauty. The scene suggests a secluded, serene environment.</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td><b>The second image</b> presents a cascading waterfall, framed in monochrome, offering a sense of timelessness and tranquility. The waterfall descends through rocky formations before settling into a small pool at the bottom. The foreground features smooth rocks, polished by flowing water, which lends texture and depth to the black-and-white photograph. The dark shadows and bright highlights create stark contrasts, enhancing the visual impact. The surrounding area is densely wooded, indicating a secluded spot within a forest.</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td><b>The third image</b> displays a vividly colorful landscape dominated by an expansive lake flanked by tall, rugged cliffs. The cliffs exhibit a range of warm hues, from deep reds to oranges, basked in the glow of a low-hanging sun. The sky is a brilliant blue with scattered clouds, which together with the reflections on the water, create a picturesque scene. Reeds in the foreground frame the serene water, whose stillness mirrors the sky and cliffs, suggesting peace and prismatic beauty.</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Task Instruction</td>
    <td colspan="3">Retrieve an image-description pair depicting cascading water features in different lighting and weather conditions.</td>
  </tr>
  <tr>
    <td>Query</td>
    <td colspan="3">Picturesque waterfall with vibrant forest and sunlight</td>
  </tr>
  <tr>
    <td>Positive document</td>
    <td colspan="3">This image captures a waterfall in a forested area, presented in a monochrome format. It highlights the natural setting with cascading water over rocky terrain, creating an air of serenity. The use of black and white emphasizes light contrasts and textures of the water and surroundings, offering a classic, peaceful portrayal of nature unaffected by human interference, emphasizing timeless tranquility.</td>
  </tr>
  <tr>
    <td>Hard negative document</td>
    <td colspan="3">In this photograph, a beautiful lake sits surrounded by striking cliffs under a bright blue sky. The cliffs are illuminated by the sun, showcasing a range of warm colors that contrast with clear blue reflections on the water surface. Framing the lake are tall reeds, contributing to a sense of isolation and calm are very good.</td>
  </tr>
</table>

Figure 6: An example of the synthetic Retrieval IT2IT data (part 1). This part includes the input images, the multi-aspect descriptions, and the initially generated data.
An Example of Synthetic Retrieval IT2IT Data (Evaluation & Refinement)

<table>
  <tr>
    <th>Evaluation</th>
    <td>
      <b>Relevance:</b> The task effectively links the images through their feature of cascading water though the third doesn't fit.<br>
      <b>Plausibility:</b> The negative stands apart from the cascading water concept but holds some scenic elements.<br>
      <b>Clarity:</b> The task is clearly defined.<br>
      <b>Diversity:</b> There is a good variation in image scenes using water as a theme.
    </td>
  </tr>
  <tr>
    <th>Possible Improvements</th>
    <td>Ensure the negative image relates closer to cascading water but under different weather conditions or lighting. Provide more differentiation between positive and negative descriptions. Revise watermark elements for image clarity.</td>
  </tr>
  <tr>
    <th>Revised Task Instruction</th>
    <td>Retrieve an image-description pair depicting waterfalls in different lighting or weather conditions.</td>
  </tr>
  <tr>
    <th>Revised Query</th>
    <td>Vibrant sunlight waterfall with lush surrounding forest</td>
  </tr>
  <tr>
    <th>Revised Positive document</th>
    <td>This image captures a waterfall in a forested area, presented in a monochrome format. It highlights the natural setting with cascading water over rocky terrain, creating an air of serenity. The use of black and white emphasizes light contrasts and textures of the water and surroundings, offering a classic, peaceful portrayal of nature unaffected by human interference, emphasizing timeless tranquility.</td>
  </tr>
  <tr>
    <th>Revised Hard negative document</th>
    <td>The photo showcases a breathtaking desert landscape under a vibrant sky. The stillness of the clear, reflective lake beneath towering orange-red cliffs suggests peace. Foregrounding tall reeds framing this water body adds depth without cascading water, contrasting yet resembling a waterfall.</td>
  </tr>
</table>

Figure 7: An example of the synthetic Retrieval IT2IT data (part 2). This part includes the evaluation, possible improvements, and the revised data.
An Example of Synthetic Classification IT2T Data

<table>
  <tr>
    <th>Image (Query)</th>
    <td><img src="page_312_370_184_184.png" alt="Movie poster featuring a dramatic scene with a man and woman in a dystopian landscape"></td>
  </tr>
  <tr>
    <th>Description</th>
    <td>The image is a movie poster featuring a dramatic scene with a dusky, smoky background. The foreground shows a man in a sharp, dark suit and a woman in a leather outfit, both shown from the waist up. They are positioned closely, facing the viewer. Above them, the text 'NICOLAS CAGE' is prominently displayed. The background depicts a dystopian, smoky landscape with industrial structures, flames, and dark billowing smoke, giving an impression of chaos and destruction. The middle lower section features the title 'THE HUMANITY BUREAU' in bold, white capital letters. Below the title, there are smaller texts with names 'Sarah Lind', 'Hugh Dillon', and more, alongside the phrase 'Escape from New America'. The color scheme is predominantly dark, with flashes of fiery orange.</td>
  </tr>
  <tr>
    <th>Task Instruction</th>
    <td>Identify the genre this movie poster belongs to</td>
  </tr>
  <tr>
    <th>Input Text</th>
    <td>Destruction backdrop</td>
  </tr>
  <tr>
    <th>Label</th>
    <td>Science Fiction</td>
  </tr>
  <tr>
    <th>Misleading Label</th>
    <td>Romantic Comedy</td>
  </tr>
  <tr>
    <th>Evaluation</th>
    <td>
      <b>Relevance:</b> The poster's imagery and text signaling a dystopian theme are closely related to the 'Science Fiction' label.<br>
      <b>Plausibility:</b> The misleading label 'Romantic Comedy' might be incorrect due to the intense imagery of destruction, however, it still bears a relationship to movie genres.<br>
      <b>Clarity:</b> The task is clear in instructing the identification of film genre based on visual cues.<br>
      <b>Diversity:</b> The data introduces variety in illustration, steering clear of repetitive genre classification.
    </td>
  </tr>
  <tr>
    <th>Possible Improvements</th>
    <td>To enhance data quality, add more challenging input text variations to test genre recognition effectiveness and to diversify task styles.</td>
  </tr>
  <tr>
    <th>Revised Task Instruction</th>
    <td>Determine the movie genre based on the visual elements in the poster.</td>
  </tr>
  <tr>
    <th>Revised Input Text</th>
    <td>Chaotic landscape scene</td>
  </tr>
  <tr>
    <th>Revised Label</th>
    <td>Science Fiction</td>
  </tr>
  <tr>
    <th>Revised Misleading Label</th>
    <td>Adventure</td>
  </tr>
</table>

Figure 8: An example of the synthetic Classification IT2T data.
<table>
  <tr>
    <th>Image (Query)</th>
    <td><img src="page_324_312_256_192.png" alt="A young boy sitting in a rustic wooden crate filled with apples, eating an apple." /></td>
  </tr>
  <tr>
    <th>Description</th>
    <td>The image features a young boy sitting in a rustic wooden crate filled with apples, with a specific focus on him biting into a green apple. The boy’s light blonde, wavy hair frames his face as he appears engaged in eating. He is dressed in a striped shirt with blue, green, and yellow hues, paired with blue jeans, adding a playful and casual vibe. The wooden crate, slightly weathered and grayish, holds several red and green apples, emphasizing the theme of harvest. Surrounding the boy are lush apple trees, their branches laden with ripe red and green apples and vibrant green leaves. The background is filled with soft sunlight filtering through the orchard, creating a bright and cheerful setting. The natural tones of green and red apples contrast with the boy’s colorful outfit, making him the focal point. The scene suggests a connection to outdoor activities like apple picking and highlights the freshness of rural environments.</td>
  </tr>
  <tr>
    <th>Question</th>
    <td>In the image, there is a child dressed in a brightly striped shirt standing near a large container filled with colorful fruits. Based on the scene depicted, what type of fruit is the child likely surrounded by, considering the visible tree and its fruit characteristics?</td>
  </tr>
  <tr>
    <th>Positive Answer</th>
    <td>Apples</td>
  </tr>
  <tr>
    <th>Negative Answer</th>
    <td>Bananas</td>
  </tr>
  <tr>
    <th>Evaluation</th>
    <td><b>Relevance</b> of the question and answers is high, focusing specifically on the child and the surrounding fruits, while maintaining consistency with the image content.<br>The hard negative answer, although certainly wrong, could mislead due to its <span style="color:red">plausible</span> nature, but overall is less fitting given the absence of bananas in the scene.<br>The data displays <b>diversity</b> through detailed description without repetitive elements.</td>
  </tr>
  <tr>
    <th>Possible Improvements</th>
    <td>Consider more deeply associating the description of the tree’s characteristics to add context that makes the negative answer slightly more challenging, without sacrificing the overall accuracy of context. Ensuring a balance between misleading potential and implausibility would enhance this data further.</td>
  </tr>
  <tr>
    <th>Revised Question</th>
    <td>What type of fruit, seen in the trees and collected in the container, is prominently featured around the child who is standing next to the wooden box in this lush outdoor setting?</td>
  </tr>
  <tr>
    <th>Revised Positive Answer</th>
    <td>Apples</td>
  </tr>
  <tr>
    <th>Revised Negative Answer</th>
    <td>Oranges</td>
  </tr>
</table>

Figure 9: An example of the synthetic VQA IT2T data.
E  Detailed Results

In this section, we present the detailed comparisons of mmE5 to baseline models on both zero-shot and supervised finetuning settings on the MMEB benchmark (Jiang et al., 2024b). Due to space limitation, we omit the detailed results of partially supervised finetuning models.
<table>
  <tr>
    <th rowspan="2">Task</th>
    <th colspan="8">Zero-shot Setting Models</th>
    <th colspan="6">Supervised Finetuning Models</th>
  </tr>
  <tr>
    <th>CLIP</th>
    <th>OpenCLIP</th>
    <th>SigLIP</th>
    <th>BLIP2</th>
    <th>MagicLens</th>
    <th>E5-V</th>
    <th>MMRet</th>
    <th>mmE5</th>
    <th>VLm2Vec</th>
    <th>MMRet</th>
    <th>mmE5</th>
  </tr>
  <tr>
    <th colspan="13">Classification (10 tasks)</th>
  </tr>
  <tr>
    <td>ImageNet-1K</td><td>55.8</td><td>63.5</td><td>45.4</td><td>10.3</td><td>48.0</td><td>9.6</td><td>49.1</td><td>68.8</td><td>74.5</td><td>58.8</td><td>77.6</td>
  </tr>
  <tr>
    <td>N24News</td><td>34.7</td><td>38.6</td><td>13.9</td><td>36.0</td><td>33.7</td><td>23.4</td><td>45.8</td><td>54.5</td><td>80.3</td><td>71.3</td><td>82.1</td>
  </tr>
  <tr>
    <td>HatefulMemes</td><td>51.1</td><td>51.7</td><td>47.2</td><td>49.6</td><td>49.0</td><td>49.7</td><td>51.0</td><td>55.0</td><td>67.9</td><td>53.7</td><td>64.3</td>
  </tr>
  <tr>
    <td>VOC2007</td><td>50.7</td><td>52.4</td><td>64.3</td><td>52.1</td><td>51.6</td><td>49.9</td><td>74.6</td><td>73.9</td><td>91.5</td><td>85.0</td><td>91.0</td>
  </tr>
  <tr>
    <td>SUN397</td><td>43.4</td><td>68.8</td><td>39.6</td><td>34.5</td><td>57.0</td><td>33.1</td><td>60.1</td><td>72.7</td><td>75.8</td><td>70.0</td><td>77.9</td>
  </tr>
  <tr>
    <td>Place365</td><td>28.5</td><td>37.8</td><td>20.0</td><td>21.5</td><td>31.5</td><td>8.6</td><td>35.3</td><td>39.7</td><td>44.0</td><td>43.0</td><td>42.6</td>
  </tr>
  <tr>
    <td>ImageNet-A</td><td>25.5</td><td>14.2</td><td>42.6</td><td>3.2</td><td>8.0</td><td>2.0</td><td>31.6</td><td>46.1</td><td>43.6</td><td>36.1</td><td>56.7</td>
  </tr>
  <tr>
    <td>ImageNet-R</td><td>75.6</td><td>83.0</td><td>75.0</td><td>39.7</td><td>70.9</td><td>30.8</td><td>66.2</td><td>86.2</td><td>79.8</td><td>71.6</td><td>86.3</td>
  </tr>
  <tr>
    <td>ObjectNet</td><td>43.4</td><td>51.4</td><td>40.3</td><td>20.6</td><td>31.6</td><td>7.5</td><td>49.2</td><td>74.8</td><td>39.6</td><td>55.8</td><td>62.2</td>
  </tr>
  <tr>
    <td>Country-211</td><td>19.2</td><td>16.8</td><td>14.2</td><td>2.5</td><td>6.2</td><td>3.1</td><td>9.3</td><td>35.1</td><td>14.7</td><td>14.7</td><td>34.8</td>
  </tr>
  <tr>
    <td><i>All Classification</i></td><td>42.8</td><td>47.8</td><td>40.3</td><td>27.0</td><td>38.8</td><td>21.8</td><td>47.2</td><td>60.7</td><td>61.2</td><td>56.0</td><td>67.6</td>
  </tr>
  <tr>
    <th colspan="13">VQA (10 tasks)</th>
  </tr>
  <tr>
    <td>OK-VQA</td><td>7.5</td><td>11.5</td><td>2.4</td><td>8.7</td><td>12.7</td><td>8.9</td><td>28.0</td><td>56.6</td><td>69.0</td><td>73.3</td><td>67.9</td>
  </tr>
  <tr>
    <td>A-OKVQA</td><td>3.8</td><td>3.3</td><td>1.5</td><td>3.2</td><td>2.9</td><td>5.9</td><td>11.6</td><td>50.0</td><td>54.4</td><td>56.7</td><td>56.4</td>
  </tr>
  <tr>
    <td>DocVQA</td><td>4.0</td><td>5.3</td><td>4.2</td><td>2.6</td><td>3.0</td><td>1.7</td><td>12.6</td><td>81.3</td><td>52.0</td><td>78.5</td><td>90.3</td>
  </tr>
  <tr>
    <td>InfographicsVQA</td><td>4.6</td><td>4.6</td><td>2.7</td><td>2.0</td><td>5.9</td><td>2.3</td><td>10.6</td><td>44.0</td><td>30.7</td><td>39.3</td><td>56.2</td>
  </tr>
  <tr>
    <td>ChartQA</td><td>1.4</td><td>1.5</td><td>3.0</td><td>0.5</td><td>0.9</td><td>2.4</td><td>2.4</td><td>35.2</td><td>34.8</td><td>41.7</td><td>50.3</td>
  </tr>
  <tr>
    <td>Visual7W</td><td>4.0</td><td>2.6</td><td>1.2</td><td>1.3</td><td>2.5</td><td>5.8</td><td>9.0</td><td>40.4</td><td>49.8</td><td>49.5</td><td>51.9</td>
  </tr>
  <tr>
    <td>ScienceQA</td><td>9.4</td><td>10.2</td><td>7.9</td><td>6.8</td><td>5.2</td><td>3.6</td><td>23.3</td><td>47.3</td><td>42.1</td><td>45.2</td><td>55.7</td>
  </tr>
  <tr>
    <td>VizWiz</td><td>8.2</td><td>6.6</td><td>2.3</td><td>4.0</td><td>1.7</td><td>2.6</td><td>25.9</td><td>54.0</td><td>43.0</td><td>51.7</td><td>52.8</td>
  </tr>
  <tr>
    <td>GQA</td><td>41.3</td><td>52.5</td><td>57.5</td><td>9.7</td><td>43.5</td><td>7.8</td><td>41.3</td><td>65.4</td><td>61.2</td><td>59.0</td><td>62.1</td>
  </tr>
  <tr>
    <td>TextVQA</td><td>7.0</td><td>10.9</td><td>1.0</td><td>3.3</td><td>4.6</td><td>3.2</td><td>18.9</td><td>83.1</td><td>62.0</td><td>79.0</td><td>83.5</td>
  </tr>
  <tr>
    <td><b>Avg.</b></td><td>9.1</td><td>10.9</td><td>8.4</td><td>4.2</td><td>8.3</td><td>4.9</td><td>18.4</td><td>55.7</td><td>49.9</td><td>57.4</td><td>62.7</td>
  </tr>
  <tr>
    <th colspan="13">Retrieval (12 tasks)</th>
  </tr>
  <tr>
    <td>VisDial</td><td>30.7</td><td>25.4</td><td>21.5</td><td>18.0</td><td>24.8</td><td>9.2</td><td>62.6</td><td>39.1</td><td>80.9</td><td>83.0</td><td>73.7</td>
  </tr>
  <tr>
    <td>CIRR</td><td>12.6</td><td>15.4</td><td>15.1</td><td>9.8</td><td>39.1</td><td>6.1</td><td>65.7</td><td>41.6</td><td>49.9</td><td>61.4</td><td>54.9</td>
  </tr>
  <tr>
    <td>VisualNews_t2i</td><td>78.9</td><td>74.0</td><td>51.0</td><td>48.1</td><td>50.7</td><td>13.5</td><td>45.7</td><td>51.2</td><td>75.4</td><td>74.2</td><td>77.7</td>
  </tr>
  <tr>
    <td>VisualNews_i2t</td><td>79.6</td><td>78.0</td><td>52.4</td><td>13.5</td><td>21.1</td><td>8.1</td><td>53.4</td><td>64.9</td><td>80.0</td><td>78.1</td><td>83.4</td>
  </tr>
  <tr>
    <td>MSCOCO_t2i</td><td>59.5</td><td>63.6</td><td>58.3</td><td>53.7</td><td>54.1</td><td>20.7</td><td>68.7</td><td>55.0</td><td>75.7</td><td>78.6</td><td>76.2</td>
  </tr>
  <tr>
    <td>MSCOCO_i2t</td><td>57.7</td><td>62.1</td><td>55.0</td><td>20.3</td><td>40.0</td><td>14.0</td><td>56.7</td><td>59.1</td><td>73.1</td><td>72.4</td><td>73.6</td>
  </tr>
  <tr>
    <td>NIGHTS</td><td>60.4</td><td>66.1</td><td>62.9</td><td>56.5</td><td>58.1</td><td>4.2</td><td>59.4</td><td>58.9</td><td>65.5</td><td>68.3</td><td>68.8</td>
  </tr>
  <tr>
    <td>WebQA</td><td>67.5</td><td>62.1</td><td>58.1</td><td>55.4</td><td>43.0</td><td>17.7</td><td>76.3</td><td>82.9</td><td>87.6</td><td>90.2</td><td>88.1</td>
  </tr>
  <tr>
    <td>FashionIQ</td><td>11.4</td><td>13.8</td><td>20.1</td><td>9.3</td><td>11.2</td><td>2.8</td><td>31.5</td><td>21.6</td><td>16.2</td><td>54.9</td><td>28.6</td>
  </tr>
  <tr>
    <td>Wiki-SS-NQ</td><td>55.0</td><td>44.6</td><td>55.1</td><td>28.7</td><td>18.7</td><td>8.6</td><td>25.4</td><td>58.8</td><td>60.2</td><td>24.9</td><td>65.2</td>
  </tr>
  <tr>
    <td>OVEN</td><td>41.1</td><td>45.0</td><td>56.0</td><td>39.5</td><td>1.6</td><td>5.9</td><td>73.0</td><td>67.6</td><td>56.5</td><td>87.5</td><td>77.3</td>
  </tr>
  <tr>
    <td>EDIS</td><td>81.0</td><td>77.5</td><td>23.6</td><td>54.4</td><td>62.6</td><td>26.8</td><td>59.9</td><td>55.2</td><td>87.8</td><td>65.6</td><td>83.6</td>
  </tr>
  <tr>
    <td><b>Avg.</b></td><td>53.0</td><td>52.3</td><td>31.6</td><td>33.9</td><td>35.4</td><td>11.5</td><td>56.5</td><td>54.7</td><td>67.4</td><td>69.9</td><td>71.0</td>
  </tr>
  <tr>
    <th colspan="13">Visual Grounding (4 tasks)</th>
  </tr>
  <tr>
    <td>MSCOCO</td><td>33.8</td><td>34.5</td><td>46.4</td><td>28.9</td><td>22.1</td><td>10.8</td><td>42.7</td><td>59.0</td><td>80.6</td><td>76.8</td><td>85.0</td>
  </tr>
  <tr>
    <td>RefCOCO</td><td>56.9</td><td>54.2</td><td>70.8</td><td>47.4</td><td>22.8</td><td>11.9</td><td>69.3</td><td>78.9</td><td>88.7</td><td>89.8</td><td>92.7</td>
  </tr>
  <tr>
    <td>RefCOCO-matching</td><td>61.3</td><td>68.3</td><td>50.8</td><td>59.5</td><td>35.6</td><td>38.9</td><td>63.2</td><td>80.8</td><td>84.0</td><td>90.6</td><td>88.9</td>
  </tr>
  <tr>
    <td>Visual7W-pointing</td><td>55.1</td><td>56.3</td><td>70.1</td><td>52.0</td><td>23.4</td><td>14.3</td><td>73.5</td><td>71.2</td><td>90.9</td><td>77.0</td><td>92.3</td>
  </tr>
  <tr>
    <td><b>Avg.</b></td><td>51.8</td><td>53.3</td><td>59.5</td><td>47.0</td><td>26.0</td><td>19.0</td><td>62.2</td><td>72.5</td><td>86.1</td><td>83.6</td><td>89.7</td>
  </tr>
  <tr>
    <th colspan="13">Final Score (36 tasks)</th>
  </tr>
  <tr>
    <td>All IND Avg.</td><td>37.1</td><td>39.3</td><td>32.3</td><td>25.3</td><td>31.0</td><td>14.9</td><td>43.5</td><td>57.2</td><td>67.5</td><td>59.1</td><td>72.4</td>
  </tr>
  <tr>
    <td>All OOD Avg.</td><td>38.7</td><td>40.2</td><td>38.0</td><td>25.1</td><td>23.7</td><td>11.5</td><td>44.3</td><td>60.4</td><td>57.1</td><td>68.0</td><td>66.6</td>
  </tr>
  <tr>
    <td>All Avg.</td><td>37.8</td><td>39.7</td><td>34.8</td><td>25.2</td><td>27.8</td><td>13.3</td><td>44.0</td><td>58.6</td><td>62.9</td><td>64.1</td><td>69.8</td>
  </tr>
</table>

Table 7: Detailed results of zero-shot setting and supervised setting models on each dataset of MMEB (Jiang et al., 2024b).

Reviewer paper summaries: The work presents a comprehensive study of the current literature on MLLM-based embedding models and proposes a multilingual and multimodal MLLM model and a synthetically generated training dataset.
One of the most important contributions to this work is the dataset.
Summary of strengths: 1) Extensive experimental setup with multiple ablation studies. I appreciate the extensive experimental setup, ablation studies and hyperparameter search that was performed.
2) Useful models and datasets to foster research in this field.
3) Deep presentation and study of the state-of-the-art.
Summary of weaknesses: 1) Some technical details are missing.
2) Poor evaluation of the synthetic data generation framework without human validation.
3) Poor discussion of the premises of the work.
4) The multilingual evaluation in task-diversity w.r.t. English with MMEB is quite limited.
5) The authors mention that they will release the source code, model, and data. However, since the dataset is one of this work's most important contributions, it would have been appreciated if they could have provided a small sample during submission.
Comments, suggestions and typos: 1) The authors start from the following premises:
- "the limited labeled multimodal data often hinders embedding performance. Recent approaches have leveraged data synthesis to address this problem, yet the quality of synthetic data remains a critical bottleneck"
- "These vision-language models (VLMs) mostly rely on high-quality human-labeled datasets to achieve robust embedding capabilities. Such datasets suffer from data scarcity because they require high costs of multimodal annotations."

However, the paper never builds on these premises. The authors never mention the total cost of the data generation process in the main text or the appendix.
Furthermore, despite the reliance on synthetically generated data, no humans were involved in manually verifying a subset of instances.
This also makes it unclear how effective the Data Synthesis Framework process is. It would have been nice if humans could have been involved in assessing the quality of the framework.
In addition, the Data Synthesis Framework relies solely on GPT-4o, a proprietary model, rather than an open-source one.
However, I appreciate that some ablation studies were performed on the framework's components.

2) Some technical details that are missing: 
2.1) How the negative mining process was performed for the synthetically generated dataset is unclear. The authors only mention that "Then, for the modality types involving images on the document side (e.g., IT→IT), we use a small embedding model, jina-clip-v2 (Koukounas et al., 2024), to retrieve a similar positive image d+i and a hard negative image di efficiently.". The technical details regarding this process are not in the main text or the appendix. How is the negative image chosen? Additionally, the authors describe how negative mining was performed with MMEB "Each training sample incorporates one hard negative document. Hard negatives are mined for each subset of MMEB using VLM2Vec-LoRA, with the 70th position in the ranking list selected as the hard negative sample.". It is unclear why this description is missing for their dataset.
2.2) In one of their ablations, the authors mention that they studied the effect of different batch sizes and temperature values. However, they never report anywhere which batch size and temperature were used for the experimental setup of Section 4.1. Appendix B.2, which contains the hyperparameters used for fine-tuning, only mentions LoRA rank, learning rate and gradient accumulation steps.
2.3) The authors mention that they used gradient accumulation. However, to my knowledge, the contrastive learning objective doesn't work correctly with gradient accumulation to increase the training batch size, hence why GradCache was introduced [Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. 2021. Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 316–321, Online. Association for Computational Linguistics.]. Is there any reason why the authors didn't use GradCache like it has been done in VLM2Vec?
Overall assessment scores: 3
Metareview score: 3
Metareview: This submission studies how to synthesize multi-modal data to improve multimodal multilingual embeddings. It identify three key criteria for deriving high-quality synthetic data, and it then accordingly synthesize data which the work uses to train a multimodal multilingual E5 model (mmE5). The model outperforms a series of baselines on MMEB and XTD benchmarks.

The authors response to the reviewers is in general satisfactory and there is not major weakness left. Recommend to be accepted to Findings since it has good soundness but relatively lower excitement scores.
Summary of reasons to publish: The three reviewer in general recognize the soundness of the methodological design for data synthesis, are satisfied with the extensive experiments, and acknowledge that the work may benefit the related research and the community.