Universal Multimodal Retrieval with Multimodal LLMs

## Abstract

Abstract State-of-the-art retrieval models typically address a straightforward search scenario, where retrieval tasks are fixed (e.g., finding a passage to answer a specific question) and only a single modality is supported for both queries and retrieved results.
This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs), enabling a broader search scenario, termed universal multimodal retrieval, where multiple modalities and diverse retrieval tasks are accommodated.
To this end, we first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks.
Our empirical results show that the fine-tuned MLLM retriever is capable of understanding challenging queries, composed of both text and image, but underperforms a smaller CLIP retriever in cross-modal retrieval tasks due to modality bias from MLLMs.
To address the issue, we propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers.
Second, we propose to continually fine-tune the universal multimodal retriever to enhance its text retrieval capability while maintaining multimodal retrieval capability.
As a result, our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR, which spans multiple domains and tasks, while also surpassing the state-of-the-art text retrieval model, NV-Embed-v1, on MTEB retrieval benchmark.
Finally, we explore to prompt the off-the-shelf MLLMs as the zero-shot rerankers to refine the ranking of the candidates from the multimodal retriever.
We find that through prompt-and-reranking, MLLMs can further improve multimodal retrieval when the user queries (e.g., text-image composed queries) are more complex and challenging to understand.
These findings also pave the way to advance universal multimodal retrieval in the future.
We release the model weights at: https://huggingface.co/nvidia/MM-Embed .

## 1 Introduction

Information retrieval is crucial for a variety of downstream tasks, such as question answering (Kwiatkowski et al., ), fact-checking (Thorne et al., ), and retrieval-augmented generation (Lewis et al., ).
Existing state-of-the-art retrievers often focus on narrow scenarios.
For example, LLM-based retrievers (Wang et al., ; Lee et al., ; Meng et al., ; Moreira et al., ) are limited to text-to-text retrieval tasks, where both the query and the retrieved results are text-only.
Recent work on multimodal retrieval (Zhang et al., ; Jiang et al., ) focuses on specific tasks and assumes a homogeneous document format. However, in real-world applications, documents and queries often consist of diverse formats or modalities, such as text, images, and interleaved text and images.
To advance information retrieval and support broader search scenarios, this work explores the use of multimodal LLMs (MLLMs; Dai et al., ; Liu et al., ; ) for universal multimodal retrieval, accommodating diverse user-instructed tasks with multimodal queries and documents, as illustrated in Figure 1.

We first explore to fine-tune MLLM-based bi-encoder retrievers with instructions as a guide (Asai et al., ) on 16 multimodal retrieval tasks from M-BIER (Wei et al., ).
We find that MLLM-based retrievers significantly outperform CLIP-based retrievers in the challenging tasks, where interleaved text–image queries are given, such as visual question answering and composed image retrieval (tasks 3 and 7 in Figure 1).
However, MLLM-based retrievers underperform in cross-modal retrieval tasks due to the *modality bias* from MLLMs.
That is, given a text-based query with the instruction to retrieve an image (e.g., task 9 in Figure 1), an MLLM-based retriever tends to retrieve a relevant text-only rather than documents with images, especially when we improve the MLLM-based retriever’s text retrieval capability.
To address the issue, we propose modality-aware hard negative mining in Section 4.1.1 and continual text-to-text retrieval fine-tuning in Section 4.1.2.
Our final retriever, coined MM-Embed, is the first state-of-the-art universal multimodal retriever while maintaining competitive text-to-text retrieval performance across diverse tasks.

Finally, we explore to prompt MLLMs as zero-shot rerankers.
Surprisingly, we find that the zero-shot MLLM-based rerankers can further boost retrieval accuracy in the tasks, where user queries are interleaved text–image and more challenging to understand.
For example, in the composed image retrieval dataset, CIRCO (Baldrati et al., ), the zero-shot rerankers are able to refine the ranked lists and significantly boosts the accuracy (mAP@5) over 7 points from the existing state-of-the-art composed-image retriever (Zhang et al., ) and our universal multimodal retrievers.
This finding indicates that there is still room for improvement in such challenging tasks in order to tackle universal multimodal retrieval.
Also, knowledge distillation from zero-shot or few-shot MLLM-based rerankers to retrievers is a promising direction.

We summarize our contributions as follows: *i)* We present a study on applying MLLMs to universal multimodal retrieval.
*ii)* We are the first to build MLLM-based universal multimodal retrievers.
Notably, our MM-Embed, initialized from the existing best-performing text retriever (NV-Embed-v1; Lee et al., ), not only achieves state-of-the-art results in universal multimodal retrieval benchmark, M-BEIR (Wei et al., ), but also surpasses NV-Embed-v1 in text-to-text retrieval tasks on MTEB.
*iii)* We are the first work to explore prompting MLLMs for zero-shot reranking.
With a zero-shot MLLM-based reranker, we are able to boost the ranking accuracy over 7 points upon state-of-the-art retrievers in the composed image retrieval task, CIRCO (Baldrati et al., ).

We organize the rest of the paper as follows.
We discuss related work in § 2.
We introduce the definition of universal multimodal retrieval in § 3 and present the proposed method in § 4.
We report experiment results in § 5 and conclude the paper in § 6.

Figure: Figure 1: Illustration of universal multimodal retrieval, where diverse tasks with instructions, queries and documents with multimodal formats are supported. In this work, we explore to fine-tune MLLM-based universal multimodal retriever, MM-Embed, and prompt an MLLM for reranking.
Refer to caption: x1.png

## 2 Related Work

##### Instruction-Aware Dense Representation Learning.

Asai et al. () is the first work to identify the implicit search intent behind each retrieval task and propose to fine-tune a retriever to learn diverse retrieval tasks with handwritten task instructions.
Su et al. () and existing state-of-the-art LLM-based text embedding models (Wang et al., ; Meng et al., ; Lee et al., ) adopt this approach to broader tasks beyond text retrieval, such as text classification and clustering.
Recently, Wei et al. () propose a universal multimodal retrieval dataset, M-BEIR, and find that instruction-aware dense retrieval fine-tuning is crucial to tackle universal multimodal retrieval.

##### Vision-Language Models for Multimodal Retrieval.

With the advance of pre-trained vision-language models (Radford et al., ; Li et al., ), research focus shifts from single-modal (Bajaj et al., ; Fu et al., ) to cross-modal (Lin et al., ; Han et al., ; Liu et al., ) or more complex multimodal retrieval tasks (Liu et al., ; Wu et al., ; Baldrati et al., ).
However, the aforementioned tasks assume homogeneous modality for queries and documents, limiting its application.
Liu et al. () take one step further to tackle the retrieval scenario involving candidate pool with heterogeneous modalities but still limit to single retrieval task.

Wei et al. () extend the study to a more general scenario, where retrievers are required to deal with queries, candidate pool in heterogeneous modalities and diverse retrieval tasks.
However, the study is limited to CLIP-based retrievers and ignores important text-to-text retrieval tasks, such as fact checking (Thorne et al., ) and entity retrieval (Hasibi et al., ).
While Koukounas et al. () aim to fine-tune a CLIP-based retriever with both strong text-to-text and multimodal retrieval capability, they only consider simple multimodal retrieval tasks: image-caption retrieval (Young et al., ; Lin et al., ).
Concurrent to our work, Jiang et al. () propose to fine-tune MLLMs on NLI dataset (Bowman et al., ) and demonstrate their transferability to multimodal retrieval.
In this paper, we are the first to study how to fine-tune an MLLM-based universal multimodal retriever while maintaining strong text-to-text retrieval capability.
Also, we are the first to explore prompting MLLMs as zero-shot rerankers in diverse multimodal retrieval tasks.

## 3 Universal Multimodal Retrieval

Following the framework of Lin et al. (), we formulate the task of retrieval as follows: given a query $q$, the goal is to retrieve a ranked list of candidates $\{c_{1},c_{2},\cdots c_{k}\}\in C$ to maximize some ranking metrics, such as nDCG, where $C$ is the collection of documents.
In this work, we borrow the setting of universal multimodal retrieval from Wei et al. (), where user queries and candidates may consist of a text, image or interleaved text–image; i.e., $q\in\{q^{\text{txt}},q^{\text{img}},(q^{\text{txt}},q^{\text{img}})\}$; $c\in\{c^{\text{txt}},c^{\text{img}},(c^{\text{txt}},c^{\text{img}})\}$.
Additionally, there are multiple search intents behind a search query, which can be elaborated by task-specific instructions (Asai et al., ).
For example, in task 1 and 2 of Figure 1, given the same image as a query, the search intent is to find an image caption and similar image, respectively.
Thus, in universal multimodal retrieval, given a multimodal query and task instruction $inst$, we aim to retrieve a list of candidates from a pool of multimodal documents to maximize a specified ranking metric.
Note that we only consider text and image in this work while more modalities, such as audio and video can be included, which we leave for future work.

## 4 Method

In this section, we describe our approach to universal multimodal retrieval by leveraging multimodal LLMs (MLLMs).
In Section 4.1, we first fine-tune an MLLM-based retriever to project multimodal user queries, along with task instructions, into the same semantic space as multimodal documents, enabling $k$-nearest neighbor search (Johnson et al., ).
In Section 4.2, we present our method for using MLLMs to rerank the top-$k$ candidates retrieved by the universal multimodal retriever.

### 4.1 Fine-tuning Multimodal LLMs for Universal Multimodal Retrieval

We fine-tune an MLLM-based retriever parameterized by $\theta$ (i.e., $\eta^{\theta}$) under the guidance of task-specific instructions, aiming to capture the implicit intents behind retrieval tasks.
Specifically, given a user query $q_{i}$ with the specified task instruction $inst_{i}$ and its relevant and negative candidates, $c^{+}_{i}$ and $c^{-}_{i}$, we minimize the InfoNCE loss (Gutmann & Hyvärinen, ):

$$ $\displaystyle NCE=-\frac{1}{|\mathcal{B}|}\sum_{i=1}^{|\mathcal{B}|}\log\frac{ \exp{(\eta^{\theta}(inst_{i},q_{i})\cdot\eta^{\theta}(c_{i}^{+})/\tau)}}{\sum_ {c^{\prime}\in\mathcal{D}_{\mathcal{B}}}{\exp(\eta^{\theta}(inst_{i},q_{i}) \cdot\eta^{\theta}(c^{\prime})/\tau)}},$ (1) $$

where $\mathcal{D}_{\mathcal{B}}=(c^{+}_{1},c^{-}_{1},\cdots,c^{+}_{|\mathcal{B}|},c^
{-}_{|\mathcal{B}|})$ includes all the positive and negative documents for all the queries in the mini batch $\mathcal{B}$, $\eta^{\theta}(\cdot)\in\mathbb{R}^{d}$ is a normalized vector and $\tau$ is the temperature.

#### 4.1.1 Modality-Aware Hard Negative Mining

Prior work (Karpukhin et al., ; Xiong et al., ; de Souza P. Moreira et al., ) has demonstrated that hard negative mining significantly improves representation learning for text-to-text retrieval.
In the previous retrieval setting, where the corpus consists of documents with a homogeneous modality, a document is considered a hard negative if it lacks the required information but is still retrieved by a model.
However, in the scenario of universal multimodal retrieval, where the corpus contains documents involving diverse modalities, the users’ desired modality as specified in task instructions (i.e., text, image or interleaved text–image) should be taken into consideration.
For example, as shown in Figure 1, the first and second users issue the same query along with different instructions, requiring the documents to be in the format of text and image, respectively.
To address this, we propose modality-aware hard negative mining to guide models in retrieving candidates that meet both the users’ information needs and their preferred modality.

Specifically, we first fine-tune an MLLM-based retriever using random negatives; i.e., $\mathcal{D}_{\mathcal{B}}=(c^{+}_{1},\cdots,c^{+}_{|\mathcal{B}|})$.
The fine-tuned model is denoted as $M^{\text{rand}}$.
For each query $q_{i}$ and its associated instruction $inst_{i}$ in the training set, we generate two types of negatives from the top-50 candidates retrieved by $M^{\text{rand}}$:
*i*) negatives with incorrect modality ($C^{1}_{i}$), where the candidate ranks higher than the labeled positive but has a different modality from the desired one, and
*ii*) negatives with unsatisfactory information ($C^{2}_{i}$), where the candidate ranks lower than $k^{\prime}$ but has the same desired modality.
Note that setting $k^{\prime}$ to a small number may include false positives while setting $k^{\prime}$ to large number would make the negative samples too easy.
Thus, in our experiment, following the prior work (Chen et al., ; Lin et al., ), we set $k^{\prime}=45$.
While training, given the query $q_{i}$ with the associated instruction $inst_{i}$, we generate a triplet, $((inst_{i},q_{i}),c_{i}^{+},c_{i}^{-})$, by sampling hard negative $c^{-}_{i}$ from either $C^{1}_{i}$ or $C^{2}_{i}$ with the same probability.
We denote the models fine-tuned with modality-aware hard negatives as $M^{\text{hard}}$.
We refer readers to Fig. 2 in the Appendix for examples of both types of negative samples.

#### 4.1.2 Continual Text-to-Text Retrieval Fine-Tuning

Since text-to-text retrieval remains one of the most commonly used retrieval tasks, we further fine-tune $M^{\text{hard}}$ on diverse public text-to-text retrieval tasks, including MS MARCO (Bajaj et al., ), HotpotQA (Yang et al., ), Natural Question (Kwiatkowski et al., ), PAQ (Lewis et al., ), StackExchange (Stack-Exchange-Community, ), Natural Language Inference (Bowman et al., ), SQuAD (Rajpurkar et al., ), ArguAna (Wachsmuth et al., ), BioASQ (Nentidis et al., ), FiQA (Maia et al., ), and FEVER (Thorne et al., ).
As these datasets do not contain negative samples, we employ the fine-tuned LLM-based retriever (NV-Embed-v1; Lee et al., ) to mine hard negatives in our experiments (see de Souza P. Moreira et al. () for details).

During the continual fine-tuning stage, we uniformly sample triplets from both the universal multimodal and text-to-text retrieval training data.
Note that for each query $q_{i}$ in universal multimodal retrieval training data, we use $M^{\text{hard}}$ to mine the second-type hard negatives $C_{i}^{2}$ again.
Since no first-type hard negatives (i.e., $C_{i}^{1}=\emptyset$) are mined by $M^{\text{hard}}$, we retain the first-type hard negative mined by $M^{\text{rand}}$.

### 4.2 Prompting Multimodal LLMs for Reranking

Prior work (Sun et al., ; Jin et al., ) has demonstrated that instruction fine-tuned LLMs can be prompted to rerank candidates in text-to-text retrieval tasks.
In this work, we prompt LLaVa-Next (Liu et al., ) to further rerank the top-10 retrieved candidates by universal multimodal retrievers.
Following the approach in Nogueira et al. (), we frame the reranking task as a series of true-false questions.
Specifically, given a query and retrieved candidate, we prompt LLaVa-Next to determine whether the retrieved candidate satisfies the given query by answering “True” or “False”.
For example, in the image caption retrieval (task 1 in Figure 1), given an image query, $q^{\text{img}}$, and a retrieved text-based candidate, $c^{\text{txt}}$, we use the below prompt: “$<q^{\text{img}}>$\nCaption:$<c^{\text{txt}}>$\nDoes the above daily life image match the caption? True or False”.
Additionally, in the visual question answering retrieval (task 7 in Figure 1), given a visual question, $<$Qry image$><$Qry text$>$, and a retrieved text-based candidate, $<$Doc text$>$, we use the below prompt: $<$Qry image$>$\nQuestion:$<$Qry text$>$\nAnswer:$<$Doc text$>$\nDoes the answer correctly answer the question? True or False.
We refer readers to Table 13 in the Appendix for the specific prompts used in different multimodal retrieval tasks.

To compute relevance scores, we apply the Softmax operation over the logits of the “True” and “False” tokens, using the probability of the “True” token as the relevance score for reranking.
Our preliminary study in Section 5.3.3 shows that zero-shot MLLM-based rerankers mainly improve the tasks, where queries are interleaved text–image, such as composed image retrieval and visual question answering as shown in the tasks 3, 5,7 and 8 of Figure 1.

## 5 Experiments

### 5.1 Datasets and Models

##### Multimodal Retrieval Dataset.

We evaluate models’ universal multimodal retrieval capability using M-BEIR dataset (Wei et al., ), which is constructed from 10 datasets with 16 diverse multimodal retrieval tasks across 4 domains listed in Table 8 (in the Appendix).(^1^11[https://huggingface.co/datasets/TIGER-Lab/M-BEIR](https://huggingface.co/datasets/TIGER-Lab/M-BEIR))
We train our models on the M-BEIR 1.1M training queries and evaluate models’ effectiveness on the 190K test queries.
Following the global evaluation setting of M-BEIR dataset, for each query, candidates are retrieved from a merged candidate pool of 5.6M multimodal documents spanning all 10 datasets.
We report the averaged Recall@5 (R@5) as retrieval accuracy across all test queries in each dataset, except for Fashion200K and FashionIQ, where we report Recall@10 (R@10).
We refer readers to Wei et al. () for more details on the construction of M-BEIR dataset.

##### Text-to-Text Retrieval Dataset.

While M-BEIR contains WebQA dataset for text-to-text retrieval evaluation, we conduct a more comprehensive text-to-text retrieval evaluation using MTEB dataset (Muennighoff et al., ).
Specifically, we evaluate our models on 15 diverse text retrieval datasets.(^2^22The 15 retrieval datasets in MTEB are derived from public datasets in BEIR (Thakur et al., ), excluding BioASQ, Signal-1M, TREC-NEWS, Robust04.)
Following the established procedure, we report the averaged nDCG@10 across the 15 text retrieval datasets.
Note that unlike in M-BEIR, where candidates are retrieved from a merged pool across all tasks, in the MTEB retrieval tasks, we retrieve candidates from separate corpora for each task.

##### Backbone Model Choices.

In this work, we utilize two representative backbones of vision–language models to build universal multimodal retrievers, CLIP (Radford et al., ) and LLaVa-Next (Liu et al., ).
For CLIP, we initialize from CLIP-large model and employ the best-performing modeling approach from Wei et al. (), denoted as CLIPSF.(^3^33[https://huggingface.co/openai/clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14))
This method fuses input image and text features by separately encoding each input (query or document) image and text into separate vectors, which are then summed to create a fused vector (Liu et al., ).

LLaVa-Next (Liu et al., ) is a multimodal LLM (MLLM), which integrates a CLIP image encoder, LLM and a vision–language MLP projector to align image features to the input embedding space of the LLM.
We use LLaVa-Next with Mistral 7B (Jiang et al., ) as the backbone LLM.(^4^44[https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf))
We experiment with three variants: (1) LLaVa-E: the $<$eos$>$ token embedding is used to aggregate information from the multimodal input, a method commonly employed in prior work for text retrieval (Wang et al., ; Ma et al., );
(2) LLaVa-P: the MLLM is prompted to summarize each multimodal query (or document) input in one word, using embedding for the last token to encode multimodal input;(^5^55We refer readers to Table 12 in the Appendix for the prompt and more detail from the prior work (Zhuang et al., ; Jiang et al., ).)
(3) NV-Embed-v1: The LLM from LLaVa-Next is replaced by the fine-tuned LLM-based text retrieval model NV-Embed-v1 (Lee et al., ) while all other components (i.e., image encoder and vision–language MLP projector) remain unchanged.(^6^66[https://huggingface.co/nvidia/NV-Embed-v1](https://huggingface.co/nvidia/NV-Embed-v1))
Note that the backbone of NV-Embed-v1 is Mistral 7B.
The instructions for LLaVa-E (or NV-Embed-v1) and LLaVa-P are illustrated in Table 11 and 12 (in the Appendix), respectively.
For reranking experiments, we also utilize LLaVa-Next with Mistral 7B and the prompts are listed in Table 13 (in the Appendix).

##### Retriever Training Details.

For each backbone, we start from fine-tuning $M^{\text{rand}}$ with random negatives; i.e., $\mathcal{D}_{\mathcal{B}}=(c^{+}_{1},\cdots,c^{+}_{|\mathcal{B}|})$ in Eq. 1.
The fine-tuned model is denoted $M^{\text{rand}}$.
For CLIP backbone, following (Wei et al., ), we fine-tune CLIPSF for 20 epochs with learning rate $1e-5$.
For LLaVa-Next backbone, we fine-tune models for 2 epochs with learning rate $1e-4$.
Note that for LLaVa-Next backbone, we only fine-tune the vision–language projector and LoRA ($r=8,\alpha=64$) added on the language model.
At the stage of fine-tuning $M^{\text{hard}}$ with hard negatives, we mine the two types of hard negatives following Section 4.1.1 using each retriever.
Then, we fine-tune each retriever using its own mined hard negatives with the same training procedure as the first stage; i.e., $\mathcal{D}_{\mathcal{B}}=(c^{+}_{1},c^{-}_{1},\cdots,c^{+}_{|\mathcal{B}|},c^
{-}_{|\mathcal{B}|})$ in Eq. 1.
We fine-tune models with the batch size of $128\times 8$ and $64\times 8$ when using random and hard negatives, respectively.
When GPU memory is not enough for the designated batch size, we use gradient accumulation.
Note that we initialize retriever using the pre-trained model rather than $M^{\text{rand}}$.
We denote the models fine-tuned with random and hard negatives $M^{\text{rand}}(\cdot)$ and $M^{\text{hard}}(\cdot)$, respectively.
We refer readers to the Appendix A.1 for more detail.

To enhance text-to-text retrieval capability, we continuously fine-tune $M^{\text{hard}}$(NV-Embed-v1) with learning rate $2e-5$ using the mixture of training data from M-BEIR and public text retrieval datasets aforementioned in Section 4.1.2 for 4.5K steps.
The final model is coined MM-Embed.

**Table 1: Main results. Following Wei et al. (), we report R@5 for all the datasets, except for Fashion200K and FashionIQ, where we report R@10. The tasks of single-modal and multi-modal queries denote tasks 1–5 and 6–8, respectively. For MTEB text retrieval (Muennighoff et al., ), we report nDCG@10 averaged from 15 retrieval tasks (detailed in Appendix Table 9).**
| Task | Dataset | $M^{\text{rand}}$ | $M^{\text{hard}}$ | MM-Embed |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIPSF | LLaVa-E | LLaVa-P | NV-Embed-v1 | CLIPSF | LLaVa-P | NV-Embed-v1 |  |  |  |
| 1. $q^{\text{txt}}\rightarrow c^{\text{img}}$ | VisualNews | 43.8 | 33.2 | 34.2 | 32.1 | 42.7 | 39.7 | 41.1 | 41.0 |
| MSCOCO | 72.0 | 69.3 | 70.8 | 64.6 | 69.2 | 73.8 | 72.7 | 71.3 |  |
| Fashion200K | 16.4 | 13.5 | 13.3 | 10.4 | 19.7 | 17.4 | 18.6 | 17.1 |  |
| 2. $q^{\text{txt}}\rightarrow c^{\text{txt}}$ | WebQA | 83.2 | 88.6 | 88.8 | 92.1 | 88.2 | 93.6 | 95.6 | 95.9 |
| 3. $q^{\text{txt}}\rightarrow(c^{\text{img}},c^{\text{txt}})$ | EDIS | 46.5 | 55.9 | 56.6 | 55.1 | 54.2 | 68.8 | 69.8 | 68.8 |
| WebQA | 76.0 | 80.3 | 81.6 | 81.3 | 80.1 | 84.9 | 84.8 | 85.0 |  |
| 4. $q^{\text{img}}\rightarrow c^{\text{txt}}$ | VisualNews | 39.5 | 32.4 | 33.3 | 30.4 | 40.6 | 39.4 | 41.4 | 41.3 |
| MSCOCO | 91.0 | 91.8 | 92.2 | 90.3 | 88.5 | 89.5 | 88.9 | 90.1 |  |
| Fashion200K | 17.2 | 13.9 | 14.7 | 13.2 | 20.0 | 17.5 | 19.9 | 18.4 |  |
| 5. $q^{\text{img}}\rightarrow c^{\text{img}}$ | NIGHTS | 31.6 | 31.8 | 30.7 | 30.4 | 31.9 | 31.8 | 31.1 | 32.4 |
| 6. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{txt}}$ | OVEN | 40.4 | 37.9 | 39.1 | 36.3 | 40.9 | 42.9 | 42.6 | 42.1 |
| InfoSeek | 26.1 | 31.0 | 32.9 | 33.3 | 27.6 | 37.2 | 35.8 | 42.3 |  |
| 7. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{img}}$ | FashionIQ | 24.2 | 27.4 | 27.0 | 26.0 | 21.7 | 25.8 | 26.6 | 25.7 |
| CIRR | 43.2 | 48.1 | 45.4 | 45.3 | 38.3 | 49.5 | 50.8 | 50.0 |  |
| 8. $(q^{\text{img}},q^{\text{txt}})\rightarrow(c^{\text{img}},c^{\text{txt}})$ | OVEN | 60.9 | 61.6 | 62.6 | 61.7 | 61.6 | 63.9 | 63.5 | 64.1 |
| InfoSeek | 45.9 | 50.3 | 50.0 | 53.4 | 47.1 | 54.4 | 53.5 | 57.7 |  |
| M-BEIR Avg. | All | 47.4 | 47.9 | 48.3 | 47.2 | 48.3 | 51.9 | 52.3 | 52.7 |
| Single-modal Qry | 51.7 | 51.0 | 51.6 | 50.0 | 53.5 | 55.6 | 56.4 | 56.1 |  |
| Multi-modal Qry | 40.1 | 42.7 | 42.8 | 42.7 | 39.5 | 45.6 | 45.5 | 47.0 |  |
| MTEB Text Retrieval Avg. | - | - | - | - | - | 46.4 | 49.7 | 60.3^$\ast$ |  |

### 5.2 Main Results

##### Universal Multimodal Retrieval.

Table 1 reports the retrieval accuracy of different retrievers.
In M-BEIR evaluation, we observe that when fine-tuning with random negatives, LLaVa-P achieves the highest overall retrieval effectiveness.
This result indicates that LLaVa-P effectively aggregates multimodal input information into a single word representation.
While MLLM-based retrievers outperform CLIPSF on tasks involving multi-modal queries, they still lag behind CLIPSF on tasks with single-modal queries, especially in cross-modality retrieval; i.e., tasks 1 and 4.
In addition, NV-Embed-v1 reaches the best text-to-text retrieval accuracy on WebQA task2.

Observing from the models fine-tuned with hard negatives, MLLM-based retrievers show significant retrieval accuracy improvements, particularly in tasks involving single-modal queries.
On the other hand, CLIPSF does not show similar improvement.
This could attribute to the fact that CLIP has been well pre-trained for cross-modal retrieval whereas MLLM-based retrievers, fine-tuned with contrastive learning objective for only 2 epochs, may still be underfitting.
Fine-tuning with hard negatives accelerates contrastive learning of MLLM-based retrievers.

**Table 2: Retrieval analysis on MSCOCO. M.A.@1 denotes the modality accuracy of the top-1 candidate.**
| Task | Metric | $M^{\text{rand}}$ | $M^{\text{hard}}$ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIPSF | LLaVa-E | LLaVa-P | NV-Embed-v1 | CLIPSF | LLaVa-P | NV-Embed-v1 |  |  |
| 1. | R@1 | 42.6 | 33.9 | 41.7 | 14.1 | 45.8 | 50.7 | 49.8 |
| R@5 | 72.0 | 69.3 | 70.8 | 64.6 | 69.2 | 73.8 | 72.7 |  |
| M.A.@1 | 92.6 | 79.9 | 91.0 | 42.1 | 98.3 | 100.0 | 100.0 |  |
| 4. | R@1 | 72.3 | 73.0 | 73.4 | 69.3 | 63.8 | 72.7 | 72.4 |
| R@5 | 91.0 | 91.8 | 92.2 | 90.3 | 88.5 | 89.5 | 88.9 |  |
| M.A.@1 | 98.7 | 99.2 | 99.8 | 96.3 | 94.2 | 100.0 | 100.0 |  |

Table 2 reveals another factor contributing to the lower retrieval accuracy of MLLM-based retrievers for single-modal queries: text retrieval bias.
This issue is particularly obvious for NV-Embed-v1.
We compare models’ retrieval accuracy on text–image and image–text retrieval (tasks 1 and 4) on MSCOCO.
The comparison shows that $M^{\text{rand}}$(LLaVa-E) and $M^{\text{rand}}$(NV-Embed-v1) exhibit significant lower modality accuracy (M.A.@1) than $M^{\text{rand}}$(CLIPSF) in the text-to-image retrieval task.
Most erroneous top-1 retrieved candidates from the MLLM-based retrievers are relevant texts rather than images (see Figure 2 in the Appendix).
This result indicates that MLLM-based retrievers have a bias toward relevant text rather than images.
This issue can be mitigated by our proposed modality-aware hard negative mining.

Finally, we observe that $M^{\text{hard}}$(NV-Embed-v1) outperforms $M^{\text{hard}}$(LLaVa-P) in text-to-text retrieval tasks (i.e., WebQA task 2 and MTEB); however, compared to the original NV-Embed-v1 (Lee et al., ), the score on MTEB retrieval tasks drops almost 10 points.
After continual fine-tuning (detailed in Section 4.1.2), the final model, MM-Embed, not only surpasses NV-Embed-v1 in MTEB but also maintains strong multimodal retrieval capability.
We attribute the improvement in text-to-text retrieval to the effective hard negatives mined by NV-Embed-v1 aforementioned in Section 4.1.2.
Notably, continual fine-tuning significantly enhances multimodal retrieval performance in InfoSeek (col 8 vs 7 in Table 1), highlighting its effectiveness in improving the model’s ability to handle knowledge-intensive multimodal retrieval tasks.

**Table 3: Experiments of zero-shot reranking on tasks 6–8 from M-BEIR.**
| Task | Dataset | $M^{\text{hard}}$ (NV-Embed-v1) | MM-Embed |  |  |
| --- | --- | --- | --- | --- | --- |
| Ret. | Rerank | Ret. | Rerank |  |  |
| 6. | OVEN | 42.6 | 44.3 | 42.1 | 43.5 |
| InfoSeek | 35.8 | 37.1 | 42.3 | 43.1 |  |
| 7. | FashionIQ | 26.6 | 20.0 | 25.7 | 19.0 |
| CIRR | 50.8 | 48.6 | 50.0 | 48.2 |  |
| 8. | OVEN | 63.5 | 65.8 | 64.1 | 65.9 |
| InfoSeek | 53.5 | 54.5 | 57.7 | 57.3 |  |

##### Zero-Shot Reranking.

Table 4 reports the reranked results from the top-10 retrieved candidates of $M^{\text{hard}}$(NV-Embed-v1) and MM-Embed on the tasks involving multi-modal queries.
We observe accuracy improvements in visual question answering retrieval tasks (i.e., OVEN and InfoSeek), but no improvement on composed image retrieval tasks (i.e., FashionIQ and CIRR).
However, as shown in Table 8 (in the Appendix), compared to OVEN and InfoSeek, FashionIQ and CIRR only have one relevance label per query.
We hypothesize that there may be additional relevant positives that are not labeled.
We refer readers to Figure 3 in the Appendix for case studies.

We conduct experiments on the composed image retrieval dataset with high-quality human annotations, CIRCO (Baldrati et al., ) validation set, consisting of 219 queries and 123K candidates in total.
On average, 4.2 positives are labeled by humans per query.
Table 4 reports mAP@5 for various retrievers and their reranking results.
We directly use the models and code provided by the authors to get the results of MagicLens (Zhang et al., )(^7^77[https://github.com/google-deepmind/magiclens](https://github.com/google-deepmind/magiclens)) and E5-V (Jiang et al., )(^8^88[https://github.com/kongds/E5-V](https://github.com/kongds/E5-V)) retrievers.
For our retrievers fine-tuned on M-BEIR, $M^{\text{rand}}$(CLIPSF), $M^{\text{hard}}$(LLaVa-P), $M^{\text{hard}}$(NV-Embed-v1) and MM-Embed, we directly use the same instructions as CIRR in M-BEIR for query encoding.
We first observe that our MLLM-based retrievers outperform MagicLens and E5-V.
More importantly, reranking upon the top-10 retrieved candidates from the different retrievers significantly improves mAP@5 by at least 7 points.
The result demonstrates the effectiveness of prompting an MLLM as a reranker in composed image retrieval tasks.

### 5.3 Ablation Studies

#### 5.3.1 Is Fine-Tuning with Instruction Necessary?

**Table 5: Ablation study on fine-tuning NV-Embed-v1 w/o (✗) and w/ (✓) instructions.**
| Task | Dataset | zero-shot | fine-tuning |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP | LLaVa-P | NV-Embed-v1 | NV-Embed-v1 |  |  |  |  |
| ✗ | ✗ | ✗ | ✓ | ✗ | ✓ |  |  |
| 1. $q^{\text{txt}}\rightarrow c^{\text{img}}$ | VisualNews | 40.9 | 11.7 | 15.3 | 17.4 | 33.1 | 38.7 |
| MSCOCO | 55.4 | 58.1 | 64.2 | 59.9 | 76.7 | 82.8 |  |
| Fashion200K | 8.9 | 2.4 | 4.2 | 3.2 | 12.3 | 15.6 |  |
| 4. $q^{\text{img}}\rightarrow c^{\text{txt}}$ | VisualNews | 42.0 | 6.3 | 6.5 | 5.9 | 29.3 | 37.2 |
| MSCOCO | 79.6 | 66.8 | 70.6 | 68.2 | 88.9 | 93.0 |  |
| Fashion200K | 7.7 | 2.9 | 4.0 | 3.6 | 12.0 | 16.8 |  |
| 5. $q^{\text{img}}\rightarrow c^{\text{img}}$ | NIGHTS | 25.4 | 28.4 | 29.3 | 27.7 | 31.6 | 30.9 |

We fine-tune NV-Embed-v1 with random negatives on the M-BEIR subtasks listed in Table 5 and evaluate models’ retrieval accuracy on the development queries from each subtask.
Note that, for simplicity, we encode only the corpus specific to each dataset, containing documents of the targeted modality.
For example, when evaluating retrieval accuracy for VisualNews task 1, we encode the 542K images from VisualNews (see Table 8 in the Appendix) as the index rather than the entire 5.6M documents from M-BEIR.
We also report CLIP and LLaVa-P (w/o instruction) zero-shot retrieval effectiveness as a reference point.(^9^99We follow Jiang et al. () to prompt LLaVa-Next to output one word embedding for each query and document. i.e., $<$txt$>$\nSummary above sentence in one word:; $<$img$>$\nSummary above image in one word:.)

From Table 5, we observe that NV-Embed-v1, as a zero-shot MLLM-based retriever, outperforms LLaVa-P and even competes CLIP in the tasks in Miscellaneous domain (i.e., MSCOCO and NIGHTS).
This result indicates that a fine-tuned MLLM-based text retriever is capable to perform multimodal retrieval tasks (same finding in (Jiang et al., )).
Although incorporating task instructions with queries degrades the retrieval effectiveness (col 4 vs 3), the model fine-tuned with instructions significantly outperforms the one fine-tuned without instructions (col 6 vs 5).
This indicates that task instructions can help elicit models’ task- or domain-specific knowledge for diverse multimodal retrieval tasks.

#### 5.3.2 Effectiveness of Continual text-to-text retrieval Fine-Tuning

In this section, we study the best strategy to enhance models’ capabilities in both multimodal and text-to-text retrieval.
We begin by fine-tuning NV-Embed-v1 on both training data for universal multimodal retrieval and text-to-text retrieval (detailed in Section 4.1.2) for 2K steps.
As shown in Table 6, joint fine-tuning for both tasks allows the model to maintain its text retrieval capability (row 3 vs 1), although it results in a drop of over 2 points in multimodal retrieval accuracy (row 3 vs 2).
In contrast, consciously fine-tuning $M^{\text{hard}}$(NV-Embed-v1) for addition 2K steps significantly boosts its text-to-text retrieval capability with a slight drop of 0.8 points in multimodal retrieval (row 5 vs 4).(^10^1010Note that MM-Embed in Table 1 is fine-tuned with the same condition with total 4.5K steps.)

**Table 6: Ablation study to enhance model’s text-to-text retrieval capability.**
| Initialization | Training data | M-BEIR$\ast$ | BEIR$\ast$ |  |
| --- | --- | --- | --- | --- |
| Multimodal | Text-to-Text |  |  |  |
| NV-Embed-v1 | - | - | - | 62.9 |
| ✓ | ✗ | 54.3 | 51.7 |  |
| ✓ | ✓ | 52.2 | 63.0 |  |
| $M^{\text{hard}}$ (NV-Embed-v1) | - | - | 56.4 | 51.7 |
| ✓ | ✓ | 55.6 | 63.1 |  |

This experiment shows that continuously fine-tuning a multimodal retriever to enhance its text-to-text retrieval is more effective than fine-tuning a retriever on all the retrieval tasks simultaneously.
This finding suggests that a more optimized curriculum learning strategy (Bengio et al., ) could further improve performance in universal multimodal retrieval, a direction we leave for future work.

#### 5.3.3 Study on Prompting MLLMs for Reranking

**Table 7: Reranking study on top-10 retrieved candidates from $M^{\text{rand}}$(CLIPSF) on M-BEIR development query set.**
| Task | Dataset | Ret. | Rerank |  |
| --- | --- | --- | --- | --- |
| 7B | 34B |  |  |  |
| 1. $q^{\text{txt}}\rightarrow c^{\text{img}}$ | VisualNews | 44.2 | 38.8 | 42.5 |
| MSCOCO | 72.0 | 68.0 | 69.7 |  |
| Fashion200K | 17.8 | 14.7 | 15.6 |  |
| 2. $q^{\text{txt}}\rightarrow c^{\text{txt}}$ | WebQA | 78.2 | 79.2 | 82.9 |
| 3. $q^{\text{txt}}\rightarrow(c^{\text{img}},c^{\text{txt}})$ | EDIS | 48.3 | 46.5 | 47.4 |
| WebQA | 78.2 | 67.7 | 68.3 |  |
| 4. $q^{\text{img}}\rightarrow c^{\text{txt}}$ | VisualNews | 37.4 | 29.3 | 29.8 |
| MSCOCO | 91.0 | 87.3 | 89.0 |  |
| Fashion200K | 17.3 | 9.9 | 12.0 |  |
| 5. $q^{\text{img}}\rightarrow c^{\text{img}}$ | NIGHTS | 32.1 | 29.4 | 32.7 |
| 6. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{txt}}$ | OVEN | 40.6 | 43.2 | 43.7 |
| InfoSeek | 25.6 | 28.4 | 29.0 |  |
| 7. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{img}}$ | FashionIQ | 32.5 | 21.5 | 23.4 |
| CIRR | 52.4 | 54.1 | 54.2 |  |
| 8. $(q^{\text{img}},q^{\text{txt}})\rightarrow(c^{\text{img}},c^{\text{txt}})$ | OVEN | 60.6 | 63.8 | 63.7 |
| InfoSeek | 45.3 | 48.7 | 50.5 |  |

In this section, we study the reranking effectiveness of MLLMs on all the tasks in M-BEIR dataset.
Specifically, for each development query, we rerank the top-10 retrieved candidates from $M^{\text{rand}}$(CLIPSF).
As shown in Table 7, prompting LLaVa-Next for reranking further boosts the ranking accuracy in tasks 6–8, which involve multimodal queries (except for FashionIQ).
However, the reranking degrades accuracy in tasks 1–5 which involve single-modal queries (except for WebQA task 2).
This trend persists even after scaling the reranker from 7B to 34B (col 3, 2 vs 1).(^11^1111In the experiment, we use the model from [https://huggingface.co/llava-hf/llava-v1.6-34b-hf](https://huggingface.co/llava-hf/llava-v1.6-34b-hf))
We hypothesize that it is challenging for bi-encoder models to encode multimodal queries, such as visual question answering and composed image retrieval.
Prompting an MLLM as a reranker in a zero-shot or few-shot manner, or distilling the reranked results into a bi-encoder retriever is a promising solution.

## 6 Conclusion and Future Work

In this paper, we present techniques for advancing information retrieval with multimodal large language models (MLLMs).
We first study fine-tuning MLLM-based retrievers to tackle a general information retrieval scenario: universal multimodal retrieval, where models are required to deal with diverse retrieval tasks, multimodal queries and documents.
Our study shows that MLLM-based retrievers exhibit *modality bias* in cross-modal retrieval tasks compared to CLIP-based retrievers.
To address the issue, we propose modality-aware hard negative mining, which significantly improves our MLLM-based retrievers’ accuracy by 5 points in M-BEIR dataset, a benchmark for universal multimodal retrieval.
Additionally, with our proposed continual fine-tuning, our MLLM-based retriever, MM-Embed, is the first model to yield state-of-the-art retrieval accuracy in universal multimodal retrieval tasks while maintaining strong text-to-text retrieval capability (ranked top-5 on MTEB retrieval task leaderboard).
Finally, we explore to prompt MLLMs as reranker in M-BEIR tasks.
We find that MLLMs can be used as zero-shot rerankers to further boost retrieval accuracy in the challenging tasks, which require the understanding of multimodal queries, such as visual question answering and composed image retrieval.
For example, our zero-shot MLLM-based reranker improves the retrieval accuracy upon the state-of-the-art retrievers by over 7 points in CIRCO.

Our work also suggests two promising future directions: (1) Distilling our MLLM-based retriever, MM-Embed, to smaller multimodal retrievers, such as CLIP (Radford et al., ) or BLIP (Li et al., ); (2) Distilling MLLM-based reranker to retriever to further improve its retrieval capability in tasks involving multimodal queries.
In addition, recent work (Ma et al., ; Faysse et al., ) has demonstrated that MLLMs can be fine-tuned to tackle visual document retrieval tasks, which could be integrated into universal multimodal retrieval.

## Appendix A Appendix

**Table 8: M-BEIR dataset statistics.**
| Task | Dataset | Domain | # Query | # Relevance / Query | # Candid. |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Train | Dev | Test | Train | Dev | Test |  |  |  |  |
| 1. $q^{\text{txt}}\rightarrow c^{\text{img}}$ | VisualNews (Liu et al., ) | News | 99K | 20K | 20K | 1.0 | 1.0 | 1.0 | 542K |
| MSCOCO (Lin et al., ) | Misc. | 100K | 24.8K | 24.8K | 1.0 | 1.0 | 1.0 | 5K |  |
| Fashion200K (Han et al., ) | Fashion | 15K | 1.7K | 1.7K | 3.3 | 3.1 | 2.8 | 201K |  |
| 2. $q^{\text{txt}}\rightarrow c^{\text{txt}}$ | WebQA (Chang et al., ) | Wiki | 16K | 1.7K | 2.4K | 2.0 | 2.0 | 2.0 | 544K |
| 3. $q^{\text{txt}}\rightarrow(c^{\text{img}},c^{\text{txt}})$ | EDIS (Liu et al., ) | News | 26K | 3.2K | 3.2K | 2.6 | 2.6 | 2.6 | 1M |
| WebQA (Chang et al., ) | Wiki | 16K | 1.7K | 2.4K | 1.4 | 1.4 | 1.4 | 544K |  |
| 4. $q^{\text{img}}\rightarrow c^{\text{txt}}$ | VisualNews (Liu et al., ) | News | 100K | 20K | 20K | 1.0 | 1.0 | 1.0 | 537K |
| MSCOCO (Lin et al., ) | Misc. | 113K | 5K | 5K | 5.0 | 5.0 | 5.0 | 25K |  |
| Fashion200K (Han et al., ) | Fashion | 15K | 4.8K | 4.8K | 1.0 | 1.0 | 1.0 | 61K |  |
| 5. $q^{\text{img}}\rightarrow c^{\text{img}}$ | NIGHTS (Fu et al., ) | Misc. | 16K | 2K | 2K | 1.0 | 1.0 | 1.0 | 40K |
| 6. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{txt}}$ | OVEN (Hu et al., ) | Wiki | 150K | 50K | 50K | 8.5 | 10.0 | 9.9 | 676K |
| InfoSeek (Chen et al., ) | Wiki | 141K | 11K | 11K | 6.8 | 6.7 | 6.5 | 611K |  |
| 7. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{img}}$ | FashionIQ (Wu et al., ) | Fashion | 16K | 2K | 6K | 1.0 | 1.0 | 1.0 | 74K |
| CIRR (Liu et al., ) | Misc. | 26K | 2K | 4K | 1.0 | 1.0 | 1.0 | 21K |  |
| 8. $(q^{\text{img}},q^{\text{txt}})\rightarrow(c^{\text{img}},c^{\text{txt}})$ | OVEN (Hu et al., ) | Wiki | 157K | 14.7K | 14.7K | 17.8 | 17.5 | 17.7 | 335K |
| InfoSeek (Chen et al., ) | Wiki | 143K | 17.6K | 17.6K | 9.1 | 7.5 | 7.5 | 481K |  |
| M-BEIR (Wei et al., ) | 4 domains | 1.1M | 182K | 190K | 6.5 | 5.9 | 5.7 | 5.6M |  |

**Table 9: Detailed results on MTEB retrieval tasks.**
| Model | AA | CF | CQ | DB | Fe | FQ | HQ | MS | NF | NQ | Qu | SD | SF | T2 | TC | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NV-Embed-v1 (Lee et al., ) | 68.2 | 34.7 | 50.5 | 48.3 | 87.8 | 63.1 | 79.9 | 46.5 | 38.0 | 71.2 | 89.2 | 20.2 | 78.4 | 28.4 | 85.9 | 59.4 |
| $M^{\text{hard}}$ (LLaVa-P) | 38.6 | 20.4 | 38.0 | 36.9 | 78.1 | 36.2 | 61.2 | 23.2 | 35.1 | 45.1 | 86.1 | 19.2 | 72.7 | 27.7 | 77.2 | 46.4 |
| $M^{\text{hard}}$ (NV-Embed-v1) | 37.2 | 30.8 | 44.0 | 44.3 | 86.4 | 45.5 | 70.6 | 34.2 | 37.4 | 49.7 | 86.9 | 13.9 | 64.1 | 23.5 | 76.7 | 49.7 |
| MM-Embed | 69.0 | 39.3 | 49.7 | 50.6 | 92.6 | 60.1 | 81.4 | 45.1 | 40.5 | 70.6 | 88.7 | 21.8 | 78.3 | 31.1 | 85.4 | 60.3 |

### A.1 Implementation Details

We implement our training and inference using Tevatron (Gao et al., ).
For CLIP-based retrievers, we follow all the settings from Wei et al. ().
For MLLM-based retriever, we fine-tune models with DeepSpeed Zero 2 (Rajbhandari et al., ) and gradient checkpointing.
During fine-tuning on M-BEIR training data, we set maximum length for queries and documents to 128.
While continual fine-tuning on both M-BEIR and text-to-text retrieval training data, we set maximum length for queries and documents to 128 and 512, respectively.
All fine-tuning are conducted on 8$\times$80GB A100 GPUs.
Note that image input only occupies single token length after being tokenized; however, each image will be converted to multiple image tokens.
Thus, the actual input length to MLLM is longer than the maximum length we set.
To speed fine-tuning and inference for MLLM-based retrievers, we only use the global image patches, which occupy 576 (24$\times$24) image tokens.

### A.2 Baseline Reproducing

**Table 10: A comparison of $M^{\text{rand}}$(CLIPSF) fine-tuned by us and Wei et al. ().**
| Task | Dataset | $M^{\text{rand}}$(CLIPSF) |  |
| --- | --- | --- | --- |
| Wei et al. () | Ours |  |  |
| M-BEIR Avg. | All | 47.4 | 47.4 |
| Single-modal Qry | 52.5 | 51.7 |  |
| multi-modal Qry | 39.1 | 40.1 |  |

Since we implement our fine-tuning and inference following the setting from Wei et al. (), our fine-tuned $M^{\text{rand}}$(CLIPSF) should be equal to CLIPSFfrom Wei et al. ().
In Table 10, we compare the results from our fine-tuned $M^{\text{rand}}$(CLIPSF) and the checkpoint provided by the authors.(^12^1212[https://huggingface.co/TIGER-Lab/UniIR/blob/main/checkpoint/CLIP_SF/clip_sf_large.pth](https://huggingface.co/TIGER-Lab/UniIR/blob/main/checkpoint/CLIP_SF/clip_sf_large.pth))

**Table 11: NV-Embed-v1 (and LLaVa-E) instructions for M-BEIR and MTEB, which are from Wei et al. () and Lee et al. (), respectively. For all the candidates, we use the prompt to generate the embedding: $<c^{\text{img}}>$\n$<c^{\text{txt}}><$eos$>$.**
| Task | Dataset | M-BEIR task instruction |
| --- | --- | --- |
| 1. $q^{\text{txt}}\rightarrow c^{\text{img}}$ | VisualNews | Identify the news-related image in line with the described event.\nQuery: $<q^{\text{txt}}><$eos$>$ |
| MSCOCO | Find me an everyday image that matches the given caption.\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| Fashion200K | Based on the following fashion description, retrieve the best matching image.\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| 2. $q^{\text{txt}}\rightarrow c^{\text{txt}}$ | WebQA | Retrieve passages from Wikipedia that provide answers to the following question.\nQuery: $<q^{\text{txt}}><$eos$>$ |
| 3. $q^{\text{txt}}\rightarrow(c^{\text{img}},c^{\text{txt}})$ | EDIS | Find a news image that matches the provided caption.\nQuery: $<q^{\text{txt}}><$eos$>$ |
| WebQA | Find a Wikipedia image that answers this question.\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| 4. $q^{\text{img}}\rightarrow c^{\text{txt}}$ | VisualNews | Find a caption for the news in the given photo.\nQuery: $<q^{\text{img}}><$eos$>$ |
| MSCOCO | Find an image caption describing the following everyday image.\nQuery: $<q^{\text{img}}><$eos$>$ |  |
| Fashion200K | Find a product description for the fashion item in the image.\nQuery: $<q^{\text{img}}><$eos$>$ |  |
| 5. $q^{\text{img}}\rightarrow c^{\text{img}}$ | NIGHTS | Find a day-to-day image that looks similar to the provided image.\nQuery: $<q^{\text{img}}><$eos$>$ |
| 6. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{txt}}$ | OVEN | Retrieve a Wikipedia paragraph that provides an answer to the given query about the image.\nQuery: $<q^{\text{img}}>$\n$<q^{\text{img}}>$$<$eos$>$ |
| InfoSeek | Retrieve a Wikipedia paragraph that provides an answer to the given query about the image.\nQuery: $<q^{\text{img}}>$\n$<q^{\text{img}}>$$<$eos$>$ |  |
| 7. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{img}}$ | FashionIQ | Find a fashion image that aligns with the reference image and style note.\nQuery: $<q^{\text{img}}>$\n$<q^{\text{img}}>$$<$eos$>$ |
| CIRR | Retrieve a day-to-day image that aligns with the modification instructions of the provided image.\nQuery: $<q^{\text{img}}>$\n$<q^{\text{img}}>$$<$eos$>$ |  |
| 8. $(q^{\text{img}},q^{\text{txt}})\rightarrow(c^{\text{img}},c^{\text{txt}})$ | OVEN | Retrieve a Wikipedia image-description pair that provides evidence for the question of this image.\nQuery: $<q^{\text{img}}>$\n$<q^{\text{img}}>$$<$eos$>$ |
| InfoSeek | Retrieve a Wikipedia image-description pair that provides evidence for the question of this image.\nQuery: $<q^{\text{img}}>$\n$<q^{\text{img}}>$$<$eos$>$ |  |
| Task | Dataset | MTEB task instruction |
| 9. $q^{\text{txt}}\rightarrow c^{\text{txt}}$ | ArguAna | Given a claim, find documents that refute the claim\nQuery: $<q^{\text{txt}}><$eos$>$ |
| Climate-FEVER | Given a claim about climate change, retrieve documents that support or refute the claim\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| CQADupStack | Given a question, retrieve detailed question descriptions from StackExchange that are duplicates to the given question\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| DBPedia | Given a query, retrieve relevant entity descriptions from DBPedia\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| FEVER | Given a claim, retrieve documents that support or refute the claim\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| FiQA | Given a financial question, retrieve user replies that best answer the question\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| HotpotQA | Given a multi-hop question, retrieve documents that can help answer the question\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| MSMARCO | Given a web search query, retrieve relevant passages that answer the query\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| NFCorpus | Given a question, retrieve relevant documents that best answer the question\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| Natural Questions | Given a question, retrieve Wikipedia passages that answer the question\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| Quora | Find questions that have the same meaning as the input question\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| SCIDOCS | Given a scientific paper title, retrieve paper abstracts that are cited by the given paper\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| SciFact | Given a scientific claim, retrieve documents that support or refute the claim\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| Touch´e-2020 | Given a question, retrieve detailed and persuasive arguments that answer the question\nQuery: $<q^{\text{txt}}><$eos$>$ |  |
| TREC-COVID | Given a query on COVID-19, retrieve documents that answer the query\nQuery: $<q^{\text{txt}}><$eos$>$ |  |

**Table 12: LLaVa-P instructions for M-BEIR and MTEB. [image], [text] and [image,text] are used to inform LLaVa-P the user desired modality. For all the candidates, we use the prompt to generate the embedding: $<c^{\text{img}}>$\n$<c^{\text{txt}}>$\nDescribe the above in one word:**
| Task | Dataset | M-BEIR task instruction |
| --- | --- | --- |
| 1. $q^{\text{txt}}\rightarrow c^{\text{img}}$ | VisualNews | [image] $<q^{\text{txt}}>$\nDescribe the news-related caption in one word: |
| MSCOCO | [image] $<q^{\text{txt}}>$\nDescribe the everyday caption in one word: |  |
| Fashion200K | [image] $<q^{\text{txt}}>$\nDescribe the fashion description in one word: |  |
| 2. $q^{\text{txt}}\rightarrow c^{\text{txt}}$ | WebQA | [text] $<q^{\text{txt}}>$\nAnswer the question using Wikipedia in one word: |
| 3. $q^{\text{txt}}\rightarrow(c^{\text{img}},c^{\text{txt}})$ | EDIS | [image,text] $<q^{\text{txt}}>$\nDescribe the news-related caption in one word: |
| WebQA | [image,text] $<q^{\text{txt}}>$\nAnswer the question using Wikipedia in one word: |  |
| 4. $q^{\text{img}}\rightarrow c^{\text{txt}}$ | VisualNews | [text] $<q^{\text{img}}>$\nDescribe the news-related image in one word: |
| MSCOCO | [text] $<q^{\text{img}}>$\nDescribe the everyday image in one word: |  |
| Fashion200K | [text] $<q^{\text{img}}>$\nDescribe the fashion image in one word: |  |
| 5. $q^{\text{img}}\rightarrow c^{\text{img}}$ | NIGHTS | [image] $<q^{\text{img}}>$\nDescribe the everyday image in one word: |
| 6. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{txt}}$ | OVEN | [text] $<q^{\text{img}}>$\n$<q^{\text{txt}}>$\nAnswer the question based on the image from Wikipedia in one word: |
| InfoSeek |  |  |
| 7. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{img}}$ | FashionIQ | [image] $<q^{\text{img}}>$\nChange the style of this shirt/dress/toptee to $<q^{\text{txt}}>$\nDescribe this modified shirt/dress/toptee in one word: |
| CIRR | [image] $<q^{\text{img}}>$\nModify this image with $<q^{\text{txt}}>$\nDesribe modified image in one word: |  |
| 8. $(q^{\text{img}},q^{\text{txt}})\rightarrow(c^{\text{img}},c^{\text{txt}})$ | OVEN | [image,text] $<q^{\text{img}}>$\n$<q^{\text{txt}}>$\nAnswer the question based on the interleaved image-text passage from Wikipedia in one word: |
| InfoSeek |  |  |
| Task | Dataset | MTEB task instruction |
| 9. $q^{\text{txt}}\rightarrow c^{\text{txt}}$ | ArguAna | [text] $<q^{\text{txt}}>$\nGiven a claim, generate a document that refute the claim in one word: |
| Climate-FEVER | [text] $<q^{\text{txt}}>$\nGiven a claim about climate change, generate a document that supports or refutes the claim in one word: |  |
| CQADupStack | [text] $<q^{\text{txt}}>$\nDescribe the StackExchange question in one word: |  |
| DBPedia | [text] $<q^{\text{txt}}>$\nGiven a query, generate a relevant entity description from DBPedia in one word: |  |
| FEVER | [text] $<q^{\text{txt}}>$\nGiven a claim, generate a document that supports or refutes the claim in one word: |  |
| FiQA | [text] $<q^{\text{txt}}>$\nAnswer the financial question in one word: |  |
| HotpotQA | [text] $<q^{\text{txt}}>$\nAnswer the multi-hop question in one word: |  |
| MSMARCO | [text] $<q^{\text{txt}}>$\nAnswer the web search query in one word: |  |
| NFCorpus | [text] $<q^{\text{txt}}>$\nAnswer the question in one word: |  |
| Natural Questions | [text] $<q^{\text{txt}}>$\nAnswer the question using Wikipedia in one word: |  |
| Quora | [text] $<q^{\text{txt}}>$\nDescribe the question in one word: |  |
| SCIDOCS | [text] $<q^{\text{txt}}>$\nGiven a scientific paper title, generate a paper abstract that is cited by the given paper in one word: |  |
| SciFact | [text] $<q^{\text{txt}}>$\nGiven a scientific claim, generate a document that support or refute the claim in one word: |  |
| Touch´e-2020 | [text] $<q^{\text{txt}}>$\nAnswer the question with detailed and persuasive arguments in one word: |  |
| TREC-COVID | [text] $<q^{\text{txt}}>$\nAnswer the query on COVID-19 in one word: |  |

**Table 13: Prompts for reranking tasks in M-BEIR .**
| Task | Dataset | Prompt |
| --- | --- | --- |
| 1. $q^{\text{txt}}\rightarrow c^{\text{img}}$ | VisualNews | $<c^{\text{img}}>$\nNews:$<q^{\text{txt}}>$\nDoes the above News image match the News story? True or False |
| MSCOCO | $<c^{\text{img}}>$\nCaption:$<q^{\text{txt}}>$\nDoes the above daily-life image match the caption? True or False |  |
| Fashion200K | $<c^{\text{img}}>$\nDescription:$<q^{\text{txt}}>$\nDoes the above image match the cloth style description? True or False |  |
| 2. $q^{\text{txt}}\rightarrow c^{\text{txt}}$ | WebQA | Question: $<q^{\text{txt}}>$\nAnswer: $<c^{\text{txt}}>$\nDoes the answer correctly answer the question? True or False |
| 3. $q^{\text{txt}}\rightarrow(c^{\text{img}},c^{\text{txt}})$ | EDIS | Question: $<q^{\text{txt}}>$\nAnswer: $<c^{\text{txt}}>$\nDoes the answer correctly answer the question? True or False |
| WebQA |  |  |
| 4. $q^{\text{img}}\rightarrow c^{\text{txt}}$ | VisualNews | $<q^{\text{img}}>$\nNews:$<c^{\text{txt}}>$\nDoes the above News image match the News story? True or False |
| MSCOCO | $<q^{\text{img}}>$\nCaption:$<c^{\text{txt}}>$\nDoes the above daily-life image match the caption? True or False |  |
| Fashion200K | $<q^{\text{img}}>$\nDescription:$<c^{\text{txt}}>$\nDoes the above image match the cloth style description? True or False |  |
| 5. $q^{\text{img}}\rightarrow c^{\text{img}}$ | NIGHTS | $<q^{\text{img}}>$\n$<c^{\text{img}}>$\nDoes the above two images have the same scene? True or False |
| 6. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{txt}}$ | OVEN | $<q^{\text{img}}>$\nQuestion:$<q^{\text{txt}}>$\nAnswer:$<c^{\text{txt}}>$Does the answer correctly answer the question? True or False |
| InfoSeek |  |  |
| 7. $(q^{\text{img}},q^{\text{txt}})\rightarrow c^{\text{img}}$ | FashionIQ | $<c^{\text{img}}>$\nCaption:$<q^{\text{txt}}>$\nDoes the above caption describe the modification of the image? True or False |
| CIRR |  |  |
| 8. $(q^{\text{img}},q^{\text{txt}})\rightarrow(c^{\text{img}},c^{\text{txt}})$ | OVEN | $<q^{\text{img}}>$\nQuestion:$<q^{\text{txt}}>$\nAnswer:$<c^{\text{txt}}>$Does the answer correctly answer the question? True or False |
| InfoSeek |  |  |

Figure: Figure 2: Examples of modality-aware negative samples mined by $M^{\text{rand}}$(NV-Embed-v1). We observe that negative samples with incorrect modality show similar semantic meaning as queries while negative samples with unsatisfactory information needs show less accurate information compared to the correct answers
Refer to caption: x2.png

Figure: Figure 3: Top-1 candidates for the tasks of composed image retrieval and reranking. In many cases, retrieval and reranking yields different top-1 results from labeled positives but appears to be correct since each query only has single labeled positive candidate (see Table 8).
Refer to caption: x3.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: please try to address the weaknesses.
soundness: 3
strengths: 1)	This paper explores the use of multimodal LLMs for universal multimodal retrieval, which is an interesting and potential research topic for the retrieval field.
2)	In this paper, the authors explore universal multimodal retrieval from three aspects, including modality-aware hard negative mining, continual text-to-text retrieval fine-tuning, and prompting MLLMs as zero-shot rerankers. These components are designed to improve the effectiveness of MLLM-based universal multimodal retrieval, while maintaining strong text-to-text retrieval capability.
3)	Extensive results demonstrate the effectiveness of the proposed method.
summary: In this paper, the authors focus on developing multimodal LLMs (MLLMs) for universal multimodal retrieval. Instead of directly fine-tuning MLLM-based bi-encoder retrievers with instructions as a guide on multiple multimodal retrieval tasks, the authors propose modality-aware hard negative mining and continual text-to-text retrieval fine-tuning to address the modality bias problem from MLLMs. Furthermore, the authors explore to prompting MLLMs as zero-shot rerankers, which can further boost retrieval accuracy in complex retrieval tasks. Extensive experimental results demonstrate the effectiveness of the proposed method.
weaknesses: 1)	The details of the proposed method are not clear and confusing. In sec. 4.1.1, it seems random negatives are denoted as c_i^{\times} symbol, while the generated triplets use c_i^{\times} again, does it mean positive sample or random negative?
2)	In sec. 4.2, the authors prompt LLaVa-Next for reranking, while not introducing the detailed MLLMs adopted in sec. 4.1. Are there two multimodal LLMs needed for the proposed method? 
3)	It will be better to provide the framework figure for the whole method. 
4)	In Table 4, it is not clear which results denote the reranking results, it seems the last row does not outperform the fifth row.
5)	As a retrieval method, the efficiency of retrieval time and memory is as important as effectiveness. The authors should analyze the efficiency of the unimodal retrieval mechanism.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: See weaknes.
soundness: 2
strengths: 1. The topic explored by the authors is a great one, namely general multimodal retrieval.
2. The method proposed by the author is effective to a certain extent.
3. The authors describe the experiment in detail and I think it is reproducible.
summary: This paper explores a more practical scenario in information retrieval, namely general multimodal retrieval. The author proposes a new fine-tuning pipeline and re-ranking method to improve the performance of specific retrieval tasks. However, the organization of the experimental results of this paper is rather coarse, and the performance presented makes me concerned about its generalization and robustness. It is recommended that the author further improve the presentation of results and methods. However, it is undeniable that the research topic can bring contribution and inspiration to the multimodal community.
weaknesses: 1. Unfortunately, the authors are not the first to use MLLM for re-ranking. It is recommended that the author faithfully investigate related research, e.g., [1]. The authors should narrow down the scope of the study, i.e., composed image retrieval.
2. The author's claim in line 184 should be verified experimentally using either very large k or very small k.
3. Line 306: The author's fine-tuned batch size for the two types of models is different. Will this affect the effect of mining negative samples?
4. I personally think that CLIP is a product from 3 years ago, and more advanced models should be used, such as BLIP or an improved version of CLIP. Moreover, the comparative experiment lacks experimental results of some recent methods that solve specific tasks, and it is impossible to quantitatively feel the advantages of the proposed MLLM-based method. The current organization is more like a complete ablation experiment.
5. From the results in Table 4, the full version re-ranking results do not seem to be optimal, which makes it difficult to explain the effectiveness of the method's continuous fine-tuning on some tasks. The improvement margin of its full version seems to be even worse than M^hard, which makes it hard to convince me.
6. From the studied results, the author expects to achieve universal retrieval, but as far as the experimental results (e.g., Table 7) are concerned, the proposed method will reduce the performance in some benchmarks. This makes me concerned about the generalization and robustness of its method.

[1] MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to the Weaknesses section and address my concerns.
soundness: 3
strengths: 1.	The paper is well-motivated to use multimodal LLMs for advancing information retrieval and support broader retrieval scenarios.
2.	The experiments are relatively comprehensive to validate the effectiveness of the proposed method.
summary: The paper presents techniques for advancing information retrieval with multimodal large language models (MLLMs). It first finetunes MLLM-based retrievers to tackle universal multimodal retrieval tasks and shows that MLLM-based retrievers exhibit modality bias in cross-modal retrieval tasks compared to CLIP-based retrievers. To address the issue, it proposes modality-aware hard negative mining and continual fine-tuning, and the MLLM-based retriever, UniEmb, yields state-of-the-art retrieval accuracy in universal multimodal retrieval tasks while maintaining strong text-to-text retrieval capability. Moreover, it explores to prompt MLLMs as a re-ranker to further boost retrieval accuracy.
weaknesses: 1.	The paper claims that existing retrieval models typically addressed search scenario where retrieval task are fixed and only a single modality is supported for both queries and retrieved results, but a series of retrieval models for multimodal retrieval are listed in related work.
2.	The paper claims that it is the first work to explore prompting MLLMs fir zero-shot reranking, but as far as I know, the work [a] has used MLLMs for reranking, and the differences should be explained in detail.
3.	Some expressions are confused. For example, in Line 308 of Page 6, “we initialized retriever using the pretrained model”, does the “retriever” refer to the model to be fine-tuned with hard negatives?
4.	It is not easy to understand Figure 1, where the task definition of Task 1-13 is missing. The relationship between the upper left and lower right sub-figure is not clearly illustrated.
5.	How is the performance when using fine-tuned MLLMs for reranking? Is it better than reranking in a zero-shot manner for tasks that involve single-modal queries?
6.	More qualitative cases are suggested to be provided to better illustrate the effects, such as how modality-aware hard negative mining alleviates modality bias. 

[a] Qu L, Li H, Wang T, et al. Unified Text-to-Image Generation and Retrieval[J]. arXiv preprint arXiv:2406.05814, 2024.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. In Table 1, why is the M^rand column for MTEB Text Retrieval empty? Are these models not evaluable on MTEB? I believe that comparing M^rand with M^hard would more effectively demonstrate the modality bias issue and the effectiveness of the proposed method.
2. In Table 4, why does re-ranking show limited improvement for knowledge-related multimodal retrieval datasets like OVEN and InfoSeek? and even a decline in performance on some tasks (e.g., task1 and task3)? How do the authors explain this phenomenon, and could it be related to the modality of the retrieval task?
3. In line 343, the authors mention that MLLM-based retrievers tend to retrieve text over images. Could this be related to the sampling of negative samples during training? If there are more image samples in the dataset, this could lead to a higher proportion of images as negative samples, which could further induce bias. I suggest conducting a statistical analysis on the modality distribution of the data samples.
soundness: 3
strengths: 1. The paper identifies an interesting issue of modality bias in multimodal retrieval based on multimodal large language models, and it proposes an effective hard sample mining strategy to address this issue while preserving text retrieval capabilities.
2. The paper introduces a zero-shot re-ranking approach using multimodal large language model, which, to some extent, improves multimodal retrieval accuracy.
3. The evaluation is thorough, using widely adopted M-BEIR and MTEB datasets, and the paper provides detailed parameters for reproducibility, lending credibility to the experimental results.
summary: This paper proposes a general retrieval method for multimodal retrieval tasks, named UniEmb, based on multimodal large language models. The paper identifies the phenomenon of modality bias in multimodal retrieval and finds that current retrieval methods based on multimodal large language are lacking in text retrieval performance. To address these issues, the authors propose a continued fine-tuning method based on hard negative samples, as well as a strategy that uses multimodal large language for further re-ranking. The proposed method is comprehensively evaluated on two widely adopted datasets, M-BEIR and MTEB, with results showing that UniEmb improves text retrieval performance while maintaining the effectiveness of multimodal retrieval.
weaknesses: 1. The proposed methods for hard sample mining and re-ranking using large multimodal models are straightforward and intuitive, and may lack some novelty.
2. There is no direct comparison with similar baseline models, such as E5-V, on the M-BEIR dataset.

Meta-review: This paper explores trained and zero-shot versions of mixed-modality retrieval with help from MLLMs for reranking. The authors show effectiveness through extensive experiments. Some concerns exist about novelty (in using MLLM for re-ranking), experiments and writing/clarity (method details, relationship to prior work), but these are addressed in the rebuttal phase. All reviewers recommend acceptance, although mostly lean only very weakly positive.

Decision: Accept (Poster)