MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS
Authors: Sheng-Chieh Lin
Chankyu Lee
Mohammad Shoeybi
Jimmy Lin
Bryan Catanzaro
Wei Ping
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal
embedding model
retriever
LLM
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: State-of-the-art retrieval models typically address a straightforward search scenario, in which retrieval tasks are fixed (e.g., finding a passage to answer a specific question) and only a single modality is supported for both queries and retrieved results. This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs), enabling a broader search scenario, termed universal multimodal retrieval, where multiple modalities and diverse retrieval tasks are accommodated. To this end, we first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks. Our empirical results show that the fine-tuned MLLM retriever is capable of understanding challenging queries, composed of both text and image, but it underperforms compared to a smaller CLIP retriever in cross-modal retrieval tasks due to the modality bias exhibited by MLLMs. To address the issue, we propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers. Second, we propose continuously fine-tuning the universal multimodal retriever to enhance its text retrieval capability while preserving multimodal retrieval capability. As a result, our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR, which spans multiple domains and tasks, while also surpassing the state-of-the-art text retrieval model, NV-Embed-v1, on the MTEB retrieval benchmark. Finally, we explore prompting the off-the-shelf MLLMs as zero-shot rerankers to refine the ranking of the candidates from the multimodal retriever. We find that, through prompt-and-reranking, MLLMs can further improve multimodal retrieval when the user queries (e.g., text-image composed queries) are more complex and challenging to understand. These findings also pave the way for advancing universal multimodal retrieval in the future. We release the model weights at: https://huggingface.co/nvidia/MM-Embed.

Published as a conference paper at ICLR 2025
MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMs
Sheng-Chich Lin +1,
Chankyu Lee
Mohammad Shoeybi
Jimmy Lin
Bryan Catanzaro
Wei Ping
NVIDIA
University of Waterloo
ABSTRACT
State-of-the-art retrieval models typically address straightforward search sce- nario, in which retrieval tasks are fixed (e.g.. finding a passage to answer a spe- eific question) and only single modality supported for both queries and re- trieved results. This paper introduces techniques for advancing information re- trieval with multimodal large language models (MLLMs), enabling broader search scenario, termed universal multimodal retrieval, where multiple modali- ties and diverse retrieval tasks accommodated. To this end, first study fine-tuning an MLLM as bi-encoder retriever on 10 datasets with 16 retrieval tasks. Our empirical results show that the fine-tuned MILM retriever capa- ble of understanding challenging queries, composed of both text and image, but it underperforms compared to a smaller CLIP retriever in cross-modal retrieval tasks due to the modality bias exhibited by MILMs. To address the issue, we propose modality-aware hard negative mining to mitigate the modality bias ex- hibited by MLLM retrievers. Second, propose continuously fine-tuning the universal multimodal retriever to enhance its text retrieval capability while pre- serving multimodal retrieval capability. As result, model, MM-Embed, achieves ofte--oftthe-rtt performance on the multimodal retrieval benchmark M- BEIR, which spans multiple domains and tasks, while also surpassing the state- of-the-art text retrieval model, NV-Embed-vl, on the MTEB retrieval benchmark. Finally, we explore prompting the off off-the-shelf MLLMs as zero-shot rerankers to refine the ranking of the candidates from the multimodal retriever. We find that, through prompt-and -er-anannnnnngg. MLLMs can further improve multimodal re- trieval when the user queries (e.g., text-image composed queries) are more com- plex and challenging to understand. These findings also pave the way for advane- ing universal multimodal retrieval in the future. We release the model weights at: bttps://nuggingfaacs cw/ dia MM Embed
INTRODUCTION
Information retrieval is crucial for variety of downstream tasks, such question answer- ing (Kwiatkowski al. 2019), fact-checking (Thorne et al., 2018), and retrieval-augmmented gener- ation (Lewis et al. 2020). State-of-the-att retrievers often focus narrow scenarios. For example, LLM-based retrievers (Wang et al. 2023 Lee et al. 2024 Meng et al. 2024) are limited to text-to- text retrieval tasks, in which both the query and the retrieved results text-only. Recent work on multimodal retrieval (Zhang et al. 2024 Jhang et al. 2024) focuses on specific tasks and assumes a homogeneous document format. However, in real-world applications, documents and queries often incorporate diverse formats or modalities, such as text, images, and interleaved text-image content. To advance information retrieval and support broader search scenarios. this work explores leverag-
ing multimodal LLMs (MLLMs; Liu et al. 2023a 2024 Dai et al. 2024) for universal multimodal
"Sheng-Chieh Lin did this work during intersshi at NVIDIA. Correspondence to: Sheng-Chich Lin ($269lin @wwaterloo.ca). Wei Ping (wping@nvidia.com

---

Published as a conference paper at ICLR 2025
retrieval, accommodating diverse iser-instructed tasks with multimodal queries and documents, as illustrated in Figure
We first explore fine-tuning MLLM-based bi-encoder retrievers with instructions guide (Asai (ct al., 2023) on 16 multimodal retrieval tasks from M-BIER (Wei et al. 2023). We find that MLLM- based retrievers significantly outperform CLIP--ased retrievers in the challenging tasks that involve interleaved text-image queries, such as visual question answering and composed image retrieval (tasks and 7 in Fig However, MLLM-based retrievers underperform in cross-modal retrieval tasks due to the modality bias exhibited by MLLMs. That is, given a text-based query with the instruction to retrieve an image (e.g.. task in Figure D an MLLM-based retriever tends retrieve a relevant text-only document rather than one containing images, especially when we enhance the retriever's text retrieval capability. To address the issue, we propose modality-aware hard negative mining (Section 4.1.1 and continuously fine-tuning for text-to-text retrieval (Section 4.1.2). Our final retriever, coined MM-Embed, is the first universal multimodal retriever to achieve state-of-the- art performance while maintaining competitive text-to-text retrieval across diverse tasks.
Finally, we explore prompting MLLMs as zero-shot rerankers. Surprisingly, we find that the zero- shot MLLM-based rerankers can further boost retrieval accuracy in the tasks where user queries are interleaved text-image and particularly challenging to understand. For example, in the composed image retrieval dataset, CIRCO (Baldrati et al. 2023), the zero-shot reranker refine the ranked lists and significantly boost the accuracy (mAP@ 5) over 7 points from the existing state-of-the-an composed-iman retriever (Zhang et al. (024) and our universal multimodal retrievers. This finding indicates that there is still room for improvement in such challenging tasks to advance universal multimodal retrieval. Moreover, knowledge distillation from zero-shot few-shot MLLM-based rerankers to retrievers is a promising direction.
We summarize our contributions as follows: i) We present study on applying MLLMs to universal multimodal retrieval. ii) We are the first to develop MLLM-base M-ased universal multimodal retrievers. Notably, our MM-Embed, initialized from the existing best-performing text retriever (NV-Embed- v1:[Lee et al.) 2024). not only achieves state-of- of-the-art results in the universal multimodal retrieval benchmark M-BEIR (Wei et al., 2023). but also surpasses NV-Embed-v1 in text-to-text retrieval tasks MTEB. iii) We explore prompting MLLMs as zero-shot rerankers in various multimodal retrieval tasks. Surprisingly, we find that zero-shot MLLM-basee rerankers can improve ranking accuracy over strong retrievers in challenging tasks involving interleaved -ext-image queries.
We organize the rest of the paper as follows: We discuss related work in 2
We introduce the
definition of universal multimodal retrieval in $ $ Jndd present the proposed method in $ 4 We report experimental results in 5 and conclude the paper in $
RELATED WORK
instruction-Aware Dense Representation Learning. Asai et al. (2023) is the first study to iden- tify the implicit search intent behind each retrieval task and proposes fine-tuning retriever to learn diverse retrieval tasks with handwritten task instructions. SSu et al. (2023) and existing state-of-the- art LLLM-based text embedding models (Wang et al. [023] Meng al. 2024 Lee al.) 2024) adopt this approach to broader tasks beyond text retrieval, such as text classification and clustering. Re- cently, Wei et al. (2023) propose a universal multimodal retrieval dataset, M-BEIR, and find that catruction-wware dense retrieval fine-tuning is crucial to tackle universal multimodal retrieval.
Vision-Language Models for Multimodal Retrieval. With the advancement of pre- vision-language models (Radford et al. 2021 Li et al., 2022), the research focus shifts from single- modal (Bajaj et al., 2016) Fu et al. (003) to cross-modal (Lin et al.)
2014 Han et al., 2017 Liu
et al. 202 a or more complex multimodal retrieval tasks (Liu et al. 2021b: Wu al. 2021 Bal-
drati et al. 2023). However, the aforementioned tasks assume a homogencous modality for queries and documents, thus limiting their application. Liu et al. (2023c) take one step further to tackle the retrieval scenario involving a candidate pool with heterogeneous modalities but still limit itself to a single retrieval task.
Wei et al. (2023) extend the study to more general scenario, where retrievers are required to deal with queries, candidate pool with heterogeneous modalities and diverse retrieval tasks. However,
2

---

Published as a conference paper at ICLR 2025
Task
Instruction
Query
falkwing
Fime-tnne
Candidatos
sull eenylle eage
aucic vtlippd pvopyday imags
MLLM
Insaee
a sove cupiic dapplaycl
MM-Embod
Image
Fir Brsinee tilky sl oe speght you
Wha baakiaa
Rut a Willew kaage dourlit pab
Find mumplo Legiin
(a) Fine-tuning retrievers
Reraniked rewlt
Candidate
Prompe:
Query:
Maatio du buElin
sniceer the
Tru:: 0.7
Candidate
Candidate 0
halking
Reranker
answer
Does correctly
Candidats
(b) Prompting and reranking
Illustration of universal multimodal retrieval (a). where each task consists of task- specific instruction and query. Both queries and candidate documents are in heterogeneous formats (i.e., text, image or, interleaved text-image). In this work, we explore (a) fine-tuning MI.LM-based universal multimodal retrievers and (b) prompting pre-trained MILLMs for zero-shot reranking over retrieved candidates. We adopt LLa Va-Next (Liu et al. 2024 as our MLLM backbone.
Figure 1:
the study is limited to CLIP-based retrievers and ignores important text-to- to--ttttttetteee retricval tasks, such as fact-checking (Thorne et al. 2018) and entity retrieval (Hasibi al. 2017). While Koukounas al. (2024) aim to fine-tune Ca.de-based retriever with both strong text-to-text and mu multimodal retrieval capabilities, they only consider simple multimodal retrieval tasks: image-caption retrieval (Young) ct al. 2014, Lin et al. 2014). Concurrent with our work, Jiang et al. (2024) propose to fine-tune MLLMs on the NLI dataset (Bowman et al., 2015) and demonstrate their transferability to multi- modal retrieval. In this paper, we are the first to study how to fine-tune an MLLM- -ased universal multimodal retriever while maintaining strong text-to-text retrieval capability.
Prompting Multimodal LLMs for Reranking. Instruction tuning has enabled large language models (LLMs) to tackle a wide range of tasks in zero-shot setting. Building on this, prior studies have investigated prompting LLMs text reranking (Ma al. 2023 Sun al. 2023 Zhuang et al., 2024b). In this work, we extend this line of research into multimodal LLMs, exploring their potential as zero rerankers for multimodal tasks. Notably, Qu et al.) (2024) introduce a framework using multimodal LLMs for zero-shot reranking through generative retrieval approach (Li et al., 2024). However, their method constrained retrieval tasks with text-only queries. contrast, our ap- proach broadens the scope by prompting multimodal LLMs to handle diverse multimodal reranking tasks, accommodating queries and documents that can in text, image, or interleaved text-image formats. This generalization enables versatile applications in multimodal ranking settings.
UNIVERSAL MULTIMODAL RETRIEVAL
Following the framework of Lin et (222 ), we formulate the task of retrieval follows: given a query q, the goal is to retrieve ranked list of candidates {cl ( ranking metries such as nDCG, where C' the collection of documents. In this work, we bor- row the setting of universal multimodal retrieval from Wei et al. (2023). where user queries and candidates may consist of text, image or interleaved text-image; i.c., q E {chh _! (cM c. Additionally, there are multiple search intents behind search query, which can be elaborated through specific instructions (Asai et al. (023). For example, in tasks 1 and 2 of Figure given the same image query, the search intent to find an image caption and similar image, respectively. Thus, in universal multimodal retrieval, given a multimodal query and task instruction inst, we aim retrieve list of candidates from pool of multimodal documents
E C' to maximize some
3

---

Published as a conference paper at ICLR 2025
maximize specified ranking metric. Note that only consider text and image in this work while other modalities, such as audio and video can be included, which we leave for future work.
4 METHOD
In this section, we describe our approach to universal multimodal retrieval by leveraging multimodal LLMs (MLLMs), specifically Va-Neee (Liu et al. 2024). In Section [.11 we first fine-tune an MLLM-based retriever to project multimodal user queries, along with task instructions, into the same semantic space as multimodal documents, enabling n-nearest neighbor search (Johnson al, 2021]. In Section 4.2 we present our method to use MLLMs to rerank the top-k candidates retrieved by the universal multimodal retriever.
4.1 FINE-TUNING MULTIMODAL LLMS FOR UNIVERSAL MULTIMODAL RETRIEVAL
We fine-tune an MLLM-based retriever parameterized by 0 (i.e., 17") under the guidance of task- specific instructions, aiming to capture the implicit intent behind retrieval tasks. Specifically, given a user query qi with the specified task instruction insl, and its corresponding relevant candidate, ot we minimize the contrastive loss (Gutmann & Hyvärinen 2010):
[8]
exp (Tf (inst; g.) of 1/7) EeeD (ep(nn(inste 9:) -T (c)/T)
L
log
(1)
B
where r-) Rd is normalized vector and is the temperature scaling factor. Ideally, D includes all the candidate documents. However, including all the candidate documents is not computationally feasible; thus, mining informative negative candidates as alternative D is crucial for success- ful contrastive learning. In this work, we propose modality-award aware negative mining for contrastive learning in the scenario for universal multimodal retrieval.
4.1.1 MODALITY- HARD NEGATIVE MINING
Prior work (Karpukhin et al. 2020 Xiong al. 2021 de Souza Moreira al. 2024 has demon- strated that hard negative mining significantly improves representation learning for text-to-text re- trieval. In the traditional retrieval setting, where the corpus consists of documents with homoge- neous modality, a document is considered hard negative if it lacks the required information but still retrieved by a model. However, the scenario of universal multimodal retrieval, where the corpus contains documents involving diverse modalities, user's desired modality as specified in task instructions (i.c., text, image or interleaved text-image) should be taken into consideration. For example, shown in Figure T the first and second users issue the same query along with different instructions, requiring the documents in text and image formats, respectively. To address this, we propose modality-awar hard negative mining to guide models retrieving candidates that meet both the users' information needs and their preferred modality.
Specifically, we first fine-tune MLLM-based retriever by using in-batch samples as random neg- atives; i.e., D = (ct, (TB) in Eq. 4 The candidate documents in the mini-batch, except c,", are considered random negatives for (insti, q:). The fine-tuned model is denoted as M For each query 4: and its associated instruction inst; in the training set, we generate two types of negative samples from the top-50 candidates retrieved by A/rand i) negatives with an incorrect modality (C.) ) where the candidate ranks higher than the labeled positive but has different modality from the de- sired one, and ii) negatives with unsatisfactory information (C?), where the candidate ranks lower than position k but has the same desired modality.
Previous studies on text retrieval (Xiong et al. 2021 de Souza Moreira et al. 2024) have shown that setting k' to small number may introduce false positives while setting k' large number could make the negative samples too easy. In our experiment, we set k' = 45 following the state- of-the-art text retrieval training in Lin al. (202)) During training, given the query d with the associated instruction inst,, we generate triplet. (inst, q:), c c, by sampling a hard negative from either C. or C² with the same probability; i.c., D = (ct. ..... ( in Eq. ( Thus, in the setting of hard negative mining, the negatives mined for (inst,, gi) include i) the hard
4

---

Published as a conference paper at ICLR 2025
negative of and ii) all positives and hard negatives from other queries, which are considered random negatives. Note that the setting of hard negative mining includes twice as many candidate documents as random negative mining under the same batch size [B]. For fair comparison, we use 2 [B] and [B] when fine-tuning with random and hard negatives, respectively. Figure [ the Appendix showcases both types of negative samples. We observe that the negatives from C1 are sentences that are semantically similar to the queries but do not match the user's desired modality.
4.1.2 CONTINUOUS TEXT-TO- TEXXT RETRIEVAL FINE- TUNING
Since text-to-text retrieval remains one of the most widely used retrieval tasks, we further fine- tune M on a diverse set of public text-to-text retrieval tasks, including MS MARCO (Ba-) jaj et al. 2016). HotpotQA (Yang et al., 2018). Natural Question (K (Kwiatkowski et al. 2019). PAQ (Lewis et al. (02T), StackExchange Fxchan-Ex Siiraangee-Cmmnttyy. (023), Natural Language Inference (Bowman et al.] (2015). SQuAD (Rajpurkar al. 2016), ArguAna (Wachsmuth et al. 2018 BioASQ (Nentidis et al.) 2023). FIQA (Maia et al., 2018), and FEVER (Thorne et al,
2018 As these datasets do not contain negative samples, employ the fine-tuned LLM-based
retriever (NV-Embed-vy) Lee et al. 2024) to mine hard negatives in our experiments (see de Souza
P. Moreira et al. (2024) for details).
During continuous fine-tuning, we uniformly sample triplets from both universal multimodal and text-to-text retrieval training data. Note that for each query qi in the universal multimodal retrieval training data, we use M to mine second-type hard negatives C. again. Since no first-type hard negatives (i.e., C. = 0) are found by Mfhard we retain the first-type hard negatives mined by Mrnd
4.2 PROMPTING MULTIMODAL LLMS FOR RERANKING
Prior work (Sun et al. 2023 Jin et al. 2024) has demonstrated that instruction-fine-une LLMs can be prompted to rerank candidates in text-to-text retrieval tasks. In this work, we directly prompt pre-trained LL-Vt-Next (i.e., the same MLLM backbone for retrievers but without fine-tuning) to further rerank the top 10 retrieved candidates from universal multimodal retrievers. Following the approach of Nogueira et al. (2020), we frame the reranking task as a series of True/False questions. Specifically, given a query and retrieved candidate, we prompt Va-Next to determine whether the retrieved candidate satisfies the query by answering "Truc" or "False". For example, in the image caption retrieval (task in Figure 1,, given an image query, q gn and a retrieved text-based candidate, >\ıCaationn=< c > >'nDoes the above daily-life image visual question answering retrieval (task 5 in Figure D. given visual question, <Qry image>< texi>, and retrieved text-based candidate, <Doc lext>, we use the following prompt: <Qry bonng>>\/QQuttton <Qry texi>> nAnnwerr < Doc text>> nDoes the answer correctly answer the question? True False. We refer readers to Table I8 in Appendix for the specific prompts used in different multimodal retricval tasks.
chi, we use the following prompt: "< qTTRR >>
match the caption? True or False". Additionally,
To compute relevance scores, we apply the Soîtmax operation to the logits of the "Truc" and "False" tokens, and use the probability of the "True" token as the relevance score for reranking. Our preliminary study in Section 5.3.3 shows that zero-shot MLLM- based rerankers mainly improve the tasks where queries are interleaved text and images, such as composed image retrieval and visual question answering, shown in the tasks 3, 5, and 6 of Figure I
EXPERIMENTS
5.1 DATASETS AND MODELS
Multimodal Retrieval Dataset. We evaluate models' universal multimodal retrieval capabilities using the M-BEIR dataset (Wei et al. 2023), which is constructed from 10 datasets with 16 diverse multimodal retrieval tasks across domains, as listed in Appendix Table 10 We train our models on the M- M-BEIR 1.1M training queries and evaluate their effectiveness the 190K test queries. Fol- lowing the global evaluation setting of the M-BEIR dataset, for each query, candidates are retrieved from unified candidate pool of 5.6M multimodal documents spanning all 10 datasets. We report the average Recall @5 (R (5) as the retrieval accuracy across all test queries in each dataset, except
https::/huugiingfacccmo
BEIR
5

---

Published as a conference paper at ICLR 2025
for FaShion200K and FashionIQ. for which we report Recall @ 10 @R@ 10). We refer readers to Wei ct al. (2023) for more details on the construction of the M- M-BEIR dataset.
Text-to-Text Retrieval Dataset. While M-BEIR contains the WebQA dataset for text-to-text re- trieval evaluation, conduct a more comprehensive text-to-text retrieval evaluation using the MTEB dataset (Muennighoff et al.] 2023). Specifically, we evaluate our models on 15 diverse text retrieval datasets Following the established procedure, we report the average nDCG@ 10 across the 15 text retrieval datasets. Note that, unlike in M-BEIR, where candidates are retrieved from a unified pool across all tasks, in the MTEB retrieval tasks, candidates are retrieved from separate corpora for each task.
Backbone Model Choices. In this work, we utilize two representative backbones of vision- language models build universal multimodal retrievers, CLIP (Radford al.) 202I) and LL Next (Liu et al. 2024). For CLIP, we initialize from the CLIP-Large model and employ the best- performing modeling approach from Wei et al. (2023). denoted CLIP st- 3] This method fuses the input image and text features by separately encoding each input (query or document) image and text into distinct vectors, which are then summed to create a fused vector (Liu et al. 2023c). Addition- ally, we report results for BLIP (Li et al. (022). which integrates text information into the image encoder through cross-attention. We use BLIPFF from Wei et al. (2023), which fine-tuned on the M-BEIR dataset with random negative
LLa Va-Net (Liu et al., 2024) is a multimodal LLM (MLLM), which integrates a CLIP image en- coder, LLM and vision- languaee MLP projector to align image features to the input embedding space of the LLM. We use LLaVt-Next with the Mistral 7B (Jingg et al.] 2023) as the backbone L We experiment with three variants: (1) Va-E: the <eos>> token embedding is used to aggregate information from the multimodal input, a method commonly employed in prior work on text retrieval (Wang al. 2023 Ma et al., 2024b); (2) LLa Va-P: the MILM is prompted to summa- rize each multimodal query (or document) input in one word, using embedding of the last token to encode multimodal input O (3) NVEmb: The LLM from LLa Va-Next replaced by the fine-tuned LLM-based text retrieval model NV-Embed-vl (Lee et al. 2024) while all other components (i.e., image encoder and vision-language MLP projector) remain unchanged
? Note that the backbone of NV-Embed-vl is also Mistral 7B. The instructions for LLa Va-E (or NVEmb) and LLaVa-P are illustrated in Appendix Table [6 and [7] respectively. For reranking experiments, we also utilize with Mistral 7B, and the prompts are listed in Appendix Table 18
Retriever Training Details. For each backbone, we start by fine-tuning M with random neg- atives; i.e., D = (ej , ( in Eq. (I.. The fine-tuned model is denoted M For the CLIP backbone, following (Wei et al. 2023), we fine-tune CLIPS  for 20 epochs with a learning rate of 1e 5. For I. Va-Next backhone, we fine-tune the models for 2 epochs with learning rate of le- - 4. Note that for Va-Nex backbone, we only fine-tune the vision-language projector and LoRA (r 8,a
64) added to the language model. At the stage of fine-tuning Mfhand with hard negatives, we mine the two types of hard negatives following Section 4.1.J using each retriever. Then, we fine-tune each retriever using its own mined hard negatives with the same training proce- dure as the first stage; i.e., D = (c" C ................ B ( in Eq. 1). We fine-tune the models with the batch size of 128 8 and 64 x 8 when using random and hard negatives, respectively. When GPU memory is insufficient for the designated batch size, we use gradient accumulation. Note that when fine-tuning M we initialize the models using the pre-trained model, rather than continuously fine-tuning mn We denote the models fine-tuned with random and hard negatives as M and Mhard, (-), respectively. We refer readers to Appendix A.1 for more details.
The 15 retrieval datasets in MTEB are derived from public datasets in BEIR Thakur al. 2021 excluding BioASQ. SiM TREC-NEWS, Robust04,
1iip_fflarqe.11
https://huggings http:/giiinnaac...///aaaaaaaa/ee httpps//wanngnnnnaeeeoooe1eeeees http://oooo.aa........e......aa.
We refer readers Appendix Table 17 for the prompt and more details from the prior work (Zhuang et al.
2024a
Diang et al. 2024).
https::/ hwg.guuggggg.g wng www.c/..o/vvviiaacooccoooo
https://huggingaace...

---

Published as a conference paper at ICLR 2025
Table I: Main results on retrieval. Following Wei al. (203), we report R@5 for all the datasets,
except for Fashion200K and FashionIQ, for which we report R @ 10. The tasks of single-modal and multi-modal queries are tasks 1-5 and 6-8, respectively. For MTEB text retrieval, we report nDCG @ 10 averaged across 15 retrieval tasks (detailed in Appendix T5). See more comparison in Appendix Table and 13
Agua
Task
Dataset
MM-Embed
CL
BLIP
LLaVa-E
LLaVa-P
NVEmb
CLIPss
LLaVa-P
NVEmb
43.8
31.2
14.2
39.7
9t chr
MSCOCO
72.0
75.6
69.3
70.8
64.6
69.2
73.8
72.7
71.3
Fashion200K
1644
25
13.5
13.3
10.4
19.7
17.4
18.6
17.1
2. 9t 1
WebQA
83.2
79.5
88.6
88.8
92.1
88.2
93.6
93.5
95.9
EDIS
46.5
50.3
55,9
56.6
55.1
54.2
68.8
69.8
68.8
WebQA
76.0
79.7
80.3
81.6
81.3
80.1
84.9
84.8
85.0
VisualNews
39.5
21.1
32.4
33.3
30.4
40.6
39.4
41.4
41.3
MSCOCO
Fashion200K
17.2
27.6
13.9
14.7
13.2
20.0
17.5
19.9
18.4
NIGHTS
31.6
33.0
31.8
30.7
30.4
31.9
31.8
31.1
32.4
6. (f"",1"")) > dm
OVEN
40.4
38.7
37.9
39.1
36.3
40.9
42.9
42.6
42.1
InfoSeek
26.1
19.7
31.0
32.9
33.3
27.6
37.2
35.8
42.3
7. (f"")"""))
FashionIQ
27.4
27.0
25.8
CIRR
43.2
$1.4
48.1
45.4
45.3
38.3
49.5
50.8
50.0
(f"""""")) (ctre
OVEN
61.6
62.6
63.9
63.5
InfoSeek
43.9
27.7
50.3
50.0
53.4
47.1
54.4
$3.5
57.7
All
47.4
43.3
47.9
48.3
47.2
48.3
31.9
32.3
52.7
M-BEIR Avp.
Single-modal Qry
51.7
51.0
51.6
50.0
53.5
55.6
56.4
56.1
Mulii-modal Qry
40.1
37.3
42.7
42.8
42.7
39.5
45.6
45.5
47.0
MMcenighoff
2023]
Retrivaal Avg.
46.4
49.7
60.3
ranked top-5 on MTEB retrivval task leaderboard. NV-Emhod-vi
LLo
2024
scores 59.36 MTEB retrieval task.
To enhance text-to-text retrieval capability, we continuously fine-tune (NVard( with learn- ing rate of 2e 5 using mixture of training data from M-BEIR and public text retrieval datasets mentioned in Section 4. 1.2 for 4.5K steps. The final model is called MM-Embed.
5.2
MAIN RESULTS
Universal Multimodal Retrieval. Table pressnns the retrieval accuracy of different retrievers. In the M-BEIR evaluation, we observe that when fine-tuning with random negatives, Va-P achieves the highest overall retrieval effectiveness. This result indicates that LLaVa-P effectively aggregates multimodal input information into a single word representation. While MLLM based retrievers outperform CLIPsF on tasks involving multimodal queries, they still lag behind CLIPsH on tasks with single-modal queries, especially in cross-modality retrieval (i.e., tasks and 4). In addition, NVEmb achieves the best text-to-text retrieval accuracy on WebQA Task 2. It is worth noting that although BLIPFF performs the worst overall, it demonstrates notably strong performance in the fashion domain (e.g., Fashion200K and FashionIQ) but performs worse in news domain (c.g., VisualNews), likely due to differences in the text-image pairs used for pre-training between CLIP and BLIP.
Observing the models fine-tuned with hard negatives, MLLM- retrievers show significant im- provements in retrieval accuracy, particularly in tasks involving single-modal queries. On the other hand, CLIPsr does not show similar improvement. This could be attributed to the fact that CLIP has been well pre-trained for cross-modal retrieval, whereas MLLM-basee retrievers, fine-tuned with a contrastive learning objective for only 2 epochs, may still be underfitting. Fine-tuning with hard negatives accelerates the contrastive learning process of MLLM-based retrievers.
Table 2 reveals another factor contributing to the lower retrieval accuracy of MLLM-based retrievers for single-modal queries: text retrieval bias. This is- sue is particularly obvious for NVEmb. We compare models' retrieval accuracy on text-image and image- text retrieval (tasks and 4) on MSCOCO. The compar-
Table 2: Retrieval analysis on MSCOCO. M.A.@ denotes the modal- ity accuracy of the top-1 candidate. More results of M.A. are pre- sented in Appendix Table 14
M 1
M
Task Metrie
CLIPsF LLaVa-E LLaVa-P NVEmb
CLIPsF
LLaVa-P NVEmb
R@I
33.9
41.7
14.1
45.8
50.7
49.8
1.
R@s
72.0
69.3
70.8
64.6
69.2
73.8
72.7
M.A.@1
92.6
79.9
91.0
42.1
98.3
100.0
100.0
R@1
72.3
73.0
73.4
69.3
63.8
72.7
72.4
4. R@5
91.0
91.8
92.2
90.3
88.5
89.5
88.9
M.A.@.
98.7
99.2
99.8
96.3
94.2
100.0
100.0

---

Published as a conference paper at ICLR 2025
Table 4: Results of zero-shot reranking on composed
Table 3: Results of zero-shot reranking on
image retrieval task, CIRCO (Baldrati et al. 2023).
tasks 6-8 from the M-BEIR dataset.
Retrieval Model
Retrieval
Rerank
Ma (NVEmb)
MM-Embed
Task Dataset
MagicLens (Zhang et al.
2024
24.9
32.4
Retrieval
Rerank
Retricval
Rerank
E5-V (Jiang al. 2024
19.1
31.0
6. InfoSock
OVEN
35.8
42.6
37.1
44.3
42.3
42.1
43.1
43.5
(CLIPs
12.7
31.6
FashionlQ
26.6
20.0
25.7
19.0
BLIPfr (Wei ct al.
2023
26.6
36.1
7.
CIRR
50.8
48.6
50.0
48.2
M VL-aV
29.0
37.9
8.
OVEN
63.5
65.8
64.1
65.9
M (NVEmb)
32.4
40,9
InfoSock
53.5
54.5
57.7
57.3
MM-Embed
32.3
39.9
ison shows that (LLaVa-B) and (NVEn)) exhibit significantly lower modality accuracy (M.A. @ 1) than Mraand (CLIPs( in the text-to-image retrieval task. Most incorrectly retrieved top-1 candidates from the MLLM-based retrievers are relevant texts rather than images (see Appendix Figure This result indicates that MLLM-based retrievers have a bias toward relevant text rather than images, phenomenon not caused by random sampling, as evidenced in Appendix Table II This issue can be mitigated by our proposed modality-awar hard negative mining.
Finally, we observe that (NVEm)) performs worse in text-to-text retrieval tasks compared to (Nemn)))m)) but still outperforms M (LLa (i.c., WebQA task 2 and MTEB) However, compared the original NVEmb (Lee et al.] 2024), its score on MTEB retrieval tasks drops by almost 10 points. After continuous fine-tuning (detailed in Section 4.1.2). the final model, MM- Embed, not only surpasses NVEmb in MTEB but also retains strong multimodal retrieval capability. We attribute the improvement in text-to-text retrieval to the effective hard negatives mined by NV- Embed-vl as mentioned in Section 4.1.2 Notably, continuous fine-tuning significantly enhances multimodal retrieval performance in InfoSeek (col 8 vs 7 in Table [], highlighting its effectiveness in improving the model's 's ability to handle knowledge-intenss
multimodal retrieval tasks.
Zero-Shot Reranking. Table reports the reranked results of the top- retrieved candidates from M (NVEm)) and MM-Embed on the tasks involving multimodal queries. We observe accuracy improvements in visual question answering retrieval tasks (i.e.. OVEN and InfoSeck), but no im- provement in composed image retrieval tasks (i.e., FashionIQ and CIRR), However, as shown in Appendix Table TO FashionlQ and CIRR, compared to OVEN and InfoSeek, only have one rele- vance label per query. We hypothesize that there may be additional relevant positive samples that are not labeled. We refer the reader to Appendix Figure 3 for case studies.
We conduct experiments the CIRCO (Baldrati et al., 2023) validation set, composed image retrieval dataset with high-quality human annotations, consisting of 219 queries and 123K candi- dates in total. On average, 4.2 positive samples are labeled by humans per query. Table 4 presents mAP@5 for various retrievers and their reranking results. We directly use the models and code provided by the authors to obtain the results of MagicLens (Zhang et al. 2024 and E5-V [Jiang et al. 2024 10 retrievers. For our M-BEIR fine-tuned retrievers-Mi (CLIP M Va-),, M 1 (NVEmb) and MM-em we directly use the same instructions as CIRR in M-BEIR for query encoding. First, we observe that our MLLM-based retrievers outperform MagicLens and E5- V. More importantly, reranking the top-10 10 retrieved candidates from different retrievers significantly improves mAP@5 by at least points. This result demonstrates the effectiveness of prompting an MLLM as a reranker in composed image retrieval tasks.
5.3
ABLATION STUDIES
5.3.1 Is FINE-TUNING WITH INSTRUCTION NECESSARY?
We fine-tune NVEmb with random negatives on the M-BEIR subtasks listed in Table 5 and eval- uate the models' retrieval accuracy on the development queries from each subtask. Note that, for
We hypothesize that the decline of M (NVEmb) in text-to-text retrieval tasks results from the mitigation of text retrieval bias after fine-tuning with modality-aware hard negatives.
/tte////oot///////oooooooooooooe thub. deepm wwappmmin///mee nd/magg clllclleee clens
8

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study on fine-tuning NVEmb w/o (X) and w/ (V) instructions.
zero-shot
line lieettuing
Task
Dataset
CLIP
LLaVa-P
NVEmb
NVEmb
X
X
/
x
VisualNews
40.9
11.7
15.3
17.4
33.1
38.7
1. ghhl + clung
MSCOCO
55.4
58.1
64.2
59.9
76.7
82.8
Fashion200K
8.9
2.4
4.2
3.2
12.3
15.6
VisualNews
42.0
6.3
6.5
5.9
29.3
37.2
4. q -> cl
MSCOCO
79.6
66.8
70.6
68.2
88.9
93.0
Fashion200K
7.7
2.9
4.0
3.6
12.0
16.8
5. qm + c
NIGHTS
25.4
28.4
29.3
27.7
31.6
30.9
simplicity, encode only the datasct-specific corpus containing documents of the targeted modal- ity. For example, when evaluating retrieval accuracy for VisualNews Task I, we encode the 542K images from VisualNews (see Appendix Table I0] into the index rather than the entire 5.6M doc- uments from M-BEIR. We also report the CLIP and LLaVa-P (w/o instruction) zero-shot retrieval effectiveness reference point [1]
From 3 we observe that NVEmb, as zero-shot MLLM-based retriever, outperforms LLaV and even competes with CLIP in the tasks in tasks within the Miscellaneous domain (i.e., MSCOCO and NIGHTS). This result indicates that a fine-tuned MLLM-based text retriever is capable of per- forming multimodal retrieval tasks (the rating task instructions with queries degrades the model's 's retrieval effectiveness (col 4 vs 3). the model fine-tuned with instructions significantly outperforms the one fine-tuned without instructions (col 6 vs 5). This indicates that task instructions can help elicit a model's 's task- or domain knowledge for diverse multimodal retrieval tasks.
finding as in (Jiang et al. 2024)). Although incorpo-
5.3.2 EFFECTIVENESS OF CONTINUOUS TEXT- TEXT-TO-TEXS TO-TEXT RETRIEVAL. FINE-TUNING
In this section, we study the best strat- egy enhance a model's capabilities both multimodal and text-to-text retrieval. We begin by fine-tuning NVEmb on train- ing data for both universal multimodal re- trieval and text-to-text retrieval (detailed in Section 4.1.2 for 2K steps. As shown in Table 6 joint fine-tuning allows the model to maintain its text retrieval capa- bility (row vs 1), though it results in a drop of 2 points in multimodal re- trieval accuracy (row 3 vs 2). In contrast, explicitly fine-tuning hard (NVEmb) for an addition 2K steps significantly boosts its text-to-text retrieval capability with a slight drop of 0.8 points multimodal retrieval (row vs 4) This experiment shows that continuously fine-tuning multimodal retriever to enhance its text-to-text retrieval more effective than fine-tuning a retriever across all retrieval tasks simulta- neously. This finding suggests that a better optimized curriculum learning strategy (Bengio et al.) 2009) could further improve performance in universal multimodal retrieval, a direction we leave for future work.
Table 6: Ablation study to enhance model's text-to-text retrieval capability.
Training data
Initialization
M-BEIR+ BEIR+
Multimodal Text-to-
62.9
NVEmb
54.3
$1.7
/
52.2
63.0
M (NVEmb)
56.4
51.7
/
55.6
63.1
For M-BEIR, we only evaluate on the tasks with single- modality queries (i.c., tasks 1-5) while for BIER, we evaluate on 7 tasks: ArguAna. FiQA, NFCorpus, Quora, SCIDOCS. SciFact and TREC-COVID.
5.3.3 STUDY ON PROMPTING MLLMS FOR RERANKING
In this section, we study reranking effectiveness of MILMs on all the tasks the M-BEIR dataset. Specifically, for each development query, rerank the top- retrieved candidates from M Mrad ( (C). As shown in Table 7 prompting LLaVa-Next for reranking further boosts the rank-
11 We follow Jiang et al. (024) to prompt La-VN-Nex to output one word embedding for each query and document, i.c., <txt>> SSSmmary above sentence in one word:; <img>> <img>>\,nmme AnSummryy above image in one word:. 12 Note that MM-Fmbed in Table is fine-tuned under the same conditions for total of 4.5K steps.

---

Published as a conference paper at ICLR 2025
Table 7: Reranking study on top-10 retrieved candidates from Mc the M-BEIR de- velopment query set.
Task
Dataset
Retrieval
Rerank
7B
34B
VisualNews
44.2
38.8
42.5
1. 8r + clmp
MSCOCO
72.0
68.0
69.7
Fashion200K
17.8
14.7
15.6
2. gd + clu
WebQA
78.2
79.2
82.9
3. qmt [clmxx
EDIS
48.3
46.5
47.4
WebQA
78.2
67.7
68.3
VisualNews
37.4
29.3
29.8
4. gmm - c5
MSCOCO
91.0
87.3
89.0
Fashion200K
17.3
9.9
12.0
5. qlms > dime
NIGHTS
32.1
29.4
32.7
6. (gimss, sl dh
OVEN
40.6
43.2
43.7
InfoScck
25.6
28.4
29.0
7. (giins quil) > dime
FashionIQ
32.5
21.5
23.4
CTRR
52.4
54.1
54.2
8. (dl"ss g(a)) + (dime, chi)
OVEN
60.6
63.8
63.7
InfoScek
45.3
48.7
50.5
ing accuracy in tasks 6-8, which involve multimodal queries (except for FashionIQ). However, the reranking degrades accuracy in tasks 1-5 which involve single-modal queries (except for WebQA task 2). This trend persists even after scaling the reranker from 7B to 34B (col 3, 2 vs D We hy- pothesize that MILM rerankers, as a more robust cross-encoder compared to a bi-encoder retriever, excel at challenging tasks involving multimodal queries, even in zero-shot manner. However, zero- shot rerankers fail to leverage task- or Nomain-speciine knowledge, which limits their performance on relatively simple tasks involving single-modal queries. The relevance signals between queries and documents in the News, Miscellaneous, and Fashion domains can vary significantly. Thus, optimiz- for MILMSs to better capture domain- task-specific knowledge
ing prompts or instruction-tunie offers promising direction for improving reranking accuracy.
6 CONCLUSION AND FUTURE WORK
In this paper, we present techniques for advancing information retrieval using multimodal large lan- guage models (MLLMs). We first study fine-tuning MLLM-based retrievers tackle a general in- formation retrieval scenario: universal multimodal retrieval, where models are required to deal with diverse retrieval tasks, multimodal queries, and documents. Our study shows that MLLM-bane re- trievers exhibit a modality bias in cross-modal retrieval tasks compared to CLIP-based retrievers. To address the issue, we propose modality-awar hard negative mining, which significantly improves our Ml-se- retrievers' accuracies by 5 points in the M-BEIR dataset, benchmark for univer- sal multimodal retrieval. Additionally, with our proposed continuous fine-tuning, our MLLM-based retriever, MM-Embed, is the first model to yield state-of-the-e retrieval accuracy in universal mul- timodal retrieval tasks while maintaining strong text-to-text retrieval capability (ranked top-5 on the MTEB retrieval tasks leaderboard). Finally, we explore prompting MLLMs as rerankers in M-BEIR tasks. We find that MLLMS can be used as zero-shot rerankers to further boost retrieval accuracy in challenging tasks that require the understanding of multimodal queries, such visual question swering and composed image retrieval. For example, our zero-shot MLLL--bsse reranker improves the retrieval accuracy over the state-of-the-at retrievers by over 7 points in CIRCO.
Our work also suggests two promising future directions: (1) Distilling our MLLM-base retriever. MM-Embed, into smaller multimodal retrievers, such as CLIP (Radford et al., 2021) or BLIP (Li et al. 2022). to achieve better retrieval efficiency (see the efficiency comparisons in Appendix A.2
(2) Distilling an MLse- reranker into a retriever to further improve its retrieval capability in tasks involving multimodal queries. Other directions, such as iterative retrieval with relevance feedback (Han al. 2024) and generative retrieval (Qu al., 2024), are also worth exploring for universal multimodal retrieval tasks. In addition, recent work (Ma et al.) 2024a Faysse et al.) (024) has demonstrated that MLLMs can be fine-tuned to tackle visual document retrieval tasks, which could be incorporated into universal multimodal retrieval tasks.
We use 1ava-hf/1lase 6-34t-hf in the experiment.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannanch Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. In Findings of ACL, pp. 3650-3675, 2023.
Payal Bajaj. Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. MS MARCO: A human generated machine reading compre- hension dataset. arXiv:1611.09266 2016.
Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In Proc. ICCV, pp- 15338-15347, 2023.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proc. ICML, pp. 41-48, 2009.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. Proc. EMNLP, pp. 632-642, 2015.
Yingshan Chang, Guihong Cao, Mridu Narang. Jianfeng Gao, Hisami Suzuki, and Yonatan Bisk. WebQA: Multihop and multimodal QA. In Proc. CVPR, pp. 16474-16483, 2022.
Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wi Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? In Proc. EMNLP, pp. 14948 44948-14968, 2023.
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Tuomas Rinta- maki, Mohammad Shoeybi. Bryan Catanzaro, and Wei Ping. NVLM: Open frontier-class multi- modal LLMs. arXiv:2409. 14002 2024.
Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. NV-Retriever: Improving text embedding models with effective hard-negative mining. (o0...2407.il.... 15831, 2024.
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. ColPali: Efficient document retrieval with vision language models, wwXiv:2407.01449 2024.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. DreamSim: Learning new dimensions human visual similarity using synthetic data. In Proc. NeurlPS, pp. 50742-5077777 50768. 2023.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Tevatron: An efficient and flexible toolkit for neural retrieval. In Proc. SIGIR, pp. 3120-3124, 2023.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. AISTATS, pp. 297-304, 2010.
Donghoon Han, Eunhwan Park, Gisang Lee, Adam Lec, and Nojun Kwak. MERLIN: Multimodal embedding refinement via LLM-based iterative navigation for tixt- retrieval-rerank pipeline. In Proc. EMNLP, pp. 547-562, 2024.
Xintong Han, Zuxuan Wu, Phoenix X. Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S. Davis. Automatic spatially-aware fashion concept discovery. In Proc. ICCV, pp. 1472-1480, 2017.
Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. DBpedia-Entity v2: test collection for entity search. Proc. SIGIR, pp. 1268-1268, 2017.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal. Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wo Wei Chang. Open-domain Visual Entity Recognition: Towards recognizing millions of Wikipedia entities. In Proc. ICCV, 2023.

---

Published as a conference paper at ICLR 2025
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie- Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas 2023.
Wang, Timothée Lacroix, and William EI Sayed. Mistral 7B.
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. ES-V: Universal embeddings with multimodal large language models. arxiv:2407. / /250, 2024.
Can Jin, Hongwu Peng, Shiyu Zhao, Zhenting Wang, Wujiang Xu, Ligong Han, Jiahui Zhao, Kai Zhong, Sanguthevar Rajasckaran, and Dimitris N. Metaxas. APEER: Automatic prompt engi- neering enhances large language model reranking. arXiv:2406.14449 2024.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, pp. 535-547, 2021.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proc. EMNLP, pp. 6769-6781, 2020.
Andreas Koukounas, Georgios Mastrapas, Michael Günther, Bo Wang, Scott Martens, Isabelle Mohr, Saba Sturua, Mohammad Kalim Akram, Joan Fontanals Martínez, Saahil Ognawala, Su- sana Guzman, Maximilian Werk, Nan Wang, and Han Xiao. Jina CLIP: Your CLIP model is also your text retriever. arXiv:2405.20204 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, pp. 452-466, 2019.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. NV-Embed: Improved techniques for training LLMs as generalist embedding models. arXiv:2405.17423 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proc. NeurIPS, pp. 9459-9474, 2020.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. PAQ: 65 million probably-asken -sske questions and what you can with them. Transactions of the Association for Computational Linguistics, pp. 1098-1115, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre- training for unified lasion-language understanding and generation. In Proc. ICML, pp. 12888- 12900, 2022.
Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou, From matching generation: survey on generative information retrieval. arXiv:2404.J 14851, 2024.
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. Pretrained Transformers for Text Ranking: BERT and Beyond. Morgan & Claypool, 2021.
Sheng- Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. How Train your DRAGON: Diverse augmentation towards generalizable dense retrieval. In Proc. Findings of EMNLP, pp. 6385-6400, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. Proc. ECCV, pp. 740-755, Cham, 2014.
12

---

Published as a conference paper at ICLR 2025
Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual News: Benchmark and challenges in news image captioning. In Proc. EMNLP, pp. 6761-6771, 2021a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proc. NeurIPS, volume 36, pp. 34892-34916, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, January 2024. URL https:
//llava-v1. git.tiithhbbb. httpo//oo////g//////////
Siqi Liu, Weixi Feng, Tsu-Jui Fu, Wenhu Chen, and William Wang. EDIS: Entity- image search over multimodal web content. In Proc. EMNLP, pp. 4877-4894, 2023b.
Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv, Zhiyuan Liu, and Ge Yu. Universal vision-language dense retrieval: Learning unified representation space for multi-modal retrieval. In Proc. ICLR, 2023c.
Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained vision-and-language models. In Proc. ICCV, pp. 2021b.
Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. Zero-shot listwise document rerank- ing with large language model. arXiv:2305.0211 2023.
Xueguang Ma, Sheng- Chich Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. In Proc. EMNLP, pp. 6492-6505, 2024a.
Xueguang Ma, Liang Wang. Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning LLaMA for multi- stage text retrieval. In Proc. SIGIR, pp. 2421-2425, 2024b.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. WWW'18 open challenge: Financial opinion mining and question an- swering. In Companion Proceedings of the The Web Conference 2018, pp. 1941-1942, 2018.
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. SFR- Embedding-2: Advanced text embedding with multi-stage training, 2024. URL https:
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text embed- ding benchmark. In Proc. FACI, pp. 2014-2001 20377, 2023.
Anastasios Nentidis, Georgios Katsimpras, Anastasia Krithara, Salvador Lima López, Eulália Farré- Maduell, Luis Gasco, Martin Krallinger, and Georgios Paliouras. Overview BioASQ 2023: The cleventh BioASQ challenge on large-scale biomedical semantic indexing and question answering. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pp. 227-250, 2023.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with pre- trained sequeencetto-sequence model. In Proc. Findings of EMNLP, pp. 708-718, 2020.
Leigang Qu, Haochuan Li, Tan Wang, Wenjie Wang, Yongqi Li, Liqiang Nie, and Tat-Seng Chua. Unified text-to-image generation and retrieval. 0585042406005814 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proc. ICMI., pp. 8748-8763 2021.
Samyam Rajbhandari,, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: memory optimizations toward training trillion parameter models. In Proc. of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. EMNLP, pp. 2383-2392, 2016.
13

---

Published as a conference paper at ICLR 2025
Stack-Exchangee Baack--Echange-Coomminn Stack Exchange data dump, Sep 2023. URL https://archive. ottp//ertaiilssttttk----xhhnne
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction-finetuned text embeddings. In Proc. Findings of ACL, pp. 1102-1121, July 2023.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGPT good at search? Investigating large language models as re- ranking agents. In Proc. EMNLP, pp. 14918-14937, 2023.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation information retrieval models. In Proc. NeurlPS, 2021.
James Thorn,, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: large- scale dataset for Fact Extraction and VERification. In Proc. NAACL, pp. 809-819, June 2018.
Henning Wachsmuth, Shahbaz. Syed, and Benno Stein. Retrieval of the best counterargument with- out prior topic knowledge. In Proc. ACI. pp. 241-251, 2018.
Liang Wang. Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models. arXiv:2401.00366 2023.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. UnilR: Training and benchmarking universal multimodal information retrievers.
arxiv:2311.1716,, 2023.
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. The Fashion IQ dataset: Retrieving images by combining side information and relative natural language feedback. In Proc. CVPR, 2021.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Amold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In Proc. ICLR, 2021.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: dataset for diverse, explainable multi-hop question 2018.
answering. In Proc. EMNLP, pp.
Peter Young. Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, pp. 67-78, 2014.
Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming-Wer Wei Chang. MagicLens: Self-supervised image retrieval with open open-ended instructions. In Proc. ICML, pp. to appear. 2024.
Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. PromptReps: Prompting large language models to generate dense and sparse representations for zero-shot doc- ument retrieval. In Proc. EMNLP, pp. 4375 13911, 2024a.
Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. setwise approach for effective and highly efficient zero-shot ranking with large language models. In Proc. SIGIR, pp- 38-47, 2024b.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. IMPLEMENTATION DETAILS
We implement our training and inference using Tevatron (aoo et al. 2023). For CLIP-based retriev- ers, we follow all the settings from Wei et al. (2023). For the MLLM-based retrievers, we fine-tune models with DeepSpeed Zero (Rajbhandari al.) 2020) and gradient checkpointing. During fine- tuning on the M-BEIR training data, we set the maximum length for queries and documents to 128. During continuously fine-tuning on both M-BEIR and text-to-text retrieval training data, we set maximum length for queries and documents to 128 and 512, respectively. All fine-tuning is con- ducted on x A A00 GPUs. Note that image input only occupies a single token length after being tokenized; however, each image will be converted to multiple image tokens. Thus, the actual input length for the MILMs is longer than the maximum length we set. To speed up fine-tuning and inference for MILM- retrievers, only use the global image patches, which occupy 576 (24 x 224) image tokens.
A.2 RETRIEVAL EFFICIENCY COMPARISONS
Table Retrieval efficiency comparisons on M-BEIR dataset.
Storage (GBs)
Latency (ms)
Retriever
Index
Encoding (1x 550h 99th perc.) Vector search
CLIPsF
16
26/ 27 39
BLIPFF
37 / 38 / 44
MM-Embed
86
194
33
Table compares the retrieval efficiency in terms of storage and latency for different retrievers adopted in the paper. We measure the index storage required for the 5.6M documents from the M-BEIR dataset. As for retrieval latency, we measure the latencies of query encoding and vector search. For query latency, randomly sample 100 queries from each test query pool in the 16 M-BEIR tasks and measure the per query encoding and vector search latency with a batch size of 1. Since query encoding latency varies with query length, report the latency th 50th and 99th percentiles. The latency is measured using one thread on a Linux machine with a 2.2 GHz Intel Xeon Silver 4210 CPU and NVIDIA RTX A6000 GPUs, respectively. Note that we perform brute-force search on the sharded index with two GPUs since the full index from UniEmb cannot be loaded into single A6000 GPU.
A.3 BASELINE REPRODUCING
Since we implement our fine-tuning and inference following the settings fine-tuned (CLIP sF) should be equivalent to CL Wei al. (2023). In Table we compare the results of our fine-tuned M (CLIPs)F with the checkpoint pro- vided by the authors 14
Table 9: A comparison of ra (CLIPs) fine-tuned by us and Wei et (2023).
from Wei al. (2023),
Mrand
M (CLIP sp)
Task
Dataset
Wei et al. (2023)
Ours
All
47.4
47.4
M-BEIR Avg. Single-modal Qry
52.5
51.7
multi- modill Qry
39.1
40.1
c1 clip_sf www.sfaargrg... pth
15

---

Published as a conference paper at ICLR 2025
Table 10: M-BEIR dataset statistics.
# QQuery
Relevance Query Candid.
Domain
Train
Dev
Test
Train
Dev
Test
VisualNews Liu et
2021a
News
99K
20K
20K
1.0
542K
11 - clng
MSCOCO (Lp
Misc.
100K
24.8K
24.8K
Fushion200k (Han al
2017
Fashion
15K
1.7K
1.7K
3.3
3.1
2.8
201K
2. 9th + da
WebQA Chang ct al 2022
Wiki
16K
1.7K
2.4K
2.0
2.0
2.0
544K
3. 9t - (etmee c
(Liu et 20235
News
26K
3.2K
3.2K
2.6
WehQA Chang
2022
Wiki
16K
1.7K
2.4K
1.4
1.4
1.4
544K
VisualNews cl 20.2
News
100K
20K
20K
1.0
1.0
1.0
537K
4. gl cM
MSCOCO [1 2010
Mise.
113K
5K
5.0
25K
Fushion200K
Fashion
15K
4.8K
4.8K
61K
5. gmm cmn
NIGHTS (Fu al.
MUR4
Mise.
16K
2K
2K
1.0
1.0
1.0
40K
6. (6 (ime,0a ( d
OVEN Hu al 2026
Wiki
150K
30K
50K
8.5
10.0
9.9
676K
InfoSeck (Chon al. 2023
Wiki
141K
11K
11K
6.5
611K
7. (q (aa)) -
Fushionl(
Fashion
16K
6K
74K
CIRR (Liu arai 202410
Mise.
26K
2K
4K
1.0
1.0
1.0
21K
8. (g-mE,
(elmee 200)
OVEN Fiu al 2046
Wiki
157K
14.7K
14.7K
17.8
17.5
17.7
335K
InfoSeck
2023
Wiki
143K
17.6K
481K
M BEIR (Wei al. 2023
domains
1.M
182K
190K
6.5
5.9
5.7
5.6M
Table 11: Document modality statistics of positive samples in M-BEIR train set. We observe that there more text-modal samples the positive samples of the M-BIER training set. The statistics indicate that text modality has a higher probability of being randomly sampled as negatives than other modalities (i.e., image and interleaved text- eext-image); thus, the modality bias of MLLM- based retrievers is not from the sampling bias of the random sampling strategy.
Document modality
Text
Image
Interieaved text-image
number
568,757
364,903
399,130
percentage
42.6%
27.4%
29.9%
Table 12: A comparison with other existing state-of-the-art multimodal retrieval models on M-BEIR
dataset.
Task
Dataset
E5-V
(Jang al.
2024
MagicLens (Zhang et al. 2024
MM-Embed
All
11.5
5.8
52.7
M-BEIR Avg.
Single-modal Qry
14.6
8.1
56.1
Multi-modal Qry
6.3
2.0
47.0
Table 13: A comparison with other existing state-of-the-art multimodal retrieval models on M-BEIR subtasks in a local evaluation setting (Wei et al. 2023), where retrieval is conducted only among the corpus of each dataset consisting of less than IM candidate documents with single modality.
Task
Dataset
ES-V (Jang
2024
MagicLens Zhang al. 2024]
MM--mmbed
1. gt + cimg
MSCOCO
75.8
82.7
g + cht
WebQA
47.9
96.6
4. gi= + cls
MSCOCO
83.4
17.4
91.0
5. gim + cimg
NIGHTS
26.7
14.1
32.7
(gims, ) + can
FashionIQ
8.4
13.8
26.0
CIRR
30.1
37.5
53.0
16

---

Published as a conference paper at ICLR 2025
Retrieval models' top-1 modality accuracy (M.A. @ 1). We observe that most MLLM- based retrievers suffer from low modality accuracy Task 1 due modality bias, particularly in NVEmb due to its superior text retrieval capability. This issue can be resolved with our modality- aware hard negative mining. Even though MM-Emmed exhibits strong text retrieval effectiveness, no modality bias is observed.
Table 14:
M
Mare
Task
Dataset
MM-Embed
CL
LLaVa-E
LLaVa-P
NVEmb
CLIPss
LaaVa-P NVEmb
VisualNews
0.97
0.82
0.94
0.76
0.98
1.00
1.00
1.00
1. r0th + dim
MSCOCO
0.93
0.80
0.91
0.42
0.98
0,99
1.00
1.00
Fashion200K
0.97
1.00
1.00
0.78
1.00
1.00
1.00
0.99
2. nzt c
WebQA
1.00
1.00
1.00
1.00
1.00
0.99
1.00
3. ge >> (dmm,.ca)
EDIS
0.94
1.00
0.96
1.00
0.90
0.99
1.00
0.99
WebQA
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
VisualNews
0.33
0.90
0.84
0.84
0.97
1.00
1.00
1.00
4. glll -t ct
MSCOCO
0.99
0.99
1.00
0.96
0.94
0.99
1.00
0.99
Fashion200K
0.99
0.99
1.00
0.99
1.00
1.00
1.00
.mm + clm
NIGHTS
1.00
1.00
1.00
1.00
1,00
1.00
0.99
1.00
6. (6/""1,0"")) + e
OVEN
1,00
1.00
1.00
1.00
1.00
1.00
1.00
0.99
InfoScek
0.94
1.00
1.00
1.00
0.97
0.99
1.00
0.99
7. (gfim, (00) + clme
FashionlQ
0.99
1,00
1.00
1.00
1.00
1.00
1.00
1.00
CIRR
1.00
1.00
1.00
1.00
1.00
1.00
1.00
8. (gime, (gt)) + (c c
OVEN
1.00
1.00
1.00
1.00
1.00
1.00
1.00
InfoSeck
0.98
1,00
1.00
1.00
1.00
1.00
1.00
1.00
Table 15: Detailed results on MTEB retrieval tasks.
Model
AA
CF
CQ
DB
Fe
FQ
HQ
MS
NF
NQ
Qu
SD
SF
T2
TC
Avg.
NVEmb Lee et al. 2024
68.2
34.7
50.5
48.3
87.8
63.1
79.9
46.5
38.0
71.2
89.2
20.2
78.4
28.4
85.9
59.4
M
(l-PaVa-P)
48.4
12.9
34.0
34.0
52.2
33.7
50.1
12.3
30.4
36.5
83.8
17.9
72.3
73.4
19.6
40.8
MBnnd
(NVEmb)
$1.5
23.7 43.6
44.9
78.6
46.5
70.2
32.5
38.9
54.1
87.5
20.3
74.5
83.4
23.4
51.6
Mhand
(LLaVa-P)
38.6
20.4
38.0
36.9
78.1
36.2
61.2
23.2
35.1
45.1
86.1
19.2
72.7
27.7
77.2
46.4
Mann (NVEmb)
37.2
30.8
44.0
44.3
86.4
45.5
70.6
34.2
37.4
49.7
86.9
13.9
64.1
23.5
76.7
49.7
MM-Embed
69.0
39.3
49.7
50.6
92.6
60.1
81.4
45.1
40.5
70.6
88.7
21,8
78.3
31.1
85.4
60.3
Dataset Legend: AA=ArguAna, CF=CIimaa--FEVER, CQ=CQADuuptakk, DR=DBPedia, Re=FEVER, FQ=FIQA, HQ=HotpotQA, MS=MMMARCO. NI=NNForpus, NQ-NaturiQQuestionn, Qu=Quora, SD=SCIDOCS, Sc=Sciaact. T2=Touché-2022 TC=TREC-COVID
Table 16: NVEmb (and Va-EE instructions for M-BEIR and MTEB, which are from Wei et al. 2023) and Lee et al. (2024), respectively. For all the candidates, we use the prompt to generate the embedding: < ch n d <eos>.
M.BER task
MizzaNews
Plud ow everydey lange die ant denershld
MSCOCO
coption 'atlarry <g" 5-caes>
Roaed on the julmning fottion dorrijtian mirere bes waiking wuge VaQurry <gg" >icreso
WiRQA
Remie prosager jost Wilpois daa poorie munoee thr fullnnnn quomon untaary S-oma
EDIS
Find a atno image thst warder iis pronded option iraan. e 5oors
WorDA
Witipedda isage duat morern awstin VeQrary
3-0009
VissalNews
caprio for oner is given ploto inthern Sneess
isuge ogeise descriiing folleeing ronslly tsuge HtOwar 5-0ewp
Falin2l0X
NIGHTS
6.(9"5.0"" -5 dM
OVEN
Ininleck
Rominea Wilyydil
qurry de (aQarry >/ac that senidir coneer ttyrr sao "Atuuryy
Slac
7.00"11"") 2PM
CIRK
PahiniQ
Podd juolios tha alige with the
Ainisc day to isage rha oliu mit the andiitaiion intrrciie dhe prosiee isgge W'way < slyc Setows
7
8 (("*"""")
InfoSeck
OVEN
Renies Wilgedia insge-traiiie julr dal proides esience for the prenioe of isige (aQuery 4 >las a >-00
Amorra Wolandia amage dioerptco pair that prosde endene for dhe garmue of Air usge aauarr < 5 Slac S-mev
Taak
Daisari
MTTB bak intractio
AiasAAn
Uneen oloin And disason that roste de clwn siwer
Chivat-FEVERR
Grees rinte about elaut clatgy moneer decaston that sgeer or rnfar the clateinuurp < seoet
Grves gueatee risee deented gusttes doceproer frnm kentge that deplicttrs pos gwaterl-ggurr p-ter
Gives qvery revirne refeew esuly desrippiiee DBPelirls < 5-eees>
REVER
FQQA
Grees Auuncia guomon,
applay that best sunver de gurmin stwnn: 4d S-cre
HepergA
Ghess rubii mop gersion ieview decassur dor on bel ounee de garninl ('wayy " >-cros
ASMARCC
Give web reach garry. mories mensat parsages that otoue the guer) "'wuer"" < ge 5-ceer
NFCoppas
Green gozation remiew reiraat dosusceos duat bot acrer dhe quomee AQuar. d S-cees
Quees
Aad gerstioss
ptherry Greet a irtougic poper aits, rimin pojer dbnner dhat cire by give super-trQweyy i g picrat>
SITIDCS
Seiact
Green u scientji cliis octitee docaments tha supoot or rojnt nDarry i d" Cosss.
Groes queaiven remee deunted pomast arganan that anner gurdion eqwey 5" ><ow>
IREC-DDVVI
guury
quel auraterr
17

---

Published as a conference paper at ICLR 2025
Table 17: Va-P instructions for M-BEIR and MTEB. [image], [text] and [image,text] are used to inform LLaV the user desired modality. For all the candidates, we use the prompt to generate the embedding: dtn > >> ct
> ADescribe the above in one word:
Dntawr
M-RRIR task instraction
[insgge yeAneentr nesiemmdne reption one sond
MSCDCD
jinuge] DeDeeenil everyday coptton wom!
FastionI0KK
[inuge] SeDeecnib the jusiis deacriuiin our wond!
WEBQA
[hatt 4 Vdssser the quistion asing Widjpedia
word
FIDIS
junageetet)) sest/ 44 Stteeeentt the reso-risted raplise one word
WiHQA
jusage tes))
Siydaaner the question asing Wikipolla
Visualleess
Heatt 5t Htlescrthe the news related image one word!
6f4 2
MSCUCU
Hear)
inteecrttee the everyday tagge vs one werd!"
Fashion200g
Hear! i 4 50 iDDescrthe the Sashion image in one vond:
5. 1y
NIGHTS
(unage) 4 > -nffesrebbe the everyday usage oue used!
OVEN
InloScck
[nee] 5. >lec grdh o >'ydamer the question bated on the image from Wilppook in one wand
Fishoon]]
(imgge) 8 > hhrggee the styte of this jiirsfreedshpire <. g"d > >'DDDrennte shis muxdfed shirsdessshpppee in one word. Tinage! 2 (iMosllyy this image with 6" 5 unDesrid moditedd image one word.
CIRR
InfeSeek
OVEN
(enage pin 4tth s'yddamerr the gaestios based or the intereused inage-satt pasagge frome Wilipodie in word:
Arguttaa
Heat < 2000hi crelun, gentrate docament that reiu clam word:
changs
nsgiorts
cisie wond:
CQADugSinkk
(ust)
Sucteschung quezion
D6Palia
SiGhee guery, gennatr minusr enlly deseiisiin fexe 788Paalu voml aupoont refutes claim fuat S'yGhee fiuncial questio vord.
FEVER
FIQA
HotporQA
(was) >'dAamer the molii-hp
wond-
MSMARCD
(wat) >'pdameer the soun
word:
NPCoepue
(waa) >'pdaneer the question
word
NuttniQurttioon
(laat >'ndanner the qussiio
Wilipelii ooe uordl
Quora
(teat) >'pDeeenbb the gvernos one want
SICDOCS
(1aa) >1pGihen ariennfi paper tile gesenste paper abmocc that is cited by the ghos paper in wond.
SeiPart
(1st) S1oGio wriennffc class genmnnte docament that support rnfoar the clain one wand:
Touch 1-3020
(est) > S'AAacerr the gurstion with drinled and persncire orgunents in one ward:
TREC-COVI
(eat) 4d >'ddaner the query on COVID-19 one word.
Table 18: Prompts for reranking tasks in M-BEIR.
Task
Datasel
Prompi
VisualNews
< Ce Vnvewas gt Silttooer the above News image manch the News story? True or Fulse
1. dmg
MSCOCO
>aCCaptoon of "'yDhoes above delly-life image mutch oxpioo? True False > VaDeserpptonnn < g" S'yDooss the above image mutch the cloth stlle description True or False Question: q" > >\achewer:: < d" >>yDDess the answer correctly anser the question? True or False
Pashion200K
2. 2. + dm
WebQA
3. g0 - ("
WchQA
EDIS
Question: < th >iadhawer:
>-pDoes unswer correctly answer the question? True False
VisualNews
< giae > >-NNeewess 2" S >'DDorr the above News image murch the Neus story? Truc or False
4.' (9
MSCOCO
gine >/nCpption d "'nDooes the above daily-lfee image match caption? Trme False
Fashion200K
< gr >ineeccrption 2 nooes above image match cloth style descripiion? True or Folse
5. - aa1.
NIGHTS
< 4 sinc 6o > VaDoes the abous teo images have the nime scene? True or Faliz
("") pM
OVEN
InfaSeck
< nr
>/nQuestionn< 9t sAnneees >Does autwer correctly answer the quistion? True False
7. (g""", (0) - drg
CIRR
FashionIQ
< cv
>raCaution o² Dooss the above coption deserthe the undification of the image? True or False
("") + (cme, (M))
OVEN
InfoSock
>lnQueetion:: 8 SAnnwer: >Does the outwer correctty onswer the question? Trme Folse
18

---

Published as a conference paper at ICLR 2025
Instruction: Find me an everyday image that Instruction: Identify the news-related image matches the given caption.
Instruction: Find me an everyday image that matches the given caption.
line with described Query: The Q Street NW entrance to the Dupont Circle station.
Query: A man brushes his teeth while
Query: A large tow truck towing double decker
woman wraps
Correct Answer
C. Negative e samples
modality
behind
wraps towel around herself.
Dupont Cirole metro station, Street escalator.
A truck towing a double decker
honself in towel in the background.
A man brushing his seech with woman in wrapping
Metro.
Riding escalator to Q Street exit of Dupous Cirole
A tow truck fromt of double docker bus.
C²: Negative samples with unsatisfactory information needs
AN ON
NOVN
Figure 2: Examples of modality-aware negative samples mined by (aa(m)))))) We observe that negative samples with incorrect modality show similar semantic meaning to queries, while negative samples with unsatisfactory information needs provide less accurate information compared to the correct answers.
19

---

Published as a conference paper at ICLR 2025
M-BEIR CIRR Task 7
Query
Answer
Retrieval
Reranking
Human animal from different specie
Sume breed dog, focus on its head.
the white plate white background, clean.
M-BEIR FashionlQ Task
Query
Answer
Retrieval
Reranking
Is shiny and silver with shorter sleeves and fit and flare.
Is grey with black design and is light printed short dress.
solid red color and shorter and tighter with blue and white.
Figure 3: Top-1 candidates for the tasks of composed image retrieval and reranking. In many cases, retrieval and reranking yield different top-1 results from labeled positives, but these results appear to be correct since each query only has single labeled positive candidate (see Table 10..
20


Reviews:
[{"review_id": "ptRkVrM0d3", "reviewer": "Reviewer_RfVM", "summary": "In this paper, the authors focus on developing multimodal LLMs (MLLMs) for universal multimodal retrieval. Instead of directly fine-tuning MLLM-based bi-encoder retrievers with instructions as a guide on multiple multimodal retrieval tasks, the authors propose modality-aware hard negative mining and continual text-to-text retrieval fine-tuning to address the modality bias problem from MLLMs. Furthermore, the authors explore to prompting MLLMs as zero-shot rerankers, which can further boost retrieval accuracy in complex retrieval tasks. Extensive experimental results demonstrate the effectiveness of the proposed method.", "questions": "please try to address the weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1)\tThis paper explores the use of multimodal LLMs for universal multimodal retrieval, which is an interesting and potential research topic for the retrieval field.\n2)\tIn this paper, the authors explore universal multimodal retrieval from three aspects, including modality-aware hard negative mining, continual text-to-text retrieval fine-tuning, and prompting MLLMs as zero-shot rerankers. These components are designed to improve the effectiveness of MLLM-based universal multimodal retrieval, while maintaining strong text-to-text retrieval capability.\n3)\tExtensive results demonstrate the effectiveness of the proposed method.", "weaknesses": "1)\tThe details of the proposed method are not clear and confusing. In sec. 4.1.1, it seems random negatives are denoted as c_i^{\\times} symbol, while the generated triplets use c_i^{\\times} again, does it mean positive sample or random negative?\n2)\tIn sec. 4.2, the authors prompt LLaVa-Next for reranking, while not introducing the detailed MLLMs adopted in sec. 4.1. Are there two multimodal LLMs needed for the proposed method? \n3)\tIt will be better to provide the framework figure for the whole method. \n4)\tIn Table 4, it is not clear which results denote the reranking results, it seems the last row does not outperform the fifth row.\n5)\tAs a retrieval method, the efficiency of retrieval time and memory is as important as effectiveness. The authors should analyze the efficiency of the unimodal retrieval mechanism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZiVA3OQ8Vy", "reviewer": "Reviewer_yjMm", "summary": "This paper explores a more practical scenario in information retrieval, namely general multimodal retrieval. The author proposes a new fine-tuning pipeline and re-ranking method to improve the performance of specific retrieval tasks. However, the organization of the experimental results of this paper is rather coarse, and the performance presented makes me concerned about its generalization and robustness. It is recommended that the author further improve the presentation of results and methods. However, it is undeniable that the research topic can bring contribution and inspiration to the multimodal community.", "questions": "See weaknes.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The topic explored by the authors is a great one, namely general multimodal retrieval.\n2. The method proposed by the author is effective to a certain extent.\n3. The authors describe the experiment in detail and I think it is reproducible.", "weaknesses": "1. Unfortunately, the authors are not the first to use MLLM for re-ranking. It is recommended that the author faithfully investigate related research, e.g., [1]. The authors should narrow down the scope of the study, i.e., composed image retrieval.\n2. The author's claim in line 184 should be verified experimentally using either very large k or very small k.\n3. Line 306: The author's fine-tuned batch size for the two types of models is different. Will this affect the effect of mining negative samples?\n4. I personally think that CLIP is a product from 3 years ago, and more advanced models should be used, such as BLIP or an improved version of CLIP. Moreover, the comparative experiment lacks experimental results of some recent methods that solve specific tasks, and it is impossible to quantitatively feel the advantages of the proposed MLLM-based method. The current organization is more like a complete ablation experiment.\n5. From the results in Table 4, the full version re-ranking results do not seem to be optimal, which makes it difficult to explain the effectiveness of the method's continuous fine-tuning on some tasks. The improvement margin of its full version seems to be even worse than M^hard, which makes it hard to convince me.\n6. From the studied results, the author expects to achieve universal retrieval, but as far as the experimental results (e.g., Table 7) are concerned, the proposed method will reduce the performance in some benchmarks. This makes me concerned about the generalization and robustness of its method.\n\n[1] MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E91O6G2iiG", "reviewer": "Reviewer_ESAe", "summary": "The paper presents techniques for advancing information retrieval with multimodal large language models (MLLMs). It first finetunes MLLM-based retrievers to tackle universal multimodal retrieval tasks and shows that MLLM-based retrievers exhibit modality bias in cross-modal retrieval tasks compared to CLIP-based retrievers. To address the issue, it proposes modality-aware hard negative mining and continual fine-tuning, and the MLLM-based retriever, UniEmb, yields state-of-the-art retrieval accuracy in universal multimodal retrieval tasks while maintaining strong text-to-text retrieval capability. Moreover, it explores to prompt MLLMs as a re-ranker to further boost retrieval accuracy.", "questions": "Please refer to the Weaknesses section and address my concerns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper is well-motivated to use multimodal LLMs for advancing information retrieval and support broader retrieval scenarios.\n2.\tThe experiments are relatively comprehensive to validate the effectiveness of the proposed method.", "weaknesses": "1.\tThe paper claims that existing retrieval models typically addressed search scenario where retrieval task are fixed and only a single modality is supported for both queries and retrieved results, but a series of retrieval models for multimodal retrieval are listed in related work.\n2.\tThe paper claims that it is the first work to explore prompting MLLMs fir zero-shot reranking, but as far as I know, the work [a] has used MLLMs for reranking, and the differences should be explained in detail.\n3.\tSome expressions are confused. For example, in Line 308 of Page 6, “we initialized retriever using the pretrained model”, does the “retriever” refer to the model to be fine-tuned with hard negatives?\n4.\tIt is not easy to understand Figure 1, where the task definition of Task 1-13 is missing. The relationship between the upper left and lower right sub-figure is not clearly illustrated.\n5.\tHow is the performance when using fine-tuned MLLMs for reranking? Is it better than reranking in a zero-shot manner for tasks that involve single-modal queries?\n6.\tMore qualitative cases are suggested to be provided to better illustrate the effects, such as how modality-aware hard negative mining alleviates modality bias. \n\n[a] Qu L, Li H, Wang T, et al. Unified Text-to-Image Generation and Retrieval[J]. arXiv preprint arXiv:2406.05814, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WurUS0FsnD", "reviewer": "Reviewer_8yb7", "summary": "This paper proposes a general retrieval method for multimodal retrieval tasks, named UniEmb, based on multimodal large language models. The paper identifies the phenomenon of modality bias in multimodal retrieval and finds that current retrieval methods based on multimodal large language are lacking in text retrieval performance. To address these issues, the authors propose a continued fine-tuning method based on hard negative samples, as well as a strategy that uses multimodal large language for further re-ranking. The proposed method is comprehensively evaluated on two widely adopted datasets, M-BEIR and MTEB, with results showing that UniEmb improves text retrieval performance while maintaining the effectiveness of multimodal retrieval.", "questions": "1. In Table 1, why is the M^rand column for MTEB Text Retrieval empty? Are these models not evaluable on MTEB? I believe that comparing M^rand with M^hard would more effectively demonstrate the modality bias issue and the effectiveness of the proposed method.\n2. In Table 4, why does re-ranking show limited improvement for knowledge-related multimodal retrieval datasets like OVEN and InfoSeek? and even a decline in performance on some tasks (e.g., task1 and task3)? How do the authors explain this phenomenon, and could it be related to the modality of the retrieval task?\n3. In line 343, the authors mention that MLLM-based retrievers tend to retrieve text over images. Could this be related to the sampling of negative samples during training? If there are more image samples in the dataset, this could lead to a higher proportion of images as negative samples, which could further induce bias. I suggest conducting a statistical analysis on the modality distribution of the data samples.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper identifies an interesting issue of modality bias in multimodal retrieval based on multimodal large language models, and it proposes an effective hard sample mining strategy to address this issue while preserving text retrieval capabilities.\n2. The paper introduces a zero-shot re-ranking approach using multimodal large language model, which, to some extent, improves multimodal retrieval accuracy.\n3. The evaluation is thorough, using widely adopted M-BEIR and MTEB datasets, and the paper provides detailed parameters for reproducibility, lending credibility to the experimental results.", "weaknesses": "1. The proposed methods for hard sample mining and re-ranking using large multimodal models are straightforward and intuitive, and may lack some novelty.\n2. There is no direct comparison with similar baseline models, such as E5-V, on the M-BEIR dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)