U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs
Authors: Xiaojie Li
Chu Li
Shi-Zhe Chen
Xi Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Universal multimodal retrieval
Multimodal large language models
Embedding
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Universal multimodal retrieval (UMR), which aims to address complex retrieval tasks where both queries and candidates span diverse modalities, has been significantly advanced by the emergence of MLLMs. While state-of-the-art MLLM-based methods in the literature predominantly adopt contrastive learning principles, they often differ in their specific training recipes. Despite their success, the mechanisms underlying their retrieval capabilities remain largely unexplored, potentially resulting in suboptimal performance and limited generalization ability. To address these issues, we present a comprehensive study aimed at uncovering the key factors that drive effective embedding learning for UMR using MLLMs. We begin by implementing a general MLLM-based embedding learning pipeline, and systematically analyze the primary contributors to high-performing universal retrieval systems. Based on this, we explore various aspects of the details in embedding generation and training strategies, including progressive transition, hard negative mining and re-ranker distillation. Notably, our findings reveal that often-overlooked factors can have a substantial impact on model performance. Building on these discoveries, we introduce a unified framework termed U-MARVEL (Universal MultimodAl RetrieVal via Embedding Learning), which outperforms state-of-the-art competitors on the M-BEIR benchmark by a large margin in supervised settings, and also exhibits strong zero-shot performance on several tasks such as composed image retrieval and text-to-video retrieval. These results underscore the generalization potential of our framework across various embedding-based retrieval tasks.

Published as a conference paper at ICLR 2026
U-MARVEL: UNVEILING KEY FACTORS FOR UNI- VERSAL MULTIMODAL RETRIEVAL VIA EMBEDDING LEARNING WITH MLLMS
Xiaojie Li
Chu Li
Shi-Zhe Chen
Xi Chen
Tencent PCG Nanjing University *ByteDance
{chaxili, shi shizhechen, www.nuccceeeeeenee.....m.
Lichuebytedancce.cm
ABSTRACT
Universal multimodal retrieval (UMR), which aims to address complex retrieval tasks where both queries and candidates span diverse modalities, has been signif- icantly advanced by the emergence of MLLMs. While state-of-the-art MLLM- based methods in the literature predominantly adopt contrastive learning prin- ciples, they often differ in their specific training recipes. Despite their suc- cess, the mechanisms underlying their retrieval capabilities remain largely un- explored, potentially resulting in suboptimal performance and limited generaliza- tion ability. To address these issues, we present a comprehensive study aimed uncovering the key factors that drive effective embedding learning for UMR using MLLMs. We begin by implementing general MLLM-based embed- ding learning pipeline, and systematically analyze the primary contributors to high-performing universal retrieval systems. Based on this, we explore vari- ous aspects of the details embedding generation and training strategies, in- cluding progressive transition, hard negative mining and re-ranker distillation. Notably, our findings reveal that often-overlooked factors can have a substan- tial impact model performance. Building these discoveries, a unified framework termed U-MARVEL (Universal MultimodAl Retrie via Embedding Learning). which outperforms state-of-the-art competitors the M- BEIR benchmark by a large margin in supervised settings, and also exhibits strong zero-shot performance on several tasks such as composed image retrieval and text- to-video retrieval. These results underscore the generalization potential of our framework across various embedding- based retrieval tasks. Code is available https: //tpt/gggtttt/tt//// com/ chax /1/UMMARVEI
introduce
INTRODUCTION
Multimodal information retrieval (Liu et al. 2022) has emerged as pivotal research direction at the intersection of multimodal understanding and information retrieval. It serves cornerstone for wide range of modern Al applications, including retrieval-augmenter generation (RAG) (Jiang
al. 2023 Cong al. 2023), text-image retrieval (Baldrati et 2023 Zhang et al. 2024b). and visual question answering (VQA) (Chow et al.] 2024 Chun et al. 2021). While existing methods such as CLIP (Radford et al. 2021). BLIP (Li et al. 2022 2023a), and CoCa (Yu et al.) 2022) have demonstrated impressive performance in cross-modal retrieval, they often struggle to address the diverse and complex requirements encountered in real-world scenarios, ranging from fine-grained instruction following multi-turn interleaved interactions.
To tackle these challenges, the research community has increasingly focused on universal multi- modal retrieval (UMR) (Wei al. 2024), depicted in Fig. [[),, which aims to develop unified, cnstruction-uudddd retrievers capable of handling diverse retrieval tasks across multiple modalities. Recent advancements in this area, such as LamRA (Liu al. 2024b), MM-Embed (Lin al. 2024).
'Equal contribution, work done Tencent PCG.
I Project lead.
1 "oorrsponniigg authors.

---

Published as a conference paper at ICLR 2026
Instructio::
Contractive
How to distil one model?
Find on image dust meets the requinmmmets Querys make yellow
negative mining?
Leariig
Candidates pool
How generate embeddings?
LLM LORA
Encwlledgs Disillation
YES
NO
Iestraction:
projector
Tokenizer
LLM
LORA
Image
hyperparameters?
Vision Encoder
000 projector
g Q
Tokenizer
Instruction:
snower
Bo Tony Stari
How to design transition strategles?
PROMPT]
Vision Encoder
Query:
U-MARVEL
Image
Instruction
Text
Query
Candidate
Recall Model
Rerank Model
(a) Universal Multimodal Retrieval
(b) The Framework for U-MARVEL
Figure I: U-MARVEL Exploration: We conduct a comprehensive study on effective design princi- ples for embedding model architectures and optimal strategies for training embedding models.
GME (Zhang et al.) 2024d). UniME (Gu al. 2025) and VLM2VEC (Jiang et al. 2024b), have shown promising results on UMR powered by MILMs. However, most of the existing approaches often directly adapt MILMs to embedding tasks without systematically examining training schemes tailored for MLLM-based embedding models. Consequently, the impact of design decisions on re- trieval performance remains inadequately understood, highlighting the need for more systematic investigation-a gap that motivates our study.
In this paper, aim address these issues by systematically exploring the design principles for building high-performing universal multimodal retrievers with pre-trained MLLMs (Fig. [b)). To this end, we first implement simple yet effective embedding learning pipeline based on contrastive learning, following the existing approaches in the literature. Subsequently, we conduct a compre- hensive study to investigate three key aspects for training high-performance retrieval models built on MLLMs. Specifically, we explore: (I) how to adapt decoder-only MLLMs into instruction-guider embedding models; (2) how to train MLLM-based embedders within the context contrastive learning framework; and (3) how to distill the recall-then-rerank paradigm into single model to further enhance retrieval performance. Our findings highlight several overlooked factors critical to retrieval performance, including the superiority of bidirectional attention with mean pooling over last-token mechanisms, and the interactions among parameters of hatch size, learning rate, and temperature coefficient in contrastive learning. We also discover that integrating hard negative min- ing and knowledge distillation effectively mitigates computational inefficiencies in the recall-then- rerank pipeline, while simultancously improving retrieval efficiency and accuracy.
Motivated by these insights, we present U-MARVEL, a unified framework for learning universal multimodal embeddings tailored UMR tasks. Our method achieves significant improvements over state-of of-the-art UMR approaches on the M-BEIR benchmark (Wei et al. (2024) in supervised settings, and also demonstrates strong zero-shot generalization abilities on tasks such as text-to- video retrieval and composed image retrieval.
The contributions of this paper can be summarized as follows. (1) We conduct comprehensive
exploration of the design space for MLLM-based universal retrieval, uncovering key factors that improve performance and providing actionable insights for future research. The findings presented in this paper aim to advance the effectiveness of UMR tasks within the research community. (2) We introduce U-MARVEL, a unified framework that achieves tate-of--hhe-ar performance both supervised and shot settings, showcasing its effectiveness across wide range of retrieval tasks.
RELATED WORK
Multimodal Large Language Models. Large language models (LLMs) (Achiam et al. 2023
Yang et al. 2024| Liu et al. 22024a Touvron et al. 2023), including multimodal LLMs (Zhang al. 2024c Yao et al. 2024h, have achieved remarkable success across a wide range general tasks. For instance, LLaVA (Liu al. 2023a) integrates image and text processing for tasks like visual question answering, image description, and visual reasoning. The Qwen-VL series (Wang
et al. 2024a Bai et al. 2025) enhances this by supporting flexible image resolutions and extending
2

---

Published as a conference paper at ICLR 2026
capabilities video understanding through temporal modeling and localization. IntervVL3 (Zhu iet al. 2025) unifies multimodal and linguistic learning by leveraging both multimodal data and text corpora in single pre-training stage.
Multimodal retrieval aims to align diverse modalities (e.g., text, images, or their combinations) within a shared embedding space for semantic similarity matching. Early works (Radford et al. 202T; Li et al.; 2022; Yu et al. (2022) leveraged large-scale contrastive learning to align image-tex( text pairs, enabling zero-shot classification and gross-modal retrieval, but struggled with complex multimodal inputs. Recent advancements have addressed these limitations through innova- tive approaches based MILMMs (Jiang et al. 2024a Wei et al. 2024 Jiang et al. 2024b Lan et al,, 2025). LamRA (Liu et al. 2024b) adapts MILM into retrievers using lightweight LoRA modules with joint pointwise and listwise reranking; MM-Embed (Lin et al. 2024) improves alignment via MILM fine-tuning and modality-aware hard negative mining: GME (Zhang et al. 2024d) devel- ops unified multimodal retrieval framework leveraging MLLMs with synthetic data augmentation; and UniME (Gu et al., 2025) combines textual knowledge distillation from LLM teachers with hard negative-enhanced instruction tuning, achieving robust cross-modal alignment for diverse retrieval tasks. While these works provide a strong foundation for UMR, the design space for MLLLM-bssed embedding models remains underexplored. Unlike LLMs that generate outputs via autoregressive decoding, embedding models encode features of the entire input sequence, necessitating specialized strategies for feature extraction and tailored training methodologies. This work addresses these gaps through a systematic investigation of the design decisions for MLLM-basee embedding models.
Multimodal Retrieval.
RECIPE FOR BUILDING U-MARVEL
In this section, we conduct a comprehensive study identify the key factors that drive effective embedding learning for UMR through MLLMs. Following the mainstream approaches, we first establish simple pipeline. Unless stated otherwise, we finetune the pre- pre-trained
Twen2-VL-7BB
Instruct (Wang et al. 2024a) as the embedding model with LoRA (Hu et al., (2022) by contrastive learning on M-BEIR dataset (Wei et al. 2024). Specifically, the InfoNCE loss (Oord et al. 2018
selected as our training objective:
exp(sim(e,) t/T))
LIsfONCE: = log Σ. exp(sim(eg ee)/T)
(1)
where eg. et, represent the embeddings of the query, the positive candidate, and any sample from the candidate set, respectively. 5 the temperature parameter. sim(..-) means the cosine similarity; More details are given in Appendix [C] Throughout the ablation studies, we report the Average Recall as the evaluation metric, which computed as the mean of Recall or Recall 10 across the benchmark tasks (c.f. Appendix A. We then explore three major axes of design decisions the following subsections:
We investigate the adaptation of decoder-only MILMss into instruction-awart embedders.
We explore how to train embedding models within the framework of contrastive learning.
We validate that the retrieval performance can be further enhanced through distillation from the recall-then-rerank paradigm.
3.1 How TO ADAPT MILMS INTO EMBEDDING MODELS?
LLMs generate output tokens in an autoregressive manner, whereas embedding models encode holis- tie representations of input sequences, reflecting fundamental differences their training objectives. Adapting LLMs into embedding models thus requires specialized training strategies. In this subsec- tion, we examine this adaption from three aspects as follows.
3.1.1 EMBEDDING EXTRACTION
Finding 1: Generating embeddings with bidirectional attention and mean pooling outperforms the common approach of using compression prompts with the last token mechanism.
3

---

Published as a conference paper at ICLR 2026
We first investigate the method for extracting embeddings of the token sequence of multimodal queries using MLLMs. Following prior approaches, the instruction is directly concatenated with the query. Existing methods can basically be divided into two categories based the way embeddings are extracted. (1) Last token: previous works (Liu et al. 2024b Jiang et al., 2024b) Zhang et al.) 2024d, Diang et al., 2024a Gu al., (025) typically employ prompt for compression. Generally, for multimodal input, the prompt is "<image> <text>... <image>><>>> Sreema-Smarrize above image and sentence in one word: emb", where <image> and <text>> are the placeholders. In this way, the feature of last token (i.e, <emb>> is used as the output embedding of the input sequence. (2) Mean token: an alternative method, adopted by MM LMM-Ebeed (Lin et al.] (024). transforms the unidirectional attention mechanism into a bidirectional one and applies mean pooling to the features from the last hidden states to derive the embedding for the entire sequence.
Table 1: Comparison of last token and mean token mechanisms
Causal or Bidirectional
Last
Compression Prompt ("in one word") )
Local Avg.
Global
Mean token
Avg.
Causal Bidirectional
Last token
56.6
54.8
Last token
55.5
$2.6
Causal
Mean token
33.7
27.6
Bidirectional
Bidirectional
Mean token
X
57.2
55.2
We conducted detailed comparative experiments, as presented in Table (1) The experimental re- sults reveal that the effectiveness of the compression prompt exhibits dual dependency. On one hand, it establishes a strong coupling with the last token mechanism, as evidenced by ID-0 and ID-2, where replacing last token with
token results in a significant performance drop of 22.9% 27.2%. On the other hand, causal attention enhances the impact of the compression prompt (ID--) vs. ID-1) bidirectional attention.
demonstrating 1.1 %/2.2% improvement tional attention partially decouples the dependency between the compression prompt and the last to- ken mechanism. For instance, ID-3 shows an 18%/18.55 improvement compared ID-2. However, when combined with mean pooling, bidirectional attention remains constrained by the limitations of the compression prompt. (3) Furthermore, when compression prompt removed from bidirectional attention with mean token mechanism (ID-4), it achieves a modest improvement of 0.6%//0.49 over ID-0, which is commonly used in the literature.
(2) Interestingly, bidirec-
In conclusion, generating embeddings using bidirectional attention combined with mean pooling from the sequence demonstrates superior performance compared to the commonly employed ap- proach of combining compression prompts with the last token mechanism. This difference may stem from the last token embedding being affected by recency bias, leading to an excessive reliance on the output of the final token. Notably, this finding aligns with the conclusions of NV-Embed (Lee) jet al. 2024), yet diverges from those presented in GME (Zhang et al. 2024d), offering a distinct perspective architectural design and performance optimization.
3.1.2 INSTRUCTION INTEGRATION
Finding 2: Masking instruction tokens during mean pooling enhances embedding performance.
Motivated by prior studies on text embeddings (Lee al. 2024 BehnamGhader et al. 2024 we mask out the instruction tokens during the mean pooling process, as these tokens have already in- fluenced the output features through self-attention. Experimental results show that ID-0 achieves 0.1 1/0..%%%% improvement compared to ID-1 and 2.0% improvement compared to ID-2 in Table 2 incorporates instruction information during forward propagation thought bidirectional self-attention mechanism. Therefore, by filtering out instruction tokens and focusing on feature comparisons between the query and candidate, the approach mitigates calculation bias. While the numerical improvement is modest, this step effectively eliminates theoretical instruction bias without compro- mising performance.
We hypothesize that this improvement arises primarily from that: the query inherently
4

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of instruction integra-
Table 3: Comparison of progressive transi-
tion
tion
ID
Instructions
Masking
Local Avg.
Global Avg.
Methods
Local Avg.
Global Avg.
V
57.3
55.5
Instruction Tuning M-BEIR
56.6
53.9
V
57.2
55,2
[0] + text-only retrieval
57.3
55,5
X
55.3
43.4
2
[1] + text image retrieval
57.7
55.8
3.1.3 PROGRESSIVE TRANSITION
Finding 3: Progressive transition effectively adapts decoder-only MLLMs to embedding mod- els through stepwise training.
We set a baseline by vanilla instruction tuning on multimodal retrieval data from the M-BEIR dataset, presented in Table under ID-0, Following prevalent approaches (Liu et al. 2024b Jiang et al. 2024a), we use text-only retrieval data for pre-training, which improves performance on vari- retrieval tasks. Training the NLI dataset (Gao al. 2021) significantly enhances model's performance, as indicated by ID-1 in Table 3 Despite these improvements, a notable gap persists when transitioning from single-modality, single-instruction tasks multimodal, multi-instruction tasks. To address this gap, we propose a progressive approach incorporating an additional pre- training step using text-image retrieval data from the CC3M dataset (Sharma et al. 2018). This step further enhances performance, as demonstrated by ID-2 in Table 3 Since LLM was origi- nally trained with causal attention, switching to bidirectional attention degrades text-visual encoder alignment. We speculate that text-based fine-tuning strengthens the text encoder's semantic rep- resentation, while image-text pairs further align the text-visual encoders. During experiments, we train the model with NLI using unidirectional InfoNCE, followed by CC3M using bidirectional In- foNCE. Furthermore, we further analyze the impact of training data selection in Appendix D which reveals that the concise text in CC3M aligns better with retrieval tasks compared to other datasets (c.g., ShareGPT4V (Hurst et al.
2024) and TART (Asai et al. 2023
Therefore, we formally propose the progressive transition method, adopting stepwise training strategy to adapt decoder-only MILMs for embedding tasks. Following the principle of advancing from simpler more complex tasks (Bengio al.] (2009), this phased approach ensures smooth transition through three key steps: (1) Text Retrieval Adaptation: Learning retrieval tasks using text-only datasets establish foundational capabilities. (2) Cross-modal Alignment: Advancing to cross cross-modal retrieval tasks through text-image paired data. (3) Instruction -unee Multimodal Retrieval: Mastering cettructtionguudee. multimodal retrieval tasks with comprehensive training on multimodal datasets.
3.2 How TO TRAIN MLLM-BAN -BASED EMBEDDERS BY INFONCE?
In this subsection, we investigate how MLLM-based embedding models should be trained under the contrastive learning framework, following the paradigm adopted by most existing methods, Despite its simplicity, we identify that several overlooked factors exert a critical impact on the final perfor- mance, including interactions among the training parameters of InfoNCE and the control of noise in hard negative mining during the continual training process.
3.2.1 INTERACTIONS AMONG LARGE BATCH SIZE, LEARNING RATE AND TEMPERATURE
Finding 4: Increasing batch size yields performance gains, but these improvements plateau without appropriate learning rate scaling. Additionally, learnable temperature parameters play a pivotal role enhancing the effectiveness of contrastive learning.
Our experimental results dispute the widely held belief that "increasing batch size directly improves performance" indicating that this assumption requires closer examination. By analyzing the results presented in Table 4 we identify three key findings follows. (1) Experiments with ID-0, ID-2, and ID-4 indicate that increasing the batch size does improve model performance; however, the performance gains tend to plateau the batch size continues grow, as observed in comparisons
5

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of performance when increasing batch size
ID
Batch Size
Tomp (r)
LR (1)
Local Avg.
0 480
0.05 (lixed)
1 10-4
57.2
1 1920
0.05 (fixed)
1 10
57.4
2 1920
0,05 (fixe)
2 10
58.3
3 3840
0.05 (fixed)
2.8 x 10
58.5
3840
0.05 (fixed)
x 10
58.9
5
5760
0.05 (fixed)
6 10
58.7
7680 6
0.05 (fixed)
6 1071
58.9
7
10 8 7680 0.05 (leamable)
0.05 (leamable)
4 x 10
60.3
between ID-5 and ID-6. (2) Consistent with prior research (Goyal et al., 2017) that highlights the necessity of scaling the learning rate (Ir) alongside batch size, contrastive learning experiments reveal similar trends. Specifically, simply increasing the batch size without adjusting the learning rate results in marginal improvements (ID-0 ID-1). In contrast, applying scaling rule (ID-2) significantly boosts performance to 58.3%, which aligns with the findings from previous works. (3) Our investigation into the effects of temperature parameters reveals that employing a learnable temperature significantly enhances performance. The learnable mechanism effectively optimizes the sharpness of the probability distribution during training, superior fixed temperature settings used in prior works such as LamRA (Liu et al.) 2024b) and MM-Ember (Lin et al. 2024). Empiri- cal results (comparing ID-4/II-66 with ID-7/ID-8) demonstrate that this dynamic adjustment yields consistent performance gains (e.g., +1 1 .44% in Local Avg), while also exhibiting stable and adaptive convergence across different batch sizes. More detailed ablation study on learnable temperature is provided in Appendix D
3.2.2 CONTINUAL TRAINING WITH HARD NEGATIVE MINING
Finding 5: Hard negatives may hinder convergence during training. Filtering false negatives and mixing random in-batch negatives help balance difficulty and improve performance.
The initial random negative sampling yields basic discriminative ability but struggles with challeng- ing false positives: negatives that exhibit high semantic similarity yet are incorrect matches. Recent studies have employed hard negative mining (Lin et al. 20244] Lan et al. 2025) or hard negative reweighting in loss computation (Hou & Li 2023: Radenovic et al. 2023 Zhuang et al., 2024) to address this issue. In this work, we explore related settings. However, as shown in Table for ID-0, directly selecting the top-k hard negatives proved to be excessively challenging, resulting in complete failure. Simply setting & to a larger value (e.g., 50) prevents model collapse but fails to yield any performance improvement. Additionally, incorporating in-batch negatives results in a per- formance decline compared the baseline in this table, observed by ID-1. We hypothesize that some hard negatives are false negatives, which can adversely affect model training. Specifically, naively maximizing the loss all hard negatives often leads to model collapse because the model is forced to push away semantically valid candidates that are labeled as negatives due to dataset noise. To address this issue, we propose straightforward filtering method that eliminates hard negatives with scores exceeding a predefined threshold. This approach led to significant improvement, as demonstrated by ID-2.
We apply a hard negative mining strategy as follows: (1) Feature Extraction: Using the model from the previous training stage, we extract features for all queries and candidates, compute similarity scores (excluding positive matches), and rank candidates to identify the most challenging negatives per query: (2) Filtered Hard Negative: For each query, we filter out negative samples that exceed predefined threshold (considered as false negatives) and then select the top-k negative samples as hard negatives; (3) Balanced Training: To prevent model convergence difficulties from excessive hard negatives and to accelerate training, we select a suitable k-value and mix these hard negatives with other in-batch negatives. We then perform continual finetuning by InfoNCE. In our experi- ments, we set = 5 and the threshold to 0.7, filtering out any negative samples with scores ceeding this limit. The choice of for hard negative mining strikes a balance between training efficiency and sample quality.

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of different strategies in hard negative mining
Methods
Local Arg.
Global Avg.
Progressive Transition (buseline)
in batch neg
60.6
58.7
only top-k & hard neg
failed
failed
in-batch neg top-k bard neg
57.4
55.4
in-batch nog filtered top-k & hard neg
61.7
59.9
Teacher
cand1]
@ 50
Retrieval model
top-k
Reranking model
cand2
49
Rerank
36
query candidates
ca md
pos
query
D: in batch neg
A pos
Traditional
Improved
hard neg
Distilation
Distilation
blank: ignored + hard neg
0 0
Figure 2: Distillation Illustration. It shows how the teacher model generates scores, and compares the traditional and improved distillation methods.
3.3 WILL RERANKER DISTILLATION IMPROVE PERFORMANCE?
Finding 6: The improved distillation approach reduces computational overhead while increas- ing feature diversity, thereby enabling resource- -effrrceeef and effective distillation.
Some studies (Liu al. 2024b Lin 2024) further refine retrieval results by applying rerank- ing step on top of the recall model. The reranker can be either a trained model (Liu et al.) 2024b) or a zero-shot reranker (Lin et al.] (024) based on MILMs. While this recall-then-rerant strategy often improves retrieval performance, it introduces additional inference latency and increases sys- tem complexity. In this work, we exploit to distill this cascaded pipeline into a single unified model, thereby simplifying the system while maintaining high performance.
Inspired by (Liu et al., 2024b), we train a generative reranker based on the decoder-only MILM architecture. The training data is prepared by: (I) extracting embeddings for all queries and can- didates via the recall model from Sec. 3.2.2 then (2) constructing pairs for each query consisting of a positive match and the top-50 most challenging negatives. The model is prompted to output "YES" for positives and "NO" for negatives, optimized with next-token prediction loss to maximize the likelihood of correct response. More detailed specifics can be found in Appendix [
After training the reranker, we employ it to refine the ranking results from the recall model, further improving the retrieval performance. Specifically, the recall model from Sec. 3.2.2 first retrieves the top-k candidates, then each (query, candidate) pair is processed by the reranker sequentially. The reranker is prompted to judge each candidate with "YES" or "NO" response, where the probability of "YES" serves as the reranker's 's score. To further enhance overall performance, we combine the scores from both models through linear interpolation: Smalt: = a S + (1 - a) Srerank, where Smult: becomes the final relevance score, and a is weight hyperparameter (fixed at 0.5). This completes our recall-then-rerank pipeline, producing optimized query-candidate matching scores.
We distill this entire pipeline as teacher model into single student model, shown in Figure 2 where the combined knowledge is compressed into one unified model by using KI. divergence de- fined as Eq. 2
Σ Smudts log Smatts (i)
Cdissill DKI
|| Ssingle)
Ssingte(i)
where Smult: represents the multi-model ensemble scores and Saingle the single model's predictions. before computing the KL divergence. In traditional distillation
The scores are frameworks, the base model typically aligns with the reranker. Distillation samples are structured as (query, pos), with similarity matrices incorporating in-batch negatives. However, recall- -hen- rerank inference over this similarity score matrix incurs prohibitive computational costs, making impractical for deployment.
7

---

Published as a conference paper at ICLR 2026
We propose improved distillation method to address this limitation. Specifically, we construct samples in the form of (query, positive, top-k hard negatives), and confine distillation to the scope of positive and hard negative samples. In this way, it significantly reduces computational overhead while substantially increasing the diversity of features encountered during model training. Thus, it achieves both resource efficiency and performance gains critically, it makes distillation practically feasible. Furthermore, besides the data and base model, a key factor contributing to the effectiveness of this distillation method lies in its tight alignment with the recall-then-rerant inference pipeline, depicted in Figure 2 (1) Retrieving with a hard negative model; (2) Reranking the top-k results and generating integrated scores. Our distillation method mirrors this workflow: it adopts the hard negative model as the base and distills top-k scores from the integrated model.
To further validate its efficiency and effectiveness, we provide the following mathematical analysis. Let n. denote the number of queries in batch, and N the total number of queries. In traditional distillation, the computational complexity of inferring the similarity matrix is N (2n + n²) = (2++)), and the number of features observed per training epoch is 2NN. The total time for inference and training is (6 + N. In our improved method, the cost of computing the similarity matrix is reduced to N. x (m(k + 3)) = (k + 3), and the number of features observed during training increases to (k: + 2). The total time is reduced to (3k + 7).N. Substituting our actual parameters, n = 60 x 64 and k = 50, the improved method reduces the total time to 3x50+7 6- = 4.1% of the original, while the number of features seen during training increases by 26.0x. In terms of actual training time, improved distillation completes in just 14 hours, compared to a theoretical cost of over 340 hours for the traditional method. This substantial efficiency gain makes MLLM distillation practically feasible, transforming it from intractable task to manageable one. More detailed information can be found in Appendix and Table 14
n
Table 6: Comparison of distillation from rerankers
Methods
Local Avg.
Global Avg.
Hard Negative Mining (haseline)
61.7
59.9
Rocall-Reranker
64.5
61.7
Distillation
63,2
60.7
Continue-hard
60.0
As shown in Table 6 the Rereakl-erranee" represents the model we aim to distill, and the "Distil- lation" method achieves an improvement of 1.5% over the baseline. However, distillation data are generated from the top-k retrieval results from baseline model and correspond to hard negative samples for it. We designed experiment "Continue-hard" for validation. Unlike the "Distillation" stage, the "Continue-hard" model continues training by treating the training data as hard negatives. The "Continue-hard"" method demonstrates an improvement of 0.5%/0. 1% over the baseline; how- ever, it exhibits decline of 1.0%/0.7% when compared to the "Distillation" approach. It validates that the performance improvement is clearly attributable to the distillation process. Notably, con- trary to the instability observed in Section 3.2.2 with the "only top-k hard negatives" setting, the "Continue-hard" strategy (k = 50) employed here remains stable and achieves slight gains. This stability is primarily attributed to the strong initialization: starting from converged Recall model provides robust feature representations that are more resilient to hard negatives. Additionally, the larger value helps mitigate the noise introduced by false negatives.
3.4 U-MARVEL FRAMEWORK
Motivated by the findings presented above, we introduce a unified framework, term U-MARVEL (Universal MultimodAl Retrie via Embedding Learning). which consists of the following three stages: (1) Progressive transition: the model is progressively fine-tuned on the increasing complexity levels of the retrieval data, which enables the model to gradually adapt to retrieval tasks. (2) Hard Negative Mining and Fusion Reranker Model: building on Progressive transition, we first train a hard negative model for recall and a reranker model for ranking, and then linearly combine them to obtain a recall-reranker model. (3) Distillation: we perform improved knowledge distillation on the reranker model. Detailed experimental settings and results for U-MARVEL are provided in Table 12 in Appendix [ and Table in Appendix D. respectively. Additionally, to demonstrate the generalization ability of our framework across different model sizes, we also provide results using the -i.en3-VI..44B-nsstete (Team 2025) backbone in Table 16 Appendix D
8

---

Published as a conference paper at ICLR 2026
4 COMPARISONS WITH STATE-OF-THE-ART METHODS
In this section, we compare U-MARVEL with state-of-the-art methods in both supervised and un- supervised settings.
4.1 M-BEIR EVALUATION
We first compare U-MARVEL with competitors on M-BEIR benchmark (Wei et al. 2024) in both lo- cal and global pool settings, details of this dataset can be found in Appendix A For fair comparison, we also present results of the tecall-then-rerank cascade pipeline, denoted as U-MARVEL." similar to that used in LamRA (Liu al. 2024b), details seen in Appendix [ As shown in Table 7 and 17 U-MARVEL establishes new state-of-the-art, Beyond the raw metrics, three key observations highlight the efficacy of our design: 1) Efficiency-perfrmmee performance trade-off: In the single-model set- ting. U-MARVEL significantly outperforms existing sate-of-the-ane approaches by large margin. Furthermore, while comparisons with stronger baselines like LamRA (Liu et al.] 2024b) typically involve computationally expensive two-stage retrieval process. U-MARVEL achieves comparable performance using only single-stage inference. This dual advantage of superior accuracy and
ficiency is directly attributable to our improved distillation strategy (Sec. [.3 which successfully transfers the discriminative power the reranker into the embedding space. 2) Robustness across settings: As detailed in Table 17 and Appendix D our method maintains its superiority even in the more challenging "Global Pool" setting, where candidates from all tasks are mixed. It suggests that method helps the model learn more robust, task-agnostic features rather than overfitting to specific task distributions. 3) Contribution of Components: The ablation study in Table 15 (Ap- pendix (c further reveals that our model already surpasses existing state-of-the-art methods in the single-mode setting upon the completion of "Progressive Transition". Subsequently, "Hard Nega- tive Mining" provides an additional boost in discrimination, while the "Distillation" stage critical for closing the gap with reranking models.
Table 7: Comparisons with SoTA approaches on M-BEIR benchmark in local pool setting.
Methods
d'
o
+
d
q - (g', d') + d (g',4) d')
+ d' Avg.
VN CO
F200 WQ
ES
WQ
VN R@5
CO
F200
ON
InS
FQ
CR
ON
InS R
R
R
R
RO'S
5
R
R
R
R
R
R
R
R
R
R@s
Single
(Radoord
(221
43.3
61.1
6.6
36.2
43.3
45.1
41.3
79
7.7
26.1
24.2
20.5
7
13.2
38 8
26.4
32.5
SigLIP Zana et si 2013
30.1
75.7
36.5
39.8
27
43.5
30.8
88.2
34.2
28.9
29.7
25.1
14.4
22.7
41.7
27.4
37.2
Unil-Bun (ey C 2024
23.4
79.7
26.1
80
50.9
79.8
22.8
89.9
28.9
33
41
22.4
29.2
52.2
55.8
33
46.8
UnilR-CLI Werre
42.6
81.1
18
84.7
59.4
78.7
43.1
92.3
18.3
32
45.5
27.9
24.4
44,6
67.6
48.9
50.6
28.7
81.2
30.4
33.2
76.2
U-MARVEL
47.3
84.4
33.6
97.1
78.8
88.5
47.3
93.5
35.1
34.2
62.5
58.3
36,4
60.7
79.4
74.7
63.2
RRranner
LamitA Liu
85.2
32,9
75.8
87.7
36.1
33.5
U-MARVEL
49.4
85.6
34,2
98.5
81.4
89.4
50.5
88.4
37.7
34.7
63.7
62.9
38.2
63.2
80.8
78.9
64.8
4.2 ZERO-SHOT EVALUATION
We evaluate the performance of U-MARVEL with several unseen datasets in zero-shot manner. Details regarding the datasets and evaluation metrics are provided in Appendix B As presented in Table U-MARVEL achieves state-of-the-art performance on 9 out of 12 tasks, highlighting its exceptional zero-shot capabilities compared to leading methods such as VLM2Vec (Jiang et al. 2024b), UniME (Gu et al. 2025). and LamRA (Liu et al.] 2024b). This verifies that our model effectively transfers knowledge from the diverse M-BEIR tasks to unseen domains, likely due to the high diversity of the progressive training curriculum that prevents catastrophic forgetting the pre- trained knowledge. Furthermore, we evaluate U-MARVEL on the zero-shot text-to-vide retrieval task. The results on the MSVD and MSR- VTT datasets, summarized in Table demonstrate that U- MARVEL outperforms VLM2 (Jiag et al.) 2024b), LamRA (Liu et al. 2024b), and LLaVE-7B (Lan et al., 2025). However, is worth noting that U-MARVEL exhibits degraded performance, which may be attributed to fail to model temporal contexts, such action sequence progression.

---

Published as a conference paper at ICLR 2026
Table 8: Comparisons with SoTA approaches zero-shot image and text benchmarks.
Methods
d o
o' + (d'dd) - d quale -
[q] 6 d')
+
ITM
U-IK"
Flickr STAV U-IK" Flickr CIRCO" GCIS
V-Dia" R@
M.FIQ" R
C.Neg S-Cre" Acc. Acc.
R R@1
R@ RØ
RØ1 R@I MAP@5 R@
Single model
CLIP-I. (Radford et [202]]
52.8
67.3
81.8
68.7
87.2
13.3
23.7
17.7
66.7
73
UniTR CTaP We
2024
85.8
75
78.7
84,1
78.4
94,2
12.5
16.8
26.8
19.4
79.9
80.1
ES-V Jiang
202231
86.7
84
79.5
82.4
86.2
24.8
54.6
83.2
84.7
Marictenc-11 if
59.3
62.7
75.9
VL.M2Vec (Jann en da
90.7
90.8
85.8
84.7
90.5
79.5
UniME (Gil o al (0
97.2
95.9
81.9
93.9
95.2
93.4
85
LamRA-Rar ér 2024h
93.3
95.1
82.8
88.1
94.3
92.7
33.2
18.9
62.8
60.9
79.6
85.8
U.MARVEL
Rerunker
I LamRA (Lán
2024h
97.9
98.8
88.1
96.5
98
97.6
42.8
24.8
70.9
63.9
85.9
93.5
U MARVEL
97.8
97.7
88.5
98.9
98.2
95.1
46.0
22.6
77.6
66.3
86.1
93.4
Table 9: Comparisons with SoTA approaches zero-shot text-to-video retrieval benchmarks.
MSR-VTT
MSVD
Model
R@l
R@5
R@
R@1
R@5
RØ 10
Zero-shot (fnetumed with texr-video data)
Intern (Wnng et 2022
40.0
65.3
74.1
43.4
69.9
79.1
VICLIP Wane of 2015
42.4
49.1
UMI-L
42.6
Inter V8022, 0b Wang et
20246
55.9
78.3
85.1
59.3
84.4
89.6
Zero shot (fnettmed only image data)
VLM2
2024bb
43.5
LamRA de 20220)
44.7
78.6
52.4
79.8
LLaVE-"B Laan et 2n 2025
46.8
71.1
80.0
52.9
80.]
87.0
U-MARVEL
47.2
72.0
80,2
54.6
80.9
87.7
U-MARVEL
47.5
69.7
78.5
53.1
79.9
87.1
CONCLUSION AND LIMITATIONS
This paper systematically explores the design principles for building high-performing universal mul- timodal retrievers with MLLMs. Through comprehensive study, we identified key factors that significantly impact retrieval performance, including embedding generation strategies, progressive adaptation techniques, and training recipes such as hard negative mining and re-ranker distillation. Based on these insights, we introduce U-MARVEL, a unified framework that achieves state-of-the- art performance on the M-BEIR benchmark in supervised settings and demonstrates strong zero-shot generalization across diverse tasks. The results highlight the generalization potential of U-MARVEL and its ability to address the complex requirements of real-world retrieval scenarios. We believe that our study contributes valuable knowledge to the research community and paves the way for future advancements in universal multimodal retrieval.
Despite the promising results achieved by U-MARVEL, there remain several limitations that can be improved. First, it focuses text and image modalities, leaving the inclusion of other modalities (e.g., audio) for future work. Second, while effective in zero-shot scenarios, its integration with LLMs retrieaal-augmente generation (RAG) applications remains underexplored. Lastly, exper- iments were limited to 7B-sized models due to submission time constraints; future work will extend support models of various sizes, enabling flexible trade-offs between performance and cost.
ETHICS STATEMENT
This study strictly adheres to the full requirements of the Code of Ethics outlined by the International Conference Learning Representations (ICLR). We have thoroughly read and adhered to this code during the research design, data handling, experimental execution, and manuscript writing phases, ensuring that the entire research process aligns with academic integrity and ethical guidelines.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of this study, we fixed the global random seed to 42 across Python, NumPy, and PyTorch environments. This configuration minimizes stochasticity in model initial- ization, data sampling, and training dynamics. Consequently, the results reported Tables [ and Tables 13 reflect deterministic runs, aligning with standard practices in large-scale model research where computational constraints render multi-seed training infeasible. To facilitate direct replication, the source code and datasets are publicly available via the GitHub repository linked in the Abstract. Additionally, detailed experimental environment configurations are included in the Appendix.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08744 2023.
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. In Findings of the Association for Computational Linguistics: ACL. 2023, pp. 3650-3675, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Alberto Baldrati, Lorenzo Agnolucei, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15338-15347, 2023.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapa- dos, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint Stt.:::2044000661. 2024.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16495-16504, 2022.
David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 190-200, 2011.
Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Wei Chang. Can pre- pre-trained vision and language models answer visual information-seeking questions? arXiv preprint Xrxi::2302. 11713, 2023.
Wei Chow, Juncheng Li, Qifan Yu, Kaihang Pan, Hao Fei, Zhiqi Ge, Shuai Yang, Siliang Tang, Hanwang Zhang, and Qianru Sun. Unified generative and discriminative training for multi-modal large language models. arXiv preprint arXiv:2411.00304, 2024.
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8415-8424, 2021.
Xin Cong, Bowen Yu, Mengcheng Fang, Tingwen Liu, Haiyang Yu, Zhongkai Hu, Fei Huang, Yongbin Li, and Bin Wang. Universal information extraction with meta-pretrained self-retrieval. In Findings of the Association for Computational Linguistics: ACL 2023. pp. 4084-4100, 2023.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 226-335, 2017.

---

Published as a conference paper at ICLR 2026
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306.09344. 09344, 2023.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings, arXiv preprint arXiv:2104.0m 08821, 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint arXiv: 1tkdv::700..0000 02677, 2017.
Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. Breaking the modality barrier: Universal em- bedding learning with multimodal 1lms. arXiv preprint arXiv:2504. 2025.
Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. Automatic spatially-aware fashion concept discovery. In Proceedings of the IEEE international conference on computer vision, pp. 1463-1471, 2017.
Pengyue Hou and Xingyu Li. Improving contrastive learning of sentence embeddings with focal- infonce. arXiv preprint 1rXiv:2310006911 2023.
Cheng-Ye Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compasitionality. Advances in neural informa- fion processing systems, 36:31096-31116, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal. Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wes Wei Chang. Open-domain visual entity recognition: Towards recogniz- ing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12065-12075, 2023.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407. 12580, 2024a.
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun. Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7969-7992, 2023.
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingho Zhou, and Wenhu Chen. Vim2vec: Training lisgun-language models for massive multimodal embedding tasks. arXiv preprint arXiv:2410.05 160, 2024b.
Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. Llave: Large language and vision embedding models with hardness weightee contrastive learning. arXiv preprint arXiv:2503.04812 2025.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Nv-embed: Improved techniques for training Iims as generalist embedding models. arXiv preprint arXiv:2405.I /7428, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- -mage pre- training for unified languaele understanding and generation. In International conference on machine learning, pp.
PMLR, 2022.
12

---

Published as a conference paper at ICLR 2026
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image image pre-training with frozen image encoders and large language models. In International conference machine learning. pp. 19742.. PMLR, 2023a.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficien video foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19948-19960, 2023b.
Sheng-Chied Chigh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms. arXiv preprint 24100:2411.02571, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part 13, pp. 740-755. Springer, 2014.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning. arXiv preprint arXiv:2010.037443, 2020.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023a.
Siqi Liu, Weixi Feng, Tsu-jui Fu, Wenhu Chen, and William Yang Wang. Edis: Entity-driven image search over multimodal web content. arXiv preprint arXiv:2305.13631, 2023b.
Yikun Liu, Pingan Chen, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant. arXiv preprint arXiv:2412.01720. 0222200 2024b.
Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv, Zhiyuan Liu, and Ge Yu. Universal vision-language dense retrieval: Learning unified representation space for multi-modal retrieval. arXiv preprint arXiv:2209. 001.0,019..... 2022.
Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Tency, and Stephen Gould. Image retrieval on real-life images with pre-trained lision-and-languall models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2125-21344 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- 2018.
tive coding. arXiv preprint arXiv:
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to-phrass -o-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp- 2641-2644 2015.
Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, distillation, and hard nega- tives for vision-language pre-training. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 6967-6977. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PmLR, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp- 2556-2565, 2018
13

---

Published as a conference paper at ICLR 2026
Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparn Bharati. Learn" no" to say" yes" better: Improving vision-language models via negations. arXiv preprint arXiv:2403.20312. 2024.
Zhun Sun and Chao Li. Analyzing the impact of learnable softmax temperature in contrastive visual- textual alignment systems: Benefits, drawbacks, and alternative approaches. Transactions Machine Learning Research, 2024.
Qwen Team. Qwen3 technical report, 2025. URL /
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 2023.
2023.
Sagar Vaze, Nicolas Carion, and Ishan Misra. Genecis: A benchmark for general conditional im- age similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-lange language model's perception of the world at any resolution. arXiv preprint vrXi:2249.122111 2024a
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang. et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint 222iv:211..03113 2022.
Yi Wang. Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understand- ing and generation. arXiv preprint arXiv:2307.06442 2023.
Yi Wang, Kunchang Li, Xinhao Li. Jiashuo Yu, Yinan He. Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Interviide22: Scaling foundation models for multimodal video understanding. In European Conference Computer Vision, pp. 396 416. Springer, 2024b.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, pp. 387-404. Springer, 2024.
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Roge- rio Feris. Fashion iq: A new dataset towards retrieving images by natural language feedback. In Proceedings of the IEEE/CVF Conference computer vision and pattern recognition, pp- 11307-11317, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-588858 2288-5296,225 2016.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint AXiv:24/2.151155 2024.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint
arXiv:2408.. 01..8018000 01800, 2024.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprint 2rXiv:2205.01917, 2022.
Yifei Yuan and Wai Lam. Conversational fashion image retrieval via multiturn natural language feedback. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 839-848 2021.
14

---

Published as a conference paper at ICLR 2026
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In Proceedings of the IEEE/CVE international conference on computer vision, pp. 11975-11986, 2023.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision, pp. 310-325. Springer, 2024a.
Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming- Wei Chang. Magiclens: Self-supervised image retrieval with open-ended instructions. arXiv preprint arXiv:2403. 19651, 2024b.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, al. model supporting long-contextual input and output. arXiv preprini arXiv:2407.03320 2024c.
A versatile large vision language
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Mcishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal llms. arXiv preprint ar A4 16855, 2024d.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
Haojie Zhuang, Wei Emma Zhang, Jian Yang, Weitong Chen, and Quan 7. Sheng. Not all negatives are equally negative: Soft contrastive learning for unsupervised sentence representations. In Pro- ceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 2024.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A M-BEIR DATASET & METRICS
Datasets. We evaluate the retrieval performance of our method with the M-BEIR (Wei et al. 2024) benchmark. This dataset consist of possible multimodal retrieval tasks involving texts and images. It is constructed based on 10 different datasets from 4 domains. Moreover, according to the datasets and task types (i.e., the modalities of queries and candidates), the multi-modal retrieval tasks are further subdivided into 16 types.
Regarding the structure of the dataset, the M-BEIR consists of three key components: instructions, query sets, and candidate pools. Each task has its independent query set and candidate pool, and the query set and candidate pool of each task contain only single modality. Additionally, each task is equipped with 4 instructions, which clearly specify the target modality of the task. Regarding the scale of the dataset, the training set contains 1.33 million queries and 1.93 million candidates, while the test set contains 0.19 million queries and 5.6 million candidates. In addition, besides the local query sets and candidate pools for each task, both the training set and the test set also have global query sets and global candidate pool, that is, the query sets and candidate pools of the 16 tasks are combined. Table TO provides detailed presentation of the information about the M-BEIR dataset.
Table 10: The overview of M-BEIR training//alidationn/es set.
Task
Dutasel
Instruction (shown 1 out of 4)
Domain
Train
Dev
Test
Pool
VisuiNNews, et bnzo
Idesrify news-rclated image match with the description
News
99K
20X
20K
542K
1. e
MSCOCO Lm
2015
Find an everyday image mutch with caption
Micc.
100K
2488
24.8K
5K
200%
7017
Based fashion descrigtion, retrieve matched image
Fashion
15K
1.7K
1.7K
201K
2.0
WehQA (Chang
Find parngraph from Wikipedii
the question
Waki
1.7K
544K
3. + (d.2)
EDIS [a
Find
image masching with cuption
News
3.2K
WebtO Ching aca
2022
Find Wiki image that answer the question
Wiki
1.7K
403K
VisualNews
(ianet m 2000
Provide a news-reltted caption for the disphayed image
News
100K
20K
20K
537K
4.
MSCOCO Li
Find a caption describe the an image
Mac.
113K.
5K
5K
25K
Fashion:00X pal er 10
2017
Find description for the fashion item in the image
Fashion
15K
4.8K
4.8K
61K
5.
NIGHTS (Fu
2023
Find image that is identical to the given image
Misc.
15K
2X
2K
40K
OVEN Hu
D125
Ratriove
the given query wes given question about image
image
Waki
150K
50X
SOK
676K
InfoSeck CRREELUR
Find article
141K
7.64.)) -
10 at
CIRR Liu kr 2041
D121
Find image to mutch feshicn image and style note I'm looking for a similar everyday image with the described changes
Fashion
MisC:
26K
2K
2K
4K
74K
21K
(/ (e.c)
OVEN Hu s 2012
Find a Wiki image-text pair to answer question regording an image Wiki
157K
14.7K
14.7K
335K
InfoSeck Clen al. 2024
Find a Wiki image- test pair to answers question about this image Wiki
143K
17.6K
17.6K
481K
10 datesets
64 instructions
domains
1.1M
182K 190K
5.6M
Metrics. Following the evaluation protocols by M-BEIR benchmark, we use the Recall@ 10
(R @ 10) for Fashion200K (Han et al. 2017) and FashionIQ (Wu et al. 202T). and the Recall @5 (R@5) for the remaining datasets. We report the average score of all test queries as the criterion for measuring the retrieval accuracy.
The evaluation has two kinds of settings: Local and Global pool settings. The local setting indicates that each task conducts retrieval operations within its own candidate pool and the retrieved results will not have modality errors. The global setting means that the retrieval is carried out within the global candidate pool formed by combining all task candidate pools. In this case, modality errors will occur in the retrieved results.
Note that the first columns of Table 15 and 10 indicates the retrieval task type: ge for text queries, q' for image queries, cl for text candidates, and for image candidates.
Datasets used in our experiments include VisualNews (VN), MSCOCO (CO). Fashion200K (F200), WebQA (WQ), EDIS (ES), NIGHTS (NS), OVEN (ON), InfoSeek (InS), FashionlQ (FQ), and CIRR (CR) in Table 7 and
B ZERO-SHOT DATASETS& & METRICS
Datasets.
We have examined the zero-shot capabilities of U-MARVEL in retrieval tasks, specif- ically focusing on image-text cross-modal retrieval and text-video retrieval. For tasks of cross- modal retrieval between images and texts, we use the following datasets: ShareGPT4VV (Hurst et al.)
16

---

Published as a conference paper at ICLR 2026
2024). Urban- LK (Zhang al. 2024a). CIRCO Ballrraii et al.
2023 Flickr (Plummer al. 2015),
GeneCIS Vaze et al., (023), Visual Dialog (Das et al. 2017 7. Multi-round Q ( Yuan & Lam 2021), CC-Neg (Singh et al., 2024), and Sugar-Crepe (Hsich al. (023). For text- video retrieval, the datasets used are MSR- VTT (Xu et al. 2016) and MSVD (Chen & Dolan 2011). We present the details of the Unseen Dataset in Table 11 Although some of them are actually adapted from MSCOCO or FashionIQ, we still treat these datasets as unseen datasets due to the significant dif- ferences in their captions or query formats, following (Liu al. 2024b). For instance, the captions in Urban K consist of extended captions generated by GPT-4V (Hurst et al. 2024), while the query format CIRCO combines a reference image with a relative caption. These differences create substantial disparity compared to the original COCO (Lin et al. (014) dataset. Table [1 provides detailed presentation of the information about the zero-shot datasets.
Metries. The first of Table indicates the retrieval task type: q' for text queries, q' for im- age queries, qlialog for dialog queries, (a' @ g¹) for multi-interleaved image-text queries, e for text candidates, ct for image candidates, and ITM for the Image-Text Matching task. Datasets used in our experiments include ShareGPT4v (ST4V), Urban-IK* (U-IK*), Flickr, CIRCO (CIRCO*), GeneCIS* (GCIS*), Visual Dialog* (V-Dia*), Multi-round FashionIQ* (M-FIQ*), CC-Neg (C- Neg), and Jugar-Crrepe (S-Cre*) in Table 8 indicates that the images in these datasets are sourced from COCO FashionIQ.
Table 11: Summary of the evaluation benchmarks. Queries represents the number of test queries, and # Candidates denotes the number of test candidates per query.
Benchmnark
Zero shot N Queries
8 Candidates
ShareGPPaa (Hurst all 2024
1K
1K
Urban-1K (Ziann RUAe
IK
1K
Flickr30K PPummre
2013
IK
5K
CIRCO [Bauri 2i 2012
80D
120K
GeneCIS Vaze 23463
8K
10 15
Visual Distog-1iis
20
2K
2K
CC-Neg (Singh ei al. 202-4
40K
Sugar- Crépe (Fiste 0
7.5K
2
MSR-VII (X 2016)
670
27K
MSVD (Chèn Dotan
1K
C IMPLEMENTATION DETAILS
We select Qwen2-VL. VL-7B- Insttructt (Wang et al. 2024a) the pretrained MILMM backbone instan- tiate both of the retriever and the reranker. Throughout all experiments, we froze the visual side and employed LoRA (Hu al., 2022) to fine-tune the LLM component of the model. For embedding models, the InfoNCE loss (Oord et al.) 2018) is selected as our training objective:
exp(sim( )/T)
LInfONCE: = log
(3)
Σ; exp(ssmm(e (e:. e/)/T)
where eg, et ea represent the embeddings of the query, the positive candidate, and any sample from the candidate set, respectively. is the temperature parameter. sim(.,) means the cosine similarity.
Recall-Rerank Model. Some studies (Liu et al., 2024b Lin et al. 2024) further refine retrieval results by applying reranking step top of the recall model. Inspired by (Liu et al., 2024b), randomly sample top-k hard negatives from the training hard negative stage to construct data triplets (query, positive, hard negative). Based on the progressive transition model, perform supervised fine-tuning of the model shown in Figure [bb using the loss function defined in Eq. 4 Detailed training configuration are shown in Table 12
Crerank (8) log Po(y query + candidate)
(4)
where y = yes if the candidate belongs to the of positive candidates, and y = no otherwise.
score
PoESSS query candidate)
(5)
17

---

Published as a conference paper at ICLR 2026
During inference, adopt the reranker's predicted probability of the token "YES" the relevance score, as shown in Eq. | After training the reranker, we employ the hard negative mining model for recall, obtaining the top-k candidates (C1, C2 Ca........ ...,..))).))))... Cx) along with their corresponding retrieval scores Sreall = (#1. 82,...,888) 8k) for each query. These candidates are then evaluated using the reranker model, which produces scores Srerank (s/ ,2.....,88,, The final ranking is generated
by combining both sets of scores through Eq. 6 producing optimized query-candidate matching scores.
This completes our recall-then-rerank pipeline,
S Smulti Srecall + (1 a). Sreaank,
(6)
where a is set 0.5.
Improved Distillation. For the distillation stage, we use the following loss function:
|| Ssingle) Σ (inutt(()) log Smatts (i)
Laisill D
Ssingle(t)
where Smult: represents the modei ensemble scores and Ssingle the single model's predictions. The scores are softmax-normin before computing the KL divergence.
Table 12: Training Configurations
Training Stage
Temp
LR (n)
Batch Size
GPU number
LoRA para.
epoch number
Progressive Transition (stage-1) 0.05 (fixed)
2 10-1
576
B
tank(6)))x(128)
2
Progressive Transition (stago-2)
0.05 (fixed)
10
720
16
(2)k(128))((26))
Progressive Transition (stage-3)
0.05 0.05->0.0206*
4 10
3840
64
rank(128)a<(256)
Hard-neg mining
0.0206-0.0188-
1.5 10
1408
64
rank( (256)
Distillation
0.0188-20.020
10
64
rank(128)a(256))
Reranker Training
128
64
mmn((12))((56))
Leamable tomperature parameter
U-MARVEL.*. We use the U-MARVEL model the initial retrieval system and employ its Top- 100 candidate results to train a point-wise reranking model following the methodology presented in Figure bb) and Eq. as the loss function. Subsequently, the embedding similarity by the U-MARVEL model are combined with the reranking scores from the point-wise reranking model through weighted fusion mechanism compute the final ranking score:
produced
==a SU-MARVEL (1 (1-a) a.) point- seranking model:
(8)
where weighting hyperparameter, set to 0.5. Based on this final score, the candidate is reranked to generate the final ranked list. We refer to this unified framework as U-MARVEL The detailed training configurations are provided Table 12
D SUPPLEMENTAL EXPERIMENTAL RESULTS
We present the retrieval performance of each stage in U-MARVEL in Table 15) which provides a detailed ablation study of our framework. The results clearly demonstrate the effectiveness of the proposed design, showcasing significant perfor- mance improvements achieved through the progressive transition mechanism, hard negative mining, and reranker distillation. These components collectively contribute to the robustness and efficiency of U-MARVEL, highlighting the importance of each stage in enhancing retrieval capabilities and validating the overall framework design.
Effectiveness of Three Key Stages in U-MARVEL.
Comparison of M-BEIR (global). Table 17 presents a comprehensive comparison of our pro- posed U-MARVEL framework with of-he-of-the-rr methods on the M-BEIR benchmark under the global pooling setting. The results demonstrate that, within the single-model category, U-MARVEL significantly outperforms leading approaches such as MM-Embed (Lin et al. 2024) and LamRA- Ret (Liu et al. 2024b), by large margin. Furthermore, when integrated into recall-then-rerank pipeline, the enhanced variant U-MARVEL* achieves the best overall performance, further solidi- fying the effectiveness of our framework. These findings highlight the versatility and superiority of U-MARVEL across different retrieval paradigms, showcasing its potential for advancing state-of- the-art retrieval systems.
18

---

Published as a conference paper at ICLR 2026
Held-out Experiments M-BEIR. To evaluate the cross-task generalization ability of the pro- posed method, we trained the model on the five tasks, reserving the remaining tasks for evaluation. As presented in Table [8] our method achieves an average improvement of 2.4%/1.6% across the three held-out tasks compared with SoTA approaches.
three tasks as held-out
Analysis on Training Data for Progressive Transition. Our method adopts a progressive train- ing pipeline: NLI + CC3M - M-BEIR, achieving scores of 57.7%/55 8.6 as shown in Table 3
During the progressive training phase, we attempted to incorporate high-quality image-text pairs from ShareGPTTV, dataset characterized by highly detailed image captions. However, contrary to expectations, utilizing ShareGPT4V data- independently or mixed with CC3M- in performance degradation on the M-BEIR benchmark, as demonstrated in Table 13 esize that this decline stems from a significant distribution shift: the dense, exhaustive descriptive text in ShareGPT4V contrasts sharply with the concise, query-oriented text typical of retrieval tasks (e.g., M-BEIR). This discrepancy likely hinders the model's alignment with the target retrieval ob- jective. Furthermore,
We hypoth-
explored alternative progressive path: NLI - TART (Asai et al.) 2023) - M-BEIR, which also led to a substantial decrease in performance. We conjecture that the diverse text instruction tasks within TART interfere with the model's effective alignment for multimodal retrieval.
Analysis on Learnable Temperature of InfoNCE. We analyze the impact of the learnable tem- perature parameter initialized at 0.05 in Progressive Transition (Stage-3).
Superior Performance over Fixed :: Learnable r consistently outperforms static settings. In Table [ ID-7 (batch size 3840) and ID-8 (batch size 7680) achieve scores of 60.1% 1% and 60.3%, respectively, surpassing the fixed T = 0.05 baseline (ID-4 and ID-6, both at 58.9%). We further conducted an experiment fixing т 0.02 (aligned with the converged value of ID-7), which yielded a suboptimal score of 59.8%. This comparison confirms that dynamic adjustment adapts better to the optimization landscape than even a carefully manually tuned static value.
Adaptive Convergence: We observe stable convergence of r across different batch sizes. Specifically, we observed convergence to 0.0206 for batch size 3840 and 0.0195 for batch size 7680. The slight decrease in the larger batch setting aligns with findings in (Sun & Li) 2024) where sharper distribution is necessitated handle increased batch noise.
Stage-wise Evolution: As detailed in Table 12 fluctuates adaptively across stages (Stage-3: 0.0206 - Hard-neg: 0.0188 > Disuilattin: 0.0204). Notably, т drops during Hard-negative Mining increase sensitivity to hard gradients.
Analysis on Generalization Ability of U-MARVEL. To investigate the generalization ability of U-MARVEL across different MLLM backbones, we extended our framework to the Qwen3-VL-4B model. As presented in Table 16 the experimental results align consistently with the conclusions drawn from the 7B model, showing steady performance improvements across the three training stages-Pres Progressve Transition, Hard Negative Mining, and Distillation. This validates that proposed training recipe is robust and agnostic to specific model architectures. Remarkably, despite having significantly fewer parameters, the Qwen3- VL.44B model trained with U-MARVEL achieves a Local Average Recall of 58.8% and a Global Average Recall of 56.2% on the M-BEIR benchmark. These results not only demonstrate the effectiveness of our method but also establish new state- of-the-art, surpassing the previous best- best-performing method, LamRA-Ret (Liu et al.) 2024b) (Local: 56.6%. Global: 54.9%) with a larger 7B backbone. This underscores the strong generalization potential of U-MARVEL in enabling smaller models to achieve superior retrieval capabilities in both local and global search scenarios.
Table 13: Comparison of Progressive Transition with Different Training Data
Training Pipeline
Avg.
Global Avg.
NLI -hareGPP4v-+ M-BEIR
56,9
$4.0
NLI CC3M BEBR
57.2
55.2
NLI TART - M-BEIR
50.3
36.6
19

---

Published as a conference paper at ICLR 2026
E DISTILLATION COMPUTATIONAL COST ANALYSIS
The distillation process consists of two main parts: (1) The teacher model infers a similarity ma- trix; (2) The student model is trained to extract features, compute scores, and lear from the teacher model's similarity matrix. In terms of computational cost, the recall stage primarily focuses on ex- tracting the feature representations of the samples. On the other hand, the rerank stage primarily concerned with calculating the similarity scores between the query and candidates. Despite the dif- ferences in tasks, the inference time for single pass through both stages is approximately the same. Moreover, we assume that during the distillation phase, the training time for feature by the student model approximately twice that of inference, it includes both forward and backward propa- gation. Table [44 presents detailed comparison between the traditional and improved distillation methods. We adopt the following notation for our analysis:
N: The total number of queries in the dataset (assuming ratio of 1:1 with positive samples, the candidate pool size is also N).
n: The number of queries in a single batch.
The number top-k hard negatives selected in our improved method.
E. TEACHER MODEL INFERENCE STAGE
This stage subdivided into Recall and Rerank phases.
Both the traditional and improved methods require feature extraction for all queries and candidate documents to compute retrieval scores. With N queries and N positive candidates, the total inference cost is N + N 2N. For the improved method, the additional overhead of nearest neighbor search to identify top-k hard negatives is computationally negligible compared to feature extraction and is therefore omitted.
Recall:
Rerank: This phase focuses on computing query-candidate similarity scores.
Traditional Method: Computes scores for all pairs within a batch (utilizing in-batch nega- tives), resulting in x similarity matrix per batch. The total cost is N. x n² n.N. Improved Method:
Computes scores strictly for the query, the positive sample, and the mined hard negatives, resulting in an n (k + 1) matrix per batch. The total cost is x nk + 1) (k + 1)N.
E.2 STUDENT MODEL TRAINING STAGE
The student model processes the query and positive sample pairs, to- taling 2NV features. Applying the training cost multiplier (factor of for forward/backwwrd passes), the total training cost is 2 2N = 4NN
Traditional Method:
The input augments the query and positive sample with & hard neg- atives, totaling (A: + N) features. Consequently, the training cost is 2 x (k: 2) (2k + 4.
Improved Method:
E.3 OVERALL COMPLEXITY
Aggregating the costs from both stages, the total computational complexity is (n + 6)N for the traditional method and (3k + 7).N for the improved method. Given that the batch size is typically much larger than the number of hard negatives & (i.e., n > k), our improved method significantly reduces the computational burden, particularly during the Rerank phase.
F THE USE OF LLMS
We only used GPT for language polishing after completing the first draft, and there other scenarios involving the use Large Language Models (LLMs) in this work.
20

---

Published as a conference paper at ICLR 2026
Table 14: Comparison of Computational Cost Between Traditional and Improved Distillation.
represents the number of queries in batch, and N denotes the total number of queries in the dataset.
Ttem
Traditional Method
Improved Method
Recall
2N
2N
Teacher model inference
Rerank
(n²) = nN (n 2)N
(n(k + 1)) = (k + 1)N
Total (ifeeenfeeence Stage)
(k 3)N
Extract Features
2)N
Student model training
Total time Training Stage)
4N
(2k 4)N
Overall
Overall Total
(n + 6)
(3k 7)N
Table 15: U-MARVEL: Retrieval performance of each stage on M-BEIR benchmark
Local
Global
Task
Dataset
Progressive Hard-neg Distillation mining
Progressive transition
Hard-neg mining
Distillation
VisualNews
46.4
47.5
47.3
46.3
47.5
47.2
d' +
MSCOCO
84.0
83.6
84.8
77.3
79.4
72.8
Fashiio200K
34.4
35.6
33.6
34.3
35.4
33.3
d'
WebQA
92.4
95.2
97.1
92.2
95.0
96.7
d' + (d,2)
65.6
72.9
78.8
65.4
72.9
78.7
WebQA
84.6
86.9
88.5
83.8
86.6
87.7
VisualNews
45.7
47.9
47.3
45.4
47.8
47.2
d' d
MSCOCO
93.3
93.1
93.5
93.2
93.1
93.5
Fashion200K
35.4
35.6
35.1
35.3
35.6
34.9
8
NIGHTS
31.4
33.0
34.2
31.4
33.0
34.0
(6.4)
OVEN
56.5
58.9
62.5
$1.3
52.2
58.3
InfoSeek
55.1
52.2
58.3
50.8
47.5
52.2
(d'4)
FashionIQ
CIRR
58.7
35.4
57.2
34.8
60.7
36.4
35,3
55.9
34.7
55.2
36.0
36.0
(d
(d.0)
OVEN
78.0
80.8
79.4
71.9
73.8
73.1
60.6
61.7
63.2
58.7
59.9
60.7
Table 16: Ablation study on the backbone architecture. Retricval performance of U-MARVEL on
the M-BEIR benchmark using Qwen3-VL-4B.
Local
Global
Task
Dalaset
Progressive transition
Hard-neg neg Distillation mining
Progressive transition
Hard-neg Distillation mining
VisualNeww
35.8
36.6
36.2
35.7
36.5
36.0
o'
MSCOCO
83.8
80.
82.6
77.4
74.4
73.2
Fashion200K
27.6
27.6
d' +
WebQA
96.8
d' + (d,e)
EDIS
60.1
62.4
66.2
59.7
62.3
65.9
WehQA
86.0
VisualNews
34.9
35.6
MSCOCO
93.5
93.9
93.9
93.4
93.9
93.9
Fashiio200K
27.3
28.2
26.5
27.3
28.2
26.3
ddd.
NIGHTS
32.4
31.9
33.3
32.3
31.9
33.2
(d'44)
OVEN
51.8
55.0
57.9
46.7
47.7
$3.7
InfoSock
46.5
46.2
56.8
41.4
40.9
47.7
(4'
FashionIQ
35.4
34,9
36.0
35.3
34.8
35.8
CTRR
60.3
54.2
60.5
57.7
52.8
57.2
("' (d)
OVEN
71.1
75.2
73.3
65.1
67.3
67.1
InfoSock
63.7
64.0
70.1
59.3
58.9
62.7
Avg.
56.2
56.7
$8.8
54.2
54.6
$6.2
21

---

Published as a conference paper at ICLR 2026
Table 17: Comparisons with SoTA approaches on M-BEIR benchmark in global pool setting
Methods
d'
(c.d)
o
+
g' -
(q',
+ (g', e' Avg.
VN RØ5
CO
F200 WQ
ES
WQ
VN
CO
F200 NS
ON
InS
FQ
ON R@5
InS RØ5
R R
ROS
ROS
RØB
R@5
R
RO
R
RAB5
R
R
R
Single model
UnilR BLIP
Wei al
2024
23
75.6
25.4
79.5
50.3
79.7
21.1
88.8
27.6
38.7
19.7
28.5
51.4
57.8
27.7
45.5
UnIR--CLIP We aC
42.6
77.9
17.8
84.7
39.4
78.8
42.8
92.3
17.9
32
39.2
24
24.3
43.9
60.2
44.6
48,9
MM-Emmed
Lib
2024
41
71.3
17.1
95.9
68.8
85
41.3
90.1
18.4
32.4
42.1
42.3
25.7
50
64.1
57.7
52.7
U-MARVEL
47.2
72.8
33.3
96.7
78.7
87.7
47.2
93.5
34,9
34,0
58.3
52.2
36,0
56.0
73.1
69.2
60.7
Renanker
LamRA Liu
58.7
74
U-MARVEL
48.8
70.1
33.8
98.3
80.8
88.3
49.8
86.0
36.8
34.8
58.7
56.9
37.4
58.4
74.9
73.8
61.8
Table 18: Held-out task generalization experiments were conducted on M-BEIR. The asterisk indicates that the model was trained on the remaining five tasks, without any exposure to the three held-out tasks.
o' d
(d'
(o'.d')
(c),
Methods
NIGHTS
OVEN RØ5
InfoSock
OVEN RØ5
IntoSock R@5
Local Avg.
Supervised
MailR-BLIP (Wii al UnilR-CLIP
2024
33.0
41.0
22,4
55.8
33.0
37.0
32.0
45.5
27.9
67.6
48.9
44.4
Zemosshot
LamRA-Ret"
LLu et al
2024b
27.2
44.7
44.0
62.8
49.5
45.6
U-MARVEL
31.5
44.3
49.2
62.6
$2.6
48.0
LamRA* Li
2024b
29.2
46.9
54.2
65.1
59.1
50.9
U.MARVEL
32.1
49.0
55.3
64.9
61.2
52.5
22


Reviews:
[{"review_id": "cxsUYMsnDt", "reviewer": "Reviewer_kXJc", "summary": "The authors perform a systemic investigation to identify an optimized configuration for universal multimodal retrieval (UMR) with a 'general' MLLM-based pipeline. Specifically, they finetune a Qwen2-VL-7B-Instruct model with LoRA as the retriever-reranker embedding model (with distillation) using contrastive learning (InfoNCE) on the M-BEIR dataset (Figure 1b). Based on this 'basic' architecture, they explore three primary questions: (1) how to adapt decoder-only MLLMs into instruction-aware embedding function, (2) how to train embedding models with contrastive learning, and (3) the value of distillation in recall-then-rerank methods. \n\nFor (1), the first question addressed is whether to use {last token, mean token}, {unidirectional, bidirectional}, and the use of a compression prompt instruction (i.e., one-word) in extracting embeddings from the MLLM [bidirectional, mean pooling wins]. The second question addressed for (1) is instruction inclusion and their masking out during mean pooling [masking out instructions during mean pooling wins]. The third question is testing progressive transition in fine-tuning MLLMs with step wise training (instruction tuning -> text-only retrieval (NLI) -> text-image retrieval (CC3M)) [it works].\n\nFor (2), the first question asked is the hyperparameter grid search over {batch size, learnable temperature, learning rate} [increasing batch size with appropriate rate scaling and learnable temperature parameters enhance effectiveness]. The second question asked for (2) explores hard negative mining, in-batch negatives, and negative filtering when training InfoNCE [in-batch negatives and filtered top-k hard negatives wins (there are some details on the hard negative filtering strategy)].\n\nFor (3), they show that reranker distillation reduces computational requirements and increases feature diversity (i.e., effective distillation via a teacher-student model). This section has a lot of technical details (especially if also considering the appendices) [the method distills the teacher model with minimal performance degredatation]\n\nPut all of this together, and you have U-MARVEL, which is shown to outperform several single model (e.g., LamRA-retriever) and recall-then-rerank (e.g., LamRA) models on M-BEIR in local and global pool settings. On zero-shot settings with unseen datasets, U-MARVEL outperforms all single model settings, is competitive with LamRA on the recall-then-rerank setting for image/text retrieval, and outperforms all baseline models on text-to-video retrieval benchmarks.", "questions": "Some questions of mine would include:\n- Statistical significance of the empirical results\n- Based on your experience in conducting all of these experiments, what are some directions you are thinking about for further improvements that may lead to notable improvements (if any)?\n- (For an appendix), were there any additional directions explored that led to inconclusive or interesting negative results?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "Strengths of this work include:\n- Variants of the base architecture captures most state-of-the-art systems for universal multi-modal retrieval with VLLM backbones.\n- The configuration/hyperparameter search space addresses several relevant questions and provides good evidence for particular design choices.\n- The resulting empirical performance is strong across multiple datasets and settings and is compared to multiple recent strong baseline systems.", "weaknesses": "Weaknesses of this work include: \n- While this is partially attributable to space limitations, it was difficult to assess the soundness of the precise method without reading the appendices (I would make some different choices in terms of details, etc.). In the same vein, many of the empirical results in the appendices are useful and not incorporated into the discussion. Finally, with respect to the empirical results, the discussion/analysis is mostly 'just' restating the tables with limited interpretation.\n- Maybe I am missing a description of the notation in the empirical results, but I wasn't able to discern statistical significance. Many of the results don't appear statistically significant. \n- As the authors are aware, limiting experiments to 7B models is limiting as I would expect that the findings may differ for larger models (specifically with respect to instruction following). While these are comparable to other models in the literature, as this work follows these, it is going to be necessary to make these comparisons soon. \n- Modulo the ranker distillation details, there is limited methodological novelty. This is a classic \"optimize all the pieces\" of a architecture to create a definitive strong baseline for this moment in time that more innovative variants can directly compare to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zpc2Y0mHz2", "reviewer": "Reviewer_4154", "summary": "The paper introduces a novel framework, U-MARVEL, aimed at improving Universal Multimodal Retrieval (UMR) using Multimodal Large Language Models (MLLMs). While current state-of-the-art UMR methods have made significant strides, they often face limitations in terms of retrieval capability, generalization, and the underlying mechanisms driving performance. The authors systematically explore the key factors that contribute to effective embedding learning and retrieval performance, such as embedding generation strategies, progressive training transitions, hard negative mining, and re-ranker distillation. They present U-MARVEL, a unified framework that integrates these findings, demonstrating superior performance over existing methods on the M-BEIR benchmark and strong zero-shot capabilities for tasks like text-to-video retrieval.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of U-MARVEL as a unified framework is a significant contribution. It successfully integrates multiple advanced techniques to create a more efficient and effective retrieval system, and the results demonstrate clear improvements in performance compared to the state-of-the-art methods.\n2. The framework is validated through extensive experiments on the M-BEIR benchmark, where U-MARVEL outperforms other methods in multiple retrieval tasks. It also shows strong zero-shot generalization, which is a critical aspect of real-world applications.", "weaknesses": "1. Retrieval and reranking are common pipelines in information retrieval. The authors apply this framework to MLLMs, seemingly leveraging the powerful capabilities of MLLMs to obtain better embeddings. The authors should further clarify the differences and novelty of the MLLM-based framework compared to traditional retrieval frameworks.\n2. The authors use Qwen2-VL-7B as the base model for experiments. However, the performance of this retrieval framework on other MLLMs is unclear. The authors should further validate the generalization ability of this retrieval framework.\n3. From Table 3 in the paper, I do not see a significant gain from the progressive transition design. The authors should further analyze this in the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0HX2sZGytt", "reviewer": "Reviewer_WToA", "summary": "This paper presents a systematic empirical study on using multimodal large language models as embedding models for Universal Multimodal Retrieval. The authors implement a baseline contrastive learning pipeline (Qwen2-VL-7B with LoRA) and investigate three main axes: (1) adapting decoder-only MLLMs into instruction-aware embedders (embedding extraction strategies, instruction handling, progressive transition); (2) training MLLM-based embedders under InfoNCE (batch size / learning rate / temperature interactions, hard negative mining and filtering); and (3) compressing the recall-then-rerank pipeline into a single model via improved distillation. Key findings include that bidirectional attention + mean pooling outperforms last-token/compression prompt schemes, masking instruction tokens during pooling helps, learnable temperature improves contrastive learning, and filtered hard negatives stabilize training. Based on these insights the authors propose U-MARVEL, and demonstrate substantial gains on M-BEIR and many zero-shot benchmarks.", "questions": "1.\tIn Section 3.1.2, the paper discusses the effectiveness of multi-stage training. However, it is not clear whether this effectiveness comes from the increased amount of training data or from the multi-stage training strategy itself. Is the comparison and the resulting conclusion objective and fair under this ambiguity?\n2.\tIn Section 3.1.1, the paper states that the presence or absence of the Compression Prompt has a significant impact on model performance. Is this effect truly as substantial as reported? Based on my experience, the impact may not be that pronounced.\n3.\tHow exactly is the similarity threshold chosen for filtering false negatives in hard-negative mining? Is it dataset-specific, and how sensitive are results to this threshold?\n4.\tWhat is the initialization and optimizer settings of the learnable temperature? Do you observe stable convergence of τ across different batch sizes? Has the variation process of the temperature been recorded？\n5.\tIn the recall-then-rerank pipeline, what is the value of α? Does it vary during training or remain fixed? How is its magnitude determined?\n6.\tIn Section 3.2.2, what is the value of k used in the top-k selection? There are several unclear parameter choices throughout the paper — could the authors specify the exact values of these parameters and the criteria used for their selection?\n7.\tIn Section 3.3, the “continue-hard” experiments show that using only hard negatives leads to performance improvement, while in Section 3.2.2, using only hard negatives resulted in a failure. These conclusions appear inconsistent — is this discrepancy caused by differences in experimental settings or by the choice of data?\n8.\tThe distillation method proposed in Section 3.3 appears relatively conventional — can it truly be considered an innovation? Moreover, the paper only provides a theoretical derivation of the time savings. How does the proposed method’s performance compare with traditional approaches in practice, and is there any empirical comparison of the actual time consumption?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1.\tthe paper offers actionable, well-motivated findings (embedding extraction, instruction masking, progressive transition, filtering of hard negatives, distillation).\n2.\tbackbone, datasets, and training configs are explicitly listed (Tables in Appendix). This makes replication feasible.", "weaknesses": "1.\tLimited novelty, this work is just a conbination of existing tricks in retreiver training, some techs are already presented and disscussed by previous works, such as mean token and bidirection attention [1], learnable temperature [2]. Porting them from LLM to MLLM offers little scientific significance.\n2.\tInsufficient detail on hard-negative filtering criteria, the choice and sensitivity of the threshold for filtering presumed false negatives needs more analysis.\n3.\tSome experimental comparisons are not entirely fair, and certain conclusions appear to be inconsistent across experiments. Please refer to the Questions section for detailed discussion.\n4.  Lacks sufficient details regarding experimental implementation, parameter selection, and description of observed phenomena, such as why masking instruction tokens during mean pooling enhances embedding performance. In such cases, a deeper analysis is required; merely stating the results is insufficient for a valid research contribution.\n\n[1] LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (BehnamGhader, COLM 2024)\n\n[2] Analyzing the Impact of Learnable Softmax Temperature in Contrastive Visual-Textual Alignment Systems: Benefits, Drawbacks, and Alternative Approaches (Sun and Li, TMLR 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYkPmUOuZO", "reviewer": "Reviewer_SuPC", "summary": "This paper investigates key design decisions for universal multimodal retrieval (UMR) with MLLMs. \nThe analysis spans three axes: (1) embedding generation choices, (2) Contrastive learning training factors (i.e., batch size, learning rate, temperature; hard-negative mining), and (3) re-ranker distillation into a single model. \nBased on these findings, the authors propose U-MARVEL, a unified embedding-learning framework that targets single-model retrieval on M-BEIR.", "questions": "1. L413, 414: there should be a space between method name and inline citation.\n2. L268-269: please make the claim (\"we argue that...\") more rigorous or use evidence to back it.\n3. How exactly do you filter hard negatives? (i.e., what threshold value and how do you select the threshold)\n4. Can you provide empirical results (e.g., latency) to compare efficiency between traditional distillation and the improved distillation?\n5. Please fix the typo \"exihibits\".", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Practically impactful problem: design decisions in universal multimodal retrieval are often overlooked and this work advocates for more attention on this, in which can be a good initiative for UMR deployability.\n2. Performance: the proposed U-MARVEL framework improves retriever performance on M-BEIR, particularly in single model retrieval.", "weaknesses": "1. Writing clarity: some paragraphs can be revised to improve readability. (e.g., L44: define concrete examples for “diverse and complex requirements in the real-world” in the intro; L161: state the exact metric used for Table 1 (and other tables))\n2. Evidence strength: e.g., Gains of 0.1/0.3 (ID-0 vs ID-1) are tiny -- consider significance tests or a larger sample. L263: The square-root rule is argued via only two datapoints (ID-0 vs ID-2); The “from simple to complex” motivation would benefit from a curriculum-learning citation or an ablation isolating each step’s contribution.\n3. Some motivations needs to be justified: e.g., L224: \"following the principle of advancing from simpler to more complex tasks\" would benefit from either a curriculum-learning citation or an ablation isolating each step’s contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)