MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction
Authors: Zilin Xiao
Qi Ma
Mengting Gu
Chun-cheng Jason Chen
Xintao Chen
Vicente Ordonez
Vijai Mohan
Venue: ICLR 2026 Oral (2026)
Keywords: multimodal retrieval
information retrieval
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Universal multimodal embedding models have achieved great success in capturing semantic relevance between queries and candidates. However, current methods either condense queries and candidates into a single vector, potentially limiting the expressiveness for fine-grained information, or produce too many vectors that are prohibitively expensive for multi-vector retrieval. In this work, we introduce MetaEmbed, a new framework for multimodal retrieval that rethinks how multimodal embeddings are constructed and interacted with at scale. During training, a fixed number of learnable Meta Tokens are appended to the input sequence. At test-time, their last-layer contextualized representations serve as compact yet expressive multi-vector embeddings. Through the proposed Matryoshka Multi-Vector Retrieval training, MetaEmbed learns to organize information by granularity across multiple vectors. As a result, we enable test-time scaling in multimodal retrieval where users can balance retrieval quality against efficiency demands by selecting the number of tokens used for indexing and retrieval interactions. Extensive evaluations on the Massive Multimodal Embedding Benchmark (MMEB) and the Visual Document Retrieval Benchmark (ViDoRe) confirm that MetaEmbed achieves state-of-the-art retrieval performance while scaling robustly to models with 32B parameters. Code is available at https://github.com/facebookresearch/MetaEmbed.

Published as a conference paper at ICLR 2026
METAEMBED: SCALING MULTIMODAL RETRIEVAL AT TEST-TIME WITH FLEXIBLE LATE INTERACTION
Zilin Xiao! Qi Ma' Mengting Gu's Jason Chen', Xintao Chen', Vicente Ordonez', Vijai Mohan
Meta Superintellignnc Labs Riice University
min@meta com
ABSTRACT
Universal multimodal embedding models have achieved great success cap- turing semantic relevance between queries and candidates. However, current methods either condense queries and candidates into a single vector, potentially limiting the expressiveness for fine-grained information, or produce too many vectors that are prohibitive for multi-vector retrieval. In this work, we intro- duce METAEMBED, new framework for multimodal retrieval that rethinks how multimodal embeddings are constructed and interacted with at scale. During training, a fixed number of learnable Meta Tokens are appended the input sequence. At test-time, their last-layer contextualized representations serve as compact yet expressive multi-vector embeddings. Through the proposed Ma- tryoshka Relti-Vector Retrieval training, MEtA learns to organize infor- mation by granularity across multiple vectors. As a result, we enable test-time scaling in multimodal retrieval where users can balance retrieval quality against efficiency demands by selecting the number of tokens used for indexing and re- trieval interactions. Extensive evaluations ding Benchmark (MMEB) and the Visual Document Retrieval Benchmark (Vi- DoRe) confirm that METAEMBED achieves of-te-of-the-rr retrieval performance while scaling robustly to models with 32B parameters. Code is available at https: //gs::///iii www.b..huu....... com facebookeekkooo resea rch a Embed.
the Massive Multimodal Embed-
INTRODUCTION
Multimodal embedding models play essential role in image search (Gordo et al., 2016), visual question answering (Hu et al., 2018; Zheng et al., 2021) and visual document retrieval (Faysse et al., 2025), where models project heterogeneous inputs into a unified representation space. While existing methods, including CLIP (Radford et al., 2021), BLIP (Li et al., 2022) and SigLIP (Zhai et al., 2023) have demonstrated superior performance in cross-modal retrieval, their performance remains limited in scenarios where the inputs involve complex and diverse instructions. Thanks to recent advances in building embeddings through foundation vision-language models (VLMs), one could apply contrastive learning on the extracted embedding from the hidden states of the last layer of VLM learn mcaningful multimodal representations while retaining pre-trained knowledge.
Despite growing progress in multimodal embedding VLMs, the common practice of condensing the entire query and candidate into single vector is not an optimal choice, as fine-grained details are lost between modalities (Yao et al., 2022: Thrush et al., 2022) and this process has theoretical limitations (Weller et al., 2025). In text retrieval, ColBERT (Khattab Zaharia, 2020) introduced multi-vector late interaction mechanism that retains multiple token-level embeddings and uses lightweight scoring between query and document token representations. This approach preserves significantly more contextual information than single-vector methods while still allowing indepen- dent encoding of queries and documents, and has motivated recent trend of devising multi-vector embeddings for multimodal retrieval (Faysse et al., 2025; Xu et al., 2025; Günther et al., 2025).
However, multi-vector methods incur substantial efficiency costs in terms of index size, retrieval latency and feasibility. In these methods, each image is encoded into hundreds of patch embeddings, and each query text into several token embeddings, all which must be stored and compared during retrieval. This results in large index sizes and slower retrieval processing. Moreover, multimodal-to-

---

Published as a conference paper at ICLR 2026
Score
contrastive
Image
contrastive
Murden
Text
Not Supported
Score
VLM
VLM
VLM
VLM
Meta Tokens
(4 Spal
Mosiin
Score
(4t ze Gus
contrastive
VLM
VLM
Score
Figure I: Upper Left: Single vector retrieval method computes a score for each pair of query and candidate and uses contrastive objective to maximize the score for corresponding pairs. Upper Right: Multi-vector retrieval aggregates maximum similarities across vector pairs before training. Lower: META structures query and candidate vectors into hierarchical nested groups and trains coarse-to- to-finnt multi-vector embeddings that enable scalable and flexible retrieval.
multimodal retrieval becomes impractical when both the query and candidate sides contain images, as similarity computation for each query-candidate pair involves interactions between thousands of query tokens and thousands of candidate tokens, making both training and inference prohibitive due to computational demands.
In this work, we propose METAEMBED scalable late-interaction training recipe that advances multimodal retrieval with flexible multi-vector method, illustrated in Figure 1. Instead of encoding the query and candidate into one vector, we introduce small number of learnable Meta Tokens appended to the input sequence of the query and candidate. Their last-layer hidden states serve as set of contextualized representations for late interaction, namely Meta Embeddings. To enable flexible late interaction, where users can trade off retrieval accuracy against computational budget and retrieval latency, draw inspiration from Matryoshka Representation Learning (Kusupati et al., 2022) and design the Matryoshka Multi- Retrieval (MMR) module METAEMBED. By performing contrastive learning across parallel nested groups of representations at training-time, the model learns coarse-to-fine multi-vector embeddings that can be selectively utilized for late interactions depending on the computation budget test-time. Increasing the number of Meta Embeddings used at indexing improves the retrieval quality at the cost of index storage budget and retrieval latency, thus enabling test-time scaling in multimodal retrieval.
We first validate METAEMBED on the Massive Multimodal Embedding Benchmark (MMEB) (Jiang al., 2024) and Visual Document Retrieval Benchmarks (ViDoRee) v2 (Faysse et al., 2025; Macé et al., 2025), which represent comprehensive suite of retrieval tests covering images, text and visual documents. Our experiments show that METAEMBED achieves state-of-the-art retrieval per- formance across diverse scenarios. To further examine its generality and training scalability, evaluate METAEMBED with different VLM architectures and model sizes. Notably, test-time scala- bility remains effective even at 32B scale, with minimal diminishing returns models grow larger. We hope METAEMBED charts a path toward multimodal retrieval systems that are both accurate and deployable scale, advancing the pursuit of generality, efficiency, and flexibility.
RELATED WORK
Multimodal Embedding. These methods aim to project heterogeneous inputs into a shared rep- resentation space for cross-modal understanding and retrieval (e.g Frome al. (2013); Kiros et al.
2

---

Published as a conference paper at ICLR 2026
(2014); Faghri al. (2018)). More recent large scale models such CLIP (Radford et al., 2021), MetaCLIP (Xu et al., 2024; Chuang et al., 2025), BLIP (Li et al., 2022) and SigLIP (Zhai et al., 2023) encode each modality independently and apply contrastive training to enforee cross-modal align- ment. More recent methods are built upon stronger VLMSs (Xiao et al., 2024; Kong et al., 2025; Qin et al, Ju & Lee, 2025; Lin et al., 2025; Xiao et al., 2025). For instance, VIM2 (Jiang al., 2025) adapts Phi-3.5-V (Abdin et al., 2024), VLM2Vec-V2 (Meng et al., 2025) and GME (Zhang al., 2024b) builds on Qwen2 (Wang al., 2024) and LLaVE (Lan et al., 2025) finetunes on LLaVA (Li et al., 2024). Beyond architectural choices, the community has also explored innovative strategies in data construction and training. MegaPairs (Zhou et al., 2024) and mmE5 (Chen al., 2025b) curate large-scale synthetic data with sophisticated pipelines to support contrastive training. UniME (Gu et al., 2025) achieves strong results through diverse data combined with teacher model distillation. B3 (Thirukovalluru et al., 2025) incorporates novel insights into batch mining tech- niques. MoCa (Chen et al., 2025a) used continual pre- pre-training to produce bidirectional embeddings. Nevertheless, many existing multimodal retrieval methods predominantly rely single-vector trieval, which hinders further scaling as embedding size becomes a bottleneck.
Multi- Vector Retrieval. Multi-vector retrieval refers to family of dense retrieval methods that represent queries and documents with multiple embeddings rather than a single vector (Tolias et al., 2016; Tan et al., 2019; Ren et al., 2017), with ColBERT (Khattab & Zaharia, 2020) being suc- cessful recent example of this paradigm by introducing a late interaction framework. While many variants have been proposed to improve multi-vector retrieval efficiency through approximation (Lee et al., 2023; Engels et al., 2023; Jayaram et al., 2024) and compression (Santhanam et al., 2022b;a; Li al., 2023), naive ColBERT-style methods such ColPali, ColQwen (Faysse et al., 2025) and oth- ers (Günther et al., 2025; Xu et al., 2025) still remain dominant in the context of text-image retrieval. However, these approaches do not support multimodal- tooudallt--ultmmmddd retrieval, since introducing hundreds of image tokens on the query side renders both training and inference computationally prohibitive, highlighting the need for our proposed METAEMBED framework.
Matryoshka Representation Learning. Matryoshka Representation Learning (MRL) (Kusupati al.. 2022) was introduced encode features at multiple granularities within a single vector in nested structure. Popular text-only single-vector retrieval models (Zhang et al., 2025; Günther al., 2025) natively support MRL, enabling retrieval to dynamically select the first few dimensions according to the available computational budget. Although prior work (Cai et al., 2025) has ap- plied Matryoshka methods for token budgeting in VLM generation, to best of our knowledge, METAEMBED presents the first work that leverages such framework for multi-vecter retrieval and achieves successful test-time scaling.
3 METHODOLOGY
In this section, we first revisit the definition of multimodal retrieval and how late interaction works to utilize multiple vectors for retrieval. Then we introduce the METAEMBED recipe, its model architecture, and how it enables test-time scaling in multimodal retrieval.
3.1 PRELIMINARIES
Problem Definition. Multimodal retrieval consists of retrieving relevant content across different modalities, where the query q can be text gt, image q or a combination of both (gr,g). And the retrieval candidates e can likewise be of any modality or multimodal combination. Given a query q and set of N candidates {c1, C2..... CN); ). multimodal retrieval model typically defines a similarity function s(q. c) to measure the relevance between q and a candidate c. The retrieved top-1 prediction c* is then determined by:
arg max s(q.c). cE((I...,ex))
Late Interaction. For query q and a candidate c. let their multi-vector representations be denoted as Eg E IR xD and E, E NuDD, where D is the embedding dimension, and N,- Nr are the number of token-level vectors for the query and the candidate, respectively. The late interaction
3

---

Published as a conference paper at ICLR 2026
operator L[(g, d) captures the most informative alignment by selecting, for each query vector Eq its maximum similarity (dot product) with the document vectors E and summing across all query vectors:
N₂
LI(g,d) d) - Σ
E E (1) JEIL,N
max
(2)
3.2 OUR DESIGN
EMBED Recipe. METAEMBED is designed scalable late- nate-interaction retrieval model that introduces small number of learnable Meta Tokens appended to the input sequence of both queries and candidates. These Meta Tokens are processed jointly with the original input by an underly- ing Vision-Language Model (VLM), and their final hidden states serve Meta Embeddings. Unlike patch- or token-level embeddings, Meta Embeddings provide a set of compressed yet expressive vec- tors that capture fine-grained semanties through contextualization. This design drastically reduces the number of vectors required for retrieval while maintaining retrieval quality.
Formally, let a VLM with parameters 9 define conditional probability distribution pe (y x, I) where x = [.i,..., Vn] is the query or document text prompt and I are associated input im- ages. METAEMBED augments the input with learnable Meta Tokens: queries use M2 RR, and candidates use Mc E RRtxD For an input (x, the transformer consumes z
v t; M,: Mc E R(Ptn=RRtR.)xDD where and t are P visual patches tokens and text inputs. The model produces last-layer hidden states H (z E (R(P+*+R,++R.
where Fo denotes the transformer network parameterized by 0. We extract the final hidden states the Meta Tokens positions to obtain query-side embeddings E (44) E R or candidate-side embed- meta
dings E(C) E RRRx xD, followed by L2 normalization. Each (n) and E constitutes a compact, contextualized multi-vector representation produced two separate forward passes of the VLM.
Matryoshka Multi-Vector Retrieval (MMR). With E(e)
meta E RRRx and E(C) E RR&xD avail- able, we can compute late-interaction score between query q and a candidate follows:
R.
s(q. -) = Σ
(E() .E.))
max JEL.R.
While effective, using all vectors for every instance is not flexible: the index size scales O(N x Re x D) for N candidates, and the scoring cost scales as OR2, x R. x D) per pair. We therefore seek mechanism that maintains strong retrieval quality under tight resources and scales to higher accuracy as more compute is allocated. Inspired by Kusupati et al. (2022), we impose prefix-nested structure Meta Embeddings that the first few vectors form coarse summary, and additional vectors refine the representation. Concretely, fix G group sizes for queries so that
1 (1)
(G) Rg:
and for candidates that
S r( < < (2)
r Ra-
For any input, define the g-th group of query embeddings as E(q.0) = (4) (1:11) , and similarly for candidates E(c.)) F(c)
[ (1:r(
::]. We then compute group- specifce late-interaction scores
max (E(L)
(E(g.)) E(..i)
(4)
During training, optimize contrastive objectives across all groups in parallel, encouraging each prefix to be discriminative on its own while remaining consistent with larger prefixes.
Training Objective. Let B (b) ())) be minibatch where each query has cor- responding positive candidate (b) and one additional hard negative ((b,-), For each group g, we define the similarity scores between query u and candidate as follows:
S (00) c(o)).
(5)
4

---

Published as a conference paper at ICLR 2026
Index
Scoring Latency
Retrieval
Nesting
79.0
1.67
Recneval Accurccy
754
AUA
734
6.25 ms
700
Retrivaa eval Cost
Illustration of test-time scaling with varying retrieval budgets. Left: METAEMBED con- structs a nested multi-vector index that can be retrieved flexibly given different budgets. Mid- dle: How the scoring latency grows with respect to the index size. Scoring latency is reported with 100,000 candidates per query on an A100 GPU. See $5 for full efficiency analysis. Right: METAEMBED-7B performance curve with different retrieval budgets. See Figure (b) for full met-
Figure 2:
rics.
with J > 0 as temperature nyper--arameter. For query u. the denominator of the softmax spans (i) all in-batch candidates (c(1),... .c,'",,...... ,,BB) and (ii) its explicit hard negative c(u,-) The InfoNCE loss (Oord et al., 2018) for group is:
B
exp(S
NCR = - ==1
Σ
log
(6)
exp((!!)) + E vou exp() The final loss combines all groups with group- grecifc syper-paramete W/ as importance scales:
Linal Σ 9=1
Wg
NCT
Test-time Scaling. The nested design yields simple accuracy-efficieny knob, as illustrated in Figure 2. At indexing time, one may store only the first (!9) vectors for each candidate. query time, the system selects (r/10) ,((2)) based on latency constraints and computes s(9) c) for scoring. Coarse prefixes (g small) ideal for fast retrieval scoring, while larger prefixes (g large) improve precision at the expense of additional compute. Because those groups are optimized in parallel, we can seamlessly adjust the retrieval granularity and budget without retraining the system by selecting a different group size at test-time. In later sections, we refer to the selected combination of group sizes (r(9) r(())) the retrieval budget
4 EXPERIMENTS
In this section, we first introduce experimental settings, including models, training data and bench- marks. We then report comprehensive results to showcase effectiveness and robustness of METAEMBED.
4.1 SETTINGS
To evaluate the effectiveness of METAEMBED as training recipe, we conduct experiments various VLMs of different sizes, including Qwen2.5-VL (Bai al., 2025), PaliGemma (Beyer et al., 2024) and Llama-3.2-Vision (Grattafiori et al., 2024). Qwen2.5-VL and PaliGemma represent unified multimodal architectures that process text and vision inputs, while Llama-3. 3.2ma-3.2--i0 Vision represents cross-attention. based designs where visual information is integrated into the language model through cross-attention layers. In this section, META EMBED33... -7B and -32B refer to models finetuned on Qwen2.5-VV backbones and METAEMBED- 1B is finetuned on the Llama-3.2- Vision model.
Models.
Training. We train METAEMBED-P7 on 32 NVIDIA H100 SXM5 96GB GPUs for 3,500 steps with global batch size of 2, 048, which leads to 30 hours for training. Appendix A.1 has training
5

---

Published as a conference paper at ICLR 2026
Table 1: Precision (%) results on MMEB, which includes 36 tasks across four categories: Clas- sification, Visual Question Answering (VQA), Retrieval, and Visual Grounding. IND and OOD represent the in in-domain average and out-of-domain average metries, respectively. Bold denotes the best scores in the subset and the second-best scores are highlighted with underline.
Per Meta-Task Score
Average Score
Models
Size
Classification
VQA
Retrieval
Grounding
IND
OOD
Overall
Baseline Models
CLIP
428M
19.7
53.2
62.2
47.6
42.8
45.4
MagicLens
613M
38.8
8.3
35.4
26.0
27.8
UnilR
428M
42.1
15.0
60.1
62.2
42.8
ABC
7B
60.0
31.0
MM-EMBED
7B
48.1
32.2
63.8
57.8
50.0
GME
7B
56.9
41.2
67.8
53.4
55.8
7B
61.2
49.9
67.4
86.1
67.5
57.1
62.9
VIM2Vec-V2
2B
62.9
56.3
69.5
77.3
64.9
MMRet
7B
56.0
57.4
69.9
83.6
68.0
59.1
64.1
mmE5
11B
62.7
89.7
72.4
66.6
69.8
MoCa-3B
3B
59.8
62.9
70.6
88.6
72.3
61.5
67.5
MoCa-7B
7B
65.8
64.7
75,0
92.4
74.7
67.6
71.5
B3-7B
7B
70.0
66.5
74.1
84.6
75.9
67.1
72.0
METAEMBED PaliGemma Initialized
METAEMBED-3HD Gemma
3
64.9
53.5
70.9
79.5
68.6
61.3
65.4
METAEMBED
Llama- 3.2 Vision Initialized
METAEMBED-IIB
11B
66.4
74.3
91.6
65.7
64.3
METAEMBED
Qwen2.5 VL Initialized
METAËMBED-3B 3B
3B
62.7
68.1
71.9
79.6
73.5
63.8
69.1
METAEMBED-7TH
7B
71.3
74.2
78.7
85.4
81.8
70.0
76.6
METAFMBBED32B
32B
73.7
78.6
78.9
88.1
82.8
73.7
78.7
details for other variants. We use LoRA (Hu et al., 2022) with rank of 32 and scaling factor 32 in all training. For models with Matryoshka Multi-Vector Retrieval, we empirically choose G = 5 group sizes of (Ty, re) as {(1, 1), (2, 4), (4, 8), (8, 16), (16, 64)} and discuss other group size options in $4.3. Group-specific hyper- papeeepar wy in Equation is set to following Kusupati al. (2022). Contrastive training temperature 7 is set to 0.03. We only incorporate MMEB-train (Jiang al., 2025) and ViDoRe-train (Faysse et al., 2025) with one explicit hard negative from Chen et al. (2025a) for training all variants of METAEMBED.
Evaluation. We assess the general multimodal embedding ability of METAEMBED on the Massive Multimodal Embedding Benchmark (MMEB) (Jiang et al., 2024) and use Precision @ as the eval- uation metric. MMEB is an established benchmark covering tasks across four types, including classification, visual question answering (VQA) e.g. ScienceQA (Lu et al., 2022), VizWiz (Gu- rari et al., 2018), CharQA (Masry et al., 2022), retrieval across a variety
domains e.g. Visual News (Liu et al., 2021), FashionIQ (Wu al., 2021), OvenWiki (Hu et al., 2023), and visual grounding e.g. COCO (Lin et al., 2014), RefCOCO (Kazemzadeh et al., 2014; Yu et al., 2016). In addition, to compare with existing multi-vector solutions text-image retrieval,
evaluate
METAEMBED on Visual Document Retrieval Benchmarks (ViDoRe) v2 (Macé et al., 2025) and use average NDCG@5 the metric. ViDoRe (Faysse al., 2025) first introduced to benchmark visual document retrieval capabilities in different domains, and its v2 version mitigates performance saturation by including more generalized settings and incorporating multilingual subsets. We refer to Appendix A.4 for an introduction to selected baseline methods.
4.2 MAIN RESULTS
We report the overall multimodal embedding performance of different METAEMBED variants and baseline methods MMEB in Table 1. Similarly, present the visual document retrieval per- formance of METEEMBED and baselines on ViDoRe in Table 2. All METAEMBED results are reported with 16 query-side vectors and 64 candidate-side vectors, i.e. (Fg.Te) (16, 64), and

---

Published as a conference paper at ICLR 2026
Table 2: N (%) results on the ViDoRe v2 benchmark, which covers 7 tasks on visual doc- ument retrieval. "Syn" denotes synthetic data, "Mul" indicates multilingual tasks, and "Bio" refers biomedical domains.
Models
Size
ESG.Human
Eco
Bio
ESG_Syn ESG_Syn.Muu Bio_Mul
Eco
Avg.
Single-Vess Vector Retrieval
SigLIP
652M
28.8
14.0
33.8
19.8
21.9
18.2
29.8
23.8
VLM2Vec
7B
33.9
42.0
38.8
36.7
38.4
29.7
51.4
38.7
VisRAG-Ra
3B
53.7
48.7
54.8
45.9
46.4
47.7
59.6
51.0
GME
7B
65.8
56.2
64.0
54.3
56.7
55.1
62.9
59.3
mmE5
11B
52.8
44.3
51.3
55.1
54.7
46.8
48.6
50.5
MoCa-3B
3B
63.3
57.3
62.5
58.3
54.8
59.8
62.8
59.8
MoCa-7B
7B
58.8
57.6
63.2
55.3
51.4
61.3
63.8
58.8
Multi- Vector Retrieval
ColPali
3B
51.1
49.9
59.7
57.0
55.7
56.5
51.6
54.5
ColQwen2
2B
62.2
53.2
61.8
53.4
54,2
56.5
61.5
57.5
METAEMBED
META 3B
3B
63.7
55.5
61.7
62.6
57.4
58.7
62.3
60.3
METAEMBED-7B
7B
62.9
54.2
65.0
62.9
61.1
61.9
60.9
61.3
will discuss the impact of the number of Meta Embeddings used in $4.3. We conclude key observa- tions from those metries follows.
METAEMBED delivers substantial improvements over the best existing single-vector baselines at comparable model sizes. At the 3B scale, METAEMBED achieves 69.1 overall on MMEB, already surpassing MoCa-3B (67.5) with +1.6% relative improvement. At 7B, the margin widens: METAEMBED reaches 76.6, outperforming MoCa-7B (71.5) and mmE5 (69.8) by over 5-7 points. Scaling further to 32B yields 78.7 overall, a clear improvement over both the strongest baselines and our smaller variants. Importantly, the relative gains of METAEMBED increase with model size while the 3B variant offers competitive results, the 7B and 32B models establish new state-of-the- art performance with the gap over baselines widening as scale increases. This trend suggests that METAEMBED scales more favorably than prior approaches, with benefits compounding in larger regimes.
The choice of VL.M backhone has a pronounced effect on META EMBED performance across tasks. METAEMBED- IB with Llama-35 3.2- Fision backbone shows strong grounding and solid clas- sification abilities, but its VQA score drops sharply to 42.1 more than 32 points lower than the 7B model (74.2). This limitation caps its overall score at 65.1 despite ex- celling in other subtasks. In contrast, Qwen2.5-VL initialization consistently delivers balanced im- provements across all metrics: METAEMBED-78 achieves 76.6, and scaling further to 32B pushes the state of the to 78.7, with especially strong VQA and retrieval capabilities. We notice that if the underlying base model itself struggles on some domains when used as generative model, such a weakness directly propagates into METAEMBED as an embedding model. For example, Huang al. (2025) suggests that Llama-3.2 3.---.22Viii Disi-iisiio--- IB is less competitive in most zero-shot VQA benchmarks, and such weakness is inherited in METAEMBED-
demonstrates strong retrieval performance on ViDoRe v2, particularly in mul- tilingual and biomedical domains, despite not being trained on multilingual data. Even the 3B scale, METAEMBED matches or surpasses much larger baselines, showing robustness across all seven evaluation tracks. When scaled to 7B, the model yields further gains, with the largest improve- ments appearing in multilingual and biomedical domains. This is especially noteworthy given that no explicit multilingual data was included during training, suggesting that METAEMBED effectively retains and leverages cross-lingual capabilities from its backbone.
4.3 ABLATION STUDIES
To better understand METAEMBED, we design comprehensive ablation studies to investigate its test-time scaling capabilities, the effectiveness of MMR and its robustness across different models.

---

Published as a conference paper at ICLR 2026
73.0
79.0
11.2
76.8
/12
66.2
Preckiond1
69.4
69.1
75.4
75.4
75.8
+6.6
644
68.8
65.4
65.8
66-7
07.5
65.8
71.8
71.3
72,0
72.9
+5.0
71.8
78.6
72.4
74.2
72.1
62.6
60.8
60.0
60.6
162.2
+500
64.0
70.0
70.0
50.0
11.11 (2.4) 14.00 18,101 (16.64)
0111 (2:4)
14.00
(8,16)116,64)
(1.11 (2:4)
15,00 18. 18.10) 18616)16664)
(1.1) (2.4) (4.10 (1.161)
(6) Metaambeed-00
(6) Metaammeed-70
[c] MataEmbed-322
(6) MetaEmbed-111
Figure 3: Impact of retrieval budget on MMEB across METAEMBED of varying model sizes. Re- trieval budget is denoted as (fg, i.e. tuple of the number of Meta Embeddings used query and candidate side. Increasing the retrieval budget from (1,1) to (16,64) consistently improves per- formance for all model sizes, with larger gains observed in higher-capacity models. The dashed green lines indicate the best single-vector retrieval performance and red arrows indicate the absolute gain (in percentage points) between METAEMBED and single-vector retrieval.
78.7(+2.1)
79.0
76.6(+7.5)
5)
How does the performance scales with the retrieval cost? We present the performance plots of different METAEMBED models with respect to retrieval cost in Fig. 3. Data points in each plot correspond to METAEMBED performance with a specific model size with varying test-time retrieval budgets. The dashed green line marks the best-perfornance single-vector retrieval model with identical training settings. The plots show that across model sizes, the curves rise steadily more retrieval budget is allo- cated. While the improvement is modest for smaller models as METAEMBED-3B shows +3.3 points relative gain against the single-vector method, we observe that becomes more notice- able the base model size grows and METAEMBED brings the most pronounced improvements on the 32B model with a +6.6 points gain.
75.8
72.6
69.1
72.1
694
(+6:0)
[00.5]
662
MetaEmbed O Single-Vector
65.6
62.0
(a) METAEMBED with (16, 64) re- trieval budget shows less diminish- ing returns as model size scales. Green numbers denote the gain compared to the preceding model size.
Does METAΣMBED EMBED apply to pre-trained VLMS of different sizes and architectures? Fig. (d) already demonstrates that METAEMBED- 11B shows ad advantages when finetuned on differ- ent VLM, LIama-3. 2- Vision- showing the robustness of our method across architectures. Another key observation is that METAEMBED makes more effective use of larger model capacity compared to single-vector methods. Fig. 4a presents the perfor- mance of the two approaches on MMEB under identical train- ing settings as model size increases. We find that METAEMBED achieves more substantial gains than single-vector retrieval. No- tably, the improvement of the single-vector baseline from 7B to 32B is no longer statistically significant while METAEMBED still holds noticeable gain.
90.0
86.8
88.7
85.1
85.0
81.7
88.4
80.0
77.9
75.0
78.3
78.5
70.0
w/ MMR
68.9
(1.1) 12,4) (4.8) (8.16) (16.69)
(b) Average NDCG@5 (%) Vi- DoRe v1 benchmark with varying retrieval budgets on METAEMBED- 3B with and without MMR design.
How effective is the MMR design? To investigate how MMR functions, i.e. how it organizes query and candidate information in a nested order of importance, we use average NDCG@5 on ViDoRe-vi 1 and report test-time scaling curves on two variants of METAEMBED-33B with and with- out MMR in Fig. 4b. If MMR is not enabled during training, the flexibility of METAEMBED will be severely constrained, as evidenced by the substantial performance drop under low retrieval budgets. For example, the performance drop hits 9.0 points when using METAEMBED without MMR single-vector retrieval model, i.e. (ra. ro) = (1, 1). Although the gap narrows as retrieval budget in- creases, the model with MMR consistently outperforms the non-MMR model as shown in the figure. Surprisingly, we find that even at the full budget (rg. rz) = (16, 64), the MMR model still performs slightly better, demonstrating that MMR does not sacrifice the original multi vector retrieval ability at full scale. We additionally report the test-time scaling results on MMEB in Appendix A.2 where
Figure 4: Ablation studies.
MMR shows negligible performance degradation.
8

---

Published as a conference paper at ICLR 2026
Table 3: Efficiency analysis of METAEMBED-TH with different retrieval budgets on an A100 GPU with 100,000 candidates per query with scoring batch size of 1,000. Query encoding and index gen- eration latency are omitted because they remain the same for all variants. Latency refers specifically to scoring latency and mean and standard deviation of latency reported with 10 runs. Index is stored and compared with bf bfoatl 1 precision (Wang & Kanwar, 2019).
Retrieval Budget Scoring FLOPs (G) Latency (ms) Index Memory (GiB) MMEB Ace (%)
(1,
0.71
1..7
0.68
71.3
(2
5.73
1.66.0.12
2.67
72.0
(4,8)
22.94
1.67
5.34
72.9
(8, 16)
91.75
1.92.10.1
10.68
74.3
(16,66 64)
733.89
6.25
42.72
76.6
5 DISCUSSION
In this section, we mainly discuss the efficiency of METAEMBED as a flexible multi-vector retrieval method, with a focus on index memory consumption and latency under varying retrieval budgets.
typical online retrieval process consists three stages: (a) Query encoding, where the query is processed by the encoder to obtain contextualized embeddings. (b) Scoring, where query em- beddings are compared with candidate document embeddings in the index. For single-vector dense retrieval, this is dot-product operation between pairs of vectors, while in multi-vector retrieval such as METAEMBED it requires late interaction (e.g., MaxSim in Eq. 2) between multiple embeddings. sorted based the scores.
(c) Ranking, a lightweight operation where candidate documents
We report the efficiency analysis of METAEMBED-7B in Table 3 with the following observations:
1. Although the number of scoring FLOPs grows substantially with larger retrieval budgets, the scoring stage itself not compute-bounded until the extreme case (16, 64). The measured latencies remain nearly flat across moderate budgets, demonstrating that GPU throughput can accommodate the additional FLOPs without becoming a bottleneck.
The relative contribution of scoring cost the overall retrieval pipeline is negligible compared to query encoding. For instance, encoding an image query of 1024 tokens requires 42.72 TFLOPs and 788ms. These figures are orders of magnitude larger than the scoring costs reported in Table 3, indicating that efficiency improvements should primarily target encoding rather than scoring with small number of candidates.
As flexible multi-vector retrieval method, index memory consumption can grow proportionally with the retrieval budget. While this can present challenges for large deployments, the issue can be mitigated by using balanced retrieval budget more frequent offloading of index data to CPU memory.
Overall, these findings suggest that MEtAEMBED is efficient in practice. Query encoding dominates latency, scoring is lightweight under most realistic budgets with a small number of candidates, and memory scaling can be controlled by either selecting balanced retrieval budgets system-level strategies such as CPU swapping.
6 CONCLUSION
We present METAEMBED, a new paradigm for multimodal retrieval that rethinks the construction and interaction of embeddings at scale. By leveraging small set of learnable Meta Tokens and train- ing them through our proposed Matryoshka Multi- Vector Retrieval (MMR) framework, METAΣM- BED organizes information into coarse-to-fine levels of granularity. This design enables flexible late interaction that balances retrieval accuracy, index size, and latency unlocking test-time scalability for multimodal retrieval. We believe METAEMBED opens path toward more general, efficient, and controllable multimodal retrieval, bridging the gap between fine- fine-graine grained expressiveness and large-scale deployability.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Anshumali Shrivastava, Xu Han, Norman Huang and Xueyuan Su for insightful discus- sions and support. V. Ordonez is supported by National Science Foundation CAREER Award No. 2201710 and support from the Ken Kennedy Institute at Rice University.
ETHICS STATEMENT
The training and evaluation of METAEMBED rely existing multimodal datasets. While these datasets are widely adopted in the research community, they may contain biases (e.g.. cultural, de- mographic, or linguistic) that can propagate into the learned representations, We acknowledge this risk and emphasize that users should carefully consider dataset composition and potential down- stream fairness impacts when deploying such systems.
REPRODUCIBILITY STATEMENT
Experimental setups (backbones, data, metrics, and budgets) are detailed in $4.2 and $4.3. Main MMEB and ViDoRe results appear in Table and Table 2, while test-time scaling and ablations are summarized in Figures 2, 3, 4a, and 4b. Implementation and compute details (optimizers, LoRA settings, group sizes, temperatures, hardware) are provided in Appendíx A. (see Table 4); additional ablation tables and comparisons are in Appendix A.2, and baseline descriptions are compiled in Appendix A.4. Efficiency measurements and index sizing used for latency/memory reporting are discussed in $5 and tabulated in Table 3,
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jiangiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie. Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. ArXiv preprint, abs/2502. 2025.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. ArXiv preprint, abs/2407.07726 2024.
Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. In The Thirleenth International Conference Learning Representations, 2025.
Haonan Chen, Hong Liu, Yuping Luo, Liang Wang. Nan Yang, Furu Wei, and Zhicheng Dou. Moca: Modality-aware continual pre-training makes better bidirectional multimodal embeddings. ArXiv preprint, abs/2506.231 15, 2025a.
Haonan Chen, Liang Wang. Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. mme5: Improving multimodal multilingual embeddings via high-quality synthetic data. ArXiv preprint, abs/2502.08468. 2025b.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. ArXiv preprint, abs/1 2os/1604.06174, 2016.
10

---

Published as a conference paper at ICLR 2026
Yung-Sung Chuang. Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, et al. Meta clip 2: A worldwide scaling recipe. ArXiv preprint, abs/2500.22066. 2025.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009). 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE Computer Society, 2009, doi: 10. 200.0/CCR..000950000000
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson. Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. IEEE Transactions on Big Data, 2025.
Joshua Engels. Benjamin Coleman, Vihan Lakshman, and Anshumali Shrivastava. DESSERT: an efficient algorithm for vector set search with vector set queries. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural In- formation Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Fartash Faghri, David Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: improving visual- semantic embeddings with hard negatives. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, pp. 12. BMVA Press, 2018
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
Andrea Frome, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc' Aurelio Ranzato, and Tomás Mikolov. Devise: A deep visual-semantic embedding model. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Pro- cessing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 2121-2129, 2013.
Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. Deep image retrieval: Learning global representations for image search. In European conference on computer vision, pp. 241- 257. Springer, 2016.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024.
Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long. Yingda Chen, Weidong Cai, and Jiankang Deng. Breaking the modality barrier: Universal em- bedding learning with multimodal llms. ArXiv preprint, Abs/2504.17433) 2025.
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jicbo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 3608-3617. IEEE Computer Society, 2018. doi: 109/CVR... 2018.00380.
Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, and Han Xiao. jina- embeddings-v4: Universal embeddings for multimodal multilingual retrieval, 2025.

---

Published as a conference paper at ICLR 2026
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Hexiang Hu, Wei-Lun Chao, and Fei Sha. Learning answer embeddings for visual question an- swering. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 5428-5436. IEEE Computer Society, 2018. doi:
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang- Open-domain visual entity recognition: Towards recogniz- ing millions of wikipedia entities. In IEEE/CVF International Conference Computer Vi- IEEE, 2023. doi:
sion, ICCV 2023, Paris, France, October 1-6, 2023, pp. LL11991CCV511070022..111
Chengyue Huang. Yuchen Zhu, Sichen Zhu, Jingyun Xiao, Moises Andrade, Shivang Chopra, and Zsolt Kira. Mimicking or reasoning: Rethinking multi-modal in-context learning in vision- language models. ArXiv preprint, abs/2506.07936, 2025.
Rajesh Jayaram, Laxman Dhulipala, Majid Hadian, Jason Lee, and Vahab Mirrokni. MUVERA:
multi-vector retrieval via fixed dimensional encoding. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, NeurIPS 2024, Vancouver BC, Canada, December 10 15, 2024, 2024.
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. ArXiv preprint, abs/2410.05160, 2024.
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. VLM2vec: Training vision-language models for massive multimodal embedding tasks. In The Thirteenth International Conference on Learning Representations, 2025.
Yeong-Joon and Seong- Whan Lee. From generator to embedder: Harnessing innate abilities of multimodal lims via building zero-shot discriminative embedding model. ArXiv preprint, abs/2508. 005/2250.00055,, 2025.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Referring to objects in photographs of natural scenes. In Alessandro Moschitti, Bo Pang, and Walter Daele- mans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 787-798, Doha, Qatar, 2014. Association for Computational Linguis- tics. doi: 10.3115/v1/DI4 1086.
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Jimmy Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (eds.), Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 48. ACM, 2020. doi: 10.1145/337221134400777
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Jamie Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. Unifying visual-semantic embed- dings with multimodal neural language models, 2014.
Fanheng Kong, Jingyuan Zhang, Yahui Liu, Hongzhi Zhang, Shi Feng, Xiaoeui Yang, Daling Wang, Yu Tian, Fuzheng Zhang, Guorui Zhou, et al. Modality curation: Building universal embeddings for advanced multimodal information retrieval. ArXiv preprint, abs/2505. 196.0 2025.
12

---

Published as a conference paper at ICLR 2026
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard- -nyderr,,, Kaifeng Chen, Sham M. Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Sys- tems 35: Annual Conference Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. Llave: Large language and vision embedding models with hardness- weighted contrastive learning. ArXiv preprint, 0272550304812, 2025.
Jinhyuk Lee, Zhuyun Dai, Sai Meher Karthik Duddu, Tao Lei, Iftekhar Naim, Ming- Wei Chang. and Vincent Zhao. Rethinking the role of token retrieval in multi-vector retrieval. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llavt-next-interlleae:: Tackling multi-image, video, and 3d in large multimodal models. ArXiv preprint, abs/2407 00789999.7 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping ereguage-imagge pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 12888-12900. PMLR, 2022.
Minghan Li, Sheng-Chieh Lin, Barlas Oguz, Asish Ghoshal, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. CITADEL: Conditional token interaction via dynamic lexical routing for efficient and effective multi-vector retrieval. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 11891-11907, Toronto, Canada, 2023. Association for Computational Linguistics. doi:
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: experiences on accelerating data parallel training. Proceedings of the VLDB Endowment, ((12):3005-3013 2020.
Sheng-chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping- Mm-embed: Universal multimodal retrieval with multimodal 1lms, 2024.
Sheng-Chich Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS. In The Thirteenth International Conference Learning Representations, 2025.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp- 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1.
Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6761-6771, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguisties. doi: 18.18653/v 1/2021 emnlp-
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Con- ference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
13

---

Published as a conference paper at ICLR 2026
Quentin Macé, António Loison, and Manuel Faysse. Vidore benchmark v2: Raising the bar for
visual retrieval. ArXiv preprint, abs/2505.17166. 2025.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A bench- mark for question answering about charts with visual and logical reasoning. In Smaranda Mure- san, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Lingnistics: ACL 2022, pp. 2263-2279, Dublin. Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v 1/2022 1/t.:/5////202.....dnnngga///ll findings--all.17.
Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, et al. V1m2vec-v2: Advancing multimodal embedding for videos, images, and visual documents. ArXiv preprint, abs/2507 00490,, 2025.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, ArXiv preprint, abs/1807.00 0037488 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett Advances in Neural Information Process- ing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp.
2019.
Jiajun Qin, Yuan Pu, Zhuolun He, Seunggeun Kim, David Z. Pan, and Bei Yu. Unimoca: Unified modality completion for robust multi-modal embeddings. ArXiv preprint, abs/2505. 11815, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021.
Zhou Ren, Hailin Jin, Zhe Lin, Chen Fang, and Alan L. Yuille. Multiple instance visual-semantic embedding. In British Machine Vision Conference 2017, BMVC 2017, London, UK, September 4-7, 2017. BMVA Press, 2017.
Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. Plaid: an efficient en- gine for late interaction retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 1747-1756, 2022a.
Keshav Santhanam, Omar Khattab, Jon Saad-Fald Falcon, Christopher Potts, and Matei Zaharia. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.). Proceedings of the 2022 Con- ference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 3715-3734, Seattle, United States, 2022b. Association for Computa- tional Linguistics. doi:
Jan Luca Scheerer, Matei Zaharia, Christopher Potts, Gustavo Alonso, and Omar Khattab. Warp: An efficient engine for multi-vecter retrieval. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2504-2512, 2025.
Benjamin Schneider, Florian Kerschbaum, and Wenhu Chen. ABC: Achieving better control of visual embeddings using VLLMs. Transactions on Machine Learning Research, 2025. ISSN www.eeewwwww..eeeteeeeeeeeeee..e
2835-8856. URL https:////......
Chull Hwan Song, Jooyoung Yoon, Taeback Hwang, Shunghyun Choi, Ycong Hyeon Gu, and Yannis Avrithis. On train-test class overlap and detection for image retrieval. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 17375- 2024.
14

---

Published as a conference paper at ICLR 2026
Fuwen Tan, Paola Cascante- Cascante-Bonilla, Bonilla, Xiaoxiao Guo, Hui Wu, Song Feng, and Vicente Ordonez. Drinl-down: Interactive retrieval of complex scenes using natural language queries. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 2647-2657, 2019.
Raghuveer Thirukovalluru, Rui Meng. Ye Liu, Mingyi Su, Ping Nie, Semih Yavuz, Yingbo Zhou, Wenhu Chen, Bhuwan Dhingra, et al. Breaking the batch barrier (b3) of contrastive learning via smart batch mining. ArXiv preprint, 0tbb/2505.112999 11293, 2025.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. -238,, 2022.
Giorgos Tolias, Ronan Sicre, and Hervé Jégou. Particular object retrieval with integral max- max-pooling of CNN activations. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Reprexentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. ArXiv preprint, htw/2409.1219)) 2024.
Shibo Wang and Pankaj Kanwar. Bf The secret to high performance on cloud tpus. Google Cloud Blog, 2019.
Yichuan Wang, Shu Liu, Zhifci Li, Yongji Wu, Ziming Mao, Yilong Zhao, Xiao Yan, Zhiy- ing Xu, Yang Zhou, Ion Stoica, al. Leann: A low-storage vector index. arXiv preprint orXiv:2506.08277 2025.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In Ales Leonardis, Elisa Ricei, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gil Varol (eds.), Computer Vision ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXXVII, volume 15145 of Lecture Notes in Computer Science, pp. 387-404 Springer, 2024. doi:
001779978-3-011-3222
23.
Orion Weller, Michael Boratko, Iftekhar Naim, and Jinhyuk Lee. On the theoretical limitations of embedding- retrieval. ArXiv preprint, abs/2508.21038. 2025.
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogério Feris. Fashion IQ: A new dataset towards retrieving images by natural language feedback. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 202 pp. 11307-11317. Computer Vision Foundation / IEEE, 2021. doi: 00.110//CPR44443..00 22222
01115.
Zilin Xiao, Ming Gong, Paola Cascante-Bonilla Xingyao Zhang, Jie Wu, and Vicente Ordonez. Grounding language models for visual entity recognition. In European Conference on Computer Vision, pp. 393 411. Springer, 2024.
Zilin Xiao, Pavel Suma, Ayush Sachdeva, Hao-Jen Wang, Giorgos Kordopatis Giorgos To- lias, and Vicente Ordonez. Locore: Image re-ranking with long-context sequence modeling. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9580-9590, 2025.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Yo-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
15

---

Published as a conference paper at ICLR 2026
Mengyao Xu, Gabriel Moreira, Ronay Ak, Radek Osmulski, Yauhen Babakhin, Zhiding Yu, Benedikt Schifferer, and Even Oldridge. Llama nemoretriever colembed: Top-performing text- image retrieval model. ArXiv preprint, abs/2507.055133 2025.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: fine-grained interactive language-image pre-training. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling con- text in referring expressions. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 69 85, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46475-6
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 11941-11952. IEEE, 2023. doi: 10.1100// 2023.01100.
Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming-Wer Wei Chang- Magiclens: Self-supervised image retrieval with open-ended instructions. In Forty-first International Conference on Machine Learning. JCML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a.
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal 1lms, 2024b.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. ArXiv preprint, abs/2506.05 2025.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, al. Pytorch fsdp: Experiences scaling fully sharded data parallel. Proceedings of the VLDB Endowment, 16(12):3848 -8688, 2023.
Wenfeng Zheng, Lirong Yin, Xiaobing Chen, Zhiyang Ma, Shan Liu, and Bo Yang. Knowledge base graph embedding module design for visual question answering model. Pattern recognition, 220:10810 10110081111111 2021.
Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval. ArXiv preprint, abs/a412..141///1 2024.
16

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 IMPLRMENTATION DETAILS
Table 4: Training details of METAEMBED variants.
Batch Size
Learning Rate
Training Cost
Embedding Dim
METAEMBED-3B Gemma
2,048
1 1x 10-4
32 H100s for 14h
2,048
METAEMBED-3B
2,048
1 10-4
32 H100s for 23h
2,048
EETAEMB
1,536
1 10-4
32 H100s for 30h
3,584
METAEMBED-IID METAEMBED-32B
1,024
1 10
32 H100s for 10h
4,096
1,536
1 10-5
64 H100s for 25h
5,120
We list the details of each METAEMBED variant in Table We reserve 1% training data from each subset of MMEB-train as evaluation split and training was early stopped when evaluation loss stops dropping. All models are trained with gradient checkpointing (Chen al., 2016) reduce memory usage. We use Adam optimizer (Kingma, 2014) with betas of 0.9, 0.999 and no weight training configurations. We use
decay and apply linear learning rate warmup of 500 steps LoRA (Hu et al., 2022) with rank of 32 and scaling factor = 32 in all training. Contrastive training temperature T is set to 0.03. Training was conducted using PyTorch (Paszke et al., 2019) 2. 6.0+cu124 4 and FlashAttention 2.0 (Dao, 2024). For the 3B configuration,
adopt Distributed Data Parallel (Li et al., 2020) while for all other model sizes we use Fully Sharded Data Parallel (FSDP) (Zhao al., 2023) v2. To prevent distributed hanging when training samples contain no images under FSDP, we pad those samples with a placeholder image to ensure the visual encoder activated on each GPU. We the default system prompts of the base models. Each training query in MMEB-train (Jiang et al., 2024) contains a task-specific text instruction, and we keep them during training following common practice in the community. Those instructions are reused during training and evaluation on the same type of tasks.
For efficiency analysis implementation in $5, we store the index in native PyTorch tensors and use PyTorch torch. einsum to implement late interaction operation following the practice in Faysse al. (2025). This controlled setup allows us to compare the cost of different retrieval budgets with- out additional lestem-level engineering. In a realistic large-scale deployment, one would typically rely on specialized similarity-search frameworks such as FAISS (Douze et al., 2025) or systems op- timized for multi-vector indexes (e.g.. WARP (Scheerer et al., 2025), LEANN (Wang et al., 2025)), which provide more efficient memory management and higher throughput. Therefore, the latencies reported in Table 3 should be interpreted as conservative upper bounds rather than results from an optimized production system.
A.2 DETAILED MMEB ABLATION RESULTS
To further compare the superiority of METAEMBED against both single-vector and multi-vector retrieval methods, we design the following baselines based on the same pre-trained models for fair comparison:
single-last: a single-vector retrieval method that uses the last-token hidden state from the last layer as the retrieval representation.
single-mean:
similar to the above, but applies average pooling over all last-layer hidden states to obtain the retrieval representation.
3. s1lit-(16, 64): a simple multi-vector retrieval baseline where the query-side last-layer hidden states evenly partitioned into 16 segments, with the mean of each segment taken as 16 query vectors; the same process produces 64 candidate-side vectors. This method does not introduce additional parameters, making it a suitable fixed-length multi-vector retrieval baseline.
Our findings indicate that METAEMBED goes beyond test-time scaling advantages mentioned in the main text: it consistently outperforms the top single-vector method as well as naive multi-vector baseline. Moreover, the MMR design brings no statistically significant loss, demonstrating that adds flexibility while maintaining retrieval quality.
17

---

Published as a conference paper at ICLR 2026
Table 5: Comparison between METAEMBED and single-vector multi-vector retrieval models trained with identical settings. NoMMR indicates Matryoshka Multi- Vector Retrieval (MMR) is disabled. A denotes the difference to the best single-vector retrieval method, i.e. single-last.
3B
7B
32B
11B
Type
ММEB
A
MMEB
A
MMEB
A
MMEB
A
single-last
65.6
71.6
0
72.1
60.1
0
single-mean
65.2
-0.4
71.2
-0.4
71.1
-1.0
58.1
-2.0
split-(16, 64)
64.2
-1.4
70.1
-1.5
70.5
-1.6
56.0
-4.1
METAEMBED
(1,
65.4
-0.2
71.3
-0.3
72.4
+0.3
60.0
-0.1
(2, 4)
65.8
+0.2
72.0
+0.4
73.2
+1.1
60.6
+0.5
(4,8)
66.7
+1.1
72.9
+1.3
74.2
+2.1
62.2
+2.1
(8, 16)
67.5
+1,9
74.3
+2.7
75.8
+3.7
63.3
+3.2
(16, 64)
69.1
+3.5
76.6
+5.0
78.7
+6.6
65.1
+5.0
NoMMR-(16, 64)
69.3
+3.7
77.0
+5.4
79.1
+7.0
66.2
+6.1
A.3 МMEB PER-TASK SCORE
We present the per-task results of METAEMBED and baseline models on the MMEB bench- mark (Jiang et al., 2024) in Table 6. While the in-domain (IND) and out-of-domain (OOD) setup follow the previous multimodal retrieval literatures (Jiang et al., 2025; Chen et al., 2025a; Thiruko- valluru al., 2025; Meng et al., 2025), we believe such an evaluation protocol is subject known issues arising from the train-test class overlap. As the anonymous reviewer pointed out, Object- Net (Barbu et al., 2019) and ImageNet (Deng et al., 2009) have 113 overlapped classes. Song al. (2024) demonstrates that such overlap can bias performance estimates. Therefore, we call for more principled evaluation protocols in the topic of multimodal retrieval.
A.4 BASELINE METHOD INTRODUCTION
For clarity and completeness, we provide short introductions to the baseline methods considered in Table 1. All performance metries reported on baseline methods are directly taken from the corre- sponding original papers or Chen et al. (2025a).
CLIP (Radford et al., 2021). CLIP is dual-encoder trained with contrastive learning on 400M image-text pairs. It learns aligned representations for both modalities, enabling strong zero-shot classification and retrieval capabilities.
MagicLens (Zhang et al., 2024a). Magiclenns is lightweight dual-encoder for instruction- guided image retrieval. It is trained self-supervised manner on roughly 36.7M (query-image, text instruction, target-image) triplets mined from co-occurring web images.
UnilR (Wei et al., 2024). UnilR is unified, instruction-guica multimodal retriever that handles eight retrieval task formats spanning text, image, and mixed-modality queries/candidates. It jointly trained on ten heterogeneous datasets, showing robust in-distribution performance and zero-shot generalization across tasks.
ABC (Schneider et al., 2025). ABC is an open-source multimodal embedding model designed to unify visual and textual representations under explicit natural language control. To evaluate in- struction sensitivity, the authors introduce CtrlBench, benchmark requiring retrieval conditioned on subtle, interleaved visual and linguistic cues. ABC demonstrates strong generalization on CtrlBench, validating its ability to produce controllable, high-fidelity visual embeddings. also achieves good performance MMEB in zero-shot settings.
18

---

Published as a conference paper at ICLR 2026
Table 6: Detailed performance on 36 MMEB tasks. Table style and baseline performance are adopted from Chen et al. (2025a). Rows in yellow indicate metrics of an OOD task.
Task
CLIP OpenCLIP VeM2 MMRet mmES MoCa-3B MoCa-7B METAËMBED-7E
Classification (10 tasks)
ImageNet-1K
55.8
63.5
74.5
58.8
77.6
75.4
78.0
88.1
N24News
34.7
38.6
80.3
71.3
82.1
80.9
81.5
84.3
HatefulMemes
51.1
51.7
67.9
53.7
64.3
70.6
77.6
78.7
VOC2007
50.7
52.4
91.5
85.0
91.0
87.0
90.0
94.1
SUN397
43.4
75.8
70.0
77.9
74.8
76.8
83.3
Place365
28.5
44.0
43.0
42.6
43.0
48.6
ImageNet-A
25.5
14.2
43.6
36.1
56.7
39.7
52.7
60.9
ImageNei-R
75.6
83.0
79.8
71.6
86.3
75.4
83.0
88.1
ObjectNett
43.4
51.4
39.6
55.8
62.2
31.3
45.2
56.8
Country-211
19.2
14.7
14.7
34.8
24.0
30.4
35.4
Classification
42.8
61.2
56.0
VQA (10 tasks)
OK-VQA
7.5
73.3
A-OKVQA
3.8
3.3
54.4
56.7
56.4
54.6
57.1
69.1
Doc
4.0
5.3
52.0
78.5
90.3
93.0
94.3
96.5
Infographics
4.6
4.6
30.7
39.3
56.2
67.7
77.2
82.1
ChartQA
1.4
1.5
34.8
41.7
50.3
64.1
69.8
78.3
7
4.0
49.8
49.5
51.9
72.9
ScienceQA
9.4
10.2
42.1
45.2
55.7
45.4
59.2
57.4
VizWiz
8.2
6.6
43.0
51.7
52.8
52.3
46.2
55.5
GQA
41.3
52.5
61.2
59.0
62.1
66.9
71.6
67.2
TexIVQA
7.0
10.9
62.0
79.0
83.5
83.1
75.8
85.8
Aug. VQA
9.1
10.9
49.9
57.4
62.7
62.9
64.7
74.1
Retrieval (12 tasks)
VisDial
30.7
80.9
83.0
73.7
84.5
CIRR
12.6
49.9
61.4
VisualNews. 12i
78.9
74.0
75.4
74.2
77.7
74.4
78.2
80.6
NewwalNews i2t
79.6
78.0
80.0
78.1
83.4
77.8
83.1
84.3
MSCOCO.1.
59.5
63.6
75.7
78.6
76.2
76.4
79.8
85.2
MSCOCO
57.7
62.1
73.1
72.4
73.6
72.6
73.9
82,2
NIGHTS
60.4
65.5
68.3
75.5
WebQA
67.5
62.1
87.6
90.2
88.1
90.6
91.4
93,0
FashionIQ
11.4
13.8
16.2
54.9
28.6
22.2
28.9
34.1
Wiki-SS-No
55.0
44.6
60.2
24.9
65.2
73.3
82.7
83.6
OVEN
41.1
45.0
56.5
87.5
77.3
75.9
80.4
83.3
EDIS
81.0
87.8
65.6
83.6
96.9
Avg. Retrieval
53.0
52.3
67.4
69.9
71.0
70.6
75.0
78.6
Visual Grounding (4 tasks)
MSCOCO
33.8
80.6
76.8
84.6
78.0
RefCOCO
56.9
54.2
88.7
89.8
92.7
92.1
94.0
94.7
ReiCOCO-matehe
61.3
68.3
84.0
90.6
88.9
92.8
95.5
93.4
Visual7W-poinnee
55.1
56.3
90.9
77.0
92.3
89.5
95.3
87.2
Avg. Grounding
51.8
53.3
86.1
83.6
89.7
88.7
92.4
88.3
Final Score (36 tasks)
All IND Avg.
37.1
39.3
67.5
59.1
72.4
72.3
74.7
81.8
OOD Avg.
38.7
57.1
68.0
67.6
71.0
All Tasks Avg.
37.8
39.7
62.9
64.1
69.8
67.5
71.5
76.6
MM-Embed converts multimodal large language model into universal bi-encoder for retrieval. It is fine-tuned with modality- hard negatives across diverse retrieval datasets to improve cross-modal alignment.
MM-Fmbed (Lin al., 2024).
GME (Zhang et al., 2024b). The General Multimodal Embedder is trained large synthetic dataset containing diverse multimodal queries and documents. introduces fused-modal training examples (mixed text- imxte inputs) to enable universal any-to-any modality retrieval.
VLM2Vec (Jiang et al., 2025). VLM2Vec transforms a pretrained vision- languaee model into universal embedding model through instruction-tunal contrastive learning. It is trained the Mas- sive Multimodal Embedding Benchmark (MMEB), covering tasks across classification, VQA, retrieval, and grounding.
19

---

Published as a conference paper at ICLR 2026
MMRet (Zhou al., 2024). MMRet builds on the MM-Embed framework but introduces further refinements negative sampling and large-scale fine-tuning on massive synthetic dataset, making one of the strongest retrieval-centric embedding models in prior work.
mmES (Chen et al., 2025b), mmE5 extends the multilingual text embedding model ES into the multimodal setting. It trained with multilingual and multimodal signals, leveraging synthetic image-text pairs and hard negatives, and achieves strong state-of-the- results on MMEB prior to more recent models.
MoCa (Chen et al., 2025a). MoCa (Modality-aware Causal Pre-training) introduces two-stage process: modality-aware continual pre-training to adapt causal vision language models for bidirec- tional encoding, followed by heterogeneous contrastive fine-tuning across text, image, and mixed modality pairs. We evaluate both MoCa-3B and MoCa-7B, which show competitive overall perfor- mance among baselines MMEB.
B3 (Thirukovalluru et al., 2025) B3-7B is 7B p-rammetter instruction-tuie multimodal retriever with specialized batch mining techniques. achieves strong results on MMEB and serves as an additional competitive baseline.
A.5 THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language models (LLMs) solely for minor language editing purposes, such as catch- ing grammar errors and improving elarity of expression. The LLM did not contribute research ideation, analysis, or substantive writing of the paper. All conceptual development, methodology. results, and interpretations conducted entirely by the authors.
20


Reviews:
[{"review_id": "iKCAHAdsqB", "reviewer": "Reviewer_L8o6", "summary": "The authors propose MetaEmbed which uses grouped multimvector embeddings to scale the \"test-time compute\" used during retrieval. MetaEmbed introduces a tradeoff between resources (computation, memory), and multimodal retrieval quality. The approach is validated on both general multimodal retrieval tasks (MMEB) and document retrieval (ViDoRe).", "questions": "Some questions can be found in the section above, in addition:\n\n- Can the authors provide a per-task breakdown of MMEB? Although averages over MMEB tasks are often reported, not all tasks in MMEB are equally useful when evaluating a model, therefore a per-task breakdown would be useful.\n   \n- How are the text queries templated into context? Are task-specific system prompts reused between training and evaluation?\n\n- A comparison between the late interaction approach and a single vector that controls for dimension would be interesting (potentially created by concatenating across the sequence dimension).\n\n[1] R. Meng et al., “VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents,” 2025, doi: 10.48550/arxiv.2507.04590.  \n[2] C. Jia et al., “Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,” 2021, doi: 10.48550/arxiv.2102.05918.  \n[3] M. Douze et al., “THE FAISS LIBRARY,” IEEE transactions on big data, pp. 1–17, 2025, doi: 10.1109/TBDATA.2025.3618474.  \n[4] B. Schneider, F. Kerschbaum, and W. Chen, “ABC: Achieving Better Control of Visual Embeddings using VLLMs,” Transactions on Machine Learning Research, vol. 2025-, 2025.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The manuscript is clearly written and well structured.\n    \n- The introduction of a latency-accuracy tradeoff mechanism into multimodal retrieval is novel and well-motivated.\n    \n- The authors demonstrate the value of MetaEmbed across a good variety of multimodal embeddings tasks (both general and visual documents).\n    \n- The MMEB and ViDoRe results are mostly fair, with the majority of compared works having similar training distributions.", "weaknesses": "- The training data has large distribution overlap with evaluation tasks (especially on key tasks, like ImageNet-1K, ViDORe). Therefore, it is unclear how well the model will generalize.\n    \n- It is unclear in tables 1 and 2 which models are trained on MMEB/ViDoRe (and therefore, strongly advantaged). For clarity, it would be valuable to specify which models are in-distribution vs out-of-distribution when reporting performance.\n    \n- Several relevant training settings are not included in appendix A.1, such as Optimizer, optimizer hyperparameters, weight decay, learning rate scheduling, was temperature fixed [1] or trained [2]?\n    \n- The analysis of latency is limited and potentially misleading. How is the index stored? A realistic setup would use an appropriate framework like FAISS [3]. Can the authors provide more details about their inference setup (table 3) and justify its relationship to a realistic deployment?\n    \n- A common use-case would be short text queries and a much larger candidate pool, in which case scoring latency may become significant.\n    \n- The device memory requirements for the best performing group settings are very significant, even for the small candidate pool size. This is both a constraint in itself, and can incur significant performance penalties if the index cannot fit into GPU memory.\n\nMinor:\n\n- The reproducibility statement says: “Implementation and compute details (optimizers, LoRA settings, group sizes, temperatures, hardware) are provided in Appendix A.1.” However, some of these details are found elsewhere, whereas others seem to be missing entirely. Ideally, these settings should all be in A.1, even if it results in some duplication from the main text.\n\n- Potentially relevant related work: [4] seems to corroborate the importance of VLM backbone choice for multimodal retrieval, and may be worth including in section 4.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SKaUkKqgX3", "reviewer": "Reviewer_dVyq", "summary": "This paper introduces a framework designed to resolve the critical trade-off between expressiveness and efficiency in multimodal retrieval by using a small set of learnable \"Meta Tokens\" to form a compact multi-vector representation. The central contribution is the Matryoshka Multi-Vector Retrieval (MMR) training objective, which organizes information into a coarse-to-fine structure. This enables a single model to be flexibly scaled at test-time to meet varying accuracy and efficiency demands without retraining.\nOverall, this is a quality paper with interesting findings that could have significant applications for the research community. Although some questions and concerns remain, as detailed below, I am willing to raise my score if these points are adequately addressed.", "questions": "- (Regarding the 2nd weakness) To better understand the mechanism behind METAEMBED, could you provide some qualitative analysis of the learned Meta Embeddings? For instance, what kinds of visual or textual concepts do different tokens appear to specialize in? Does the first token of an MMR-trained model consistently capture more global, summary-level information compared to later tokens, which might focus on finer details?\n\n-The choice of MMR group sizes {(1, 1), (2, 4), (4, 8), (8, 16), (16,64)} appears somewhat heuristic. How sensitive is the model's performance to this specific configuration? For example, what happens if a more linear scaling is used, and what was the process for selecting this particular set of nested budgets?\n\n-The split-(16,64) baseline in Table 5 performs worse than the single-last baseline. This is counter-intuitive, as one might expect a multi-vector representation, even a naive one, to retain more information. Does this imply that without explicit training tailored for late interaction, simply partitioning a representation is detrimental?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Novel Combination of Existing Concepts: novel synthesis of learnable tokens, late-interaction, and Matryoshka learning, resulting in the technically solid Matryoshka Multi-Vector Retrieval objective.\n\n- The experimental evaluation is thorough and convincing, demonstrating state-of-the-art performance on diverse benchmarks like MMEB and ViDoRe v2.", "weaknesses": "- Lack of Interpretability and Qualitative Analysis: The \"Meta Tokens\" are treated as a black box. The paper offers no analysis or visualization to explain what these tokens learn, missing an opportunity to provide deeper scientific insight beyond a strong engineering result.\n\n- A key motivation presented in the introduction of the paper is solving the efficiency problem for multimodal-to-multimodal (universal) retrieval, yet no experiments on such a task are presented. This leaves a core claim of the paper unsubstantiated by empirical evidence. Specifically, as far as I know, the evaluation on the MMEB and ViDoRe v2 benchmarks lacks an image-to-image retrieval task (which is different from the visual document retrieval task or image+text retrieval task), which would be a key area for qualitative analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cU6s6g04pu", "reviewer": "Reviewer_g6en", "summary": "The paper introduces METAEMBED, a novel framework designed to enhance multimodal retrieval by addressing the key challenges of scalability, efficiency, and fine-grained retrieval accuracy. Existing methods either compress queries and candidates into a single vector, losing fine-grained information, or generate too many vectors, leading to computational inefficiencies. METAEMBED introduces Meta Tokens, which are small, learnable tokens appended to both queries and candidates, and their final contextualized representations serve as compact yet expressive multi-vector embeddings during retrieval. By employing a Matryoshka Multi-Vector Retrieval (MMR) mechanism, the model organizes embeddings into hierarchical groups, enabling flexible scaling at test-time where users can adjust retrieval quality against computational resources.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The introduction of Meta Tokens and the Matryoshka Multi-Vector Retrieval (MMR) approach provides a unique solution to the scalability and efficiency issues in multimodal retrieval, allowing flexible scaling at test-time.\n2. METAEMBED demonstrates remarkable scalability, allowing users to adjust retrieval quality based on computational constraints without retraining the model, a critical advancement for real-world deployment.\n3. The paper provides extensive evaluations across various model sizes and VLM architectures, showcasing METAEMBED's robustness and effectiveness in diverse scenarios, including challenging tasks like visual document retrieval and multimodal embedding.", "weaknesses": "1. The design intuition of meta embedding is somewhat similar to the concept of query extension in the early information retrieval field. Could the authors further differentiate between these two design intuitions?\n2. The approach of using embedding with different granularities for retrieval is promising. Have the authors explored or statistically analyzed which granularity of embedding provides better discrimination? This could offer valuable guidance for future embedding extraction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PSyCfy9f5X", "reviewer": "Reviewer_rgEq", "summary": "This paper introduces METAEMBED, a multimodal embedding learning framework designed for scalable late interaction. METAEMBED adds a small number of learnable Meta Tokens to multimodal inputs and trains them using a Matryoshka Multi-Vector Retrieval (MMR) strategy, which organizes embeddings hierarchically to enable test-time scaling. Specifically, MMR enables test-time scaling balancing retrieval accuracy and latency. \nThe authors evaluate METAEMBED on MMEB and ViDoRe v2 benchmarks across multiple vision-language backbones (e.g., Qwen2.5-VL, PaliGemma, and LLaMA-3.2-Vision), showing state-of-the-art retrieval performance and flexible inference efficiency across different VLM varying backbones and model sizes.", "questions": "1. In Table 2, ColQwen2 should correspond to a 3B model rather than 2B.\n2. “P” in L182 likely refers to number of visual patch tokens and should be denoted.\n3. How do you empirically choose a group size of 5 and with these configurations for MRL?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. Strong performance: METAEMBED demonstrates strong performance across diverse domains and setups over baseline retrievers.\n2. General applicability across VLM backbones and model sizes: METAEMBED is evaluated on Qwen2.5-VL, PaliGemma, and LLaMA-3.2-Vision, show that it generalizes to different model architectures and parameter scales.\n3. Comprehensive analysis of METAEMBED: the authors provide comprehensive ablations such as the effectiveness of MMR component, efficiency analysis, analysis on different VLM backbones.", "weaknesses": "1. Limited originality beyond adaptation of MRL: The main innovation lies in adapting Matryoshka Representation Learning (MRL) to multi-vector retrieval. While this adaptation is meaningful, much of the practical foundation (i.e., multimodal representation learning with Matryoshka nesting) derives from prior work (Cai et al. 2025).\n2. Insufficient analysis of efficiency in training: Although efficiency is discussed in the context of query encoding at test-time (Table 3). Comparing computational costs in training across different retriever methods can further strengthen the paper, as additional training costs incurred could be a trade-off for enabling flexible test-time compute.\n3. Empirical gaps in hyperparameter justification: the paper fixes the group sizes to be G=5 without ablation on group size for MRL. A sensitivity study on different group configurations would strengthen the argument for generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)