Think Then Embed: Generative Context Improves Multimodal Embedding

## Abstract

Abstract There is a growing interest in Universal Multimodal Embeddings (UME), where models are required to generate task-specific representations. While recent studies show that Multimodal Large Language Models (MLLMs) perform well on such tasks, they treat MLLMs solely as encoders, overlooking their generative capacity. However, such an encoding paradigm becomes less effective as instructions become more complex and require compositional reasoning. Inspired by the proven effectiveness of chain-of-thought reasoning, we propose a general Think-Then-Embed (TTE) framework for UME, composed of a reasoner and an embedder. The reasoner MLLM first generates reasoning traces that explain complex queries, followed by an embedder that produces representations conditioned on both the original query and the intermediate reasoning. This explicit reasoning step enables more nuanced understanding of complex multimodal instructions. Our contributions are threefold. First , by leveraging a powerful MLLM reasoner, we achieve state-of-the-art performance on the MMEB-V2 benchmark, surpassing proprietary models trained on massive in-house datasets. Second , to reduce the dependency on large MLLM reasoners,
we finetune a smaller MLLM reasoner using high-quality embedding-centric reasoning traces,
achieving the best performance among open-source models with a 7 % 7\% absolute gain over recently proposed models. Third , we investigate strategies for integrating the reasoner and embedder into a unified model for improved efficiency without sacrificing performance.

## 1 Introduction

Multimodal embedding-based retrieval has emerged as a popular and effective solution for handling diverse data types such as text, images, and videos (Radford et al., 2021; Li et al., 2023a; Yu et al., 2022; Li et al., 2023b). Traditionally, these models focus on learning general-purpose representations that capture content similarity across modalities.
Recently, there has been growing interest in instruction-aware Universal Multimodal Embeddings (UME) (Jiang et al., 2024; Gu et al., 2025). UMEs are powerful representations because they bridge the gap between general-purpose content similarity and user-specific requirements. This capability unlocks a wide range of applications, such as retrieving documents or images based on nuanced queries, powering personalized recommendation systems, supporting multimodal search with complex transformations, and tackling knowledge-intensive tasks where the same input may yield different embeddings depending on the downstream objective. To advance this direction, recent benchmarks like MMEB (Meng et al., 2025) aggregate diverse, instruction-driven retrieval tasks—including VQA, grounding, and document retrieval—where both the input and the user instruction determine the retrieval target.

In this context, leveraging multimodal large language models (MLLMs) as encoders for UME is a promising new approach (Jiang et al., 2024; Gu et al., 2025; Lan et al., 2025). MLLMs can generate instruction-aware representations that incorporate long context and nuanced user guidance (Peng et al., 2024). However, while state-of-the-art models on these benchmarks have advanced through techniques such as hard negative sampling (Lee et al., 2025; Thirukovalluru et al., 2025; Lan et al., 2025; Lin et al., 2025), additional training stages (Gu et al., 2025; Yu et al., 2025; Lin et al., 2025), external data (Yu et al., 2025; Chen et al., 2025b), and improved embedding extraction (Faysse et al., 2024), a central challenge remains: understanding instructions of varying complexity that require different levels of reasoning. Addressing this challenge calls for leveraging the generative capacity of MLLMs, rather than restricting them to encoder-only models.

Figure: Figure 1: Given a multi-modal input, we want to first think about the desired embedding content. The representation is conditioned on both original input and the thinking result.
Refer to caption: x1.png

Our approach is motivated by a closer examination of the tasks in MMEB-v2 (Meng et al., 2025), revealing that many, such as VQA, classification, visual grounding, and composed retrieval, require substantial reasoning capabilities. For example, the first case in Fig. [1](https://arxiv.org/html/2510.05014v3#S1.F1) (from the MMEB-v2 RefCOCO dataset) involves retrieving the image region referred to by the query “vehicle second closest to camera”. Rather than simply encoding the query and matching it to the target image, we argue that it is beneficial for the model to first *reason* about the query—identifying, for instance, that it refers to the vehicle with “bright yellow on the upper half and deep blue on the lower”. For such complex tasks, we believe that explicitly augmenting multi-step reasoning and contextual grounding enables the model to achieve more fine-grained and accurate retrieval.

To address these challenges, we propose Think-Then-Embed (TTE) for UME. The key idea is to introduce an explicit thinking stage before embedding, in which the model generates reasoning traces based on the given instruction. Prior work has shown that intermediate reasoning processes, such as Chain-of-Thought (CoT) (Wei et al., 2023b), can significantly improve the accuracy of language model generation. We hypothesize that a similar benefit can be achieved for multimodal representation learning, since the embeddings produced by MLLMs are inherently conditioned on the sequence of preceding tokens. In this work, we investigate how explicit chain-of-thought reasoning can enhance universal multimodal embeddings, by enabling the model to better interpret and follow complex task instructions.

In this paper, we made several contributions. First, we proposed a Think-Then-Embed (TTE) framework that by exploiting reasoning traces generated by a powerful reasoner (e.g. Qwen2.5 VL 72B), it can achieve state-of-the-art performance on MMEB V2 (Meng et al., 2025) with a smaller embedder (Qwen2 7B), surpassing close-sourced models trained with additional data. This demonstrates that CoT can also benefit representation learning.

Second, we study a recipe to show how a backbone model can be effectively used both as a reasoner and as an embedder. We experimented with two approaches to achieve this. The first approach uses the backbone as a zero-shot reasoner while fine-tuning a copy of the same backbone as the embedder. This method yields noticeable improvements on smaller datasets, but the gains diminish as the dataset size increases. Motivated by this limitation, our second approach involves fine-tuning a second copy of the backbone (same size as the embedder, either 2B or 7B) as a strong reasoner, using reasoning traces generated by the 72B reasoner model as the training data. The resulting SFTed reasoner performs effectively across the entire MMEB v2 benchmark, achieving the best performance among all open source models.

Finally, to improve efficiency of both inference and number of parameters in the TTE framework, we explore the capabilities of instruction reasoning and embedding creation within a single unified model. Although prior work (Yu et al., 2025) also investigated the unification of generation and contrastive learning, it treats generation and embedding as separate, unrelated tasks.
In contrast, our focus is on enhancing embedding through reasoning, that is, generate first, then embed. We focus on two strategies for this unification: (1) joint contrastive-autoregressive training of the same backbone for reasoning and embedding tasks, and (2) a two-stage autoregressive-then-contrastive training with a dedicated embedding head on top of the reasoner. Empirically, we find that the two-stage approach consistently outperforms joint training, almost halving the overall parameters, while not degrading end-to-end TTE retrieval performance.

## 2 Related Work

Multimodal Embeddings. The multimodal representation paradigm has been popularized by large-scale foundational models such as CLIP (Radford et al., 2021; Xu et al., 2024), BLIP (Li et al., 2023a), and SigLIP (Zhai et al., 2023), which encode images and texts using separate uni-modality encoders and enforce alignment using contrastive objectives.

Universal Multimodal Embedding.
Recently there has been growing interest in developing Universal Multimodal Embedding (UME) (Gu et al., 2025; Jiang et al., 2024; Meng et al., 2025), where the embedding depends both on the query and task instruction. Representative examples include VLM2Vec (Jiang et al., 2024), which proposes the Massive Multimodal Embedding Benchmark (MMEB), comprising a wide range of cross / multimodal retrieval tasks, as well as non-conventional retrieval tasks such as VQA, grounding, and classification. Subsequently, MMEB-V2 (Meng et al., 2025) is proposed, extending MMEB-V1 to include video and visual document (visdoc) tasks.

MLLM-based Embedding Models. Recent studies have gone beyond dual encoder setups, building embedding models directly on top of powerful Multimodal Language Models (MLLMs) (Jiang et al., 2024; Meng et al., 2025; Gu et al., 2025; Yu et al., 2025; Thirukovalluru et al., 2025). Existing work on MLLM-based embedding typically explores a training strategy such as text-continual contrastive training (Gu et al., 2025; Yu et al., 2025; Chen et al., 2025a), hard negative mining approach (Lee et al., 2025; Lan et al., 2025; Thirukovalluru et al., 2025), additional training data (Zhou et al., 2024; Chen et al., 2025b), and architectural design (e.g. with / without causal mask) (Lee et al., 2025; Chen et al., 2025a). However, these methods all treat MLLMs solely as encoders, while overlooking their generative capability obtained from pre-training. In contrast, we explore an orthogonal approach, where we leverage MLLMs for both generative reasoning and representation learning.

LLM-based Query Rewriting. Frequently used in text-based retrievals, query rewriting (Ma et al., 2023) is an effective approach to improve retrieval accuracy. Popular trends include prompt-based rewriting (Wilson et al., 2025; Ye et al., 2023), iterative or Reinforcement Learning (RL)-guided (Cao et al., 2025; Zhu et al., 2025) query rewritings, and multi-reformulation approaches (Kostric and Balog, 2024; Dhole et al., 2024). However, these works all focus on text-based retrievals using external retrievers, where no learned embedding is involved.

Recently, Bai et al. (2025b) applied the idea of query rewriting to uni-modal encoder-based models (e.g. CLIP) for text-to-video retrieval, by enriching textual queries with additional context to bridge information asymmetry. In contrast, our work focuses on leveraging reasoning to improve the quality of general instruction-following multimodal embeddings, which can apply to both query and target.

## 3 Preliminary: MLLM-based Universal Multimodal Embedding

We briefly introduce MLLM-based Universal Multimodal Embedding (UME). In UME, every query $q$ and target $t$ is a triplet of an optional visual input, a textual input, and a pre-defined task instruction, written as $\langle\mathcal{V},\mathcal{T},[\texttt{Ins}]\rangle$.
Both query and target are passed into the same MLLM separately to obtain the corresponding embedding, defined by the equation below:

$$ $\mathbf{h}_{q}=\texttt{Pooling}(f_{\theta}(\mathcal{V}_{q},[\texttt{Ins}_{q}],\mathcal{T}_{q})),\ \ \ \mathbf{h}_{t}=\texttt{Pooling}(f_{\theta}(\mathcal{V}_{t},[\texttt{Ins}_{t}],\mathcal{T}_{t})),$ $$

where $f_{\theta}$ denotes the MLLM embedder and Pooling refers to the pooling operation for aggregating MLLM’s hidden states into the final embedding. Following previous works (Jiang et al., 2024; Gu et al., 2025), we use the last token’s hidden state for final embedding. During training, we adopt the standard uni-directional $q\rightarrow t$ InfoNCE loss as follow:

$$ $\mathcal{L}_{\text{InfoNCE}}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\phi(\mathbf{h}^{i}_{q},\mathbf{h}^{i}_{t})}{\sum_{j=1}^{N}\phi(\mathbf{h}_{q}^{i},\mathbf{h}_{t}^{j})},\ \ \ \ \phi(\mathbf{h}_{q},\mathbf{h}_{t})=\text{exp}(\frac{1}{\tau}\cos(\mathbf{h}_{q},\mathbf{h}_{t})),$ $$

where $\cos$ denotes the cosine similarity function, and $\tau$ is the temperature hyper-parameter.

Figure: (a) Existing MLLM-based embedding approach. The MLLM embedder directly encode visual and textual queries, producing the embedding from the last token.
Refer to caption: x2.png

## 4 Think-then-Embed

We propose our Think-then-Embed (TTE) framework for enhancing Universal Multimodal Embeddings (UME) by incorporating an explicit reasoning step prior to the encoding step. TTE consists of a reasoner, that generates an Embedding-Centric Reasoning (ECR) trace, and an embedder, that produces task-specific representations conditioned on both the original input and the reasoning trace generated by the reasoner. Figure [2](https://arxiv.org/html/2510.05014v3#S3.F2) compares the standard approach of directly encoding multimodal inputs for embedding ([2(a)](https://arxiv.org/html/2510.05014v3#S3.F2.sf1)) with our proposed approach TTE ([2(b)](https://arxiv.org/html/2510.05014v3#S3.F2.sf2), [2(c)](https://arxiv.org/html/2510.05014v3#S3.F2.sf3)).

Formally, given a multimodal input $\langle\mathcal{V},\mathcal{T},[\texttt{Ins}]\rangle$, we obtain its embedding $\mathbf{h}$ by:

$$ $\mathbf{h}=\texttt{Pooling}(f_{\theta}(\mathcal{V},[\texttt{Ins}],\mathcal{T},\psi)),\quad\psi=g_{\omega}(\mathcal{V},[\texttt{Ins}],\mathcal{T}),$ $$

where $g_{\omega}$ and $\psi$ are the reasoner MLLM and its embedding-centric reasoning trace, respectively.

### 4.1 Embedding-Centric Reasoning

Here, we introduce the concept of Embedding-Centric Reasoning (ECR), a form of intermediate reasoning tailored to improve embedding quality. Inspired by chain-of-thought (CoT) reasoning (Wei et al., 2023b) in complex problem solving, ECR are generative reasoning traces that explicitly support the production of target embeddings. For example, for VQA query, the ECR represents the model’s step-by-step reasoning to understand the query, whereas for grounding query, it captures a detailed description of the referred object along with its surrounding visual context.

Formally, for a batch of $(q,t)$ pair, $\mathbf{h}_{q}$ and $\mathbf{h}_{t}$ are conditioned on $\psi$, the ECR $\psi$ can be described as

$$ $\psi^{\star}_{i}\in\arg\max_{\psi}\;\log\frac{\phi(\mathbf{h}^{i}_{q},\mathbf{h}^{i}_{t})}{\sum_{j}\phi(\mathbf{h}_{q}^{i},\mathbf{h}_{t}^{j})}.$ $$

In this work, we simplify the learning of ECR, by using manually designed task prompt and format to prompt reasoner MLLM to generate ECR. We formulate ECR in the form of $\texttt{<think>}\cdots\texttt{</think> Final Reasoning}$, where the reasoner model first outputs some intermediate CoT, then generates the final reasoning. The content for CoT and reasoning can vary in terms of task types. For instance, for QA-based query, the CoT is the standard reasoning process; for simple embedding tasks such as visual document embedding, the CoT is a detailed description of the visual input, and the reasoning is as the summary. While it is also possible to directly optimize reasoner MLLM for ECR generation use retrieval signals (e.g. RL-based finetuning), we will leave these explorations for future work. By conditioning the embedder on these task-aligned reasoning traces, we enable the model to construct more semantically aligned and task-aware embeddings.

Inference cost. We note that, CoT is known to have higher inference cost. In the following subsections, we explore different approaches to construct the reasoner $g_{\omega}$ that can lower the cost but still improve the performance significantly.

### 4.2 TTE with Teacher Reasoner

For our TTE framework, we employ a powerful MLLM (e.g., Qwen2.5-72B) as the teacher reasoner, while keeping the embedder model lightweight (Qwen2-2B or 7B). We refer to this setup as TTEt. We utilize the ECR reasoning traces generated by the reasoner for both training and evaluation of the embedder.
Although this setup may seem computationally expensive, many real-world retrieval tasks only require a one-time offline inference step to generate ECR (e.g., detailed descriptions of visual documents) when constructing the retrieval index. Importantly, the reasoner is not involved in online retrieval for existing data points, and only needs to be run once for new data points.
We empirically show that TTEt achieves superior performance on MMEB-v2 without requiring additional training data, advanced training techniques or changes to the model architecture.

### 4.3 TTE with Finetuned Reasoner

While our TTE framework supports seamless integration with large MLLMs as the teacher reasoner, we are also interested in exploring
whether comparable performance gains can be achieved with a smaller reasoner $-$ specifically, by using the same MLLM backbone as the embedder. As an initial study (Fig. [4](https://arxiv.org/html/2510.05014v3#S7.F4)), we employ the backbone model itself to generate zero-shot ECRs, which are then used to train the embedder for retrieval. This setup results in noticeable performance gains across classification, retrieval, VQA, and grounding tasks.

Encouraged by these results, we further finetune a dedicated reasoner, initializing it from the same backbone as the embedder (2B or 7B).
We finetune the small reasoner using ECR traces generated by the teacher reasoner (e.g., Qwen2.5-72B).
Given a multimodal input $\langle\mathcal{V},[\texttt{Ins}],\mathcal{T}\rangle$, we maximize the conditional likelihood of the ECR $\psi$ by optimizing the standard negative log-likelihood (NLL) loss in standard LLM autoregressive finetuning:

$$ $\mathcal{L}_{\text{SFT}}(\omega)=-\frac{1}{T}\sum_{t=1}^{T}\log p_{\omega}\!\left(\psi_{t}\,\middle|\,\mathcal{V},[\texttt{Ins}],\mathcal{T},\psi_{t^{\prime}<t}\right).$ $$

Figure: (a)
Refer to caption: x5.png

### 4.4 TTE with Unified Reasoner and Embedder

The TTE framework described so far rely on separately trained reasoners and embedders, where the reasoner’s ECR tokens are encoded together with the query to form embeddings. To reduce the computational overhead introduced by the additional reasoner, we explore unifying reasoning and embedding generation within a shared backbone, enabling embeddings to be produced in a single forward pass. We consider two approaches: (1) joint SFT-contrastive training of a shared reasoner–embedder backbone, and (2) a reasoner augmented with an embedding head. We begin with (1), training reasoning and embedding jointly in a multi-task learning setup. However, this configuration consistently resulted in performance degradation, due to the challenges in training curriculum. The implementation details and results for this approach are provided in Appendix [10](https://arxiv.org/html/2510.05014v3#S10). For the rest of the section, we focus on design (2), as it offers greater flexibility in training control.

Reasoner with Embedding Heads. Our key design principle is that hidden states generated during reasoning can also be reused for downstream representation extraction. However, the two tasks should have specialized parameters so that the embedding training objective will not interfere with reasoning, and vice versa. We adopt a two-stage training strategy for the unified model. In the first stage, we fully fine-tune the backbone as a reasoner on the curated ECR dataset. In the second stage, we freeze the backbone and train only the embedding head on top of it. This design enables the unified model to produce embeddings in a single forward pass, as illustrated in Figure [2](https://arxiv.org/html/2510.05014v3#S3.F2)c.

A Systematic Study of Pluggable Embedding Head.
We present the first systematic study of embedding head designs that can be seamlessly plugged into a frozen backbone MLLM. This stands in contrast to the focus of prior LLM-based embedding models (Lee et al., 2025; Lin et al., 2025; Jiang et al., 2024), which primarily rely on either full finetuning or parameter-efficient approaches based on LoRA (Hu et al., 2021). As illustrated in Fig. [3](https://arxiv.org/html/2510.05014v3#S4.F3), we evaluate four embedding head designs:
(1) Learnable attention queries over backbone hidden states (Fig. [3(a)](https://arxiv.org/html/2510.05014v3#S4.F3.sf1)); (2) Learnable latent context with backbone hidden states as queries;
(3) A QFormer-style (Li et al., 2023a) embedding head patched onto the last $n$ layers of the backbone (Fig. [3(c)](https://arxiv.org/html/2510.05014v3#S4.F3.sf3)). (4) Repeat the last $n$ layers of the backbone to formulate N new layers as the embedding head (Fig. [3(d)](https://arxiv.org/html/2510.05014v3#S4.F3.sf4)).

We present a detailed experimental comparison of the four designs in Section [7.3](https://arxiv.org/html/2510.05014v3#S7.SS3).

## 5 Experiments

### 5.1 Experimental Setup

Datasets. We conduct our experiments on both MMEB-V2 (Meng et al., 2025) and MMEB-V1 (Jiang et al., 2024) datasets. MMEB-V1 comprises 20 in-distribution (IND) tasks and 16 out-of-distribution (OOD) tasks, spanning 4 meta-tasks: classification, VQA, retrieval, and grounding. MMEB-V2 extends MMEB-V1 by introducing additional video and visual document (visdoc) retrieval tasks.
Video tasks including video QA, classification, retrieval, and video moment retrieval. Visdoc involves supportive document retrieval, where the task is to retrieve supporting documents given the query.
Overall, MMEB-V2 includes a total of 78 test tasks. We report NDCG@5 for visdoc retrieval and Precision@1 for image and video tasks (Meng et al., 2025).

Models. We employ Qwen2-VL 2B and 7B as the backbone, and introduce three variants of TTE: TTE with a large teacher reasoner (TTEt), TTE with a small, SFT-ed student reasoner (TTEs), and TTE with a unified reasoner and embedder (TTEu). We compare our approach against several baselines, including dual-encoder methods such as CLIP (Radford et al., 2021), SigLIP (Zhai et al., 2023), UniIR (Wei et al., 2023a), and MagicLens (Zhang et al., 2024).
We also directly compare with VLM2Vec-V1 (Jiang et al., 2024) and VLM2Vec-V2 (Meng et al., 2025) with Qwen2-VL as the backbone MLLM. Additionally, we evaluate against recent MLLM-based embedding approaches, including UniME (Gu et al., 2025), LLaVE (Lan et al., 2025), and B3 (Thirukovalluru et al., 2025).

Training Details. For both MMEB-V1 and MMEB-V2, we use a global batch size of $8192$
, with a learning rate of $2e^{-4}$ for the backbone MLLM and $5e^{-4}$ for the embedding head. The temperature for contrastive loss is set to $0.02$. We train the backbone using LoRA (Hu et al., 2021) with rank $16$ and alpha $64$. Following VLM2Vec, we employ GradCache (Gao et al., 2021) to increase the per-device batch size. We train for $1$ epoch on MMEB-V1, and $2.3$ epochs for MMEB-V2. For MMEB-V2, we follow the same data weighing setup as in VLM2Vec-V2, and adopt its interleaved sampling strategy, where one global batch is splitted into $n$ sub batches, each from one dataset.

For supervised finetuning of the ECR Reasoner, we perform full parameter finetuning of the MLLM while keeping the visual encoder frozen, using a learning rate of $2e^{-5}$, a global batch size of $128$, and training for an epoch. We use DeepSpeed (Aminabadi et al., 2022) for optimizer offloading.

## 6 Results

### 6.1 Main results

**Table 1: Results on the MMEB-V1 benchmark (Jiang et al., 2024). The scores are averaged per meta-task. Best/2nd-best performance across both 2B and 7B categories are in bold/underlined.**
| Model | Backbone | Per Meta-Task Score |  | Average Score |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Classification | VQA | Retrieval | Grounding |  | IND | OOD | Overall |  |  |
| # of datasets $\rightarrow$ |  | 10 | 10 | 12 | 4 |  | 20 | 16 | 36 |
| Encoder-Based Baselines |  |  |  |  |  |  |  |  |  |
| CLIP (Radford et al., 2021) | - | 42.8 | 9.1 | 53.0 | 51.8 |  | 37.1 | 38.7 | 37.8 |
| SigLIP (Zhai et al., 2023) | - | 40.3 | 8.4 | 31.6 | 59.5 |  | 32.3 | 38.0 | 34.8 |
| UniIR ($\text{CLIP}_{\text{SF}}$) (Wei et al., 2023a) | - | 44.3 | 16.2 | 61.8 | 65.3 |  | 47.1 | 41.7 | 44.7 |
| MagicLens (Zhang et al., 2024) | - | 38.8 | 8.3 | 35.4 | 26.0 |  | 31.0 | 23.7 | 27.8 |
| $\sim$ 2B Model Size |  |  |  |  |  |  |  |  |  |
| VLM2Vec-V1 (Jiang et al., 2024) | Qwen2VL | 59.0 | 49.4 | 65.4 | 73.4 |  | 66.0 | 52.6 | 59.3 |
| UniME (Gu et al., 2025) | LLaVA-1.6 | 54.8 | 55.9 | 64.5 | 81.8 |  | 68.2 | 52.7 | 64.2 |
| LLaVE (Lan et al., 2025) | Aquila-VL | 62.1 | 60.2 | 65.2 | 84.9 |  | 69.4 | 59.8 | 65.2 |
| B3 (Thirukovalluru et al., 2025) | Qwen2VL | 67.0 | 61.2 | 70.9 | 79.9 |  | 72.1 | 63.1 | 68.1 |
| (Ours) $\text{TTE}_{s}$ | Qwen2VL | 67.6 | 62.7 | 70.7 | 80.0 |  | 71.4 | 65.2 | 68.7 |
| (Ours) $\text{TTE}_{u}$ | Qwen2VL | 69.7 | 60.8 | 71.4 | 78.4 |  | 71.6 | 65.1 | 68.8 |
| (Ours) $\text{TTE}_{t}$ | Qwen2VL | 72.6 | 74.3 | 72.6 | 85.2 |  | 80.5 | 67.0 | 74.5 |
| $\sim$ 7B Model Size |  |  |  |  |  |  |  |  |  |
| VLM2Vec-V1 (Jiang et al., 2024) | Qwen2VL | 62.6 | 57.8 | 69.9 | 81.7 |  | 65.2 | 56.3 | 65.8 |
| UniME (Gu et al., 2025) | LLaVA-OV | 66.8 | 66.6 | 70.5 | 90.9 |  | 74.6 | 65.8 | 70.7 |
| LLaVE (Lan et al., 2025) | LLaVA-OV | 65.7 | 65.4 | 70.9 | 91.9 |  | 75.0 | 64.4 | 70.3 |
| B3 (Thirukovalluru et al., 2025) | Qwen2VL | 70.0 | 66.5 | 74.1 | 84.6 |  | 75.9 | 67.1 | 72.0 |
| QQMM (Xue et al., 2025) | LLaVA-OV | 69.9 | 70.0 | 72.1 | 86.0 |  | 77.2 | 66.6 | 72.5 |
| (Ours) $\text{TTE}_{s}$ | Qwen2VL | 70.5 | 71.5 | 73.5 | 83.9 |  | 76.8 | 68.9 | 73.3 |
| (Ours) $\text{TTE}_{u}$ | Qwen2VL | 71.6 | 72.0 | 72.5 | 82.1 |  | 76.7 | 68.7 | 73.2 |
| (Ours) $\text{TTE}_{t}$ | Qwen2VL | 77.4 | 78.6 | 75.9 | 89.0 |  | 82.7 | 73.3 | 78.5 |

Results on MMEB-V1. Table [1](https://arxiv.org/html/2510.05014v3#S6.T1) compares the performance of our models with recent approaches on MMEB-V1. The encoder-based baselines are evaluated in a zero-shot setting, while the remaining models are trained on the MMEB-V1 training data. Our proposed approach TTE achieves the best performance in both the 2B and 7B categories, with TTEt outperforming the next best approach by $6\%$. Compared to VLM2Vec-V1, TTEs, TTEu and TTEt achieve substantial improvements of $7.5\%$, $7.4\%$ and $12.7\%$, respectively, on the 7B embedding model.

Results on MMEB-V2. Table [2](https://arxiv.org/html/2510.05014v3#S6.T2) summarizes results on MMEB-V2 dataset across the entire leaderboard, at the time of submission. TTEt-7B achieves highest overall score of $71.5\%$ on the MMEB-V2 leaderboard, surpassing recent SOTA models trained with massive external data (e.g., seed-1.6-embedding). Without relying on the teacher reasoner, TTEs achieves best performance across open-sourced models, for both 2B and 7B variants. Compared to baseline VLM2Vec-V2, TTE shows substantial gain. For instance, TTEs and TTEt improve 2B baseline by $5.1\%$ and $10.6\%$.

Taking a closer look at the performance across modality and tasks, we can observe notable improvement against VLM2Vec-V2 baseline from VQA and classification-based tasks. For instance, TTEt-7B improves video QA performance significantly against VLM2Vec-V2. As for retrieval, we observe larger improvements on video-based retrieval ($+5\%$) as compared to image-based retrieval (e.g. $+2\%$ on TTEt-2B), possibly due to video-based retrieval is more challenging than image-based, and therefore signifying the role of teacher-generated ECR.

**Table 2: Results on the MMEB-V2 benchmark (Meng et al., 2025). Best/2nd-best performance across all models are in bold/underlined. Task abbreviations: CLS (classification), QA (question answering), RET (retrieval), GD (grounding), MRET (moment retrieval), VDR (ViDoRe), VR (VisRAG), and OOD (out-of-domain).**
| Model | Image | Video | VisDoc | All |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLS | QA | RET | GD | Overall | CLS | QA | RET | MRET | Overall | VDRv1 | VDRv2 | VR | OOD | Overall |  |  |
| # of Datasets $\rightarrow$ | 10 | 10 | 12 | 4 | 36 | 5 | 5 | 5 | 3 | 18 | 10 | 4 | 6 | 4 | 24 | 78 |
| Close-sourced Models or w/ Additional Data |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| seed-1.6-embedding | 76.1 | 74.0 | 77.9 | 91.3 | 77.8 | 55.0 | 60.9 | 51.3 | 53.5 | 55.3 | 89.5 | 60.8 | 87.9 | 44.4 | 76.8 | 71.3 |
| RzenEmbed-v1-2B | 65.3 | 61.7 | 73.8 | 77.9 | 68.5 | 45.6 | 47.5 | 38.3 | 36.7 | 42.6 | 87.0 | 57.6 | 85.4 | 43.3 | 74.4 | 64.4 |
| Ops-MM-embedding-v1-2B | 68.1 | 65.1 | 69.2 | 80.9 | 69.0 | 53.6 | 55.7 | 41.8 | 33.7 | 47.6 | 87.0 | 57.6 | 85.4 | 43.3 | 74.4 | 63.4 |
| GME-2B | 54.4 | 29.9 | 66.9 | 55.5 | 51.9 | 34.9 | 42.0 | 25.6 | 32.4 | 33.9 | 86.1 | 54.0 | 82.5 | 43.1 | 72.7 | 54.1 |
| ColPali v1.3-3B | 40.3 | 11.5 | 48.1 | 40.3 | 34.9 | 26.7 | 37.8 | 21.6 | 25.5 | 28.2 | 83.6 | 52.0 | 81.1 | 43.1 | 71.0 | 44.4 |
| Ops-MM-embedding-v1-7B | 69.7 | 69.6 | 73.1 | 87.2 | 72.7 | 59.7 | 62.2 | 45.7 | 43.2 | 53.8 | 80.1 | 59.6 | 79.3 | 43.3 | 70.3 | 67.6 |
| RzenEmbed-v1-7B | 69.8 | 68.7 | 76.8 | 85.7 | 73.6 | 52.8 | 56.2 | 41.9 | 41.8 | 48.9 | 89.5 | 60.8 | 87.9 | 44.4 | 76.8 | 68.9 |
| GME-7B | 57.7 | 34.7 | 71.2 | 59.3 | 56.0 | 37.4 | 50.4 | 28.4 | 38.2 | 38.6 | 89.4 | 55.6 | 85.0 | 44.4 | 75.2 | 57.8 |
| LamRA-Qwen2-7B | 59.2 | 26.5 | 70.0 | 62.7 | 54.1 | 39.3 | 42.6 | 24.3 | 34.6 | 35.2 | 22.0 | 11.5 | 37.4 | 21.0 | 23.9 | 40.4 |
| LamRA-Qwen2.5-7B | 51.7 | 34.1 | 66.9 | 56.7 | 52.4 | 32.9 | 42.6 | 23.2 | 37.6 | 33.7 | 56.3 | 33.3 | 58.2 | 40.1 | 50.2 | 47.4 |
| Models Trained on MMEB V2 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| VLM2Vec-V2-2B | 62.9 | 56.3 | 69.5 | 77.3 | 64.9 | 39.3 | 34.3 | 28.8 | 38.5 | 34.9 | 75.5 | 44.9 | 79.4 | 39.4 | 65.4 | 58.0 |
| (Ours) $\text{TTE}_{s}$-2B | 67.9 | 66.6 | 70.2 | 84.1 | 70.1 | 47.3 | 49.1 | 34.4 | 33.2 | 32.1 | 77.5 | 53.2 | 83.2 | 41.1 | 68.8 | 63.1 |
| (Ours) $\text{TTE}_{t}$-2B | 76.6 | 76.8 | 71.5 | 87.2 | 76.1 | 56.1 | 65.3 | 34.1 | 33.8 | 47.9 | 81.1 | 62.4 | 84.7 | 43.2 | 72.6 | 68.6 |
| VLM2Vec-V2-7B | 65.7 | 61.5 | 70.0 | 85.2 | 68.1 | 45.9 | 33.9 | 27.6 | 39.3 | 36.4 | 78.8 | 52.6 | 82.7 | 42.1 | 69.3 | 61.2 |
| (Ours) $\text{TTE}_{s}$-7B | 69.7 | 72.4 | 74.0 | 90.6 | 74.2 | 49.1 | 60.6 | 36.4 | 37.2 | 46.8 | 84.1 | 62.7 | 91.9 | 47.6 | 76.4 | 68.6 |
| (Ours) $\text{TTE}_{t}$-7B | 76.7 | 78.6 | 74.3 | 89.3 | 77.8 | 57.5 | 68.2 | 38.0 | 39.3 | 52.0 | 83.7 | 63.6 | 91.4 | 50.6 | 76.8 | 71.5 |

## 7 Ablations

**Table 3: Effect of finetuning $\text{TTE}_{s}$ under MMEB-V1. w/ SFT-ed Reasoner denotes SFT-ed/zero-shot generation of ECR.**
| w/ SFT-ed<br>Reasoner | Per Meta-Task Score |  | Average Score |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLS | VQA | Ret | GD |  | IND | OOD | All |  |
| 2B Model |  |  |  |  |  |  |  |  |
| Baseline | 59.0 | 49.4 | 65.4 | 73.4 |  | 66.0 | 52.6 | 59.3 |
| ✗ | 62.0 | 57.3 | 62.7 | 73.9 |  | 65.1 | 58.7 | 62.2 |
| ✓ | 67.6 | 62.7 | 70.7 | 80.0 |  | 71.4 | 65.2 | 68.7 |
| 7B Model |  |  |  |  |  |  |  |  |
| Baseline | 62.6 | 57.8 | 69.9 | 81.7 |  | 65.2 | 56.3 | 65.8 |
| ✗ | 65.5 | 66.9 | 68.4 | 78.2 |  | 71.7 | 64.0 | 68.3 |
| ✓ | 70.5 | 71.5 | 73.5 | 83.9 |  | 76.8 | 68.9 | 73.3 |

### 7.1 Understanding the role of TTE Reasoner and ECR

Figure: Figure 4: Baseline (2B) with/without zero-shot ECR on MMEB-V1.
Refer to caption: x9.png

Effect of zero-shot TTE reasoner. We begin by analyzing the impact of the model’s inherent reasoning capability to the retrieval performance. We first prompt the MLLM reasoner (Qwen2VL 2B) to generate ECR in a zero-shot setting. The generated ECR is then combined with the raw query for embedder training using the same Qwen2VL 2B backbone.
We conduct the ablation on a subset of datasets spanning different tasks within MMEB V1, where constrastive training is performed individually for each dataset. Fig. [4](https://arxiv.org/html/2510.05014v3#S7.F4) shows the corresponding results.
We can observe that incorporating self-generated zero-shot ECR leads to notable performance improvements across all tasks.

Effect of finetuning TTE reasoner. While we find that the zero-shot reasoner can generally improve embedding performance, its effectiveness can still be bottlenecked by the quality of the zero-shot ECR.
In Table [3](https://arxiv.org/html/2510.05014v3#S7.T3), we present ablation results on finetuning the TTE reasoner on MMEB-V1, where we observe a significant improvement after finetuning, with an absolute gain of over $6\%$.

Effect of intermediate CoT in ECR. To examine the role of reasoning within ECR, we conduct an ablation by including or excluding intermediate CoT during training and testing, with results shown in Table [4](https://arxiv.org/html/2510.05014v3#S7.T4). Overall, incorporating CoT leads to consistent performance gains. We observe clear improvements on classification, retrieval, and grounding, with only marginal gains on VQA—an expected outcome since VQA primarily requires the final answer, whereas retrieval and grounding benefit from richer intermediate context. Notably, the improvements are larger for 7B than 2B models, likely due to the stronger language understanding capacity of larger models.

Analysis on the role of ECR.
We conducted more analysis to understand the role of ECR in Fig. [5](https://arxiv.org/html/2510.05014v3#S7.F5).

A natural question is whether the ECR text alone is sufficient for retrieval, potentially eliminating the need to train a dedicated encoder. However, our experiments on MMEB show that this is not the case. Specifically, when we use the same ECR and apply a text-to-text (T2T) retrieval baseline—by directly encoding the ECR with a strong off-the-shelf text encoder, Jina-V3 (Sturua et al., 2024) – the performance is significantly worse.

Another concern is that imperfect ECRs might introduce noise that propagates into the Embedder, degrading retrieval performance. Interestingly, we find that our TTE Embedder is robust to such noise: it learns to extract useful information from the ECR without blindly relying on it. In fact, although ECRs alone do not yield high retrieval precision, they provide valuable signals that our Embedder can leverage. We observe similar trends across all task types. These findings indicate that our Embedder is robust to noisy ECRs, and retrieval performance can benefit even from imperfect reasoning traces.

### 7.2 Visualization

In Fig. [6](https://arxiv.org/html/2510.05014v3#S7.F6) we show a side-by-side $t$-SNE visualization comparing embedding produced by the baseline VLM2Vec and TTEt on MMEB-V1. For each task type, we select $3$ distinct subtasks. We clearly observe larger overlap between query and target embedding for TTEt, as compared to VLM2Vec.

Figure: Figure 5: Results on T2T evaluation on generated ECR, versus $\text{TTE}_{s}$ and VLM2Vec-V1 on MMEB V1.
Refer to caption: x10.png

Figure: (a) VQA
Refer to caption: x11.png

### 7.3 Embedding Head Design

**Table 5: Embedding head ablations on MMEB-V1.**
| Model | Per Meta-Task Score |  | Average Score |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLS | VQA | Ret | GD |  | IND | OOD | Overall |  |
| $\text{TTE}_{s}$ | 67.6 | 62.7 | 70.7 | 80.0 |  | 71.4 | 65.2 | 68.7 |
| w/ Simple Attention Pooler |  |  |  |  |  |  |  |  |
| $\text{n_queries}=1$ | 48.0 | 45.4 | 45.8 | 53.1 |  | 50.5 | 42.9 | 47.1 |
| $\text{n_queries}=8$ | 48.3 | 46.1 | 45.9 | 60.1 |  | 51.3 | 44.3 | 48.2 |
| $\text{n_queries}=16$ | 49.7 | 47.6 | 47.8 | 56.7 |  | 53.1 | 44.5 | 49.3 |
| w/ NV-Embed-style Pooler |  |  |  |  |  |  |  |  |
| $\text{n_latent_value}=512$ | 39.3 | 46.3 | 42.7 | 57.6 |  | 48.2 | 39.7 | 44.4 |
| w/ QFormer-style Embedding Head |  |  |  |  |  |  |  |  |
| $\text{n_layers/last_n}=4/4$ | 65.2 | 57.8 | 68.0 | 79.8 |  | 68.8 | 61.8 | 65.7 |
| $\text{n_layers/last_n}=8/8$ | 65.8 | 60.7 | 69.8 | 78.7 |  | 70.0 | 63.6 | 67.1 |
| w/ Self-Initialized MHSA |  |  |  |  |  |  |  |  |
| $\text{n_layers/last_n}=4/4$ | 67.0 | 59.3 | 70.2 | 78.5 |  | 71.2 | 62.2 | 67.2 |
| $\text{n_layers/last_n}=8/8$ | 69.7 | 60.8 | 71.4 | 78.4 |  | 71.6 | 65.1 | 68.8 |
| $\text{n_layers/last_n}=8/16$ | 70.8 | 58.1 | 68.8 | 83.4 |  | 69.0 | 66.8 | 68.0 |

In Table [5](https://arxiv.org/html/2510.05014v3#S7.T5), we investigate the design of embedding heads. We first explore simple trainable pooler, including an attention pooler with learnable queries and an NV-Embed–style pooler. However, both approaches perform significantly worse than TTEs, and increasing the number of queries ($\text{n_queries}=1\rightarrow 16$) yields little improvement. We hypothesize that the hidden states from the final layer may not be optimal for capturing latent representations, as these layers are primarily tuned for providing discriminative features for token predictions.

To better exploit representations from a frozen MLLM, we shift focus to earlier layers (e.g., the 8th-to-last) and equip them with more expressive multi-layer embedding heads. Specifically, we compare a QFormer-style head with a self-initialized multi-head self-attention (MHSA) module. Empirically, the latter achieves superior performance compared to the QFormer-style head, which requires training from scratch. When self-initialized with the last $\text{n_layers}=8$, the MHSA head reaches performance on par with a separately fine-tuned embedder (TTEs).

## 8 Conclusion

We propose Think-Then-Embed (TTE), a general framework for universal multimodal embedding that leverages a reasoner to “think” before predicting embeddings with an embedder. We first show that using a multimodal LLM as the reasoner can substantially boost the performance of a smaller embedder, demonstrating that CoT-style reasoning also benefits multimodal representation learning. To improve efficiency, we then distill a compact reasoner from the reasoning traces of the large model. The distilled reasoner and the embedder can be trained from the same backbone, yielding both capacity and efficiency gains. Finally, we improve the integration of reasoning and embedding by introducing a pluggable embedding head on top of the reasoner. This design enables embeddings to be produced in a single forward pass, further improving efficiency while halving model parameters. Extensive experiments on the MMEB-V1 and MMEB-V2 benchmarks show that our approach significantly outperforms a range of baselines and recent methods without requiring additional data, validating the effectiveness and robustness of the proposed TTE framework.

## References

- Aminabadi et al. (2022)
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He.
Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022.
[https://arxiv.org/abs/2207.00032](https://arxiv.org/abs/2207.00032).
- Bai et al. (2025a)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report, 2025a.
[https://arxiv.org/abs/2502.13923](https://arxiv.org/abs/2502.13923).
- Bai et al. (2025b)
Zechen Bai, Tianjun Xiao, Tong He, Pichao Wang, Zheng Zhang, Thomas Brox, and Mike Zheng Shou.
Bridging information asymmetry in text-video retrieval: A data-centric approach, 2025b.
[https://arxiv.org/abs/2408.07249](https://arxiv.org/abs/2408.07249).
- Cao et al. (2025)
Zhiyu Cao, Peifeng Li, and Qiaoming Zhu.
Icr: Iterative clarification and rewriting for conversational search, 2025.
[https://arxiv.org/abs/2509.05100](https://arxiv.org/abs/2509.05100).
- Chen et al. (2025a)
Haonan Chen, Hong Liu, Yuping Luo, Liang Wang, Nan Yang, Furu Wei, and Zhicheng Dou.
Moca: Modality-aware continual pre-training makes better bidirectional multimodal embeddings, 2025a.
[https://arxiv.org/abs/2506.23115](https://arxiv.org/abs/2506.23115).
- Chen et al. (2025b)
Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou.
mme5: Improving multimodal multilingual embeddings via high-quality synthetic data, 2025b.
[https://arxiv.org/abs/2502.08468](https://arxiv.org/abs/2502.08468).
- Dhole et al. (2024)
Kaustubh D. Dhole, Ramraj Chandradevan, and Eugene Agichtein.
Generative query reformulation using ensemble prompting, document fusion, and relevance feedback, 2024.
[https://arxiv.org/abs/2405.17658](https://arxiv.org/abs/2405.17658).
- Faysse et al. (2024)
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDELOT, and Pierre Colombo.
Colpali: Efficient document retrieval with vision language models.
In *The Thirteenth International Conference on Learning Representations*, 2024.
- Gao et al. (2021)
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan.
Scaling deep contrastive learning batch size under memory limited setup, 2021.
[https://arxiv.org/abs/2101.06983](https://arxiv.org/abs/2101.06983).
- Gu et al. (2025)
Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng.
Breaking the modality barrier: Universal embedding learning with multimodal llms, 2025.
[https://arxiv.org/abs/2504.17432](https://arxiv.org/abs/2504.17432).
- Hu et al. (2021)
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models, 2021.
[https://arxiv.org/abs/2106.09685](https://arxiv.org/abs/2106.09685).
- Jiang et al. (2024)
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen.
Vlm2vec: Training vision-language models for massive multimodal embedding tasks.
*arXiv preprint arXiv:2410.05160*, 2024.
- Kostric and Balog (2024)
Ivica Kostric and Krisztian Balog.
A surprisingly simple yet effective multi-query rewriting method for conversational passage retrieval, 2024.
[https://arxiv.org/abs/2406.18960](https://arxiv.org/abs/2406.18960).
- Lan et al. (2025)
Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su.
Llave: Large language and vision embedding models with hardness-weighted contrastive learning, 2025.
[https://arxiv.org/abs/2503.04812](https://arxiv.org/abs/2503.04812).
- Lee et al. (2025)
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping.
Nv-embed: Improved techniques for training llms as generalist embedding models, 2025.
[https://arxiv.org/abs/2405.17428](https://arxiv.org/abs/2405.17428).
- Li et al. (2023a)
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023a.
[https://arxiv.org/abs/2301.12597](https://arxiv.org/abs/2301.12597).
- Li et al. (2023b)
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He.
Scaling language-image pre-training via masking, 2023b.
[https://arxiv.org/abs/2212.00794](https://arxiv.org/abs/2212.00794).
- Lin et al. (2025)
Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping.
Mm-embed: Universal multimodal retrieval with multimodal llms, 2025.
[https://arxiv.org/abs/2411.02571](https://arxiv.org/abs/2411.02571).
- Ma et al. (2023)
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan.
Query rewriting for retrieval-augmented large language models, 2023.
[https://arxiv.org/abs/2305.14283](https://arxiv.org/abs/2305.14283).
- Meng et al. (2025)
Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, and Semih Yavuz.
Vlm2vec-v2: Advancing multimodal embedding for videos, images, and visual documents, 2025.
[https://arxiv.org/abs/2507.04590](https://arxiv.org/abs/2507.04590).
- Peng et al. (2024)
Letian Peng, Yuwei Zhang, Zilong Wang, Jayanth Srinivasa, Gaowen Liu, Zihan Wang, and Jingbo Shang.
Answer is all you need: Instruction-following text embedding via answering the question.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 459–477, 2024.
- Plummer et al. (2016)
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.
Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models, 2016.
[https://arxiv.org/abs/1505.04870](https://arxiv.org/abs/1505.04870).
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision, 2021.
[https://arxiv.org/abs/2103.00020](https://arxiv.org/abs/2103.00020).
- Sturua et al. (2024)
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, and Han Xiao.
jina-embeddings-v3: Multilingual embeddings with task lora, 2024.
[https://arxiv.org/abs/2409.10173](https://arxiv.org/abs/2409.10173).
- Thirukovalluru et al. (2025)
Raghuveer Thirukovalluru, Rui Meng, Ye Liu, Karthikeyan K, Mingyi Su, Ping Nie, Semih Yavuz, Yingbo Zhou, Wenhu Chen, and Bhuwan Dhingra.
Breaking the batch barrier (b3) of contrastive learning via smart batch mining, 2025.
[https://arxiv.org/abs/2505.11293](https://arxiv.org/abs/2505.11293).
- Wei et al. (2023a)
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen.
Uniir: Training and benchmarking universal multimodal information retrievers, 2023a.
[https://arxiv.org/abs/2311.17136](https://arxiv.org/abs/2311.17136).
- Wei et al. (2023b)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.
Chain-of-thought prompting elicits reasoning in large language models, 2023b.
[https://arxiv.org/abs/2201.11903](https://arxiv.org/abs/2201.11903).
- Wilson et al. (2025)
Raymond Wilson, Chase Carter, and Cole Graham.
Contextualizing search queries in-context learning for conversational rewriting with llms, 2025.
[https://arxiv.org/abs/2502.15009](https://arxiv.org/abs/2502.15009).
- Xu et al. (2024)
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer.
Demystifying clip data, 2024.
[https://arxiv.org/abs/2309.16671](https://arxiv.org/abs/2309.16671).
- Xue et al. (2025)
Youze Xue, Dian Li, and Gang Liu.
Improve multi-modal embedding learning via explicit hard negative gradient amplifying, 2025.
[https://arxiv.org/abs/2506.02020](https://arxiv.org/abs/2506.02020).
- Ye et al. (2023)
Fanghua Ye, Meng Fang, Shenghui Li, and Emine Yilmaz.
Enhancing conversational search: Large language model-aided informative query rewriting, 2023.
[https://arxiv.org/abs/2310.09716](https://arxiv.org/abs/2310.09716).
- Yu et al. (2025)
Hao Yu, Zhuokai Zhao, Shen Yan, Lukasz Korycki, Jianyu Wang, Baosheng He, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, and Hanchao Yu.
Cafe: Unifying representation and generation with contrastive-autoregressive finetuning, 2025.
[https://arxiv.org/abs/2503.19900](https://arxiv.org/abs/2503.19900).
- Yu et al. (2022)
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.
Coca: Contrastive captioners are image-text foundation models, 2022.
[https://arxiv.org/abs/2205.01917](https://arxiv.org/abs/2205.01917).
- Zhai et al. (2023)
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.
Sigmoid loss for language image pre-training, 2023.
[https://arxiv.org/abs/2303.15343](https://arxiv.org/abs/2303.15343).
- Zhang et al. (2024)
Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming-Wei Chang.
Magiclens: Self-supervised image retrieval with open-ended instructions, 2024.
- Zhou et al. (2024)
Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong.
Megapairs: Massive data synthesis for universal multimodal retrieval, 2024.
[https://arxiv.org/abs/2412.14475](https://arxiv.org/abs/2412.14475).
- Zhu et al. (2025)
Changtai Zhu, Siyin Wang, Ruijun Feng, Kai Song, and Xipeng Qiu.
Convsearch-r1: Enhancing query reformulation for conversational search with reasoning via reinforcement learning, 2025.
[https://arxiv.org/abs/2505.15776](https://arxiv.org/abs/2505.15776).

## 9 Additional Results and Analysis

### 9.1 Results on I2T/T2I retrieval.

We evaluate zero-shot I2T and T2I retrieval on Flickr30k (Plummer et al., 2016) dataset and show the results for MSCOCO retrieval. The results are provided in Tab. [6](https://arxiv.org/html/2510.05014v3#S9.T6). We can see both TTEs and TTEt surpass baseline VLM2Vec and the recently proposed B3 (Thirukovalluru et al., 2025) by a large margin. For instance, on MSCOCO T2I, the 7B TTEt achieves 82.0 Precision@1, surpasses B3 (62.8) by almost 20% absolute.

**Table 6: Results on I2T/T2I retrieval on MSCOCO and zero-shot Flickr30K dataset. We report R@1 and R@5 for Flickr30k, and R@1 for MSCOCO.**
| Model | Flickr30k-I2T |  | Flickr30k-T2I |  | MSCOCO |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| R@1 | R@5 |  | R@1 | R@5 |  | I2T | T2I |  |
| $\sim$ 2B Models |  |  |  |  |  |  |  |  |
| CLIP(ViT-BigG/14) | 92.9 | - |  | 79.6 | - |  | 67.3 | 51.3 |
| UniME | 88.2 | - |  | 77.0 | - |  | 66.8 | 49.8 |
| B3++ | 94.9 | - |  | 82.8 | - |  | 73.6 | 59.0 |
| VLM2Vec | 89.1 | 98.7 |  | 68.8 | 90.4 |  | 68.6 | 71.5 |
| (Ours) $\text{TTE}_{s}$ | 94.8 | 99.5 |  | 83.6 | 95.2 |  | 75.5 | 74.8 |
| (Ours) $\text{TTE}_{t}$ | 95.2 | 99.5 |  | 84.2 | 95.5 |  | 77.7 | 78.0 |
| $\sim$ 7B Models |  |  |  |  |  |  |  |  |
| EVA-CLIP | 94.5 | - |  | 80.3 | - |  | 70.1 | 52.0 |
| UniME (Llava-Next) | 93.4 | - |  | 81.9 | - |  | 70.1 | 53.7 |
| B3 7B | 95.9 | - |  | 85.5 | - |  | 77.6 | 62.8 |
| VLM2Vec | 94.6 | 99.5 |  | 80.3 | 95.0 |  | 73.5 | 78.2 |
| (Ours) $\text{TTE}_{s}$ | 95.8 | 99.6 |  | 86.3 | 97.6 |  | 76.8 | 80.7 |
| (Ours) $\text{TTE}_{t}$ | 96.5 | 99.6 |  | 86.8 | 98.3 |  | 78.5 | 82.0 |

**Table 7: Ablation on the MMEB V1 with or without noisy ECR data for contrastive training.**
| Model | w/ Noisy CL ECR | Per Meta-Task Score |  | Average Score |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLS | VQA | Ret | GD |  | IND | OOD | All |  |  |
| 2B Model Size |  |  |  |  |  |  |  |  |  |
| $\text{TTE}_{s}$ | ✗ | 66.8 | 62.1 | 69.5 | 78.6 |  | 70.6 | 64.1 | 67.7 |
| $\text{TTE}_{s}$ | ✓ | 67.6 | 62.7 | 70.7 | 80.0 |  | 71.4 | 65.2 | 68.7 |
| $\text{TTE}_{t}$ | ✗ | 71.5 | 72.1 | 72.3 | 82.4 |  | 79.4 | 65.3 | 73.1 |
| $\text{TTE}_{t}$ | ✓ | 72.6 | 74.3 | 72.6 | 85.2 |  | 80.5 | 67.0 | 74.5 |
| 7B Model Size |  |  |  |  |  |  |  |  |  |
| $\text{TTE}_{s}$ | ✗ | 69.3 | 70.1 | 72.3 | 82.2 |  | 75.4 | 67.6 | 71.9 |
| $\text{TTE}_{s}$ | ✓ | 70.5 | 71.5 | 73.5 | 83.9 |  | 76.8 | 68.9 | 73.3 |
| $\text{TTE}_{t}$ | ✗ | 76.1 | 76.0 | 74.5 | 85.2 |  | 80.8 | 71.3 | 76.6 |
| $\text{TTE}_{t}$ | ✓ | 77.4 | 78.6 | 75.9 | 89.0 |  | 82.7 | 73.3 | 78.5 |

### 9.2 Constructing ECR dataset

We construct two separate high-quality ECR for contrastive training for TTE Embedder, and Supervised Finetuning (SFT) for TTE Reasoner, using powerful MLLMs such as Qwen2.5-72B (Bai et al., 2025a). For each task in MMEB, we manually design prompts that take in both query and the ground-truth target. For contrastive training, we aim to blend in a certain amount of “noise” in the training dataset to improve robustness of TTE Embedder against incorrect ECR generation. To prevent label leakage and overfitting, we explicitly instruct the MLLM to rewrite the answers while performing reasoning. Additionally, we only keep $50\%$ of the ground-truth in the training data. For test set, we adopt similar prompts but without the ground-truth target. The process for SFT ECR dataset is similar, except we do not ask the MLLM to rewrite the ground-truth, as we want the TTE Reasoner to learn to output the exact ground-truth.

In Tab. [7](https://arxiv.org/html/2510.05014v3#S9.T7) we show ablation study on whether to apply noisy ECR dataset construction method (i.e. with rephrasing and 50% non-ground-truth ECR) for contrastive training. We can observe that by applying noisy ECR data construction for contrastive training, we can obtain 1-2% performance gain across both TTEs and TTEt, for both model size.

## 10 Joint Contrastive-Autoregressive Training

Model and implementation details
Here we explore unifying text generation with embedding, by jointly optimizing for both contrastive and autoregressive objective:

$$ $\mathcal{L_{\text{joint}}}=\lambda\mathcal{L}_{\text{InfoNCE}}+\mathcal{L}_{\text{SFT}}$ $$

where $\lambda$ is a hyper-parameter controlling the weight of contrastive (InfoNCE) loss. During training we use the second last token as the embedding token, since it is the last token generated during inference. We apply a simple MLP block on top of the token to obtain the embedding.

Training details. We use the same set of hyper-parameters as used in baseline contrastive training: LoRA with rank and alpha equals to 16 and 64, learning rate of $2e^{-4}$, a global batch size of 8192, and train for one epoch.

**Table 8: Ablation on the MMEB V1 with joint SFT+CL training, using Qwen2 2B. Baseline denotes VLM2Vec V1 with Qwen2 2B.**
| Model | $\lambda$ | Per Meta-Task Score |  | Average Score |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | CLS | VQA | Ret | GD |  | IND | OOD | All |  |
| Baseline | - | 57.6 | 47.5 | 65.0 | 71.6 |  | 61.5 | 55.1 | 58.9 |
| SFT+CL | 1 | 50.7 | 43.2 | 59.5 | 67.1 |  | 55.6 | 48.3 | 52.3 |
| SFT+CL | 10 | 51.4 | 45.8 | 59.2 | 68.6 |  | 55.3 | 50.2 | 54.7 |
| SFT+CL | 100 | 52.4 | 48.3 | 61.5 | 68.8 |  | 56.6 | 51.9 | 56.2 |

We provide results in Tab. [8](https://arxiv.org/html/2510.05014v3#S10.T8). We can see incorporating SFT objective leads to performance degradation against baseline. As we increase $\lambda$ (reducing impact of SFT), the performance is partially recovered. We conjecture that there may exist conflicting gradients between contrastive and SFT objectives. It could also be due to the expressiveness of the last layer’s hidden states as embedding: the last layer in MLLMs may not be optimal for embedding as they are heavily guided to produce discriminative features for token classification.

## 11 Prompt templates and ECR Samples

We provide sample prompts for prompting the teacher MLLMs (e.g. InternVL3 78B) to generate ECR. Examples are shown in Fig. [8](https://arxiv.org/html/2510.05014v3#S11.F8), Fig. [9](https://arxiv.org/html/2510.05014v3#S11.F9) and Fig. [7](https://arxiv.org/html/2510.05014v3#S11.F7). We also provide sample generated ECR in Tab. [10](https://arxiv.org/html/2510.05014v3#S11.T10), Tab. [9](https://arxiv.org/html/2510.05014v3#S11.T9) and Tab. [11](https://arxiv.org/html/2510.05014v3#S11.T11).

**Table 9: Examples of ECRs for VQA datasets in MMEB V1 benchmark.**
| Dataset | Input Text | Input Image | ECR |
| --- | --- | --- | --- |
| A-OKVQA | Given the image and the below question, answer the question based on the image. Let’s think step by step.<br>Question: What is the man by the bags awaiting? |  | <think>The man is standing by bags with a luggage cart on a street, a common scenario for someone waiting for transportation. The context suggests he is awaiting a cab.</think>Answer: cab |
| ChartQA | Given the image and the below question, answer the question based on the image. Let’s think step by step.<br>Question: Is the value of Favorable 38 in 2015? |  | <think>The Favorable line is the green one. In 2015, the Favorable value is 38.</think>Answer: Yes |
| Visual7W | Given the image and the below question, answer the question based on the image. Let’s think step by step.<br>Question: What is written on the white square on the bus? |  | <think>The image shows a double-decker bus with a large white square on its side that clearly displays the words “FOX’S GINGER BISCUITS” in bold, black letters.</think>Answer: FOX’S GINGER BISCUITS |

**Table 10: Examples of ECRs for Classification datasets in MMEB V1 benchmark.**
| Dataset | Input Text | Input Image | Embedding-Centric Reasoning |
| --- | --- | --- | --- |
| ImageNet-1K | Given an image, identify the main object category it belongs to. Let’s think step by step. |  | <think>The image mainly shows a hand plane with a metallic body and wooden handles, a tool used for smoothing or shaping wood.</think>Answer: hand plane |
| N24News | Given an image and its associated news text, identify the main domain of the news. Let’s think step by step.<br>News text: Danai Gurira and Andrew Holland in a theater at New York University, where they met in the Tisch Graduate Acting Program. |  | <think>The image shows two individuals in a theater setting, and the caption mentions they are actors who met at a graduate acting program, which is related to the performing arts.</think>Answer: Arts & Entertainment |
| VOC2007 | Given an image, identify the main object category it belongs to. Let’s think step by step. |  | <think>The image mainly shows a car with racing stripes and branding, identifying it as an automobile.</think>Answer: car |
| HatefulMemes | Given an image, determine if it contains hateful speech. Let’s think step by step. |  | <think>The image shows a cute squirrel on top of a tree trunk. The text on the image is: "you’re a special kinda nuts aren’t you?" The text uses humor and wordplay referencing nuts and squirrels, which is harmless. There’s no indication of harm or offense intended.</think>Answer: No |

**Table 11: Examples of ECRs for Retrieval datasets in MMEB V1 benchmark.**
| Dataset | Input Text | Input Image | ECR |
| --- | --- | --- | --- |
| VisualNews_i2t | Given a news image, first generate a concise and informative description of the news image, then generate a summarization based on the description. Let’s think step by step. |  | <think>The image shows a man with a mustache wearing a dark suit and a purple tie with a pin or badge on his jacket. He appears to be in a formal setting, possibly outside a building. His expression is somber, suggesting a serious or official event. Based on his appearance and the context, this is likely a depiction of a prominent political figure at a significant moment.</think>Answer: Turkish President Recep Tayyip Erdogan is seen during a state visit or formal event. |
| WebQA | Given an wiki image and a short text description, first generate a detailed description of the image and text, then generate a summarization based on the description. Let’s think step by step.<br>Text: 1913-D Barber half obverse. |  | <think>The image shows a close-up of a silver Barber half obverse minted in 1913. The coin features a profile of Lady Liberty facing right, adorned with a laurel wreath and a ribbon tied behind the neck. Above the portrait, the words “IN GOD WE TRUST” are inscribed, and thirteen stars encircle the top and left side of Lady Liberty. The year “1913” is prominently displayed below the neck. The design is intricate, with fine details visible on the hair and leaves.</think>Answer: The image shows a 1913-D Barber half obverse featuring Lady Liberty with a laurel wreath and the inscription “IN GOD WE TRUST.” |
| MSCOCO_t2i | Given a COCO-style caption, first use your knowledge to expand it into a more detailed and concise description of the target image, then generate a summarization based on the description. Let’s think step by step.<br>Caption: A skateboarder in mid air following a jump form cincrete. |  | <think>The caption describes a skateboarder who has launched off a concrete surface. Given the mid-air position, the skateboarder is likely performing a trick such as an ollie or kickflip. The skateboarder is in the midst of executing the trick, and the concrete background suggests an urban skate park or street setting.</think>Answer: A skateboarder captured in mid-air, performing a daring trick off a concrete ramp in an urban skate park, surrounded by smooth pavement and scattered wheels |
| RefCOCO (Grounding) | Given an image and a query, identify the object or region in the image that the query refers to, and generate a concise description of the object or region. Let’s think step by step.<br>Query: vehicle second-closest to camera |  | <think>The image shows a black and white photograph of a small white dog. The dog is standing or walking on a light-colored surface, surrounded by a darker background. It has a round, fluffy appearance and is wearing a collar. The dog’s features are somewhat blurred, making it difficult to discern fine details beyond its general shape and posture.</think>Answer: A small white dog standing on a light surface. |

Figure: Figure 7: Prompt template for generating ECR data using teacher MLLMs (e.g. InternVL3 78B) for VQA tasks. Left: prompt template without ground-truth. Right: prompt template with ground-truth and rephrasing.
Refer to caption: x15.png

Figure: Figure 8: Prompt template for generating ECR data using teacher MLLMs for i2t tasks.
Refer to caption: x16.png

Figure: Figure 9: Prompt template for generating ECR data using teacher MLLMs for t2v tasks.
Refer to caption: x17.png

## 12 Full results

We present the full results on MMEB V2 (Jiang et al., 2024) in Tab. [13](https://arxiv.org/html/2510.05014v3#S12.T13). For VLM2Vec 2B baseline, we use the officially reported number from VLM2Vec V2 (Meng et al., 2025). For VLM2Vec 7B, since it is not reported, we use our reproduced version under our training hyper-parameters.

**Table 12: Full results on MMEB V2. Visrag results are shown on the next table.**
|  | ColPali v1.3 | GME-2B | GME-7B | LamRA<br>Qwen2.5 | VLM2Vec<br>2B | VLM2Vec<br>7B | TTEs-2B | TTEt-2B | TTEs-7B | TTEt-7B |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Avg - All (78 tasks) | 44.4 | 54.1 | 57.8 | 47.4 | 58.0 | 61.2 | 63.1 | 68.6 | 68.6 | 71.5 |
| Avg - Image (36 tasks, Hit@1) | 34.9 | 51.9 | 56.0 | 52.4 | 64.9 | 68.1 | 70.1 | 76.1 | 74.2 | 77.8 |
| Avg - Video (18 tasks, Hit@1) | 28.2 | 33.6 | 38.4 | 33.6 | 34.6 | 36.4 | 41.3 | 48.8 | 46.8 | 51.9 |
| Avg - Visdoc (24 tasks, NDCG@5) | 71.0 | 72.7 | 75.2 | 50.2 | 65.4 | 69.3 | 68.8 | 72.1 | 76.4 | 76.8 |
| I-CLS (10) | 40.3 | 54.4 | 57.7 | 51.7 | 62.9 | 65.7 | 67.9 | 76.6 | 69.7 | 76.7 |
| I-QA (10) | 11.5 | 29.9 | 34.7 | 34.1 | 56.3 | 61.5 | 66.6 | 76.8 | 72.4 | 78.6 |
| I-RET (12) | 48.1 | 66.9 | 71.2 | 66.9 | 69.5 | 70.0 | 70.2 | 71.5 | 74.0 | 74.3 |
| I-VG (4) | 40.3 | 55.5 | 59.3 | 56.7 | 77.3 | 85.2 | 84.1 | 87.2 | 90.6 | 89.3 |
| V-CLS (5) | 26.7 | 34.9 | 37.4 | 32.9 | 39.3 | 45.9 | 47.3 | 56.1 | 49.1 | 57.5 |
| V-QA (5) | 37.8 | 42.0 | 50.4 | 42.6 | 34.3 | 33.9 | 49.1 | 65.3 | 60.6 | 68.2 |
| V-RET (5) | 21.6 | 25.6 | 28.4 | 23.2 | 28.8 | 27.6 | 33.2 | 34.1 | 36.4 | 37.6 |
| V-MR (3) | 25.5 | 31.1 | 37.0 | 37.2 | 36.8 | 39.3 | 32.1 | 33.8 | 37.2 | 39.3 |
| VD-Vidore-V1 (10) | 83.6 | 86.1 | 89.4 | 56.3 | 75.5 | 78.8 | 77.5 | 81.1 | 84.1 | 83.7 |
| VD-Vidore-V2 (4) | 52.0 | 54.0 | 55.6 | 33.3 | 44.9 | 52.6 | 53.2 | 59.9 | 62.7 | 63.6 |
| VD-VisRAG (6) | 81.1 | 82.5 | 85.0 | 58.2 | 79.4 | 82.7 | 83.2 | 84.7 | 91.9 | 91.4 |
| VD-OOD (4) | 43.1 | 43.1 | 44.4 | 40.1 | 39.4 | 42.1 | 41.1 | 43.2 | 47.6 | 50.6 |
| ImageNet-1K | 42.4 | 58.3 | 64.6 | 58.9 | 80.8 | 82.5 | 83.3 | 83.1 | 84.3 | 84.6 |
| N24News | 25.5 | 50.1 | 50.5 | 29.8 | 72.9 | 80.1 | 78.6 | 83.1 | 83.1 | 81.8 |
| HatefulMemes | 50.6 | 52.5 | 53.6 | 51.3 | 56.3 | 67.9 | 64.0 | 78.2 | 67.4 | 75.8 |
| VOC2007 | 69.8 | 75.9 | 80.3 | 78.7 | 85.0 | 84.2 | 86.3 | 87.6 | 86.6 | 84.8 |
| SUN397 | 56.1 | 67.3 | 69.5 | 66.5 | 71.0 | 73.0 | 77.5 | 78.0 | 78.9 | 79.3 |
| Place365 | 27.5 | 35.8 | 39.1 | 37.4 | 35.9 | 41.7 | 45.7 | 59.8 | 44.6 | 64.1 |
| ImageNet-A | 14.9 | 28.8 | 41.2 | 36.3 | 47.4 | 49.6 | 50.9 | 69.8 | 60.4 | 73.0 |
| ImageNet-R | 64.6 | 78.6 | 83.9 | 77.0 | 89.3 | 88.4 | 89.7 | 90.2 | 90.5 | 90.5 |
| ObjectNet | 45.6 | 70.6 | 69.0 | 59.4 | 65.2 | 66.3 | 74.1 | 74.4 | 72.6 | 72.7 |
| Country211 | 6.0 | 26.5 | 24.8 | 21.7 | 25.2 | 23.7 | 28.5 | 62.0 | 29.0 | 60.3 |
| OK-VQA | 9.4 | 29.9 | 33.2 | 39.9 | 51.5 | 57.3 | 68.4 | 80.1 | 74.7 | 83.2 |
| A-OKVQA | 6.6 | 18.6 | 21.0 | 34.1 | 43.6 | 50.2 | 57.1 | 75.0 | 66.1 | 76.9 |
| DocVQA | 11.3 | 29.8 | 41.4 | 37.1 | 90.1 | 93.5 | 94.2 | 94.4 | 95.6 | 95.6 |
| InfographicsVQA | 5.0 | 11.6 | 20.3 | 23.7 | 58.8 | 69.3 | 65.6 | 81.8 | 77.5 | 81.6 |
| ChartQA | 5.7 | 13.4 | 17.8 | 15.0 | 47.4 | 56.8 | 57.5 | 81.7 | 70.9 | 83.1 |
| Visual7W | 6.1 | 16.2 | 22.2 | 24.6 | 52.9 | 55.3 | 54.1 | 70.7 | 57.9 | 73.3 |
| ScienceQA | 16.3 | 27.3 | 28.0 | 31.3 | 38.2 | 46.4 | 50.7 | 64.7 | 60.0 | 67.8 |
| VizWiz | 27.6 | 37.0 | 39.0 | 32.0 | 43.3 | 44.5 | 55.1 | 55.5 | 53.8 | 55.4 |
| GQA | 8.3 | 75.1 | 76.9 | 57.4 | 64.9 | 64.5 | 77.0 | 77.1 | 80.9 | 81.3 |
| TextVQA | 18.8 | 39.7 | 46.8 | 46.1 | 72.2 | 77.0 | 86.2 | 87.3 | 87.0 | 87.5 |
| VisDial | 41.2 | 48.1 | 60.8 | 62.5 | 82.7 | 82.3 | 81.2 | 81.5 | 84.4 | 84.9 |
| CIRR | 8.2 | 44.2 | 54.9 | 44.7 | 57.5 | 61.1 | 59.4 | 64.2 | 65.1 | 67.1 |
| VisualNews_t2i | 50.1 | 74.7 | 79.7 | 70.1 | 74.5 | 73.2 | 72.8 | 74.9 | 78.5 | 79.4 |
| VisualNews_i2t | 47.6 | 78.3 | 83.6 | 74.2 | 78.2 | 80.4 | 76.5 | 76.6 | 81.3 | 81.7 |
| MSCOCO_t2i | 59.2 | 68.1 | 71.2 | 65.7 | 75.3 | 75.8 | 75.2 | 75.7 | 77.9 | 78.4 |
| MSCOCO_i2t | 49.9 | 63.1 | 57.7 | 71.1 | 71.4 | 72.6 | 71.1 | 72.2 | 73.1 | 73.1 |
| NIGHTS | 65.5 | 67.0 | 67.6 | 64.4 | 68.6 | 66.5 | 70.8 | 68.0 | 69.8 | 70.2 |
| WebQA | 53.8 | 88.8 | 91.4 | 85.7 | 90.6 | 90.1 | 90.4 | 90.4 | 90.8 | 91.1 |
| FashionIQ | 5.9 | 32.9 | 37.8 | 33.4 | 19.5 | 24.9 | 26.3 | 28.9 | 29.7 | 30.2 |
| Wiki-SS-NQ | 80.5 | 73.9 | 78.2 | 67.0 | 66.9 | 72.2 | 64.2 | 64.9 | 70.5 | 71.2 |
| OVEN | 50.0 | 72.3 | 75.1 | 84.8 | 64.3 | 67.0 | 67.6 | 67.9 | 72.7 | 69.4 |
| EDIS | 64.7 | 91.8 | 96.0 | 78.7 | 84.1 | 73.6 | 87.0 | 92.8 | 93.9 | 95.2 |
| MSCOCO | 36.7 | 28.6 | 31.4 | 36.0 | 67.1 | 73.0 | 67.7 | 82.5 | 74.1 | 84.9 |
| RefCOCO | 64.5 | 55.9 | 60.9 | 57.1 | 87.1 | 93.1 | 91.4 | 89.4 | 97.7 | 92.1 |
| RefCOCO-Matching | 3.9 | 73.3 | 78.4 | 82.6 | 85.8 | 93.1 | 95.0 | 90.3 | 96.3 | 91.6 |
| Visual7W-Pointing | 56.1 | 64.1 | 66.5 | 51.2 | 69.2 | 81.7 | 82.5 | 86.5 | 94.3 | 88.4 |
| K700 | 23.4 | 35.2 | 39.7 | 32.1 | 38.0 | 53.6 | 49.6 | 48.2 | 55.0 | 55.6 |
| SmthSmthV2 | 25.1 | 29.9 | 30.6 | 25.3 | 42.8 | 46.6 | 50.4 | 59.4 | 44.9 | 55.3 |
| HMDB51 | 24.8 | 43.4 | 47.9 | 33.8 | 40.9 | 43.2 | 52.5 | 64.2 | 51.7 | 63.9 |
| UCF101 | 49.4 | 52.4 | 54.7 | 53.0 | 60.0 | 66.9 | 58.3 | 75.5 | 64.2 | 78.6 |
| Breakfast | 10.9 | 13.6 | 14.3 | 20.1 | 14.8 | 19.4 | 25.4 | 33.2 | 29.7 | 34.2 |
| MVBench | 33.7 | 37.5 | 46.6 | 37.6 | 33.7 | 32.8 | 48.5 | 62.0 | 59.5 | 65.3 |
| Video-MME | 30.6 | 34.3 | 39.2 | 35.1 | 30.7 | 33.1 | 45.8 | 58.9 | 53.1 | 62.1 |
| NExTQA | 35.2 | 39.5 | 53.6 | 44.9 | 20.9 | 21.0 | 53.8 | 74.0 | 70.1 | 73.6 |
| EgoSchema | 38.4 | 40.8 | 46.8 | 47.0 | 34.0 | 34.4 | 36.4 | 58.2 | 55.6 | 62.8 |
| ActivityNetQA | 51.3 | 58.0 | 65.6 | 48.5 | 52.3 | 48.0 | 60.8 | 73.6 | 64.6 | 77.1 |
| DiDeMo | 22.8 | 22.0 | 26.4 | 22.8 | 30.4 | 31.6 | 33.5 | 34.1 | 34.9 | 36.3 |
| MSR-VTT | 17.6 | 27.3 | 31.8 | 25.0 | 28.3 | 31.6 | 34.8 | 36.6 | 37.6 | 39.5 |
| MSVD | 45.4 | 47.6 | 49.7 | 41.9 | 48.1 | 46.7 | 56.5 | 57.1 | 58.5 | 59.4 |
| VATEX | 16.7 | 23.0 | 24.9 | 18.7 | 26.5 | 19.0 | 25.6 | 26.4 | 31.0 | 32.6 |
| YouCook2 | 5.3 | 7.9 | 9.1 | 7.5 | 10.6 | 9.2 | 15.8 | 16.3 | 19.9 | 20.3 |
| QVHighlight | 19.9 | 43.6 | 59.5 | 60.9 | 49.4 | 58.2 | 38.9 | 40.3 | 51.0 | 52.7 |
| Charades-STA | 29.0 | 14.9 | 14.0 | 18.8 | 20.2 | 19.3 | 19.5 | 21.4 | 18.9 | 23.0 |
| MomentSeeker | 27.6 | 34.8 | 37.4 | 31.8 | 40.8 | 40.6 | 37.7 | 39.6 | 41.5 | 42.2 |

**Table 13: Visrag results on MMEB V2.**
|  | ColPali v1.3 | GME-2B | GME-7B | LamRA<br>Qwen2.5 | VLM2Vec<br>2B | VLM2Vec<br>7B | TTEs-2B | TTEt-2B | TTEs-7B | TTEt-7B |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ViDoRe_arxivqa | 81.7 | 82.8 | 86.9 | 53.0 | 80.6 | 81.5 | 80.7 | 81.8 | 84.6 | 83.4 |
| ViDoRe_docvqa | 56.6 | 53.1 | 57.5 | 25.4 | 44.9 | 45.7 | 44.5 | 40.7 | 46.0 | 45.8 |
| ViDoRe_infovqa | 84.9 | 90.2 | 91.6 | 72.3 | 83.7 | 86.4 | 84.8 | 83.4 | 88.7 | 87.6 |
| ViDoRe_tabfquad | 86.9 | 93.3 | 94.6 | 66.1 | 89.2 | 91.4 | 88.4 | 90.1 | 94.7 | 92.0 |
| ViDoRe_tatdqa | 70.9 | 69.9 | 74.1 | 25.9 | 43.8 | 50.8 | 50.4 | 51.8 | 59.4 | 56.1 |
| ViDoRe_shiftproject | 75.1 | 89.5 | 96.8 | 27.3 | 60.8 | 71.8 | 65.2 | 79.3 | 81.6 | 81.8 |
| ViDoRe_artificial_intelligence | 95.7 | 97.5 | 99.6 | 72.0 | 88.5 | 91.0 | 91.9 | 97.3 | 98.1 | 98.2 |
| ViDoRe_energy | 94.7 | 91.9 | 95.3 | 65.2 | 86.5 | 86.8 | 88.7 | 93.4 | 93.5 | 96.2 |
| ViDoRe_government_reports | 93.6 | 94.6 | 98.8 | 72.2 | 85.0 | 87.8 | 86.9 | 95.3 | 96.7 | 97.7 |
| ViDoRe_healthcare_industry | 95.9 | 98.7 | 99.3 | 83.8 | 92.2 | 95.1 | 92.8 | 97.9 | 97.9 | 98.6 |
| ViDoRe_esg_reports_human_labeled_v2 | 51.3 | 61.0 | 63.4 | 33.0 | 45.6 | 55.7 | 59.0 | 64.9 | 69.4 | 70.9 |
| ViDoRe_biomedical_lectures_v2_multilingual | 54.7 | 54.0 | 49.5 | 35.9 | 44.3 | 54.0 | 52.0 | 55.0 | 60.8 | 62.8 |
| ViDoRe_economics_reports_v2_multilingual | 49.0 | 50.2 | 54.2 | 31.9 | 43.0 | 51.9 | 49.8 | 53.8 | 60.4 | 56.3 |
| ViDoRe_esg_reports_v2_multilingual | 52.9 | 50.7 | 55.4 | 32.5 | 46.6 | 48.8 | 52.1 | 65.8 | 60.3 | 64.3 |
| VisRAG_ArxivQA | 80.9 | 82.0 | 87.4 | 37.7 | 76.9 | 79.5 | 78.5 | 84.0 | 94.5 | 92.4 |
| VisRAG_ChartQA | 78.2 | 79.9 | 81.9 | 65.9 | 84.4 | 82.9 | 84.4 | 85.2 | 91.2 | 95.0 |
| VisRAG_MP-DocVQA | 86.8 | 84.4 | 89.2 | 54.5 | 71.8 | 81.5 | 79.2 | 80.0 | 90.1 | 87.0 |
| VisRAG_SlideVQA | 95.0 | 93.4 | 94.5 | 76.5 | 91.5 | 91.3 | 92.3 | 93.4 | 95.6 | 94.9 |
| VisRAG_InfoVQA | 85.7 | 91.4 | 93.5 | 73.3 | 85.7 | 89.8 | 87.2 | 87.8 | 93.0 | 92.2 |
| VisRAG_PlotQA | 60.3 | 64.1 | 63.4 | 41.2 | 66.1 | 71.5 | 77.5 | 77.8 | 86.9 | 87.1 |
| ViDoSeek-page | 22.2 | 21.6 | 23.2 | 23.1 | 21.9 | 21.7 | 22.6 | 22.9 | 35.0 | 45.2 |
| ViDoSeek-doc | 83.7 | 83.6 | 83.9 | 80.3 | 80.2 | 82.2 | 82.0 | 83.2 | 84.4 | 84.6 |
| MMLongBench-page | 14.2 | 15.8 | 16.2 | 13.5 | 11.9 | 14.8 | 12.9 | 17.8 | 20.7 | 22.0 |
| MMLongBench-doc | 52.5 | 51.4 | 54.3 | 43.5 | 43.7 | 49.7 | 47.0 | 48.8 | 50.4 | 50.6 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * The grounding task score does not show an advantage compared to LLaVA-OV-based models in Table 1, even though Qwen2VL is claimed to have better grounding ability.
* Why does the embedder use Qwen2VL-7B? Is it superior to Qwen2.5-VL-7B?
soundness: 3
strengths: The method is simple yet effective. Involving cot process in retrieval shows significant performance gains on benchmarks. The author designed various experiments and methods to demonstrate this, and the experimental results performed well on the MMEB benchmarks.
summary: This paper introduces Think-Then-Embed (TTE), a universal multimodal embedding framework that uses a reasoner to "think" before generating embeddings with an embedder. Leveraging a multimodal large language model (MLLM) as the reasoner significantly enhances the performance of smaller embedders, showing that chain-of-thought (CoT) reasoning benefits multimodal representation learning. To boost efficiency, we distill a compact reasoner from the large model’s reasoning traces, allowing both the reasoner and embedder to share the same backbone for improved capacity and speed. Additionally, a pluggable embedding head on the reasoner enables single-pass embedding generation, reducing parameters and further increasing efficiency. Extensive experiments on MMEB benchmarks demonstrate that TTE outperforms existing methods without extra data, confirming its effectiveness and robustness.
weaknesses: * Some terms should be unified: ‘Fig.’ in line 62 and ‘Figure 2’ in line 177 should be consistent.

* I recommend that the author distinguish between MLLM and LLM. For example, ‘Qwen2.5-VL-72B’ in line 84 and ‘Qwen2.5-72B’ in line 
212 are confusing. These names should be unified in both the introduction and the methods section.

* Figure 5 looks strange; the results on different benchmarks do not show temporal relationships clearly. A bar chart or radar chart would be more appropriate.

* The grounding task score does not show an advantage compared to LLaVA-OV-based models in Table 1, even though Qwen2VL is claimed to have better grounding ability.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - The “contrastive generative embedding” objective is vaguely described. It is unclear whether gradients flow through the reasoning generator (is it frozen or partially fine-tuned?).
- The distinction between “reasoning trace” and “explanation text” is blurred—are they generated with the same prompt template or via different decoding strategies?
soundness: 3
strengths: - The writing is mostly clear and logically structured; the introduction and motivation are strong. Figures and pipeline diagrams are helpful in explaining the “think → embed” flow.
- The overall framework is technically coherent and easy to reproduce: the authors provide clear training and inference procedures.
- The formulation of generative embeddings based on explicit reasoning traces is intuitive and connects language modeling with representation learning in a principled way.
summary: This paper proposes Think-Then-Embed (TTE), a novel framework for multimodal reasoning that introduces an explicit generative embedding step between reasoning and representation. Instead of directly encoding multimodal inputs (e.g., image-text pairs) into a single joint embedding, the method first conducts latent reasoning generation through an autoregressive module and then projects the reasoning trace into a compact embedding space via a learned encoder. This “reasoning-before-embedding” paradigm aims to enhance reasoning fidelity and interpretability while maintaining embedding efficiency. Experiments are conducted on multiple multimodal benchmarks (e.g., ScienceQA, OK-VQA, and MM-CoT-style datasets). Results show consistent improvements over baseline embedding methods such as CLIP, BLIP-2, and reasoning-augmented contrastive approaches. The authors claim that TTE achieves comparable or better performance with smaller embedding dimensionality and provides interpretable intermediate reasoning traces.
weaknesses: - Based on Tables 1 and 2, $\text{TTE} _\text{T}$ performs significantly better than the other two paradigms. However, it is unclear whether this advantage stems from the use of a stronger teacher model. Did the authors conduct any ablation studies using different teacher models to isolate this effect?
- If reasoning traces are generated by a large pretrained LLM (e.g., GPT-4V or Qwen2-VL), it is unclear whether they contain test-relevant information from pretraining corpora, which may bias the results.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. From the comparison between TTE_t and TTE_s, as well as the ablation results before and after fine-tuning TTE_s, it seems that the quality of ECR plays a crucial role in the embedder’s performance. However, the paper claims that the embedder is stable even with imperfect ECR, which appears somewhat contradictory. Could the authors provide a more detailed explanation or analysis to clarify this point?
2. The MMEB-V2 benchmark results do not include TTE_u. Since the results on MMEB-V1 suggest that TTE_u performs similarly to TTE_s, including such data would make the paper more complete and convincing.
soundness: 3
strengths: 1. The paper provides a clear motivation and detailed explanation of the proposed method, with well-defined formulations for both the task and the training process.
2. Based on the intuition for retrieval tasks that require reasoning, as well as the strong results of TTE_t, the proposed Think-Then-Embed (TTE) framework appears highly effective.
summary: This paper introduces a Universal Multimodal Embedding framework, Think-Then-Embed(TTE), which first generates reasoning before producing embeddings. TTE exploits the generative reasoning ability of MLLMs to enhance embedding quality. Specifically, a powerful MLLM serves as a Teacher Reasoner to produce high-quality CoT traces (embedding-centric reasoning in this paper), guiding a smaller embedder to learn more discriminative representations. To reduce reliance on large models, the authors further distill a compact MLLM reasoner from the teacher and train a unified model that functions simultaneously as a reasoner and an embedder. Experiments show that TTE achieves state-of-the-art results on MMEB-V1 and MMEB-V2 benchmarks. Ablation studies confirm the effectiveness and stability of ECR  for complex instruction embedding.
weaknesses: 1. The analysis of ECR stability is only described qualitatively and lacks experimental support.
2. The MMEB-V2 benchmark seems to be missing some evaluation data. 

Please refer to the Questions section for details.

Things to improve the paper that did not impact the score:
1. In Table 1, there are several mistakes in the use of bold and underline to highlight the first and second-best results.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * It would be interesting to see whether using traces from more powerful reasoning models, like GPT-o3, can further improve the embedder performance.

* Does the performance scale with the size of the MLLM? And does a better MLLM, measured by standard benchmarks like MMMU, results in better embedding?
soundness: 4
strengths: * The paper shows that using CoT traces improves multimodal retrieval without using additional data.

* The paper comprehensively examines different settings to incoporate the CoT traces, including separate reasoning models, distilled reasoning models, and unified reasoning-embedding models. This highlights the flexibility of apply CoT for better multimodal embedding.

* The ablations clearly demonstrate the effectiveness of incoporating CoT to improve multimodal embeddings by comparing with baselines without CoT.
summary: This paper investigates using multimodal CoT traces to improve multimodal embedding for retrieval tasks. The paper explores diverse training settings, including using separate or unified models to combine CoT and embedding, and the design choices of a good embedding head to extract information from CoT latents. Through comprehensive experiments, the paper demonstrates the effectiveness of CoT for generating multmodal embedding compared to both decoder-encoder and multimodal llm based methods.
weaknesses: * The paper's method is not specifically tied to multimodal, or vision-language. The same approach can be applied to text-only retrieval. It would be great to clarify on how multimodal is important here, or demonstrate the method on text-only datasets.

* The paper only used Qwen-VL series as the base models, but the method should be generalizable to most MLLMs. It would be great to show results on other base models.

* While the authors hyphosize that the last hidden states might not be optimal, it is not properly justified in the paper. The improved results from Q-Former or self-initialized could be from the extra trainable parameters. A deeper analysis, like comparing the performance of each layer, would strengthen the design choices of the embedder head.


Decision: Accept (Poster)