MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval
Authors: Yeong-Joon Ju, Ho-Joong Kim, Seong-Whan Lee
Accepted at: acl25

MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval

Yeong-Joon Ju1, Ho-Joong Kim1, Seong-Whan Lee1
1Department of Artificial Intelligence, Korea University
{yj_ju, hojoong_kim, sw.lee}@korea.ac.kr

Abstract
Recent multimodal retrieval methods have endowed text-based retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment. They often directly fuse the two modalities for cross-reference during the alignment to understand multimodal queries. However, existing methods often overlook crucial visual information due to a text-dominant issue, which overly depends on text-driven signals. In this paper, we introduce MIRe, a retrieval framework that achieves modality interaction without fusing textual features during the alignment. Our method allows the textual query to attend to visual embeddings while not feeding text-driven signals back into the visual representations. Additionally, we construct a pre-training dataset for multimodal query retrieval by transforming concise question-answer pairs into extended passages. Our experiments demonstrate that our pre-training strategy significantly enhances the understanding of multimodal queries, resulting in strong performance across four multimodal retrieval benchmarks under zero-shot settings. Moreover, our ablation studies and analyses explicitly verify the effectiveness of our framework in mitigating the text-dominant issue. Our code is publicly available: https://github.com/yeongjoonJu/MIRe

1 Introduction
Information retrieval aims to fetch relevant information from a large collection given a user query, underpinning numerous NLP tasks such as search engines, open-domain question answering (Chen, 2017; Zhu et al., 2021), and fact-checking (Thorne et al., 2018). Beyond conventional methods based on lexical similarities (e.g., TF-IDF and BM25 (Robertson et al., 2009)), embedding-based retrieval methods (Lee et al., 2019; Karpukhin et al., 2020; Izcard et al., 2022; Chen et al., 2024) have achieved rich semantic matching by learning high-dimensional representations of queries and passages via large-scale pre-training. However, they focus on textual queries, struggling to address multimodal queries that encompass both textual and visual information.

![Effect of the text-dominant issue in multimodal query retrieval.](page_1016_328_377_377.png)

Figure 1: Effect of the text-dominant issue in multimodal query retrieval.

In real-world scenarios, users often include visual references in their queries (e.g., complex objects or named entities depicted in an image), which are difficult to represent by text alone fully (Liu et al., 2023). Recent multimodal retrieval methods (Lin et al., 2023; Luo et al., 2023; Lin et al., 2024; Zhou et al., 2024a,b) have endowed text-based retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment. Most existing methods directly fuse the two modalities for cross-reference during visual-text alignment to enhance the understanding of multimodal queries. For instance, Luo et al. (2023) and Zhou et al. (2024a) facilitate modality interaction through early token fusion, where visual representations are prepended before passing through self-attention layers in the query encoder. Similarly, Lin et al. (2024) integrate modalities within the multimodal query using a cross-attention mechanism, where the textual query embeddings function as keys and values.

However, these methods often overlook crucial visual information due to a text-dominant issue induced by excessive reliance on text-driven signals
during the alignment stage. In this stage, the retriever over-relies on textual similarities, thereby hindering proper visual alignment. Consequently, the retriever assigns high scores to irrelevant passages when textual cues are ambiguous. Fig. 1 shows the effect of the text-dominant issue. The baseline, which is trained with a direct fusion of textual features, fails to retrieve the desired passages due to its excessive reliance on text when the textual query becomes partially ambiguous (e.g., replacing a specific term like ‘beverage’ with a more generic word like ‘object’). This text-dominant issue is further amplified through pre-training datasets constructed such that pseudo-queries are extracted from passages (Luo et al., 2023). Datasets obtained via this approach contain text-based queries that alone are sufficient to match relevant passages. This hinders visual-text alignment by relying on the high contextual similarity between textual queries and passages, even in the absence of visual information. This issue highlights the need for a retrieval framework that leverages multimodal queries by mapping both visual and textual cues into a linguistic space, capturing complementary interactions between these modalities without over-relying on textual features alone.

To address these issues, we introduce MIRe, a retrieval framework that achieves modality interaction without fusing textual features during the alignment stage. Instead of directly merging both modalities, MIRe allows the textual query to attend to patch-level visual embeddings without feeding text-driven signals back into the visual representations. We then fuse the two modalities during the relevance scoring stage based on a late-interaction mechanism (Khattab and Zaharia, 2020). This design alleviates the dependency on text-driven signals in the context of knowledge retrieval using a multimodal query. Furthermore, we construct a pre-training dataset by transforming multimodal query-response pairs into extensive passages via our response-to-passage conversion process that utilizes solely a text retrieval model. The constructed dataset requires the integration of both modalities to match a desired passage during training, enabling the model to link image understanding with complex textual queries. Our experiments demonstrate that our pre-training strategy significantly enhances multimodal query understanding for knowledge retrieval, resulting in strong performance across four multimodal retrieval benchmarks under zero-shot settings.

2 Related Work

Traditional methods such as TF-IDF and BM25 (Robertson et al., 2009) rely on keyword matching to retrieve relevant content but often fail to capture the deeper semantics underlying queries and documents. Beyond the surface-level lexical similarities, dense retrieval methods (Lee et al., 2019; Karpukhin et al., 2020; Izacard et al., 2022; Chen et al., 2024; Ni et al., 2022) leverage high-dimensional embedding models for richer semantic matching.

The transition from traditional text queries to multimodal queries has marked a significant evolution in information retrieval (Luo et al., 2021a). Early methods focused on converting images into textual representations, such as captions (Qu et al., 2021; Gao et al., 2022) and object tags (Gui et al., 2022; Yang et al., 2022). EnFoRe (Wu and Mooney, 2022) and DEDR (Salemi et al., 2023) improve image-query representations derived from a multimodal encoder with generated entities and captions, respectively. The OVEN dataset (Hu et al., 2023) has also provided insights into multimodal entity recognition. Whereas most of these approaches utilize DPR (Karpukhin et al., 2020) based on a single embedding for retrieval, FLMR (Lin et al., 2023) refines multimodal queries by incorporating RoIs and generated captions with the late-interaction mechanism. ReViz (Luo et al., 2023) represents an end-to-end multimodal retrieval system that removes the dependency on intermediate modules by pre-training on the VL-ICT, which automatically constructs a pre-training dataset by applying the Inverse Cloze Task (ICT) (Lee et al., 2019) to a multimodal knowledge base. UniIR (Wei et al., 2024) proposes an instruction-guided multimodal retriever along with its benchmark. They design two variants of the model architecture for modality interaction: score-level fusion and feature-level fusion based on CLIP and BLIP (Li et al., 2022). VISTA (Zhou et al., 2024a) introduces an in-depth fusion strategy by prepending visual tokens to the input of a text retrieval model to enhance multimodal understanding. PreFLMR (Lin et al., 2024) extends FLMR to investigate the scalability of multimodal retrievers under the late-interaction mechanism. In contrast to previous methods that rely heavily on text information within multimodal queries, we address the text-dominant issue in multimodal query representations caused by the direct intervention of textual features. We also adopt the
late-interaction mechanism to fuse modalities during the scoring stage.

3 Method

In this section, we first define the problem of knowledge retrieval with multimodal queries. Next, we describe the architecture of our retrieval model and our data construction method.

3.1 Problem Definition

Given a multimodal query \( Q = (I, T) \), the primary objective of our retriever \( \mathcal{R} \) is to retrieve a set of relevant passages \( K = \{ D_1, D_2, \ldots, D_n \} \) from a knowledge base \( U \), where \( I \) and \( T \) denote an image and a textual query, respectively. Each \( D_i \) corresponds to a passage of text. To achieve this goal, \( \mathcal{R} \) should encode the multi-modal query \( Q \), integrating both the image and text modalities.

3.2 Background: Late Interaction in Retrieval

Late interaction (Khattab and Zaharia, 2020) is a retrieval strategy that preserves token-level embeddings for both queries and passages, enabling more fine-grained matching compared to single-vector retrieval. This mechanism defers the aggregation of embeddings to the scoring phase, retaining token-level signals. The retrieval model generates a set of low-dimensional embeddings \( E = \{ e_1, ..., e_l \} \) for tokens in both the query and the passage. Then, the final relevance score between query embeddings \( E_Q \) and document embeddings \( E_D \) is computed via the following MaxSim operation:

\[
r_{Q,D} = \sum_{i=1}^{l_Q} \max_{j=1}^{l_D} \left( E_Q \cdot E_D^T \right),
\]

where \( l_Q \) and \( l_D \) denote the number of tokens in the query and the document, respectively. Each query token is matched with its most relevant document token. In our MIRE framework, we extend this mechanism to handle retrieval with multimodal queries. Our rationale for this adoption is to mitigate the overemphasis on textual features during alignment by maintaining distinct representations for each modality.

3.3 Model Architecture

We detail our model architecture, focusing on how it integrates visual and textual features for multi-modal query retrieval.

Textual Embeddings. We employ a pre-trained text retriever \( \mathcal{R}_T \) to encode the input textual query \( T \) and passage \( D \), utilizing multi-vector representations under the late-interaction mechanism. The text encoder generates token-level embeddings \( E_t \in \mathbb{R}^{l_t \times d_t} \), where \( l_t \) denotes the number of tokens in \( T \) and \( d_t \) represents the embedding dimension.

Visual Embeddings. We use ViT (Dosovitskiy et al., 2021) to encode image \( I \). We adopt two kinds of visual embeddings: (1) global embeddings \( V_g \) derived from the CLS token, representing the overall content of the image, and (2) token-level embeddings \( V_m \) extracted from the penultimate layer of ViT, representing individual patches of the image. The global embedding \( V_g \in \mathbb{R}^{d_v} \) is directly projected into the latent space of the text retriever \( \mathcal{R}_T \) via a two-layer perception, producing embedding with dimension of \( \mathbb{R}^{l_g \times d_t} \), where \( l_g \) is the pre-defined number of tokens. Subsequently, the projected \( V_g \) is reshaped into token-level embeddings \( E_g \in \mathbb{R}^{l_g \times d_t} \).

Query-guided Attentive Pooling. Our architecture aims to achieve modality interaction without directly incorporating textual features in the pre-training stage for multimodal alignment, thereby mitigating the text dominance issue. To this end, we introduce a query-guided attentive pooling module employing an attention layer. This module retrieves visual information required by the textual query \( T \) from \( V_m \in \mathbb{R}^{l_v \times d_t} \) and then aggregates the visual information based on its relevance to tokens within the textual query, where \( l_v \) denotes the number of image patches. Employing \( E_t \) as query vectors, attention scores \( \mathcal{A} \in \mathbb{R}^{h \times l_t \times l_v} \) are calculated as follows:

\[
\mathcal{A} = \text{Softmax} \left( \frac{E_t \cdot \mathcal{K}_m^\top}{\sqrt{d_t}} \right),
\]

where \( \mathcal{K}_m \in \mathbb{R}^{h \times l_v \times d_t} \) denotes key vectors of \( V_m \) projected by a linear layer and split into \( h \) tokens

![Overview of the MIRE architecture. This figure illustrates the interaction between the text encoder \( \mathcal{R}_T \) and the vision encoder \( \mathcal{R}_V \).](page_1047_186_393_312.png)

Figure 2: Overview of the MIRE architecture. This figure illustrates the interaction between the text encoder \( \mathcal{R}_T \) and the vision encoder \( \mathcal{R}_V \).
Visual dialog datasets

Split turns & Filter out “yes/no” answers

What brand of wine is this?
Chateau garraud

Chateau garraud (brand, wine)

Add information for simple answers

Image

What type of aircraft is shown in the image?

Question

The image shows a large passenger jet belonging to China airlines.

Response

Wikipedia

Synthesized passage

The Commercial Aircraft Corporation of China, Ltd. (COMAC) is a Chinese state-owned aerospace manufacturer established on 1 May 2008 in Shanghai. The company has a registered capital of RMB 19 billion (as of May 2008). The image shows a large passenger jet belonging to China airlines. China Airlines Flight 676 (CAL676, C676) was a scheduled international passenger flight operated by China Airlines on 1 February 2003. A 340-600 jet airliner operating the flight crashed into a road and residential area in Taoyuan County (now Taoyuan City), near Chang Kai-shek International Airport (present-day Taoyuan International Airport). The aircraft, registered as B-1814, was en route from Ngurah Rai Airport in Bali, Indonesia to Taipei, Taiwan.

China Airlines (CAL), is the state-owned flag carrier of the Republic of China (Taiwan), and one of its two major international carriers. The airline was established on 1 May 2008. China Airlines Flight 676 (CAL676, C676) was a scheduled international passenger flight. On Monday, 1 February 2003, a China Airlines Boeing 747-400 (B-1814) crashed into a road and residential area in Taoyuan County (now Taoyuan City), near Chang Kai-shek International Airport (present-day Taoyuan International Airport). The aircraft, registered as B-1814, was en route from Ngurah Rai Airport in Bali, Indonesia to Taipei, Taiwan.

Top-K retrieved passages

Response-to-Passage conversion

Figure 3: Our data construction process. Starting with visual dialogue datasets, our process involves two steps to convert the dialogue tasks to knowledge retrieval tasks. After preprocessing, we transform responses into a passage format by unifying the response and relevant passages retrieved from Wikipedia.

for each embedding within \( V_m \). Then, the attended visual output \( E_m \in \mathbb{R}^{h \times d_v} \) is calculated with value vectors \( \mathcal{V}_m \in \mathbb{R}^{h \times l_v \times d_v} \) of \( V_m \) as follows:

\[
E_m = \mathrm{Linear}\left( \frac{1}{l_v} \sum_i^{l_v} (\mathcal{A} \cdot \mathcal{V}_m) \right),
\]

where \( \mathcal{V}_m \) is produced via operations identical with \( K_m \). Unlike the standard cross-attention mechanism, we apply mean-pooling along the sequence dimension without a residual connection, yielding \( h \) visual embeddings. In this modality interaction process, we only leverage textual embeddings \( E_t \) to calculate \( \mathcal{A} \) as relevance scores for \( T \) without direct fusion. Our module alleviates the text dominance issue by explicitly limiting textual information during visual representation alignment, as demonstrated in our empirical analyses (see Sec. 5).

3.4 Dataset Construction
We aim to train our model to comprehend images based on textual queries, thereby enabling effective multimodal query understanding. To achieve this goal, we leverage existing multimodal question-response datasets, such as visual instruction-following data and VQA data. These datasets consist of query-response pairs where each pair is associated with a single image. In each pair, the response provides a concise and image-specific answer that directly addresses the textual query. Thus, the response serves as a clear bridge between the visual content and the query, explicitly linking image understanding to the language of the query. However, despite the explicit information provided by these responses, the datasets are not directly suitable for training the retriever \( \mathcal{R} \) because of the inherent difference between concise responses and more expansive passages. In practice, responses can be matched with queries without ambiguity, whereas real-world retrieval tasks demand the identification of relevant information embedded within broader documents that often contain noisy content. To bridge this gap, we transform query-response pairs into a format suitable for multimodal retrieval tasks via response-to-passage conversion, as illustrated in Fig. 3.

Response-to-Passage Conversion. Let a multimodal query-response pair \( S \) as follows:

\[
S = \{(I, T), R\},
\]

where \( R \) represents the response. We first attain multiple QA pairs for a single image from samples with several turns in source datasets. We divide the response \( R \) into two types: (1) detailed responses and (2) simple responses with a single word or a phrase. The simple responses often lack sufficient context to facilitate effective knowledge retrieval. Thus, we compensate simple responses with nouns extracted from the textual query \( T \). Note that we filter out pairs of responses that do not contribute to knowledge-based retrieval, such as simple affirmations and negations (e.g., “yes” or “no”).

The nature of the data \( S \) guarantees a high correlation between \( (I, T) \) and \( R \) since the responses contain information conditioned on the given multimodal query while the textual queries have restrictive information. Thus, we utilize the response \( R \) to
transform the response into an informative passage. From an arbitrary knowledge base \( U \), we retrieve relevant passages using the response \( R \) as the query. Specifically, we obtain the top-\( k \) passages:

\[
\{D_1, D_2, \ldots, D_k\} = \text{Retrieve}_{\mathcal{R}_T}(R, U, k),
\]

where \( \text{Retrieve}_{\mathcal{R}_T} \) denotes the retrieval function that returns the top \( k \) relevant passages from the knowledge base \( U \) based on the query \( R \) using the text retriever \( \mathcal{R}_T \). To maintain contextual relevance with the multimodal query (\( I, T \)), we then augment the response \( R \) by combining it with the retrieved passages:

\[
R' = [D_1; R; D_2; \ldots; D_k].
\]

This conversion strategy yields training data that more closely mimic the complexity and noise of real-world documents. Consequently, the retriever is exposed to more challenging and realistic scenarios during training, enabling it to effectively integrate visual cues and ultimately achieve more robust retrieval performance.

3.5 Training and Inference
We deal with passages including the golden answers to a given question \( Q \) as relevant passages \( K \). To train our model, we employ in-batch negative sampling, which treats all passages in a training batch except for a passage \( D \) belonging to \( K \) as negative passages \( \bar{K} \) for \( Q \). We optimize our model by minimizing the following contrastive loss \( \mathcal{L}_{CL} \) over the dataset \( \mathcal{D} \):

\[
\mathcal{L}_{CL} = - \sum_{D} \log \frac{\exp(r_{Q,D}/\tau)}{\exp(r_{Q,D}/\tau) + \sum_{D \in \bar{K}} \exp(r_{Q,D}/\tau)},
\]

where \( \tau \) is the temperature parameter that regulates the influence of penalties on negative samples. During the alignment stage, all parameters of \( \mathcal{R}_T \) and \( \mathcal{R}_V \) are frozen, preserving the established text retrieval performance. To focus on visual alignment, we exclude the textual embeddings \( E_t \) from the final query embedding \( E_Q \), using only the visual features \( [E_g; E_m] \) as \( E_Q \). We also integrate a subset of a multimodal knowledge base, WiT (Srinivasan et al., 2021), into our dataset to enrich the world knowledge learned during alignment. Note that this addition does not affect multimodal query understanding because the dataset consists solely of pairs of an image and a passage (i.e., it does not include a textual query). For such data, we simply assign dummy prompts for multimodal queries (e.g., What is the core object or subject shown here?). We discuss this integration in Sec. 5 in detail.

After the alignment stage, we add textual embeddings \( E_t \) to \( E_Q \) when training on downstream tasks and the inference stage. For efficient retrieval, all passages within knowledge base \( U \) are pre-indexed using PLAID (Santhanam et al., 2022a), identical to ColBERTv2 (Santhanam et al., 2022b).

4 Experiments

4.1 Setup
Benchmarks. We employ four benchmarks for knowledge retrieval with multimodal queries: two variants of OK-VQA (Marino et al., 2019), ReMuQ (Luo et al., 2023), and E-VQA (Mensink et al., 2023). For OK-VQA, we use two versions based on different knowledge bases: OKVQA-GS, a corpus collected using Google search API as introduced in Luo et al. (2021a), and OKVQA-WK11M, a corpus containing 11 million Wikipedia passages compiled by Qu et al. (2020).

Metrics. We evaluate retrieval performance using Mean Reciprocal Rank at 5 (MRR@5), Recall@k (R@k), and Pseudo Recall@k (PR@k) across four benchmarks. MRR@5 measures the ranking quality of the first relevant passage. For OKVQA-GS and E-VQA, which do not provide explicit ground-truth passages, we compute PR@5 by checking whether retrieved documents contain the correct answer. For OKVQA-WK11M and ReMuQ, we evaluate R@k by verifying whether the target passages appear in the top-\( k \) results.

Implementation Details. Our pre-training dataset is synthesized from three visual instruction datasets (Zhang et al., 2023; Wang et al., 2023; Liu et al., 2024) and two VQA datasets (Singh et al., 2019; Biten et al., 2019), resulting in 1.35 million QA pairs, each paired with an image after preprocessing. We sampled to have no more than 12 question-response pairs per image. For the response-to-passage conversion, we utilize 6 million Wikipedia articles released by Chen et al. (2023) as our data pool. We retrieve three candidate passages for each response using ColBERTv2, trained with the MS MARCO Passage Ranking task (Nguyen et al., 2016). Each passage is truncated to three sentences, and the response is inserted between the first and second passages to ensure contextual consistency. We also added 0.5 million pairs randomly sampled from WiT.

For our base model, we adopt CLIP ViT-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">OKVQA-GS</th>
    <th colspan="4">OKVQA-WK11M</th>
    <th colspan="4">ReMuQ</th>
    <th colspan="4">E-VQA</th>
  </tr>
  <tr>
    <th>MRR@5</th>
    <th>PR@5</th>
    <th>PR@10</th>
    <th>MRR@5</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>MRR@5</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>MRR@5</th>
    <th>PR@5</th>
    <th>PR@10</th>
    <th>MRR@5</th>
    <th>PR@5</th>
    <th>PR@10</th>
  </tr>
  <tr>
    <td>CLIP (Radford et al., 2021)</td>
    <td>19.08</td>
    <td>34.54</td>
    <td>50.48</td>
    <td>16.45</td>
    <td>29.81</td>
    <td>43.0</td>
    <td>0.34</td>
    <td>0.78</td>
    <td>1.36</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>FLMR (Lin et al., 2023)</td>
    <td>38.15</td>
    <td>57.25</td>
    <td>69.42</td>
    <td>32.56</td>
    <td>50.61</td>
    <td>62.58</td>
    <td>66.67</td>
    <td>72.10</td>
    <td>74.95</td>
    <td>29.97</td>
    <td>42.0</td>
    <td>50.75</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ReViz (Luo et al., 2023)</td>
    <td>45.77</td>
    <td>64.05</td>
    <td>75.39</td>
    <td>44.03</td>
    <td>62.43</td>
    <td>73.44</td>
    <td>23.61</td>
    <td>39.43</td>
    <td>46.77</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>UniIR (Wei et al., 2024)</td>
    <td>53.27</td>
    <td>73.94</td>
    <td>84.19</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>79.15</td>
    <td>84.34</td>
    <td>86.62</td>
    <td>31.59</td>
    <td>44.21</td>
    <td>53.31</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>VISTA (Zhou et al., 2024a)</td>
    <td>55.33</td>
    <td>72.83</td>
    <td>81.61</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>78.32</td>
    <td>84.21</td>
    <td>87.03</td>
    <td>33.9</td>
    <td>47.73</td>
    <td>56.72</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>PreFLMR* (Lin et al., 2024)</td>
    <td>59.38</td>
    <td>76.83</td>
    <td>84.34</td>
    <td>45.68</td>
    <td>63.85</td>
    <td>73.64</td>
    <td>52.27</td>
    <td>54.31</td>
    <td>55.06</td>
    <td>30.92</td>
    <td>41.71</td>
    <td>49.44</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>MIRe</td>
    <td><b>63.03</b></td>
    <td><b>80.48</b></td>
    <td><b>88.15</b></td>
    <td><b>51.15</b></td>
    <td><b>70.71</b></td>
    <td><b>81.25</b></td>
    <td><b>83.06</b></td>
    <td><b>86.84</b></td>
    <td><b>88.56</b></td>
    <td><b>41.88</b></td>
    <td><b>54.24</b></td>
    <td><b>61.01</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>w/ ViT-large</td>
    <td>63.17</td>
    <td>81.13</td>
    <td>88.72</td>
    <td>50.64</td>
    <td>69.92</td>
    <td>80.18</td>
    <td>82.56</td>
    <td>86.48</td>
    <td>88.17</td>
    <td>44.92</td>
    <td>57.65</td>
    <td>64.40</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 1: Zero-shot performance of MIRe and comparison methods. Note that FLMR was only pre-trained on the WiT dataset. PreFLMR* were trained using our dataset and experimental settings. Bold indicates the highest performance.

base (Radford et al., 2021) as a vision encoder and ColBERTv2 as a text encoder based on BERT-base (Devlin et al., 2019). The number of tokens for visual embeddings \( E_g \) and \( E_m \) are set to 16 and 12, respectively. The value for \( E_m \) is determined by the number of heads \( h \) in the interaction module. The dimension of the final embeddings \( d_f \) is set to 128, consistent with the text encoder. Our base model has 211M parameters. Further implementation details are provided in Appendix A.

Comparison Methods. We benchmark our MIRe model against a diverse set of baseline models that employ pre-training stages for visual-text alignment: CLIP (Radford et al., 2021), FLMR (Lin et al., 2023), ReViz (Luo et al., 2023), PreFLMR (Lin et al., 2024), and VISTA (Zhou et al., 2024a). UniIR (Wei et al., 2024), which requires an explicit instruction input, is also included; we follow their protocol and use the instruction "Retrieve a passage that answers the given query about the image" during evaluation. Both FLMR and PreFLMR utilize the same vision and text encoders as our model, where FLMR was pre-trained with a subset of the WiT dataset. For direct comparison, PreFLMR was trained using the same pre-training procedure as our model, thereby highlighting the distinct advantages of our model architecture. For zero-shot (ZS) evaluation, we exclude baselines requiring external supervision for fairness, while for few-shot (FS) evaluation, models utilizing supervision datasets are included.

4.2 Main Results

Zero-shot Retrieval Performance. Tab. 1 shows that our method achieves superior zero-shot retrieval performance across all four benchmarks, significantly outperforming the comparison models. Despite employing a two-stage training strategy and directly optimizing the vision encoder for retrieval, VISTA still underperforms relative to our approach. Even though PreFLMR was trained under the same settings as our model, it exhibits a significant performance gap compared to our model. These results validate the effectiveness of our modality interaction approach. Our method also benefits from increased model capacity. The variant employing a larger vision encoder (ViT-large) shows similar performance to the standard model, but it further outperforms the standard model in E-VQA.

Fine-tuning on Downstream Tasks We further demonstrate the adaptability of our model and the effectiveness of our pre-training task by fine-tuning models on downstream tasks. Tab. 2 demonstrates remarkable adaptability when fine-tuned on downstream tasks. On the OKVQA-GS dataset, our model substantially outperforms all state-of-the-art models. On the ReMuQ dataset, our model still delivers strong performance, showing its competitive results. It is important to note that our method achieved higher performance on ReMuQ than VISTA in the zero-shot setting, which suggests that our pre-training and modality interaction approach endow our model with strong generalization capabilities. Notably, the variant without pre-training clearly lags behind the pre-trained model, highlighting the crucial role of our pre-training task. Furthermore, employing a larger vision encoder (ViT-large) yields additional improvements on OKVQA-GS, demonstrating the scalability of our approach. Overall, these results confirm that our model not only excels in zero-shot settings but also adapts effectively to fine-tuning on downstream tasks.

4.3 Ablation Studies

Our ablation studies, summarized in Tab. 3, reveal that each component in our framework plays a sig-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">OKVQA-GS</th>
    <th colspan="4">ReMuQ</th>
  </tr>
  <tr>
    <th>PR@5</th>
    <th>PR@10</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>PR@5</th>
    <th>PR@10</th>
    <th>R@5</th>
    <th>R@10</th>
  </tr>
  <tr>
    <td>FLMR (Lin et al., 2023)</td>
    <td>70.63</td>
    <td>81.23</td>
    <td>62.76</td>
    <td>74.67</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VRR (Luo et al., 2021b)</td>
    <td>71.5</td>
    <td>81.5</td>
    <td>-</td>
    <td>-</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ReViz (Luo et al., 2023)</td>
    <td>73.35</td>
    <td>83.17</td>
    <td>23.61</td>
    <td>39.43</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GeMKR (Long et al., 2024)</td>
    <td>78.6</td>
    <td>86.2</td>
    <td>90.3</td>
    <td>92.7</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VISTA (Zhou et al., 2024a)</td>
    <td>82.06</td>
    <td>90.11</td>
    <td><b>96.3</b></td>
    <td><b>97.3</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Ours w/o Pre-training</td>
    <td>74.26</td>
    <td>84.07</td>
    <td>92.44</td>
    <td>94.38</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>83.59</b></td>
    <td><b>90.59</b></td>
    <td>94.40</td>
    <td>96.20</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/ ViT-large</td>
    <td>84.66</td>
    <td>91.30</td>
    <td>94.38</td>
    <td>96.18</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 2: Fine-tuning performance on two tasks.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">OK-GS</th>
    <th colspan="2">OK-WK</th>
    <th colspan="2">ReMuQ</th>
    <th colspan="2">E-VQA</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th>PR@5</th>
    <th>PR@10</th>
    <th>PR@5</th>
    <th>PR@10</th>
    <th>PR@5</th>
    <th>PR@10</th>
    <th>PR@5</th>
    <th>PR@10</th>
    <th></th>
  </tr>
  <tr>
    <td>Base</td>
    <td><b>63.03</b></td>
    <td><b>51.15</b></td>
    <td>83.06</td>
    <td>41.88</td>
    <td><b>59.78</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o WiT</td>
    <td>62.54</td>
    <td>50.53</td>
    <td>82.63</td>
    <td>40.88</td>
    <td>59.15</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o R2P</td>
    <td>60.43</td>
    <td>42.93</td>
    <td>81.87</td>
    <td>38.13</td>
    <td>55.84</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/ Single T</td>
    <td>59.72</td>
    <td>49.09</td>
    <td>79.27</td>
    <td>29.29</td>
    <td>54.34</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/ Residual</td>
    <td>61.65</td>
    <td>47.95</td>
    <td>80.47</td>
    <td><b>43.06</b></td>
    <td>58.28</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o \( E_m \)</td>
    <td>60.19</td>
    <td>47.23</td>
    <td>81.70</td>
    <td>39.01</td>
    <td>57.03</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/ \( E_t \)</td>
    <td>51.38</td>
    <td>42.13</td>
    <td>71.69</td>
    <td>32.80</td>
    <td>49.50</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="5">IF</td>
    <td>w/o \( E_m \)</td>
    <td>60.43</td>
    <td>44.13</td>
    <td>85.10</td>
    <td>42.4</td>
    <td>58.02</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o \( E_t \)</td>
    <td>58.4</td>
    <td>44.61</td>
    <td><b>85.91</b></td>
    <td>40.24</td>
    <td>57.29</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o \( E_t \) & \( E_m \)</td>
    <td>52.46</td>
    <td>36.0</td>
    <td>71.69</td>
    <td>42.48</td>
    <td>50.66</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o \( E_t \)</td>
    <td>36.99</td>
    <td>36.68</td>
    <td>2.73</td>
    <td>11.39</td>
    <td>21.95</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 3: Ablation Studies. Retrieval performance (MRR@5) in zero-shot settings across four datasets. "PT" and "IF" indicate ablations performed at the pre-training and inference stages, respectively.

significant role in achieving robust zero-shot retrieval performance. We examine the contributions of our design from three perspectives: the dataset, model architecture during alignment, and the embeddings used at inference.

Dataset. In the pre-training stage (PT), omitting external knowledge from the WiT dataset causes only a slight performance drop, underscoring its supportive role (see Sec. 5). In contrast, training the model on original responses without applying the response-to-passage conversion (R2P) results in a substantially larger decline. These observations indicate that the R2P mechanism is essential for enhancing visual-text alignment and overall knowledge retrieval. We also investigate the effect of multiple QA pairs per image. As shown in Tab 3, although sampling a single QA pair per image keeps the total number of images, this variant (w/ Single T) significantly degrades retrieval performance, suggesting the presence of hard-negative effects beyond simple visual-image alignment.

Model. We further examine how directly fusing text features during the alignment process affects performance. When we add a residual connection to our model architecture before sequential-wise pooling (w/ Residual), we observe a performance drop, indicating a slight exacerbation of the text-dominant issue. Moreover, when text features are allowed an even more direct influence, by setting \( E_Q = [E_g, E_m, E_t] \) during alignment, the performance degrades considerably.

Embeddings \( E_Q \). At the inference stage (IF), our analysis shows that each embedding type plays a unique and complementary role. Removing either the modality-specific embedding (w/o \( E_m \)) or the general embedding (w/o \( E_g \)) leads to a moderate decline in performance, suggesting that both capture distinct yet essential aspects of the data. However, removing these components simultaneously causes a sharper performance drop. Notably, omitting the text embedding (w/o \( E_t \)) results in severe degradation of retrieval accuracy, indicating that \( E_t \) is indispensable for maintaining semantic coherence. This clear hierarchy in the impact of each embedding underscores their distinct functions and the need for their balanced integration.

![UMAP visualization of embedding space](page_1012_120_388_246.png)
(a) UMAP visualization

![Attention visualization for each token](page_1012_370_388_246.png)
(b) Attention visualization for each token (averaged across heads)

Figure 4: Visualization of multimodal query processing, illustrating the alignment between textual and visual modalities.
<table>
  <tr>
    <th>Dataset</th>
    <th>OKVQA-GS</th>
    <th>ReMuQ</th>
    <th>E-VQA</th>
    <th>Infoseek</th>
  </tr>
  <tr>
    <td>FLMR (Lin et al., 2023)</td>
    <td>57.25</td>
    <td>72.10</td>
    <td>42.0</td>
    <td><b>42.93</b></td>
  </tr>
  <tr>
    <td>w/o WiT</td>
    <td><b>81.11</b></td>
    <td><b>87.45</b></td>
    <td>51.95</td>
    <td>37.15</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>80.48</td>
    <td>86.84</td>
    <td><b>54.24</b></td>
    <td>42.61</td>
  </tr>
  <tr>
    <td>w/ WiT (0.5M)</td>
    <td>79.63</td>
    <td>86.59</td>
    <td><b>54.05</b></td>
    <td><b>44.01</b></td>
  </tr>
</table>

Table 4: Zero-shot retrieval performance (R@5) under knowledge integration settings using WiT data.

![Training loss curves and R@5 on OK-VQA](page_184_370_370_246.png)

Figure 5: Training convergence and retrieval performance. All models were trained for only one epoch under the same settings.

5 Discussion

Effect of Query-guided Attentive Pooling To demonstrate the effectiveness of MIRe in capturing modality interactions, we visualize the embeddings and attention maps of multimodal queries on a controlled dataset. We synthesized 224 images with the prompt ‘A dog and a cat in an image’ using Diffusion-XL (Podell et al., 2024), and conditioned the embeddings \( E_m \) on three distinct textual prompts: (1) What is the dog or puppy’s species?, (2) What is the cat or kitty’s species?, and (3) Where is the place in the image? In Fig. 4(a), the UMAP clustering (McInnes et al., 2018) of \( E_q \) and \( E_m \) illustrates MIRe effectively separates visual embeddings based on the query’s intent. Additionally, Fig. 4(b) visualizes attention patterns of our pooling module, revealing how the model attends to specific visual patches relevant to each query. These results demonstrate that MIRe enhances interactions between textual and visual modalities.

Effect of Knowledge Integration. We further assess MIRe’s capacity for external knowledge integration by incorporating the WiT dataset and analyzing its effect on retrieval performance, particularly on the Infoseek dataset (Chen et al., 2023). As shown in Tab. 4, Ours w/o WiT falls short on Infoseek relative to FLMR while competitively performing on other benchmarks. Notably, FLMR was learned with a subset of WiT without modality interaction. When we integrate external knowledge using 0.5 million WiT data, our model’s performance on Infoseek is substantially improved to 42.61, bringing it on par with FLMR. Moreover, further increasing the WiT data to 1.0 million boosts the R@5 on Infoseek to 44.01. These findings, however, reveal that the performance gains observed on Infoseek are largely driven by its heavy reliance on external knowledge, which raises concerns about the generality of evaluation protocols that depend on such background information.

Text-dominant Issue. We analyze how certain pre-training strategies and model architectures exacerbate reliance on textual features during multimodal alignment. In Fig. 5(a), both PreFLMR and w/ Residual exhibit faster loss convergence compared to our model, suggesting that directly leveraging text features accelerates optimization. However, as shown in Fig. 5(b), the accelerated convergence does not translate to improved performance, with PreFLMR and w/ Residual underperforming relative to our model. The text-dominant issue is further exacerbated when using VL-ICT, introduced in ReViz, which constructs pseudo-queries from passages. Such behavior reveals the text-dominant issue, where excessive dependence on text features during alignment hinders the model’s ability to fully leverage multimodal information. Fig. 6 illustrates this effect by visualizing the alignment of multimodal query embeddings \( Q \) with passage embeddings \( D \). In (a), when \( E_t \) is explicitly used during alignment \( Q \) embeddings (orange) remain largely separated from the passage space, indicating poor alignment. In contrast, (b) demonstrates that our method effectively incorporates \( Q \) embeddings (green) into the linguistic space, improving alignment. These results suggest that excessive reliance on text features inhibits the multimodal query embeddings from adapting properly to the passage space.

![Comparison of Embedding Distribution](page_1012_120_370_246.png)

Figure 6: Comparison of Embedding Distribution. (a) with \( E_t \) during alignment, where query embeddings (\( Q \), orange) remain distinct from passage embeddings (\( D \), blue); (b) our method, where \( Q \) (green) is better integrated into the textual space.
6 Conclusion

We introduced MIRe, a novel retrieval framework designed for multimodal query retrieval without fusing textual features during the alignment stage. Our query-guided attentive pooling module allows textual embeddings to attend to visual patches while preventing text-driven signals from dominating the visual representations. We also constructed a pre-training dataset by converting concise question-answer pairs into extended passages, thereby exposing the model to more realistic retrieval tasks. Our extensive experiments demonstrate that MIRe consistently outperforms existing methods under both zero-shot and fine-tuned settings. Ablation studies further validate that each component of MIRe is crucial for achieving robust multimodal query retrieval.

7 Limitations

Despite the promising results, our work has several limitations that point to potential directions for future research. First, while our approach demonstrates strong performance across general-domain benchmarks, it remains untested in specialized domains (e.g., medical or legal documents), where multimodal content may exhibit more complex and domain-specific features. Second, we have not explored synergy with retrieval-augmented generative (RAG) frameworks, which typically prepend retrieved passages to a language model for downstream generation tasks. Although we believe our retrieval improvements would benefit RAG-based methods, in line with findings from Kim et al. (2024) showing that stronger retrievers enhance downstream generation, fully validating our approach in a RAG pipeline is left for future work. Finally, our current data construction method focuses on retrieval from large yet homogeneous corpora; adapting the framework to more diverse or dynamically changing knowledge sources may require additional techniques to handle domain shifts or continuously updated information.

8 Acknowledgment

This work was supported by the Agency For Defense Development by the Korean Government(UI247035TF)

References

Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marcà Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. 2019. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291–4301.

D Chen. 2017. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051.

Jianyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics (ACL), pages 2318–2335.

Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. 2023. Can pre-trained vision and language models answer visual information-seeking questions? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 14948–14968.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 4171–4186.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR).

Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. 2022. A thousand words are worth more than a picture: Natural language-centric outside-knowledge visual question answering. arXiv preprint arXiv:2201.05299.

Liangke Gui, Borui Wang, Qiuyuan Huang, Alexander Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2022. KAT: A knowledge augmented transformer for vision-and-language. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 956–968.

Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. 2023. Open-domain visual entity recognition: Towards recognizing millions of Wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12065–12075.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research (TMLR).

Vladimir Karpukhin, Barlas Ogun, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781.

Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the International ACM SIGIR conference on research and development in Information Retrieval (SIGIR), pages 39–48.

Jaehyun Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin. 2024. Sure: Summarizing retrievals using answer candidates for open-domain QA of LLMs. In The International Conference on Learning Representations (ICLR).

Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 6086–6096.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning (ICML), pages 12888–12900.

Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. 2023. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems (NeurIPS), 36.

Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. 2024. PreFLMR: Scaling up fine-grained late-interaction multi-modal retrievers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 5294–5316.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306.

Siqi Liu, Weixi Feng, Tsu-Jui Fu, Wenhui Chen, and William Wang. 2023. EDIS: Entity-driven image search over multimodal web content. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4877–4894.

Xinwei Long, Jiali Zeng, Fandong Meng, Zhiyuan Ma, Kaiyan Zhang, Bowen Zhou, and Jie Zhou. 2024. Generative multi-modal knowledge retrieval with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 38, pages 18733–18741.

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR).

Man Luo, Zhiyuan Fang, Tejas Gokhale, Yezhou Yang, and Chitta Baral. 2023. End-to-end knowledge retrieval with multi-modal queries. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 8573–8589.

Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. 2021a. Weakly-supervised visual-retriever-reader for knowledge-based question answering. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6417—6431.

Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. 2021b. Weakly-supervised visual-retriever-reader for knowledge-based question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6417–6431.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3195–3204.

Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. 2018. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, page 861.

Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. 2023. Encyclopedia VQA: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3113–3124.

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human-generated machine reading comprehension dataset.

Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9844–9855.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2024. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The International Conference on Learning Representations (ICLR).
Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W Bruce Croft, and Mohit Iyyer. 2020. Open-retrieval conversational question answering. In Proceedings of the International ACM SIGIR conference on research and development in Information Retrieval (SIGIR), pages 539–548.

Chen Qu, Hamed Zamani, Liu Yang, W Bruce Croft, and Erik Learned-Miller. 2021. Passage retrieval for outside-knowledge visual question answering. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 1753–1757.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748–8763.

Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, pages 333–389.

Alireza Salemi, Juan Altmayer Pizzorno, and Hamed Zamani. 2023. A symmetric dual encoding dense retrieval framework for knowledge-intensive visual question answering. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 110–120.

Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022a. Plaid: an efficient engine for late interaction retrieval. In Proceedings of the ACM International Conference on Information & Knowledge Management (CIKM), pages 1747–1756.

Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022b. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 3715–3734.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8317–8326.

Krishna Srinivasan, Karthik Ramam, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), page 2443–2449.

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 809–819.

Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. 2023. To see is to believe: Prompting GPT-4V for better visual instruction tuning. arXiv preprint arXiv:2311.07574.

Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhui Chen. 2024. Unir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision (ECCV), pages 387–404.

Jialin Wu and Raymond Mooney. 2022. Entity-focused dense passage retrieval for outside-knowledge visual question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8061–8072.

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI), volume 36, pages 3081–3089.

Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. 2023. LLAVAR: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107.

Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. 2024a. VISTA: Visualized text embedding for universal multi-modal retrieval. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 3185–3200.

Tianshuo Zhou, Sen Mei, Xinze Li, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, Yu Gu, and Ge Yu. 2024b. MARVEL: Unlocking the multi-modal capability of dense retrieval via visual module plugin. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 14608–14624.

Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774.

A Appendix

A.1 Training and Inference Details

In all experiments, we train models using the AdamW optimizer (Loshchilov and Hutter, 2019) with warm-up steps on a machine with 4 RTX A6000 GPUs. We chose model checkpoints based
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="7">Hyperparameter</th>
  </tr>
  <tr>
    <th>LR</th>
    <th># Epochs</th>
    <th># Batch per GPU</th>
    <th># Global Batch</th>
    <th># Warm-up</th>
    <th>\( \tau \)</th>
  </tr>
  <tr>
    <td>Pre-training (R2P)</td>
    <td>1e-4</td>
    <td>5</td>
    <td>128</td>
    <td>512</td>
    <td>300</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td>OKVQA-GS</td>
    <td>5e-5</td>
    <td>15</td>
    <td>128</td>
    <td>512</td>
    <td>10</td>
    <td>0.8</td>
  </tr>
  <tr>
    <td>ReMuQ</td>
    <td>5e-5</td>
    <td>5</td>
    <td>128</td>
    <td>512</td>
    <td>10</td>
    <td>0.8</td>
  </tr>
</table>

Table 5: Summary of hyperparameters utilized for training. The LR denotes the learning rate.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="3">Size</th>
  </tr>
  <tr>
    <th>#Train</th>
    <th>#Test</th>
    <th>KB \( U \)</th>
  </tr>
  <tr>
    <td>OKVQA-GS</td>
    <td>8,958</td>
    <td>5,046</td>
    <td>166,390</td>
  </tr>
  <tr>
    <td>OKVQA-WK11M</td>
    <td>-</td>
    <td>2,523</td>
    <td>11,000,000</td>
  </tr>
  <tr>
    <td>ReMuQ</td>
    <td>8,418</td>
    <td>3,609</td>
    <td>195,387</td>
  </tr>
  <tr>
    <td>E-VQA</td>
    <td>-</td>
    <td>3,750</td>
    <td>51,462</td>
  </tr>
</table>

Table 6: Summary of dataset statistics for evaluation. This table presents the distribution of training and testing instances alongside the size of the knowledge bases for each dataset employed in our study. GS and KB denote the corpus collected from the Google Search API and used knowledge base, respectively.

<table>
  <tr>
    <th>Statistic</th>
    <th>Counts</th>
  </tr>
  <tr>
    <td># Total data</td>
    <td>1,356,536</td>
  </tr>
  <tr>
    <td># Images</td>
    <td>264,262</td>
  </tr>
  <tr>
    <td># Max. queries per image</td>
    <td>12</td>
  </tr>
  <tr>
    <td># Avg. queries per image</td>
    <td>8.32</td>
  </tr>
  <tr>
    <td># Queries requiring description</td>
    <td>230,877 (17.02%)</td>
  </tr>
  <tr>
    <td># Other types of queries</td>
    <td>1,125,659 (82.98%)</td>
  </tr>
</table>

Table 7: Statistics of our constructed dataset.

on the validation loss. We set hyperparameters for each dataset as shown in Tab. 5.

Pre-training. We used \( E_Q = [E_g; E_m] \) without \( E_t \) to align visual embeddings with the linguistic space during the pre-training stage. In this stage, we only tuned the mapping network, such as a MLP layer for \( E_g \) and the query-guided attentive pooling module. PreFLMR and MIRe were set with the same hyperparameters.

Fine-tuning. For fine-tuning our model on downstream tasks, we tuned all parameters of our model except for the vision model in all experiments. Since the parameters of the vision model are not updated during training, we cached the outputs of the vision model before training. In our setting, training one epoch for our dataset took about 20 minutes on 4 RTX A6000 GPUs, where one epoch encompasses 3625 steps. We detail statistics of benchmark datasets in Tab. 6.

![UMAP visualization of embeddings extracted using the Contriever model, comparing Wikipedia documents (purple) and LLaVA responses (red).](page_1012_1012_388_388.png)

Figure 7: UMAP visualization of embeddings extracted using the Contriever model (Izacard et al., 2022), comparing Wikipedia documents (purple) and LLaVA responses (red). The separation between clusters highlights the structural and semantic differences.

Inference. Passages within the knowledge base were pre-indexed, following the method established by the previous work (Santhanam et al., 2022b). The indexing process consists of three critical steps: centroid selection, passage encoding, and index inversion. To enhance storage efficiency, embeddings were compressed to 2 bits per dimension. In the OK-VQA dataset using a corpus collected from Google search API, the retrieval time of MIRe and ColBERTv2 is approximately 0.085 seconds and 0.081 seconds per query on one RTX A6000 GPU, respectively. Thus, MIRe spends slightly more time retrieving relevant passages with multimodal queries, compared to the base text retriever.

A.2 Details for Our Dataset

To construct our dataset, we employ three visual instruction datasets (Zhang et al., 2023; Wang et al., 2023; Liu et al., 2024) and two VQA datasets (Singh et al., 2019; Biten et al., 2019). Initially, samples were split into individual turns. We removed turns with responses shorter than 30 characters only for detailed responses. Subsequently, we edited responses containing simple affirmations ("yes", "no") and excluded samples for tasks irrelevant to retrieval tasks (e.g., location and count),
<table>
  <tr>
    <th>Source Dataset</th>
    <th># Data</th>
    <th># Images</th>
    <th># Avg. \( S \) per \( I \)</th>
    <th># Max. \( S \) per \( I \)</th>
  </tr>
  <tr>
    <td>ST-VQA (<b>Biten et al., 2019</b>)</td>
    <td>25,154</td>
    <td>18,518</td>
    <td>1.36</td>
    <td>7</td>
  </tr>
  <tr>
    <td>TextVQA (<b>Singh et al., 2019</b>)</td>
    <td>26,406</td>
    <td>18,913</td>
    <td>1.40</td>
    <td>2</td>
  </tr>
  <tr>
    <td>LLaVAR (<b>Zhang et al., 2023</b>)</td>
    <td>42,690</td>
    <td>19,787</td>
    <td>2.16</td>
    <td>7</td>
  </tr>
  <tr>
    <td>Instruct4V (<b>Wang et al., 2023</b>)</td>
    <td>222,711</td>
    <td>26,663</td>
    <td>8.35</td>
    <td>12</td>
  </tr>
  <tr>
    <td>LLaVA-1.5 (<b>Liu et al., 2024</b>)</td>
    <td>1,017,622</td>
    <td>158,429</td>
    <td>6.42</td>
    <td>12</td>
  </tr>
  <tr>
    <td>Subset of WiT (<b>Srinivasan et al., 2021</b>)</td>
    <td>500,000</td>
    <td>500,000</td>
    <td>1</td>
    <td>1</td>
  </tr>
</table>

Table 8: <b>Statistics of each source dataset within our dataset.</b> \( S \) per \( I \) denotes the number of queries per image.

<table>
  <tr>
    <th rowspan="2">Source Dataset</th>
    <th colspan="2">OKVQA-GS</th>
    <th colspan="2">OKVQA-WK11M</th>
    <th colspan="2">ReMuQ</th>
    <th colspan="2">E-VQA</th>
  </tr>
  <tr>
    <th>PR@5</th>
    <th>PR@10</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>PR@5</th>
    <th>PR@10</th>
  </tr>
  <tr>
    <td>ST-VQA (<b>Biten et al., 2019</b>)</td>
    <td>72.29</td>
    <td>81.45</td>
    <td>57.35</td>
    <td>67.97</td>
    <td>85.79</td>
    <td>87.81</td>
    <td>51.01</td>
    <td>58.37</td>
  </tr>
  <tr>
    <td>TextVQA (<b>Singh et al., 2019</b>)</td>
    <td>72.18</td>
    <td>81.75</td>
    <td>57.83</td>
    <td>69.20</td>
    <td>86.03</td>
    <td>87.97</td>
    <td>51.41</td>
    <td>58.80</td>
  </tr>
  <tr>
    <td>LLaVAR (<b>Zhang et al., 2023</b>)</td>
    <td>73.11</td>
    <td>82.62</td>
    <td>60.88</td>
    <td>71.90</td>
    <td>86.34</td>
    <td>88.39</td>
    <td>51.89</td>
    <td>58.75</td>
  </tr>
  <tr>
    <td>Instruct4V (<b>Wang et al., 2023</b>)</td>
    <td>78.72</td>
    <td>86.88</td>
    <td>65.83</td>
    <td>75.90</td>
    <td>86.01</td>
    <td>87.81</td>
    <td>52.0</td>
    <td>59.25</td>
  </tr>
  <tr>
    <td>LLaVA-1.5 (<b>Liu et al., 2024</b>)</td>
    <td>79.41</td>
    <td>87.77</td>
    <td>68.05</td>
    <td>78.32</td>
    <td>86.56</td>
    <td>88.31</td>
    <td><b>52.56</b></td>
    <td><b>59.92</b></td>
  </tr>
  <tr>
    <td>Total</td>
    <td><b>81.11</b></td>
    <td><b>88.84</b></td>
    <td><b>70.55</b></td>
    <td><b>82.20</b></td>
    <td><b>87.45</b></td>
    <td><b>88.45</b></td>
    <td>51.95</td>
    <td>59.12</td>
  </tr>
</table>

Table 9: <b>Zero-shot performance by each source dataset.</b> We apply our response-to-passage conversion process to each source dataset. Note that we did not add WiT data in this experiment.

where we automatically filtered out based on specific phrases.

Fig. 7 illustrates there exists a clear distinction between the concise responses and more expansive passages, supporting our perspective. After the pre-processing, we refined the data through a response-to-passage conversion using ColBERTv2, a text retriever trained on the MS MARCO Passage Ranking task (Nguyen et al., 2016). Responses were converted into passages using a pool of 6 million Wikipedia documents (Chen et al., 2023), with textual queries limited to 128 tokens. As shown in Fig. 8, our constructed dataset is featured by pairs of multimodal queries and passages including responses to different queries about the same image, advancing the capability to retrieve relevant information from multimodal queries. This process yielded a total of 1.36 million QA pairs; further data statistics are provided in Tab. 7 and Tab. 8.

Table 9 summarizes the zero-shot retrieval performance for each source dataset. The results demonstrate that our conversion process effectively leverages complementary strengths from various datasets, underscoring the robustness of our approach. Additionally, when unifying WiT data, we assigned textual queries by randomly sampling from the following prompts: “What is the main object?”, “Identify the subject of this image.”, “Who or what is the subject in this picture?”, “Identify the main entity.”, and “What is the core object or subject shown here?”.

A.3 Architectural Differences: MIRe vs. PreFLMR

Our architecture differs from PreFLMR in several important aspects. In PreFLMR, visual tokens serve as queries, and the hidden states of the text encoder act as keys and values within a cross-attention mechanism. Attention scores \( \mathcal{A} \in \mathbb{R}^{H \times Q \times K} \) are computed using these components, and value vectors \( \mathcal{V} \) are derived from the truncated text encoder hidden states, which may cause information loss. The resulting outputs have dimensions \([H, Q, D]\) and are reshaped to \([Q, H \cdot D]\) before undergoing further reduction by the ColBERT head.

In contrast, MIRe uses textual embeddings \( E_t \) from the ColBERT head as queries, with visual tokens split into multiple heads for keys and values. Here, \( Q \) corresponds to the number of textual tokens and \( K \) to the number of visual tokens, reversing the modality roles compared to PreFLMR. Value vectors are obtained directly from the visual embeddings without truncation. Output aggregation is performed by mean-pooling across the sequence dimension, producing representations of shape \([H, D]\), which are passed through a linear layer without further dimensionality reduction.
<table>
  <tr>
    <th>Image I</th>
    <th>Question T</th>
    <th>Passage D</th>
  </tr>
  <tr>
    <td rowspan="2">
      <img src="page_184_352_370_180.png" alt="A bus driving down a street">
    </td>
    <td>What feature can be seen on the back of the bus?</td>
    <td>In bus advertising, buses and their related infrastructure is a medium commonly used by advertisers to reach the public with their message. Usually, this takes the form of promoting commercial brands, but can also be used for public campaign messages. <b>The back of the bus features an advertisement.</b> School bus advertising is a form of advertising in the United States in which advertising space is sold on the sides of school buses. ...</td>
  </tr>
  <tr>
    <td>What are the colors of the bus in the image?</td>
    <td>Red & White Services was a bus company operating in south east Wales and Gloucestershire, England between 1929 and 1978. Red & White evolved into Red & White United Transport Ltd, formed in 1937, which owned bus and road freight companies in the United Kingdom and Southern Africa. <b>The bus is white and red.</b> Red Jammers are the vintage White Motor Company/Bender Body Company Model 706 buses used at Glacier National Park in the United States ...</td>
  </tr>
  <tr>
    <td rowspan="2">
      <img src="page_184_576_370_180.png" alt="Two birds standing on a sandy beach">
    </td>
    <td>What type of surface are the birds standing on?</td>
    <td>The Sandy Island and Low Rock Important Bird Area comprises two small islands with a collective area of 9 lying at the western end of the Gulf of Carpentaria in the Northern Territory of Australia. They lie south-west of Groote Eylandt off the coast of Arnhem Land, with the nearest settlement there the Numbulwar community. <b>The birds are standing on a sandy beach.</b> Sandy Island is a 20 (ha) island lying about 5 (km) off the coast of Windy Harbour in south-west Western Australia, and near Point D' ...</td>
  </tr>
  <tr>
    <td>Are there any specific objects near the birds in the image?</td>
    <td>In agriculture, poultry litter or broiler litter is a mixture of poultry excreta, spilled feed, feathers, and material used as bedding in poultry operations. This term is also used to refer to unused bedding materials. <b>There are straws lying on the sand near the birds.</b> Bat-fowling is an archaic method of catching birds at night, while they are at roost. The process involves lighting straw or torches near their roost. After awakening them from their roost, the birds fly toward the flames, ...</td>
  </tr>
  <tr>
    <td rowspan="2">
      <img src="page_184_792_370_180.png" alt="A person playing tennis on a court">
    </td>
    <td>What is the man doing in the image?</td>
    <td>A passing shot is a forceful shot, as in tennis or team handball, that travels to one side out of the reach of one's opponent. In tennis, this shot is generally a groundstroke and is used when one's opponent is running to the net or if they are at the net already. ... <b>The man is playing tennis near the net and getting ready to hit a ball, he might have just made a play, and he is attempting to return the ball to continue the tennis match.</b><br> Gamesmanship is the use of dubious (although not technically illegal) methods to win or gain a serious advantage in a game or sport. ...</td>
  </tr>
  <tr>
    <td>Is there any official or umpire present in the image?</td>
    <td>A challenge is a request made to the holder of a competitive title for a match between champion and challenger, the winner of which will acquire or retain the title. ... <b>There is an official looking on</b> indicating that the tennis match is likely a formal or competitive one. The tennis scoring system is a standard widespread method for scoring tennis matches, including pick-up games. Some tennis matches are played as part of a tournament, which may have various categories, such as singles and doubles. The great majority are organised as a single-elimination tournament, with competitors being eliminated after a single loss, and the overall winner being the last competitor without a loss. A tournament is a competition involving at least three competitors, all participating in a sport or game. More specifically, the term may be used in ...</td>
  </tr>
</table>

Figure 8: Examples for our dataset. The figure illustrates samples in the dataset, where the red-colored text denotes inserted responses.

Reviewer paper summaries: The paper introduces MIRe, a novel multimodal retrieval framework designed to address the text-dominant issue, where excessive reliance on textual cues often leads to neglect of visual information while retrieval. Unlike traditional methods that fuse text and visual modalities early in the process, MIRe employs a fusion-free approach allowing textual queries to attend to visual embeddings without feeding text-driven signals back into visual representations. The authors also create a specialized pre-training dataset by transforming concise multimodal question-answer pairs into extended passages, enhancing the retrieval model's ability to comprehend multimodal queries. Experimental results indicate MIRe significantly outperforms existing models across multiple benchmarks in both zero-shot and fine-tuned settings. Ablation studies validate the importance of each component in MIRe, especially highlighting the effectiveness of its modality interaction strategy.
Summary of strengths: 1. MIRe exhibits strong performance in comparison to SOTA retrievers across a bunch of datasets in both zero and few shot setup.
2. The proposed dataset can be valuable in training multimodal retrievers. 
3. The ablation study shows how different components in data, model, and training recipes help achieve strong multimodal retrieval performance.
Summary of weaknesses: 1. Section 3.3 requires elaboration for a comprehensive understanding of the proposed method. In particular the calculation of attention scores, ($V_m \in \mathbb{R}^{l_v \times d_v}$ $\rightarrow$ $K_m \in \mathbb{R}^{h \times l_v \times d_t}$) require more explanation regarding the dimensions. Also, Do the authors employ only one layer of Query-guided Attention? 

2. Looking at the proposed architecture of the PreFLMR and MIRe, it seems like the only difference between the approaches is that MIRe gets rid of the residual connections in the cross-attention block of the PreFLMR. In that case what is the difference between PreFLMR and w-residual version of the MIRe? Also, can the authors present a more apple-to-apple comparison against the original PreFLMR (not retrained matching the setup of the authors)? PreFLMR uses a different set of pertaining datasets and has 2 stages of intermediate pre-training instead of 1 in the MIRe. What if the original PreFLMR can overcome the text dominance using their setup and training strategies? 

3. There are some inconsistencies in the dataset and model selections for various experiments. In zero-shot (ZS) and few-shot (FS) experiments, authors are using different sets of baselines (+CLIP and +PreFLMR for ZS; +VRR and +GeMKR for FS). A more consistent selection of baselines should have been more appropriate for better comparison. Also, once again makes me wonder why the authors chose to reproduce PreFLMR for zero-shot results instead of directly using the model weights, rather it would be interesting to see fine-tuned PreFLMR results since it is the strongest baseline in zero-shots? One more interesting finding from Tab 1 and 2 is, FLMR (slightly) and ReViz (largely) loses performance on the ReMuQ datasets, while not required, but an intuitive explanation of this might strengthen the case for MIRe. In Tab 4, the authors suddenly introduce the InfoSeek dataset for wit experiment, while this was not used elsewhere, any particular reason for using InfoSeek for this experiment? VL-ICT dataset was again abruptly introduced in section 5, without any proper explanation of what this dataset looks like and how it is used in the training?

4. In the dataset generation, how the passages are created does not sound natural and is something we would face in the wild. The generated passages can contain high amount of overlapping information from top retrieved passages and the placement of the actual response in the curated passage makes it even more unnatural. If there is any particular reason for generating such un-natural query-document pairs, that should be elaborated upon. Otherwise a better approach for constructing these passages would be to feed in the R and the Ds into some LLM and generate a cohesive passage that might mimic passages in the wild.
Comments, suggestions and typos: 1. The paper introduces an interesting case study of how existing retrievers fail to successfully retrieve quality results when the key terms are replaced by more generic words, (e.g. beverage $\rightarrow$ object), but do not follow up on this case study in the results and analysis section. `Text-dominant` issue is discussed on a very high level in section 5, where it is shown that using PreFLMR or w/residual connection leads to a faster convergence but lower performance than MIRe, and also excluding $E_t$ during alignment leads to better clustering of document and query embeddings. While all these are empirical evidence for text dominance problem but following up on the case-study, a small controlled study showing how replacing keywords cannot affect the performance of MIRe in comparison to other SOTA retrievers would be interesting.

2. How was CLIP used as a baseline for zero shot? Since, CLIP generates individual embeddings for text and image, how were they combined?

3. It would be interesting to see the comparison of the model sizes/parameters of MIRe and the baselines. 

4. The table 6 introduces different seed datasets for dataset generation. Additionally, a more comprehensive explanation of all the datasets including these ones and the evaluation datasets in the appendix would be very much helpful for the understanding of the readers.

5. I, as a reader, would love to see more retrieval examples for the MIRe and baselines in the appendix to get a qualitative idea of the model performances. Maybe for some type of questions the text-dominance is more acute than the others. Also, some negative results would provide immense insights into how the model is actually performing beyond numbers.
Overall assessment scores: 3.5
Metareview score: 3.5
Metareview: This paper introduces a retrieval framework MIRe that enables modality interaction without integrating textual features during the alignment process. Unlike traditional approaches, MIRe allows the textual query to attend to visual embeddings without feeding text-driven signals back into the visual representations. To support this, the authors also construct a pre-training dataset for multimodal query retrieval by converting concise question-answer pairs into extended passages. Experimental results show that the pre-training strategy substantially improves the understanding of multimodal queries, achieving strong performance on four multimodal retrieval benchmarks in zero-shot settings.
Summary of reasons to publish: Strength of this work consistently identified by the reviewers includes:
- Problem importance: The text-dominant issue is indeed a widespread issue, and addressing it would more effectively reduce the semantic loss of visual information caused by visual-text alignment. 
- Techniques contribution: The authors proposed a Fusion-Free Modality Interaction method which is conceptual novelty.
- Dataset: The proposed dataset can be valuable in training multimodal retrievers.
- Evaluation Results: Strong results and comprehensive evaluation on 4 datasets with baseline models, including three widely used metrics. And the authors provided detailed analysis and ablation studies.