Enhancing Multimodal Retrieval via Complementary Information Extraction and Alignment
Authors: Delong Zeng, Yuexiang Xie, Yaliang Li, Ying Shen
Accepted at: acl25

Enhancing Multimodal Retrieval via Complementary Information Extraction and Alignment

Delong Zeng1,* Yuexiang Xie2 Yaliang Li2 Ying Shen1,3,†
1School of Intelligent Systems Engineering, Sun Yat-sen University 2Alibaba Group
3Guangdong Provincial Key Laboratory of Fire Science and Intelligent Emergency Technology
zengdlong@mail2.sysu.edu.cn, {yuexiang.xyx, yaliang.li}@alibaba-inc.com
sheny76@mail.sysu.edu.cn

Abstract

Multimodal retrieval has emerged as a promising yet challenging research direction in recent years. Most existing studies in multimodal retrieval focus on capturing information in multimodal data that is similar to their paired texts, but often ignores the complementary information contained in multimodal data. In this study, we propose CIEA, a novel multimodal retrieval approach that employs Complementary Information Extraction and Alignment, which transforms both text and images in documents into a unified latent space and features a complementary information extractor designed to identify and preserve differences in the image representations. We optimize CIEA using two complementary contrastive losses to ensure semantic integrity and effectively capture the complementary information contained in images. Extensive experiments demonstrate the effectiveness of CIEA, which achieves significant improvements over both divide-and-conquer models and universal dense retrieval models. We provide an ablation study, further discussions, and case studies to highlight the advancements achieved by CIEA. To promote further research in the community, we have released the source code at https://github.com/zengdlong/CIEA.

1 Introduction

Retrieval-Augmented Generation (RAG) (Lewis et al., 2020b; Gao et al., 2023; Yi et al., 2024) has recently attracted widespread attention for its role in enhancing large language models (LLMs) (Touvron et al., 2023; Bai et al., 2023; Chung et al., 2024; Brown et al., 2020) by providing up-to-date information and alleviating hallucination issues. Most existing studies focus on retrieving textual information (Cheng et al., 2023; Shi et al., 2024; Wang et al., 2024a; Zhu et al., 2025), leveraging text corpora to provide factual support for model responses. As the scale of multimodal data continues to grow, effectively supporting multimodal retrieval to provide knowledge beyond text has emerged as a promising yet challenging research area (Zhao et al., 2023; Kuang et al., 2025).

To promote the progress of multimodal retrieval, researchers have employed captioning models to convert multimodal data into text (Wu et al., 2023; Baldrati et al., 2023; Mahmoud et al., 2024), enabling the application of existing retrieval techniques developed for text corpora. However, these approaches can be heavily influenced by the effectiveness of the captioning models, which might result in the loss of crucial information during the conversion process (Che et al., 2023; Li et al., 2024b). Recent studies propose representation-based approaches, wherein textual and multimodal information are mapped into a unified embedding space for knowledge retrieval, after a separate encoding process (Radford et al., 2021; Zhang et al., 2021) or a project-based joint encoding process (Zhou et al., 2024; Li et al., 2024a).

Q: Is the Rainbow Bridge always lit at night with the same colors?
Gloden image documents: √
Caption: Rainbow Bridge, Odaiba, Tokyo - Sony A7R [1811229514]
Caption: Tokyo Bay and Rainbow bridge from Odaiba
Confused image documents: ×
Caption: Bridge,Tokyo Bay from Odaiba Rainbow Bridge [Tokyo] and Northern Tokyo Bay
Caption: Rainbow Bridge 3 Statue of Liberty of Odaiba &amp; Rainbow Bridge,

Figure 1: An example of multimodal retrieval. The images contain critical information that can assist in query responses, which might not be present in the text.
In this study, we focus on a novel representation-based approach that aims to transfer multimodal data into the latent vector space of language models. We identify a critical limitation of existing representation-based methods (Wang et al., 2024b; Zhou et al., 2024): they tend to focus on capturing the information in multimodal data that is similar to their textual counterparts (such as captions or associated text) while neglecting the complementary information in multimodal data. For example, as shown in Figure 1, a textual description paired with an image might reference the Rainbow Bridge. While the image can indeed provide similar visual details of Rainbow Bridge, supplementary information such as the nightscape and the color of the bridge can be also important. For the representation-based approach, extracting and preserving such complementary information in multimodal data can enhance the quality of the responses to queries that cannot be fully resolved using textual information alone.

Inspired and motivated by the above insights, we propose a novel multimodal retrieval approach that involves Complementary Information Extraction and Alignment, denoted as CIEA. Specifically, we adopt a language model to transform text into a latent space, while employing the CLIP model (Radford et al., 2021; Li et al., 2023) and a projector to map image information into the same unified latent space. Then, we design a complementary information extractor, which identifies the differences in representations between the images and the text in the documents. Based on these differences, we update the representations of the images to integrate complementary information. Furthermore, we introduce a novel optimization method tailored for CIEA by constructing two complementary contrastive loss functions: one that ensures the semantics integrity of the learned representations, and another that enhances the extraction of complementary information from the images.

Extensive experiments are conducted to demonstrate the effectiveness of CIEA in multimodal retrieval. The experimental results show that the proposed method achieves noticeable improvements compared to both divide-and-conquer models and universal dense retrieval models. An ablation study is carried out to highlight the contributions of the different components of CIEA. Besides, we provide further discussions on the effect of the language model and include some case studies for a better understanding of CIEA.

2 Related Work

Conventional retrieval models (Lewis et al., 2020b; Yu et al., 2023) encode queries and documents into vectors via language models (Chen et al., 2024), trained with contrastive learning and retrieved using KNN (Su et al., 2022). Multimodal retrieval, compared to text-only retrieval, incorporates multiple modalities of information, necessitating effective utilization strategies for information from different modalities (Yuan et al., 2021). One approach involves using caption models to convert information from other modalities into text, effectively converting multimodal retrieval tasks into text-only retrieval tasks (Baldrati et al., 2023). However, such an approach may lead to information loss (Che et al., 2023; Li et al., 2024b).

Another approach is to employ representation models, where visual and textual encoders encode information separately before fusion (Radford et al., 2021). Early studies adopt a divide-and-conquer strategy, encoding each modality separately and concatenating vectors to fuse information, which potentially causes modality competition. To tackle this, UniVL-DR (Liu et al., 2023b) proposes a universal multimodal retrieval framework by encoding queries and documents into a unified embedding space for retrieval, routing, and fusion. Some existing studies (Wang et al., 2022a,b) involve training large models to unify text and visuals. However, differing representations complicate the acquisition of sufficient data for effective semantic understanding (Lu et al., 2023).

Recent studies (Zhou et al., 2024; Wang et al., 2024b; Li et al., 2024a, 2023) propose a project-based framework, which leverages models like CLIP (Radford et al., 2021) to convert visual inputs into feature sequences and introduce projector layers to align these sequences with language model embeddings. Such a framework facilitates the comparison of visual and textual content at the embedding level and converts visual information into language model “tokens”, capitalizing on knowledge infused during the training of language models. For example, MARVEL (Zhou et al., 2024) utilizes the project-based framework for capturing multimodal information within the output space of the language model; MCL (Li et al., 2024a) adds a retrieval token for enhancing the model’s performance in representation learning.

Although remarkable progress has been made, visual information remains a novel and underex-
A. Project Based Module

Caption: Madison square garden Madison Square Garden in New York.

B. Representations Of Documents

C. Optimization

Query comp: In <mask>, what movie was advertised on a poster outside in October 2003?
Caption: Madison square garden Madison Square Garden in New York.
Query: In Madison Square Garden in New York, what movie was advertised on a poster outside in October 2003?

Figure 2: The overall architecture of CIEA. The upper part illustrates how queries and multimodal documents are transformed into a unified latent space, while the lower part details the optimization process of CIEA.

plored component for language models, which motivates us to provide effective solutions for processing visual information to attain a comprehensive representation for enhancing multimodal retrieval.

3 Methodology

3.1 Preliminary
In this study, we focus on multimodal retrieval, which aims to find one or more relevant documents from a knowledge base in response to a given text query \( q \). A knowledge base consisting of a total of \( N \) documents can be denoted as \( D = \{d_1, d_2, \ldots, d_N\} \). Each document might contain text and images, i.e., \( d_m = \{t_m, i_m\} \), where \( t_m \) and \( i_m \) represent the text and images in the document, respectively.

The main objective of multimodal retrieval is to align the representations of the query with the corresponding multimodal documents, ensuring that similar items are closely matched in a unified latent space. To achieve this, a multimodal encoder is employed to encode both the query and the documents, transforming them into dense representations, which can be given as:

\[
\mathbf{q} = \mathrm{Encoder}(q),
\]
\[
\mathbf{d}_m = \mathrm{Encoder}(d_m), \forall d_m \in D.
\] (1)

After that, the similarities between the query and documents are measured via cosine similarity:

\[
\cos(\mathbf{q}, \mathbf{d}_m) = \frac{\mathbf{q} \cdot \mathbf{d}_m}{\|\mathbf{q}\| \|\mathbf{d}_m\|}.
\] (2)

In the following subsections, we provide a detailed introduction of the proposed Complementary Information Extraction Alignment, denoted as CIEA. The overall architecture of CIEA is illustrated in Figure 2. Specifically, we first encode the queries and multimodal documents via a language model and a CLIP model, as shown in Section 3.2. Then, in Section 3.3, we design a complementary information extractor to capture the complementary information contained in the images. The optimization designed for the proposed CIEA is introduced in Section 3.4.

3.2 Representations of Queries and Multimodal Documents
Firstly, as shown in part A of Figure 2, we transform both the queries and the multimodal documents into their dense representations. Inspired by previous studies (Zhou et al., 2024; Li et al., 2024a, 2023), we utilize a transformer-based language model as the backbone of the encoder to encode both text and images. A transformer-based
language model typically consists of an embedding layer followed by several transformer blocks. We denote the embedding layer as Emb(\cdot) and the transformer blocks as Trans(\cdot).

The text query \( q \) can be transformed into dense representations via language models, which can be given as:
\[
\mathbf{q} = \mathrm{Trans}(\mathrm{Emb}(q)),
\]
where \( \mathbf{q} \in \mathbb{R}^{l_q \times d} \), \( l_q \) denotes the number of tokens in the query, and \( d \) denotes the dimension of the dense representations.

For a document containing both text and images, i.e., \( d_m = \{ t_m, i_m \} \), the text \( t_m \) and image \( i_k \) are processed separately before being fused. Specifically, we begin by feeding the text within the document to the embedding layer:
\[
\mathbf{t}_m = \mathrm{Emb}(t_m),
\]
where \( \mathbf{t}_m = [t_m^{(1)}, t_m^{(2)}, \ldots, t_m^{(l_t)}] \in \mathbb{R}^{l_t \times d} \) and \( l_t \) denotes the number of token in the text.

The images in the documents can be typically represented as RGB multi-channel matrices. We adopt a frozen CLIP (Radford et al., 2021) visual encoder to transform these matrices into a set of semantic vectors. Such a process involves dividing an image into multiple patches, with each patch representing a different region of the image, which can be formally given as:
\[
\mathbf{i}_m = \mathrm{CLIP}_{\mathrm{Visual}}(i_m),
\]
where \( \mathbf{i}_m \in \mathbb{R}^{l_i \times d_{\mathrm{clip}}} \), \( l_i \) denotes the number of patches, and \( d_{\mathrm{clip}} \) denotes the hidden dimension of CLIP visual encoder.

Note that the hidden dimension of these image representations might be different from that of text representations, i.e., \( d_{\mathrm{clip}} \neq d \). Therefore we adopt a linear layer as the projector for further alignment:
\[
\widetilde{\mathbf{i}}_m = \mathrm{Proj}(\mathbf{i}_m),
\]
where \( \widetilde{\mathbf{i}}_m = [\widetilde{\mathbf{i}}_m^{(1)}, \widetilde{\mathbf{i}}_m^{(2)}, \ldots, \widetilde{\mathbf{i}}_m^{(l_i)}] \in \mathbb{R}^{l_i \times d} \).

3.3 Complementary Information Extractor

The proposed CIEA aims to capture the complementary information contained in images, i.e., the information not encompassed by the text in a document. To achieve this, we design a complementary information extractor that captures the differences between images and textual content. Specifically, we calculate the patch-level distances between text and image representations, identifying the maximum value to serve as a difference measurement, which is similar to those adopted in BLIP-2 (Li et al., 2023). Formally, the difference between the \( j \)-th patch of image and the text within document \( m \), denoted as \( r_{m,j} \), can be calculated as:
\[
r_{m,j} = -\max_{c \in [l_t]} \cos(\widetilde{\mathbf{i}}_m^{(j)}, \mathbf{t}_m^{(c)}).
\]
The obtained differences are normalized into a range of [0, 1] for effectively weighing the image patches, as given by:
\[
w_{m,j} = \frac{1 + r_{m,j}}{2}.
\]
We adopt an attention layer for re-weighting the patches of the image based on the calculated \( w_m = w_{m,j} \forall j \in [l_i] \). The attention scores of embeddings are multiplied by the corresponding weights for normalization, which can be given as:
\[
\mathbf{Q}_m = \widetilde{\mathbf{i}}_m \mathbf{W}_Q, \quad \mathbf{K}_m = \widetilde{\mathbf{i}}_m \mathbf{W}_K, \quad \mathbf{V}_m = \widetilde{\mathbf{i}}_m \mathbf{W}_V,
\]
\[
\widehat{\mathbf{i}}_m = \mathrm{softmax}(\frac{\mathbf{Q}_m \mathbf{K}_m^T \cdot \mathbf{w}_m}{\sqrt{d}})\mathbf{V}_m,
\]
where \( \mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V \) are learnable matrices and "." here stands for the broadcasting method.

After that, as shown in part B of Figure 2, the embeddings of image and text are concatenated together and fed into transformer blocks for multimodal knowledge fusion. The obtained dense representation of the document \( d_m \) can be given as:
\[
\mathbf{d}_m = \mathrm{Trans}([\mathbf{e}_{\mathrm{start}} \oplus \widehat{\mathbf{i}}_m \oplus \mathbf{e}_{\mathrm{end}} \oplus \mathbf{t}_m]),
\]
where \( \oplus \) denotes the concatenate operation, and \( \mathbf{e}_{\mathrm{start}} \) and \( \mathbf{e}_{\mathrm{end}} \) denote the embeddings of special tokens <start> and <end>, respectively, for distinguishing the text and image embeddings explicitly.

Finally, for a given query, we rank the documents based on the similarities between their representations (refer to Eq.(10)) and the query representation (refer to Eq.(3)) to obtain the relevant documents.

3.4 Optimization

The trainable parameters of CIEA framework include the projector (as defined in Eq. (6)), the added attention weights \( \mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V \) for re-weighting (as defined in Eq. (9)), and the transformer layers of the language models (as defined in Eq. (10)). In this section, we introduce how to optimize these trainable parameters in CIEA.
<table>
  <tr>
    <th></th>
    <th>mrr@10</th>
    <th>ndcg@10</th>
    <th>mrr@20</th>
    <th>ndcg@20</th>
    <th>rec@20</th>
    <th>rec@100</th>
  </tr>
  <tr>
    <td>BM25</td>
    <td>22.11</td>
    <td>22.92</td>
    <td>22.80</td>
    <td>25.41</td>
    <td>46.27</td>
    <td>62.82</td>
  </tr>
  <tr>
    <td>CLIP-DPR</td>
    <td>37.35</td>
    <td>37.56</td>
    <td>37.93</td>
    <td>40.77</td>
    <td>69.38</td>
    <td>85.53</td>
  </tr>
  <tr>
    <td>BM25 & CLIP-DPR</td>
    <td>42.27</td>
    <td>41.58</td>
    <td>42.79</td>
    <td>44.69</td>
    <td>73.34</td>
    <td>87.50</td>
  </tr>
  <tr>
    <td>BM25 & CLIP (oracle)</td>
    <td>61.05</td>
    <td>58.18</td>
    <td>61.37</td>
    <td>60.45</td>
    <td>80.82</td>
    <td>90.83</td>
  </tr>
  <tr>
    <td>VinVL-DPR</td>
    <td>38.14</td>
    <td>35.43</td>
    <td>38.74</td>
    <td>37.79</td>
    <td>53.89</td>
    <td>69.42</td>
  </tr>
  <tr>
    <td>CLIP-DPR</td>
    <td>48.83</td>
    <td>46.32</td>
    <td>49.34</td>
    <td>49.11</td>
    <td>69.84</td>
    <td>86.43</td>
  </tr>
  <tr>
    <td>UniVL-DR</td>
    <td>62.40</td>
    <td>59.32</td>
    <td>62.96</td>
    <td>61.22</td>
    <td>80.37</td>
    <td>89.42</td>
  </tr>
  <tr>
    <td>T5-ANCE</td>
    <td>64.13</td>
    <td>62.03</td>
    <td>64.40</td>
    <td>64.02</td>
    <td>83.81</td>
    <td>92.07</td>
  </tr>
  <tr>
    <td>MARVEL</td>
    <td>65.43</td>
    <td>63.07</td>
    <td>65.67</td>
    <td>65.00</td>
    <td>84.32</td>
    <td>92.27</td>
  </tr>
  <tr>
    <th>CIEA (ours)</th>
    <th><b>66.16</b></th>
    <th><b>63.89</b></th>
    <th><b>66.41</b></th>
    <th><b>65.85</b></th>
    <th><b>85.43</b></th>
    <th><b>92.75</b></th>
  </tr>
</table>

Table 1: Comparisons on the WebQA-Multi dataset. The best results are marked with bold.

Firstly, we identify the text segments from the queries in the training set that correspond to the text in the ground truth documents. For example, as shown in Figure 2, we mask out “Madison Square Garden in New York” that appears in the caption. We match the token IDs generated by the tokenizer of the language model for the query and the text in document for such identification.

After that, we replace these identified text segments with special tokens <mask> while preserving the remaining parts of the query. We denote the processed query as \( q_{comp} \), whose representations can be obtained in a similar manner to the computation defined in Eq. (3):

\[
q_{comp} = \mathrm{Trans}(\mathrm{Emb}(q_{comp})).
\]

We also construct negative samples for the queries to apply a contrastive loss for optimization. The documents annotated as relevant to the query are denoted as \( D^+ \), while \( N \) negative documents are sampled from the knowledge base and denoted as \( D^- = \{d_1^-, d_2^-, \ldots, d_N^-\} \). The contrastive loss can be defined as:

\[
\mathcal{L}_c = - \log \frac{e^{\cos(q, d^+)/\tau}}{e^{\cos(q, d^+)/\tau} + \sum_{D^-} e^{\cos(q, d^-)/\tau}},
\]

where \( \tau \) denotes the temperature, \( \cos(\cdot) \) denotes the cosine similarity function, \( q \) denotes the representation of the query, \( d^+ \) and \( d^- \) denote the representations of positive documents and negative documents, respectively.

Meanwhile, to enhance the extraction of complementary information from images within the documents, we construct another contrastive loss function based on \( q_{comp} \) and the representations of images, which can be given as:

\[
\mathcal{L}_{comp} = - \log \frac{e^{\cos(q_{comp}, d_{img}^+)/\tau}}{e^{\cos(q_{comp}, d_{img}^+)/\tau} + \sum_{D^-} e^{\cos(q_{comp}, d_{img}^-)/\tau}},
\]

where \( d_{img} = \mathrm{Trans}(\hat{i}_k) \) denotes the representations of the images in the documents. The intuition behind the above loss function is that we promote the representations of images to align with the information contained in the query that is not present in the text within documents.

Finally, the training objective of the proposed CIEA can be given as:

\[
\mathcal{L} = \mathcal{L}_c + \lambda \cdot \mathcal{L}_{comp},
\]

where \( \lambda \) is a hyperparameter that balances the contributions of the two contrastive losses.

4 Experiments

4.1 Settings

Datasets We conduct extensive experiments on the WebQA (Chang et al., 2022) and EDIS (Liu et al., 2023a) dataset to compare the proposed method against baselines. WebQA is an open-domain multimodal question answering dataset in which each query is associated with one or more relevant documents. There are 787,697 documents containing only textual information, while 389,750 documents contain both text and images. For clarity, we refer to this dataset as WebQA-Multi in this study. Besides, to better show the effectiveness of multimodal retrieval, we extract all documents containing images and construct the WebQA-Image
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="4">WebQa-Image</th>
    <th colspan="4">EDIS</th>
  </tr>
  <tr>
    <th>mrr@10</th>
    <th>ndcg@10</th>
    <th>mrr@20</th>
    <th>ndcg@20</th>
    <th>mrr@10</th>
    <th>ndcg@10</th>
    <th>mrr@20</th>
    <th>ndcg@20</th>
  </tr>
  <tr>
    <td>CLIP-DPR</td>
    <td>59.78</td>
    <td>61.05</td>
    <td>60.2</td>
    <td>63.28</td>
    <td>62.52</td>
    <td>39.19</td>
    <td>62.90</td>
    <td>44.24</td>
  </tr>
  <tr>
    <td>UniVL-DR</td>
    <td>65.95</td>
    <td>67.33</td>
    <td>66.24</td>
    <td>69.01</td>
    <td>62.89</td>
    <td>39.50</td>
    <td>63.29</td>
    <td>44.54</td>
  </tr>
  <tr>
    <td>T5-ANCE</td>
    <td>64.38</td>
    <td>65.65</td>
    <td>64.73</td>
    <td>67.52</td>
    <td>63.95</td>
    <td>39.74</td>
    <td>64.30</td>
    <td>43.76</td>
  </tr>
  <tr>
    <td>MARVEL-DPR</td>
    <td>62.51</td>
    <td>63.48</td>
    <td>62.90</td>
    <td>65.54</td>
    <td>63.60</td>
    <td>37.76</td>
    <td>63.98</td>
    <td>41.89</td>
  </tr>
  <tr>
    <td>MARVEL</td>
    <td>66.43</td>
    <td>67.60</td>
    <td>66.76</td>
    <td>69.40</td>
    <td>67.00</td>
    <td>42.19</td>
    <td>67.28</td>
    <td>46.63</td>
  </tr>
  <tr>
    <td>CIEA (ours)</td>
    <td><b>67.40</b></td>
    <td><b>68.77</b></td>
    <td><b>67.74</b></td>
    <td><b>70.51</b></td>
    <td><b>68.11</b></td>
    <td><b>42.57</b></td>
    <td><b>68.42</b></td>
    <td><b>47.19</b></td>
  </tr>
</table>

Table 2: Comparisons on the WebQa-Image and EDIS. The best results are marked with bold.

dataset. EDIS retrieves images via textual queries, with 26,000 training, 3,200 development, and 3,200 test samples from 1M image corpus (each paired with a text caption). More statistics of datasets can be found in Appendix A.

Metrics To evaluate the effectiveness of the proposed method and baselines in multimodal retrieval, we adopt six widely-used metrics, including MRR@10, MRR@20, NDCG@10, NDCG@20, REC@20, and REC@100. The MRR (Bajaj et al., 2016) and NDCG (Taylor et al., 2008) metrics are calculated using the official code1.

Implementation Details In the experiments, we employ T5-ANCE (Yu et al., 2023) as the foundational language model and CLIP as the visual understanding module to implement the proposed CIEA model to enhance the image understanding capabilities. The document representation capabilities of T5-ANCE are enhanced by using ANCE (Xiong et al., 2021) to sample negative documents based on T5 (Raffel et al., 2020). We truncate the text input length to 128 tokens. During training, we adopt the AdamW (Loshchilov and Hutter, 2019) optimizer with a maximum of 40 epochs, a batch size of 64, a learning rate of 5e-6, and set the temperature hyperparameter \( \tau \) to 0.01. The hyperparameter \( \lambda \) is set to 0.0011 for WebQA-Multi, 0.019 for WebQA-Image, and 0.001 for EDIS, respectively. More implementation details can be found in Appendix B.

4.2 Baselines

We compare CIEA with two different types of baselines: divide-and-conquer models and universal dense retrieval models.

Divide-and-conquer models suggest retrieving image and text documents separately and then merging the retrieval results. We employ various single-modal retrievers to instantiate the divide-and-conquer models, including UniVL-DPR (Zhang et al., 2021), CLIP-DPR (Radford et al., 2021), and BM25 (Robertson et al., 2009). The results of multi-modality retrieval are combined based on their uni-modal rank reciprocals or oracle modality routing. The latter approach demonstrates the maximum potential performance of our divide-and-conquer models in retrieval tasks.

Regarding universal dense retrieval models, we adopt the following recent studies as baselines: (i) Pre-trained multimodal alignment models VinVL-DPR (Zhang et al., 2021) and CLIP-DPR (Radford et al., 2021), which are trained using the DPR framework; (ii) UniVL-DR (Liu et al., 2023b), which employs modality-balanced hard negatives to train text and image encoders, and uses image language adaptation techniques to bridge the modality gap between images and text; (iii) MARVEL (Zhou et al., 2024), a project-based approach that encodes image documents with CLIP and projects them into the embedding layer of a language model for fusion.

Comparisons The experimental results on WebQA and EDIS are shown in Table 1 and 2, respectively. From these results we can observe that, the proposed CIEA achieves noticeable improvements across various metrics compared to both divide-and-conquer models and universal dense retrieval models. CIEA achieves outperformance on both WebQA and EDIS, demonstrating its effectiveness in capturing crucial information from both text and images for enhancing multimodal retrieval. As shown in Tables 1, compared to universal dense retrieval models that employ projectors for language model grounding (e.g., MARVEL, which shows improvement over text-only methods with a 1.3 MRR@10 improvement over T5-ANCE), CIEA
<table>
  <tr>
    <th>Language model</th>
    <th>setting</th>
    <th>mrr@10</th>
    <th>ndcg@10</th>
    <th>mrr@20</th>
    <th>ndcg@20</th>
    <th>rec@20</th>
    <th>rec@100</th>
  </tr>
  <tr>
    <td rowspan="3">T5-ANCE</td>
    <td>Text only</td>
    <td>64.13</td>
    <td>62.03</td>
    <td>64.40</td>
    <td>64.02</td>
    <td>83.81</td>
    <td>92.07</td>
  </tr>
  <tr>
    <td>Project-based</td>
    <td>65.43</td>
    <td>63.07</td>
    <td>65.67</td>
    <td>65.00</td>
    <td>84.32</td>
    <td>92.27</td>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td><b>66.16</b></td>
    <td><b>63.89</b></td>
    <td><b>66.41</b></td>
    <td><b>65.85</b></td>
    <td><b>85.43</b></td>
    <td><b>92.75</b></td>
  </tr>
  <tr>
    <td rowspan="3">BERT</td>
    <td>Text only</td>
    <td>61.39</td>
    <td>58.20</td>
    <td>61.69</td>
    <td>60.32</td>
    <td>79.59</td>
    <td>89.10</td>
  </tr>
  <tr>
    <td>Project-based</td>
    <td>63.03</td>
    <td>60.20</td>
    <td>63.31</td>
    <td>62.37</td>
    <td><b>81.80</b></td>
    <td><b>90.78</b></td>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td><b>63.63</b></td>
    <td><b>60.46</b></td>
    <td><b>63.90</b></td>
    <td><b>62.49</b></td>
    <td>81.51</td>
    <td>90.57</td>
  </tr>
  <tr>
    <td rowspan="3">BART</td>
    <td>Text only</td>
    <td>60.14</td>
    <td>56.82</td>
    <td>60.49</td>
    <td>59.14</td>
    <td>78.41</td>
    <td>89.25</td>
  </tr>
  <tr>
    <td>Project-based</td>
    <td>62.67</td>
    <td>59.70</td>
    <td>62.98</td>
    <td>62.04</td>
    <td>81.77</td>
    <td>90.77</td>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td><b>63.73</b></td>
    <td><b>60.43</b></td>
    <td><b>64.04</b></td>
    <td><b>62.72</b></td>
    <td><b>81.96</b></td>
    <td><b>90.98</b></td>
  </tr>
  <tr>
    <td rowspan="3">GPT2</td>
    <td>Text only</td>
    <td>54.68</td>
    <td>51.49</td>
    <td>55.07</td>
    <td>53.77</td>
    <td>72.91</td>
    <td>84.86</td>
  </tr>
  <tr>
    <td>Project-based</td>
    <td>58.59</td>
    <td>55.00</td>
    <td>58.92</td>
    <td>57.15</td>
    <td>75.84</td>
    <td>86.79</td>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td><b>59.25</b></td>
    <td><b>55.35</b></td>
    <td><b>59.59</b></td>
    <td><b>57.62</b></td>
    <td><b>76.47</b></td>
    <td><b>86.92</b></td>
  </tr>
  <tr>
    <td rowspan="3">GPT2-LARGE</td>
    <td>Text only</td>
    <td>63.58</td>
    <td>60.44</td>
    <td>63.86</td>
    <td>62.45</td>
    <td>81.24</td>
    <td>89.98</td>
  </tr>
  <tr>
    <td>Project-based</td>
    <td>64.33</td>
    <td>61.98</td>
    <td>64.60</td>
    <td>62.04</td>
    <td>83.97</td>
    <td>92.11</td>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td><b>65.38</b></td>
    <td><b>63.00</b></td>
    <td><b>65.62</b></td>
    <td><b>64.96</b></td>
    <td><b>84.96</b></td>
    <td><b>92.68</b></td>
  </tr>
</table>

Table 3: Experiments results when using different language models. The Text only setting denotes we only use the document’s captions for training without incorporating image information, while the Project-based setting is the most common projection method without visual alignment.

<table>
  <tr>
    <th></th>
    <th>mrr@10</th>
    <th>ndcg@10</th>
    <th>rec@20</th>
    <th>rec@100</th>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td><b>66.03</b></td>
    <td><b>63.70</b></td>
    <td><b>85.14</b></td>
    <td><b>92.63</b></td>
  </tr>
  <tr>
    <td>w/o image query</td>
    <td>65.66</td>
    <td>63.44</td>
    <td>84.74</td>
    <td>92.54</td>
  </tr>
  <tr>
    <td>w/o attention</td>
    <td>65.90</td>
    <td>63.39</td>
    <td>84.95</td>
    <td>92.60</td>
  </tr>
  <tr>
    <td>Base</td>
    <td>65.30</td>
    <td>63.19</td>
    <td>84.49</td>
    <td>92.40</td>
  </tr>
</table>

Table 4: Ablation study on WebQa-multi. Here, "w/o" stands for "without," and "Base" refers to the setting where both modules are removed.

achieves a further 0.74 MRR@10 improvement, with analogous enhancements replicated in Table 2. These CIEA-driven advancements underscore the critical benefits of complementary information alignment - a framework that enables superior visual information modeling and enhances the representational power of project-based models.

Ablation Study We conduct an ablation study to demonstrate the contributions of the image query and the attention layer. Specifically, we remove the attention layer designed for re-weighting (refer to Eq. (9)) and the utilization of image query (refer to Eq. (13)), respectively. We also perform base setting by removing these two components.

The experimental results are shown in Table 4. Although the removal of the image query does not lead to a decrease in rec@100, other metrics exhibit varying degrees of decline, particularly rec@20, which drops from 85.14 to 84.74. On the other hand, the removal of the attention mechanism results in larger significant decreases in ndcg@10 metrics, highlighting the importance of complementary information extraction. Further, we also compare the baseline configuration with the simultaneous removal of both modules. The results indicate that removing a single module yields improvements in various metrics compared to the baseline, demonstrating that both the image query and the attention layer contribute to enhancing retrieval accuracy in multimodal retrieval. More results can be found in Appendix C.

4.3 Further Discussions

The Effects of Language Models in CIEA To provide further discussions regarding the effects of language models used in the proposed CIEA, we conduct experiments with different language models as the backbone, including T5-ANCE (Yu et al., 2023), BART (Lewis et al., 2020a), BERT (Devlin et al., 2019), GPT-2, and GPT-2-LARGE (Radford et al., 2019). These language models cover three mainstream architectures, i.e., encoder-decoder, encoder-only, and decoder-only. More implementation details can be found in Appendix D.

The experimental results in Table 3 demonstrate the significant impacts of the backbone language model on multimodal retrieval effectiveness. For text-only retrieval, language models with larger parameter sizes, such as GPT-2 LARGE
<table>
  <tr>
    <th>Weights</th>
    <th>MRR@10</th>
    <th>NDCG@10</th>
    <th>REC@20</th>
    <th>REC@100</th>
  </tr>
  <tr>
    <td>Dissimilar</td>
    <td><b>66.03±0.012</b></td>
    <td><b>63.70 ± 0.030</b></td>
    <td><b>85.14±0.052</b></td>
    <td><b>92.63±0.008</b></td>
  </tr>
  <tr>
    <td>Similar</td>
    <td>65.60±0.038</td>
    <td>63.35±0.005</td>
    <td>85.00±0.012</td>
    <td>92.61±0.003</td>
  </tr>
</table>

Table 5: Experimental results with different weights on WebQA-Multi.

and T5-ANCE, achieve better overall performance. Both project-based models and CIEA benefit from powerful backbones, showing performance gains through image-derived information. Notably, CIEA maintains competitive results across different backbones compared to project-based approaches, confirming its robustness.

Attention Weight Selection The application of maximum cosine similarity for computing patch-level similarities is designed to extract image regions exhibiting lower correspondence with textual descriptions. To determine the optimal extraction approach, we perform empirical experiments that retained original cosine values rather than their complements. As demonstrated in Table 5, explicit dissimilarity computation has proven more effective for capturing fine-grained patch-level image-text mappings.

Computational Efficiency Although the proposed dual losses might increase training complexity, the computation of the losses is relatively independent and would not lead to a multiplicative increase in overall complexity. The proposed method achieves similar computational efficiency compared to MARVEL, which is also the project-based method. For example, with the sample devices, the training on the WebQA-Multid dataset with 4,966 samples with MARVEL needs around 6.1 minutes while that of CIEA is around 6.5 minutes.

Case Study We conduct case studies on the WebQA dataset for better understanding. We apply cosine similarity to identify the words from the language model’s vocabulary that are closest to the embeddings projected from images, showing what information is extracted from images. We compare the proposed CIEA and the strongest baseline MARVEL. The results are illustrated in Figure 3, which indicate that MARVEL focuses on image information that is close to the text, while CIEA captures more supplemental information from images. For example, CIEA captures some terms, such as green, leaf, and clock, that are semantically related to the image but are not mentioned in the caption. The term clock in the second case and flowers in the third case are relevant to the query, which can assist the model in efficiently locating and retrieving relevant content. These results further confirm the advances of CIEA for multimodal retrieval. Discussions regarding the failure cases are provided in Appendix E, which further illustrate the model’s behavior in challenging scenarios and inspire future improvements.

5 Conclusion

In this paper, we propose CIEA to enhance the effectiveness of multimodal retrieval. The main idea of CIEA is to enhance the capturing of complementary information in multimodal data. Specifically, CIEA utilizes language models and CLIP to transform multimodal documents into a unified latent space. For complementary information extraction, we calculate the patch-level distances between text and images, which are then used to re-weight the image representations. Regarding the optimization of CIEA, besides applying a contrastive loss for learning the semantics of text, we also encourage the alignment of image representations with the complementary information in queries. We conduct a series of experiments to demonstrate the advantages of CIEA compared to two different types of multimodal retrieval approaches. Further discussions on the effect of language models show the robustness of CIEA, and several case studies are included for better understanding.

Limitations

Multimodal retrieval is currently in a phase of rapid development. In this paper, our exploration is limited to queries that contain only text, with a focus on extracting information from images. Expanding the proposed approach to include more types of multimodal data, such as audio and video, is a promising direction for future research. Besides, although we conduct experiments with various language models, we are constrained by computational resources and have not yet explored larger-sized models.
Caption: Rainbow bridge, Odaiba, Tokyo - Sony A7R (11812224916)
Query: Are the flowers of the American Wintergreen or the Stinking willie bigger?
MARVEL nearest token: ['wort', 'seeds', 'pseudo', 'bacteria', 'behalf', 'berries', 'bacter', 'bacterial', 'alga', 'indigenous', 'rum']
CIEA nearest token: ['wort', 'green', 'seeds', 'it', 'Seed', 'pseudo', 'behalf', 'leaf', 'buds', 'flower', 'cum', 'berries']

Caption: Durgin Park 2009 Photograph of the entrance to Durgin Park, one of the oldest restaurants in Boston, located near Faneuil Hall.
Query: What color is the clock located directly outside of Durgin Park in Boston?
MARVEL nearest token: ['brewery', 'the', 'Lake', 'bou', 'stau', 'restaurant', 'restaur', 'bakery', 'urmätareale', 'residents', 'store', 'dairy', 'Eröffnung', 'Historical', 'situatud']
CIEA nearest token: ['brewery', 'bou', 'stau', 'restaurant', 'clock', 'bakery', 'Uhr', 'locaux', 'operates', 'locuri', 'yolk', 'store', 'outlets', 'see', 'museum']

Caption: Saxifraga bryoides (habitus) Mossy Saxifrage near the Aletsch Glacier, Switzerland
Query: How many flowers grow on each flower stem of the mossy saxifrage?
MARVEL nearest token: ['bacteria', 'the', 'behalf', 'berries', 'bacter', 'omato', 'cum', 'indigenous', 'rum']
CIEA nearest token: ['cum', 'flowers', 'Flowers', 'alt', 'behalf', 'berries', 'timp', 'individuals', 'young', 'Blue', 'concerns', 'omato', 'buds', 'um', 'acum', 'rum']

Caption: Portrait de la marquise d’Orvilliers, Jacques-Louis David, 1790 Portrait of one of two sisters of the Genevan banker Jacques Rilliet
Query: Did the woman in the Elisabeth Vigée-Lebrun Self-Portrait and the woman in Portrait de la marquise d’Orvilliers both wear a black dress with a red ribbon?
MARVEL nearest token: ['dominant', 'the', 'behalf', 'often', 'Mary', 'acum', 'Deborah', 'ographer', 'altar', 'mother', 'Francis', 'actress', 'Sfant']
CIEA nearest token: ['connaissance', 'dress', 'behalf', 'expressive', '$75', 'Dorothy', 'Deborah', 'ographer', 'mother', 'altar', 'Francis', 'acum', 'actress', 'Sfant']

Figure 3: Case studies. The nearest token represents the embeddings in the vocabulary that are closest to the visual embeddings, with duplicates removed. The words in red represent terms related to the image that are not found in MARVEL within CIEA.

Acknowledgement

This work is supported by Key-Area Research and Development Program of Guangdong Province (No.2024B1111100001).

References

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.

Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. 2023. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15338–15347.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901.

Yingshan Chang, Mridu Narang, Hisami Suzuki, Gui-hong Cao, Jianfeng Gao, and Yonatan Bisk. 2022. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16495–16504.

Chang Che, Qunwei Lin, Xinyu Zhao, Jiaxin Huang, and Liqiang Yu. 2023. Enhancing multimodal understanding with clip-based image-to-text transformation. In Proceedings of the 2023 6th International Conference on Big Data Technologies, pages 414–418.

Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754–17762.

Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. 2023. Lift yourself up: Retrieval-augmented text generation with self-memory. Advances in Neural Information Processing Systems, 36:43780–43799.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.
2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171–4186.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Jiayi Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. 2025. Natural language understanding and inference with mlm in visual question answering: A survey. ACM Computing Surveys, 57(8):1–36.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020b. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pages 19730–19742.

Wei Li, Hehe Fan, Yongkang Wong, Yi Yang, and Mohan Kankanhalli. 2024a. Improving context understanding in multimodal large language models via multimodal composition learning. In International Conference on Machine Learning, pages 27732–27751.

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2024b. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26763–26773.

Siqi Liu, Weixi Feng, Tsu-Jui Fu, Wenhui Chen, and William Wang. 2023a. Edis: Entity-driven image search over multimodal web content. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4877–4894.

Zhenghao Liu, Chenyan Xiong, Yuanhuizi Lv, Zhuyuan Liu, and Ge Yu. 2023b. Universal vision-language dense retrieval: Learning a unified representation space for multi-modal retrieval. In International Conference on Learning Representations.

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.

Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36:43447–43478.

Anas Mahmoud, Mostafa Elhoushi, Amro Abbas, Yu Yang, Newsha Ardalani, Hugh Leather, and Ari S Morcos. 2024. Sieve: Multimodal dataset pruning using image captioning models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22423–22432.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.

Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024. Replug: Retrieval-augmented black-box language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, pages 8364–8377.

Xi’ao Su, Ran Wang, and Xinyu Dai. 2022. Contrastive learning-enhanced nearest neighbor mechanism for multi-label text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 672–679.

Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka. 2008. Sofrank: optimizing non-smooth rank metrics. In Proceedings of the 2008 International Conference on Web Search and Data Mining, pages 77–86.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Jianfeng Wang, Zhengyuany Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Li-juan Wang. 2022a. GIT: A generative image-to-text transformer for vision and language. Transactions on Machine Learning Research.

Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, et al. 2024a. Searching for best practices in retrieval-augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17716–17736.

Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2022b. SimVLM: Simple visual language model pretraining with weak supervision. In International Conference on Learning Representations.

Ziyang Wang, Heba Elfardy, Markus Dreyer, Kevin Small, and Mohit Bansal. 2024b. Unified embeddings for multimodal retrieval via frozen llms. In Findings of the Association for Computational Linguistics, pages 1537–1547.

Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. 2023. Cap4video: What can auxiliary captions do for text-video retrieval? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10704–10713.

Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations.

Zihao Yi, Jiariu Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. 2024. A survey on recent advances in llm-based multi-turn dialogue systems. arXiv preprint arXiv:2402.18013.

Shi Yu, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2023. Openmatch-v2: An all-in-one multi-modality plm-based information retrieval toolkit. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3160–3164.

Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. 2021. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6995–7004.

Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579–5588.

Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, and Shafiq Joty. 2023. Retrieving multimodal information for augmented generation: A survey. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4736–4756.

Tianshuo Zhou, Sen Mei, Xinze Li, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, Yu Gu, and Ge Yu. 2024. Marvel: Unlocking the multi-modal capability of dense retrieval via visual module plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 14608–14624.

Xiangrong Zhu, Yuexiang Xie, Yi Liu, Yaliang Li, and Wei Hu. 2025. Knowledge graph-guided retrieval augmented generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, pages 8912–8924.
A Datasets

<table>
  <tr>
    <th>Modality</th>
    <th>Documents</th>
    <th colspan="3">Queries</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Train</th>
    <th>Dev</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>Image</td>
    <td>389,750</td>
    <td>16,400</td>
    <td>2,554</td>
    <td>2,511</td>
  </tr>
  <tr>
    <td>Text</td>
    <td>787,697</td>
    <td>15,366</td>
    <td>2,446</td>
    <td>2,455</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>1,177,447</td>
    <td>31,766</td>
    <td>5,000</td>
    <td>4,966</td>
  </tr>
</table>

Table 6: Statistics of the adopted WebQA.

WebQA is an open-domain question answering dataset, where each query is associated with one or more multimodal documents to assist in generating responses. WebQA can be divided into three partitions: train, dev, and test, with data statistics as shown in Table 6. The retrieval corpus contains 389,750 image documents with visual information and captions, as well as 787,697 plain text documents. Our partitioned WebQA-Image focuses solely on image documents for testing the effectiveness of CIEA in multimodal modeling. In contrast, WebQA-Multi retrieves from a total of 1,177,447 documents to assess whether the model can maintain an accurate representation of text information.

WebQA is used by many baselines, such as MARVEL (Zhou et al., 2024). ClueWeb22-MM is also utilized in MARVEL. We have applied for this dataset from Carnegie Mellon University; unfortunately, our request was denied due to export control restrictions. Therefore, we also use EDIS to evaluate our methods. EDIS (Liu et al., 2023a) is a comprehensive dataset consisting of 1 million image-text pairs sourced from Google. The dataset includes a training set of 26,000 pairs, accompanied by validation and test sets, each containing 3,200 pairs. With a high entity count of 4,03, EDIS reflects a diverse range of semantic content, making it a valuable resource for research in image-text retrieval tasks.

B Implementation Details

In our experiment, we use T5-ANCE (Yu et al., 2023) as the base language model and utilize CLIP as the visual understanding module to implement our CIEA model, enhancing the image understanding capability of T5-ANCE. For easier comparison, we initialize our projector using the pre-trained projector from MARVEL. The visual encoder is initialized using the clip-vit-base-patch32 checkpoint from OpenAI\footnote{https://huggingface.co/sentence-transformers/clip-ViT-B-32}. We truncate the text input length to 128 tokens.

During the training process, we use the AdamW (Loshchilov and Hutter, 2019) optimizer, set the maximum training epochs to 40, with a batch size of 64, a learning rate of 5e-6, and the temperature hyperparameter \( \tau \) set to 0.01. We follow the setup of UniVL-DR (Loshchilov and Hutter, 2019) by training with the ANCE sampling method. Starting from the CIEA-DPR model, fine-tuned with negative examples from within the batch, we continue to train CIEA-DPR to achieve a balanced modality with difficult negative examples. For negative sampling in evaluation, we shuffle the training set and select other samples within the same batch as negative examples in DPR, while in the ANCE approach, we utilize the top 100 samples with the highest similarity retrieval by CIEA-DPR as hard negatives. All models are evaluated every 500 steps, with early stopping set at 5 steps. For the parameter \( \lambda \), we perform a grid search and select the parameter that yielded the lowest loss on the validation set, setting it to 0.0011 for WebQA-Multi, 0.019 for WebQA-Image, and 0.001 for EDIS. The results of various baselines for the WebQA-Multi dataset are provided in the MARVEL’s paper(Zhou et al., 2024), while the experimental results for other baselines are reproduced using the open-source code from the original papers. All experiments are conducted on a single NVIDIA A100.

C Statistical significance

To avoid potential concerns that the results are justified, we repeat the experiments with different random seeds and report the average values and standard deviation values in Table 8. The results demonstrate the proposed CIEA achieves significant and consistent improvements compared to MARVEL. Besides, different modules make positive contributions to the overall performance of ICEA.

D The Effects of Language Models

The project-based approach relies on language models as the backbone, making the performance of the language model one of the key factors influencing retrieval effectiveness. To validate whether our method can function effectively with different language models as the backbone, we con-
<table>
  <tr>
    <th>Language model</th>
    <th>setting</th>
    <th>mrr@10</th>
    <th>ndcg@10</th>
    <th>mrr@20</th>
    <th>ndcg@20</th>
    <th>rec@20</th>
    <th>rec@100</th>
  </tr>
  <tr>
    <td rowspan="3">T5-ANCE</td>
    <td>Text only</td>
    <td>52.67</td>
    <td>45.69</td>
    <td>53.08</td>
    <td>48.48</td>
    <td>67.70</td>
    <td>82.07</td>
  </tr>
  <tr>
    <td>Project-based</td>
    <td>56.91</td>
    <td>54.26</td>
    <td>57.31</td>
    <td>56.88</td>
    <td>77.20</td>
    <td>89.26</td>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td><b>57.49</b></td>
    <td><b>54.80</b></td>
    <td><b>57.86</b></td>
    <td><b>57.24</b></td>
    <td><b>77.21</b></td>
    <td><b>89.41</b></td>
  </tr>
  <tr>
    <td rowspan="3">BERT</td>
    <td>Text only</td>
    <td>39.50</td>
    <td>36.23</td>
    <td>40.06</td>
    <td>38.69</td>
    <td>55.37</td>
    <td>72.72</td>
  </tr>
  <tr>
    <td>Project-based</td>
    <td>52.65</td>
    <td>49.41</td>
    <td>53.21</td>
    <td>51.97</td>
    <td>71.16</td>
    <td>85.40</td>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td><b>53.40</b></td>
    <td><b>49.96</b></td>
    <td><b>53.83</b></td>
    <td><b>62.56</b></td>
    <td><b>71.95</b></td>
    <td><b>85.72</b></td>
  </tr>
  <tr>
    <td rowspan="3">BART</td>
    <td>Text only</td>
    <td>24.52</td>
    <td>21.79</td>
    <td>25.04</td>
    <td>23.98</td>
    <td>36.91</td>
    <td>54.66</td>
  </tr>
  <tr>
    <td>Project-based</td>
    <td>50.46</td>
    <td>47.72</td>
    <td>50.94</td>
    <td>50.04</td>
    <td>69.40</td>
    <td>83.69</td>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td><b>50.97</b></td>
    <td><b>48.06</b></td>
    <td><b>51.43</b></td>
    <td><b>50.72</b></td>
    <td><b>70.44</b></td>
    <td><b>84.81</b></td>
  </tr>
  <tr>
    <td rowspan="3">GPT2</td>
    <td>Text only</td>
    <td>20.38</td>
    <td>18.62</td>
    <td>20.86</td>
    <td>20.19</td>
    <td>29.97</td>
    <td>44.33</td>
  </tr>
  <tr>
    <td>Project-based</td>
    <td><b>45.94</b></td>
    <td>42.45</td>
    <td>46.46</td>
    <td>45.08</td>
    <td>63.16</td>
    <td>78.34</td>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td><b>45.82</b></td>
    <td><b>42.60</b></td>
    <td><b>46.51</b></td>
    <td><b>45.13</b></td>
    <td><b>63.46</b></td>
    <td><b>78.91</b></td>
  </tr>
  <tr>
    <td rowspan="3">GPT2-LARGE</td>
    <td>Text only</td>
    <td>37.25</td>
    <td>34.52</td>
    <td>37.67</td>
    <td>37.11</td>
    <td>54.37</td>
    <td>73.08</td>
  </tr>
  <tr>
    <td>Project-based</td>
    <td>51.44</td>
    <td>48.98</td>
    <td>51.89</td>
    <td>51.64</td>
    <td>72.06</td>
    <td>86.61</td>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td><b>54.28</b></td>
    <td><b>51.81</b></td>
    <td><b>54.68</b></td>
    <td><b>54.39</b></td>
    <td><b>74.94</b></td>
    <td><b>88.52</b></td>
  </tr>
</table>

Table 7: Results of different language models on DPR training framework.

<table>
  <tr>
    <th>Model</th>
    <th>MRR@10</th>
    <th>NDCG@10</th>
    <th>REC@100</th>
  </tr>
  <tr>
    <td><b>CIEA</b></td>
    <td>66.03±0.012</td>
    <td>63.70±0.030</td>
    <td>92.63±0.008</td>
  </tr>
  <tr>
    <td>w/o image query</td>
    <td>65.66±0.008</td>
    <td>63.44±0.072</td>
    <td>92.54±0.003</td>
  </tr>
  <tr>
    <td>w/o attention</td>
    <td>65.90±0.098</td>
    <td>63.39±0.003</td>
    <td>92.60±0.001</td>
  </tr>
  <tr>
    <td><b>MARVEL</b></td>
    <td>65.30±0.017</td>
    <td>63.19±0.005</td>
    <td>92.40±0.012</td>
  </tr>
</table>

Table 8: Standard Deviation Values on WebQa-multi.

duct experiments using T5-ANCE (Yu et al., 2023), BART (Lewis et al., 2020a), BERT (Devlin et al., 2019), GPT-2, and GPT-2-LARGE (Radford et al., 2019), encompassing encoder-decoder, encoder-only, and decoder-only architectures. For the encoder-decoder model, we follow MARVEL’s approach (Zhou et al., 2024) and encode the multi-modal document input into the encoder, while inputting a ‘s’ character into the decoder segment to use its final hidden layer representation as the document representation. For the encoder-only model, we use the representation of the first character, and for the decoder-only model, we use the final hidden layer representation of the last token, as each token only computes attention with preceding tokens. Due to the large number of parameters in GPT-2 LARGE, we set its batch size to 16 to avoid memory issues, while setting the batch size to 64 for the others. Besides the results trained with the ANCE negative sampling method presented in Table 3, we also include the results trained using the in-batch negative sampling method of DPR in Table 7. It is evident that while ANCE consistently outperforms DPR, CIEA surpasses both simple project-based and text-only methods under both negative sampling approaches, demonstrating that the proposed method aligns better with image information.

E Failure Case Analysis

We provide a failure case analysis in Table 9 for a better understanding of the proposed method and for promoting further research. Each case presents the ground-truth image-caption pair alongside the top three retrieved candidates from CIEA, with all instances classified as retrieval failures due to the absence of ground-truth matches among the top results. The first two cases illustrate the model’s partial success in capturing color attributes (e.g., “orange” and “green”), while it might struggle to establish precise color-object mappings (e.g., “ceiling” and “tiles”), especially within multi-object environments. The third case reveals more fundamental limitations regarding action interpretation, as the retrieved results exhibit minimal relevance to the target action “relax”. These failures highlight persistent challenges in visual-semantic alignment, particularly in the contextual binding of object attributes and the dynamic interpretation of action representations.
Question: Which restaurant on Orchard Road in Singapore has a tiered orange ceiling?

<table>
  <tr>
    <th>Golden</th>
    <th>Retrieval (Top 3)</th>
  </tr>
  <tr>
    <td>Coffee Club, Orchard Road (8171776629)<br>Coffee Club at Wheelock Place, Singapore.</td>
    <td>Orchard Road 12, Xmas, Dec 06 Orchard Road, Christmas Light-up 2006, Singapore.<br>Orchard Road, Xmas, Dec 06 Orchard Road, Christmas Light-up 2006, Singapore.<br>Orchard Road 15, Xmas, Dec 06 Orchard Road, Christmas Light-up 2006, Singapore.</td>
  </tr>
</table>

Question: Which Buddha statue has green tiles in front of it?

<table>
  <tr>
    <th>Golden</th>
    <th>Retrieval (Top 3)</th>
  </tr>
  <tr>
    <td>Buddha statue no 1<br>Buddha statue at Hussain sagar</td>
    <td>Buddha eden (34)<br>Buddha 1251876<br>Building the Buddha – panoramio</td>
  </tr>
</table>

Question: Where can people relax in the grass next to the Eifel Tower?

<table>
  <tr>
    <th>Golden</th>
    <th>Retrieval (Top 3)</th>
  </tr>
  <tr>
    <td>Eiffel Tower, Paris 17 September 2010.</td>
    <td>FW Eiffelturm.<br>Berlin , Mitte , Alexanderplatz , fernsehturm - panoramio.<br>Eiffelturm.</td>
  </tr>
</table>

Table 9: Failure Case Analysis.

Reviewer paper summaries: Key Objective:
The paper claims to introduce a novel framework, Complementary Information Extraction and Alignment (CIEA), for improving multimodal retrieval by addressing a critical limitation in existing methods: most current methods emphasize the similarities between text and image representations while neglecting the unique, complementary information offered by each modality. This complementary information can enhance retrieval tasks, especially in scenarios where text alone cannot fully resolve a query.

Approach: The proposed framework -
1. Maps multimodal data (text and images) into a unified latent space using a transformer-based language model for text and the CLIP model for images.
2. Extracts and preserves complementary information in image representations using a complementary information extractor. This extractor identifies differences between image and text representations, re-weights image embeddings, and aligns them with textual queries.
3. Optimizes the alignment process using 2 contrastive loss functions:
    * One ensures the semantic integrity of learned representations.
    * The other promotes the alignment of image representations with the complementary information from queries.

The novelty and main contribution of the paper centers on step # 2 and #3.


Experiment Scope:
1. Datasets: 2 datasets derived from the WebQA dataset:
    * WebQA-Multi, containing both text-only and multimodal documents.
    * WebQA-Image, focusing exclusively on image-containing documents.
2. Baseline approaches: divide-and-conquer (e.g., BM25, CLIP-DPR), universal dense retrieval models (e.g., MARVEL, UniVL-DR) and project-based methods using CLIP and language models.
3. Metrics: MRR@10, MRR@20, NDCG@10, NDCG@20, REC@20, REC@100

Findings and Contributions:
1. Effectiveness: CIEA achieves improvements in retrieval performance by capturing complementary information, demonstrating its value over methods that focus solely on modality-specific features or universal representations.
2. Robustness: The framework is tested with multiple language models (e.g., T5-ANCE, BERT, BART, GPT-2) and shows consistent improvements, highlighting its flexibility and effectiveness across architectures.
3. Case Studies: Visualized examples illustrate how CIEA captures unique, relevant image details (e.g., colors, objects) that other models miss, validating its ability to enhance multimodal retrieval.
Summary of strengths: 1. Novelty: The method extracts and aligns patch level distance between text and image representations which offers a fresh perspective for improving retrieval performance. The dual contrastive loss adds practical novelty to the optimization.
2. Strong empirical results: The method works well on both mixed-modality and image-centric retrieval tasks and outperforms competitive baselines. The robustness adds on top of that.
3. Detailed ablation study and analysis helps understand the contribution of each key component like complementary information extractor or the attention mechanism. The paper provides examples (Figure 3) where CIEA captures semantically relevant image details (e.g., "flowers," "clock," "green") that are absent in MARVEL's retrieval outputs, highlighting it’s effectiveness.
4. The paper is well written. It clearly explains the current gaps, paper’s scope, the claims and contributions. The paper clearly differentiates its contributions from existing methods. It builds upon established concepts (e.g., CLIP, contrastive loss, unified latent spaces) but extends them with its 2 contributions. It doesn't claim to overhaul the entire multimodal retrieval paradigm but rather to address a specific gap. The figures give a good intuition into the use case. 
5. The authors acknowledge limitations and provide future exploration paths in audio/video modalities and larger LLMs.
6. The proposed framework is grounded in widely used architectures (e.g., CLIP, transformer-based models). The findings show the potential for other application areas, such as multimodal question answering and recommendation systems but it needs to supported through experiments.
7. Reproducibility: Key Implementation Details Provided:
    * Model Architecture: Clear description of the overall architecture (Figure 2), Detailed equations for each component (Equations 1-14), Specific description of how text and image inputs are processed
    * Training hyper-params details: AdamW Optimizer, learning rate: 5e-6, batch size: 64 (16 for GPT-2-LARGE), maximum training epochs: 40, temperature parameter τ: 0.01, early stopping: 5 steps with evaluation every 500 steps, λ hyperparameter values: 0.0011 for WebQA-Multi and 0.019 for WebQA-Image
    * Model Components: Base language models, visual encoder: clip-vit-base-patch32, text input length truncation: 128 tokens
Summary of weaknesses: 1. Limited dataset diversity: The evaluation is only based on WebQA. To establish the complete efficacy of this method, it should be evaluated on the other datasets used by the baselines e.g. ClueWeb22-MM.
2. The paper does not discuss the computational efficiency of the proposed framework, particularly the impact of the complementary information extractor and dual contrastive losses on training and inference time.
3. While the complementary information extractor is described as a core innovation, the paper lacks a strong theoretical explanation for why it improves retrieval. For instance, the decision to compute patch-level differences using maximum cosine similarity (Equation 7) is introduced but not justified in depth. The methodology (Section 2.3) mentions re-weighting patches based on cosine similarity, but there is no discussion of why this approach is optimal.
4. While the paper provides detailed metrics (e.g., MRR@10, NDCG@10), the qualitative analysis of CIEA’s advantages is limited. The case studies in Section 3.4 highlight successful retrievals, but failure cases are not discussed, leaving gaps in understanding scenarios where CIEA might underperform. Figure 3 illustrates cases where CIEA retrieves additional relevant tokens, but there is no analysis of cases where the complementary information extractor might introduce noise or fail to enhance retrieval.
5. Some implementation details are vague: Specifics of the projector, exact procedure for negative sampling in evaluation, specific configuration or parameters for baseline models, how the complementary information extractor is implemented in practice, gpu memory requirements  etc.
6. Though the paper acknowledges in the "Limitations" section but does not provide any preliminary experiments or evidence to suggest adaptability beyond text-image modalities.
Comments, suggestions and typos: - Will be better to add Related work section after Introduction than later in the paper.
- Include more implementation details and failure analysis
Overall assessment scores: 3.5