Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval
Authors: Zhenghao Liu
Chenyan Xiong
Yuanhuiyi Lv
Zhiyuan Liu
Ge Yu
Venue: ICLR 2023 poster (2023)
Keywords: Multi-Modal Retrieval
Dense Retrieval
Universal Embedding Space
Modality-Balanced Hard Negative Training
Image Verbalization

Abstract: This paper presents Universal Vision-Language Dense Retrieval (UniVL-DR), which builds a unified model for multi-modal retrieval. UniVL-DR encodes queries and multi-modality resources in an embedding space for searching candidates from different modalities. To learn a unified embedding space for multi-modal retrieval, UniVL-DR proposes two techniques: 1) Universal embedding optimization strategy, which contrastively optimizes the embedding space using the modality-balanced hard negatives; 2) Image verbalization method, which bridges the modality gap between images and texts in the raw data space. UniVL-DR achieves the state-of-the-art on the multi-modal open-domain question answering benchmark, WebQA, and outperforms all retrieval models on the two subtasks, text-text retrieval and text-image retrieval. It demonstrates that universal multi-modal search is feasible to replace the divide-and-conquer pipeline with a united model and also benefits single/cross modality tasks. All source codes of this work are available at https://github.com/OpenMatch/UniVL-DR.

Published as a conference paper at ICLR 2023
UNIVERSAL VISION-LANGUAGE DENSE RETRIEVAL: LEARNING A UNIFIED REPRESENTATION SPACE FOR MULTI-MODAL RETRIEVAL
Ge Yu² 1 DDpartment of Computer Science and Technology, Northeastern University, Shenyang. China "Microsoft Research, Redmond, USA 'Department of Computer Science and Technology, Tsinghua University, Beijing, China Institute for Artificial Intelligence. Tsinghua University, Beijing, China State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China
Zhenghao Liu'
Chenyan Xiong"
Yuanhuiyi Lv+
Zhiyuan Liu
ABSTRACT
This paper presents Universal Vision-I Language Dense Retrieval (UniVL.DRR), which builds unified model for multi-modal retrieval. UniVL-DR encodes queries and multi-modality resources in an embedding space for searching candidates from different modalities. To learn a unified embedding space for multi-modal retrieval, UniVL-DR proposes two techniques: 1) Universal embedding optimization strategy, which contrastively optimizes the embedding space using the modality- hard negatives; 2) Image verbalization method, which bridges the modality gap between images and texts in the raw data space. UniVL-DR achieves the state-of- the-art on the multi-m modal open-domain question answering benchmark, WebQA, and outperforms all retrieval models the two subtasks, text-text retrieval and text-image retrieval. It demonstrates that universal multi-modal search is feasible replace the divide-and-conquer pipeline with united model and also benefits single/cross modality tasks. All source codes of this work are available at https: 7/gi //github. wtt.:/c hhhpbb.cco//ppeeaaacccuunW--
INTRODUCTION
Although search engines primarily focus textual data (Singhal al., 2001). multi-media is necessary to satisfy user needs during retrieval. A user query can be answered by the information in variant formats, such text document, picture. The growth of multi-media content has been one of the most notable trends on the internet (Mei et al., 20144), and various studies have proved that users prefer more vivid multi-media content in search results (Datta al. 2008).
Current multi-media search systems often employ a divide-and-conquer approach. As shown in Figure (aa they first conduct search in individual modalities, including text, image, video, etc. (Bajaj et al.) 2016 Grubinger et al. 2008 Kwiatkowski et al. 2019; Awad et al. 202I). and then fuse
the retrieval results from various verticals together. e.g.. building another ranking layer on top of
these single/cross modality retrievers (Escalante et al.) 2008 Grubinger et al.) 2008). Both relevance modeling and retrieval result fusion are usually entwined to achieve more accurate multi-modal retrieval results. However, due to the modality gap, they can be only pipeline-modeke in divide-and- conquer, making challenging to fuse retrieval results from different modalities.
In this paper, we explore the potential of universal multi-modal retrieval to build end-to-end model and retrieve multi-modality documents for user queries. Illustrated in Figure T(b) universal multi-moda retrieval maps queries and multi-modality resources to universal embedding space and retrieves multityodality candidates via KNN search. As a result, the relevance modeling, croas-modality matching, and retrieval result fusion done by one model.
More specifically, we propose Universal Vision-Language Dense Retrieval (UniVL-DR) model to get the representations of queries, texts. and images and learn tailored vision-language embedding space for multi-modaa retrieval, UniVL-DR optimizes the vision- languagee embedding space using hard negatives (Xiong al. 2021a) and balances the modalities of these negatives to alleviate

---

Published as a conference paper at ICLR 2023
Image Retrieval
Query
Image Retrieval
Query
Yanadu Guemmee, For ta, with the pouring concrille
Images
Text IR
coin to feature an African Assertan the fooker
honored
Ranking List
Text IR
Ranking List
The National Air and Space Museum of the Smitosoman Instrution, nlso called the Air
KNN Search
Texts
Sub-Ranking Lists
Divide-and and-Conquurr Multi-Media Search.
(b) Universal Vision-Language Search.
Figure I: Different Architectures of Multi-Modal Retrieval Systems.
the modality preference of multi-modal retrievers. Furthermore, UniVL-DR introduces an image verbalization method, which regards language kind of mentalese (Cavanagh 2021) and mitigates the modality gap between images and texts. Our image verbalization method first aligns the semantics of image captions and figure pixels (Huang et 2021a), and then paraphrases the image facts. helps to bridge language and vision understanding modules of UniVL-DR via natural language.
To build a multi-modal retrieval benchmark, leverage multi-modal question answering (QA) benchmark WebQA (Chang et al. (0022) and convert it to a standard open-domain setting: retrieving multi-modality candidates from text and image collections for a user query. Divide-and-come andee-uuee an intuitive way to build multi- modal retrieval system and we pre-route queries to oracle modality to show the upper bound performance of such a system. Compared with the lndide-and-conuuer system, UniVL-DR addresses the retrieval result fusion challenge, achieves state-of-the-an multi-modal retrieval performance, and brings more than 5% improvement in single/cross modality retrieval.
Our experiments show that UniVL-DR learns effective embedding space for multi-modal retrieval by separating texts and images into different areas and guiding queries to return candidates from corresponding modalities. Our further analyses show that UniVL-DR can alleviate overfit single- modality signals by balancing hard negatives during training and bridging the modality gap between vision and language by verbalizing images. All experimental results show that learning one universal representation space is starting to benefit single- modality tasks pretraining representation models on multi-modality and using our techniques can learn additional signals from multi-moaalities, overcome the modality boundary, and provide convincing gains in single/multimmodaiiy tasks.
2 RELATED WORK
Document retrieval is typical single modality retrieval task, which aims to return related documents for user queries and can be tackled with dense retrievers (Xiong et al. 2021b Lewis et al.) 2020 Zhan al. 202 Li al. 2021b Yu et al. (202I). Dense retrievers encode queries and documents with pretramed language models (Dellin et 2019) and map them in an embedding space to conduct an efficient search. The query and document encoders are usually contrastively trained with bn-hatch negatives, BM25 retrieved negatives, and hard negatives (Karpukhin et al. 2020 Xiong et al. 2021a).
Recently, lots of work has focused on multi-mode retrieval tasks, which retrieve texts and images to satisfy the multi-modality information needs of users (Hannan et al. 2020 Singh et al. 2021 Talmor jet al. 202T Chang al.] 2022). WebQA (Chang al. (022). an open-domain multi- modll question answering benchmark, is built to encourage the following work to represent multi-mod knowledge in a unified space and answer user queries with the information from attribute modalities. It is more realistic setting, which avoids synthesizing queries with templates (Talmor et al. 202 and downplays the role modality disambiguatto (Hannan et al. (0020) in the multi-modalitt modeling.
To search information from large-scale multi-monl multi-modality sources, WebQA (Chang et al. (022 employs a divide-and lal--aaddccouuuue pipeline to search text and image candidates with BM25 and CLIP (Radford et al. 2021) and then fuse these retrieval results using a vision-language model. However, single-
2

---

Published as a conference paper at ICLR 2023
Query: how many years did william bradford serve governor plymouth colony?
Query: woman wearing her cutting Retrieval Candidates:
Query; What water-related object sitting in the Torre del Relaj?
Retrieval Candidates:
Retrieval Candidates:
Imagel
Imape2
Images
Text1: William Bradford was an English Separatist leader in Leiden and Holland.
He served as Plymouth Calony Gavernor five times covering about thirty years
Image Retrieval
between 1621
Query:
Text Retrieval +
lorre Reloj Spanish historic center Cartagena Indias.
Text2: William Bradford was passenger on the Mayflower in 1620. He travelled to the New World to live.
main city gate
Retrieval Candidates:
Text2: The Torre del Reloj is clock tower
Text3: William m Bradford was governor of Plymouth founded by people called
Text1: A woman wearing net on head
kenwwn as Arquillo Clock, and one the most emblematic buildings Chiclana.
for years.
cutting cake.
Text2: A baker woman preparing bread
Text3: Other landmarks the city include the Torre del Reloj (Clock Tower).
Puritans.
dough on a tray with wax paper.
(a) Text-Tex: Retrieval.
(b) Cross Modality Retrieval.
(c) Multi-Modis Retrieval.
Figure 2: Examples of Different Retrieval Tasks.
modality retrievers, such as BM25 and CLIP, usually show distinct retrieval effectiveness (Chang et al. 2022), leading to modality discrimination during fusing retrieval results from different modalities,
When building a unified multi-modal retriever, vision-language pretraining (VLP) is crucial to learn universal representations for texts and images, which has also shown success lots of vision- language benchmarks (Uppal et al.) [022] Han et al. 2020) Khan et al., 2021 Du et al.) 2022). Most VLP approaches encode texts and images and pretrain encoders with two tasks: masked token 202I). These VLP methods teach vision- lasion-language
prediction and text- text-image matching (Zhang et al.
models learn the semantic alignments between texts and images, as well as encode images with the regional features of detected objects (Chen et al.
[019 Lu et al. 2019; Tan and Bansal (2019] SSu
al., 2020; Li al. 2019 2021a Cho al. 2021 Hu al. 2020 Gan et al.] or the whole image features (Xu et al. 2021 Kim et al.
Huang et al. 2021b Wang et al. 2021).
3 MULTI-MODAL RETRIEVAL TASK
As shown in Figure 3 we compare different retrieval tasks and tell apart the differences between multi-modal retrieval and other two tasks, single modality retrieval and cross modality retrieval.
Single Modality Retrieval. Single modality retrieval focuses on conducting relevance searching in modality space, which includes text-text retrieval and image-image retrieval. Text-text re- trieval (Bajaj et al. 2016) aims to search relevant candidates from the text collection T = (Ti,... T, answer a query q. And image-image retrieval (Yoon et al. 2021) focuses more on returning similar images from the image collection I = {/,,.., Im] for the given image Ij.
Cross Modality Retrieval. The cross modality retrieval, e.g. MSCOCO (Chen et al. 2015) and Flickr30K (Young et al., 2014), contains two subtasks: text-image retrieval and image-text retrieval. Given an image caption T. or an image 1j. these tasks require retrieval models conduct cross- modality matching between images and captions, aiming to search candidates from images I = (I...., Im,) } or image captions T = = [TI.... ..., Tn respectively. Such cross- modalityy interactions are built to align semantics between captions and images, which is distinct from the search relevance.
Multi-Modal Retrieval. Given a query 9. the multi-modal retrieval task (Chang al., 2022) helps users uncover the information from multi-modality sources D = (Ti..., Tn, T,..,,... )-
Different from single/cross modality retrieval, multi-modal retrieval aims at returning relevant candidates from the multi-modality documents D. The retrieval results may consist of texts, images, or a mixture of them according to user query q. Different from existing text and sketch base image retrieval (Sangkloy et al. 2022 Dutta and Akatal 2020; Dey et al., 2018 Mai et al. 2017). the multi- modal retrieval focuses more on relevance modeling between queries and documents, single/cross modality matching, and modality routing, making this task more challenging. Moreover, we can pre-route queries to a single modality and convert the multi-modal retrieval to two subtasks, text-text retrieval and text-image retrieval, which single and cross modality retrieval tasks.
3

---

Published as a conference paper at ICLR 2023
Push Away
Query
Query
Align
Text
Encoder
Document
Training with Modality-
Image Caption
Balanced Negatives
Image Caption
Verbalize
Ground Truth
Image Document
Picture
Image
Image
Feature
Encoder
Image Feature
Figure 3: The Architecture of UniVL-DR.
4 UNIVSEARCH BY LEARNING A UNIFIED EMBEDDING SPACE
This section describes our Universal Vision-Language Dense Retrieval (UniVL-DR). As shown in Figure given query q and multi-modality documents D {d Text .... fext= d image ...y d Image directly encodes query q, text document dex and image document d image in one embedding space, which conducts relevance modeling, modality routing, and result fusion in such a space (Sec. [4.1].
Texts and images usually have different understanding mechanisms, making it difficult to tackle multi-modality tasks. Nevertheless, language and vision can be commonly translated as type of mentalese better communicate between different modules in our brains (Cavanagh (02T), thus a unified representation method has the ability to break the boundary of different modalities and benefit vision- lasoun-language learning. To build unified multi-modal retrieval system, UniVL-DR learns a universal embedding space by contrastively optimizing vision-language representations using hard negatives with galanced-modality sampling (Sec. (4.2) and bridging the modality gap via verbalizing the picture to paraphrase pixel semantics in the raw text space (Sec.
4.1 MULTI-MODALITY DENSE RETRIEVAL
UniVL-DR gets representations of queries, image documents and text documents with two encoders: TextEnocder and ImgEncoder. Specifically, image document d' image consists of a picture I, and an image caption Cj, thus we utilize ImgEncoder and TextEnocder encode I, and C,.
Query Encoding, UniVL-DR directly encodes the query to get its representation qे: Texedereee
(1)
Text Document Encoding. To represent text documents, UniVL-DR also leverages the TextEnocder to encode the i-th text document d fext as d'
di Text (nooddrrerr
(2)
Image Document Encoding. Different from text documents, image documents can be represented by picture features and image captions and the textual captions can help better understand the semanties of image documents (Baldrati et al. 2022). Thus, Uni UL-DL encodes picture 1, and image caption Cj and then sums these embeddings to get the representation d) d' of j-th image document:
d Limage ImggEnocder(1) + TeatEnoccerr(C))
(3)
The representations d image and d d of image document and text document use the same Tert Enocder encode their textual information, which bridges different modalities in the text space and helps to build a universal embedding space for multi-modality retrieval.
The cosine similarity score f(g, d) of query q and document candidate E D can be calculated to estimate the relevance between and d:
Multi-modality Document Retrieval.
fq. d) cos(q.
(4)
where q' and d are the representations of q and d. The efficient similarity calculation between queries and the multi-modality documents can be provided by FAISS (Johnson al. 2019).
4

---

Published as a conference paper at ICLR 2023
4.2 UNIVERSAL REPRESENTATION LEARNING
UniVL-DR employs a vision- languaae model, CLIP (Radford et al., 2021), to learn universal representations for queries and multi-modality documents, which knowledgeable about cross- modality retrieval. UniVI.-DR optimizes the universal embedding space through training with modality-baland hard negatives, which avoids overfitting to the signals of single-modaliyy during multi-modal co-training.
Given the query q and its relevant candidate dt E D, the embedding space can be optimized by sampling hard negatives D and minimizing the following contrastive training loss L:
log ellg. + Ex ED el(g,d)//
h2
(5)
flg, dt 1/T + log(e
log(e (aid 1/5
Mand in
e maa
Exlips
Linage
where 7 is the temperature to scale the similarity score. During training, we in fact maximize LAig and minimize Limage and Liex, which make queries closer related documents and away from unrelated documents. If 11 >> k2 or k2 > k1. we can achieve smaller loss Limage + Lext by simply making queries far away from the image collection or the text collection. Such behavior can win lower loss T. but overfits the ranking features from single/cross modality matching, leading to modality discrimination during retrieval. Our modalittybalannee negative training strategy keeps k 22 k to better train the modality selection ability of retrievers.
4.3 IMAGE VERBALIZATION FOR EXPANSION
UniVL-DR provides another way bridge the modality gap between texts and images by verbalizing picture pixel features, including image caption and query generation methods.
Following Li al. 2020), we can represent picture Ij using detected objects O (O,,..., Or}.
For each image object O; we can get its pixel feature O and the predicted class Ö. Then UniVL-DR uses vision-language model, such as VinVL (Zhang et al.) (022). to verbalize image documents. Specifically, we generate potentially matched captions or related queries as the image verbalization results ((J),, according to the picture I or the image document d'mage {4g Cj}.
We can first feed the predicted classes (Ó::..; O,} and regional features (Ō o,} of detected objects into image verbalization models. Then we train the model to generate image caption Cj:
x [CLS]]
O;; (SEP):
(6)
or replace the detected object classes {ō.;..: Õ.} in the input sequence X² with the image caption Cj to generate related query g of the image document di d'mage
X [CLS]; q: (SEP); Cj; [SEP]: O
where is the concatenation operation, and [CLS] and [SEP] are special tokens. During training or inference, we utilize Masked Language Modeling (MI.) (Devlin et al. (019)) to mask and predict some all of the tokens of image caption C and query in the inputs X; and Xy. aiming to train image verbalization models or generate verbalized captions and queries.
Finally, we enhanced the representations of image documents by expending their text representations Cj by expanding the raw caption Cj with image verbalization results V(J;):
C = Cý; [SEP]; V(I)),
where the enhanced text representation Cj is used to replace the raw caption C. in E.q. 3 during encoding the image document d i'mnge
5

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTAL METHODOLOGY
This section describes the dataset, baselines, some vision language models used in our experiments, and implementation details.
Dataset. A molti-hop and multi modal open domain question answering dataset WebQA (Chang et al. 2022) is used our experiments. We process the WebQA dataset an open domain retrieval setting and show the details in Appendix A.1
Evaluation Metrics We use NDCG@ K, MRR K, Recall @ @20, and Recall @ 100 as the evaluation metrics. K can be 10 and 20. And we regard MRR @ 10 our main evaluation (Bajaj et al. 2016).
Vision-Language Models. our experiments, we employ two state-of-the-art vision-language models, VL (Zhang et al. (2021) and CLIP (Radford et al. (02T) to implement different retrieval models in our experiments. VLnVL Zhang et al. (2021) inherits Oscar (Li et al. 2020) architecture, which extracts object tags and region features to represent images, and learns cross modal representa- tions by aligning semantics between images and texts. Different from VIn CLIP (Radford et al., 2021] utilizes dual encoder to project images and texts in the same semantic space for computing their similarity scores and is trained on a large-scale dataset Weblmage'Tex, that contains 400 million image-text pairs. It has shown strong effectivvness cross-modality retrieval.
Baselines. Our baselines contain several models in the settings of single modality retrieval, divide- and-conquer, and universal multi-modal retrieval.
Single modality retrieval. In this setting, we represent image documents with captions and em- ploy text retrievers, BM25 and DPR (Karpukhin al., 2020) as baselines. DPR is trained with NQ (Kwiatkowski et al. 2019), which is similar to the textual source of WebQA. Then we continu- ously train DPR with bn-batch and hard negatives to implement NQ-DPR and NQ-ANCE models.
Divide soride-and-orn and-conquer. We first employ three widely used retrievers, BM25, VinVL-DPR, and CLIP- DPR, conduct text-text retrieval and text-image retrieval. Then the multi-modality retrieval results are fused according to their uni-modal rank reciprocals or oracle modality routing. The latter one shows the upper bound of the retrieval performance of our divide-and-comee and--que models.
Multi-modal retrieval. In our experiments, we also build two multi-mode retrieval baselines: VinVL- DPR and CLIP-DPR. VinVL-DPR and CLIP-DPR represent image documents with caption and picture features. And then they optimize VLP models, VLn (Zhang et al.) 2021 and CLIP (Radford ct al. 202T). with in-batch negatives to learn universal representations for multi-modal retrieval.
Implementation Details. During training UniVL-DR, employ the text and image encoders from CLIP, truncate the text with the max length of 77 and set the batch size to 64, learning rate=5e 6, max training epoch to 20, and the temperature hyperparameter retrieve Top 100 documents using CLIP-DPR and sample two hard negatives of different modalities (k 1) from these candidates. All models are tuned with Adam W optimizer, are evaluated per 500 steps, and set early stop step as 5. More experimental details are shown in Appendix A.2
0.01. In our experiments,
6 EVALUATION RESULTS
In this section, we study the performance of UniVL-DR, its advantages in multi-modal retrieval, the effectiveness of our modality-balanced balanced hard negative training strategies, and how our image verbalization methods bridge the modality gap between texts and images.
6.1 OVERALL PERFORMANCE
The multi-modal retrieval performance of different models is shown in Table
Our UniVL-DR outperforms all baselines with more than 7% improvement on ranking evaluation, recalls more than 6% relevant multi modditt documents, and even outperforms the divide-and- conquer model guided by oracle modality routing. Such significant improvements illustrate the effectiveness of UniVI-DDR in building multi-modal retrieval system.
https://github. httpaatthhhhcccooopppaaii/L/11

---

BM25 and Published as a conference paper at ICLR 2023
Setting
Model
MRR#10 10
NDCG@ TO
MRR
NDCG@20
Rec
Rec 100
BM25
53.75
51.72
DPR (Zero Shor) Karpukhin et 2020
22,72
20.06
23.14
21.79
32.78
45.43
Single Modality
CLIP (Zero-Shot) Rantfrre
18.16
16.76
18.60
18.27
27.97
39.83
(Text Only)
Karpdkinn
NQ-DPR (Kanyakin al 2020
41,88
39.65
42.44
42.35
61.71
78.57
NQ-ANCE (Xtong 2040
45.54
42.05
45.93
43.83
58.42
69.31
VInYL-DPK
22.11
22.92
22.80
25.41
46.27
62.82
Divide-Conquer
CLIP-DPR
31.35
37.56
37.93
41..7
69.38
85.53
CLIP-DPR
BM25 CI JP-DPR (Omale Modality)
61,05
58.18
61.37
60.45
80.82
90.83
CLIP (err-Zee-Sh
10.59
8.69
10.80
9.52
14.32
20.21
UnivScarch
VInVI.-DPRR
37.19
CLIP-DPR
48.83
46.32
49.34
49.11
69.84
86.43
UniVL-DR
62.40
62.69
61.22
Table 1: Multi-Modal Retrieval Performance. VinVL-DPR, CLIP-DPR, NQ-DPR and BERT-DPR trained with in-batch negatives, while NQ-ANCE is trained with hard negatives.
Similar to UniVL-DR, BM25 learns universal textual representations for image/text documents and shows strong ranking effectiveness. To build a driidd-andd-ooquee system, CLIP-DPR to implement text-text and text-image retrievers and then fuse the results from different retrievers. With the help of oracle modality routing, the divide-and-conquee system shows better ranking results and recalls more relevant documents than BM25. Nevertheless, this system shows distinct performance when using the uni-modal rank reciprocals to route queries, showing the challenge of fusing retrieval results in divide-and-conuue with this problem by learning universal representations for queries and multi-modality documents, which unifies the multi- modality relevance modeling and retrieval result fusion. Thanks to our multi modaitty training strategies, UniVL-DR achieves more than 10% improvement on multi-modal retrieval than CLIP-DPR, The following experiments further explore how UniVL-DR learns universal representations for multi-modal retrieval and bridges the gap between images and texts.
CLIP-DPR and UniVL-DR can deal
6.2 ABLATION STUDIES
The ablation studies are conducted to study model performance on multi-mode retrieval. And we also evaluate the effectiveness of UniVL-DR on both text-text and text-image retrieval tasks, which aims showing the influence of multi-modal learning on these single/cross modality retrieval tasks.
As shown in Table 2 we evaluate the retrieval effectiveness of different vision-language models, VinVLL and CL.R-DPR. They are trained with in-batch negatives on text-text/image and multi- modal retrieval tasks. In the single/cross modality setting, we fine-tune vision-language models with a group of queries that only contain related documents in text modality or image modality. Our multi-modality training setting uses all queries to train these vision-language models and equally samples in-batch negatives from the documents of different modalities.
For both CLIP-DPR and VinVL-DPR, image captions usually more effective to represent image documents than figure features, which demonstrates the difficulty in understanding figure semantics with only figure pixels. Thus, UniVL.-DR tries verbalize the figure features by extracting the objects that appear in the figure and describing the figure facts among detected objects (Zhang et al., 2021). The image verbalization results paraphrase picture pixel facts natural language and help to enhance the textual representations of images by expanding image verbalization results to image captions. As result, UniVL-DR uses such an enhanced text representation for image documents and then employs the same module to encode text information of image documents and text documents. helps build universal representations for multi-modalin documents by breaking the modality boundary and fully using additional training signals from different modalities, making UniVL-DR achieve the best retrieval performance on multi-modal retrieval among all baseline models.
UniVL-DR also shows its advantages by outperforming all baseline models on both text-text and text-image retrieval tasks, demonstrating that multi-modality modeling indeed benefits single/cross modality retrieval. In the multi-modal retrieval setting, CLIP-DPR is converted from text-text retriever to a multi-modal retriever after adding figure features. CLIP-DPR achieves better perfor- mance the text-image retrieval task than CLIP-DPP w/o figure feature, which illustrates that models distinguish related image
image features provide additional signals to help documents. On the contrary, the multi-modal retrieval performance of CLIP-DPR is decreased,
7

---

Published as a conference paper at ICLR 2023
Model
Retrieval Performance
Sampling
Retrieval Performance
Retrieved Image (%)
Text
Image
Multi
Text
Image
Muln
Cinsle/Cruss Modulity Retrievers
In batch Training
BERT-DPR
37.09
52.34
CLIP-DPR (Random)
5177
60.61
48.83
26.82
VinVI.DDPR w/o caption
3.67
Balanced In batch
52.24
59.99
49.88
30.35
VinVL-DPR w/o fig feature
51.56
Hard Negative Training
Only Texts
54922
52.88
36.26
91,74
VinVL-DPR
25.00
48.68
Only Images
55.85
66.51
33.49
1.97
CLIP-DPR w/o caption
17.74
Texts & Image
65.15
61.64
49.53
CLIP-DPR w/o fig. feature
58.17
Text & Images
57.86
66.23
61.20
47.88
CLIP-DPR
$2.57
59.95
ANCE (Random)
59.85
64.80
61.72
50.01
Universal Multi-Modal Retrievers
Balanced In-butch
60.38
65.21
62.29
49.11
VLDVL feature
29.01
46.55
36.13
VinVL-DPR
29.95
49.65
38.14
Table 3: Effectiveness of Different Hard Negative
CLIP-DPR w/o fig feature
51.47
57.36
50.33
Training Strategies. These hard negative trained continuously fine-tuned with different numbers of hard negatives. The MRR @ 10 score and the image ratio of the Top 10 retrieved candidates shown to evaluate the modality preference.
CLIP-DPR
51.75
60.61
48.83
models start from in-batch trained ones and are
UniVT.DDR
60.72
65.57
62.40
Table 2: Retrieval Performance of Different Ab- lation Models. MRR @ 10 is used as the evalua- tion metric.
Text Image Query
(a) In-batch.
(b) Text Hard Neg.
(c) Image Hard Neg.
(d) Balanced Hard Neg.
Figure 4: Embedding Space Visualization Different Training Strategies. We randomly sample 30 image documents, 30 text documents and 30 queries. And then we visualize their embeddings learned by CLIP-DPR and the continuously trained models with different hard negative sampling strategies.
showing that CLIP-DPR fails to fuse retrieval results from different modalities. UniVL-DR uses a modality-balanced hard negative training strategy learn universal representations for queries and documents, which deals with the challenge of fusing retrieval results, helps to achieve more gains on the multi-modal retrieval task, and enhances the modality disambiguation ability.
6.3 EFFECTIVENESS OF BALANCED HARD NEGATIVE SAMPLING
In this experiment, we study the training strategies of UniVL-DR that are used in learning universal multi-modality representations and show the effectiveness of different negative sampling methods.
As shown in Table with different hard negative sampling methods, and show their performance on different retrieval tasks. Our experimental results show that the in-batch trained models prefer return text documents than image documents as the ranking results, even the anwerale queries take a larger portion (about 51.6%) in the training data. illustrates that training multi- modality retrievers with modallty-unbalanne negatives usually leads to undesired modality bias during retrieval.
start from the multi-modal retriever CLIP-DPR, continuously fine-tune it
Then we continuously train CLIP-DPR with hard negatives sampled from top-retrieved multi-modality results from CLIP-DPP and significantly improve its retrieval performance in all testing scenarios. Our modality-balanced hard negative sampling strategy achieves the best retrieval performance among all negative sampling methods, showing its important role in building universal multi-moda retrieval model. Compared with ANCE (Random), our modality-balancee sampling strategy mitigates the modality variance during contrastive training and provides more useful signals train the modality disambiguation ability universal multi-modal retrievers.
Finally, visualize the embedding space of different retrieval models in Figure 4 After training with modality- balanced hard negatives, UniVL-DR learns a more uniform and effective embedding space for multi-modal retrieval. In this embedding space, both text and image documents
8

---

Published as a conference paper at ICLR 2023
In-batch Training
Hard
Neg Training
Model
Text
Image
Muti
Text
Image
Muti
UniVL-DR
51.75
60.61
48,83
60.58
65.21
62.29
Verbalized Caption
51.51
60.57
49.49
59.86
65.87
62.24
w. Verbalized Query
52.14
59.72
50.21
60.72
65.57
62.40
Table 4: Performance of Multi-Modality Retrieval Models with Different Image Verbalization Methods, All models are evaluated with MRR@ 10.
assigned in different areas of the embedding space, and queries routed different areas for returning documents from corresponding modalities. As shown in Figure 4[b and Figure 4
when the retrieval models are only trained with hard negatives of text and image documents, the query embeddings are concentrated and respectively assigned closer to the areas of image and text documents. demonstrates that multi-modality retrieval model usually overfits the training signals of in-batch majority modality to win a lower contrastive loss during training. UniVL.-DR alleviates this problem by balancing the modalities of hard negatives in contrastive training.
6.4 BRIDGING MODDLIYY MATCHING WITH IMAGE VERBALIZATION
UniVL-DR uses image verbalization methods generate matched captions related queries to bridge the modality gap between texts and images. In this experiment, we show the effectiveness of different image verbalization strategies on text-text, text-image, and multi-modal retrieval tasks.
As shown in Table 4 our image verbalization methods demonstrate their ability on enhancing the text representations image documents by achieving better text-image retrieval results. These image verbalization methods aim to generate informative text clues to help retrievers distinguish the query- related image documents the text space. Then the text-text and multi-modal retrieval performance is also improved with the help of verbalized captions queries, showing the effectiveness of our image verbalization methods in bridging the modality gap between images and texts and benefiting the single modality tasks using additional training signals from different modalities.
Compared with verbalized captions, our query verbalization method aligns the necessary semantics in captions, e.g. mentioned entities, with image objects and verbalizes the figure pixels with the help of caption semantics. Enhancing image representations using verbalized queries usually achieves better retrieval effectiveness than using verbalized captions. It showcases that our query verbalization method can provide more meaningful text clues for relevance modeling and multi-modality learning. Moreover, some additional experiments verbalization methods. We first show the relationship between the effectiveness of verbalized queries and manual caption lengths in Appendix A.4 and then conduct some case studies in Appendix A.5 explore the characteristics of different image verbalization methods.
provided to study the effectiveness of different image
7 CONCLUSION
This paper proposes UniVL-DR, which models singe/cross modality matching and retrieval result fusion one universal embedding space. UniVL-DR proposes an effective multi-modality training strategy to learn universal representations for queries and documents, which breaks the modality boundary between vision and language, and helps to achieve state-of-the-art multi-modal retrieval performance. Our experiments show that UniVL-DDR can bridge the modality gap with image verbalization technologies and avoid overfitting the training signals of one modality by optimizing retrievers with modality-balance hard negatives.
ACKNOWLEDGMENTS
This work is supported by Beijing Academy of Artificial Intelligence (BAAI), the Natural Science Foundation of China under Grant No. 62206042, No. U1811261 and No. 62006129, the Fundamental Research Funds for the Central Universities under Grant No. N2216013, China Postdoctoral Science Foundation under Grant No. 2022M710022, and National Science and Technology Major Project (22019-V-0002-0069),

---

Published as a conference paper at ICLR 2023
REFERENCES
George Awad, Asad A Butt, Keith Curtis, Jonathan Fiseus, Afzal Godil, Yooyoung Lee, Andrew Delgado, Jesse Zhang, Eliot Godard, Baptiste Chocot, al. 2021. Treovid 2020: comprehensive campaign for evaluating video retrieval tasks across multplee application domains
Payal Bajaj, Daniel Campos, Nick Craswell. Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, al. 2016. Ms marco: A human
generated machine reading comprehension dataset
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. 2022. Effective condi-
fioned and composed image retrieval combining clip-based features] In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022,
pages 21434-21444 IEEE.
Patrick Cavanagh. 2021. The language of vision * Perception, 50(3):195-215,
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. 2022. Webqa: Multihop and multimodal qa In Proceedings of CVPR.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Uniter: Learning universal image-fext representations
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unilying viston-and-language tasks via text generation In Proceedings of ICML, pages 1931-1942.
Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z Wang. 2008. Image retrieval: Ideas, influences, and trends of the new age ACM Computing Surveys (Csur), (2):1 -60.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding In Proceedings of MACCI-HLT, pages
4171-4186.
Sounak Dey, Anjan Dutta, Suman K. Ghosh, Ernest Valveny, Josep Lladós, and Umapada Pal. 2018. Learning cross-modal deep embeddings for multi-object image retrieval using text and sketch In 24th International Conference on Pattern Recognition, ICPR 2018, Beijing, China, August 20-24, 2018, pages 916-921. IEEE Computer Society.
Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. 2022. A survey of vision-language tre-trained models
Anjan Dutta and Zeynep Akata, 2020. SSeaantiallly tied paired cycle consistency for any-shot sketch-based image retrieval Int. Comput. Vis., 128(10):664-22033
Hugo Jair Escalante, Carlos A Hérnadez, Luis Enrique Sucar, and Manuel Montes. 2008. Late
fusion of heterogeneous methods for multimedia image retrieval In Proceedings of the 1st ACM
international conference on Multimedia information retrieval, pages 172-179.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. 2020. Large-scale adversarial training for iision-and-languaal representation learning. In Proceedings of NeuriPS.
M Grubinger. P Clough, A Hanbury, and H Müller. 2008. Overview of the imageclef 2008 photo graphic retrieval task In Working Notes of the 2008 CLEF Workshop. Aarhus, Denmark.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. 2020. A survey on visual transformer
Darryl Hannan, Akshay Jain, and Mohit Bansal. 2020. Manymodalga: Modality disambiguation and QA over diverse inputs, In Proceedings of AAAI, pages 7879-7886.
Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang. Lei Zhang. Jianfeng Gao, and Zicheng Liu. 2020. Vivo Surpassing human performance in novel object captioning with visual vocabulary pre-training
10

---

Published as a conference paper at ICLR 2023
Xin Huang, Yuxin Peng, and Zhang Wen. 2021a. Visual-textual hybrid sequence matching for joint reasoning IEEE Trans. Cybern., 51(1))339222
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. 202 b. Seeing out of the box: End-to-end pre-training for vision-language representation learning In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12976-12985.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus IEEE Transactions Big Data.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering In Proceedings of EMNLP, pages 6769-6781.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. 2021. Transformers in vision: A survey ACM Computing Surveys (CSUR).
Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: "ssoo-and-languaal transformer without convolution or region supervision In Proceedings of ICML, pages 5583-5594.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research Transactions of the Association for Computational Linguistics, pages 452-466.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida I. Wang, and Luke Zettlemoyer. 2020. Pre-training via paraphrasing In Proceedings of NeurIPS.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021a. UNIMO: Towards -nified-modal understanding and generation via cross- cross-mode modal contrastive) learning In Proceedings of ACL, pages 2592-2607.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang. Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, et al. 2020. Oscar: Object-semantic aligned pre- pre-training for vision language tasks In European Conference Computer Vision, pages 121-137. Springer.
Yizhi Li, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2021b. More robust dense retrieval with contrastive dual learning In Proceedings of the 2021 ACM SIGIR International Conference Theory of Information Retrieval, pages 287-296.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for iision-and-languaal tasks In Proceedings of NeurlPS, pages 13-23.
Long Mai, Hailin Jin, Zhe L. Lin, Chen Fang, Jonathan Brandt, and Feng Liu. 2017. Spatial-semantic image search by visual feature synthesis) In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1121-1130. IEEE Computer Society.
Tao Mei, Yong Rui, Shipeng Li, and Qi Tian. 2014. Multimedia search reranking: A literature survey ACM Computing Surveys (CSUR). (3):1-38.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision In Proceedings of ICMI., pages 8748-8763.
11

---

Published as a conference paper at ICLR 2023
Patsorn Sangkloy, Wittawat Jitkrittum, Diyi Yang, and James Hays. 2022. A sketch is worth thousand words: Image retrieval with text and sketch In European Conference on Computer Vision, pages 251-267.
Christopher Sciavolino. Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centric questions challenge dense retrievers In Proceedings of EMNLP. pages 6138 6148.
Hrituaj Singh, Anshul Nasery, Denil Mehta, Aishwarya Agarwal, Jatin Lamba, and Balaji Vasan Srinivasam. 2021. MIMOQA: Multimodal imput multimodal output question answering In Proceedings NAACL-HLT, pages 5317-5332.
Amit Singhal et al. 2001. Modern information retrieval: A brief overview IEEE Data Eng. Bull., (4):35-43.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. VL-BERT pre-training of generic iisuul-linguistic representations In Proceedings of ICLR.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi. and Jonathan Berant. 2021. Mullimodalga: complex question answering
over text, tables and images In Proceedings of ICLR.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers In Proceedings of EMNLP, pages 5100-5111.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models) In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Shagun Uppal, Sarthak Bhagat, Devamanyu Hazarika, Navonil Majumder, Soujanya Poria, Roger Zimmermann, and Amir Zadeh. 2022. Multimodal research in vision and language: A review of current and emerging trends Information Fusion, pages 149-171.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak supervision
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021a. Approximate nearest neighbor negative contrastive learning for dense text retrieval| In Proceedings of ICLR.
Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick S. H. Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. 2021b. (nswering
complex open-domain questions with multi-hop dense retrieval In Proceedings of ICLR.
Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang. Wenming Xiao, and Fei Huang. 2021. -LP: End-to-end vasion-language pre-training enhanced by visual learning In Proceedings of ACI. pages 503-513.
Sangwoong Yoon, Woo- Young Kang, Sungwook Jeon, SeongEun Lee, Changjin Han, Jonghun Park, and Eun-Sol Kim. 2021. Image-to-image retrieval by learning similarity between scene graphs) In Proceedings of AAAI, pages 10718-10726.
Woo-Young
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions Transactions of the Association for Computational Linguistics, pages 67-78.
Shi Yu, Zhenghao Liu, Chenyan Xiong, Tao Feng, and Zhiyuan Liu. 2021. Few-shot conversational
dense retrieval In Proceedings of SIGIR.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2021. Optimizing dense retrieval model training with hard negatives) In Proceedings of SIGIR.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvl: Revisiting visual representations in vision-language models] In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588.
12

---

Published as a conference paper at ICLR 2023
#Queries
Modality
#Documents
Task Category
Train
Dev
Test
Image
389,750
16,400
2,554
2,511
Text- Image Retrieval
Text Total
787.697
15,366
2,446
2,455
Text-Text Retrieval
1,177,447
31,766
5,000
4,966
Multi-Modal Retricval
Table 5: Data Statistics of Vision-Language QA Benchmark WebQA in Open- Open-Domain Setting.
A APPENDIX
A.1 DATA STATISTICS
multi-hop and multi-modal open domain question answering dataset WebQA (Chang et al. 2022 is used in experiments. The dataset contains images and passages that crawled from the general Web and Wikipedia. In our experiments, we randomly sample 5,000 queries from the original training set of WebQA the development set for evaluation. All data statisties are shown in Table To build an open-domain benchmark, we collect 389,750 images and 787.697 texts as multi-modal retrieval sources. The image collection contains all images collected by the WebQA dataset, while the text collection contains all relevant passages of all 41,732 queries, which are Wikipedia snippets selected by matching noun chunks in the queries (Chang et al. 2022).
A.2 ADDITIONAL EXPERIMENT DETAILS
This subsection describes additional implementation details. In our experiments, we employ two pretrained vision-language models, VL (Zhang et al. 2022) and CLIP (Radford et al. (021). and the pretrained language model, BERT (Devlin et al. (2019) to implement different retrieval models.
VinVL-DPR. For VLnVL variant models, first detect the image objects and extract corresponding region features following Vin Then we concatenate image captions and image region features inputs feed into VIn models and get the image representations. We initialize VIn with the checkpoint trained on the MSCOCO image retrieval task and continuously train the model on the WebQA dataset with in-batch negatives. During training, we set the batch size 32, learning rate=2e 5, accumulate step as 1, and max training epoch to 30. We truncate the queries, image captions, text documents, and image region features with max lengths of 70, 70, 200, and 50.
CLIP-DPR. For training CLIP-DPR, we start from the ViT-B/32 version of CLIP and continuously train CLIP the WebQA dataset with in- in-batch negatives. We truncate texts with the max length of 77 and set accumulate step as 1, batch size to 64, learning rate=āe 6, max training epoch to 20, and the temperature hyperparameter T 0.01. The cosine annealing strategy is used schedule the learning rate during training.
BERT-DPR. We initialize our retriever with the bert-base-uncased checkpoint, which is provided by Hugginface Transformer During training, we set the batch size to 32, learning rate=5e 5, accumulate step as 1, and max training epoch to 30. We truncate the queries, text documents, and image captions with max lengths of 70, 200, and 70.
NO-DPRNQ-ANCE. NQ-DPR and NQ-ANCE start from the NQ-trained DPR model (Karpukhin ct al. 2020), which uses dual encoder architecture encode queries and documents. All experi- mental settings keep the same with BERT-DPR. Besides, NQ-ANCE is tuned with the hard negatives sampled from the Top 100 retrieved candidates of NQ-DPR Xiong et al. (2021a).
A.3 EXPERIMENTAL DETAILS OF IMAGE VERBALIZATION
The image verbalization models are used to generate potentially matched captions or related questions for an image. Our experiments start from the image caption generation model, which is trained on the MSCOCO image caption task (Zhang et al. 2021 to generate related captions or queries to verbalize images.
benchmark _
http:://giitubb.ccm//uggggggggi
13

---

Published as a conference paper at ICLR 2023
UniVL-DR
UniVL-DR w. Verbalized Query
76.0
75.3
75.0
74.5
MRR@10
74.0
73.7
73.6
73.0
72.6
72.2
72.0
71.0
70.0
Short
Medium
Long
Raw Caption Lengths
Figure 5: Performance of Multi-Modal Retrieval Models with Different Caption Lengths. The query lengths of short, medium and long groups are in the sections of [0, 10), [10, 20), and [20, +o0).
We can first directly generate image-related captions as the image verbalization results using the image caption model provided by VinVL (Zhang et al., 2021). As shown E.q. 6 we first detect the image objects in the images and then feed the predicted classes and region features of detected objects to the VInVL. model. In our experiments, we fix the parameters of the VinVL-based image caption model and generate the caption for each image.
During generating image-related queries, shown in E.q. we concatenate the image-related query, image caption, and image regional features as the input. We continuously train the VinVL-based image caption model by randomly masking the tokens queries, and optimizing vision-language models to fill in the masked positions. Different from image caption models, query generation method tries to align the semantics in image captions and image pixel features instead of mapping the predicted classes and regional image features of detected objects, which can help vision-language models better understand the image semantics (Huang et al.) 2021a).
the generated tokens up to 20 and the beam size to 5. We truncate the queries, image captions, and image region features with the max lengths of 40, 30, and 50, respectively. The mask probability is set to 0.15. More experimental details be referred to Zhang et al. (2021).
During training and inference,
A.4 IMAGE VERBALIZATION PERFORMANCE WITH DIFFERENT CAPTION LENGTHS
In this subsection, we evaluate the multi multi-modal retrieval performance of UniVL-DR with different verbalized queries. Specifically, we evaluate the effectiveness of image-verbalized queries in the multi-modal retrieval task. These image-verbalized queries are generated with the image documents with different manual caption lengths.
We group the testing examples into three categories according to the manual caption lengths of the image documents and calculate the average MRR @ 10 score for each group. The ratios are 42.33%, 36.84%, %, and 20.83% % of the short, medium, and long caption length groups.
As shown in Figure 5] the experimental results show that our query generation method mainly helps to improve the retrieval effectiveness on the queries of short length and medium length, illustrating that these generated queries can provide some crucial textual clues in image representations of shorter captions. These expanded text clues help retrieval models better understand image semantics, more effectively represent images via enhanced textual information, and conduct ross-modality matching more easily. Moreover, the queries in the medium caption length group achieve the best performance, because the image captions of medium lengths can cover more necessary text clues for generating more informative verbalization results.
14

---

Published as a conference paper at ICLR 2023
Figures
Text
Query: Does Minnetonka Khododencion
petals cup shape?
Manual Cuption: 2020-05-00 17 05 Minnetonka Rhadodendron flower along Tranquility Court
the Franklin Farm section of Oak Hill. Fairfax County. Virginia Minncionka Khododendron flower along Tranquility Court in the Franklin Farm section of Oak Hill, Fairfax County, Virginia green leaves und purple flowers.
Verbalized Capton: purple
Verbalized Query: what shape are the petalss of the minnetonka chodedendron flower?
Query: heads of
women covered in traditional elothing?
Manual Caption: Iranian family gathered logether wearing traditional clothes Nishapar-
Nownuz2014 franian family, gathered together wearing traditional clothes
Verbalized Caption: group of people in costumes standing in park.
Verbalined Query:
many people are wearing, hats the group icanian family members?
Query: At the 1928 Amstendam Olympics, what is the maximum number of buttons that you can get the Egyplian
uniform?
Manual Caption: Eggyptische atleten hij OS Amstendam 1928 Egyptian Olympic athletes, Amstendam 1928
(8941436605) htp://www www.sameetttdlpeeeee
al/commpoent//ppiooo Verbalizad Caption: group of men in suits and hats standing in a field.
memorix ... type cape? Query: What water related object is sitting in fromt of the Tarre del Kelaj?
Verbalized Query: did all the in the egyplian olympic athletes
Caption: Tore del Reloj Pluza Colén de Antofagasta (1)
Verbulized Caption: water fountain in front of a clock tower.
Verhalized Query: the fountain front the clock
plaza de reloj taller than
Query: What color the facade of bakery Sattin Fils Rethel. Frunce!
Manual Caption: Rethel-IRR 488 boulangerie Sattin-01
Verbalized Caption: red storefront on city street conter..
Verbalized Query: how many potted plants are outside of the boulangeric sattin?
Query: Does the Durham Cathedral in Linlland have any trees outside of I??
Cothedral, Durham, County Durham, England Verbulized Caption: a large building with two towers and a tree in front of
Caption: Durham Cathedral, July 2014 (04)
Verhalized Query: there any
durham cathedral which taller than the cathedral?
Table 6: Image Verbalization Cases. We highlight the matching phrases between user queries and manual captions or image verbalization results.
A.5 CASE STUDIES ON DIFFERENT IMAGE VERBALIZATION METHODS
This experiment shows some image verbalization cases in Table 6 We randomly sample queries that be answered by image documents and show the manual captions, verbalized captions, and verbalized queries of the image documents.
Overall, these cases can be categorized into two groups according to the lengths of manual captions. The first three cases are longer and more informative to describe the image facts among mentioned objects, which can be directly used in text-image relevance modeling. On the contrary, the manual captions in the last three cases are written by the most representative entities that appeared in the images, making difficult to distinguish the related images only according to these manual captions.
UniVL-DR employs two image verbalization methods to enhance the textual semantics of images. Generating image captions is the most intuitive way to paraphrase images with some pre-defined classes of image objects. Nevertheless, these object classes are too general and may be uninformative to provide semantics for matching, because the specific entities are critical to retrieving related documents in a question-answering system (Sciavolino al., 2021). Different from these verbalized captions, the verbalized queries are usually more informative and meaningful and specify the image objects by copying entity names from the manual captions, such as the names of persons, places, and buildings. These entities can be directly matched with the given queries, which benefits cross-modality matching and helps mitigate the modality gap between images and texts.
A.6 MULTI-MODAL RETRIEVAL WITH DIFFERENT IMAGE REPRESENTATION COMBINATION METHODS
In this subsection, we conduct experiments to show the effectiveness of different methods in combin- ing the representations of image captions and image features.
15

---

Published as a conference paper at ICLR 2023
Model
MRR@II 10
MRR@20
NDCG@10
NDCG@20
CLIP-DPR (Concatenation)
23.18
23.61
20.74
22.94
CLIP-DPE (Outer Product)
39.89
40.47
37.40
40.22
CLIP -PP (Sum)
48.83
46.32
49.34
49.11
Table 7: Multi-Modal Retrieval Performance with Different Representation Combining Methods. We concatenate, dot product and sum the representations the image captions and image features to build different models.
Model BM25
MRR@T
NDCG@5
NDCG@
NDCG@2
41.14
46.24
49.60
51.72
BM25 & CLIP-DPR
7.43
19.47
41.58
44.69
BM25 & CLIP-DPR (Oracle Modality)
46.19
54.20
58.18
60.45
NQ-ANCE
34.21
39.16
42.05
43.83
CLIP-DPR
34.92
42.30
46.32
49.11
UniVL-DR
47.56
55.28
59.32
61.22
Table 8: Additional Evaluations on Multi-Modal Retrieval Models.
As shown in Table 7 we evaluate the effectiveness different combination models using CLIP-DPR. We concatenate, dot product (outer product), and sum the representations of image captions and image features to conduct three models: CLIP-DPR (Concatenation), CLIP-DPR (Outer Product), and CLIP-DPE (Sum).
CLIP-DPR (Sum) shows its effectiveness by achieving the best performance among all baselines. The sum operation is commonly used semantic combination method, which is also used BERT (Devlin et al. 2019) to combine token embedding and position embedding. On the contrary, the concatenation operation usually regards the representations of image captions and image features as subvectors and separates them into subspaces, making hard to learn the semantics of image documents. On the other hand, the outer product operation conducts orthogonal representations during combining representations, which is not typical combination method.
A.7 ADDITIONAL EVALUATIONS ON MULTI-MODEL RETRIEVAL
In our experiments, we follow previous widely used retrieval benchmarks, MS MARCO (Bajaj et 2016) and BEIR (Thakur et al. 202T), and use NDCG@ @ 10/20 and MRR@ 10/20 to show the retrieval effectiveness of different retrieval models. MRR scores and NDCG scores are calculated by the MS MARCO official scripts and TREC's evaluation too
As shown in Table we also conduct some evaluations show the retrieval performance of higher- ranked candidates using MRR @ and NDCG@5. UniVL-DR also shows strong effectiveness by outperforming BM25 and CLIP-DPR with more than 6% improvements. Notably, UniVL-DR even shows better retrieval effectiveness than the BM25 & CLIP-DPR (Oracle Modality) model, which in an ideal setting. It supports our claim that multi-modality modeling can also benefit single/cross pagge//ross-modality tasks.
A.8 EXAMPLES OF HARD NEGATIVES
In this subsection, we randomly sample queries and show hard negatives in Figure 6 which are top-ranked documents using our CLIP-DPP model.
In the first case, when we ask "Is there greenery at Centennial Olympic Park?", the CLIP-DPR model can provide some image and text documents, which are regarded as hard negatives to continuously train dense retrievers. The negative images are about buildings, lawns, and trees, but these objects are not located at Centennial Olympic Park. Evidently, these negative images are on-topic with "greenery" but are not related to the given query. Training dense retrievers with these hard negatives can better teach retrievers to distinguish the subtle difference among these confusing images.
16

---

Published as a conference paper at ICLR 2023
Query1: is there greenery Centennial Olympic Park?
Golden Doc
Image Doc1
Image Doc2
Image Doc3
Text Doc1: The city's permanent memorial the 1996 Olympics is Centennial Olympic Park, which was built as focal point for the Games.
Text Doc2: Built as legacy of the 1996 Olympic Games, Centennial Olympic Park, located on 21-acre (85,000
m2) area of Downtown, is the largest downtown park in the United States developed in the last 25 years.
Text Doc3: Centennial Olympic Park was designed as the "town square" of the Olympics, and thousands of spectators had gathered for late concert and merrymaking.
Query2: The genus name Columba is Latin word means dove what, which are particularly fond of roof spaces?
Image Doc1
Image Doc2
Image Doc3
Image Doc4
Golden Doc: Pigeons particularly fond of roof spaces. These often contain tanks. Any water tank cistern on roof must, therefore, be secured and sealed off keep pigeons out them. The popularity of nesting area does not seem to be affected by the pigeons' population density.
Text Docl: The common name "columbine" comes from the Latin for "dove", due to the resemblance of the inverted flower to five doves clustered together.
Text Doc2: The genus name is Late Latin; falco derives from falx, falcis, a sickle, referring to the claws of the bird. The species name columbarius is Latin for "af doves" from "columba", "dove
Figure 6: Hard Negative Cases of Multi-Modal Retrieval. These hard negatives are top-retrieved multi-modality documents from CLIP-DPR.
For both cases, the hard negatives from different modalities showcase some necessary semanties, which are needed by the retrieval model to find relevant information. For example, text documents in Case can provide background knowledge of the Olympic Games and Centennial Olympic Park; image documents in Case 2 supply the "doves" semanties from the visual modality. These informative documents from different modalities can provide sufficient clues to guild dense retrievers to learn necessary semanties during contrastive training.
17


Reviews:
[{"review_id": "ufUAFlpqi8", "reviewer": "Reviewer_FZm3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a universal vision-language dense retrieval with two techniques, using modality-balanced hard negatives for optimization and bridging the modality gap by the image verbalization method.  Experiments are conducted on the built open-domain dataset from WebQA and compared to existing models for single-modality, divide-and-conquer, and multi-modal retrieval and ablation studies.", "summary_of_the_review": "The paper proposes to build a universal embedding space for single and cross-modality matching for retrieval tasks with a multi-modality training strategy for queries and documents and try to mitigate the modality boundary between vision and language. I have some concerns (see weakness). If my concerns are addressed, I will consider increasing the score appropriately.\n", "strength_and_weaknesses": "Strength:\nReasonable motivation and a combination approach. \n\nWeakness:\nThe proposed image verbalization method simply adopts the existing approach to generate matched captions or queries according to pictures. The performance of the combined method may be bounded by the methods of proposal generation and caption generation models. Is there a strong reason the authors choose VinVL (Zhang 2021) and MLM (Devlin 2019)? Or, it can be any arbitrary methods to achieve this.  In addition, what is the definition of hard negatives,  some visualization examples may help? What will it happen if the proposed method uses non-hard negatives compared to other methods? Are those good results from the hard negatives? \nCan the authors explain why to choose NDCG@10, instead of @1, @5, as we know that most of the time the top retrieved results are the most important to queries?  When good results on MRR@10 and @20, why BM25 & Clip-DPR Rec@20 is better a lot in Table 2? Does that imply the UniVl-DR + BM25 method can provide even better? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A6s-LiCPtqY", "reviewer": "Reviewer_qnyp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents universal vision-language dense retrieval model, which builds a unified model for multi-modal retrieval. The proposed model encodes queries and multi-modal resources in an embedding space for searching candidates from different modalities. To learn a unified embedding space for multi-modal retrieval, this work has come out with (1) universal embedding optimization strategy, which contrastively optimizes the embedding space using the modality-balanced hard negatives; (2) image verbalization method, which bridges the modality gap between images and texts in the raw data space.", "summary_of_the_review": "Taskwise, the paper is interresting. It is well presented and also the reported results are also interesting. Howevere, I have found the novelty to be limited.", "strength_and_weaknesses": "**Strengths**\n\n(1) The paper addresses a very interesting tasks of visual language research.\n(2) The paper is reasonably well presented and written in good English.\n(3) Experimental results are quite encouraging.\n\n**Weaknesses**\n\n(1) The novelty is not clear. I was expecting it to be mentioned or specified in the introduction of the paper for better understanding on the contribution of the paper. Furthermore, I am aware of the following existing papers which are also considering multi-modal embedding for different tasks, such as text [1], sketch [4] and both [3,5] based image retrieval, text and image topic modelling [2] etc.\n\n(2) It is not very clear why a simple summation of the representation of image caption and image feature works well for combining two very different types of modalities. I wonder if any other combination (concatenation, outer product etc.) has been ablated or could be interesting to try.\n\n(3) Currently a lot of progress has been made on text, sketch and both based image retrieval which should also be included and discussed within the literature of cross-modal retrieval.\n\n(4) It is not clear why [CLS] and [SEP] tokens special and different.\n\n(5) I think it is worth defining universal representation learning and image verbalization procedure.\n\n(6) In the experimental results table, citation of the baselines and SOTA methods should be given for readability.\n\n[1] Mai et al., Spatial-Semantic Image Search by Visual Feature Synthesis, CVPR, 2017.\n[2] Gomez et al., Self-supervised learning of visual features through embedding images into text topic spaces, CVPR, 2017.\n[3] Dey et al., Learning Cross-Modal Deep Embeddings for Multi-Object Image Retrieval using Text and Sketch, ICPR, 2018.\n[4] Dutta and Akata, Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-based Image Retrieval, IJCV, 2020.\n[5] Sankgkloy et al., A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch, ECCV, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "7nJR3COJk8T", "reviewer": "Reviewer_vCSr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a unified model for multi-modal retrieval. The proposed method consists of two techniques, the universal embedding optimization strategy for contrastively optimizing the embedding space, and the Image verbalization method for bridging the modality gap.", "summary_of_the_review": "This paper is marginally above the acceptance threshold of ICLR2023.\n\n=======================================================================================================\n\nI have checked the comments of the other reviewers and the feedback from the authors carefully, and I would like to keep my score. The authors have answered my concerns, especially the mathematical explanation of the universal embedding optimization strategy. ", "strength_and_weaknesses": "The main strengths of this paper can be concluded as follows:\n\n1. This paper proposes a unified model for multi-modal retrieval, which demonstrates that universal multi-modal search is feasible compared with the divide-and-conquer pipeline with a united model, and also benefits cross-modality tasks.\n\n2. The proposed method bridges the modality gap by optimizing the vision-language embedding space using hard negatives, and aligning the semantics of image captions and figure pixels, which achieves state-of-the-art performance on WebQA datasets.\n\nThe main weaknesses of this paper are as follows:\n\n1. There is still a gap between the proposed method and SOTA method in some evaluation tasks on the WebQA Dataset. The authors should provide more corresponding analyses.\n\n2. An important assumption in this paper is that the universal embedding optimization strategy can enable the optimization of the universal embedding space with modality-balanced hard negatives. The authors should provide a mathematical explanation to make the assumption more convincing.\n\n3. Relevant works such as [a] [b] should be compared to make this paper more comprehensive.\n\n[a] Effective Conditioned and Composed Image Retrieval Combining CLIP-Based Features. CVPR2022.\n[b] Visual–Textual Hybrid Sequence Matching for Joint Reasoning. TCVB2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster