Cross-Modal Retrieval Augmentation for Multi-Modal Classification
Authors: Shir Gur
Natalia Neverova
Chris Stauffer
Ser-Nam Lim
Douwe Kiela
Austin Reiter
Venue:  (2021)
Keywords: Multi-Modal
VQA
Retrieval

Abstract: Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. Here, we explore the use of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA). First, we train a novel alignment model for embedding images and captions in the same space, which achieves state-of-the-art image-caption retrieval performance w.r.t. similar methods.
Second, we show that retrieval-augmented multi-modal transformers using the trained alignment model
significantly improve results on VQA over strong baselines.
We further conduct extensive experiments to establish the promise of this approach, and examine novel applications for inference time such as hot-swapping indices.

Under review as a conference paper at ICLR 2021
CROSS-MODAL RETRIEVAL AUGMENTATION MULTI-MODAL CLASSIFICATION
FOR
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks natural lan- guage processing. Here, we explore the of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA). First, we train a novel alignment model for embedding images and captions in the same space, which achieves state-of-the-ar image-captin retrieval performance w.r.t. similar methods. Second, we show that retrieval- augmented multi-modal transformers using the trained alignment model signifi- cantly improve results on VQA over strong baselines. reporting state-of-the-eee performanee. We further conduct extensive experiments to establish the promise this approach, and examine novel applications for inference time such as hot- swapping indices.
INTRODUCTION
Neural networks augmented with non- parametric retrieval components have recently shown impres- sive results in NLP (Khandelwal et al. 2019 Guu et al., 2020 Lewis et al.) 2020 Izacard & Grave, 2020). In this work, we train state-of oftthc--rt image-caption alignment model and utilize it in var- ious aetrieval-augmmnter multi-modal transformer architectures, achieving state-ef the-art perfer- mance-on visual question answering (VQA) significant improvement over the baselines, including the winner of the VQA 2.0 2020 challenge.
Retrieval components are promising because they allow for easy revision and expansion of their memory, as compared to their parametric, pre-training counterparts. They provide more inter- pretability, as well as direct factual consistency with trusted knowledge sources. In the multi-modal setting, retrieval augmentation allows for leveraging the strengths of text-based models as evi- denced by the strong performance of BERT-based models in iision-and-lanuuage (Lu et al. 2019
Li et al. 2019b Kiela et al. 2019) via cross- translation from images to text. Being able to seamlessly "hot swap" knowledge sources without the need for re-training the model affords a unique scalability not typically seen in the traditional deep learning literature. Nearest neighbor methods are known to be strong baselines in the vision and language domain (Devlin et al. 2013).
We introduce simple yet effective novel dense cross-modal alignment architecture called DXR (Dense X-modal Retriever). DXR achieves state-of-the-arl performance on both COCO (Chen) et al. 2015) and Flickr30k (Young et al. 2014) image-caption retrieval, with respect similar meth- ods. We subsequently use DXR to augment several multi-modal transformer architectures with a retrieval component. We show that retrieval augmentation yields impressive results for variety of well-known multi-modal transformer architectures, ranging from VisualBERT (Li et al.) 2019b) and ViLBERT (Lu et al. ( which use bounding-box features-to Movie+MCAN (Jiang al., 2020) which uses grid features. We name our overall method XTRA, for X-modal Transformer Retrieval Augmentation. Specifically, our contributions are follows:
We introduce a novel image-caption retrieval architecture, DXR, that achieves state-of-the- art performance on COCO and Flickr30k, with respect similar methods.
We introduce new eetrieval-augmented multi-moda transformer architecture, XTRA, that achieves state-of the significant improvement VQA over the baselines.

---

Under review as a conference paper at ICLR 2021
To our knowledge, this is the first work to showcase the promise of hybrid parametric and non non- para-eetrie models for the vision and language domain.
We conduct extensive experiments to shed light on this novel approach. We explore dif- ferent datasets for training the alignment model, as well as the effect of in-domain versus out-of-domain retrieval indices, the index size and inference time applications. Our ex- periments show that our proposed method significantly improves over a variety of strong multi-modee baselines, and demonstrates superior results over pre-training.
2 RELATED WORK
Cross-Modal Retrieval Prior work in cross-modal retrieval can be divided into two primary cate- gories: (i) methods that use grid-features and/or vector representations of the embedding space, and (ii) methods that use detection features, sequence representations, share information between the two modalities for computing the similarity metric. The first category consists of methods such as RRF (Liu et al.) 2017) and DPC (Zheng al. 2017) which use two convolutional network branches for image and text. CMPM by Zhhng & Lu (2018) introduced a pre-trained image backbone with a Bi-directional LSTM to learn image and text embeddings. The most relevant work in this category is VSE++ (Faghri et al. 2017). which focuses on hard negative mining and ranking loss. The second category generally exploits the use of detection features, which enforces an additional complexity. Methods such as TERN (Messina et al., 2020b), TERAN (Messina et al.) (020a). SAEM (Wu et al., 2019) and MMCA ( (Wei et al., 2020), use transformer modules to obtain modality-specifff embed- dings. TERAN, as well as SCAN (Lee et al. 2018), utilize sequence similarities. SCO (Huang al.) 2018 and VSRN (Li al. 2019a)) learn, in addition to image-text alignment, to generate the caption from the image embedding. MMCA, well as CAMP (Wang et al. 2019), fuses image and text information to obtain the final embeddings. Other methods, such as Unicoder- VL (Li et al.) (020a)), Oscar (Li et al., 2020b) and UNITER (Chen et al. 2020) learn to align between image and text by using positive and negative tuples of images and captions. While these models perform well, they suffer from high computational complexity as we discuss in Sec. 3.4
External Knowledge Source Methods The use of an external knowledge source (KS) has gained much attention in the field of natural language processing (NLP), such as the work of Verga et al. (2020). Our work is inspired by that of Lewis et al. 20200), which introduced RAG, a generic approach for variety of downstream NLP tasks, that uses learned retriever (DPR by Karpukhin jtt al. (2200) to augment the inputs by marginalizing across several retrieved phrases retrieved from Wikipedia. In the multi-modi domain, previous efforts have focused on building different types of KS. such as the work of Zhu et al. (2014), Chen et al. (0013). Divvala et al. (2014), Sadeghi et al.) (2015) and /Zhu et al. (2015), which use web information for the construction of the KS. Methods that use an external KS for a downstream task use a structured KS, such as the work of Narasimhan al. (2018). Narasimhan & Schwing (2018), Wang et al./ (2015) Wang et al. (2018) and ZZuu et al.
(2017). Zhu et al. (2017) introduced an iterative method for V VQA tasks. Marino et al. (2019)
introduced OK- VQA, a novel VQA dataset that requires the use of an external KS., KS.|Fan et al. (2020)
applied KS to modal dialogue. In our work, we focus on a more natural KS, such as images and captions, which better reflect the data generated in newspapers and social media.
Multi-modal Classification In this work, we investigate the potential advantages of using an ex- ternal KS for the popular and challenging VQA domain, multi-modal classification task. Current methods for VQA use pre-training on different datasets in order to gain better performance. In our experiments, we show performance for common methods such as VisualBERT (Li et al. 2019b), which concatenates the text and image modalities as an input to a pre-trained BERT (Devlin et al.) 2018 model. ViLBERT (Lu et al. 2019)) fuses text and image modalities using co-attentional trans- former layers. Other methods such as Pythia (Jiang et al.] 2018). VLBERT (Su et al.] (019)) and MMBT Kiela et al.) 2019) can benefit from our method, as well as more recent work such as Oscar (Li et al. 2020b) and UNITER (Chen et al. 2020). which use the alignment task for pre- training their models. The currently SOTA In this paper, we choose to show our performance on the two common VisualBERT and ViLBERT models, and the winner of the VQA 2.0 2020 challange, Movic+MCAN (Jing et al., 2020). which uses grid features instead of detection features, a mod- ulated convolutional bottleneck for the image backbone, and MCAN (Yu al. 2019) for fusion. A
2

---

Under review as a conference paper at ICLR 2021
Image Encader
GM
A Nack and white dog leys fiishee
The biick and whrne dog wondh ainer person hokiing a friseee
Abbetsion
Shanc Heights
A und day with fiaited by
Fiques
Antertion
Narm
A and sitting nest frishee
boidiigg
Twxt Encoder
axother
(b)
Figure 1: (a) Cross-modal alignment architecture. We a pre-trained ResNet-152 and BERT as feature extractors with an in-batch hinge loss. (b) Sample query image and retrieved captions from the COCO dataset. Ground truth captions are colored in blue (best viewed in color).
similar method was introduced by Nguyen et al. (020). Our method is also applicable to methods such Pythia (Jiang al. 2018) and MMBT (Kiela al. 2019
3 METHOD
Our methodology is composed of two disjoint parts: (i) for given external knowledge source K, consisting of m modalities, we train a model (i.e., the Retriever) to align between the different modalities. (ii) Given knowledge source K. and an alignment model, we train downstream model (i.e., the Reader) by augmenting its inputs with extra data from K.
3.1 CROSS-MODAL ALIGNMENT
Let A. be knowledge source consisting of m modalities, where each sample 8; (sf...... ................................ STT) E IC
a tuple of m elements, corresponding to different modalities. Our aligument model encompasses m encoders Em, each composed of feature-extraction module Fm, projection layer P shared Transformer encoding layer 7 with attention pooling, and normalization layer ..
(x) (N (Fm.()))
(I)
From this point, we will consider the two-modality case of images and captions as illustrated in Fig. I For text and image feature extractors, Fi and F2, we use pre-trained BERT masked language mode Devlin et al. (0188) and a pre-trained ResNet152 CNN backbone on ImageNet, respectively. The images are represented with convolutional grid features, chosen for robustness and speed, and these are flattened across the spatial dimension. The projection layers Pm. project each modality to constant dimension d. The projected sequences are then forwarded to shared Transformer- encoding layer, and aggregated by an attention pooling layer, resulting each modality. Finally. normalize each vector using an L2 normalizer, projecting all embeddings to the unit-sphere. Following Faghri et al. (2017). we only normalize the text embeddings because of image-caption imbalance (see Sec. 4.1
vector representation for
We train our dense cross-modal retriever (DXR) using contrastive loss, specifically using an in- batch hinge penalty with hard negatives (Faghri et al. 2017). Given a batch, consisting of b samples, 81 8h. for each sample 8;, let 8) and of be the positive pairs and st and oyy; the negative pairs. We compute the pair-wise similarity between the two modalities, using a dot product, denoted by (s) ( s3) (f). The hard-negative in-batch hinge loss is then defined as:
T
of' = max T (8),83) 3F1
(2)
max क (8) sf) jfz
(3)
Liard
(7),, + E(a π ( 8?) π(3! (sss,s)) s?)
(4)
where sf' and 8²' are the hardest samples inside the batch, and is the margin constant.
3.2 INDEXING AND RETRIEVING
Following Lewis et al. (202). we use FAISS (Johnson et al. 2017) as our indexer platform for fast KNN queries. Given knowledge source K., construct an index by computing the embeddings
3

---

Under review as a conference paper at ICLR 2021
Image Features
Alignment Model
Knowledge Source
image
Image Encoder
Image Query
FAISS Indexer
$1
Knowledge Source
Images
Augment Inputs
Multi-Modal Model
Frediction
Text Encoder
Text Query
Recrieved Indices
y
Weights
Figure 2: Illustration of our end-to-end framework. The trained cross- mross-modal alignment is used to extract features as queries to a FAISS indexer. The k retrieved indices are used to access data from the external knowledge source, and augment the input by appending each of the retrievals the relative modality. For VQA, we only query the input image and retrieve k captions.
of each sample in K. using some alignment model (the Retriever), which can be trained on any arbitrary knowledge source. We introduce with two variants: we either construct separate indices Te for each of the modalities; or we construct one joint index 1x: that encompasses all modalities and where KNN query will return mixed modality result. Fig. 2 iillustrates the two independent features of the alignment model and external knowledge source.
The retrieval process then consists of input query 9, encoder Em and indexer Ix (or Ig) Ix takes as an input an embedding query 6g = (q) and k, and returns the k-nearest indices i.... in. corresponding to the k-nearest embeddings. We then index data from K,, resulting in retrieval sets T (rT r..,), one for each modality, each consisting of varying number of samples nms where E Pm k. When using 1 a single modality m is returned, resulting in rm = (rT .......7 ....)k)) For simplicity, we define the retriever by R(g, Ems Ix, k) == {r) r...,."...
3.3 END-TO-END FUSION
Let M be any multi-modal reader model, applied to a specific downstream task that takes as input (x'....,,"" consisting of m modalities and outputs prediction y. The method augments the input x by concatenating the retrieved samples to their corresponding input modalities, resulting in the augmented input x':
(x or o
2h o r o
(5)
D
The resulting end-to-end training of model M is then defined by some loss function L, minimiz- ing ((M()') y). with the same hyperparameters as in the non-retrieval augmented case. Fig. 2 illustrates the complete model.
3.4 TIME-COMPLEXITY
As introduces in Sec. 2 we consider two types of retrievers, (i) methods such ours, that Max- imum Inner Product Search (MIPS), where each modality is computed independently, and (ii) meth- ods that have entangled computation of similarity between the different modalities, e.g. they cannot compute an independent embedding. Assuming a knowledge-source of size N, and a forward-pant with (1) time-complexity. In type (i). the embeddings of the entire knowledge source need to be computed only once, as well the embeddings of the query sample. In our experiments, we use FAISS with "Hierarchical Navigable Small World" search, and shown by JJohsson et al. 7). this searching method takes O O(AD[log Nle), where A and the graph. Therefore, the total time complexity of retricving is O(AD(log N\v). On the other hand, methods of type (ii) must compute pairwise similarities between query sample, and all samples in the dataset, resulting in a O(N) run time for searching the most similar sample.
constants, and D is the degree of
As results, type (ii) methods sample we query, a O(N) is applied, which results in an inefficient and non-scalable method. On the other hand, method does not impose significant overhead.
applicable to our end-to-end fusion pipeline, where for each
4

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
In this section, we describe the two experimental settings of the alignment model and the end-to- end downstream task training and evaluation. All models and experiments are implemented and performed with the MMF library (Singh et al.) (222a).
4.1 DATASETS
We use three common datasets for training and evaluating retrieval and VQA tasks. Flickr- 30K (Young et al. 2014) composed of 30,000 images, with captions each. Following Karpathy & Fei-Fei (2015), we use 1000 images for validation and 1000 images for testing. COCO (Chen et al. 2015) a well-known dataset that contains 120,000 images, with 5 captions each. We use the splits from Karpathy & Fei-Fei 20155) as well, resulting in 80K images for training, 5K images for validation and 5K images for testing. Following Faghri et al. (2017). we add an additional 30K images for training, and uses the same 1K and 5K splits. Conceptual Captions (Sharma et al., 2018) a dataset that contains image-caption pairs, composed of 3M samples for training and 100K for validation, which use test our retrieval model.
The proposed datasets differ in two major axes (i) Size The largest knowledge-source we use is CC, which contains 3M image-caption pairs, while COCO and Flickr30K are smaller in and two orders of magnitude, respectively. (ii) Domain gap As shown in (Singh et al. 2020b), CC datasets differs both visual and textual domain from the VQA task, while COCO has the best domain match in both. Flickr30K datasets, on the other hand, is very similar to COCO, but suffers from short number of samples in an order of magnitude compared COCO.
4.2 CROSS-MODAL RETRIEVAL
In the cross-modal retrieval task, we deal with two modalities: images and captions. Bi-directional retrieval is evaluated, denoted as Text - Image and Image - Text, where the left-hand-side indi- cates the query and the other indicates the retrieved domain. For fair comparison, we only report results for methods that use grid-features and vector representations, noted in Sec and 3.4 For a full comparison with other previous methods, please see Appendix(A) Models are trained for 100K iterations with warm-up of 2k iterations, batch size of 256, and Adam optimizer with learning of 0.0001 where the (pre-trained unimodal) feature encoder's 's learning-raid rrtt is multiplied by 0.1. The hinge-loss margin hyperparameter is set 0.2.
4.3 DOWNSTREAM TASKS
After training the alignment models for each for each, as defined in Sec 3.2 Note that for COCO, we only use the training set for indexing, while for Flickr30K and CC, we use the entire of train/val/test. This is done for fair comparison with the VQA task, which uses the COCO training-set images for training. Our experiments focus on VQA as the downstream task, however we note that extension other multi-modal tasks is straightforward. The inputs of the VQA task are image and text tuples, and it is presented as classification problem over a of answers. VQA, we observe that information regarding the content of the image, such as the amount, color and location of objects is very correlated with the question and answer. Therefore, captions serve as good auxiliary information, while similar/retrieved images (e.g., to which the question does not directly refer) are less informative. For that reason, we use the separate indices variant, retrieving text captions from images to yield a cross- modal image to text translation. We experiment with all three datasets, evaluating different training and inference variants.
COCO and CC-we build indices
RESULTS
5.1 CROSS-MODAL RETRIEVAL
Tab. [ and 2 show retrieval results Flickr30K and COCO, comparing similar methods that grid-fe grid-features and vector representations for the embedding space. Reported numbers correspond to Recall-at- 1/5/10 on the test-sets. As can be seen, method significantly outperforms previous
5

---

Under review as a conference paper at ICLR 2021
Text - Image
Image - Text R@5
Method
@@
R@5
R@
R@l
R@ 10
RRF
35.4
68.3
79.9
47.6
77.4
87.1
CMPM
37.3
65.7
75.5
49.6
76.8
86.1
DPC
39.1
69.2
69.2
55.6
81.9
89.5
VSE++
39.6
69.6
79.5
52.9
79.1
87.2
DXR
50.6
78.8
86.7
65.1
87.3
92.6
Table I: Retrieval results for Flickr30K, comparing only methods that use raw images as input, and vector representations for the embedding space. Additional methods can be found in Appendix
COCO IK
COCO 5K
Text + Image
Image + Text
Text + Image
Image + Text
Method
R@1
R@5 @@ 10
R@1 R@5 R@ 10
@@I
R@5 R@ 10
R@1 R@5
R@ 10
DPC
47.1
79.9
90.0
65.6
89.8
95.5
25.3
53.4
66.4
41.2
70.5
81.1
VSE++
52.0
83.1
92.0
64.6
89.1
95.7
30.3
59.1
72.4
41.3
69.2
81.2
CMPM
44.6
78.8
89.0
56.1
86.3
92.9
22.9
50.2
63.8
31.1
60.7
73.9
DXR
56.8
88.2
94.9
67.0
93.0
97.6
33,9
64.9
77.4
44.9
75.2
84.7
Table 2: Retrieval results for COCO, comparing only methods that use raw images as input, and vector representations for the embedding space. Additional methods can be found in Appendix A
Knowledge Source
Training Type
Visual BERT
ViLBERT
Movie+MCAN
Flickr30K
XTRA 10-C
66.77
67.32
69.70
PT
64.34
68.14
CC
XTRA-10C
67.49
67.37
69.02
PT XTRA-IOC
67.53
69.17
PT
64.54
67.58
COCO
XTRA-10C
68.98
69.07
71.52
XTRA-10C
67.71
69.90
Vanilla
63.54
67.56
71.16
5-GT
69.61
71.50
71.80
Table VQA Results for three different reader models on COCO val-set. Vanilla models use pre-trained BERT model. PT Pre-Traiin Pre-Training with the knowledge source. XTRA-I0C- training via our method using the knowledge source indicated and alignment model trained on that knowledge source, using 10 retrieved captions. 5-GT- training with the 5 ground truth captions.
work in both datasets. We refer to Appendix A for comparison with additional methods. While CC not commonly used in the retrieval literature, we use for our downstream task. Using DXR, obtain the following results for CC: R@ 1: 25.1 @@5: 50.1 and R@ 10: 61.9 for Text - Image, and R 1: 25.4 R S: 50.9 and R@ 10: 61.8 for Image + Text. The alignment model trained on CC used for training in the downstream VQA task. We notice that performance degrades the dataset size increases, which could affect the downstream task since we query from the entire dataset.
5.2 VISUAL QUESTION ANSWERING
Our main results show performance on the VQA val val-set, experimenting with three common VQA methods: VisualBERT (Li et al. 2019b), ViLBERT (Lu al. 2019), and the currently state- of-the-art winner the VQA 2.0 challenge, Movie+MCAN (Nguyen et al. 2020), each along with three different knowledge sources (COCO, CC and Flickr30K). Following Jiang et al. (2020). use the val-s sal- to assist in our exhaustive ablation studies, however we report our final SOTA result on the VQA test std split. Tab. summarizes four different training settings: (i) vanilla

---

Under review as a conference paper at ICLR 2021
Model
55
Kissdedgee
Augment Ripues
Muss-Modu
R5.55
knowledee Sourte
Augement Iipues
Mu Mooit Mozel
indeoer
Source
Mopel
Indeoer
(a)
(b)
Figure 3: Two Hot-Swap configurations of the knowledge source during inference. (a) both the alignment model and the knowledge source are replaced with new ones built using new dataset. (b) only the knowledge source replaced, and the indexer built using the old alignment model.
VILBERT COCO HS2CC
VILBERT COCO HS2Flickr30k
VILBERT CC HS2COCO
VILBERT Flickr30k H52COCO
Accuracy
Accuracy
Accuracy
Accuracy
D-Domain Out-t--Dmman
n-Jomain
Jamaain Out-et-Deran
In Donain
Oum-a--ommn
Out-oh-Dama
1533 20
e Retrieved Caption
15th
Retrieved Caption
de Retrieved Caption
of Retrieved Caption
Visual BERT COCO HS2CC
Visual BERT COCO H52F akkr2kk
20
Visual BERT CC Hs2COCO
Visual BERT Flickr30k H5200CO
Accuracy
Accuracy
Accuracy
n-Bomann Dummnn
In Diaaah
2-Jamiin Out LomolDemann
F-Damu n
Traine
Traired
1533 20 0 of Retrieved Caption
I0e
538 16
10h
1510 26 of Retrieved Caption
100
# of Retrieved Caption
of Retrieved Caption
Figure 4: Hot-Swap results. Each row corresponds to a different reader model. Each graph shows (a) Training with different amount of retrieved captions. (b) Using the trained model with 10-cap, we inference with different amount of captions. (c) Hot swapping between knowledge sources.
models using pre-trained BERT; (ii) PT Task agnostic pre-Training with the knowledge source dataset (using masked language modeling); (iii) 5-GT training with the ground truth captions from COCO; (iv) XTRA-10C training via our method, using the knowledge source indicated and alignment model trained on that source, using 10 retrieved captions.
We see that using the five ground truth (GT) COCO captions as additional data (bottom row of Tab. (3.. sets a soft upper bound for our approach. On one hand, the GT captions contain relevant information about the content of the image; on the other hand, other captions from the knowledge source may additionally serve as rich, useful descriptions. We also see that our method significantly increases performance across all baselines, even with respect to pre-training. This suggests that our method serves as a good alternative for pre-training. Our best model, Movie+MCAN+XTRA. 10C, outperform Movi++MCAN on VQA2.0 test std, Our best model sets a new state-of-ther the--att on VQA2.0 test -d,, std, using Movie-MCAN-XTRA-IOON obtaining score of 73.12 (with single model performance).
5.3
HOT SWAP
Our method is devised such that querying and retrieving from the knowledge source is independent of the downstream model, enabling the swap of the aligmment model and/or knowledge source during inference. This affords interesting explorations. We describe two forms of "hot swapping": (i) the entire knowledge source with its trained alignment model are replaced with new one and corresponding alignment model we refer to this as ofut-of-dommin"; (ii) the knowledge source used for retrieving is swapped, but the alignment model remains the same as was originally trained with the downstream model. In this case, we build new retriever for the new knowledge source, using the original alignment model alignment domain with which the downstream model was trained. Fig. illustrates the two cases.
call this "in-domain". "in/out-of-domain refers to the
In Fig. 4 we show different inference results for hot swapping. All models in this experiment are trained using 10 retrieved cations. The title of each graph represents the trained model, followed by the trained knowledge source and the knowledge source to which we swap. In addition, show inference results for training with the swapped knowledge source, e.g. training with CC knowledge source and alignment model from scratch, using 10 retrievals. As can be seen, "in-domain" hot
7

---

Under review as a conference paper at ICLR 2021
Query Image
Retrieved Captions Flickr30K Hotswap
No Hotswap COCO trein set
CC Hotswap tr trainival sets
COCO val-set
tcainivalitest sete
dog that lying down sidewalk
dog soleop the streets
A dog lies down cobblesione street
dog with murele on is lying on the sidewalk
lan male balldog sleeping on sidewalk
The dog is lying on the cobblesione street
happy puppy lies the street
Cute sleeping the sidewalk
A laying the side the street
A dog is laying and resting on walkoay
dog lying on the sidewalk
A dog with a collar on lying on the street
Figure 5: Sample top-4 result for "in-domain" Hot-Swap. The model was trained using COCO as the knowledge source, and 10 retrieved captions. Left Query image from VQA val-set Columns refer to the different hot-swaps, showing retrieved captions.
VILBERT
Visual BERT
VILBERT
Accuracy
Accurast
Accuracy
Accuracy
Floo3ss
Retrieved Caption
of Retrieved Caption
Retrieved Caption
of Retrieved Caption
a
Figure 6: Ablation study of our method. (a) Training with different amount of retrieved captions. (b) Using the trained model with 10-cap, inference with different amount of captions.
swapping performance significantly higher than "out-of-domain". We hypothesize that the reader model has learned an implicit structure of the alignment space. Surprisingly, when training with COCO as the knowledge source, "in-domain" hot swapping performs similarly, for the same amount of trained retrievals (10), as training with an alternative knowledge source and alignment model. On the other hand, we observe that this suffers from a decrease in generalization due to different amounts of retrieval during inference-time. In the other direction, hot swapping COCO from CC or Flickr30K does not result in the same performance as training with COCO as the knowledge source and aligmment model, yet, performance and generalization do not degrade. Qualitative results of "in-domain" hot swapping are presented in Fig 5 As can be observed, novel information such as the fact that the image shows a "cobblestone street" retrieved from CC without having to train the alignment model on that source.
5.4 ABLATION STUDY
In this study, we explore the use of different amounts of retrieval during training and inference, as well as doing inference without retrieving which name umplugged. We further explore the relationship between pre-training and XTRA.
Number of Retrievals We experiment with different amounts of retrieved captions during training and inference. In Fig 6 (a), we show the performance our method when training with different amounts of retrieval, and different knowledge sources. As can be observed, training with 10 captions and COCO as the knowledge source results in the best performance. In Fig 6 (b), we show the inference performance for models trained using 10 retrievals. In addition, we show the inference performance of the same model, trained with random amounts of retrieval, between I and 20, on the COCO dataset (COCO 20R-C), With this, the best performance given when we inference with the same amount of trained retrievals, and this then degrades as the number of retrievals differ from how the model was trained. We also see that training with varying number of retrievals achieves better generalization to different amounts of retrievals during inference, as can be seen in Fig (b), COCO 20R-C, where performance is maintained up to 60 retrievals during inference.
Unplugged Performance One interesting observation we make is the ability to "unplug" the knowledge source by not retrieving during inference-time. Tab. 4 shows a noticeable decrease in performance, indicating the dependency of the reader the retrieved data during training. When training with COCO as the knowledge source, introducing captions that are very related to the input images is biasing the model to depend on the retrieved captions. For CC and Flickr30K, the domain
8

---

Under review as a conference paper at ICLR 2021
Model
COCO
CC
Flickr30K
Visual BERT
58.77 (68.98) 4 10.21
63.15 (67.49) | 4.34
61.86 (66.77) L 4.91
ViLBERT
45.60 (69.07) 4 23.47
63.50 (67.37) 3.87
59.34 (67.32) 4 7.98
Table 4: VQA performance using models trained with 10 retrieved caption, and evaluating with- out any retrievals ("unplugged"). The highest decrease in performance occurs for the in-domain (COCO) knowledge source where retrieved examples are most informative.
gap between the downstream task and the knowledge source lessens this gap in unplugged perfor- mance. Surprisingly, while ViLBERT performance generally better than Visual BERT, using our method, the opposite is true when unplugging the knowledge source.
External Knowledge Source Pre-training The use of retrieval mechanism over external knowledge sources raises inriguing questions, e.g.: 1) Is augmentation better than pre-training?: and 2) Can pre-trate pre-training help the external knowledge source? To address these questions, we experi- mented with two different pre- pre-training datasets commonly used for VQA- COCO and CC. Tab, suggests that for the COCO dataset, using our method significantly better than pre- pre-training alone, while using pre-training followed by XTRA causes the performance vary with respect to the reader architecture (e.g., pre-training helps XTRA with ViLBERT, but not with VisualBERT), Tab. also shows that fine-tuning our method after pre-training on the same knowledge source yields better performance over pre-training across all knowledge sources and architectures.
6 CONCLUSION
In this work, we presented a novel approach that proposes the use of external knowledge sources in multi-modal prediction models with transformer architectures. We trained a of-t-of--he-art align- ment model, DXR, for performing retrieval over external knowledge sources. We showed that our method XTRA yields gains performance when using an in-domain knowledge source on VQA. We conducted a variety of experiments to show the sensitivity and effects of the used knowledge source with various choices hyperparameters which shed further light on the different aspects of the model. Future research and applications of our method include interpretability of retrieved data and predictions for verification processes, the demonstration of increased information security by hot-swapping, and unplugged versions of models and new architectures that take advantage of out-of-domail knowledge source. We hope that our approach inspires further work in the direction of hybrid parametric non-parametric models for multi-modal problems.
REFERENCES
Xinlei Chen, Abhinav Shrivastava, and Abhinav Gupta. Neil: Extracting visual knowledge from web data. In Proceedings of the IEEE international conference on computer vision, pp. 1409-1416,
2013.
Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 1043v01554.00325, 2015.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference Computer Vision, pp. 104-120. Springer, 2020.
Jacob Devlin, Saurabh Gupta, Ross Girshick, Margaret Mitchell, and C Lawrence Zitnick. Exploring nearest neighbor approaches for image captioning. arXiv preprint a5Xiv:/505.044 04440004477.1 2015.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 180000800 2018.
Santosh K Divvala, Ali Farhadi, and Carlos Guestrin. Learning everything about anything: Webly- supervised visual concept learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3270-3277, 2014.

---

Under review as a conference paper at ICLR 2021
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives. arXiv preprint 1rXiv:1707.05612.20000 2017.
Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. Augmenting transformers with knn-based composite memory for dialogue. arXiv preprint arXiv:2004.12744, 2020.
Kelvin Guu, Kenton Lee. Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint arXiv:2002.08909 2020.
Yan Huang, Qi Wu, Chunfeng Song, and Liang Wang. Learning semantic concepts and order for image and sentence matching. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pp. 6163-6171, 2018.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arX1v:2007.01222.2000 2020.
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Mill and Xinlei Chen. In defense of grid features for visual question answering. arXiv preprint arXiv:2001.03615, 2020.
Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia vO. 1: the winning entry to the vqa challenge 2018. arXiv preprint arXiv: (xXxvv/87.00955, 2018.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-seat similarity search with gpus. arXiv preprint / 08734, 2017.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 3128-3137, 2015.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04900 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv: 1.00072, 2019.
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint 19Xiv:190, 2019.
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 201-216, 2018.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- -augeeeed--gmmeeeedd gener- ation for knowledge- knowledge-intensive inteegivve nlp tasks. arXiv preprint arXiv:2005.11400, 2020.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In AAAI, pp. 11336 11344, 2020a.
Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image- text matching. In Proceedings of the IEEE International Conference Computer Vision, pp. 4654-4662, 2019a.
Liunian Harold Li. Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: 2019b.
Xiujun Li, Xi Yin, Chumyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong. Furu Wei, et al. Oscar: Object-semantics aligned pre-training for -lanuuge tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020b.
10

---

Under review as a conference paper at ICLR 2021
Yu Liu, Yanming Guo, Erwin M Bakker, and Michael S Lew. Learning recurrent residual fu- sion network for multimodal matching. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4107-4116, 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for iision-and-languaal tasks. In Advances in Neural Information Processing Systems, pp. 13-23, 2019.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3195-3204, 2019.
Nicola Messina, Giuseppe Amato, Andrea Esuli, Fabrizio Falchi, Claudio Gennaro, and Stéphane Marchand-Maille Fine- graigee visual textual alignment for cross cross-modal retrieval using trans- former encoders. arXiv preprint S10.iv::000.0523333 2020a.
Nicola Messina, Fabrizio Falchi, Andrea Esuli, and Giuseppe Amato. Transformer reasoning net- work for image-text matching and retrieval. arXiv preprint urXiv:2004.09144 2020b.
Medhini Narasimhan and Alexander G Schwing. Straight to the facts: Learning knowledge base retrieval for factual visual question answering. In Proceedings of the European conference on computer vision (ECCV), pp. 451-468, 2018.
Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing. Out of the box: Reasoning with graph convolution nets for factual visual question answering. In Advances in neural information processing systems, pp. 2654-2665, 2018.
Duy-Kien Nguyen, Vedanuj Goswami, and Xinlei Chen. Revisiting modulated convolutions for visual counting and beyond. arXiv preprint arXiv:2004. 18833 2020.
Fereshtch Sadeghi, Santosh K Kumar Divvala, and Ali Farhadi. Viske: Visual knowledge extraction and question answering by visual verification of relation phrases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1456-1460 1464, 2015.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pp. 2556-2565, 2018
Amanpreet Singh, Vedanuj Goswami. Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Mmf: multimodal framework for vision and language research. https::/ com/ fwcebookresearcc......o
2020a.
Amanpreet Singh, Vedanuj Goswami, and Devi Parikh. Are pretraining it right? digging deeper into linio-lic pretraining. arXiv preprint arXiv:2004.0874: 2020b.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: http:/908.0800 2019.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William W Cohen. Facts as experts: Adaptable and interpretable neural memory symbolic knowledge. arXiv preprint arXiv:2007.00844 2020.
Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. Explicit knowledge- based reasoning for visual question answering. arXiv preprint arXiv:1511.02570 2015.
Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel. Fvqa: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence, 40 (10):2413-242) 2018
Zihao Wang, Xihui Liu, Hongsheng Li, Lu Sheng. Junjie Yan, Xiaogang Wang, and Jing Shao. Camp: Cross-modas adaptive message passing for text-image retrieval. In Proceedings of the IEFE International Conference Computer Vision, pp. 5764-5773, 2019.
11

---

Under review as a conference paper at ICLR 2021
Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu. Multi-modality cross attention network for image and sentence matching. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 10941-10950, 2020.
Yiling Wu, Shuhui Wang, Guoli Song, and Qingming Huang. Learning fragment self-attention embeddings for image-text matching. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 2088-2096, 2019.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6281 1221-6290, 2019.
Ying Zhang and Huchuan Lu. Deep cross-modal projection learning for image-text matching. In Proceedings the European Conference on Computer Vision (ECCV), pp. 686-701, 2018.
Zhedong Zheng, Liang Zheng, Michael Garrett, Yang, and Yi-Dong Shen. Dual-path convo- lutional image-text embedding. corr abs/1711.05533 (2017). arXiv preprint arXiv:171..05535, 2017.
Yuke Zhu, Alireza Fathi, and Li Fei-Fei. Reasoning about object affordances in a knowledge base representation. In European conference on computer vision, pp. 408-424 Springer, 2014.
Yuke Zhu, Ce Zhang, Christopher Ré, and Li Fei-Fei. Building large- scale multimodal knowledge base system for answering visual queries. arXiv preprint arXivv1507.05670, 2015.
Yuke Zhu, Joseph J Lim, and Li Fei-Fei. Knowledge acquisition for visual question answering via iterative querying. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1154-1163, 2017.
12

---

Under review as a conference paper at ICLR 2021
Text + Image
Image - Text R@5
Method
R @
R@5
R@ 10
R@D 1
R@ 10
RRF
35.4
68.3
79.9
47.6
77.4
87.1
CMPM
37.3
65.7
75.5
49.6
76.8
86.1
DPC
39.1
69.2
69.2
55.6
81.9
89.5
VSE+
39.6
69.6
79.5
52.9
79.1
87.2
DXR
50.6
78.8
86.7
65.1
87.3
92.6
TERN
41.1
71.9
81.2
53.2
79.4
86.0
SCO
41.1
70.5
80.1
55.5
82.0
89.3
SAEM
52.4
81.1
88.1
69.1
91.0
95.1
SCAN
48.6
77.7
85.2
67.4
90.3
95.8
CAMP
51.5
77.1
85.3
68.1
89.7
95.2
VSRN
54.7
81.8
88.2
71.3
90.6
96.0
TERAN
56.5
81.2
88.2
70.8
90.9
95.5
MMCA
54.8
81.4
87.8
74.2
92.8
96.4
Unicoder-Ve VL
71.5
90.9
94.9
86.2
96.3
99.0
UNITER
73.6
93.0
95.9
88.2
98.4
99.0
Table 5: Retrieval results for Flickr30K. Top methods that use raw images as input, and vector representations for embedding space. Bottom Methods that use detection features or sequence similarity measures.
COCO IK
COCO 5K
Text + Image R@1
Image - Text
Text + Image
Image + Text
Method
R@5 R@ 10
R@1 R@5 R@ 10
R@1
R@5 @@ 10
R@1 R@5 R@ 10
DPC
47.1
79.9
90.0
65.6
89.8
95.5
25.3
53.4
66.4
41.2
70.5
81.1
VSE++
52.0
83.1
92.0
64.6
89.1
95.7
30.3
59.1
72.4
41.3
69.2
81.2
CMPM
44.6
78.8
89.0
56.1
86.3
92.9
22.9
50.2
63.8
31.1
60.7
73.9
DXR
56.8
88.2
94.9
67.0
93.0
97.6
33,9
64.9
77.4
44.9
75.2
84.7
TERN
51.9
85.6
93.6
63.7
90.5
96.2
28.7
59.7
72.7
69.5 42.8
38.4
81.3
SCO
56.7
87.5
94.8
69.9
92.9
97.5
33.1
62.9
75.5
72.3
83.0
SAEM
57.8
88.6
94.9
71.2
94.1
97.7
SCAN
58.8
88.4
94.8
72.7
94.8
98.4
38.6
69.3
80.4
50.4
82.2
90.0
CAMP
58.5
87.9
95.0
72.3
94.8
98.3
39.0
68.9
80.2
50.1
82.1
89.7
VSRN
62.8
89.7
95.1
76.2
94.8
98.2
40.5
70.6
81.1
53.0
81.1
89.4
TERAN
65.0
91.2
96.4
77.7
95.9
98.6
42.6
72.5
82.9
55.6
83.9
91.6
MMCA
61.6
89.8
95.2
74.8
95.6
97.7
38.7
69.7
80.8
54.0
82.5
90.7
Unicoder- VL
69.7
93.5
97.2
84.3
97.3
99.3
46.7
76.0
85.3
62.3
87.1
92.8
UNITER
51.7
78.4
86.9
66.6
89.4
94.2
Oscar
78.2
95.8
98.3
89.8
98.8
99.7
57.5
82.8
89.8
73.5
92.2
96.0
Table 6: Retrieval results for COCO. Top methods that use raw images as input, and vector rep- resentations for the embedding space. Bottom Methods that use detection features or sequence similarity measures.
A RETRIEVAL
Tab. shows a complete comparison of the different alignment methods in the cross-modal align- ment literature. The top part corresponds to methods which use vector representations, grid-features, and not share information between the modality branches. The bottom part shows the rest of the methods.
13


Reviews:
[{"review_id": "pHTBGZTbs1t", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a method to use unstructured external knowledge sources to improve visual question answering and image-caption retrieval. The proposed method can achieve somewhat improvement for visual question answering, but drop the performance for image-caption retrieval with a more complex model. Some concerns are as follows:\n\n1. The authors claim that the proposed method achieved state-of-the-art performance on both COCO and Flickr30k  image-caption retrieval. However, their retrieval scores are lower about 10 than the state-of-the-art counterparts, such as TERAN. The statement is not correct.\n2. Although the authors stated the proposed method uses raw images as input, the adopted backbones (i.e., image/text encoders) should be frozen to extract the features for the following components in their pipeline, which is similar to the other feature-based methods (e.g., TERAN) that also can be seen as freezing their backbones (e.g., Faster R-CNN) during their training and inference stages. Thus, the inputs between the proposed method and other methods have no essential difference. What is the significance to design such a much more complex model for image-caption retrieval? What are the advantages of the proposed method comparing prior superior methods? I am confused that if it is worthy to adopt such a complex model with worse performance.\n3. It is interesting to see that the proposed method could improve the performance of VQA. However, Table 3 does not give us a throughout comparison. There are many results missed in the table, such as different training types for Flickr30K, some results for Movie+MCAN, etc. From the results, we also could draw that the improvement of the proposed method is very limited for a good VQA method, i.e., Movie+MCAN with Vanilla. The experiments could not significantly demonstrate the significance and advantages of the proposed method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5RDxihUqx8W", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores a new direction, to utilize the searched results (image caption pair) to improve downstream multimodal learning tasks. They first pre-trained a cross-modal model using the contrastive learning on the image caption dataset. Then they use the pre-trained model to search the relevant terms for image or text input, and augment the searched results as the input for the downstream multi-modal tasks.\n\nConcerns:\n\n1. The authors claim that the trained alignment model, DXR, \"achieves state-of-the-art image-caption retrieval performance\" on COCO and MIRFlickr. However, plenty of cross-modal retrieval methods achieve better performance than DXR. For example, Oscar [a] and Unicoder-VL [b] could achieve significantly better than DXR on both COCO and MIRFlickr. In my mind, this could not be regarded as a novel contribution. Also, how different cross-modal pre-trained models perform as the DXR is also an interesting direction for exploration.\n\n2. The authors claim that XTRA \"achieved state-of-the-art performance\", which is also not convincing. Oscar [a] and UNITER [c] achieve significantly better than XTRA on COCO VQA val set. Please carefully survey the literature and claim the contribution.\n\n3. The visual-linguistic pre-training methods [a,b,c] also aim to distill the external knowledge from the large-scale image caption dataset to the downstream tasks, but the solution is to provide a pre-trained model as the initialization which is pre-trained on the large scale image-caption dataset. These methods should also be surveyed in the literature, and may be compared in the experiment part.\n\n[a] Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks, ECCV 2020\n\n[b] Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training, AAAI 2020\n\n[c] UNITER: UNiversal Image-TExt Representation Learning, ECCV 2020", "extra_scores": {}, "extra_text": {}}, {"review_id": "3RkRIx0Vv60", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a cross-modal retrieval augmentation for the multi-modal classification task (VQA). The authors first introduce a transformer-based image caption retrieval architecture that achieves decent performance. Then, the authors proposed to use the retrieval model to retrieve relevant visual and textual information as augmentation. The proposed method experiment on 3 existing method (Visual Bert, ViLBERT, and Movie + MCAN) and show good improvements over the baseline model. \n\nMy major concern about this paper is the lack of novelty and experiment comparison. The proposed image caption retrieval architecture is not novel at all. Most existing method (ViLBERT, UNITER, VLBERT etc.) has a similar transformer objective, while the image fine-tuning are from Pixel Bert. In the experiment section (Table 2), the author even didn't compare these methods. \n\nIn terms of the VQA performance, pre-training on the conceptual caption actually hurt the performance of visual Bert and ViLBERT. Could the authors explain why a larger dataset can not help with the model? Is Cross-Modal Retrieval pre-training necessary for VQA?  \n\nIn terms of the speed, the deep fusion model for image retrieval is super slow, since the model need to calculate the score for each pair. What is the size of pool when computing the retrieved captions? What is the time complexity? \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper discusses the problem of how to augment cross-modal retrieval for the task of multi-modal classification -- it uses image caption pairs to improve downstream multimodal learning, and shows improvement in the task of visual question answering. However, the paper has the following weaknesses: (a) lack of novelty, (b) lack of thorough empirical evaluation, (c) the complex model did not give significant gains.
Decision: Reject