Cross-Modal Retrieval Augmentation for Multi-Modal Classification

1 INTRODUCTION . Neural networks augmented with non-parametric retrieval components have recently shown impressive results in NLP ( Khandelwal et al. , 2019 ; Guu et al. , 2020 ; Lewis et al. , 2020 ; Izacard & Grave , 2020 ) . In this work , we train a state-of-the-art image-caption alignment model and utilize it in various retrieval-augmented multi-modal transformer architectures , achieving state-of-the-art performance on visual question answering ( VQA ) significant improvement over the baselines , including the winner of the VQA 2.0 2020 challenge . Retrieval components are promising because they allow for easy revision and expansion of their memory , as compared to their parametric , pre-training counterparts . They provide more interpretability , as well as direct factual consistency with trusted knowledge sources . In the multi-modal setting , retrieval augmentation allows for leveraging the strengths of text-based models—as evidenced by the strong performance of BERT-based models in vision-and-language ( Lu et al. , 2019 ; Li et al. , 2019b ; Kiela et al. , 2019 ) —via cross-modal translation from images to text . Being able to seamlessly “ hot swap ” knowledge sources without the need for re-training the model affords a unique scalability not typically seen in the traditional deep learning literature . Nearest neighbor methods are known to be strong baselines in the vision and language domain ( Devlin et al. , 2015 ) . We introduce a simple yet effective novel dense cross-modal alignment architecture called DXR ( Dense X-modal Retriever ) . DXR achieves state-of-the-art performance on both COCO ( Chen et al. , 2015 ) and Flickr30k ( Young et al. , 2014 ) image-caption retrieval , with respect to similar methods . We subsequently use DXR to augment several multi-modal transformer architectures with a retrieval component . We show that retrieval augmentation yields impressive results for a variety of well-known multi-modal transformer architectures , ranging from VisualBERT ( Li et al. , 2019b ) and ViLBERT ( Lu et al. , 2019 ) —which use bounding-box features—to Movie+MCAN ( Jiang et al. , 2020 ) —which uses grid features . We name our overall method XTRA , for X-modal Transformer Retrieval Augmentation . Specifically , our contributions are as follows : • We introduce a novel image-caption retrieval architecture , DXR , that achieves state-of-theart performance on COCO and Flickr30k , with respect to similar methods . • We introduce a new retrieval-augmented multi-modal transformer architecture , XTRA , that achieves state-of-the-art performance significant improvement on VQA over the baselines . To our knowledge , this is the first work to showcase the promise of hybrid parametric and non-parametric models for the vision and language domain . • We conduct extensive experiments to shed light on this novel approach . We explore different datasets for training the alignment model , as well as the effect of in-domain versus out-of-domain retrieval indices , the index size and inference time applications . Our experiments show that our proposed method significantly improves over a variety of strong multi-modal baselines , and demonstrates superior results over pre-training . 2 RELATED WORK . Cross-Modal Retrieval Prior work in cross-modal retrieval can be divided into two primary categories : ( i ) methods that use grid-features and/or vector representations of the embedding space , and ( ii ) methods that use detection features , sequence representations , or share information between the two modalities for computing the similarity metric . The first category consists of methods such as RRF ( Liu et al. , 2017 ) and DPC ( Zheng et al. , 2017 ) which use two convolutional network branches for image and text . CMPM by Zhang & Lu ( 2018 ) introduced a pre-trained image backbone with a Bi-directional LSTM to learn image and text embeddings . The most relevant work in this category is VSE++ ( Faghri et al. , 2017 ) , which focuses on hard negative mining and ranking loss . The second category generally exploits the use of detection features , which enforces an additional complexity . Methods such as TERN ( Messina et al. , 2020b ) , TERAN ( Messina et al. , 2020a ) , SAEM ( Wu et al. , 2019 ) and MMCA ( Wei et al. , 2020 ) , use transformer modules to obtain modality-specific embeddings . TERAN , as well as SCAN ( Lee et al. , 2018 ) , utilize sequence similarities . SCO ( Huang et al. , 2018 ) and VSRN ( Li et al. , 2019a ) learn , in addition to image-text alignment , to generate the caption from the image embedding . MMCA , as well as CAMP ( Wang et al. , 2019 ) , fuses image and text information to obtain the final embeddings . Other methods , such as Unicoder-VL ( Li et al. , 2020a ) , Oscar ( Li et al. , 2020b ) and UNITER ( Chen et al. , 2020 ) learn to align between image and text by using positive and negative tuples of images and captions . While these models perform well , they suffer from high computational complexity as we discuss in Sec . 3.4 External Knowledge Source Methods The use of an external knowledge source ( KS ) has gained much attention in the field of natural language processing ( NLP ) , such as the work of Verga et al . ( 2020 ) . Our work is inspired by that of Lewis et al . ( 2020 ) , which introduced RAG , a generic approach for a variety of downstream NLP tasks , that uses a learned retriever ( DPR by Karpukhin et al . ( 2020 ) ) to augment the inputs by marginalizing across several retrieved phrases retrieved from Wikipedia . In the multi-modal domain , previous efforts have focused on building different types of KS , such as the work of Zhu et al . ( 2014 ) , Chen et al . ( 2013 ) , Divvala et al . ( 2014 ) , Sadeghi et al . ( 2015 ) and Zhu et al . ( 2015 ) , which use web information for the construction of the KS . Methods that use an external KS for a downstream task use a structured KS , such as the work of Narasimhan et al . ( 2018 ) , Narasimhan & Schwing ( 2018 ) , Wang et al . ( 2015 ) Wang et al . ( 2018 ) and Zhu et al . ( 2017 ) . Zhu et al . ( 2017 ) introduced an iterative method for VQA tasks . Marino et al . ( 2019 ) introduced OK-VQA , a novel VQA dataset that requires the use of an external KS . Fan et al . ( 2020 ) applied a KS to multi-modal dialogue . In our work , we focus on a more natural KS , such as images and captions , which better reflect the data generated in newspapers and social media . Multi-modal Classification In this work , we investigate the potential advantages of using an external KS for the popular and challenging VQA domain , a multi-modal classification task . Current methods for VQA use pre-training on different datasets in order to gain better performance . In our experiments , we show performance for common methods such as VisualBERT ( Li et al. , 2019b ) , which concatenates the text and image modalities as an input to a pre-trained BERT ( Devlin et al. , 2018 ) model . ViLBERT ( Lu et al. , 2019 ) fuses text and image modalities using co-attentional transformer layers . Other methods such as Pythia ( Jiang et al. , 2018 ) , VLBERT ( Su et al. , 2019 ) and MMBT ( Kiela et al. , 2019 ) can benefit from our method , as well as more recent work such as Oscar ( Li et al. , 2020b ) and UNITER ( Chen et al. , 2020 ) , which use the alignment task for pretraining their models . The currently SOTA In this paper , we choose to show our performance on the two common VisualBERT and ViLBERT models , and the winner of the VQA 2.0 2020 challange , Movie+MCAN ( Jiang et al. , 2020 ) , which uses grid features instead of detection features , a modulated convolutional bottleneck for the image backbone , and MCAN ( Yu et al. , 2019 ) for fusion . A similar method was introduced by Nguyen et al . ( 2020 ) . Our method is also applicable to methods such as Pythia ( Jiang et al. , 2018 ) and MMBT ( Kiela et al. , 2019 ) . 3 METHOD . Our methodology is composed of two disjoint parts : ( i ) for a given external knowledge source K , consisting of m modalities , we train a model ( i.e. , the Retriever ) to align between the different modalities . ( ii ) Given a knowledge source K and an alignment model , we train a downstream model ( i.e. , the Reader ) by augmenting its inputs with extra data from K . 3.1 CROSS-MODAL ALIGNMENT . LetK be a knowledge source consisting ofmmodalities , where each sample si = ( s0i , . . . , smi ) ∈ K is a tuple of m elements , corresponding to different modalities . Our alignment model encompasses m encoders Em , each composed of a feature-extraction module Fm , projection layer Pm , shared Transformer encoding layer T with attention pooling , and a normalization layer N : Em ( x ) = N ( T ( Pm ( Fm ( x ) ) ) ) ( 1 ) From this point , we will consider the two-modality case of images and captions as illustrated in Fig . 1 . For text and image feature extractors , F1 andF2 , we use a pre-trained BERT masked language model Devlin et al . ( 2018 ) , and a pre-trained ResNet152 CNN backbone on ImageNet , respectively . The images are represented with convolutional grid features , chosen for robustness and speed , and these are flattened across the spatial dimension . The projection layers Pm project each modality to a constant dimension d. The projected sequences are then forwarded to a shared Transformerencoding layer , and aggregated by an attention pooling layer , resulting in a vector representation for each modality . Finally , we normalize each vector using an L2 normalizer , projecting all embeddings to the unit-sphere . Following Faghri et al . ( 2017 ) , we only normalize the text embeddings because of image-caption imbalance ( see Sec . 4.1 ) . We train our dense cross-modal retriever ( DXR ) using a contrastive loss , specifically using an inbatch hinge penalty with hard negatives ( Faghri et al. , 2017 ) . Given a batch , consisting of b samples , s1 . . . sb , for each sample si , let s1i and s 2 i be the positive pairs and s 1 i and s 2 j 6=i the negative pairs . We compute the pair-wise similarity between the two modalities , using a dot product , denoted by π ( s1i , s 2 j ) = 〈s1i , s2j 〉 . The hard-negative in-batch hinge loss is then defined as : s2i ′ = max j 6=i π ( s1i , s 2 j ) ( 2 ) s 1 i ′ = max j 6=i π ( s1j , s 2 i ) ( 3 ) Lhard = ∑ i [ α+ π ( s1i , s 2 i ′ ) − π ( s1i , s2i ) ] + ∑ i [ α+ π ( s1i ′ , s2i ) − π ( s1i , s2i ) ] ( 4 ) where s1i ′ and s2i ′ are the hardest samples inside the batch , and α is the margin constant .

Summary: This paper proposed a cross-modal retrieval augmentation for the multi-modal classification task (VQA). The authors first introduce a transformer-based image caption retrieval architecture that achieves decent performance. Then, the authors proposed to use the retrieval model to retrieve relevant visual and textual information as augmentation. The proposed method experiment on 3 existing method (Visual Bert, ViLBERT, and Movie + MCAN) and show good improvements over the baseline model.