FILIP: Fine-grained Interactive Language-Image Pre-Training

1 INTRODUCTION . Large-scale Vision-Language Pre-training ( VLP ) models like CLIP ( Radford et al. , 2021 ) and ALIGN ( Jia et al. , 2021 ) have recently demonstrated success across various downstream tasks . They learn visual and textual representations from millions of image-text pairs collected from the Internet and show superior zero-shot ability and robustness . The core technique of these models lies in the global contrastive alignment of the images and texts through a dual-stream model . Such architecture is inference-efficient for downstream tasks like retrieval because the encoders for the two modalities can be decoupled and the image or text representations can be pre-computed offline . However , CLIP and ALIGN model the cross-modal interaction via solely the similarity of the global feature of each modality , lacking the ability of capturing finer-level information like the relationship between visual objects and textual words . In this paper , we develop a simple yet efficient cross-modal finer-grained interaction mechanism for large-scale VLP . To achieve finer-grained cross-modal interaction , previous methods mainly exploited two kinds of methods . ( 1 ) One line of work ( Chen et al. , 2020 ; Li et al. , 2020b ; Dong et al. , 2021 ; Li et al. , 2021b ; Zhang et al. , 2021 ; Zhan et al. , 2021 ) uses a pre-trained object detector to extract region-of-interest ( ROI ) features from images , and then fuses it with the paired text through a VLP model . This design complicates the pre-training due to pre-computing and storing a large number of ROI features . In addition , the zero-shot ability of these approaches is usually limited by the predefined number of classes and their performance is also restricted by the quality of the detector . ( 2 ) Another line of work ( Li et al. , 2021a ; Kim et al. , 2021 ) enforces the token-wise or patch-wise representations from both modalities into the same space and models these finer-grained interactions via cross-attention ( Li et al. , 2021a ) or self-attention ( Kim et al. , 2021 ) . However , these methods are usually less efficient in terms of both training and inference . In particular , during training , cross-attention in ( Li et al. , 2021a ) requires to be performed in an encoder-decoder structure , while the complexity of the self-attention ( Kim et al. , 2021 ) grows quadratically with the length of the prolonged concatenated sequences of both modalities . During inference , the data from both modalities are intertwined to compute the cross-attention or self-attention , and can not be pre-computed offline as dual-stream models like CLIP and ALIGN . This can be less efficient for downstream tasks like image/text retrieval and image classification . In this paper , we propose a large-scale Fine-grained Interactive Language-Image Pre-training framework named FILIP to address these limitations . Inspired by Khattab & Zaharia ( 2020 ) , we model the fine-grained semantic alignment through a novel cross-modal late interaction mechanism in the contrastive loss , instead of using cross or self-attention . Specifically , our fine-grained contrastive learning uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective . In this way , FILIP successfully leverages the finer-grained expressiveness among image patches and textual words while simultaneously gaining the ability to pre-compute image and text representations offline . Unlike Khattab & Zaharia ( 2020 ) , we discard the padded tokens and use average instead summation of token-wise maximum similarities when computing the image-text alignment , which enhances the cross-modal representation learning and stabilizes training . Furthermore , we construct a large-scale pre-training dataset named FILIP300M from the Internet . Data cleaning and image-text data augmentation are also explored and proved useful in this work . Extensive experiments show that by effectively learning fine-grained representations , FILIP achieves state-of-the-art performance on multiple downstream tasks , including zero-shot image classification and image-text retrieval . For example , FILIP reaches 77.1 % top-1 accuracy for zero-shot ImageNet classification , surpassing CLIP with less training data . Visualizations on word-patch alignment further show that FILIP learns meaningful finer-grained features with promising localization ability . 2 RELATED WORK . Vision-Language Pre-training Models . The pre-train-and-fine-tune scheme has achieved great success in the domains of natural language processing ( Devlin et al. , 2019 ; Brown et al. , 2020 ) and computer vision ( Dosovitskiy et al. , 2020 ) . It is then naturally extended to a joint cross-modal domain of Vision-and-Language Pre-training ( VLP ) . The pre-training datasets of recent VLP models include publically available datasets like YFCC100M ( Thomee et al. , 2016 ) and CC12M ( Changpinyo et al. , 2021 ) , as well as larger-scale datasets with more than 100M samples in CLIP ( Radford et al. , 2021 ) and ALIGN ( Jia et al. , 2021 ) , which are shown to be even more powerful . The pretraining tasks of VLP models can be categorized into two categories : image-text contrastive learning task and Language Modeling ( LM ) based tasks : ( i ) CLIP ( Radford et al. , 2021 ) , ALIGN ( Jia et al. , 2021 ) and UNIMO ( Li et al. , 2021b ) make use of cross-modal contrastive learning which aligns the textual and visual information into a unified semantic space ; ( ii ) VisualBERT ( Li et al. , 2019 ) , UNITER ( Chen et al. , 2020 ) , M6 ( Lin et al. , 2021 ) , and DALL-E ( Ramesh et al. , 2021 ) employ LM-like objectives , including both masked LM ( e.g. , Masked Language/Region Modeling ) , and autoregressive LM ( e.g. , image captioning , text-grounded image generation ) . On the other hand , some methods rely on a pre-trained object detection model such as Faster-RCNN ( Ren et al. , 2015 ) to extract image regional features offline , which requires extra labeled bounding-box data and makes the approach less scalable . Recent efforts such as SOHO ( Huang et al. , 2021 ) and SimVLM ( Wang et al. , 2021 ) try to eliminate this burden via visual dictionary or PrefixLM ( Raffel et al. , 2020 ) . In this paper , we directly learn fine-grained vision-language representations in an end-to-end and simpler manner while maintaining the benefit of inference efficiency . Multi-Modality Interaction Mechanism . The core of vision-language pre-training models lies in modeling the interaction between the two modalities . There are mainly two types of cross-modal interaction architectures : Single-stream models like VisualBERT ( Li et al. , 2019 ) and ViLT ( Kim et al. , 2021 ) directly concatenate the patch-wise or regional visual features and textual embeddings and feed them to the transformer-based model . Dual-stream models such as ViLBERT ( Lu et al. , 2019 ) and CLIP ( Radford et al. , 2021 ) have separate encoders for different modalities . This allows flexible use of different models for different modalities , and efficient inference for downstream tasks like image-text retrieval , through the ability of decoupling the encoders and pre-compute image/text features offline . SCAN ( Lee et al. , 2018 ) considers latent alignments between image regions and words . However , it is based on Triplet loss with a bottom-Up attention via a Faster-RCNN to extract object features while we try to directly learn to localize fine-grained object from patches . In this paper , while following the dual-stream approach for its flexible and efficient inference , we further propose a new multi-modal interaction mechanism to capture the fine-grained representations . 3 METHOD . In this paper , we propose a new cross-modal pre-training model that excels in fine-grained interaction between image encoder and text encoder for mining more detailed semantic alignment , named as FILIP , as shown in Figure 1 . Particularly , FILIP is a dual-stream model with Transformer-based image and text encoders . For the visual modality , the image encoder is a Vision Transformer ( Dosovitskiy et al. , 2020 ) which takes the concatenation of an extra [ CLS ] token embedding and linearly projected image patches as input . For the textual modality , following Radford et al . ( 2021 ) , we use the lower-cased byte pair encoding ( BPE ) ( Sennrich et al. , 2016b ) with a vocabulary size of 49,408 to tokenize the text . Each text sequence starts with [ BOS ] token and ends with [ EOS ] token . After the word embedding layer , the token embeddings are fed into a modified decoder-only Transformer model as in ( Radford et al. , 2019 ) . On top of the image and text encoders , the representations of textual tokens and visual tokens are linearly projected to the multi-modal common space , and are separately L2-normalized . Different from existing dual-stream models ( e.g. , CLIP and ALIGN ) which models cross-modal interaction via only the global features of the entire image and text sequence , we introduce a novel fine-grained contrastive learning objective equipped with cross-modal late interaction which takes into account the fine-grained interaction between image patches and textual tokens , detailed in Section 3.1 . 3.1 FINE-GRAINED CONTRASTIVE LEARNING . Contrastive representation learning has recently been found to learn better representations than its predictive counterpart in both visual ( Tian et al. , 2020 ) and vision-language cross-modal pre-training ( Radford et al. , 2021 ) . Under a general formulation of cross-modal contrastive learning ( Radford et al. , 2021 ) , we want to learn encoders fθ for image data I and gφ for text data T such that , given an image xI ∈ I , and a text xT ∈ T , the encoded representations fθ ( xI ) and gφ ( xT ) are close if they are related and far apart if not , under a distance metric . In each training batch , we sample b image-text pairs { xIk , xTk } bk=1 , For image xIk in image-text pair { xIk , xTk } , xTk is its positive , while the other texts will be used as in-batch negatives . The image-to-text contrastive loss LIk for xIk can then be formulated as LIk ( xIk , { xTj } bj=1 ) = − 1 b log exp ( sIk , k ) ∑ j exp ( s I k , j ) , where sIk , j denotes the similarity of the k-th image to the j-th text . Similarly , the text-to-image contrastive loss for xTk is LTk ( xTk , { xIj } bj=1 ) = − 1 b log exp ( sTk , k ) ∑ j exp ( s T j , k ) . The total loss of this mini-batch can be represented by L = 1 2 ∑b k=1 ( LIk + LTk ) . ( 1 ) 3.1.1 CROSS-MODAL LATE INTERACTION . From the contrastive loss ( 1 ) , the cross-modal interaction is reflected in how we compute the similarities sIi , j and s T i , j for the i-th image and j-th text . Previous methods like CLIP ( Radford et al. , 2021 ) and ALIGN ( Jia et al. , 2021 ) simply encode each image or text separately to a global feature i.e. , fθ ( xIi ) ∈ Rd and gφ ( xTj ) ∈ Rd , and compute these two similarities as sIi , j = s T i , j = fθ ( x I i ) > gφ ( x T j ) , ( 2 ) neglecting finer-grained interactions ( e.g. , word-patch alignment ) between the two modalities . To alleviate this problem , while simultaneously maintain the training and inference efficiency of dualstream models , we apply a cross-modal late interaction inspired by Khattab & Zaharia ( 2020 ) to model the token-wise cross-modal interaction . Specifically , denote n1 and n2 as the number of ( non-padded ) tokens of the i-th image and j-th text , respectively , and the corresponding encoded features are fθ ( xIi ) ∈ Rn1×d and gφ ( xTj ) ∈ Rn2×d . For the k-th visual token , we compute its similarities with all textual tokens of xTj , and use the largest one max 0≤r < n2 [ fθ ( x I i ) ] > k [ gφ ( x T j ) ] r ( 3 ) as its token-wise maximum similarity with xTj . We then use the average token-wise maximum similarity of all non-padded tokens in the image ( resp . text ) as the similarity of an image to a text ( resp . a text to an image ) . The similarity of the i-th image to the j-th text can thus be formulated as : sIi , j ( x I i , x T j ) = 1 n1 n1∑ k=1 [ fθ ( x I i ) ] > k [ gφ ( x T j ) ] mIk , ( 4 ) where mIk = argmax0≤r < n2 [ fθ ( x I i ) ] > k [ gφ ( x T j ) ] r. Similarly , the similarity of the j-th text to the i-th image is sTi , j ( x I i , x T j ) = 1 n2 n2∑ k=1 [ fθ ( x I i ) ] > mTk [ gφ ( x T j ) ] k , ( 5 ) where mTk = argmax0≤r < n1 [ fθ ( x I i ) ] > r [ gφ ( x T j ) ] k. Note that s I i , j ( x I i , x T j ) in Equation ( 4 ) does not necessarily equal sTi , j ( x I i , x T j ) in Equation ( 5 ) . Remark 1 Intuitively , the token-wise maximum similarity in Equation ( 3 ) means that for each image patch , we find its most similar textual token . Similarly , for each textual token , we also find its closest image patch . By applying this to the similarity calculation in ( 4 ) and ( 5 ) for contrastive loss ( 1 ) , the dual-stream model learns fine-grained alignment between image patches and textual tokens . The original late interaction mechanism in ( Khattab & Zaharia , 2020 ) computes the relevance score of a document to a query padded with mask tokens , as a sum of token-wise maximum similarities , and is optimized via a pairwise softmax cross-entropy loss . Though inspired from Khattab & Zaharia ( 2020 ) , our proposed cross-modal late interaction differs in several aspects . Firstly , we exclude the padded textual tokens when computing the similarity , as they harm the performance . We speculate that this is because these padded tokens also learn textual representations and will mislead the model to align image patches to these meaningless padded tokens rather than meaningful non-padded words . Secondly , when computing similarities ( 4 ) and ( 5 ) , we use the average of the token-wise maximum similarities instead of summation in ( Khattab & Zaharia , 2020 ) . This is because the number of non-padded tokens varies from text to text , and this summation over all non-padded tokens can have quite different magnitudes , leading to less stabilized training and worse final performance . These two modifications are crutial to not only the downstream tasks ’ performance , but also the quality of the word-patch alignment . A more detailed discussion can be found in Appendix A.6 . Thirdly , we optimize the late interaction mechanism via a contrastive loss ( 1 ) which is found powerful vision-language pre-training ( Radford et al. , 2021 ) instead of the original pairwise loss in ( Khattab & Zaharia , 2020 ) . Training Efficiency . Though the cross-modal late interaction is able to capture finer-grained features compared with the original loss , it relies on the token-wise representations of both modalities , and can be inefficient in terms of communication , memory and computation , especially when the batch size is large . To alleviate this problem , we utilize several methods . Firstly , we reduce the embedding size to 256 . Besides , we reduce the precision of the last-layer features of both modalities from fp32 to fp16 before node communication in a distributed learning setting , and perform the multiplication in Equations ( 4 ) and ( 5 ) under the reduced precision . In addition , since the complexity of similarity calculation scales with the sequence length of textual tokens and image patches , for each image ( resp . text ) , we select the 25 % tokens with the highest token-wise maximum similarity score ( Equation ( 3 ) ) among all texts ( resp . images ) in the same local worker before node communication , based on the intuition that each sample can be represented by a few of the most representative tokens . Effects of these modifications are studied in Section 4.4 .

Summary: The paper tackles the problem of large-scale vision-language pre-training which has been quite popular recently. The main issues identified by the paper are: 1) fine-grained information might be lost due to the use of global features, and 2) alignment between text and language is typically done using cross- or self-attention, which are inefficient. The authors proposed a dual-stream method that 1) uses visual "patch" tokens from ViT models and 2) perform late fusion of the two modalities using a variant of the method from Khattab & Zaharia (2020).