FILIP: Fine-grained Interactive Language-Image Pre-Training

1 INTRODUCTION . Large-scale Vision-Language Pre-training ( VLP ) models like CLIP ( Radford et al. , 2021 ) and ALIGN ( Jia et al. , 2021 ) have recently demonstrated success across various downstream tasks . They learn visual and textual representations from millions of image-text pairs collected from the Internet and show superior zero-shot ability and robustness . The core technique of these models lies in the global contrastive alignment of the images and texts through a dual-stream model . Such architecture is inference-efficient for downstream tasks like retrieval because the encoders for the two modalities can be decoupled and the image or text representations can be pre-computed offline . However , CLIP and ALIGN model the cross-modal interaction via solely the similarity of the global feature of each modality , lacking the ability of capturing finer-level information like the relationship between visual objects and textual words . In this paper , we develop a simple yet efficient cross-modal finer-grained interaction mechanism for large-scale VLP . To achieve finer-grained cross-modal interaction , previous methods mainly exploited two kinds of methods . ( 1 ) One line of work ( Chen et al. , 2020 ; Li et al. , 2020b ; Dong et al. , 2021 ; Li et al. , 2021b ; Zhang et al. , 2021 ; Zhan et al. , 2021 ) uses a pre-trained object detector to extract region-of-interest ( ROI ) features from images , and then fuses it with the paired text through a VLP model . This design complicates the pre-training due to pre-computing and storing a large number of ROI features . In addition , the zero-shot ability of these approaches is usually limited by the predefined number of classes and their performance is also restricted by the quality of the detector . ( 2 ) Another line of work ( Li et al. , 2021a ; Kim et al. , 2021 ) enforces the token-wise or patch-wise representations from both modalities into the same space and models these finer-grained interactions via cross-attention ( Li et al. , 2021a ) or self-attention ( Kim et al. , 2021 ) . However , these methods are usually less efficient in terms of both training and inference . In particular , during training , cross-attention in ( Li et al. , 2021a ) requires to be performed in an encoder-decoder structure , while the complexity of the self-attention ( Kim et al. , 2021 ) grows quadratically with the length of the prolonged concatenated sequences of both modalities . During inference , the data from both modalities are intertwined to compute the cross-attention or self-attention , and can not be pre-computed offline as dual-stream models like CLIP and ALIGN . This can be less efficient for downstream tasks like image/text retrieval and image classification . In this paper , we propose a large-scale Fine-grained Interactive Language-Image Pre-training framework named FILIP to address these limitations . Inspired by Khattab & Zaharia ( 2020 ) , we model the fine-grained semantic alignment through a novel cross-modal late interaction mechanism in the contrastive loss , instead of using cross or self-attention . Specifically , our fine-grained contrastive learning uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective . In this way , FILIP successfully leverages the finer-grained expressiveness among image patches and textual words while simultaneously gaining the ability to pre-compute image and text representations offline . Unlike Khattab & Zaharia ( 2020 ) , we discard the padded tokens and use average instead summation of token-wise maximum similarities when computing the image-text alignment , which enhances the cross-modal representation learning and stabilizes training . Furthermore , we construct a large-scale pre-training dataset named FILIP300M from the Internet . Data cleaning and image-text data augmentation are also explored and proved useful in this work . Extensive experiments show that by effectively learning fine-grained representations , FILIP achieves state-of-the-art performance on multiple downstream tasks , including zero-shot image classification and image-text retrieval . For example , FILIP reaches 77.1 % top-1 accuracy for zero-shot ImageNet classification , surpassing CLIP with less training data . Visualizations on word-patch alignment further show that FILIP learns meaningful finer-grained features with promising localization ability . 2 RELATED WORK . Vision-Language Pre-training Models . The pre-train-and-fine-tune scheme has achieved great success in the domains of natural language processing ( Devlin et al. , 2019 ; Brown et al. , 2020 ) and computer vision ( Dosovitskiy et al. , 2020 ) . It is then naturally extended to a joint cross-modal domain of Vision-and-Language Pre-training ( VLP ) . The pre-training datasets of recent VLP models include publically available datasets like YFCC100M ( Thomee et al. , 2016 ) and CC12M ( Changpinyo et al. , 2021 ) , as well as larger-scale datasets with more than 100M samples in CLIP ( Radford et al. , 2021 ) and ALIGN ( Jia et al. , 2021 ) , which are shown to be even more powerful . The pretraining tasks of VLP models can be categorized into two categories : image-text contrastive learning task and Language Modeling ( LM ) based tasks : ( i ) CLIP ( Radford et al. , 2021 ) , ALIGN ( Jia et al. , 2021 ) and UNIMO ( Li et al. , 2021b ) make use of cross-modal contrastive learning which aligns the textual and visual information into a unified semantic space ; ( ii ) VisualBERT ( Li et al. , 2019 ) , UNITER ( Chen et al. , 2020 ) , M6 ( Lin et al. , 2021 ) , and DALL-E ( Ramesh et al. , 2021 ) employ LM-like objectives , including both masked LM ( e.g. , Masked Language/Region Modeling ) , and autoregressive LM ( e.g. , image captioning , text-grounded image generation ) . On the other hand , some methods rely on a pre-trained object detection model such as Faster-RCNN ( Ren et al. , 2015 ) to extract image regional features offline , which requires extra labeled bounding-box data and makes the approach less scalable . Recent efforts such as SOHO ( Huang et al. , 2021 ) and SimVLM ( Wang et al. , 2021 ) try to eliminate this burden via visual dictionary or PrefixLM ( Raffel et al. , 2020 ) . In this paper , we directly learn fine-grained vision-language representations in an end-to-end and simpler manner while maintaining the benefit of inference efficiency . Multi-Modality Interaction Mechanism . The core of vision-language pre-training models lies in modeling the interaction between the two modalities . There are mainly two types of cross-modal interaction architectures : Single-stream models like VisualBERT ( Li et al. , 2019 ) and ViLT ( Kim et al. , 2021 ) directly concatenate the patch-wise or regional visual features and textual embeddings and feed them to the transformer-based model . Dual-stream models such as ViLBERT ( Lu et al. , 2019 ) and CLIP ( Radford et al. , 2021 ) have separate encoders for different modalities . This allows flexible use of different models for different modalities , and efficient inference for downstream tasks like image-text retrieval , through the ability of decoupling the encoders and pre-compute image/text features offline . SCAN ( Lee et al. , 2018 ) considers latent alignments between image regions and words . However , it is based on Triplet loss with a bottom-Up attention via a Faster-RCNN to extract object features while we try to directly learn to localize fine-grained object from patches . In this paper , while following the dual-stream approach for its flexible and efficient inference , we further propose a new multi-modal interaction mechanism to capture the fine-grained representations . 3 METHOD . In this paper , we propose a new cross-modal pre-training model that excels in fine-grained interaction between image encoder and text encoder for mining more detailed semantic alignment , named as FILIP , as shown in Figure 1 . Particularly , FILIP is a dual-stream model with Transformer-based image and text encoders . For the visual modality , the image encoder is a Vision Transformer ( Dosovitskiy et al. , 2020 ) which takes the concatenation of an extra [ CLS ] token embedding and linearly projected image patches as input . For the textual modality , following Radford et al . ( 2021 ) , we use the lower-cased byte pair encoding ( BPE ) ( Sennrich et al. , 2016b ) with a vocabulary size of 49,408 to tokenize the text . Each text sequence starts with [ BOS ] token and ends with [ EOS ] token . After the word embedding layer , the token embeddings are fed into a modified decoder-only Transformer model as in ( Radford et al. , 2019 ) . On top of the image and text encoders , the representations of textual tokens and visual tokens are linearly projected to the multi-modal common space , and are separately L2-normalized . Different from existing dual-stream models ( e.g. , CLIP and ALIGN ) which models cross-modal interaction via only the global features of the entire image and text sequence , we introduce a novel fine-grained contrastive learning objective equipped with cross-modal late interaction which takes into account the fine-grained interaction between image patches and textual tokens , detailed in Section 3.1 . 3.1 FINE-GRAINED CONTRASTIVE LEARNING . Contrastive representation learning has recently been found to learn better representations than its predictive counterpart in both visual ( Tian et al. , 2020 ) and vision-language cross-modal pre-training ( Radford et al. , 2021 ) . Under a general formulation of cross-modal contrastive learning ( Radford et al. , 2021 ) , we want to learn encoders fθ for image data I and gφ for text data T such that , given an image xI ∈ I , and a text xT ∈ T , the encoded representations fθ ( xI ) and gφ ( xT ) are close if they are related and far apart if not , under a distance metric . In each training batch , we sample b image-text pairs { xIk , xTk } bk=1 , For image xIk in image-text pair { xIk , xTk } , xTk is its positive , while the other texts will be used as in-batch negatives . The image-to-text contrastive loss LIk for xIk can then be formulated as LIk ( xIk , { xTj } bj=1 ) = − 1 b log exp ( sIk , k ) ∑ j exp ( s I k , j ) , where sIk , j denotes the similarity of the k-th image to the j-th text . Similarly , the text-to-image contrastive loss for xTk is LTk ( xTk , { xIj } bj=1 ) = − 1 b log exp ( sTk , k ) ∑ j exp ( s T j , k ) . The total loss of this mini-batch can be represented by L = 1 2 ∑b k=1 ( LIk + LTk ) . ( 1 ) 3.1.1 CROSS-MODAL LATE INTERACTION . From the contrastive loss ( 1 ) , the cross-modal interaction is reflected in how we compute the similarities sIi , j and s T i , j for the i-th image and j-th text . Previous methods like CLIP ( Radford et al. , 2021 ) and ALIGN ( Jia et al. , 2021 ) simply encode each image or text separately to a global feature i.e. , fθ ( xIi ) ∈ Rd and gφ ( xTj ) ∈ Rd , and compute these two similarities as sIi , j = s T i , j = fθ ( x I i ) > gφ ( x T j ) , ( 2 ) neglecting finer-grained interactions ( e.g. , word-patch alignment ) between the two modalities . To alleviate this problem , while simultaneously maintain the training and inference efficiency of dualstream models , we apply a cross-modal late interaction inspired by Khattab & Zaharia ( 2020 ) to model the token-wise cross-modal interaction . Specifically , denote n1 and n2 as the number of ( non-padded ) tokens of the i-th image and j-th text , respectively , and the corresponding encoded features are fθ ( xIi ) ∈ Rn1×d and gφ ( xTj ) ∈ Rn2×d . For the k-th visual token , we compute its similarities with all textual tokens of xTj , and use the largest one max 0≤r < n2 [ fθ ( x I i ) ] > k [ gφ ( x T j ) ] r ( 3 ) as its token-wise maximum similarity with xTj . We then use the average token-wise maximum similarity of all non-padded tokens in the image ( resp . text ) as the similarity of an image to a text ( resp . a text to an image ) . The similarity of the i-th image to the j-th text can thus be formulated as : sIi , j ( x I i , x T j ) = 1 n1 n1∑ k=1 [ fθ ( x I i ) ] > k [ gφ ( x T j ) ] mIk , ( 4 ) where mIk = argmax0≤r < n2 [ fθ ( x I i ) ] > k [ gφ ( x T j ) ] r. Similarly , the similarity of the j-th text to the i-th image is sTi , j ( x I i , x T j ) = 1 n2 n2∑ k=1 [ fθ ( x I i ) ] > mTk [ gφ ( x T j ) ] k , ( 5 ) where mTk = argmax0≤r < n1 [ fθ ( x I i ) ] > r [ gφ ( x T j ) ] k. Note that s I i , j ( x I i , x T j ) in Equation ( 4 ) does not necessarily equal sTi , j ( x I i , x T j ) in Equation ( 5 ) . Remark 1 Intuitively , the token-wise maximum similarity in Equation ( 3 ) means that for each image patch , we find its most similar textual token . Similarly , for each textual token , we also find its closest image patch . By applying this to the similarity calculation in ( 4 ) and ( 5 ) for contrastive loss ( 1 ) , the dual-stream model learns fine-grained alignment between image patches and textual tokens . The original late interaction mechanism in ( Khattab & Zaharia , 2020 ) computes the relevance score of a document to a query padded with mask tokens , as a sum of token-wise maximum similarities , and is optimized via a pairwise softmax cross-entropy loss . Though inspired from Khattab & Zaharia ( 2020 ) , our proposed cross-modal late interaction differs in several aspects . Firstly , we exclude the padded textual tokens when computing the similarity , as they harm the performance . We speculate that this is because these padded tokens also learn textual representations and will mislead the model to align image patches to these meaningless padded tokens rather than meaningful non-padded words . Secondly , when computing similarities ( 4 ) and ( 5 ) , we use the average of the token-wise maximum similarities instead of summation in ( Khattab & Zaharia , 2020 ) . This is because the number of non-padded tokens varies from text to text , and this summation over all non-padded tokens can have quite different magnitudes , leading to less stabilized training and worse final performance . These two modifications are crutial to not only the downstream tasks ’ performance , but also the quality of the word-patch alignment . A more detailed discussion can be found in Appendix A.6 . Thirdly , we optimize the late interaction mechanism via a contrastive loss ( 1 ) which is found powerful vision-language pre-training ( Radford et al. , 2021 ) instead of the original pairwise loss in ( Khattab & Zaharia , 2020 ) . Training Efficiency . Though the cross-modal late interaction is able to capture finer-grained features compared with the original loss , it relies on the token-wise representations of both modalities , and can be inefficient in terms of communication , memory and computation , especially when the batch size is large . To alleviate this problem , we utilize several methods . Firstly , we reduce the embedding size to 256 . Besides , we reduce the precision of the last-layer features of both modalities from fp32 to fp16 before node communication in a distributed learning setting , and perform the multiplication in Equations ( 4 ) and ( 5 ) under the reduced precision . In addition , since the complexity of similarity calculation scales with the sequence length of textual tokens and image patches , for each image ( resp . text ) , we select the 25 % tokens with the highest token-wise maximum similarity score ( Equation ( 3 ) ) among all texts ( resp . images ) in the same local worker before node communication , based on the intuition that each sample can be represented by a few of the most representative tokens . Effects of these modifications are studied in Section 4.4 .

Summary: (1) The core idea of the proposed FILIP method is to achieve word-patch alignment by token-wise similarity matrix through cross-modal late interaction by modifying only contrastive loss, which is both training and inference efficient. The authors collected FILIP300M, a large-scale cleaned image captioning dataset from the Internet for FILIP’s V-L pre-training.   (2) FILIP achieves SoTA performance on ZS image classification and ITM tasks. Visualization results show its promising ability of fine-grained (visual-textual token) classification and localization.   (3) To the best of our knowledge, FILIP is the first work with such high accuracy on F30k that not only does it reach double full mark of R@K (i.e. i2t[R@5 and R@10] are both 100.0), but also it achieves 580+ of rsum with only one single model (no ensemble).