Probing Inter-modality: Visual Parsing with Self-Attention for Vision-and-Language Pre-training

1 Introduction . Vision-Language Pre-training ( VLP ) has shown great benefits for Visual-Language ( VL ) tasks such as Visual Question Answering ( VQA ) , Visual Entailment , etc. , in many recent works [ 9 , 21 , 31 , 32 , 36 , 40 , 42 , 51 ] . VLP is designed to learn vision and language ( VL ) joint representation and alignment with a huge number of image-text pairs . It provides a powerful initialization of multimodal representation for downstream VL tasks in a fine-tuning way . In this paper , we explore how to improve the multi-modal representation which is the key challenge in VLP from the perspective of intra-modal and inter-modal learning . ∗This work was performed when Hongwei Xue and Yupan Huang were visiting Microsoft Research as research interns . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Existing works use image features as visual tokens and learn their alignments with language tokens using a multi-modal Transformer . Three main types of image representation are commonly used in VLP : region feature , grid feature and patch projection . Most VLP models [ 9 , 42 , 40 ] extract region-based image features with an off-the-shelf object detector [ 34 ] . Each visual token in VLP corresponds to a pre-defined region feature . Some recent works directly learn grid features from images by CNNs to lift restrictions of bounding boxes and pre-defined object categories [ 22 , 21 ] . They train CNN and multi-modal Transformer in an end-to-end fashion to enable visual features optimized for pre-training objectives . In addition to use CNNs for visual feature learning , a recent work ViLT [ 28 ] directly input image patch projections into the multi-modal Transformer to achieve the fastest inference speed with the lightest VLP architecture . In vision-language ( VL ) tasks , the relation between visual concepts is crucial . For example , given a question “ what is the man doing ? ” for an image of a surfing man , a VL model is expected to infer the relation of “ surfing ” from object “ man ” and object “ surfboard ” . However , existing three types of image representations fail to model the intra-vision relation . For both region feature and patch projection , each unit ( i.e. , bounding box or patch ) is independent while global relations are not encoded in the visual embedding . For grid feature learned by CNNs , the local receptive field in the convolution layer results in local features of neighbor regions . Thus the two objectives of VLP , i.e. , learning visual relation and inter-modal alignment , are encapsulated in the multi-modal Transformer network . Such design might restrict the inter-modal alignment learning in the Transformer by ignoring the specialized characteristic of each objective . On the other hand , for language , texts are highly structured and relations are explicitly provided by grammar . This inconsistent representation of different modalities distracts the multi-modal Transformer from the inter-modal alignment . To better learn visual relation and further promote inter-modal alignment , we propose a fully Transformer VLP model which adopts self-attention for visual feature learning . Self-attention breaks the spatial inductive bias and enables long-range global relation learning of visual features . Thus , multi-modal Transformer can be specialized for multi-modal joint learning . In self-attention mechanism , each visual token is an approximate weighted mixture of all tokens . The higher weight indicates higher dependency . We name this way of image feature learning as visual parsing . As visual parsing provides dependencies of each visual token pair , inter-modality learning can be further promoted by masking visual tokens with high dependency , forcing the multi-modal Transformer to have more attention on the language side . To the best of our knowledge , our proposed Masked Feature Regression ( MFR ) is first ranking-based masking mechanism in VLP that leverages the intrinsic properties of visual extractor . To promote the inter-modality learning , we propose the Inter-Modality Flow ( IMF ) metric based on Attention Flow [ 1 ] to measure the fusion of inter-modality in VLP . IMF aims to quantify the information flow between the two modalities . To verify the effectiveness of our approach , we conduct experiments on a wide range of visionlanguage tasks that are highly related to visual relation understanding and inter-modal reasoning . Our approach not only outperforms the state-of-the-art VLP models , but also shows superiority on the proposed IMF metric . We also conduct extensive ablation studies to demonstrate the effectiveness of our proposed self-attention visual parsing and parsing-based masking mechanism . We thoroughly probe the inter-modality learning in VLP from the perspective of information flow and data distribution . Our probing reveals how vision and language fuse with each other . Our contributions in this paper are summarized as follows : 1 We are the first to adopt self-attention to learn visual features for VLP , aiming to promote intermodality learning in multi-modal Transformer . Our model outperforms existing works on a wide range of vision-language tasks . 2 We propose a novel Inter-Modality Flow ( IMF ) metric to measure and reveal vision and language fusion in VLP . 3 We design the first ranking-based masking mechanism for visual self-attention module to further promote inter-modality learning , verified by well-designed ablation studies . 2 Related Work . Multi-Modal Pre-training . Pioneering works of Vision-Language Pre-training like ViLBERT [ 36 ] and LXMERT [ 42 ] adopt two-stream architecture , which utilizes two separate Transformers to encode image and text modalities , and a third Transformer for multi-modal fusion . Recent works using single-stream architecture directly fuse the two modalities in a Transformer to automatically learn inter-and-intra modality fusion [ 40 , 32 , 31 , 9 ] . The majority of existing works advocate the singlestream designs , which show stronger performance and include fewer parameters than two-stream models . For above reasons , we mainly focus on one-stream architecture in this paper . Single-stream architectures typically employ Transformer to learn multi-modal contextualized features based on the singular embedding of each modality . For vision part , most VLP works use Bottom-Up and Top-Down attention [ 2 ] to extract region-level visual features by a Faster R-CNN [ 39 ] detector pre-trained on Visual Genome dataset [ 29 ] . Region-level features may miss the contextual information out of the bounding boxes and often suffer from low quality , noisy , and over-sampled boxes . To overcome the restrictions of region-based image features , SOHO [ 21 ] designs an end-toend VLP pipeline which directly learns image embedding at pixel-level by a CNN . This end-to-end learning paradigm does not require bounding box annotations and it enables inference 10 times faster than region-based approaches . A recent work ViLT [ 28 ] directly inputs patch projections into a multi-modal Transformer to achieve a fast inference speed . While ViLT has performance gaps with existing methods on some downstream tasks like VQA . Vision Transformer . As convolutional network only captures information in a local window , it will miss correlations of long-range features . To improve the capability of encoding distant dependencies or heterogeneous interactions , some works complement CNNs by extending self-attention modules [ 17 , 44 ] . The augmentation by self-attention also benefits general visual feature extraction . This kind of method has been applied for the object detection task [ 8 , 52 , 47 ] . The pioneering work of Vision Transformer ( ViT ) [ 13 ] totally abandons convolution and applies a Transformer architecture on image patch projections for image classification . Inspired by ViT , there are some recent works studying vision Transformer for a broad range of vision tasks such as image classification [ 35 , 43 ] , object detection [ 4 ] , semantic segmentation [ 50 ] . In this paper we adopt [ 35 ] as our visual backbone . Since CNN and Transformer favor different kinds of optimizer [ 48 ] , our fully Transformer structure makes it easier to train and fine-tune . We utilize unified optimizer for vision Transformer and multi-modal Transformer . 3 Approach . 3.1 Self-Attention Visual Parser . Despite grid features overcome the restrictions of region-based image features and keep all visual information in images [ 21 ] , CNN-based image feature learning remains challenging when utilized to vision-language pre-training . Specifically , convolution network tends to focus on local regions in images , adding the learning of global visual relation to the multi-modal Transformer . Instead of concentrating on inter-modal interaction learning , multi-modal Transformer has to learn intra-model interactions in visual modality as well . For the language side , texts are highly structured and they explicitly provide some relations from grammar . For example , prepositions and verbs often indicate the relations of objects . Thus , the two modalities are fed into the multi-modal Transformer at different information levels . This inconsistency makes Transformer tend to learn stand-alone representation of each modality . As a result , the multi-modal Transformer is distracted from focusing on learning inter-modal alignment . Inspired by Vision Transformer ( ViT ) [ 13 ] , we apply self-attention for visual embedding in visionlanguage pre-training . To learn visual features , we adopt a vision Transformer V T with parameters θ : V = V T ( I , θ ) ∈ Rm×c , ( 1 ) where I is the input image . m denotes the number of visual features , and c is the dimension of hidden states . To reduce computational complexity , we adopt Swin Transformer [ 35 ] , which has a hierarchical structure . Although shifted windows makes the receptive field smaller than the whole image , the large window size and gradual down-sampling mechanism make the spatial inductive bias much lower than CNN . We also explicitly get the correlations of visual tokens . In the last layer of vision Transformer , the correlation of each token pair can be modeled by attention weights which track how features get mixed . 3.2 Inter-modality Probing . In single-stream structure for Vision-Language Pre-training , a multi-modal Transformer is used to fuse the vision and language modality . To quantify the interactions between two modalities , we propose a metric named Inter-Modality Flow ( IMF ) to probe how vision and language fuse together . Transformer is a stack of self-attention modules and feed forward neural networks . The interactions of image-and-text are learnt in self-attention modules . Following Attention Flow [ 1 ] , we first compute values in layer l after excluding shared operations on each tokens as : Tl+1 = Tl +W l attTl , ( 2 ) where Tl and Tl+1 are the input and output of layer l. W latt is layer l ’ s attention weight . To account for residual connections , weights of mixture can be represented as : Al = 0.5 ( I +W latt ) . ( 3 ) To track down the information propagating in Transformer layers , we first calculate attention flows of arbitrary two layers , e.g. , layer i and layer j where j ≥ i : Ai , j = i∏ a=j Aa . ( 4 ) This calculation sums over all possible paths between two layers , and Ai , j is the attention weight matrix of layer i ’ s input to layer j ’ s output . Specifically , when j = i , Ai , i represents attention weights in layer i . To measure the interactions between two modalities , we propose an Inter-Modality Flow F i , jinter and it is computed as the proportion of inter-modal attention among all attentions : F i , jinter = A i , j inter/ ( A i , j inter +A i , j intra ) , ( 5 ) where Ai , jinter is the summation over all attentions of visual and language inter-modality features , while Ai , jintra sums over one modality , vision or language : Ai , jinter = ∑ ϕinter Ai , jx , y ; A i , j intra = ∑ ϕintra Ai , jx , y , ( 6 ) where ϕinter = { x ∈ V , y ∈ L or x ∈ L , y ∈ V } and ϕintra = { x , y ∈ V or x , y ∈ L } . L and V are token index set of vision and text . We calculate F i , jinter on pre-training data to probe how the two modalities interact with each other .

Summary: This paper introduces Swin Transformer in the visual domain for vision-language pretraining. Previous VL pretraining models feed image tokens and text tokens together into the cross-modal transformer, and the visual relationship learning and inter-modal alignment are encapsulated in the same transformer network. The authors propose that visual relationship between different visual objects or concepts is important for inter-modal alignment learning, and there should be independent processing of visual relationship in the framework. Motivated by this hypothesis, the authors add a visual-only transformer to learn visual relationship before feeding the image and text tokens together to the cross-modal transformer. In addition, the authors propose a metric named inter-modality flow (IMF) to measure the cross-modal interactions. They also propose masked feature regression as a new pretraining task. Pretraining is conducted on in-domain datasets COCO and Visual Genome. Evaluation is conducted on three downstream tasks VQA, visual reasoning, and visual entailment.