VL-BERT: Pre-training of Generic Visual-Linguistic Representations

1 INTRODUCTION . Pre-training of generic feature representations applicable to a variety of tasks in a domain is a hallmark of the success of deep networks . Firstly in computer vision , backbone networks designed for and pre-trained on ImageNet ( Deng et al. , 2009 ) classification are found to be effective for improving numerous image recognition tasks . Recently in natural language processing ( NLP ) , Transformer networks ( Vaswani et al. , 2017 ) pre-trained with “ masked language model ” ( MLM ) objective ( Devlin et al. , 2018 ) on large language corpus excel at a variety of NLP tasks . Meanwhile , for tasks at the intersection of vision and language , such as image captioning ( Young et al. , 2014 ; Chen et al. , 2015 ; Sharma et al. , 2018 ) , visual question answering ( VQA ) ( Antol et al. , 2015 ; Johnson et al. , 2017 ; Goyal et al. , 2017 ; Hudson & Manning , 2019 ) , visual commonsense reasoning ( VCR ) ( Zellers et al. , 2019 ; Gao et al. , 2019 ) , there lacks such pre-trained generic feature representations . The previous practice is to combine base networks pre-trained for image recognition and NLP respectively in a task-specific way . The task-specific model is directly finetuned for the specific target task , without any generic visual-linguistic pre-training . The task-specific model may well suffer from overfitting when the data for the target task is scarce . Also , due to the task-specific model design , it is difficult to benefit from pre-training , where the pre-training task may well be different from the target . There lacks a common ground for studying the feature design and pretraining of visual-linguistic tasks in general . In the various network architectures designed for different visual-linguistic tasks , a key goal is to effectively aggregate the multi-modal information in both the visual and linguistic domains . For example , to pick the right answer in the VQA task , the network should empower integrating linguistic information from the question and the answers , and aggregating visual information from the input image , together with aligning the linguistic meanings with the visual clues . Thus , we seek to derive generic representations that can effectively aggregate and align visual and linguistic information . ∗Equal contribution . This work is done when Weijie Su and Xizhou Zhu are interns at Microsoft Research Asia . †Corresponding author . In the meantime , we see the successful application of Transformer attention ( Vaswani et al. , 2017 ) in NLP , together with its MLM-based pre-training technique in BERT ( Devlin et al. , 2018 ) . The attention module is powerful and flexible in aggregating and aligning word embedded features in sentences , while the pre-training in BERT further enhances the capability . Inspired by that , we developed VL-BERT , a pre-trainable generic representation for visual-linguistic tasks , as shown in Figure 1 . The backbone of VL-BERT is of ( multi-modal ) Transformer attention module taking both visual and linguistic embedded features as input . In it , each element is either of a word from the input sentence , or a region-of-interest ( RoI ) from the input image , together with certain special elements to disambiguate different input formats . Each element can adaptively aggregate information from all the other elements according to the compatibility defined on their contents , positions , categories , and etc . The content features of a word / an RoI are domain specific ( WordPiece embeddings ( Wu et al. , 2016 ) as word features , Fast R-CNN ( Girshick , 2015 ) features for RoIs ) . By stacking multiple layers of multi-modal Transformer attention modules , the derived representation is of rich capability in aggregating and aligning visual-linguistic clues . And taskspecific branches can be added above for specific visual-linguistic tasks . To better exploit the generic representation , we pre-train VL-BERT at both large visual-linguistic corpus and text-only datasets1 . The pre-training loss on the visual-linguistic corpus is incurred via predicting randomly masked words or RoIs . Such pre-training sharpens the capability of VL-BERT in aggregating and aligning visual-linguistic clues . While the loss on the text-only corpus is of the standard MLM loss in BERT , improving the generalization on long and complex sentences . Comprehensive empirical evidence demonstrates that the proposed VL-BERT achieves state-of-theart performance on various downstream visual-linguistic tasks , such as visual commonsense reasoning , visual question answering and referring expression comprehension . In particular , we achieved the first place of single model on the leaderboard of visual commonsense reasoning . 2 RELATED WORK . Pre-training for Computer Vision Prior to the era of deep networks , it is far from mature to share features among different tasks and to improve the features via pre-training . The models for various computer vision tasks are of too diverse design choices to derive a generic representation . With the success of AlexNet ( Krizhevsky et al. , 2012 ) in ImageNet ( Deng et al. , 2009 ) classification , we see the renaissance of convolutional neural networks ( CNNs ) in the vision community . Soon after that , researchers found that ImageNet pre-trained CNNs can serve well as generic feature representation for various downstream tasks ( Donahue et al. , 2014 ) , such as object detection ( Girshick et al. , 2014 ) , semantic segmentation ( Long et al. , 2015 ) , instance segmentation ( Hariharan et al. , 2014 ) . The improvement in backbone networks for ImageNet classification further improves the downstream tasks . Recently there are research works on directly training CNNs from scratch on massive-scale target datasets , without ImageNet pre-training ( He et al. , 2018 ) . They achieved performance on par with those with ImageNet pre-training . While they also note that pre-training on a proper massive dataset is vital for improving performance on target tasks with scarce data . Pre-training for Natural Language Processing ( NLP ) It is interesting to note that the development of pre-training techniques in NLP lags quite behind computer vision . There are previous research works on improving word embedding ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ; Kiros et al. , 2015 ) , which is a low-level linguistic feature representation . On top of that , numerous diverse architectures are designed for various NLP tasks . In the milestone work of Transformers ( Vaswani et al. , 2017 ) , the Transformer attention module is proposed as a generic building block for various NLP tasks . After that , a serious of approaches are proposed for pre-training the generic representation , mainly based on Transformers , such as GPT ( Radford et al. , 2018 ) , BERT ( Devlin et al. , 2018 ) , GPT-2 ( Radford et al. , 2019 ) , XLNet ( Yang et al. , 2019 ) , XLM ( Lample & Conneau , 2019 ) , and RoBERTa ( Liu et al. , 2019 ) . Among them , BERT is perhaps the most popular one due to its simplicity and superior performance . Pre-training for Visual-Linguistic Tasks . The development course of models for visual-linguistic tasks is also quite similar to those in the computer vision and NLP communities . Previously , task- 1Here we exploit the Conceptual Captions dataset ( Sharma et al. , 2018 ) as the visual-linguistic corpus , and the BooksCorpus ( Zhu et al. , 2015 ) & English Wikipedia as the text-only corpus . specific models are designed , wherein the features derived from off-the-shelf computer vision and NLP models are combined in an ad-hoc way for specific tasks . Model training is performed on the dataset for the specific task only . VideoBERT ( Sun et al. , 2019b ) is the first work seeking to conduct pre-training for visual-linguistic tasks . In it , video clips are processed by off-the-shelf networks for action recognition , and are assigned to different clusters ( visual words ) based on the derived features . The pre-training loss is incurred via predicting the cluster ids of masked video clips . Due to the abrupt clustering of the video clips , it losses considerable visual content information and hinders updating visual network parameters . In the following work of CBT ( Sun et al. , 2019a ) , such clustering mechanism is removed . Both works are applied on videos , which are of linear structure in the time dimension , same as sentences . It is highly desired to study at the well-established image-based visual-linguistic tasks . Concurrent to our work , multiple works released on Arxiv very recently also seek to derive a pretrainable generic representation for visual-linguistic tasks . Table 5 in Appendix compares among them . We briefly discuss some of these works here . In ViLBERT ( Lu et al. , 2019 ) and LXMERT ( Tan & Bansal , 2019 ) , which are under review or just got accepted , the network architectures are of two single-modal networks applied on input sentences and images respectively , followed by a cross-modal Transformer combining information from the two sources . The attention pattern in the cross-modal Transformer is restricted , where the authors believe to improve the performance . The authors of ViLBERT claim that such two-stream design is superior than a single-stream unified model . Meanwhile , in the proposed VL-BERT , it is of a unified architecture based on Transformers without any restriction on the attention patterns . The visual and linguistic contents are fed as input to VL-BERT , wherein they interact early and freely . We found that our unified model of VL-BERT outperforms such two-stream designs . VisualBert ( Li et al. , 2019b ) , B2T2 ( Alberti et al. , 2019 ) , and Unicoder-VL ( Li et al. , 2019a ) , which are of work in progress or under review , are also of unified single-stream architecture . The differences of these works are compared in Table 5 . The concurrent emergency of these research works indicates the importance of deriving a generic pre-trainable representation for visual-linguistic tasks . In addition , there are three noticeable differences between VL-BERT and other concurrent works in pre-training . Their effects are validated in Section 4.3 . ( 1 ) We found the task of Sentence-Image Relationship Prediction used in all of the other concurrent works ( e.g. , ViLBERT ( Lu et al. , 2019 ) and LXMERT ( Tan & Bansal , 2019 ) ) is of no help in pre-training visual-linguistic representations . Thus such a task is not incorporated in VL-BERT . ( 2 ) We pre-train VL-BERT on both visual-linguistic and text-only datasets . We found such joint pre-training improves the generalization over long and complex sentences . ( 3 ) Improved tuning of the visual representation . In VL-BERT , the parameters of Fast R-CNN , deriving the visual features , are also updated . To avoid visual clue leakage in the pre-training task of Masked RoI Classification with Linguistic Clues , the masking operation is conducted on the input raw pixels , other than the feature maps produced by layers of convolution . 3 VL-BERT . 3.1 REVISIT BERT MODEL . Let x = { x1 , ... , xN } be the input elements in BERT ( Devlin et al. , 2018 ) , which are of embedded features encoding sentence words . They are processed by a multi-layer bidirectional Transformer ( Vaswani et al. , 2017 ) , where the embedding features of each element are transformed layer-by-layer in the fashion of aggregating features from the other elements with adaptive attention weights . Let xl = { xl1 , ... , xlN } be the features of the l-th layer ( x0 is set as the input x ) . The features of the ( l + 1 ) -th layer , xl+1 , is computed by h̃l+1i = M∑ m=1 W l+1m { N∑ j=1 Ami , j · V l+1m xlj } Multi-head Attention , ( 1 ) hl+1i = LayerNorm ( x l i + h̃ l+1 i ) Residual Connection , ( 2 ) x̃l+1i =W l+1 2 · GELU ( W l+1 1 h l+1 i + b l+1 1 ) + b l+1 2 Feed-forward , ( 3 ) xl+1i = LayerNorm ( h l+1 i + x̃ l+1 i ) Residual Connection , ( 4 ) where m in Eq . 1 indexes over the attention heads , and Ami , j ∝ exp [ ( Ql+1m xli ) T ( Kl+1m xlj ) ] denotes the attention weights between elements i and j in the m-th head , which is normalized by∑N j=1A m i , j = 1 . W l+1 m , Q l+1 m , K l+1 m and V l+1 m are learnable weights for m th attention head , W l+11 , W l+1 2 and b l+1 1 , b l+1 2 in Eq . 3 are learnable weights and biases , respectively . Note that , the operations in Eq . 1 ∼ 4 is irrelevant to the order of input sequence , i.e . the final BERT representation of permuted input is same as the final BERT representation of the original input after the same permutation . The position of an element in BERT is encoded in its own embedding features by sequence positional embedding . Thanks to such decoupled representation , the BERT model is flexible enough to be pre-trained and finetuned for a variety of NLP tasks . In BERT pre-training , the masked language modeling ( MLM ) task is introduced . The embedded features of a certain input word would be randomly masked out ( the token embedding channels capturing the word content is replaced by a special [ MASK ] token ) . The BERT model is trained to predict the masked word from linguistic clues of all the other unmasked elements . As explained in Wang & Cho ( 2019 ) , the overall MLM-based training of BERT is equivalent to optimizing the following joint probability distribution logP ( x|θ ) = 1 Z ( θ ) N∑ i=1 log φi ( x|θ ) , ( 5 ) where φi ( x|θ ) is the potential function for the i-th input element , with parameters θ , and Z ( θ ) is the partition function . Each log-potential term log φi ( x ) is defined as log φi ( x|θ ) = xTi fi ( x\i|θ ) i , ( 6 ) where fi ( x\i|θ ) denotes the final output feature of BERT corresponding to the i-th element for input x\i , where x\i is defined as x\i = { x1 , ... , xi−1 , [ MASK ] , xi+1 , ... , xN } . The incurred MLM-based loss is as LMLM ( θ ) = −Ex∼D , i∼ { 1 , ... , N } log φi ( x ) , ( 7 ) where x is a randomly sampled sentence from the training set D , and i is a randomly sampled location for masking words . The second pre-training task , Next Sentence Prediction , focuses on modeling the relationship between two sentences . Two sentences are sampled from the input document , and the model should predict whether the second sentence is the direct successor of the first . In BERT , the sampled two sentences are concatenated into one input sequence , with special elements [ CLS ] and [ SEP ] inserted prior to the first and the second sentences , respectively . A Sigmoid classifier is appended on the final output feature corresponding to the [ CLS ] element to make the prediction . Let x be the input sequence , t ∈ { 0 , 1 } indicates the relationship between the two sentences . The loss function is defined as LNSP ( θ ) = −E ( x , t ) ∼D [ t log ( g ( xL0 ) ) + ( 1− t ) log ( 1− g ( xL0 ) ) ] , ( 8 ) where xL0 is the final output feature of the [ CLS ] element ( at the L-th layer ) , and g ( x L 0 ) is the classifier output .

Summary: This paper proposed a pre-trainable generic representation for visual-linguistic tasks call VL-BERT. VL-BERT extend BERT by changing the input from subsequent sentence to image regions and modify the caption words has the additional visual feature embedding. The authors pre-train the VL-BERT on the conceptual caption dataset and Wikipedia and book corpus dataset, empirical results show that the VL-BERT achieve the SOTA performance on the VCR, VQA and refer expression tasks.