UNITER: Learning UNiversal Image-TExt Representations

1 INTRODUCTION . Most Vision-and-Language tasks rely on joint multimodel embeddings to bridge the semantic gap between visual and textual clues in images and text , although such representations are usually tailored for specific tasks . For example , MCB ( Fukui et al. , 2017 ) , BAN ( Kim et al. , 2018 ) , DFAF ( Gao et al. , 2019 ) proposed advanced multimodal fusion methods for Visual Question Answering ( VQA ) ( Antol et al. , 2015 ) . SCAN ( Lee et al. , 2018 ) and MAttNet ( Yu et al. , 2018 ) studied learning latent alignment between words and image regions for Image-Text Retrieval ( Wang et al. , 2016 ) and Referring Expression Comprehension ( Kazemzadeh et al. , 2014 ) tasks . While each of these proposed models has pushed the state of the art on respective benchmarks , their architectures are diverse and the learned representations are highly task-specific , preventing them from being generalized to other tasks . This raises a million-dollar question : can we learn a universal image-text representation for all V+L tasks ? To answer this question , we introduce UNiversal Image-TExt Representations ( UNITER ) , a largescale pre-trained model for multimodal embedding . We adopt Transformer ( Vaswani et al. , 2017 ) as the core of our model , to leverage its elegant self-attention mechanism designed for learning contextualized representations . Inspired by BERT ( Devlin et al. , 2019 ) , which has successfully applied Transformer to NLP tasks through large-scale language modeling , we pre-train UNITER through three pre-training tasks : ( i ) Masked Language Modeling ( MLM ) conditioned on image ; ( ii ) Masked Region Modeling ( MRM ) conditioned on text ; and ( iii ) joint Image-Text Matching ( ITM ) . To further investigate the effectiveness of MRM , we propose three MRM variants : ( i ) Masked Region Classification ( MRC ) ; ( ii ) Masked Region Feature Regression ( MRFR ) ; and ( iii ) Masked Region Classification with KL-divergence ( MRC-kl ) . As shown in Figure 1 , UNITER first encodes image regions ( visual features and bounding box features ) and textual words ( tokens and positions ) into a common embedding space with Image Embedder and Text Embedder , then applies a Transformer module to learn generalizable contex- tualized embeddings for each region and word through aforementioned pre-training tasks . Compared with LXMERT ( Tan & Bansal , 2019 ) and ViLBERT ( Lu et al. , 2019 ) that use two streams ( one Transformer for each modality ) , our UNITER model can learn joint contextualized representations for image regions and textual words through a single Transformer . Besides , our masked language/region modeling is conditioned on full observation of image/text , different from other concurrent pre-trained models that apply joint random masking to both modalities . We show that the conditional masking strategy can successfully ease the missing-alignment between images and text , and obtain better joint embeddings for downstream tasks . Detailed ablation study also demonstrates that the combination of MLM+ITM+MRC-kl+MRFR yields the best pre-training performance . To demonstrate the power of UNITER , we evaluate on six V+L tasks across nine datasets , including : ( i ) VQA ; ( ii ) Visual Commonsense Reasoning ( VCR ) ( Zellers et al. , 2019 ) ; ( iii ) NLVR2 ( Suhr et al. , 2019 ) ; ( iv ) Visual Entailment ( Xie et al. , 2019 ) ; ( v ) Image-Text Retrieval ( including zero-shot setting ) ( Lee et al. , 2018 ) ; and ( vi ) Referring Expression Comprehension . Our UNITER model is trained on a large-scale V+L dataset composed of four subsets : ( i ) COCO ( Lin et al. , 2014 ) ; ( ii ) Visual Genome ( VG ) ( Krishna et al. , 2017 ) ; ( iii ) Conceptual Captions ( CC ) ( Sharma et al. , 2018 ) ; and ( iv ) SBU Captions ( Ordonez et al. , 2011 ) . Experiments show that UNITER achieves new state of the art with significant performance boost across all six downstream tasks . Moreover , training on additional CC and SBU data ( containing unseen images/text in downstream tasks ) further boosts model performance over training on COCO and VG only . Our contributions can be summarized as follows : ( i ) We introduce UNITER , a powerful UNiversal Image-TExt Representations for Vision-and-Language tasks . ( ii ) We achieve new state of the art ( SOTA ) on multiple V+L benchmarks , outperforming existing SOTA and concurrent multimodal pre-training methods by a large margin . ( iii ) We present extensive experiments and analysis to provide useful insights on the effectiveness of each pre-training task/dataset for multimodal encoder training . 2 RELATED WORK . Self-supervised learning utilizes original data as its own source of supervision , which has been applied to many Computer Vision tasks , such as image colorization ( Zhang et al. , 2016 ) , solving jigsaw puzzles ( Noroozi & Favaro , 2016 ; Trinh et al. , 2019 ) , inpainting ( Pathak et al. , 2016 ) , rotation prediction ( Gidaris et al. , 2018 ) , and relative location prediction ( Doersch et al. , 2015 ) . Recently , pre-trained language models such as ELMo ( Peters et al. , 2018 ) , BERT ( Devlin et al. , 2019 ) , GPT2 ( Radford et al. , 2019 ) , and XLNet ( Yang et al. , 2019 ) have shown great advances for NLP tasks . There are two keys to their success : effective pre-training tasks over large language corpus , and the use of Transformer ( Vaswani et al. , 2017 ) for learning contextualized text representations . More recently , there has been some concurrent work on self-supervised learning for multimodal tasks , by pre-training on large-scale image/video and text pairs , then finetuning on downstream tasks . For example , VideoBERT ( Sun et al. , 2019 ) applied BERT to learn a bidrectional joint distribution over quantized video frame features and linguistic tokens from video-text pairs . ViLBERT ( Lu et al. , 2019 ) and LXMERT ( Tan & Bansal , 2019 ) introduced the two-stream architecture , where two Transformers are applied to images and text independently , which will be fused by a third Transformer in a later stage . On the other hand , VisualBERT ( Li et al. , 2019b ) , Unicoder-VL ( Li et al. , 2019a ) , VL-BERT ( Su et al. , 2019 ) and B2T2 ( Alberti et al. , 2019 ) proposed the singlestream architecture , where a single Transformer is applied to both image and text . Specifically , LXMERT model was pre-trained with downstream tasks such as VQA ( Antol et al. , 2015 ) and GQA ( Hudson & Manning , 2019 ) , while the others were pre-trained on image-text pairs only . Our UNITER model belongs to the second family . One key difference between UNITER and the other methods is the masking approach on pre-training tasks . Instead of randomly masking both image regions and sentence words , we use conditional masking , i.e. , masking only one modality while keeping the other untainted . In addition , we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks . Another related work is DDAF ( Gao et al. , 2019 ) , which proposed a novel architecture of intermodality and intra-modality attention modules to learn the latent alignment between two modalities for VQA . Compared with Gao et al . ( 2019 ) , UNITER learns a relatively more generic V+L representation via pre-training . 3 UNIVERSAL IMAGE-TEXT REPRESENTATIONS . In this section , we first introduce the model architecture of UNITER ( Section 3.1 ) , then describe the designed pre-training tasks and V+L datasets used for pre-training ( Section 3.2 and 3.3 ) . 3.1 MODEL OVERVIEW . The model architecture of UNITER is illustrated in Figure 1 . Given a pair of image and sentence , UNITER takes the visual regions of the image and textual tokens of the sentence as the input . We design an Image Embedder and a Text Embedder to extract their respective embeddings . These embeddings are then feed it into a multi-layer self-attention Transformer to learn a cross-modality contextualized embedding between visual regions and textual tokens . Note that the self-attention mechanism in Transformer is order-less , thus it is necessary to explicitly encode positions/locations of tokens/regions as additional inputs . Specifically , in Image Embedder , we first use Faster R-CNN1 to extract the visual features ( pooled ROI features ) for each region . We also encode the location features for each region via a 7- dimensional vector2 . Both visual and location features are then fed through a fully-connected ( FC ) layer , to be projected into the same embedding space . The final visual embedding for each region is obtained by summing up the two FC outputs and then passing through a layer normalization ( LN ) layer . For Text Embedder , we follow BERT ( Devlin et al. , 2019 ) and tokenize the input sentence into WordPieces ( Wu et al. , 2016 ) . The final representation for each sub-word token3 is obtained via summing up its word embedding and position embedding , followed by another LN layer4 . We introduce three main tasks to pre-train our model : Masked Language Modeling conditioned on image regions ( MLM ) , Masked Region Modeling conditioned on input text ( with three variants ) ( MRM ) , and Image-Text Matching ( ITM ) . As shown in Figure 1 , our MRM and MLM are in analogy to BERT , where we randomly mask some words or regions from the input and learn to recover the words or regions as the output of Transformer . Specifically , word masking is realized by replacing the token with a special token [ MASK ] , and region masking is implemented by replacing the visual feature vector with all zeros . Note that each time we only mask one modality while keeping the other modality intact , instead of randomly masking both modalities like ViLBERT and LXMERT . This prevents potential miss-alignment when a masked region happens to be described by a masked word . Empirically , we show that with conditional masking , our model is able to learn better embeddings 1Our Faster R-CNN was pre-trained on Visual Genome object+attribute data ( Anderson et al. , 2018 ) . 2 [ x1 , y1 , x2 , y2 , w , h , w ∗ h ] ( normalized top/left/bottom/right coordinates , width , height , and area . ) 3We use word/sub-word and token interchangeably throughout the rest of the paper . 4We also use a special modality embedding to help the model distinguish between textual and visual input , which is similar to the ‘ segment embedding ’ in BERT . This embedding is also summed before the LN layer in each embedder . For simplicity , this modality embedding is omitted in Figure 1 . ( in Section 4.2 ) . Lastly , we also learn an instance-level alignment ( rather than token/region-level ) between the whole image and the sentence via ITM . During training , we sample both positive and negative image-sentence pairs and learn their matching scores . To pre-train UNITER with the aforementioned different tasks , we randomly sample one pre-training task for each mini-batch and train on only one objective per SGD update .

Summary: The authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).