SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

1 INTRODUCTION . Self-supervised textual representation learning ( Devlin et al. , 2018 ; Radford et al. , 2018 ; 2019 ; Liu et al. , 2019 ; Yang et al. , 2019 ; Raffel et al. , 2019 ; Brown et al. , 2020 ) based on Transformers ( Vaswani et al. , 2017 ) has pushed the state of the art on a wide range of natural language processing ( NLP ) tasks ( Rajpurkar et al. , 2016 ; Wang et al. , 2018 ; Sarlin et al. , 2020 ) . One successful approach is to first pretrain the model ( e.g . BERT ) on large-scale unlabled text corpora using masked language modeling ( MLM ) objective ( Devlin et al. , 2018 ) , followed by finetuning on downstream tasks . While this pretraining-finetuning paradigm has been widely adopted , recent work on autoregressive language models ( LM ) ( Radford et al. , 2019 ; Brown et al. , 2020 ) such as GPT-3 has shown strong performance without finetuning by utilizing few-shot prompts ( Liu et al. , 2021 ) , suggesting the text guided zero-shot generalization is a promising alternative . Motivated by the success of textual representation pretraining , various efforts have been made to build the multi-modal ( visual and textual ) counterpart . A line of work ( Tan & Bansal , 2019 ; Lu et al. , 2019 ; Li et al. , 2019 ; Chen et al. , 2020b ; Li et al. , 2020 ; Su et al. , 2020 ; Zhang et al. , 2021 ) has explored vision-language pretraining ( VLP ) that learns a joint representation of both modalities to be finetuned on vision-language ( VL ) benchmarks , such as visual question answering ( VQA ) ( Goyal et al. , 2017 ) . In order to capture the alignment between images and text , previous methods have extensively exploited two types of human-labeled datasets from multiple sources , which typically consist of the following steps . Firstly , object detection datasets are used to train a supervised object detector ( OD ) which allows further extracting region-of-interest ( ROI ) features from images . Next , datasets with aligned image-text pairs are used for MLM pretraining of a fusion model that usually takes as input the concatenation of the extracted ROI features and the paired text . In addition , due to the limited scale of human annotated data , various task-specific auxiliary losses have been introduced in order to improve performance . These design choices complicate the pretraining protocol of VLP , creating a bottleneck for further quality improvement . What is more , such pretraining-finetuning based approaches usually lack the zero-shot capability , just like their lan- guage counterparts . In comparison , another line of work ( Radford et al. , 2021 ; Ramesh et al. , 2021 ; Jia et al. , 2021 ) utilizes weakly labeled/aligned data crawled from the web to perform pretraining , achieving good performance and certain zero-shot learning capability on image classification and image-text retrieval . Nonetheless , these methods mainly focus on specific tasks of consideration and thus may not serve as a generic pretraining-finetuning representation for VL benchmarks . In light of these disadvantages of the existing techniques , we are interested in building a VLP model that : ( 1 ) can be seamlessly plugged into the pretraining-finetuning paradigm and achieve competitive performance on standard VL benchmarks ; ( 2 ) does not require a complicated pretraining protocol as in previous methods ; and ( 3 ) has the potential towards text guided zero-shot generalization in cross-modal settings . To this end , we propose SimVLM , standing for Simple Visual Language Model , which significantly simplifies VLP by solely exploiting language modeling objectives on weakly aligned image-text pairs ( Jia et al. , 2021 ) . In a nutshell , SimVLM consists of the following components : • Objective . It is trained end-to-end from scratch with a single objective of Prefix Language Modeling ( PrefixLM ) , which can not only naturally perform text generation as GPT-3 , but also process contextual information in a bidirectional manner as BERT does . • Architecture . The framework employs ViT/CoAtNet ( Dosovitskiy et al. , 2021 ; Dai et al. , 2021 ) and directly takes raw images as inputs . These models can also fit the large-scale data and are readily compatible with the PrefixLM objective . • Data . These setups relieve the requirement for object detection and allow the model to utilize the large-scale weakly labeled dataset , which has better potential towards zero-shot generalization . Not only is SimVLM simpler , requiring neither object detection pretraining nor auxiliary losses , but it also obtains better performance than previous work . Empirically , SimVLM consistently outperforms existing VLP models and achieves new state-of-the-art results on 6 VL benchmarks without additional data nor task-specific customization . Besides , it acquires stronger generalization in visual-language understanding that empowers zero-shot image captioning and open-ended VQA . In particular , SimVLM learns unified multimodal representation that enables zero-shot cross-modality transfer , where the model is finetuned on text-only data and directly evaluated on image-and-text test examples without further training . Our results suggest that generative VLP can not only match existing MLM-based methods on VL benchmarks but also demonstrate promising zero-shot potential . 2 RELATED WORK . Recent years have seen a rapid progress made in vision-language pretraining ( Uppal et al. , 2020 ; Han et al. , 2021 ; Khan et al. , 2021 ) . While a variety of approaches have been proposed , a large portion of them require object detection for image region feature regression or tagging as part of the pre-training objectives ( Tan & Bansal , 2019 ; Su et al. , 2020 ; Li et al. , 2019 ; Chen et al. , 2020b ; Gan et al. , 2020 ; Li et al. , 2020 ; Yu et al. , 2021 ; Li et al. , 2021 ; Zhang et al. , 2021 ; Hu et al. , 2021 ; Cho et al. , 2021 ) . These methods rely on a strong object detection model like Fast ( er ) R-CNN ( Ren et al. , 2015 ) , which is often trained on human annotated data sets like Visual Genome ( Krishna et al. , 2016 ) . Using such labeled training data as a prerequisite increases the cost of building the training pipeline , and makes the approach less scalable . Some recent efforts have also explored VLP without object detection module ( Xu et al. , 2021 ; Kim et al. , 2021 ; Huang et al. , 2021 ) , but they only use clean pretraining data with small scales and thus their zero-shot capability is limited . On the other hand , multiple cross-modality loss functions have been proposed as part of the training objectives , for example image-text matching ( Tan & Bansal , 2019 ; Lu et al. , 2019 ; Xu et al. , 2021 ) , masked region classification/feature regression ( Tan & Bansal , 2019 ; Chen et al. , 2020b ) , object attribute prediction ( Xu et al. , 2021 ) , contrastive loss ( Li et al. , 2020 ; 2021 ) , word-region alignment ( Chen et al. , 2020b ) word-patch alignment ( Kim et al. , 2021 ) . They are often mixed with other objectives including image caption generation and masked language modeling to form compound pre-training losses . This creates the challenge of balancing among different losses and datasets , and thus complicates the optimization procedure . Our work by contrast , follows a minimalist approach that takes raw image inputs and makes use of only the language modeling loss , without resorting to auxiliary models like faster R-CNN for image region detection . Motivated by recent works ( Radford et al. , 2021 ; Ramesh et al. , 2021 ; Jia et al. , 2021 ) that illustrate zero-shot learning in certain image-text tasks , we train our model using largescale weakly labeled data only . While concurrent work ( Shen et al. , 2021 ) has explored building on top of models pretrained with such dataset , we focus on pretraining from scratch to explore the limit of generative VLP . 3 SIMVLM . 3.1 BACKGROUND . The bidirectional Masked Language Modeling ( MLM ) has been one of the most popular selfsupervised training objectives for textual representation learning . As demonstrated by BERT ( Devlin et al. , 2018 ) , it is based on the idea of denoising autoencoder such that the model is trained to recover the corrupted tokens in a document . Specifically , given a text sequence x , a subset of tokens xm are randomly sampled and a corrupted sequence x\m is constructed by replacing tokens in xm with a special [ MASK ] token . The training objective is to reconstruct xm from the context x\m by minimizing the negative log-likelihood : LMLM ( θ ) = −Ex∼D [ logPθ ( xm|x\m ) ] , ( 1 ) where θ is the trainable parameters of the model and D is the pretraining data . This approach learns contextualized representations that can be further finetuned for downstream tasks . The MLM-style pretraining has been widely adopted in previous VLP models , whereby the input is an image-text pair and the model needs to predict masked tokens by leveraging image ROI features . Alternatively , the unidirectional Language Modeling ( LM ) trains the model to directly maximize the likelihood of the sequence x under the forward autoregressive factorization : LLM ( θ ) = −Ex∼D [ logPθ ( x ) ] = −Ex∼D [ T∑ t=1 logPθ ( xt|x < t ) ] . ( 2 ) Compared with MLM , the LM pretraining has also been shown to be highly effective for multiple NLP tasks ( Radford et al. , 2018 ) . More importantly , it facilitates the model with strong generation capability that enables text induced zero-shot generalization without finetuning ( Brown et al. , 2020 ) . While MLM has become the de facto approach in VLP models reviewed above , the generative LM has been understudied . 3.2 PROPOSED OBJECTIVE : PREFIX LANGUAGE MODELING . Motivated by the zero-shot capability introduced by pre-training with LM loss , we propose to pretain vision-language representation using the Prefix Language Modeling ( PrefixLM ) . PrefixLM differs from the standard LM such that it enables bi-directional attention on the prefix sequence ( e.g . x < Tp in Eq . ( 3 ) ) , and only conducts autoregressive factorization on the remaining tokens ( e.g . x≥Tp in Eq . ( 3 ) ) . During pretraining , a prefix sequence of tokens of ( a randomly selected ) length Tp is truncated from input sequence and the training objective becomes : LPrefixLM ( θ ) = −Ex∼D [ logPθ ( x≥Tp |x < Tp ) ] = −Ex∼D T∑ t=Tp Pθ ( xt|x [ Tp , t ] , x < Tp ) . ( 3 ) Intuitively , images can be considered as prefix for their textual descriptions as they often appear before text in a web document . Therefore , for a given image-text pair , we prepend image feature sequence of length Ti to the text sequence , and enforce the model to sample a prefix of length Tp ≥ Ti to calculate LM loss on text data only ( an example is shown in Figure 1 ) . Compared to prior MLM style VLP methods , our PrefixLM model under the sequence-to-sequence framework not only enjoys the bidirectional contextualized representation as in MLM , but also can perform text generation similar to LM .

Summary: This paper proposes a Prefix Language Modeling (PrefixLM) objective for a pretraining procedure for multiple vision-language downstream tasks and zero-shot evaluations. They argue that it successfully replaces the masked language model (MLM). This work follows the notion of weak supervision suggested by ALIGN (Jia et al., 2021), having noisy image captions and holistic labels. Surprisingly, this simple pretraining approach achieves new state-of-the-art on "a wide range of discriminative and generative vision-language benchmarks," and shows "strong generalization and transfer ability" in zero-shot settings.