SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

1 INTRODUCTION . Self-supervised textual representation learning ( Devlin et al. , 2018 ; Radford et al. , 2018 ; 2019 ; Liu et al. , 2019 ; Yang et al. , 2019 ; Raffel et al. , 2019 ; Brown et al. , 2020 ) based on Transformers ( Vaswani et al. , 2017 ) has pushed the state of the art on a wide range of natural language processing ( NLP ) tasks ( Rajpurkar et al. , 2016 ; Wang et al. , 2018 ; Sarlin et al. , 2020 ) . One successful approach is to first pretrain the model ( e.g . BERT ) on large-scale unlabled text corpora using masked language modeling ( MLM ) objective ( Devlin et al. , 2018 ) , followed by finetuning on downstream tasks . While this pretraining-finetuning paradigm has been widely adopted , recent work on autoregressive language models ( LM ) ( Radford et al. , 2019 ; Brown et al. , 2020 ) such as GPT-3 has shown strong performance without finetuning by utilizing few-shot prompts ( Liu et al. , 2021 ) , suggesting the text guided zero-shot generalization is a promising alternative . Motivated by the success of textual representation pretraining , various efforts have been made to build the multi-modal ( visual and textual ) counterpart . A line of work ( Tan & Bansal , 2019 ; Lu et al. , 2019 ; Li et al. , 2019 ; Chen et al. , 2020b ; Li et al. , 2020 ; Su et al. , 2020 ; Zhang et al. , 2021 ) has explored vision-language pretraining ( VLP ) that learns a joint representation of both modalities to be finetuned on vision-language ( VL ) benchmarks , such as visual question answering ( VQA ) ( Goyal et al. , 2017 ) . In order to capture the alignment between images and text , previous methods have extensively exploited two types of human-labeled datasets from multiple sources , which typically consist of the following steps . Firstly , object detection datasets are used to train a supervised object detector ( OD ) which allows further extracting region-of-interest ( ROI ) features from images . Next , datasets with aligned image-text pairs are used for MLM pretraining of a fusion model that usually takes as input the concatenation of the extracted ROI features and the paired text . In addition , due to the limited scale of human annotated data , various task-specific auxiliary losses have been introduced in order to improve performance . These design choices complicate the pretraining protocol of VLP , creating a bottleneck for further quality improvement . What is more , such pretraining-finetuning based approaches usually lack the zero-shot capability , just like their lan- guage counterparts . In comparison , another line of work ( Radford et al. , 2021 ; Ramesh et al. , 2021 ; Jia et al. , 2021 ) utilizes weakly labeled/aligned data crawled from the web to perform pretraining , achieving good performance and certain zero-shot learning capability on image classification and image-text retrieval . Nonetheless , these methods mainly focus on specific tasks of consideration and thus may not serve as a generic pretraining-finetuning representation for VL benchmarks . In light of these disadvantages of the existing techniques , we are interested in building a VLP model that : ( 1 ) can be seamlessly plugged into the pretraining-finetuning paradigm and achieve competitive performance on standard VL benchmarks ; ( 2 ) does not require a complicated pretraining protocol as in previous methods ; and ( 3 ) has the potential towards text guided zero-shot generalization in cross-modal settings . To this end , we propose SimVLM , standing for Simple Visual Language Model , which significantly simplifies VLP by solely exploiting language modeling objectives on weakly aligned image-text pairs ( Jia et al. , 2021 ) . In a nutshell , SimVLM consists of the following components : • Objective . It is trained end-to-end from scratch with a single objective of Prefix Language Modeling ( PrefixLM ) , which can not only naturally perform text generation as GPT-3 , but also process contextual information in a bidirectional manner as BERT does . • Architecture . The framework employs ViT/CoAtNet ( Dosovitskiy et al. , 2021 ; Dai et al. , 2021 ) and directly takes raw images as inputs . These models can also fit the large-scale data and are readily compatible with the PrefixLM objective . • Data . These setups relieve the requirement for object detection and allow the model to utilize the large-scale weakly labeled dataset , which has better potential towards zero-shot generalization . Not only is SimVLM simpler , requiring neither object detection pretraining nor auxiliary losses , but it also obtains better performance than previous work . Empirically , SimVLM consistently outperforms existing VLP models and achieves new state-of-the-art results on 6 VL benchmarks without additional data nor task-specific customization . Besides , it acquires stronger generalization in visual-language understanding that empowers zero-shot image captioning and open-ended VQA . In particular , SimVLM learns unified multimodal representation that enables zero-shot cross-modality transfer , where the model is finetuned on text-only data and directly evaluated on image-and-text test examples without further training . Our results suggest that generative VLP can not only match existing MLM-based methods on VL benchmarks but also demonstrate promising zero-shot potential . 2 RELATED WORK . Recent years have seen a rapid progress made in vision-language pretraining ( Uppal et al. , 2020 ; Han et al. , 2021 ; Khan et al. , 2021 ) . While a variety of approaches have been proposed , a large portion of them require object detection for image region feature regression or tagging as part of the pre-training objectives ( Tan & Bansal , 2019 ; Su et al. , 2020 ; Li et al. , 2019 ; Chen et al. , 2020b ; Gan et al. , 2020 ; Li et al. , 2020 ; Yu et al. , 2021 ; Li et al. , 2021 ; Zhang et al. , 2021 ; Hu et al. , 2021 ; Cho et al. , 2021 ) . These methods rely on a strong object detection model like Fast ( er ) R-CNN ( Ren et al. , 2015 ) , which is often trained on human annotated data sets like Visual Genome ( Krishna et al. , 2016 ) . Using such labeled training data as a prerequisite increases the cost of building the training pipeline , and makes the approach less scalable . Some recent efforts have also explored VLP without object detection module ( Xu et al. , 2021 ; Kim et al. , 2021 ; Huang et al. , 2021 ) , but they only use clean pretraining data with small scales and thus their zero-shot capability is limited . On the other hand , multiple cross-modality loss functions have been proposed as part of the training objectives , for example image-text matching ( Tan & Bansal , 2019 ; Lu et al. , 2019 ; Xu et al. , 2021 ) , masked region classification/feature regression ( Tan & Bansal , 2019 ; Chen et al. , 2020b ) , object attribute prediction ( Xu et al. , 2021 ) , contrastive loss ( Li et al. , 2020 ; 2021 ) , word-region alignment ( Chen et al. , 2020b ) word-patch alignment ( Kim et al. , 2021 ) . They are often mixed with other objectives including image caption generation and masked language modeling to form compound pre-training losses . This creates the challenge of balancing among different losses and datasets , and thus complicates the optimization procedure . Our work by contrast , follows a minimalist approach that takes raw image inputs and makes use of only the language modeling loss , without resorting to auxiliary models like faster R-CNN for image region detection . Motivated by recent works ( Radford et al. , 2021 ; Ramesh et al. , 2021 ; Jia et al. , 2021 ) that illustrate zero-shot learning in certain image-text tasks , we train our model using largescale weakly labeled data only . While concurrent work ( Shen et al. , 2021 ) has explored building on top of models pretrained with such dataset , we focus on pretraining from scratch to explore the limit of generative VLP . 3 SIMVLM . 3.1 BACKGROUND . The bidirectional Masked Language Modeling ( MLM ) has been one of the most popular selfsupervised training objectives for textual representation learning . As demonstrated by BERT ( Devlin et al. , 2018 ) , it is based on the idea of denoising autoencoder such that the model is trained to recover the corrupted tokens in a document . Specifically , given a text sequence x , a subset of tokens xm are randomly sampled and a corrupted sequence x\m is constructed by replacing tokens in xm with a special [ MASK ] token . The training objective is to reconstruct xm from the context x\m by minimizing the negative log-likelihood : LMLM ( θ ) = −Ex∼D [ logPθ ( xm|x\m ) ] , ( 1 ) where θ is the trainable parameters of the model and D is the pretraining data . This approach learns contextualized representations that can be further finetuned for downstream tasks . The MLM-style pretraining has been widely adopted in previous VLP models , whereby the input is an image-text pair and the model needs to predict masked tokens by leveraging image ROI features . Alternatively , the unidirectional Language Modeling ( LM ) trains the model to directly maximize the likelihood of the sequence x under the forward autoregressive factorization : LLM ( θ ) = −Ex∼D [ logPθ ( x ) ] = −Ex∼D [ T∑ t=1 logPθ ( xt|x < t ) ] . ( 2 ) Compared with MLM , the LM pretraining has also been shown to be highly effective for multiple NLP tasks ( Radford et al. , 2018 ) . More importantly , it facilitates the model with strong generation capability that enables text induced zero-shot generalization without finetuning ( Brown et al. , 2020 ) . While MLM has become the de facto approach in VLP models reviewed above , the generative LM has been understudied . 3.2 PROPOSED OBJECTIVE : PREFIX LANGUAGE MODELING . Motivated by the zero-shot capability introduced by pre-training with LM loss , we propose to pretain vision-language representation using the Prefix Language Modeling ( PrefixLM ) . PrefixLM differs from the standard LM such that it enables bi-directional attention on the prefix sequence ( e.g . x < Tp in Eq . ( 3 ) ) , and only conducts autoregressive factorization on the remaining tokens ( e.g . x≥Tp in Eq . ( 3 ) ) . During pretraining , a prefix sequence of tokens of ( a randomly selected ) length Tp is truncated from input sequence and the training objective becomes : LPrefixLM ( θ ) = −Ex∼D [ logPθ ( x≥Tp |x < Tp ) ] = −Ex∼D T∑ t=Tp Pθ ( xt|x [ Tp , t ] , x < Tp ) . ( 3 ) Intuitively , images can be considered as prefix for their textual descriptions as they often appear before text in a web document . Therefore , for a given image-text pair , we prepend image feature sequence of length Ti to the text sequence , and enforce the model to sample a prefix of length Tp ≥ Ti to calculate LM loss on text data only ( an example is shown in Figure 1 ) . Compared to prior MLM style VLP methods , our PrefixLM model under the sequence-to-sequence framework not only enjoys the bidirectional contextualized representation as in MLM , but also can perform text generation similar to LM .

Summary: The paper proposes to pre-train a generative language model conditioned on a visual input on billion-scale web image-text data. Such a model can be then transferred to various vision-and-language tasks with ease. SimVLM establishes new SotA on several new tasks and shows promising zero-shot capacity in certain tasks.