How Much Can CLIP Benefit Vision-and-Language Tasks?

Most existing Vision-and-Language ( V & L ) models rely on pre-trained visual encoders , using a relatively small set of manually-annotated data ( as compared to web-crawled data ) , to perceive the visual world . However , it has been observed that large-scale pre-training usually can result in better generalization performance , e.g. , CLIP ( Contrastive Language-Image Pre-training ) , trained on a massive amount of image-caption pairs , has shown a strong zero-shot capability on various vision tasks . To further study the advantage brought by CLIP , we propose to use CLIP as the visual encoder in various V & L models in two typical scenarios : 1 ) plugging CLIP into task-specific fine-tuning ; 2 ) combining CLIP with V & L pre-training and transferring to downstream tasks . We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data , such as BottomUp-TopDown . We achieve competitive or better results on diverse V & L tasks , while establishing new state-of-the-art results on Visual Question Answering , Visual Entailment , and V & L Navigation tasks . 1 INTRODUCTION . Vision-and-Language ( V & L ) tasks such as VQA ( Antol et al. , 2015 ) test a system ’ s ability to understand and reason about the semantics of the visual world with the help of natural language . Most V & L models rely on a visual encoder to perceive the visual world , which translates the raw pixels into vectors from a representation space . Recent work ( Anderson et al. , 2018a ; Jiang et al. , 2020 ; Zhang et al. , 2021 ) observes that the visual representation has become the performance bottleneck of V & L models and stress the importance of learning a powerful visual encoder . These high-performing visual encoders are trained on manually-annotated data with class labels ( e.g. , ImageNet ) ( Russakovsky et al. , 2015 ) or bounding boxes ( e.g. , Visual Genome ) ( Krishna et al. , 2017 ) . However , such detection or image classification data is costly to collect , and the visual representation is limited by the predefined class labels . Thus , there is a need for a visual encoder that is trained on more diverse and large-scale data sources , unbounded by a fixed set of labels , and with generalization ability to unseen objects and concepts . Recently , CLIP ( Radford et al. , 2021 ) has been proposed to learn visual concepts with language supervision . CLIP consists of a visual encoder and a text encoder . It is trained on 400M noisy imagetext pairs crawled from the Internet . Thus , the data collection process is scalable and requires little human annotation . CLIP has shown strong zero-shot capabilities on benchmarks such as ImageNet classification . We hypothesize that it also bears great potential for the V & L tasks . However , directly applying CLIP as a zero-shot model to V & L tasks proves to be difficult ( Section 5 and Kim et al . ( 2021 ) ) , as many V & L tasks require complex multi-modal reasoning . Thus , we propose to integrate CLIP with existing V & L models by replacing their visual encoder with CLIP ’ s visual encoder.1 To the best of our knowledge , we present the first large-scale empirical study on using CLIP as the visual encoder for diverse V & L tasks . We consider two typical scenarios : 1 ) we use CLIP in direct task-specific fine-tuning ( Section 3 ) ; 2 ) we integrate CLIP with V & L pre-training on image-text pairs 1Without confusion , we use the term CLIP to interchangeably refer to both the whole CLIP model ( including the text and visual encoder ) and just its visual encoder . and transfer to downstream tasks ( Section 4 ) .2 For clarity , we denote the models used in these two scenarios as CLIP-ViL ( without V & L pre-training ) and CLIP-ViLp ( with V & L pre-training ) . In direct task-specific fine-tuning , we consider three widely-adopted tasks : Visual Question Answering ( Antol et al. , 2015 ) , Image Captioning ( Chen et al. , 2015 ) , and Vision-and-Language Navigation ( Anderson et al. , 2018b ) . On all three tasks , CLIP-ViL brings sizable improvement over strong baselines , 1.4 % accuracy on VQA v2.0 , 6.5 CIDEr on COCO Captioning , and 4.0 % success rate on Room-to-Room navigation . In V & L pre-training , we replace the conventionally used region-based representation ( Anderson et al. , 2018a ) with CLIP . CLIP-ViLp performs exceptionally well on three benchmarks , including VQA v2.0 , SNLI-VE ( Xie et al. , 2019 ) , and GQA ( Hudson and Manning , 2019 ) , setting a new state-of-the-art ( SotA ) on VQA ( 76.70 % on test-std ) , and SNLI-VE ( 80.20 % on test ) . CLIP-ViLp with CLIP-Res50 outperforms models based on the widely used region-based encoder , BottomUp-TopDown ( BUTD ) ResNet101 ( Anderson et al. , 2018a ) . Moreover , CLIP-ViLp with CLIP-Res50x4 surpasses VinVLResNeXt152 ( Zhang et al. , 2021 ) , which is an extreme scale-up attempt of the region-based encoder with human-annotated data . 2 BACKGROUND AND MOTIVATION . Vision-and-Language ( V & L ) models . V & L tasks require a model to understand the visual world and to ground natural language to the visual observations . Prominent tasks include visual question answering ( Antol et al. , 2015 ) , image captioning ( Chen et al. , 2015 ) , vision-language navigation ( Anderson et al. , 2018a ) , image-text retrieval ( Wang et al. , 2016 ) and so on . V & L models designed for these tasks often consist of a visual encoder , a text encoder , and a cross-modal interaction module ( Kim et al. , 2021 ) . We illustrate the three typical training stages in Figure 1 : 1 ) the visual encoder is trained on annotated vision datasets ( Russakovsky et al. , 2015 ; Krishna et al. , 2017 ) ( denoted as visual encoder pretraining ) ; 2 ) ( optionally ) pre-training on paired image-caption data with a reconstructive objective and an image-text matching objective ( denoted as vision-and-language pre-training ) ( Lu et al. , 2019 ) ; 3 ) fine-tuning on task-specific data ( denoted as task-specific fine-tuning ) . Visual encoders in V & L models . Different models employ different visual encoders , we illustrate their architectures and pre-training processes in Figure 2 . The encoders can be categorized as follows : 1 ) region-based models such as BUTD object detector ( Anderson et al. , 2018a ; Kamath et al. , 2021 ) ; 2We distinguish between V & L pre-training and CLIP pre-training : V & L pre-training models ( Lu et al. , 2019 ) have deep interactions between modalities while CLIP follows a shallow-interaction design ( Section 2 ) . 2 ) grid-based models such as Jiang et al . ( 2020 ) that directly extract grid-like feature maps from the visual backbone ( He et al. , 2016 ; Dosovitskiy et al. , 2020 ) . The encoder is first pre-trained on human-annotated vision datasets . Region-based encoders are pre-trained with detection data such as Visual Genome ( Krishna et al. , 2017 ) . Grid-based encoders are pre-trained with image classification data such as ImageNet ( Russakovsky et al. , 2015 ) or detection data ( Jiang et al. , 2020 ) . However , these manually labeled datasets are expensive to construct and hard to scale up . They only provide supervision for a limited number of predetermined visual concepts . This motivates us to use CLIP as the visual encoder . CLIP . CLIP ( Contrastive Language-Image Pre-training ) ( Radford et al. , 2021 ) 3 falls into the line of research that learns visual representations from natural language supervision ( Desai and Johnson , 2020 ; Sariyildiz et al. , 2020 ; Jia et al. , 2021 ) . CLIP follows a “ shallow-interaction design ” , where a visual encoder and a text encoder encode an input image and text independently , and the dot-product between the two encoder ’ s output is used as the similarity score between the input image and text . It is pre-trained with a contrastive loss where the model needs to distinguish aligned pairs from randomly sampled pairs . CLIP leverages an abundantly available source of supervision without human annotation : 400M image-text pairs found across the Internet . As a result , CLIP achieves SotA performance in a range of image classification and image-text retrieval tasks in a zero-shot setting . 2.1 MOTIVATION . Despite the strong zero-shot capability of CLIP on vision tasks , CLIP does not exhibit the same level of performance on certain V & L downstream tasks . For instance , if we cast VQA 2.0 ( Goyal et al. , 2017 ) into a zero-shot image-to-text retrieval task , we only observe chance performance ( Section 5 ) . Thus , we propose to integrate CLIP ’ s visual encoder with previous V & L models ( Figure 1 ) . We consider the following CLIP variants with different visual backbones ( He et al. , 2016 ; Dosovitskiy et al. , 2020 ) ( CLIP-ResNet denoted as CLIP-Res ) : CLIP-Res50 , CLIP-Res101 , CLIP-Res50x4 , CLIP-ViT-B/16 and CLIP-ViT-B/32 . We next describe our methods in two scenarios : 1 ) direct task-specific fine-tuning ( Section 3 ) and 2 ) V & L pre-training ( Section 4 ) . 3 CLIP-VIL . In this section , we directly use CLIP as the visual encoder in task-specific models ( referred as CLIPViL ) and fine-tune on three representative tasks including Visual Question Answering ( Section 3.1 ) , Image Captioning ( Section 3.2 ) , and Vision-Language Navigation ( Section 3.3 ) . 3https : //github.com/openai/CLIP 3.1 VISUAL QUESTION ANSWERING . The task of Visual Question Answering ( VQA ) ( Antol et al. , 2015 ) is to provide the answer given an image and a related question . Various methods have been introduced ( Fukui et al. , 2016 ; Yang et al. , 2016 ; Anderson et al. , 2018a ; Jiang et al. , 2018 ; Gao et al. , 2019 ; Jiang et al. , 2020 ) . Here , we select two representative approaches ( i.e. , Pythia ( Jiang et al. , 2018 ) and MCAN ( Yu et al. , 2019 ) ) to study the impact of the CLIP visual encoders in VQA . VQA Model Visual Encoder ResultTest-dev Test-std Pythia ImageNet-Res50† 63.21 - BiTM-Res50 63.48 63.84 BiTM-Res101 63.82 64.11 VG-ResNeXt-101† 67.76 - BUTD-ResNeXt-101† 68.21 - CLIP-ViT-B/32 51.38 51.47 CLIP-ViT-B/16 54.31 54.42 CLIP-Res50 65.55 65.78 CLIP-Res101 66.76 67.14 CLIP-Res50x4 68.37 68.68 MCAN ImageNet-ResNet50 67.23 67.46 BUTD-ResNeXt-101† 72.01 - VG-ResNeXt-101† 72.59 - CLIP-ViT-B/32 56.18 56.28 CLIP-Res50 71.49 71.72 CLIP-Res101 72.77 73.19 CLIP-Res50x4 74.01 74.17 Detection data are hard to scale up and contain limited object categories , while our results suggest training visual encoders on noisy image-text data as in CLIP is promising and scalable . 3.2 IMAGE CAPTIONING . Image captioning aims at generating a natural language description of an image . Various methods have been delivered for image captioning ( Karpathy and Fei-Fei , 2015 ; Rennie et al. , 2017 ; Anderson et al. , 2018a ; Luo et al. , 2018 ; Luo , 2020 ) . We investigate the effectiveness of the CLIP model for this popular task with ( Luo , 2020 ) method . Experimental Setup . For the model architecture , we experiment with the basic Transformer model adapted from Vaswani et al . ( 2017 ) in Luo ( 2020 ) . Grid feature maps are extracted for each image . We evaluate our model on COCO dataset ( Chen et al. , 2015 ) . We use the standard automatic evaluation metrics including CIDEr ( Anderson et al. , 2016 ) , BLEU ( Papineni et al. , 2002 ) , METEOR ( Lavie and Agarwal , 2007 ) , and SPICE ( Anderson et al. , 2016 ) . The scores are obtained on Karparthy test split ( Karpathy and Fei-Fei , 2015 ) with beam search of 5 beams . Details are given in Appendix . Experimental Results . We report Image Captioning results with different models in Table 2 . Using the Transformer architecture from ( Luo , 2020 ) , we see that CLIP-Res models outperform ImageNet pre-trained alternatives for both ResNet50 ( +9.1 / +1.5 in CIDEr / SPICE ) and ResNet101 ( +9.2 / +1.5 in CIDEr / SPICE ) . It even surpasses the strong in-domain region-based feature from BUTD 4https : //github.com/facebookresearch/grid-feats-vqa and grid-based feature from BiT . As the model size grows in CLIP-ViL , the results also improve and the largest CLIP-Res50x4 achieves the best performance , although there still remains a gap to the pre-trained models that have interactive image-text pre-training phase like Oscarbase and VinVLbase . Again , CLIP-ViT variant leads to dramatically worse performance compared to other visual modules , that we will discuss in Section 5 .

Summary: This paper performs empirical analyses of applying the CLIP to various vision-and-language tasks. It demonstrates the potentials of the model in generalizing to different downstream applications, and provides some suggestions for model deployment. Experimental results show that CLIP pretraining leads to competitive performance and further combining it with V&L pretraining can outperform existing methods.