How Much Can CLIP Benefit Vision-and-Language Tasks?

Most existing Vision-and-Language ( V & L ) models rely on pre-trained visual encoders , using a relatively small set of manually-annotated data ( as compared to web-crawled data ) , to perceive the visual world . However , it has been observed that large-scale pre-training usually can result in better generalization performance , e.g. , CLIP ( Contrastive Language-Image Pre-training ) , trained on a massive amount of image-caption pairs , has shown a strong zero-shot capability on various vision tasks . To further study the advantage brought by CLIP , we propose to use CLIP as the visual encoder in various V & L models in two typical scenarios : 1 ) plugging CLIP into task-specific fine-tuning ; 2 ) combining CLIP with V & L pre-training and transferring to downstream tasks . We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data , such as BottomUp-TopDown . We achieve competitive or better results on diverse V & L tasks , while establishing new state-of-the-art results on Visual Question Answering , Visual Entailment , and V & L Navigation tasks . 1 INTRODUCTION . Vision-and-Language ( V & L ) tasks such as VQA ( Antol et al. , 2015 ) test a system ’ s ability to understand and reason about the semantics of the visual world with the help of natural language . Most V & L models rely on a visual encoder to perceive the visual world , which translates the raw pixels into vectors from a representation space . Recent work ( Anderson et al. , 2018a ; Jiang et al. , 2020 ; Zhang et al. , 2021 ) observes that the visual representation has become the performance bottleneck of V & L models and stress the importance of learning a powerful visual encoder . These high-performing visual encoders are trained on manually-annotated data with class labels ( e.g. , ImageNet ) ( Russakovsky et al. , 2015 ) or bounding boxes ( e.g. , Visual Genome ) ( Krishna et al. , 2017 ) . However , such detection or image classification data is costly to collect , and the visual representation is limited by the predefined class labels . Thus , there is a need for a visual encoder that is trained on more diverse and large-scale data sources , unbounded by a fixed set of labels , and with generalization ability to unseen objects and concepts . Recently , CLIP ( Radford et al. , 2021 ) has been proposed to learn visual concepts with language supervision . CLIP consists of a visual encoder and a text encoder . It is trained on 400M noisy imagetext pairs crawled from the Internet . Thus , the data collection process is scalable and requires little human annotation . CLIP has shown strong zero-shot capabilities on benchmarks such as ImageNet classification . We hypothesize that it also bears great potential for the V & L tasks . However , directly applying CLIP as a zero-shot model to V & L tasks proves to be difficult ( Section 5 and Kim et al . ( 2021 ) ) , as many V & L tasks require complex multi-modal reasoning . Thus , we propose to integrate CLIP with existing V & L models by replacing their visual encoder with CLIP ’ s visual encoder.1 To the best of our knowledge , we present the first large-scale empirical study on using CLIP as the visual encoder for diverse V & L tasks . We consider two typical scenarios : 1 ) we use CLIP in direct task-specific fine-tuning ( Section 3 ) ; 2 ) we integrate CLIP with V & L pre-training on image-text pairs 1Without confusion , we use the term CLIP to interchangeably refer to both the whole CLIP model ( including the text and visual encoder ) and just its visual encoder . and transfer to downstream tasks ( Section 4 ) .2 For clarity , we denote the models used in these two scenarios as CLIP-ViL ( without V & L pre-training ) and CLIP-ViLp ( with V & L pre-training ) . In direct task-specific fine-tuning , we consider three widely-adopted tasks : Visual Question Answering ( Antol et al. , 2015 ) , Image Captioning ( Chen et al. , 2015 ) , and Vision-and-Language Navigation ( Anderson et al. , 2018b ) . On all three tasks , CLIP-ViL brings sizable improvement over strong baselines , 1.4 % accuracy on VQA v2.0 , 6.5 CIDEr on COCO Captioning , and 4.0 % success rate on Room-to-Room navigation . In V & L pre-training , we replace the conventionally used region-based representation ( Anderson et al. , 2018a ) with CLIP . CLIP-ViLp performs exceptionally well on three benchmarks , including VQA v2.0 , SNLI-VE ( Xie et al. , 2019 ) , and GQA ( Hudson and Manning , 2019 ) , setting a new state-of-the-art ( SotA ) on VQA ( 76.70 % on test-std ) , and SNLI-VE ( 80.20 % on test ) . CLIP-ViLp with CLIP-Res50 outperforms models based on the widely used region-based encoder , BottomUp-TopDown ( BUTD ) ResNet101 ( Anderson et al. , 2018a ) . Moreover , CLIP-ViLp with CLIP-Res50x4 surpasses VinVLResNeXt152 ( Zhang et al. , 2021 ) , which is an extreme scale-up attempt of the region-based encoder with human-annotated data . 2 BACKGROUND AND MOTIVATION . Vision-and-Language ( V & L ) models . V & L tasks require a model to understand the visual world and to ground natural language to the visual observations . Prominent tasks include visual question answering ( Antol et al. , 2015 ) , image captioning ( Chen et al. , 2015 ) , vision-language navigation ( Anderson et al. , 2018a ) , image-text retrieval ( Wang et al. , 2016 ) and so on . V & L models designed for these tasks often consist of a visual encoder , a text encoder , and a cross-modal interaction module ( Kim et al. , 2021 ) . We illustrate the three typical training stages in Figure 1 : 1 ) the visual encoder is trained on annotated vision datasets ( Russakovsky et al. , 2015 ; Krishna et al. , 2017 ) ( denoted as visual encoder pretraining ) ; 2 ) ( optionally ) pre-training on paired image-caption data with a reconstructive objective and an image-text matching objective ( denoted as vision-and-language pre-training ) ( Lu et al. , 2019 ) ; 3 ) fine-tuning on task-specific data ( denoted as task-specific fine-tuning ) . Visual encoders in V & L models . Different models employ different visual encoders , we illustrate their architectures and pre-training processes in Figure 2 . The encoders can be categorized as follows : 1 ) region-based models such as BUTD object detector ( Anderson et al. , 2018a ; Kamath et al. , 2021 ) ; 2We distinguish between V & L pre-training and CLIP pre-training : V & L pre-training models ( Lu et al. , 2019 ) have deep interactions between modalities while CLIP follows a shallow-interaction design ( Section 2 ) . 2 ) grid-based models such as Jiang et al . ( 2020 ) that directly extract grid-like feature maps from the visual backbone ( He et al. , 2016 ; Dosovitskiy et al. , 2020 ) . The encoder is first pre-trained on human-annotated vision datasets . Region-based encoders are pre-trained with detection data such as Visual Genome ( Krishna et al. , 2017 ) . Grid-based encoders are pre-trained with image classification data such as ImageNet ( Russakovsky et al. , 2015 ) or detection data ( Jiang et al. , 2020 ) . However , these manually labeled datasets are expensive to construct and hard to scale up . They only provide supervision for a limited number of predetermined visual concepts . This motivates us to use CLIP as the visual encoder . CLIP . CLIP ( Contrastive Language-Image Pre-training ) ( Radford et al. , 2021 ) 3 falls into the line of research that learns visual representations from natural language supervision ( Desai and Johnson , 2020 ; Sariyildiz et al. , 2020 ; Jia et al. , 2021 ) . CLIP follows a “ shallow-interaction design ” , where a visual encoder and a text encoder encode an input image and text independently , and the dot-product between the two encoder ’ s output is used as the similarity score between the input image and text . It is pre-trained with a contrastive loss where the model needs to distinguish aligned pairs from randomly sampled pairs . CLIP leverages an abundantly available source of supervision without human annotation : 400M image-text pairs found across the Internet . As a result , CLIP achieves SotA performance in a range of image classification and image-text retrieval tasks in a zero-shot setting . 2.1 MOTIVATION . Despite the strong zero-shot capability of CLIP on vision tasks , CLIP does not exhibit the same level of performance on certain V & L downstream tasks . For instance , if we cast VQA 2.0 ( Goyal et al. , 2017 ) into a zero-shot image-to-text retrieval task , we only observe chance performance ( Section 5 ) . Thus , we propose to integrate CLIP ’ s visual encoder with previous V & L models ( Figure 1 ) . We consider the following CLIP variants with different visual backbones ( He et al. , 2016 ; Dosovitskiy et al. , 2020 ) ( CLIP-ResNet denoted as CLIP-Res ) : CLIP-Res50 , CLIP-Res101 , CLIP-Res50x4 , CLIP-ViT-B/16 and CLIP-ViT-B/32 . We next describe our methods in two scenarios : 1 ) direct task-specific fine-tuning ( Section 3 ) and 2 ) V & L pre-training ( Section 4 ) . 3 CLIP-VIL . In this section , we directly use CLIP as the visual encoder in task-specific models ( referred as CLIPViL ) and fine-tune on three representative tasks including Visual Question Answering ( Section 3.1 ) , Image Captioning ( Section 3.2 ) , and Vision-Language Navigation ( Section 3.3 ) . 3https : //github.com/openai/CLIP 3.1 VISUAL QUESTION ANSWERING . The task of Visual Question Answering ( VQA ) ( Antol et al. , 2015 ) is to provide the answer given an image and a related question . Various methods have been introduced ( Fukui et al. , 2016 ; Yang et al. , 2016 ; Anderson et al. , 2018a ; Jiang et al. , 2018 ; Gao et al. , 2019 ; Jiang et al. , 2020 ) . Here , we select two representative approaches ( i.e. , Pythia ( Jiang et al. , 2018 ) and MCAN ( Yu et al. , 2019 ) ) to study the impact of the CLIP visual encoders in VQA . VQA Model Visual Encoder ResultTest-dev Test-std Pythia ImageNet-Res50† 63.21 - BiTM-Res50 63.48 63.84 BiTM-Res101 63.82 64.11 VG-ResNeXt-101† 67.76 - BUTD-ResNeXt-101† 68.21 - CLIP-ViT-B/32 51.38 51.47 CLIP-ViT-B/16 54.31 54.42 CLIP-Res50 65.55 65.78 CLIP-Res101 66.76 67.14 CLIP-Res50x4 68.37 68.68 MCAN ImageNet-ResNet50 67.23 67.46 BUTD-ResNeXt-101† 72.01 - VG-ResNeXt-101† 72.59 - CLIP-ViT-B/32 56.18 56.28 CLIP-Res50 71.49 71.72 CLIP-Res101 72.77 73.19 CLIP-Res50x4 74.01 74.17 Detection data are hard to scale up and contain limited object categories , while our results suggest training visual encoders on noisy image-text data as in CLIP is promising and scalable . 3.2 IMAGE CAPTIONING . Image captioning aims at generating a natural language description of an image . Various methods have been delivered for image captioning ( Karpathy and Fei-Fei , 2015 ; Rennie et al. , 2017 ; Anderson et al. , 2018a ; Luo et al. , 2018 ; Luo , 2020 ) . We investigate the effectiveness of the CLIP model for this popular task with ( Luo , 2020 ) method . Experimental Setup . For the model architecture , we experiment with the basic Transformer model adapted from Vaswani et al . ( 2017 ) in Luo ( 2020 ) . Grid feature maps are extracted for each image . We evaluate our model on COCO dataset ( Chen et al. , 2015 ) . We use the standard automatic evaluation metrics including CIDEr ( Anderson et al. , 2016 ) , BLEU ( Papineni et al. , 2002 ) , METEOR ( Lavie and Agarwal , 2007 ) , and SPICE ( Anderson et al. , 2016 ) . The scores are obtained on Karparthy test split ( Karpathy and Fei-Fei , 2015 ) with beam search of 5 beams . Details are given in Appendix . Experimental Results . We report Image Captioning results with different models in Table 2 . Using the Transformer architecture from ( Luo , 2020 ) , we see that CLIP-Res models outperform ImageNet pre-trained alternatives for both ResNet50 ( +9.1 / +1.5 in CIDEr / SPICE ) and ResNet101 ( +9.2 / +1.5 in CIDEr / SPICE ) . It even surpasses the strong in-domain region-based feature from BUTD 4https : //github.com/facebookresearch/grid-feats-vqa and grid-based feature from BiT . As the model size grows in CLIP-ViL , the results also improve and the largest CLIP-Res50x4 achieves the best performance , although there still remains a gap to the pre-trained models that have interactive image-text pre-training phase like Oscarbase and VinVLbase . Again , CLIP-ViT variant leads to dramatically worse performance compared to other visual modules , that we will discuss in Section 5 .

Summary: This paper examines how well CLIP's visual encoders are transferred on vision-and-language (VL) tasks.   The paper conducted three without-VLP tasks: VQA, image captioning, and vision-and-language navigation, and three with-VLP tasks: VQA, SNLI-VE, and GQA to show CLIP's visual encoders' transferability,    CLIP's resnet-based visual encoders consistently outperformed their Imagenet pre-trained counterpart.   The authors found that CLIP's ViT-based visual encoder performed far worse than the resnets.   From Grad-CAM and detection fine-tuning experiments, the authors speculated that ViT's features lack localization information.