Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image  Pre-training Paradigm

Recently , large-scale Contrastive Language-Image Pre-training ( CLIP ) ( Radford et al. , 2021 ) has attracted unprecedented attention for its impressive zero-shot recognition ability and excellent transferability to downstream tasks . However , CLIP is quite data-hungry and requires 400M image-text pairs for pre-training , thereby restricting its adoption . This work proposes a novel training paradigm , Data efficient CLIP ( DeCLIP ) , to alleviate this limitation . We demonstrate that by carefully utilizing the widespread supervision among the image-text pairs , our DeCLIP can learn generic visual features more efficiently . Instead of using the single image-text contrastive supervision , we fully exploit data potential through the use of ( 1 ) self-supervision within each modality ; ( 2 ) multi-view supervision across modalities ; ( 3 ) nearest-neighbor supervision from other similar pairs . Benefiting from these intrinsic supervision , our DeCLIP-ResNet50 can achieve 60.4 % zero-shot top1 accuracy on ImageNet , which is 0.8 % above the CLIP-ResNet50 while using 7.1× fewer data . Our DeCLIP-ResNet50 outperforms its counterpart in 8 out of 11 visual datasets when transferred to downstream tasks . Moreover , Scaling up the model and computing also works well in our framework . 1 INTRODUCTION . Over the last few years , pre-trained models have greatly revolutionized computer vision ( CV ) and natural language processing ( NLP ) . The first wave of exploring pre-trained models took place in the field of CV . Deep convolutional neural nets ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; He et al. , 2016 ) are pre-trained on well-labeled ImageNet ( Deng et al. , 2009 ) and then transferred to downstream CV tasks ( Girshick et al. , 2014 ; Long et al. , 2015 ; Vinyals et al. , 2015 ) . Standardly , CV models are pre-trained to predict a fixed set of pre-defined object categories , e.g. , 1000 classes in ImageNet . However , this supervised pre-training is hard to scale since we need arduous human labeling to specify new visual concepts . When pre-training meets NLP , the intrinsic supervision within the natural language makes the pretraining more scalable ( Devlin et al. , 2018 ; Radford et al. , 2019 ; Brown et al. , 2020 ) . Witnessing the progress in NLP , researchers use natural language supervision to learn visual features . The language-image pre-training can scale up to a very large size , benefiting from abundant image-text pairs on the Internet . For instance , CLIP ( Radford et al. , 2021 ) and ALIGN ( Jia et al. , 2021 ) adopt the contrastive loss to push the embedding of matched image-text pairs together while pushing those of non-matched pairs apart . They achieve prestigious performance by learning from an enormous dataset that contains 400M/1B image-text pairs . However , these methods also require huge storage and computing resources , which is not affordable for most laboratories and companies . We argue that these prior arts only use the single image-text contrastive supervision while overlooking the widespread supervision within the pairs , thus is inefficient . Firstly , there underlies rich structural information within each modality itself ( LeCun & Misra , 2021 ) . We can tweak some words/pixels in a sentence/image while retaining a similar semantic meaning . This sort of self-supervision can be exploited to learn a more common-sense representation for each modality ( Devlin et al. , 2018 ; He et al. , 2020 ; Chen et al. , 2020a ) . Moreover , inspired C DeC 71 72 73 74 75 73.3 73.8 ImageNet1k C DeC 68 69 70 71 72 70.3 71.2 CIFAR100 C DeC 94 96 98 100 96.1 99.2 FLOWERS C DeC 86 87 88 89 90 88.2 88.7 PETS C DeC 87 88 89 90 91 88.7 89.8 CIFAR10 C DeC 70 71 72 73 74 73.3 72.8 SUN C DeC 82 84 86 86.4 82.7 FOOD101 C DeC 78 80 82 78.3 81.7 CARS C DeC 90 92 94 89.6 93.9 CALTECH C DeC 46 47 48 49 50 49.1 48.4 AIRCRAFT C DeC 74 75 76 77 78 76.4 76.8 DTD C DeC 77 78 79 80 81 79.1 79.9 Average Figure 2 : Transfer the DeCLIP-ResNet50 ( abbr . as DeC ) and CLIP-ResNet50 ( abbr . as C ) to 11 downstream visual datasets using linear probe verification . Our DeCLIP achieves better results in 8 out of 11 datasets . by contrasting multi-crops in an image ( Caron et al. , 2020 ) , we further extend the multi-view 1 supervision into our multi-modality setting . Specifically , each image is paired with multiple textual descriptions obtained via stochastic augmentations , vice versa . The benefit is intuitive : this auxiliary multi-view supervision brings more invariant and robust information . Besides these overlooked supervision , we propose a novel nearest-neighbor ( NN ) supervision from other similar pairs . This NN supervision is mainly based on the intuition that one image is likely to have other similar text descriptions among the dataset . As shown in right figure , the image with the text ’ going to see a lot of vintage tractors this week ’ can also be described by ’ vintage at tractors a gathering ’ . For this reason , we sample the NN in the embedding space and utilize them as additional supervisory signals . Aggregating these supervision leads to our novel training paradigm DeCLIP , which stands for Data efficient Contrastive Language-Image Pretraining . Extensive experiments show the effectiveness and efficiency of our DeCLIP . As shown in Fig . 1 , with a ResNet50 image encoder and a Transformer text encoder , our model can achieve 60.4 % zero-shot top1 accuracy on ImageNet , which is 0.8 % above the CLIPResNet50 while using 7.1× fewer data . Using only 88M image-text pairs , our best ResNet50/ViTB32 models boost the zero-shot performance to 62.5 % and 66.2 % , nearly 3.0 % higher than the best number reported for these two architectures . We further verify the transferability of our models on downstream tasks . As indicated in Fig . 2 , our DeCLIP-ResNet50 outperforms its counterpart in 8 out of 11 visual datasets . Moreover , Scaling up the model and computing also works well in our framework . Using 4.5× fewer data , our DeCLIP-RegNetY-64GF achieves 73.7 % zero-shot ImageNet top1 accuracy , which is on-pair with CLIP-R50×64 . Pre-trained models , code , and datasets shall be released to the community . The contributions are summarized as follows : • To the best of our knowledge , this is the first work to study self-supervision and cross-modal multi-view supervision in the million-scale image-text pre-training task . Our work opens a new direction to fully exploit the intrinsic supervision within the multi-modal data instead of scaling up data naively . • We propose novel cross-modal Nearest-Neighbor Supervision ( NNS ) to harness information from other similar pairs . The NNS can also be regarded as a semantic-level augmentation . 1View is originally a visual concept . For simplicity , we use the same term for language . 2 RELATED WORK . 2.1 PRE-TRAINED MODELS . The critical idea of pre-training is to first extract general knowledge implicitly from the massive amount of data and then transfer the knowledge to versatile downstream tasks ( Han et al. , 2021 ) . With the transformer architecture ( Vaswani et al. , 2017 ) , big pre-trained NLP models , such as BERTseries ( Devlin et al. , 2018 ; Liu et al. , 2019 ) and GPT-series ( Radford et al. , 2019 ; Brown et al. , 2020 ) , have dominated this area ( Han et al. , 2021 ) . Besides the exquisite architecture , the great success comes from the tremendous language data over the Internet and labor-free supervision ( Yang et al. , 2019 ; Radford et al. , 2019 ; Devlin et al. , 2018 ) within the language itself . In the field of CV , supervised pre-training on ImageNet is still the standard practice . While achieving great success on downstream CV tasks ( Girshick et al. , 2014 ; Long et al. , 2015 ; Vinyals et al. , 2015 ) , this supervised manner is hard to scale . To address this challenge , our DeCLIP learns directly from image-text pairs that are abundant across the Internet . More importantly , by exploiting the widespread supervision within the pairs , our DeCLIP is more data-efficient than the prior art . 2.2 SUPERVISION WITHIN DATA . Language supervision Joulin et al . ( 2016 ) ; Gomez et al . ( 2017 ) ; Zhang et al . ( 2020 ) ; Sariyildiz et al . ( 2020 ) ; Desai & Johnson ( 2021 ) demonstrate the effectiveness of learning transferable visual features from language supervision . However , these works are limited to a small dataset such as Flickr ( Young et al. , 2014 ) or COCO ( Lin et al. , 2014 ) . Pioneering work CLIP ( Radford et al. , 2021 ) and ALIGN ( Jia et al. , 2021 ) achieve prestigious performance via learning from 400M/1B image-text pairs . We are following these two works to improve their data efficiency . Visual self-supervision Our work is also highly related to self-supervised learning ( SSL ) ( LeCun & Misra , 2021 ) . Contrastive learning , as a pretext task of SSL , has achieved remarkable success in visual representation learning ( He et al. , 2020 ; Chen et al. , 2020a ; Caron et al. , 2020 ; Grill et al. , 2020 ; Chen et al. , 2020a ) . Researchers also extend contrastive learning into multi-modal settings ( Yuan et al. , 2021 ) . However , it is only limited to a small COCO dataset ( Yuan et al. , 2021 ) . Nearest-neighbor supervision Recently , researchers have exploited nearest-neighbor supervision to learn visual features ( Dwibedi et al. , 2021 ; Van Gansbeke et al. , 2021 ) . They find that using nearest-neighbor as positive samples in the contrastive loss improves the performances on multiple downstream tasks . However , they mainly focus on the single visual modality pretraining on relatively small datasets , such as ImageNet . We propose novel nearest-neighbor supervision for multi-modal learning to harness information from other similar pairs . 2.3 MULTI-MODAL LEARNING . Multi-modal learning aims to build models that can process and relate information from multiple modalities ( Baltrušaitis et al. , 2018 ) . Vision-language tasks , such as VQA ( Antol et al. , 2015 ; Gao et al. , 2019 ) , image captioning ( Xu et al. , 2015 ) , require an understanding of both textual and visual semantics . Most vision-language models use a bunch of cross-modal transformers to fuse and align the information between text and image , such as LXMERT ( Tan & Bansal , 2019 ) , UNITER ( Chen et al. , 2020b ) , ViLBERT ( Lu et al. , 2019 ) , VisualBERT ( Li et al. , 2019 ) , OSCAR ( Li et al. , 2020 ) , Pixel-BERT ( Huang et al. , 2020 ) . These methods either need an off-the-shelf object detector to extract region features or dedicated cross-modal transformer layers , significantly hindering their scalability . Our DeCLIP , by contrast , uses a simple yet effective two-tower framework with multimodal interaction only at the top . Moreover , this series of models ( Radford et al. , 2021 ; Jia et al. , 2021 ; Huo et al. , 2021 ) can perform zero-shot recognition , adapting to new categories with no seen labeled data . Shen et al . ( 2021 ) also shows that the pre-trained CLIP model can significantly benefit the downstream VQA and image caption tasks . Our DeCLIP is supposed to be compatible with more modalities , e.g. , acoustic signals ( Akbari et al. , 2021 ) . More modalities included , more correlated supervision are expected to be exploited . We hope our DeCLIP can inspire and advance the future study of multi-modal learning . correct pairings of a batch of ( image , text ) training examples . ( b ) Our DeCLIP overview . 1 means Self-Supervision ( SS ) . For image SS , we maximize the similarity between two augmented views of the same instance . For text SS , we leverage Masked Language Modeling ( MLM ) within a text sentence . 2 represents cross-modal Multi-View Supervision ( MVS ) . We first have two augmented views of both image and text , then contrast the 2 × 2 image-text pairs . 3 indicates NearestNeighbor Supervision ( NNS ) . We sample text NN in the embedding space to serve as additional supervision . The combination of the three supervision leads to efficient multi-modal learning .

Summary: The paper proposes DeCLIP to further utilize the data potential by adding three training objectives to CLIP pre-training: 1) inspired by SimSiam and BERT, self-supervised objectives are added to both image and text; 2) they generate different views for both images and text, and apply contrastive objectives; 3) they sample neighbor text as additional positive examples.  DeCLIP improves data efficiency. With web-crawled data, DeCLIP outperforms CLIP counterparts with 4.5x smaller amount of data. In addition, while introducing addition objectives and especially different views increases per-batch compute time by 1.5x, the authors show that DeCLIP still outperforms CLIP when given the same compute time budget.