Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image  Pre-training Paradigm
Authors: Yangguang Li
Feng Liang
Lichen Zhao
Yufeng Cui
Wanli Ouyang
Jing Shao
Fengwei Yu
Junjie Yan
Venue: ICLR 2022 Poster (2022)

Abstract: Recently, large-scale Contrastive Language-Image Pre-training (CLIP) has attracted unprecedented attention for its impressive zero-shot recognition ability and excellent transferability to downstream tasks. However, CLIP is quite data-hungry and requires 400M image-text pairs for pre-training, thereby restricting its adoption. This work proposes a novel training paradigm, Data efficient CLIP (DeCLIP), to alleviate this limitation. We demonstrate that by carefully utilizing the widespread supervision among the image-text pairs, our De-CLIP can learn generic visual features more efficiently. Instead of using the single image-text contrastive supervision, we fully exploit data potential through the use of (1) self-supervision within each modality; (2) multi-view supervision across modalities; (3) nearest-neighbor supervision from other similar pairs. Benefiting from intrinsic supervision, our DeCLIP-ResNet50 can achieve 60.4% zero-shot top1 accuracy on ImageNet, which is 0.8% above the CLIP-ResNet50 while using 7.1×fewer data. Our DeCLIP-ResNet50 outperforms its counterpart in 8 out of 11 visual datasets when transferred to downstream tasks. Moreover, Scaling up the model and computing also works well in our framework.

Published as a conference paper at ICLR 2022
SUPERVISION EXISTS EVERYWHERE: A DATA EFFICIENT CONTRASTIVE LANGUAGE-IMAGE PRE-TRAINING PARADIGM
Yangguang Li ", Feng Liang" Lichen Zhao Yufeng Cui', Wanli Ouyang?, Jing Shao', Fengwei Yu', Junjie Yan'
"SenseTime Research
"The University of Texas at Austin
UUniversity of Sydney
(liyangguang, zhaolichen, cuiyufenng/eeenserimm... jeffl i welliinnguteee.se... edu
ABSTRACT
Recently, large-scale Contrastive Language-Image Pre-training (CLIP) (Radford et al., 202T) has attracted unprecedented attention for its impressive zero-shot recognition ability and excellent transferability to downstream tasks. However, CLIP is quite data-hungry and requires 400M image-text pairs for pre-training. thereby restricting its adoption. This work proposes a novel training paradigm, Data efficient CL.IP (DeCLIP), to alleviate this limitation. We demonstrate that by carefully utilizing the widespread supervision among the image-text pairs, our De- CLIP can learn generic visual features more efficiently. Instead of using the single image-tex, contrastive supervision, fully exploit data potential through the use (1) self-supervision within each modality; (2) multi-view supervision across modalities; (3) nearest-neighhor supervision from other similar pairs. Benefiting from these intrinsic supervision, our DeCL.PP-ReNNtt. can achieve 60.4% zero- shot top1 accuracy ImageNet, which is 0.8% above the CLIP-ResNet50 while using 7.1x x fewer data. Our -essi---eeNe50 outperforms its counterpart in 8 out of 11 visual datasets when transferred downstream tasks. Moreover, Scaling up the model and computing also works well in our framework. Our code, dataset and models are released at: https: com/ wwwom//ennn--///De.....
INTRODUCTION
Over the last few years, pre-trained models have greatly revolutionized computer vision (CV) and natural language processing (NLP). The first wave of exploring pre-trained models took place the field of CV. Deep convolutional neural nets (Krizhevsky et al. 2012 Simonyan & Zisserman 2014 on well-labele
He et al. 2016) are to downstream CV tasks (Girshick et al.) 2014 Long et al. 2015 Vinyals et al. 2015). Standardly, CV models are pre-trained to predict a fixed set of pre-defined object categories, c.g., 1000 classes in ImageNet. However, this supervised pre-training is hard scale since need arduous human labeling to specify new visual concepts.
ImageNet (Deng et al. 2009) and then transferred
When pre-training meets NLP, the intrinsic supervision within the natural language makes the pre- training more scalable (Devlin et al. 2018 Radford et al. 2019 Brown et al.) (020). Witnessing the progress in NLP, researchers language-image pre-training can scale up to a very large size, benefiting from abundant image-text pairs on the Internet. For instance, CLIP (Radford al. (2021) and ALIGN (Jia et al. (201)) adopt the contrastive loss to push the embedding of matched image-text pairs together while pushing those of non-matched pairs apart. They achieve prestigious performance by learning from an enormous dataset that contains 400M/1B image-text pairs. However, these methods also require huge storage and computing resources, which is not affordable for most laboratories and companies. We argue
natural language supervision to learn visual features. The
"The first three authors contribute equally. The order is determined by dice rolling.

---

Published as a conference paper at ICLR 2022
ImageNetik
CIFAR100
FLOWERS
PETS
(56M, 60.4%)
X(8BM, 62.5%)
(400M, 59.6%)
312
60
data
96
Zero-shot ImageNet Topl
CIFAR10
CALTECH
50-
[15M
S8M
400M
76.4
CLIP
[55.9 44.2" 54.51 56.9 59.6
DECLIP 49.3
60.4 62.5
45-
"Ou REMMUUMIENTTTIONN
Doc
Dec
DeC
Dec
CLIP CLiP(Our reimp.) DeCLiP(Ours)
FOOD101
AIRCRAFT
SUN
Average
40
06-
60
611
73
72.
60
64
40
72
79
70.1
87
71
70
1529
56
88
400
millions image-text pairs
0
DMC
0
DvC
DMC
Zero-shot performance of CLIP- Figure 2: Transfer the ResNet50 and our ing different amounts of data. (88M, 62.5%) de- downstream visual datasets using linear probe notes the use of 88M data with top-1 accuracy verification. Our DeCLIP achieves better results 62.5% on the ImageNet-1K validation dataset. out of 11 datasets. Our model has much better data efficiency.
Figure 1:
(abbr, when us- as DeC) and CLIP-ResNethh (abbr. as C) to 11
that these prior arts only the single image-text contrastive supervision while overlooking the widespread supervision within the pairs, thus inefficient.
Firstly, there underlies rich structural information within each modality itself (LeCun & Misra 202T]. We can tweak some words/pixels meaning. This sort of self-supervision can be exploited to learn more common-sense representa- tion for each modality (Devlin et al. 2018 He et al. 2020 Chen et al. (020a).. Morcover, inspired by contrasting multi-crops in image (Caron al. 2020),
sentence/image while retaining a similar semantic
further extend the multi-view su- pervision into our multi-modality setting. Specificallyy each image is paired with multiple textual descriptions obtained via stochastic augmentations, vice versa. The benefit intuitive: this auxiliary multi-view supervision brings more invariant and robust information.
Besides these overlooked supervision, propose novel nearest-neighbor (NN) supervision from other similar pairs. This NN supervision mainly based on the intuition that one image is likely to have other similar text de- scriptions among the dataset. As shown in right fig-
ure, the image with the text "going to see a lot of vintage tractors this week' can also be described
Origin
opted for loose side parted hairstyle when she attended event.
going vintage tractors this
by 'vintage at tractors a gathering'. For this reason, we sample the NN the embedding space and uti- lize them additional supervisory signals. Aggregating these supervision leads to our novel training paradigm DeCLIP, which stands for Data efficient Contrastive Language-Iman Pretraining.
actor wore her hair framing layers during the show
vintage tractors gathering
Figure 3: Examples of Nearest Neighbor from Conceptual Cap- tions dataset.
Extensive experiments show the effectiveness and efficiency of our DeCLIP. As shown in Fig. D with a ResNet50 image encoder and a Transformer text encoder, model can achieve 60.4% zero-shot topl accuracy on ImageNet, which is 0.8% above the CLIP- ResNet50 while using 7.1x fewer data. Using only 88M image-text pairs, our best ResNet50/ViTT B32 models boost the zero-shot performance 62.5% and 66.2%, nearly 3.0% higher than the best number reported for these two architectures. We further verify the transferability of our models on downstream tasks. As indicated Fig. [ our Deste00 outperforms its counterpart out of 11 visual datasets. Moreover, Scaling up the model and computing also works well in our framework. Using 4.5x fewer data, our Reg.P---eNerY-GGGF achieves 73.7% zero-shot Ima- geNet topl accuracy, which is on-pair with CLIP-R50x66 Pre-trained models, code, and datasets shall be released to the community. The contributions are summarized as follows:
View originally visual concept. For simplicity, use the same for language.
2

---

Published as a conference paper at ICLR 2022
To the best of our knowledge, this is the first work to study self- self-supervision and cross-modal multi-view supervision in the million-scale image-text pre-training task. Our work opens a new direction to fully exploit the intrinsic supervision within the multi-modal data instead of scaling up data naively.
We propose novel cross-modal Nearest- Nearest-Neighhor Supervision (NNS) to harness information from other similar pairs. The NNS also be regarded as semantic-level augmentation.
RELATED WORK
2.1 PRE-TRAINED MODELS
The critical idea of pre-training is to first extract general knowledge implicitly from the massive amount of data and then transfer the knowledge to versatile downstream tasks (Han et al. (021). Big NLP models (Devlin et al. 2018 Brown al. 2020) yield unprecedented performance via learning from tremendous language data over the Internet and labor-free supervision within the language itself. In the field of CV, supervised pre-training on ImageNet still the standard practice. While achieving great success on downstream CV tasks (Girshick et al.) 20144] Long et al., 2015 Vinyals al. 2015). this supervised manner is hard scale. To address this challenge, our DeCLIP learns directly from image-text pairs that are abundant across the Internet. More importantly, by exploiting the widespread supervision within the pairs, our DeCLIP more data-efficient than the prior art.
2.2 SUPERVISION WITHIN DATA
Language supervision Joulin et al. 2016); Gomez et al. (2017); Zhang (2020); Sariyildiz
et al. (2020); Desai & Johnson (2021 demonstrate the effectiveness of learning transferable visual features from language supervision. Pioneering work CLIP (Radford et al. 2021) and ALIGN (Ji al. 2021) achieve prestigious performance via learning from 400M/IB image-text pairs. We are following these two works to improve their data efficiency.
Relevant concurrent work including: SLIP Mu et al. (2021) introduces self-supervision CLIP. FILIP Yao et al. (202I) leverages the finer- finer-grained grained alignment between image patches and textual words. OTTER Wu et al. (2021) 1); Cheng et al. (2021) uses online entropic optimal transport to find a soft image-text match as labels to mitigate the noise within the dataset.
Visual self-supervision Our work is also highly related self superviee learning (SSL) (LeCun & Misra 2021). Contrastive learning, as a pretext task of SSL, has achieved remarkable success in vi-
sual representation learning (He et al. 2020 Chen al. 2020a Caron et al., 22200) Grill et al. 2020
Chen et al. 2020a). Researchers also extend contrastive learning into multi-modal settings Yuan
al. 2021 However, it is only limited to small COCO dataset (uuan et al.) 2021)
Nearest-neighbor supervision Recently, researchers have exploited nearest-neighbor supervision learn visual features (Dwibedi al. 2021: Van Gansbeke al., (021). They find that using nearest-neighbor as positive samples in the contrastive loss improves the performances on multi- ple downstream tasks. However, they mainly focus on the single visual modality pretraining on relatively small datasets, such as ImageNet. We propose novel nearest-neighbor supervision for multi-modal learning to harness information from other similar pairs.
2.3
MULTI-MODAL LEARNING
Most lisiun-language models Chen al. (2020b: Lu al. 2019); ; Li et (2000) use bunch of cross-modal transformers to fuse and align the information between text and image. These methods either need an off off-tf object detector to extract region features or dedicated cross-modal trans- former layers, significantly hindering their scalability. Our DeCLIP, by contrast, uses simple yet effective two-tower framework with multi-modal interaction only at the top. Moreover, this series of models (Radford et al. 2021 Jia et al. 2021 Huo et al. (021)) can perform zero-shot recognition, adapting to new categories with no seen labeled data. [Shen et al. (2021) also shows that the pre- trained CLIP model can significantly benefit the downstream VQA and image caption tasks. Our DeCLIP is supposed to be compatible with more modalities, e.g., acoustic signals (Akbari et al, 2021). More modalities included, more correlated supervision are expected be exploited.
3

---

Published as a conference paper at ICLR 2022
0
Original Sup.
Belf-Sup
Multi- View Sup.
3
Nearest-Neighhor Sup.
cute white
white
Text cnc.
enc.
Feature Queue
NN((T)
22
22
2
2
Img enc.
7
7
Imng-
enc.
2
3
(8) CLIP & ALIGN
(b) DeCLIP
Figure 4: (a) CLIP and ALIGN jointly train an image encoder and text encoder to predict the correct pairings batch (image, text) training examples. (b) Our DeCLIP overview. ! means Self-Superrision(SS)) For image SS, we maximize the similarity between two augmented views of the same instance. For text SS, we leverage Masked Language Modeling(MLLM) within a text sentence. 22 represents cross-mos moosll Multi- View Supervision(MVSS, We first have two augmented views of both image and text, then contrast the 2 2 image-text pairs. 3 indicates Nearest- Neighbor Supervision(NNS). We sample text NN in the embedding space to serve as additional supervision. The combination of the three supervision leads efficient multi-modal learning.
APPROACH
In this section, we first revisit CLIP and denote some basic concepts, such image-text contrastive supervision (i.e., the InfoNCE loss). Next, we present the overview of our DeCLIP framework. Then introduce every auxiliary supervision: Self-Suuperviiion(SS), Multi-View Supervision(MVSS, and Nearest-Neighbor Supervision(NNS),
3.1 REVISITING CLIP
Contrastive Language-Image Pre-training (CLIP) (Radford al. 2021) aims to learn directly from the raw text about images. They use dual-encoder architecture as in Fig. 4a).. The model con- sists an image encoder (e.g., CNN (He et al. 2016) or ViT (Dosovitskiy et al. 2020 and text encoder(e.g, Transformer (Vaswani et al. 2017) or its variants (Radford et al.) 2019)), with multimodal interaction at the top. The image and text features are projected to the same dimension and followed by L2 normalization before interaction. At the training phase, contrastive objective pushes the embeddiigs of matched image-text pairs together while pushing those of non- -ot-mathhed pairs apart. In batch of N image-text pairs {(æ/ =z("), we denote x! and x:! as image and text of the ith pair. Let z and z!" be the normalized embedding of the ith image and jes text, respectively. CLIP uses InfoNCE loss (Oord et al. 2018). The loss for the image encoder can denoted as Eq.
1 log exp(sim(2)) xf)/T)
exp(sim (2/,
Here, the similarity function sim(,) is measured by dot product, and is learnable temperature variable to scale the logits. We have symmetrical loss for image and text encoder, thus the overall loss function LCLIP is the average of L, and Ly.
At the test phase, the learned text encoder synthesizes a zero-shot linear classifier by embedding the arbitrary categories of the test dataset. Because it is rare in the dataset that image caption just
4

---

Published as a conference paper at ICLR 2022
single word, CLIP uses prompts to make up the context of the category (label), such as "a photo of a {iabel}". Unless otherwise specified, we use the same prompt engineering and ensembling techniques as CLIP. Details can be found in Appendix
3.2
OVERVIEW OF DECLIP
As shown in Fig. 4b) our DeCLIP has three additional supervisory signals.
! We first use existing methods to exploit image and text Self-Supervision (SS) within its modality. For image SS, we adopt the simple yet effective SimSiam (Chen & He| (02T). The objective is to maximize the similarity between two augmented image features. For text SS, we adopt the most widely used Masked Language Modeling (MLM) (Devlin et al.] 2018) as the pre-text task. We be- lieve other kinds of self-supervised learning algorithms (e.g. MoCo (He al. 2020 SimCSE (Gao et al. )02T) are orthogonal with our framework.
While SS only focuses on a single modality, we further propose cross-modal Multi- Su- pervision (MVS). We apply stochastic data augmentations for both images and texts, resulting in two correlated views' of each example. Then, the image-text contrastive loss is calculated for all the 2 2 pairs. Worth mentioning, the original CLIP does use text augmentations and only uses random square crop image augmentations, thereby is data-hungry. The extension is instinctive and straightforward. Specifically, we contrast the pairs and resulting 3x more additional supervision.
3 We also propose novel Nearest- -eiisbb Supervision (NNS) mined in embedding space to make better use of similar text deseriptions among the dataset. In detail, we maintain fuuttot feature queue that is representative of the whole data distribution. We use the nearest-neighbor search in embedding space to get the semantically similar text descriptions. Then we use the image- text contrastive loss to get additional supervision.
3.3 SUPERVISION EXISTS EVERYWHERE
Self-Supervision within each modality Following Sim- Siam (Chen & He| 202T) (depicted in Fig) we first have two augmented views (z ) for each image. These two views are sent to the image encoder (weights are shared be- tween views). We also use the popularized nonlinear predic- tor module, which is typically a 2-layer MLP, to improve the representation quality in the encoder (Chen al. 2020a). The objective is to maximize the similarity between 2' and p', which is negative cosine similarity in this paper. To avoid the trivial "collapsing" solution, we follow (Chen & He
Img.
p'
pred
Aug. views
Similarity
Img.
Stop-grad
SimSiam Framework
2021 adopt a stop-grad technique.
Text
FE
As shown in Fig. 5jb), we follow the method in BERT (De- vlin al. 2018 for our text self-supervision. In detail, we first randomly choose 15% of all tokens in each sequence. Then the token is replaced with (1) the [mask] token 80% of the time (2) random token 10% of the time (3) the un- changed token 10% of the time. Then, the output of the lan- guage module for the corresponding token is used to predict the original token with cross-entropy loss.
MASK
enc.
white
(b) Masked Language Modeling
Figure 5: Self-Supervision with each modality. We adopt SimSiam and MLM for image and text SS.
Multi- View Supervision The authors only contrast the original text (w/o augmentation) with single 'global view' of the image in the original CLIP. However, the text annotation of the im- age might not describe the whole picture, instead depicts small local view of this image. For instance, as shown in the image with the text "a cute white cat " in Fig. 4 the central con- cept (cat) only occupies a small part of the picture. To mitigate this discrepancy, we get a closer look at the local region and utilize it as auxiliary supervision, shown in the augmented view in Fig. 4bb.. This intuitive idea is akin to the successful Malti-cron transformation (Caron al. 2020 Van Gansheke et al. 2021) in image SSL. We further extend into the multi-modal
5

---

Published as a conference paper at ICLR 2022
setting. More specifically, reuse the two image views introduced in SS, which contains the RandomResi srnddomResizedCrog policy to obtain a small local view. For text, as our goal is to understand the overall semantic meaning of sentence,
adopt text classification augmentation EDA (Wei & Zou 2019) to generate two text views. Besides the original contrastive loss hetween (z = contrast (z ),"", ( and (#1, ;²"), leading 3x diverse and high-quality additional supervision. More conveniently, they are naturally compatible with the image-text contrastive loss as denoted in Eq.
Nearest-Neighbor Supervision As shown in Fig one image is likely to have other similar text descrip- tions among the dataset. To harness the information from other pairs and go beyond single pairs, we propose using nearest-neighbor (NN) to obtain more diverse su- pervision. More formally,
Feature Queue
Portak of tean playing dog on mosdow
Nourest Neighburr''
aim find the NN fea- ture 2² of text feature =" in the embedding space. The distance between two features could be measured by a simple cosine similarity. It is infeasible to search NN in the whole million-scale dataset. Thus, we maintain a FIFO queue Q to simulate the whole data distribu- tion. The size of Q is 64K in our implementation. As shown in Fig. 6 we further get the contrastive loss be- tween (z*, x Since there are two augmented image features, also calculate the contrastive loss between (2', 2" 2.2"). Fortunately, NNS is also compatible with Eq.
Similarity
Selezt SpeecK Build
Firs--nn Fine-Out
Spacesiiip for Antrica's Retumn Moos.
Man playing with dog on the grass.
artiol performe the Restival city
Figure 6: Nearest-Neighborr Supervision. 2 is the NN of feature 2" in the embed- ding space. =T will serve as an additional objective for 2!. We use the feature-leved nearest neighbor for the text descriptions the supervision.
In summary, we denote LisS and Lyss as the loss function of image SS and text SS, respectively. LMvS is multi-view loss, and LNNS is nearest-neighbor loss.
We have the overall loss function of our
DeCLIP 2
LDELLIP = L
- LOLIP a(Liss + Ltss) BLMVS + NLNNS
(2)
EXPERIMENTS
4.1 DATASETS
Pre-training datasets We summarize our pre-training dataset as Tab. [ Our DeCLIP full data consists of two parts: open-source data and web-crawled data. The open-source data comes from three different datasets: Concep- tual Captions (CC3M) (Sharma et al.) 2018). Conceptual 12M (CC12M) (Changpinyo et al. 2021), and YFCC (Thomee et 2016). Worth mentioning, due to the download failure non- English caption, we do not obtain the com- plete data for these datasets. We further use the YFCC15M query to crawl about 59M filtered web data the Internet, together with open-source data form the 88M DeCLIP full pre-training dataset. More details can be seen in Appendix
Table Details of DeCLIP pre-training datasets.
DATASET
TRAINING SIZE
CLIP RADFORD ET AL
2021
400M
ALIGN JI
CC [HAAAAA FT Al.
2018
3M
CC-T2M CHANNGPNYY
2021
IIM
YFCC
(TOMLE Et AL
2016
15M
DECT.POPEN-SSOO OPEN-SOUKCH
DECLIP WEB CRAWLED DATA DECLIP FULL
59M
88M
Downstream datasets We assess our model performances in wider variety of distributions and tasks. Following the CLIP (Radford et al. 2021). ). we evaluate the image encoder transferability on 11 widely used downstream datasets, such as Food-101, CIFAR-10, etc. To conduct fair comparison, the metric and division for each dataset that can be collected are consistent with those of CLIP. More details of downstream datasets can be found in Tabel of Appendix

---

Published as a conference paper at ICLR 2022
Table 2: Zero-shot topl accuracy on ImageNet. Our DeCLIP shows great data-efficency.
METHOD
IMAGE ENCODER
PARAMS
TRAINING SIZE
ZERO-SHOT TOPI ACC.
CLIP
RESNET50
24M
88M
56.9
CLIP
RESNET50
24M
400M
59.6
DECLIP
RESNET50
24M
88M(L 4.5x)
62.5(1 +2.9)
CLIP
RESNETI01
42M
400M
62.2
CLIP
VIT-B/32
88M
88M
57.4
CLIP
VIT-B/32
88M
400M
63.2
DECLIP
VIT-B/32
88M
88M(L 4.5x)
66.2(1 +3.0)
CLIP
RESNET50 64 REGNETY-64GF
291M
400M
73.6
DECLIP
276M
88M(L 4.5x)
73.7(1 +0.1)
OUR REIMPLEMENTATION.
4.2 EXPERIMENTS SETUP
Following CLIP, we first consider two different architectures for the im- age encoder: a modified version of ResNet50 (Radford et al. (021) and ViT-B/32 (Dosovitskiy et al. 2020). The text encoder is Transformer (Vaswani al. (017) with the architecture modifications deseribed in Radford et al. (2019). The image and text features are projected to the same 1024 dimension, followed by L2 normalization before interaction. Benefiting from the rapid progress of large CV models, we further scale up our model. Our largest model is a RegNerY-64GF (Ra-
Network architectures
dosavovic al., 2020) Goyal et al. 2021) image encoder with a BERT (Devlin al. 2018 text encoder, which is on-pair with the largest CLIP-R50x model.
Pre-training setup For fair comparison with CLIP, we train our DeCLIP-ResNeee and DeCLIP-ViT-B/33 from scratch for 32 epochs. Unless otherwise specified, use full data, i.e.. 88M image-text pairs, to obtain the best performance. The input resolution the image encoder 224 224, and the maximum context length of the text encoder is 76. The learnable temperature parameter 7 is initialized to 0.07. The loss weights of additional supervision a, B and y are all set to 0.2. More details can be found in Appendix
Downstream evaluation setup We evaluate our model transferability by performing linear clas- sification on frozen features, i.e., the pre-trained image encoder is fixed and serves feature extractor. After feature extraction, we train the linear classifier with the L-BFGS optimizer as the same in Radford al, (202T).
4.3
MAIN RESULTS
natural language to refer to visual concepts enabling the zero-shot ability our model. As shown in Fig. [ our DeCLIP-ResNet50 consistently outperforms the 56M (29M open-source + 27M web-crawled), our model can achieve 60.4% accuracy, 0.8% above the CLIP-ResNet50, while using x.1x fewer data. As described in Tab. 2 with full data, best ResNet500ViT-B32 models boost the zero-shot performance to 62.5% and 66.2%, nearly 3.0% higher than the best number reported for these two architectures. Moreover, our DeCLIP-ResNetie is even 0.3% better than CLIP-ResNet revealing the effectiveness and efficiency of our frame- work. Scaling up model capacity works in our framework as well. Our biggest DeCLiP-RegNetY. 64GF achieves 73.7% accuracy, which is 0.1% % above CLIP ResNet50x 64 with fewer parameters.
Zero-shot recognition on ImageNet After pre-training,
across all dataset sizes. When the data amount reaches
Downstream evaluation results We report our linear probe performance on 11 downstream datasets in Tab. 3 Our DeCLIP-ReNNeth outperforms its CLIP counterpart in 8 out of 11 datasets, with a 0.8% average improvement. There are some datasets that our models perform worse than CLIP models, such as SUN and Food101. We conjecture that this is caused by the different dis- tribution of the pre-trained dataset. Interestingly, our ResNet50 and ViT-B/32 models might have distinct performance several datasets, such Pets and Aireraft. We infer that these two types

---

Published as a conference paper at ICLR 2022
Table 3: Linear probe performance on 11 downstream datasets. There are some abbreviations. C10/100 is CIFARI0/100. F101 is Food101. Flow is Flowers. Cal is Caltech. Air is Aircraft. IN ImageNet. Our DeCLIP models achieve higher average accuracy over 11 datasets.
PETS
C10
C100
SUN
F101
FLOW.
CARS
CAL.
AIR.
DTD
Z
CLIP-RESNEI30"
85.1
87.3
65.0
71.3
80.8
98.2
77.2
89.6
44.8
71.0
70.8
76.5
CLIP-ResNer50
88.2
88.7
20.3
73.3
86.4
96.1
78.3
89.
49.1
76.4
73.3
79.1
DuCLIP-ReSNEr50
88.7
89.8
71.2
72.8
82.7
99.2
81.7
93.9
48.4
76.8
74.0
79.9(7 +0.8)
85.3
91.6
74.2
72.3
80.8
97.9
77.6
93.2
47.0
72.5
70.3
78.4
CLIP-VIT-BIL
90.0
95.1
80.5
76.6
88.8
96.9
81.8
93.0
52.0
76.5
76.1
82.5
DECLIP-VIT-BE VIT-B/32
89.2
96.5
84.7
75.0
85.0
99.1
81.6
94.8
53.5
78.5
75.3
83.0 +0.5)
OUR REMPLERMETATEED CL.IP MODBI, TRAINED ON R&M DATA.
30
(27.2%, 304h)
Table 4: Ablation on additional supervision. SS/MVS/NNS denotes Self-Supervision. Multi- Supervision and Nearest- Neighbor Supervision, respectively.
Zero-shot Acc.
25
(21.7%, 416h)
6%, 208h)
DeCLIP(32 Epochs)
15
CLIP(32 Epochs)
CLIP
MVS
SS
NNS
ZERO-SHOT
CLIP(64 Epochs)
10
X
20.6
100 150 200
250 300 350 400 450 Training time (GPU hours)
24.8(1 +4.2)
25.4(1 +4.8)
V
27.2(1 +6.6)
Figure 7: Ablation on pre-training cost on CC3M dataset. The proposed method performs better with less training time.
of neural networks might have different data preferences, i.e., different feature extraction capacities when they meet the same data.
4.4 ABLATION STUDY
Ablation on additional supervision In order understand the effectiveness of each additional supervision, we conduct an ablation study as indicated in Tab, 4 We follow the DeCLIP-ResN protocol in the pre-training setup except for a smaller 1024 batch size on a smaller CC3M dataset. The single image-text contrastive supervision (CLIP) results in 20.6% zero-shot topl accuracy on ImageNet. We can observe that MVS boost amazing 4.2% improvement over the original CLIP. As discussed in Sec. 3.3 the benefits might come from two sides: 1). the MVS can look small local view of an image which might be a better fit with the text description. 2). x augmented views provide 3x diverse and high-quality additional supervision, thus leading to more robust representations. SS can further contribute an additional 0.6% improvement on the basis of MVS. We believe SS could bring more improvements with proper dedicated SSL methods. NNS further brings 1.8% improvement on the high basis of SS. We will discuss more about NNS at Sec. 4.5 On the YFCC15M dataset, our DeCLIP using full additional supervision gets significant effect improvement, Appendix shows the details.
Ablation on training cost Since we need to encode twice for each image-text pair, we admit DeCLIP needs higher training cost than CLIP. Regarding the training time, DeCLIP it- eration equals 1.5x CLIP iteration. In this ablation, we train the original CLIP-ResNet50 longer than our Dest..-ResNet50. As shown in Fig. 7 longer 64 epochs training bring about 1.1 % improvement. However, our model has 27.2% topl accuracy, which is still 5.3% higher than the time-equivalent CLIP. It reveals that framework extract richer and more representative fea- tures through our proposed supervision. We also include memory usage ablation in Appendix
4.5 ANALYSIS
Class activation maps We try to understand what renders our DeCLIP effective. As shown in Fig. R visualize the class activation maps (CAM) (Zhou et al. 2016) of different models trained
8

---

Published as a conference paper at ICLR 2022
cockstoo
coclatoo
cockstoo
lifeboat
ideeoat
lilebot
valleyhall
dining table
dining table
folding chair
Moxican hairless
wortheg
Tibetan lervior
pred
pred
Gabe
CLIP pred
pred
Figure 8: Class activation maps (CAM) for the CLIP vs. our DeCLIP model trained on the YFCC dataset. The CAMs of our model segment the complete object, while the CLIP model only looks a few components.
Original pair
Woolworth Budding Woolworth Building will ki...
Surfing Kaysks ISKILS
Black Rove And Peregrinate Stuck Vector...
support during olympic games
blue background
Slim Fir Shir
cream ice blended
Reading choir
NN
Ein show their during country
pollienn with flowers on blue tackground
the ting tings slim tht shir:
cuokiee cream and nutella wa cone
corhedrul of leaming exierior floors wonderful art deco neo got...
stand up puddle bosrds califomia pau supply
knockdowwn in action icon mobille concupt...
the special chair
CC3M
CC12M
YFCC
Webscrawled
Figure 9: Nearest neighbor samples from different datasets. As similar to the original pair, thus can provide high-quality supervision.
the NN pair is very
on the YFCC dataset. The results validate that the proposed method learns more representative features with the aid of multiple supervision.
Nearest neighbor samples In Fig we show some nearest neighbor (NN) samples from different datasets. The first row is the original image-text pair, while the second row is the NN pair. In general, we can see that the texts have similar intellectual meanings. Therefore, the NN pair can provide high-quality supervision. When taking datasets into consideration, the matching performs very well well-filtered datasets, such as CC3M and CC12M. This matching might a little worse more noisy datasets, such as YFCC and web-crawled, but it can still provide some beneficial guidance.
CONCLUSION
This paper introduces DeCLIP, Data efficient Contrastive Language-Image Pre-training paradigm. Our goal is to learn visual representations through the use of broader and scalable supervision. Specifically, instead of using the single image-text contrastive supervision, we fully exploit data potential through the use of (1) self-supervision within each modality; (2) multi-view supervision across modalities; (3) nearest-neighhor supervision from other similar pairs. Experimentally. De- CLIP shows superior effectiveness and efficiency with different types of neural nets(CNN and ViT) and different amounts of data. We hope our work could bring insights about exploiting the multi- modal data.

---

Published as a conference paper at ICLR 2022
REFERENCES
Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervvsed learning from raw video, audio and text. arXiv preprint arXiv:2104. 11178, 2021.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 2r00 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09982, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3558-3568, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning, 2020b.
Ruizhe Cheng, Bichen Wu, Peizhao Zhang, Peter Vajda, and Joseph E Gonzalez. Data-efficient language-supervised zero-shot learning with self-distillation. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3119-3124, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11162- 1167-11173,, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11999 2020.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: neighbo contrastive learning of visual representations. arXiv preprint arXiv:2104. 14548, 2021.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprini arXiv: (0t::2104.882.. 2021.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 580-587, 2014.
Lluis Gomez, Yash Patel, Marçal Rusiñol, Dimosthenis Karatzas, and CV Jawahar. Self-supervised learning visual features through embedding images into text topic spaces. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4230-4239, 2017.
10

---

Published as a conference paper at ICLR 2022
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint wrXXi:21033019880 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, et al. Pre-trained models: Past, present and future. Al Open, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng, et al. Wenlan: Bridging vision and language by large- scale multi-moda pre-training, arXiv preprint 1xxix:2103.0665 2021.
Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc V Le, Yunhsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.0500
2021.
Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In European Conference on Computer Vision, pp- 67-84. Springer, 2016.
Simon Korbllith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2661-2671, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105 1105, 2012.
Y LeCun and I Misra. Self-supervised learning: The dark matter of intelligence, 2021.
Xiujun Li, Xi Yin, Chumyuan Li, Pengchuan Zhang. Xiaowei Hu, Lei Zhang. Lijuan Wang, Houdong Hu, Li Dong. Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. arXiv preprint 1xXiv:1908.. 02265, 2019.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11); 39 39-41, 1995.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets hinguage-image pre-training. arXiv preprint 12Xi::21 2021.
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):, 2019.
11

---

Published as a conference paper at ICLR 2022
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.000
2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 10428-10436, 2020.
Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pp. 153-170. Springer, 2020.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings ACI,, 2018.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit uisionaadd-languag tasks? arXiv preprint arXiv:2107.06383, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint arXiv: 2014.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfee100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Revis- iting contrastive methods for unsupervised learning of visual representations. arXiv preprint arXiv:2/06.05967, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-60088 2017.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156-3164 2015.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv: 2019.
Bichen Wu, Ruizhe Cheng, Peizhao Zhang, Peter Vajda, and Joseph E Gonzalez. Data efficient language-supervised zero-shot recognition with optimal transport distillation. arXiv preprint arXiv:2112.09445,, 2021.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. 2021.
arXiv preprint 2rXiv22111.07700
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6995-7004, 2021.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00 00747, 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
12

---

Published as a conference paper at ICLR 2022
A
PSEUDO CODE OF DECLIP
The training pseudo code of DeCLIP is as follows:
Algorithm DeCLIP
Input: 1, 1, T, emageeencoder, encodder, Feature Feature.Queur Queue
1: function AATCH-UPDATINGG! T, T))
2:
# Get The Features of The Current Batch.
3:
Tr: I t- image-encoder (T), image
encoder() image.encoder()
4:
Ty.Ty t- teatt_nncoder((", twwttteeeccceeeee
5:
TNN f- NEAREST-NEIGHHOOK(Ettaee Queue, T))
6:
7:
# Calculate The losses.
8:
LCLIP INFONCE-LOSS(,,1I1 Ty)
9:
Lss < -MAGE-SS--LSSS// /f) TEXT-SS-LOSS(T))
10:
LMVS < INFONCE-LOSS((/, T INFONCE-LOSS((,, Tf) INFONCE-LOSS(i, .T)
11:
LNNS + INFONCE-LOSS((, (NN)
LDECLIP & (1 a - 7 + al.ss + BLMvs + YINNS
13:
14:
# Update The Network. image_encoder +- BACKKAARD-UDATEEmmmAS.
15:
LDCLIPP)
16:
test.encoder + BACKWARD- UUDNTRRRD--DDXE(ttttttt .encoder, DDCLLP
17:
Peature.Qucur FIFO-UPDATTEE eatureeeeeeaeeeeeeeeeeer Qtt.//DDDrrrreeeuuuuuuuuuu.u.rr Ty) end function
18:
19:
20: function IMAGE-SS-LOSS((,,1y
21:
ž + loagge_encoderppooj([ amage_encoder proj (iş)
22:
p.P f- image_encoder pred( imageeenoeer (
23:
2, + z.detach(), d.detoch()
24:
# Calculate the Negative Cosine Similarity loss.
25:
Limage-S9 NCS(p, 2 + NCS(p, 2)/2
26:
return T=Image-SS
27:
end function
28:
function TEXT-SS-LOSS(T))
30:
# Get The Masking GT when performing the text-encoder.
Wy. Wyt + wwwoord.fff..o feat, T Ty.makk.id
32:
W "peed &- text cetetenccoerrco encoder.preddW.
33:
# Calculate the Cross-Entropy loss.
34:
Lrext-St
CF(WW gt: W Wreea)
35:
return LT
36:
end function
B DATA AUGMENTATION
Details of SS The image SS uses SimSiam (Chen He, 2021) supervision. The prediction module is 2-layer MLP, in which the hidden dimensions 512 and output dimensions are 1024, the projection module a 3-layer MLP, in which the hidden and output dimensions 2018) as the pretext task. In detail, we first randomly choose 15% of all tokens each sequence. Then the token is replaced with (1) the [mask] token 80% of the time (2) random token 10% of the time (3) the unchanged token 10% of the time.
method as the image self-
both 1024. The text SS uses the Masked Language Model (Devlin et al.
Image augmentations The augmentation policy includes: RandomResizedCron with seale in 1..2 (Wu et al. 2018), ColorJitter containing (brightness, contrast, saturation, hue)
13

---

Published as a conference paper at ICLR 2022
strength of (0.4, 0.4, 0.4, 0.1} with applying probability of 0.8, RandomGrayscale with an applying probability of 0.2. Blurring augmentation (Chen et al. 2020a) has a Gaussian kernel with std in [0.1, 2.0], and Randomoorizontalle
Text augmentations We use the EDA (Wei & Zou 2019) method text augmentation strat- egy, which contains three types of text augmentation strategies: synonym replacement, random swap, and random deletion. Each text will randomly select one of these three types for text augmen- tation.
C PRE-TRAINING DATASETS & IMPLEMENTATION DETAILS
Open-source data. Conceptual Captions (Sharma al. 2018 is 3.3 M image caption open- source data. Due to the failure of the download link, we only download about 3M data(CC3M). Conceptual 12M (Sharma et al., 2018) contains approximately 12M of image-text pairs(CC12M), which is larger than the CC3M and covers more diverse set of visual concepts. Also, due to the failure of the download link, we only download about 1 11M data. YFCC (Thomee et al., 2016). the Yahoo Flickr Creative Commons 100M data, is a dataset for vision language tasks. We download about 86.5M data from the YFCC website and use four filtering rules filter the DeCLIP YFCC15M dataset to benchmark against CLIP YFCCI5M. The four rules are: filtering data with damaged images, filtering data without the caption, filtering data with a caption English word ratio less than 0.8, filtering data with caption only including one part of speech.
Web-crawled data. We use the user tags and machine tags of YFCCI5M to form tag list, and use WordNet (Miller 1995) to find synonyms for each tag in the tag list to form a synonym tag list. the query list to crawl images from
The tag list and synonym tag list form a query list. Then the Internet, after filtering data with smaller images, filtering data with damaged images, filtering data without the caption, and filtering data with Chinese in the caption, collect 59M web crawled the source link of pre-training datasets, and Figure shows some cases random sampled from each dataset.
data.
Table 5: The source link of DeCLIP pre-training datasets.
DATASET
DATASET API
CENCEPPUALCAAPTONNS CONCEPTUALI2M YFCC
HTTPS://AL.COOGII TTTTS//A...ooo...cccEEEERCCCCCCC
12M HITPS://WWW.GoOOL..
GOOGLE
Implementation details We train as V-B32) from scratch for 32 epochs. For R50, we use the FP16-SGD optimizer with the batch size of 10,240 (128x4 (8). Starting with an 0.01 learning rate (Ir), we first linearly increasing the Ir to 0.2 (a.k.a warm-up) in one epoch. Then we use cosine anneal decay to decrease the Ir. The weight decay is set to 0.0001. For V-B32, use hybrid FP16-AdamW-SOH optimizer with bacth size 10,240(128) x8). For the ViT image encoder, we use Adam W optimizer with the warming up from le-4 to le-3 in one epoch. The weight decay is set 0.05. For the text encoder, SGD optimizer with the Ir warming up from le-3 to 0.02 in one epoch. The weight decay is set to le-4.
(abbr. as R50) and DeCLIP- VITLP--VT-B/32 (abbr.
the
Pre-training cost Our R50 and V-B32 took 8/10 days to train 80 V000 GPUs, respectively. Our largest RegNetY-64GF took 21 days on 160 V100 GPUs, while the largest CLIP-R50xc from (Radford et al. 2021) spent 18 days 592 V100 GPUs.
14

---

Published as a conference paper at ICLR 2022
(a)
tA K00el bottl of stap-borry batsamic vhegarr top..
"a cily leatutinn street scones as well as pmall group people."
"a surfer riding seave dur ng dayliis..
"cary mominn guadied like up cocaaoo to awe the sunnise
"wl clouds Hoat acroes the wky"
"abestrat monerraotooooon nonooromme searrieos partern The style voutor"
"Mirror With Max at Trant Park."
"Spicy Rezon Bloody Mary"
over
"New Autonomous Zone"
selection teacinating
(c)
pila eaary
"Elpp coveree Kilmmanrro backgroune,
"Soul treed chickinn, man and cl chaass, and other dishis spread out on batre"
"Brcets downtown Mumihh The Munich. Bain, Bus Couch, Tramway, Light Rail Boveni Germany. limu, Public Transport. Tesine"
"High Prom
roomn
Onkor
Naray
"Daubb dia floe
"Look inside the Cedord Advanoed of Dictionary
"Stikky honey miso glazed
"Literly Mystitying Information About Marrs Cala Appy"
"Geey Fismes Chair 50 modell CSTrader"
"Cash Rox Money Doswar Ma Market Sales Etsy
Provencell
most fonder and aromatic Beet"
Figure 10: Example image-text pairs randomly sampled from the training dataset. (a) Conceptual Captions, (b) YFCC, (c) Conceptual 12M, (d) Web-crawled.
D DOWNSTREAM DATASETS & IMPLEMENTATION DETAILS
Downstream data. We begin with the 12 datasets from the well-studied evaluation suite intro- duced by Kornblith et al. (2019). Within these 12 datasets, Birdsnap can not be downloaded, and PASCAL VOC 2007 classification is replaced by more challenging ImageNet-1KK K, resulting in our
11 datasets. They are: Food- 101, CIFAR-10, CIFAR- 100, SUN397, Stanford Cars, FGVC Aircraft, Describable Textures, Oxford-IIIT IIIT Pets, Caltech- 101, Oxford Flowers 102. Tab. 6 is the detailed information of these datasets.
Table 6: Details of DeCLIP downstream datasets.
DATASET
CLASSES
TRAIN SIZE
TEST SIZE
EVALUATION METRIC
CIFARI0
10
50,000
10,000
ACCURACY
CIFARION 100
100
50,000
10,000
ACCURACY
FOOD-101
101
75,750
25,250
ACCURACY
OXFORDLLIT-
37
3,680
3,669
MEAN PER CLASS
OXFORD 102 FLOWERS
102
2,040
6,149
MEAN PER CLASS
SUN
397
19,850
19,850
ACCURACY
STANFORD CARS
196
8,144
8,041
ACCURACY
DTD
47
3,760
1,880
ACCURACY
CALTECH- 101
102
3,060
6,085
FGVC AIRCRAFT
100
6,667
3,333
MEAN PER CLASS
IMAGENETIK
1000
1,281,167
50,000
ACCURACY
15

---

Published as a conference paper at ICLR 2022
Implementation details We follow CLIP (Radford et al. 2021) to train logistic regression clas- sifier using L-BFGS, with maximum 1,000 iterations, and report the corresponding metric for each dataset. We determine the L2 regularization strength À using hyperparameter sweep on the val- idation sets over the range between 10-6 and 10,, with 96 logarithnically spaced steps. To save compute required for the sweeps, we perform parametric binary search that starts with À [10-6, 10 1002 1, 10 10', 10"] and iteratively halves the interval around the peak until it reaches a resolution of steps per decade. The hyperparameter sweeps are performed on a validation split of each dataset. For the datasets that contains validation split in addition to from the test split, we use the provided validation set to perform the hyperparameter search, and for the datasets that do not provide a validation split or have not published labels for the test data, we split the training dataset to perform the hyperparameter search and report the performance on the validation data.
E PROMPT ENGINEERING
Due to the reason that it's relatively rare in the dataset for the text to be a single word, we use prompts such as "a photo of a {label]" for zero-shot classification. For a fair comparison, we use the same prompts as proposed in |Radford et al. (2021) for the ImageNet dataset. As shown in Fig II. the prompts reduce the domain gap between the training dataset and testset, and fully consider the different situations for the picture.
a photo [labet].
a close-up photo a [label].
the origumi (label).
jong corrupted ploto the (labe
a ploto of many [label].
a black and white photo of the (label).
the (label) in a video gazae.
good photo of (Babel).
a sculpture of [label].
a painting of the (label).
shotch of (laiel).
plushic (lahel).
platoo of the hard to [labed].
painting
(label).
doodle of the (labet).
photo of nice (label].
low resolution photo the [laee].
a pizcelated photo the (label).
origami [lthel].
photo of
(labe)).
a coudoring of a (balal)).
a sculotin of the [lubd].
a low noctution plotto of (labee)).
a photo of the weid [Inist).
graffiti of a (label).
bright photo of the [label).
the toy (labe)..
the cactoon (labe)).
bad photo of the [label].
crupped photo
(label).
rendition of the (label).
of the (label).
a crogyed photo the (labal).
a plutic (labe)).
photo
clous [lalw]].
drawing
[label].
a tattoo of (labe)).
a photo of the dirty (label)-
photo a large (labe))-
photo of the large (label).
the embroiderod (lube)).
joes corrupted pioto of (Jabal).
sondition of a (belel).
black and white photo of qhiel
ploto of aadd
(label)-
bluxty photo of (label).
phato nics [inlat)..
plushic (label).
a beight photo
[labe)].
a photo the [labe).
photo
(label).
ploto [lahe)).
photoo of a cloa [labw]].
a good photo of the (babel).
a blury photo of a (bale)).
Stap of a (labe)).
photo of a dirty (luhe].
remdering of the (lbbe))
cartoon [labe)).
gralti of the (labso)).
dark photo the [label).
a [label]
game.
toy (abel).
a drawing of is [lubd).
a photo cne [label).
shetch of (label)
Stap of my [labal).
a photo of my [lahel).
a doocle of a [label).
a embovidered (label).
photo of cool [label).
the plastic (laee)).
clossopp photo of the (label). [Ishad)
piselated ploto of (label).
photo of small (label).
a ploto of the coo (aaell).
Haap the (labsl)..
batioo the (boal).
Figure 11: The prompts for zero-shot testing.
F ADDITIONAL. STUDY
Table 7: DeCLIP zero-shot performance of ImageNet topl on different training datasets.
BACKBONE
DATASET
DATA SIZE
BATCh SIZE
ZERO-SHOT
RESNET50
CONCEPTUALCAPTIONS
3M
2,048
27.8
RESNET50
CONCEPTUALI2N
11M
4,096
41.0
RESNET50
YFCC
15M
4.096
41.9
RESNET50
DECLIP OPEN-SOURG DATA
29M
6,144
49.3
Different Pre-training Datasets Data is critical for language-imane -magee pre-training task. As shown in Tab. 7 we evaluate our DeCLIP on different sources of datasets. Combining Tab and Fig we can see that when the amount of training data continues to scale up, the zero-shot recognition ability continues to improve as well. In addition, we can see that the open source data has high quality. The 29M open source data achieve 49.3% zero-shot topl accuracy ImageNet through the DeCLIP training paradigm. Our open-source data is an affordable benchmark, which would be beneficial for explorations.
16

---

Published as a conference paper at ICLR 2022
the number of image-text pairs the CLIP WFCC-15M, our YFCC data is different from CLIP. We also reproduce the naive CLIP on our YFCC-15M data, which results in 35.9% zero-shot top1 accuracy ImageNet- 1K (see Fig. g It is relatively higher than the number in CLIP paper
Our YFCC re-implementation Although
We conjecture the improvements might be caused by the different data cleaning strategies. However, DeCLIP can achieve 41.9 41.9% zero-shot accuracy which is also 6.0% higher than our CLIP re-implementation.
Table 8: DeCLIP zero-shot performance of ImageNet topl on YFCC datasets. Although we use the same amount of data, our YFCC is different with CLIP YFCC due the different data cleaning strategies.
MODEL
BACKBONE
DATASET
DATA SIZE
BATCH SIZE
ZERO-SHOT
CLIP
RESNET5O RESNET5O RESNET5O
YFCC
15M 15M
31.3
CLIP (OUR REIMP.) DECLIP
YFCC*
4,096
35.9
YFCC*
4,096
41.9(1+6.0)
Memory usage Because of the additional views, our DeCLIP is more memory-consumingg Thanks to the ICLR anonymous review comments: fairer comparison might be doubling the batch size of CLIP. Flowing the ablation study in Fig. we double the batch size and train CLIP-ResNet50 for 64 epochs. The final result 22.3% which is still 4.9% lower than our DeCLIP model. We summarize the memory usage, training cost, and the final accuracy as below. All experiments are conducted on CC-3M with V100 GPUs.
Table 9: Ablation on Memory usage.
MODEL
BATCH SIZE PER GPU
MEMORY (GB)
EPOCHS
COST (GPU HOURS)
ZERO-SHOT
CLIP- RESNET5O
128
15.8
64
416
21.7
CLIP- REP-RE
256
24.0
64
399
22.3
DECLIP-RESNIE
128
22.7
304
27.2
17


Reviews:
[{"review_id": "NkaP4s_2Iip", "reviewer": "Reviewer_a3Wz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Clear advantage of the proposed framework on data efficiency.\n- Simple and straightforward methodology.\n- Carefully designed experiments. (Comparison to CLIP with 15M data and ablation on computation time.)\n\nCons:\n- Limited novelty in terms of the method:\n    - The self-supervision and multi-view supervision are very similar to what has been used in Yuan et al.\n    - The Nearest-neighbor supervision was used in Dwibedi et al. and Van Gansbeke et al.\n- Experiments:\n    - Missing experiments:\n        - One of the biggest advantages of CLIP is its robustness to domain shift. Can the authors also provide performance on datasets like ImageNet-R, ImageNet Sketch etc.?\n        - No results on image text retrieval.\n        - The CLIP reports on 27 downstream datasets. Is there any specific reason for not comparing on all these datasets?\n    - While the authors compare with training time, another big difference is in memory consumption. With the additional views, it is equivalent to doubling the batch size. A fairer comparison should be doubling the batch size for CLIP training and train for the same number of iterations or the same amount of time. Since large batch size is important in contrastive learning, it is worth-doing to me.\n\nQuestions:\n- I believe CLIP also use random crop during training. How do the authors design the small local view (how small)? Can authors provide ablations on the effect of small size?\n    - In addition, is the augmented view the same size (in terms of network input) as the original view but just more zoomed in?\n- Did the authors do careful duplication removal?", "summary_of_the_paper": "In this paper, the authors mitigate the data-hungriness of the CLIP model. The authors propose three directions: single-modality self-supervision; multi-view multi-modality contrastive learning, and nearest-neighbor supervision. With the proposed three components, the authors can achieve better or comparable results with CLIP with more than 4x fewer data.", "summary_of_the_review": "While the novelty is limited, the authors are able to achieve good results with a much smallerset data size. Potentially, with more data, this method can achieve much more superior performance than CLIP. However, I think the insights conveyed from this paper are limited so I will choose borderline accept.\n\n\nPost-rebuttal update:\n- I am very happy with the additional results, which address most of my concerns on fair comparison with CLIP. I will remain the original rating due to the limited \"novelty\". But I would give a 7 if there is a 7 because I think this paper should be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NfwmpnGvw_s", "reviewer": "Reviewer_nUjZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- (+) The proposed DeCLIP shows better ImageNet zero-shot performances than the original CLIP models with the same parameter size but smaller training data size.\n- (+) The ablation study (Table 4) supports that the proposed three objectives are important to the final performances. Also, Figure 7 shows that the convergence speed of DeCLIP is faster than the original CLIP loss.\n\n## Weaknesses\n\n### CLIP numbers and DeCLIP numbers would not be comparable; the CLIP training environment is different from the DeCLIP training environment\n\nI fully understand that CLIP results are almost impossible to reproduce with ordinary infrastructures. It is too expensive to reproduce the original ResNet50x64 results (18 days with 592 V100 GPUs). It takes more than one week for ResNet and ViT DeCLIP training even with 80 V100 GPUs (Appendix C), and the largest RegNet DeCLIP takes three weeks with 160 V100 GPUs. Hence, I fully understand that it is very difficult to report many numbers.\n\nHowever, since this paper argues that \"CLIP is data-hungry and DeCLIP can learn strong representation as much as CLIP with much smaller dataset size\", I think that the CLIP results should be the reproduced numbers.\n\n- The original CLIP paper did not investigate the impact of dataset size (except ablation study on the YFCC-15M dataset), and there is no justification of the dataset size of 400M. Hence, I think that arguing \"CLIP needs 400M data points\" needs to be justified by the following studies (namely, in this submission). In other words, I would like to see \"reproduced CLIP results\" in Figure 1 using the authors' implementation. Since DeCLIP needs more resources than CLIP (because of additional data augmentations, computation costs, memory size, ...), I think the authors can report CLIP results for 15M / 29M / 56M / 88M as DeCLIP.\n- I presume that the reproduced CLIP environment is different from the original CLIP implementation (which is not publicly accessible yet). Table 8 amplifies my assumption; the reproduced CLIP performance is 4.6% better than the original CLIP (31.3 => 35.9). In my opinion, it could be very critical if the performance gains are originated from the DeCLIP implementation itself, not from the proposed objective functions. Hence, as I suggested in the previous bullet point, I would like to see the reproduced CLIP numbers for various dataset sizes.\n- Third, we do not have enough knowledge on **88M DeCLIP full data** collected by the authors. I wonder whether the performance improvements are from the data collection, not the proposed method. For example, 88M full data quality could be better than the original 400M dataset. As described in Section C, additional filtering processes, such as Chinese caption filtering, would improve the dataset quality compared to CLIP. It is well-known that learning with fewer clean data often outperforms learning with a large noisy dataset. To avoid such concerns, I think Figure 1, Table 2, and 3 should need additional rows \"CLIP training + 88M training data\" for each model. In my opinion, because the training datasets are different, CLIP 400M models and DeCLIP 88M models are not directly comparable in zero-shot and linear probe benchmarks.\n- Finally, CLIP did not use any data augmentation on the language domain, while DeCLIP uses the EDA augmentation for the multi-view supervision. It is not clear whether the performance improvement by MVS in Table 4 is from the EDA augmentation or the multi-view supervision. Because DeCLIP uses more data points than CLIP (Algorithm 1: Line 10), I wonder if the performance gain is from the data augmentation. CLIP + EDA augmentation experiments may answer my question.\n\nSimilarly, I would like to suggest reporting RegNetY-64GF CLIP results in Table 2.\n\n### Additional analyses may need quantitative comparisons\n\nThis weakness is not very critical, but the submission will be stronger if the authors can address this issue.\n\nFigure 8 and Figure 9 show the CAM visualizations and nearest neighbor search results by CLIP and DeCLIP. However, these results are very easy to cherry-pick; hence, these results should be supported by the quantitative comparisons.\n\nFor CAM comparisons, I would recommend weakly-supervised object localization (WSOL) benchmarks. WSOL does not require any additional training but only CAM score maps. I recommend using Choe et al. https://github.com/clovaai/wsolevaluation for comparing CLIP, and DeCLIP CAM results more concretely.\n\nNearest neighbor search performances can be easily measured by cross-modal retrieval benchmarks (in a zero-shot manner). I would recommend famous COCO or Flickr caption retrieval benchmarks. If the authors need more dense benchmarks, please refer Parekh et al. https://github.com/google-research-datasets/Crisscrossed-Captions.\n\nSince these benchmarks are available without additional expensive training, I strongly recommend reporting the results using the suggested benchmarks.\n\n## Questions\n\n- Why are there no zero-shot results for other benchmarks? I only can see the linear probe verifications (Figure 2, Table 3).\n- In Table 3, DeCLIP ViT-B/32 shows worse linear probe results on ImageNet than CLIP VIT-B/32, despite DeCLIP showing better zero-shot ImageNet results in Table 2. What is the guess by the authors for the results?\n- It seems that the proposed DeCLIP needs more computation / memory resources compared to CLIP. Could the authors provide the memory consumption comparison between CLIP and DeCLIP? I presume that with the same GPUs, DeCLIP cannot use the same batch size as CLIP.\n\n## List of suggested experiments\n\nI fully understand that the rebuttal period is too short to produce the following numbers. At least, I would expect 88M CLIP results (Table 2) for ResNet-50 and ViT-B/32.\n\n- I am willing to revise my score if these results show the same tendency as the paper claim\n  - Figure 1. Reproduced CLIP ImageNet zero-shot top-1 results for 15M / 29M / 56M / 88M (15M is already in Table 8) -- ResNet-50\n  - Table 2. Reproduced CLIP ImageNet zero-shot top-1 results for 88M with ResNet-50, (ResNet-101 -- not necessary), ViT-B/32, and RegNETY-64GF\n  - Table 3. Reproduced CLIP linear probe results on 11 downstream tasks\n- These numbers are not mandatory, but I think these results will make the submission stronger\n  - Figure 8. WSOL benchmark (https://github.com/clovaai/wsolevaluation)\n  - Figure 9. Cross-modal retrieval benchmarks (COCO Caption, Flickr Caption, Crisscrossed Caption, ...)\n  - CLIP + EDA augmentation results on 15M / 29M / 56M / 88M datasets (ImageNet zero-shot top-1)", "summary_of_the_paper": "This paper proposes a data-efficient CLIP (DeCLIP) by employing three additional objectives:\n\n- (1) self-supervision within each modality (e.g., vision domain => SimSiam, text domain => masked language model as BERT)\n- (2) Multi-view supervision (e.g., resize random crop for vision modality and EDA augmentation -- synonym replacement, random insertion, random swap, and\nrandom deletion -- for text modality)\n- (3) Nearest-neighbor supervision (e.g., using close plausible neighbors to pseudo-positives)\n\nTable 2 shows that DeCLIP shows better zero-shot ImageNet top-1 accuracies with smaller dataset sizes (400M => 88M), and Table 3 shows that DeCLIP shows better overall linear probe performances on various datasets.\n\nAlso, Table 4 and Figure 7 support that the proposed three additional objectives make the CLIP training much efficient than the vanilla version.", "summary_of_the_review": "This paper shows a strong empirical contribution; DeCLIP shows better results than CLIP with a smaller dataset size. However, I think the weakness of this paper slightly overweighs the strengths.\n\nIn particular, I am not fully convinced that DeCLIP is data-efficient and CLIP is data-inefficient. This argument should be supported by empirical comparisons, e.g., the reproduced CLIP results should be reported in Figure 1, Table 2, and 3. I listed why I think the DeCLIP results and the CLIP results are not directly comparable in the weakness section.\n\nAll technical components of DeCLIP are hard to say novel; self-supervision uses SimSiam and masked language model, multi-view supervision uses the EDA augmentation, nearest neighbor supervision can be viewed as a self-distillation method. However, the combination of known techniques can be novel if the empirical contribution is significant. I think this paper has a limited borderline novelty, but the empirical contribution can be a strength. For example, if this paper outperforms \"ViT-L/14@336px\" with much fewer data points, I think the empirical contribution exceeds its weakness. (disclaimer: I do not request ViT-L/14@336px results. I know that it is very expensive and unachievable by ordinary infrastructures) As of now, I think the empirical contribution may need more verifications, especially for the vanilla CLIP results.\n\nOverall, I recommend \"marginally below the acceptance threshold\" for my initial recommendation.\n\n-----------------------------\nPost rebuttal comment.\n\nAfter reading the responses (for raised concerns by all reviewers) and the revised paper, I think the authors address my concerns very well in general. I would like to encourage the authors includes the results in A3, A4 (not completed yet), A5, A6 in the paper. Also, I encourage the authors to add zero-shot results (Q7) in the final revision, which was not possible at the submission time (I missed it).\n\nAfter the revision, the arguments are generally well-supported (revised my score for correctness has been updated to 4 from 3). Now, I recommend \"6: marginally above the acceptance threshold\" for this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C7eEUbcCVI", "reviewer": "Reviewer_kziT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- Extensive experiments show DeCLIP achieves competitive improvements over CLIP while uses much less pre-training data, which also results in much less training time. With the same amount of pre-training data, DeCLIP achieves significant improvements over CLIP (+7%)\n- The three additions are simple yet effective. The ablation study shows that all three of them contribute to the final performance improvements.\n- Visualization show that nearest neighbor pairs can provide reasonable similar supervision signal compared to the original pair.\n- The paper is well-written and easy to follow.\n\n\nA few questions about the paper:\n- For nearest neighbor pairs, are them all sampled from the same dataset as the original image-text pair? If so, are there different FIFO queue Q for different datasets during pre-training?\n- What about other distance measures to compute nearest neighbors, for example cosine similarity? \n- Why not also sample nearest neighbor images? If added, it should add 8x more supervision than the original CLIP.\n", "summary_of_the_paper": "This paper proposes DeCLIP with three additions to the original CLIP model: (1) single-modal self supervised pre-training; (2) cross-modal contrastive pretraining across multiple views and (3)  cross-modal contrastive pretraining across nearest neighbors in feature space. DeCLIP is able to achieve higher performance on most of downstream datasets with much less pre-training data (80M vs. 400M). ", "summary_of_the_review": "This paper provides three simple yet effective additions to CLIP and show its strong performance with much less pre-training data. Overall, I feel the paper is clearly written, with extensive experimental results to support its claim.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AZrxIVnuhYG", "reviewer": "Reviewer_x3bM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPro:\nThis is the first work that I can think of that successfully combines self-supervision and supervised training, at such a large scale. While self-supervised training has shown promising results in CV, the experiments are usually done in an unsupervised/self-supervised setting where the model is trained with only raw images and then tested with linear probes. It remains less explored how one could combing self-supervision and supervised image-text pairs data and whether self-supervision is still helpful when a large amount of supervised data are available. This work provides strong empirical evidence that self-supervision can be combined with supervised data.\n\n   \n\nCon:\nTable 2 shows that DeCLIP with 88M data can outperform CLIP with 400M. However, the 88M data are collected by the authors while the 400M data are collected by CLIP authors. As shown in Table 8 in the appendix, how one process/selects the data can result in a large performance difference. \n\nI cannot safely conclude that the data efficiency/performance improvement shown in Table 2 is completely attributed to DeCLIP. It could be attributed partially to that the 88M data are a high-quality subset of the 400M data. I would recommend adding a baseline of CLIP trained with 88M data.\n\nThe authors provide fair baselines of CLIP in Figure 7 and Table 8 (in the appendix) but the dataset scale is limited. As one of the core claims is the benefit of DeCLIP under large-scale pre-training, I think providing fair baselines in Table 2 is important.\n\nMinor points:\n\n   1. Analysis lacking on why we need self-supervision signals / nearest-neighbor supervision. \n\n      As noted in Pro, this paper touches upon an important problem: the relationship between self-supervision and supervised data. \n\n      However, the authors only show empirical evidence of self-supervision can be helpful even when a large amount of supervised data are available. It would be helpful to provide some discussions. For example, does cross-view self-supervision provide signals simply not available in supervised data? Does DeCLIP do better in certain types of tasks?\n\n   2. I am under the impression that nearest neighbors are usually used to provide \"hard negatives\" in contrastive learning (e.g., in the Image Retrieval task in UNITER (Chen et al., 2019)). However, here the nearest neighbors are used as \"positive\" examples. I wonder what causes the different practices.\n\n", "summary_of_the_paper": "The paper proposes DeCLIP to further utilize the data potential by adding three training objectives to CLIP pre-training: 1) inspired by SimSiam and BERT, self-supervised objectives are added to both image and text; 2) they generate different views for both images and text, and apply contrastive objectives; 3) they sample neighbor text as additional positive examples.\n\nDeCLIP improves data efficiency. With web-crawled data, DeCLIP outperforms CLIP counterparts with 4.5x smaller amount of data. In addition, while introducing addition objectives and especially different views increases per-batch compute time by 1.5x, the authors show that DeCLIP still outperforms CLIP when given the same compute time budget.\n\n", "summary_of_the_review": "**After rebuttal: I have read the authors' rebuttal and it resolves my concerns. I will raise my score to between 6 and 8.**\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims at improving the data efficiency of pretraining in CLIP. This is a practically meaningful research direction. The proposed method is simple, even kind of straightforward and has limited innovations. It combines self-supervision within each modality, multi-view supervision across modalities, and nearest-neighbor supervision from other similar pairs. Such a combination showed strong empirical results: achieved better performance using seven times fewer data. The rebuttals resolved most critical concerns on experiments, such as fair comparisons with the original CLIP work.
Decision: Accept (Poster)