The Role of Pre-training Data in Transfer Learning
Authors: Rahim Entezari
Mitchell Wortsman
Olga Saukh
M. Moein Shariatnia
Hanie Sedghi
Ludwig Schmidt
Venue: Submitted to ICLR 2023 (2023)
Keywords: pretraining
transfer learning
supervised training
constrastive learning
clip
simclr

Abstract: The transfer learning paradigm of model pre-training and subsequent fine-tuning produces high accuracy models. However, a question remains: what data and method should be used for pre-training? We study the effect of the pre-training distribution on transfer learning in the context of image classification. Through controlled experiments, we find that the pre-training dataset is initially important for low-shot transfer. However, the differences between distributions is diminished as more data is made available for fine-tuning. Still, fine-tuning outperforms training from scratch. We also investigate dataset size and observe that larger pre-training datasets lead to better accuracy, however, the absolute accuracy difference is largest in the few-shot regime. Beyond data, we study the effect of the pre-training method, language-image contrastive vs. image-image contrastive, finding that the latter usually leads to better transfer accuracy

Under review as a conference paper at ICLR 2023
THE ROLE OF PRE-TRAINING DATA IN TRANSFER LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
The transfer learning paradigm of model pre-training and subsequent fine-tuning produces high accuracy models. However, a question remains: what data and method should be used for pre-training? We study the effect of the pre-training data distribution on transfer learning in the context of image classification, investigating what extent different pre-training datasets differ in downstream task performance. Through controlled experiments, we find that the pre-training dataset is initially important for low-shot transfer. However, the differences between distributions are diminished as more data is made available for fine-tuning. We also looked into how much is labeling worth compared to noisier but larger pre-training data. Our results ImageNet we
show that to match the performance on supervised pretraining need 15x-2200x more pre-train data from LAION for different downstream tasks. We also investigate the dataset size and observe that larger pre-training datasets lead to better accuracy, however, the absolute accuracy difference is the largest the few-shot regime. Beyond data, we study the effect of the pre-training method, language-iman contrastive vs. image-image contrastive, finding that the latter usually leads to better transfer accuracy.
INTRODUCTION
The best-performing computer vision models are produced by the transfer learning paradigm. In this two-step procedure, a model is first pre-trained on a large heterogeneous dataset. Next, the model is fine-tuned application specific data which adapts the model to problem of interest. While transfer learning is not new, it has become increasingly important with drastic improvements in the quality of pre-trained models (e.g., CLIP Radford et al.] (2021). BASIC Pham al. (202T), and Flamingo Alayrac et al. (2022)). These improvements are driven by new datasets for pre-training as well as better pre-training algorithms. This naturally leads a question:
How does the dataset and algorithm used for pre-training affect downstream transfer performance?
While related works try find the relation between pre-training and transfer performance by exploring scaling laws (Kornblihh et al. 2019 Abnar et al. 2021) or predicting transferability without actual finetuning (You et al. [022] Nguyen et al. 2020 Deshpande et al. [021 Bolya et al. [022I) we highlight that to the best of our knowledge the role of pre-training data distribution has not been investigated so far. Therefore we define specific research questions detailed below and set up systematic experiments focusing on each question, while carefully ablating the other factors.
To what extent do different pre-training datasets differ in downstream task performance? Do we expect different distributions to perform differently in the transfer setting and how does that compare to training from scratch? When controlling for size but changing the pre-train dataset, we observe noticeable differences in downstream transfer accuracy. These differences are larger in the few-shot setting when only a few examples per class are available for fine-tuning. When many images are available for fine-tuning, the difference in absolute accuracy when varying the pre-training dataset mainly evaporates. Across many downstream tasks, certain pre-training datasets (i.e, Shutterstock) consistently lead to better transfer accuracy than others (i.e., WIT). However, there are still ordering differences from one downstream task to another. Moreover, even the pre-training dataset which leads to the worst transfer accuracy still outperforms training from scratch (see Figure

---

Under review as a conference paper at ICLR 2023
How much is expensive labeling worth compared noisier but larger pre-training data? We compare different pre-training strategies: supervised pre-training on small but labeled ImageNet and semi- supervise pre- trainini on image and language pairs larger but noisier datasets. We find that pre-training on a well-curated dataset leads to better transfer accuracy than pre-training on a noisy dataset of similar size, and pre-training only on 15x-2000x larger noisy dataset can close the gap (see Figure 2
How much does increasing pre-training dataset size contribute to the performance of transfer learning? When controlling for the pre-training dataset and instead changing the size, we observe that models pre-trained on more images usually have better transfer performance. However, similarly to the aforementioned results, the absolute difference in transfer accuracy between models pre-trained on small scale and medium scale datasets diminished when fine-tuning on more data. We also observe that increasing the pre-training dataset size shows different saturation performances on target tasks, i.e. while even 100X more data does not help on transfer to some downstream tasks, including more data improve the downstream performance of others (see Figure
What is the role of pre-training method on transfer performance? ? We also examine the difference between supervised pre-training with the popular CLIP and SimCLR semi-supervised algorithms. Overall we find that the SimCLR pre- training leads to better transfer than CLIP pre-training in the low-shot regime, but that there are only small differences when many images are used for fine-tuning (see Figure
To answer these questions conduct extensive empirical investigation (over 4000 experiments) in
the context of computer vision. Our study covers 7 pre- training datasets (YFCC (Thomee et al. [0116)
LAION (Schuhmann et al., 2021), Redcaps Desai et al. ),, Conceptual captions-3m (Sharma
ct al. 2018), Conceptual captions- -2miions-1mmm (Changpinyo et al.] 2021)) WiT (Srinivasan et al.) 2021 l..
Shutterstock, ImageNet (Deng al. 2009)). tineinuning datasets (CIFARIO0 (Krizhevsky et al.
2009), DTD (Cimpoi et 2014)) Caltech-101 (Fei-Fei et al., 2004), PETS (Parkhi et al.) 2012).
REAL and CLIPART from DomainNet (Peng 2019). EuroSAT (Helber et al. 2019). Cassava
Leaf Disease Classification (Cas), and Caltech Camera Traps-20 (Beery et al. 2018) ), and two
pre-training methods CLIP (Radford et al.] (02T) and SimCLR (Chen et al.] | (020). To evaluate
transfer performance, we examine both few-shot fine-tuning and full fine-tuning.
The paper is structured as follows:
we review related work and provide relevant background on transfer learning in Section 2 followed by our experimental setup in Section 3 Section 3 details our observations relating to our research questions by measuring the downstream transfer accuracy models pre-trained on various data sources, dataset sizes, and with different pre-training losses. We discuss our findings and conclude with future research directions in Section
2 RELATED WORK
Transfer learning is widely used in deep learning research and practice and has become a cornerstone in both computer vision and natural language processing. Through the years, there have been many questions on why transfer helps and how to choose good pre-trained model to transfer from. Neyshabur et al. (200)) separated the effect of feature reuse from that of learning low-level pre-training data statistics. Their study involved models with supervised pre-trained on ImageNet. Another important question is whether transfer learning is always helpful on any downstream dataset. Raghu et al. (019) experimented with downstream medical datasets (with images coming from a distribution that is very different from that of ImageNet or other natural image datasets) and found that transfer learning from ImageNet pre-trained models shows little benefit in performance. This shows that the downstream dataset is an important factor to consider when evaluating the transfer performance of upstream models. To make it possible to more generally evaluate the visual representations of upstream models, Zhai et al. (2019) introduced the Visual Task Adaptation Benchmark (VTAB). VTAB aims to measure the adaptability of representations to diverse, unseen tasks, given only few examples from the downstream dataset.
Scaling up dataset and model size is a well-known trend for improving accuracy in both natural language processing (Kaplan et al. 2020) and computer vision (Kolesnikov et al. 2020). For instance, Kolesnikov et al. (2020) uses a weakly labeled JFT-300M dataset for pre-training An even large noisy dataset with 3.5B images from Instagram was used in (Mahajan et al. 2018). To make use of large
2

---

Under review as a conference paper at ICLR 2023
100
100
90
190
80
Finelune Rocuracy [N6
Finstune accuracy [W]
Finstune acturacy (N)
60
60
60
Pre-sai Dataseta: cattee
40
80
40
20
20
20
Shute rando 2m
Serach
203
300
200
20
Number of
Number shots
(a) CIFARIO0 100
(b) DTD
(c) CALTECHI01
100
Fintuue accunscy (6)
Fineuuee accuney [N]
Finetune ancurety [N]
30
[30)]
Ball (260)
20
kall (200)
Number of shots
Number of shots
Number shots
(d) PETS
(e) DomainNet REAL
(f) DomainNet CLIPART
Figure 1: Efffec of the pre-training data distribution. In the low-shot setting, different pre- pre-training datasets lead to noticeable differences in downstream transfer performance. If many samples are available for fine-tuning, the difference in absolute accuracy between models pre-trained on different sources largely evaporates. See Figure for an extension to more downstream datasets.
datasets without laborious labeling, researchers have proposed techniques to learn representations by unsupervised and self-supervised training. [Chnn et al. (2020); Caron et al. (202)): He al.) (2020): Grill et al. (2020) use image-only datasets to learn representations with self self-supervised loss functions, one of the most popular being the contrastive loss. Ericsson et al. (2021) studied the transfer performance of self-supervised models and found that the best self-supervised models of that time could outperform supervised pre- pre-training as an upstream source of knowledge transfer and that the performance of self- supervudde models ImageNet is indicative of downstream performance on natural image classification tasks. Similarly, [Islam et al. (20TT) found that contrastively trained models consistently outperform standard cross-entropy models in transfer learning. [ooyal al. (2021) showed that self-supervisce models outperform supervised models on ImageNet, even when trained on random and uncurated images from the web. Moreover, they showed that these models also good at few shot learning by achieving 77.9 % top-1 accuracy using only 10 % on ImageNet.
Building on contrastive techniques, Radford et al. (02T) introduced CLIP which learns joint embedding space for both images and their descriptive captions, making it possible to effectively leverage a large-scale dataset from the Internet. Flamingo (Alayrac et al.) 2022). visual language model, is another successful example in the line of multimodal models and enables visual question answering and image captioning. CLIP and similar models like ALIGN (Jia et al. (021). BA- SIC (Pham al. (0221, and LiT (Zhai et al.) 2022) demonstrated unprecedented robustness to challenging data distribution shifts. This accomplishment raised questions on the probable sources of such robustness whether this robustness is caused by language supervision, the pre-training data distribution, size, or contrastive loss functions.
Fang al. 2022) investigated this question and found that the diverse training distribution is the main cause of the robustness properties CLIP. Nguyen et al. (2022) explored the role the pre-training dataset for CLIP with testhed of six pre-training sources, finding that no single pre-training dataset
3

---

Under review as a conference paper at ICLR 2023
consistently performs best. In recent work, Santurkar et al. (2022) carefully investigated the effect of language supervision in CLIP-like models, finding an important factor the pre- training dataset is large and the captions are descriptive enough. Unlike their work, consider end-to-end fine-tuning which result in higher accuracy.
Kim et al. (2022) conducted an in-depth study of the effect of the network architecture, pre-training dataset, supervised vs self-supervised learning objectives, and different domain transfer methods on the transferability of representations to new domains. They found that the transferability of the pre-trained representations depends on factors such as the target benchmark, adaptation method, and network depth.
Our work is closely related to Abnar et al. (2021) where the authors explored how different upstream training settings affect the upstream and downstream accuracy for two upstream datasets and more than 20 downstream tasks. They showed that as the upstream accuracy increases, the transfer learning performance on downstream datasets saturates. However, the authors study only upstream models that are pre-trained with a supervised loss function on ImageNet-21K (Deng et al. 2209) or JFT- 300M (Sun et al., 2017). In this work, we extend these results to more pre-training datasets and methods. Moreover, we consider full fine-tuning in addition to few-shot transfer.
EXPERIMENTAL SETUP
Model. The main focus of this study is the CLIP model (Radford et al., 2021 This model has demonstrated unprecedented robustness natural distribution shifts (Taori et 2020 Miller et al.. 2021), and transfers well to many downstream tasks (Radford et al.) 2021] Wortsman et al. 12021) Given an image-text pair, CLIP learns a joint embedding space for both images and their captions and tries to maximize the cosine similarity between the text and image embedding for an image relative to the cosine similarity of unaligned pairs. We use the CLIP implementation from the OpenCLIP GitHub repository (Ilharco et al. (0211)
Pre-training. We mainly use ResNet-50 (He et al., 2016) as the image encoder unless stated otherwise. We vary the pre-training data distribution, curation method in Section 4.1 and pre-training dataset size in Section 4.2 to obtain different pre-trained models. We also change the contrastive loss function to SimCLR in Section 4.3 test the effect of the pre-training method on downstream transfer accuracy. Further training details are in Appendix A
Fine-tuning. For most of the experiments finetune the pretrained model end-to-end on the target transfer dataset unless otherwise stated. For each pre-trained model and downstream transfer dataset, we used large grid search over various fine-tuning hyperparameters including learning rate, batch size, and the number of epochs. We report the best-performing accuracy in the plots. Further training details are Appendix AA]
Datasets. Our scage experiments yield more than 1000 trained networks. These experiments consist of pre-training on 7 datasets and fine-tuning on 6 downstream tasks. Our pre-training datasets include:
YFCC: Our experiments mostly include YFCC-2.7M, random subset of YFCC-15M. The 15M subset of the VFCC-100M. dataset (Thomee et al. 2016) was filtered to only include images with English titles or descriptions. The dataset contains 14,829, 9,996 images with natural language captions associated with each image. The images and captions are collected from Flickr.
LAION (Schuhmann et al.) 2021): The images and corresponding alt-texts come from web pages collected by Common Crawl (Com) between 2014 and 2021. We randomly select subset of 2.7M and 15M samples for our experiments.
Redcaps Desai et al. 12021): Redcaps contains 1 11882,403 examples from 350 manually curated subreddit collected between 2008 and 2020. The subreddits are selected to contain large number of image posts that are mostly photographs and not images of people.
Conceptual Captions-3m (Sharma al. 2018): The raw descriptions in Conceptual Captions are harvested from the alt-text HTML attribute associated with web images. This dataset contains 2,799,553 samples, denoted CC 2.7m in the plots.
4

---

Under review as a conference paper at ICLR 2023
Conceptual Captions- 12m (Changpinyo et al. 2021): y: A dataset with 12 million image-text pairs. It is larger than CC.2.7m and covers a much more diverse set of visual concepts. We randomly select 2.7M samples from this dataset, denoted CC 12 2.7m.
WIT (Srinivasan et al. 2021): Image-text pairs come from Wikipedia pages. We use reference description as the source of text data and obtain 5,038,295 examples in total after filtering to include only the English language.
Shutterstock: 1 11,800,000 images and captions were crawled from the Shutterstock website in 2021, We randomly sample this dataset.
For downstream tasks we the nine different datasets (CIFARION 100 (Krizhevsky al. 2009).
DTD (Cimpoi et al.] 2014), Caltech-101 (Fei-Fei et al. 2004), PETS (Parkhi et al. 2012), REAL and CLIPART from DomainNet (Peng et al.) (019), EuroSAT (Helber al. (2019), Cassava Leaf Disease Classification (Cas), and Caltech Camera Traps-20 (Beery et al. (0188). See Appendix Section B and Table 3 for details on target transfer datasets.
EXPERIMENTS
4.1 EFFECT OF PERTAINING DATA SOURCE
In this section, we study the role of the pre-training data distribution. We do so by first pre-training on various noisy pre-training data sources and comparing downstream transfer accuracy. Next, we test the effect of pre-training on the curated dataset ImageNet Deng et al. (2009).
What is the role of the pre-training data source in transfer learning? For a controlled experiment to study the effect of the dataset on the downstream transfer accuracy, we fix the learning algorithm to CLIP, architecture to ResNet--50 He et al. (2016), and the number of images to 2.7 million. We get 2.7 million images by randomly subsampling the original datasets. Figure compares different data sources for pre-training. We make the following observations in Figure I on the effect of the pre-training data distribution on the downstream transfer accuracy:
(I) Changing the pre-training dataset leads to noticeable differences in downstream transfer per- formance. However, these differences are most pronounced in the low-shot setting. When many images available for finentuning, the difference in absolute accuracy between different pre-training models is largely diminished, For CIFAR 100, REAL, and CLIPART the model fine-tuned on the full dataset has very similar transfer performance despite different pre-training datasets. While this is not the case for DTD, CALTECH101 and PETS, these datasets have far fewer images per class available when fine-tuning on the full dataset for the downstream task, hence the difference between different pre-train datasets is still noticeable.
(2) We also notice that the ordering of models with respect to their performance on the downstream datasets is largely consistent with Shutterstock and LATON being the best performing pre-training datasets for different downstream tasks. WIT yields the worst performance in most cases.
(3) While transfer learning from large pre-training dataset outperforms training from scratch for all downstream tasks, the magnitude of the improvement varies for different datasets. For example, we observe a large improvement for PETS and CLIPART and a smaller effect of pre-training for REAL.
Do well-curated pre-training datasets lead to better transfer? There has been a significant effort to create computer vision datasets with high-qualit images and labels. On the other hand, many recent datasets are large but noisy. In this set of experiments, we are going to answer two specific questions: How much is ImageNet labeling worth? ? Is there anything special about ImageNet data distribution?
In Figure 2 we first start by pre-training ResNet-50 on ImageNet- IK using supervised cross-entropy loss and finetune the set of our target tasks as baseline. While we want to see the effect of curation, we then discard ImageNet labels and use CLIP to pre-train on ImageNet. Because the ImageNet dataset has captions,
include original Flickr captions, which reduces the size of the image and captions to 0.5M samples (more details in Appendix section C). We observe that supervised pre-training on ImageNet outperforms CLIP pretraining ImageNet with Flickr captions
5

---

Under review as a conference paper at ICLR 2023
Finetune actumny [N]
Finstune accuncy [9].
Finitune actiniy
Ducasets
40
40
LAKON,
20
20
POO mike,
An pepiiord
Mumber
Number
Number shots
(a) CIFAR 100
(b) DTD
(c) CALTECHIO1
300
100
100
80
80
00
Finstune actuinc (16)
Fineune actuinyy [16]
Finstune acturcy (18)
20
Number of shots
Number of shots
Number of shots
(d) PETS
(e) DomainNet REAL
(f) DomainNet CLIPART
Figure 2: Efffec of data curation. Pre-training on curated dataset leads better transfer accuracy than pre-training on a noisy dataset of similar size. However, pre-training on noisy dataset that is 15x larger overall performs similarly pre-training on curated dataset, especially in the high-shot transfer regime.
by a large margin all target tasks. We then use all the images from ImageNet paired with templated captions, e.g., "a photo of a class name". This allows us to have fair comparison between supervised and CLIP pre-training on ImageNet, given the same size. We observe that pre-training with new captions improves the performance of CLIP pre-training by a large margin, defeating supervised pre-training on CIFARI00. However, supervised pre-training on ImageNet still performs best for the rest of the five datasets.
we also compare ImageNet distribution with LAION Figure 2 Pre-training CLIP on LAION- Im performs similarly to ImageNet with Flickr captions(O.5mm. Interestingly for contrastive CLIP pre- training, we need 15x more data from LAION to match the performance on ImageNet with CLIP captions.
We also train CLIP models on subsampled versions of the large noisy LAION dataset of the same size 0.5m and Im. Moreover, we test the effect of pre-training on a lot of noisy data by also pre-training on 15m samples from LAION.
Figure highlights the impact of data curation the quality of the pre-trained models with respect to their transfer accuracy on downstream tasks, We observe:
(1) If we fix the number of samples to 0.5m, LAION-0.5m shows the worst performance for all downstream tasks. Comparing LAION-0.5m with INIK-Flick-Captioon shows that pre- training on clean, curated images and their associated (possibly noisy) captions improves transfer accuracy.
(2) Pre-training a CLIP model on a 2 times larger yet noisy dataset LAION- Im can outperform the NIK--Flick-Capptions yield very similar performance.

---

Under review as a conference paper at ICLR 2023
Finhune acounity (60)
11
Finsune acounnty [16]
Finstune acturacy [66]
80
40
Pretraining models "TOC2
23
20
LNDMNN
MI
Number shots
Number shots
Number
(a) CIFARIO0 100
(b) DTD
(c) CALTECHIOI
Figure 3: Effect of the pre-training dataset size. Increasing the size of the dataset used for pre- training results in better transfer accuracy on downstream tasks. However, the absolute accuracy difference is smaller in the high-shot regime, even when pre-training consists of 100x more data. Still, using an extremely large pre- pre-training dataset of LAION-2B results in significantly better accuracy on CIFAR100 100 and DTD.
(3) Next we study the effect of clean images and captions by comparing LAION- with INIK- CLIP-Captions. We observe that INIK-CLIP-Captions results better downstream transfer accuracy all transfer datasets.
(4) However, pre-training CLIP model on 15 times larger yet noisy dataset LAION-15m can outperform the I-IK-Flickr Li--Flick--Capttiin Captions on CIFARI00, and yield very similar performance on the remainder of the datasets.
(5) Similar to the findings in Figure we observe more noticeable differences in terms of the absolute accuracy difference in the low-shot regime. In the high-shot regime, the difference in absolute accuracy between the different pre-training datasets is smaller.
4.2 EFFECT OF PRE-TRAINING DATASET SIZE
To study the effect of the pre-traing pre-training dataset size on downstream transfer accuracy, we investigate different subsample sizes for the YFCC and LAION datasets. Specifically, we fix the pre-training distribution to YFCC and LAION and compare pre-training on 2.7m samples with 15m samples. We also extended our experiments see the effect of extreme sample sizes and include ViT-B/32 CLIP model trained on 2B samples from LAION, The results are presented Figure following observations:
We make
(1) Increasing the size of the dataset used for pre- pre-training results in higher downstream transfer accuracy. However, the magnitude the improvement varies across different downstream datasets. Increasing the size of the datasets subsampled from YFCC and LAION improves the downstream performance on CIFAR100 by large margin. However, the improvement is more modest for DTD and CALTECHI0L.
(2) Similarly to the findings Figure I we observe more noticeable differences in downstream transfer performance in the few-shot regime when fixing the dataset and varying the number of samples. The difference in the absolute accuracy when more data available fine-tuning usually smaller.
(3) LAION pre-training outperforms YFCC pre-training on downstream accuracy on CIFAR DTD, and CALTECH-Intereste larger size of YFCC- 15m pre-training.
on these tasks LAION-2,77 performs similarly to much
(4) Using an extremely large dataset of LAION-2B improves the performance by a significant margin in the low-shot regime for 100. While differences in absolute accuracy less when more data is available for fine-tuning, LAION-2B pre-training still performs consistently better. Moreover, the differences DTD and CALTECH101 are overall much smaller.
7

---

Under review as a conference paper at ICLR 2023
Fiituee actunity (b6)
Finelune aczuracy [9]
Fineuue accuracy [96]
Number shots (a) CIFAR100 100
Number shots (b) DTD
Number
(c) CALTECHIO!
Figure 4: Effect of the pre-training data distribution when using SimCLR as the pre-training method. Using different datasets for pre-training leads noticeable difference in downstream transfer accuracy. Similar to the previous results for CLIP pre-training, the absolute difference in downstream transfer accuracy between different pre-training datasets is smaller when many images are available for fine-tuning.
4.3 EFFECT OF PERTAINING LOSS
In this section, we investigate the effect the method used for pre-training on downstream transfer accuracy. Therefore, we change the pre-training loss from language- image contrastive (CLIP Radford it al.] (222I) to image-only contrastive (we use SimCLR (Chen et al. (2200). In contrast to previous experiments with CLIP where we fine-tuned end-to-end from the zero-shot pre-trained model, here we fine-tune using LP-FT (Kumar et al., (022)). We do this because we are no longer able to start with a zero-shot pre-trained model. When we compare to CLIP, we fine-tune both models with LP-FT to facilitate a fair comparison.
LP-FT the following two-step procedure: for each number shots we first freeze encoder and train classification head from random initialization using k examples per-class from the downstream task. In the second step, we initialize the classification head with this linear probe (LP) then unfreeze all weights and finetune (FT) the whole model.
Figure shows the results changing the pre-training dataset when SimCLR is used as the pre- training method. We make the following observations:
(1) Similarly Figure changing the pre-train dataset leads to differences in the downstream transfer performance. However, the differences in absolute accuracy are smaller when many images are available for fine-tuning.
(2) We previously observed that for CLIP, Shutterstock led better transfer accuracy in most settings. However when pre-traaee training with SimCLR, the Redcaps datasets results in higher downstream transfer accuracy.
Next, we test the difference between CLIP and SimCLR pre-training. The results are presented in Figure For a fair comparison, we finetune both CLIP and SimCLR models using LP-FT (Kumar et al. 2022). We summarize our findings below:
(1) Overall find that models pre-trained with SimCLR have better downstream transfer accuracy than models pre-trained with CLIP in the low-shot regime.
(2) Similarly to our observations regarding the effect of the pre-training data distribution (Figures and 4. the absolute accuracy difference is smaller when more data is used for fine-tuning. We note that this different from what was observed by (Santurkrr et al. (2022). However, we suspect this difference is because only a linear classifier. We are interested in modifying all parameters as this results in higher accuracy.
fine-tuning all model parameters while they consider
8

---

Under review as a conference paper at ICLR 2023
Finture accuricy (60)
Fietuee acturntyy [b6]
Finsuue acturacy [66)
50
80
Dataser:
award Senbux
Shure pertessoooi reviuue
Sancur
Reccup sie SancuR
30
Mercape tevi
Number shots (a) CIFAR 100
Number shots
Number
(b) DTD
(c) CALTECHI01
Figure 5: CLIP vs. SimCLR for pre-training. Overall we observe that SimCLR pre-training leads better downstream transfer accuracy than using CLIP pre-training on the same dataset. These differences are most substantial in the few-shot setting.
(3) The difference in downstream transfer accuracy for CLIP and SimCLR pre-training varies across different datasets. While SimCIR is only marginally better than CLIP for CIFAR 100, the difference is significantly larger for DTD and CALTECHI01, especially in the low-shot setting.
DISCUSSION, LIMITATIONS, AND FUTURE WORK
Discussion. As better pre-trained models become available, and more workloads shift from training from scratch to fine-tuning, understanding the transfer learning paradigm becomes increasingly important. Presumably, in the future, of pre-trained models will be available for download from the Internet. Therefore, researchers and practitioners will be faced with the question where to begin. It will be important to make this choice well, but also to understand to what extent this choice matters.
Overall have observed that different pre-training distributions and methods can lead differences in downstream transfer accuracy. However, these differences are largest in the few-shot transfer regime, and once many images are used for fine-tuning these differences are mostly diminished. Morcover, while different pre-training decisions lead to similar accuracy in the high-shot regime, they still outperform training from scratch in the setting we consider.
Limitations and Future work There are a number of limitations in our study. For one, we consider only end-to-end fine-tuning. We choose this because it the method of fine-tuning that produces the highest accuracy. However, if compute is limited one may choose to instead use only linear probe or other more lightweight methods of fine-tuning. So far this
addressed in our study.
Another limitation is that we do not do an exhaustive hyperparameter sweep for pre-training, While fine-tuning cheaper and we are therefore able do grid search, for pre-training we are mostly limited to using existing checkpoints. While we feel as though this reflects realistic setting, in future work our goal better understand the role hyperparameters.
In addition to mentioned limitations, future works might include understanding why supervised pre-training on smaller but well-curated ImageNet shows superior performance. The main question what is special about ImageNet- K and studied downstream tasks.
REPRODUCIBILITY AND ETHICS STATEMENS
We strongly believe in the reproducibility of results and therefore included the details of used datasets in the section 3 We also included the details of the pre-training and fine-tuning in section and Appendix section

---

Under review as a conference paper at ICLR 2023
We believe that our submission raises questions regarding the Code of Ethics public datasets and implementations. Our study also does not involve human subjects, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, diserinination concerns, privacy and security issues, legal compliance, and research integrity issues. We also referred to close related works and compare results wherever applicable.
open
REFERENCES
Makerere University Al Lab. Cassava leaf disease classification, 2021. https://ww..kgglee cessed: 2022-10-20.
htt.:///n.....aaaaaaaaaaaaaaaaaa wtt.://aaaa..aa...oooooeeee.aaaa Ac-
Common crawl. cettps://comooncrwvl...
Accessed: 2022-09-20.
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint arXiv:2110.02092 2021.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference computer vision (ECCV), pp. 456-473, 2018.
Daniel Bolya, Rohit Mittapalli, and Judy Hoffman. Scalable diverse model selection for accessible transfer learning, Advances in Neural Information Processing Systems, 34:19301-19312, 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the IEEF/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp-
1597-1607. PMLR, 2020.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111.11431 2021.
Aditya Deshpande, Alessandro Achille, Avinash Ravichandran, Hao Li, Luca Zancato, Charless Fowlkes, Rahul Bhotika, Stefano Soatto, and Pietro Perona. linearized framework and new 2021.
benchmark for model selection for fine-tuning. arXiv preprint
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5414-5423, 2021.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre- training (clip). arXiv preprint arXiv:2205.013397, 2022.
10

---

Under review as a conference paper at ICLR 2023
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesiam approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2103.01983 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-2128. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL /ttttt://////ttt ort.10...2222202 If you use this software, please cite it as below.
01todd.55433
Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, and Rogerio Feris. A broad study on the transferability of visual representations with contrastive learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
8845-8855, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling visual and dision-languag representation learning with noisy text supervision. International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint urXiv:2001.0880 2020.
Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko. A broad study of pre- pre-training for domain generalization and adaptation. arXiv preprint arXiv:2203.11819 2022.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision, pp. 491-507. Springer, 2020.
Simon Kornblith, Jonathon Shiens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 2661-2671, 2019.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 and cifar-10 (canadian institute for advanced research), 2009. URL http:://www..tt cs toronto. wtt.ronto.eeao/kkiiiiiiiii html MIT' License.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang, Fine-tuning can distort pretrained features and underperform oftttf--distibutiinn arXiv preprint arXiv:2202.10054. 2022.
11

---

Under review as a conference paper at ICLR 2023
Iyya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0390 2016.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint a 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits weakly supervised pretraining. In Proceedings of the European conference computer vision (ECCV), pp. 181-196, 2018.
John Milleer, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference Machine Learning, pp. 7721-7735. PMLR, 2021.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. arXiv preprint arXiv:21 12750, 2021.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning, pp. 7294-7305. PMLR, 2020.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. arXiv preprint arXiv:2208.0533 055116 2022.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Hicu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh- Thang Luong, Mingx- ing Tan, and Quoe V Le. Combined scaling for zero-shot transfer learning. arXiv preprint 1xXi:2211110001 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems, 32, 2019.
Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is cap- tion worth thousand images? controlled study for representation learning. arXiv preprint arXiv:2207.07635. 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarzzyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXhv:2111.021144 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565, 2018
12

---

Under review as a conference paper at ICLR 2023
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia-baad based image text dataset for multimodal multilingual machine learning. Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 2443-2449, 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 13:18583-18599 2020.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 100c:00m: The new data in multimedia research. Communications of the ACM, 5922)647-7, 2016.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-opes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv preprint urXiv:2109.01903, 2021.
https::/aarkt www./oggg//////oo//o/////ee attp//bs//2100 220.. 2109.01903
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme:
Practical assessment of pre-trained models for transfer learning. In International Conference on Machine Learning, pp.
12133-12143. PMLR, 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 19909.0440 14.0677,7 2019.
Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18123 18123-18133, 2022.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
A TRAINING DETAILS
A.1 CLIP TRAINING
Our CLIP models are trained from scratch on each of the pre-training datasets unless otherwise mentioned and follow the training code from the OpenCLIP GitHub repository[Iharco et (021). CLIP models are trained using Adam W optimizer (Loshchilov & Hutter 2017) with default Py Torch parameters B1 0.9, B2 0.999. e 10 batch size 1024, and weight decay of 0.1. For learning rate, we start with a learning rate of 10 10-3 and apply a cosine annealinn learning rate schedule (Loshchilov & Hutter 2016) with 5,000 steps warm-up. We use the same data augmentations as Raddoodrd et 2021).
A.2 SIMCLR TRAINING
Our SimCLR implementation closely follows the training code from the SLIP et al. 2021)
SimCLR models also trained for 16 epochs from scratch using AdamW optimizer (Loshchilov & Hutter 2017) with 31 = 0.9, 3g = 0.98, e = 10-$, batch size 1024, and weight decay of 0.1. we start with a learning rate of 10 3 and apply cosine-annealing
learning rate schedule (Loshchilov & Hutter 2016) with 2 epochs of warm-up. The hidden dimension of SimCLR MLP projection head is set to 4,094 and the output embedding dimension of MLP projection head is set to 256.
A.3 FINETUNING DETAIL.
Each pretrained model is finetuned on the specific downstream task for 128 epochs while the learning rate is from 0.0001, 0.0003. 0.001, 0.003 as starting and applying cosine-annealing learning rate schedule (Loshchilov & Hutter 2016) with 500 steps warm-up and batch size of 128. For each fine-tuning, we choose the best performing result on the test set among the performed grid search. We use the implementation from the WiSE-FT GitHub repository for fine-tuning, where we have only one model and a [Wortsman et al. 2021).
B EFFECT OF THE PRE-TRAINING DATA DISTRIBUTION
Here we extend the experiments in section 4.1 include three more downstream datasets. While the first six datasets in Figure I are internet-crawled datasets, these three new downstream datasets are domain-specific, i.e., the dataset created after a specific challenge defined in specific domain.
The task is to classify land use and land cover based on Sentinel-2 satellite images. The dataset covers 13 spectral bands and consists of 10 classes within total of 27,019 labeled and geo-referenced images. we create an 80%-20% random class-balanced split with the provided dataset.
EuroSAT (Helber et al. 2019):
Cassava Leaf Disease Classification (Cas): Kaggle competition, to give farmers access to methods for diagnosing plant diseases. The images are labeled healthy one of four different diseases. we split the dataset with 80%-20% random blass-balanced ratio for train and test, respectively.
The dataset contains 21,397 images from the
Caltech Camera Traps-20 (Beery et al., 2018): CCT-20 contains 57,864 images in 15 classes, taken from camera traps deployed to monitor animal populations. Classes are either single species e.g., "Coyote") or groups of species, e.g., "Bird"). CCT-20 is a subset of the iWildCam Challenge 2018, whose yearly editions have been hosted on Kaggle. Here we study the subset CCT-20 that come from the same locations including 14,07 and 16,395 images for train and test respectively.
Figure 6 compares different data sources for pre-training. While Shutterstock shows superior performance the first six datasets (except for PETS), the best pre-training distribution changes
"CCss" in the main dataset refers images from locations seen during training, and "trans" refers to new locations seen during training
14

---

Under review as a conference paper at ICLR 2023
100
90
190
80
Finelune ROCuracy [N6
Finstune accuaacy [W]
Finstune acturacy (N)
60
60
60
Pre-sah Dataseta: cattee
40
80
40
20
20
20
Siute maddoa 2m
Serach
203
300
200
Number (41
20
Number shots
(a) CIFAR100 100
(b) DTD
(c) CALTECHI01
100
Fintuue accunscy (6)
Fineuue accuney [9]
Finetune ancunity [N]
[30)]
Iall (260)
kall (200)
Number of shots
Number of shots
Number shots
(d) PETS
(e) DomainNet REAL
(f) DomainNet CLIPART
100
90
80
Finetune acturscy [Ng
Finatune actunny [N]
Finetune aosuracy (N8
80
80
erain
40
6
80
ViCIt
[2/00]
20
lul (2
(g) Camera Traps
(h) Cassava Leaf
EuroSAT
Figure 6: Effect of the pre-training data distribution. We extend Figure 1 to include three more downstream datasets of Camera Traps, Cassava Leaf, and EuroSAT. In the low-shot setting, different pre-training datasets lead to noticeable differences in downstream transfer performance. If many samples are available for fine-tuning, the difference absolute accuracy between models pre-trained on different sources largely evaporates.
between Camera Traps, Cassava Leaf, and EuroSAT. Changing the pre-training dataset leads to noticeable differences in the downstream low-shot transfer performance of nine datasets.
C EFFECT OF DATA CURATION: IMAGENET CAPTIONING
We compare CLIP models pre-trained on LAION with CLIP models pre-trained on the following two versions of the curated ImageNet dataset:
INIK-Flickr Alik--iiik:-aaptiinn:: This is a subset of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 training set, paired with the original image title, description, and
15

---

Under review as a conference paper at ICLR 2023
shot
Shots
shons
cc3m
OC129
#edrope2
2
C
finctuning
Scrotch
CCND
72.82
57
33
65
83
70
Ne0tape TM
14
1F0C2
21
3
3
3
55.39
95.20
CLEMATT
Coreestvre
CEROUE
010
CLENIT
CA CR190O
Figure 7: Effect of pre-training data distribution: a better view. We change the presentation of Figure for a better view of exact performance numbers on different data distributions and datasets.
tags from Flickr. Therefore, we can use it for CLIP pre-training. To construct this dataset, Fang al. (2022) start from 14.197.122 image URLs in the ImageNet fall 2011 release, and filter to only include images from Flickr. Next, they restrict the images to the 1,000 classes included in the 2012 ImageNet competition, run the image deduplication routine, and remove text containing profanity. As a result, the dataset of 463,622 images is left along with the newly obtained corresponding text data.
BEE-CLIP--Capppn This dataset includes all data in the ImageNet dataset, paired with templated captions, e.g.. "a photo of a classname". This allows us to use CLIP pre-training but clean images and text. In terms of ImageNet accuracy, this training scheme is very similar to standard supervised training. However, this is now a controlled experiment as we are always using CLIP pre-training.
D EFFECT OF THE PRE-TRAINING DATASET SIZE
in Figure 8 we extend the experiments in include three more downstream datasets of PETS, REAL, and CLIPART. Similar to Figure wC observe that increasing the size the dataset used for pre-training results in higher downstream transfer accuracy. However, the magnitude of the improve- ment varies across different downstream datasets. While increasing the size of the datasets improves the downstream performance more modest for PETS and REAL. Similar to Figure 3 we also observe that LAION pre-training outperforms YFCC pre-training on downstream accuracy. On these tasks, LAION-2,7m performs similarly to much larger size of YFCC- pre-training.
CLIPART(and CIFAR 100FAR100 by large margin, the improvement is
E OTHER ARCHITECTURES
In order to see the effect of architecture on the observed trends, we extend the results to the effect of pre-training distribution in Figure to include Vison Transformers. To do so, used ViT-B/32 released checkpoints trained on LA1ON-400m and OpenAI-400m, Figure showw the effect of data distribution finetune transfer to CIFAR100, DTD, and CALTECH101 when using ViT instead of
16

---

Under review as a conference paper at ICLR 2023
Finhune acouncyy [60
11
Finsune acounnty [16]
Finstune acturacy [56]
80
40
Pretraining models 21OC2.7iCLB
23
20
INDMNN
Number
Number shots
Number
(a) CIFAR 100
(b) DTD
(c) CALTECHIO1
50
10
80
Finelune acturacy (9]
Fiotune anouanry [N6
(aouraay (60)
60
50
50
Prearahning models "Poc2
40
%0
20
Number of shots
Number of shots
Number shots (f) CLIPART
(d) PETS
(c) REAL
Figure 8: Effect of the pre-training dataset size. Increasing the size of the dataset used for pre- training results in better transfer accuracy on downstream tasks. However, the absolute accuracy difference is smaller in the high-shot regime, even when pre-training consists of 100x more data. The benefit of pre-training on LAION-20 is different on target tasks. While there is major gap between LAION-2B and LAION-155 for CIFAR100, the performance on CALTECH101 saturated.
ResNet-50. While similar to Figure the difference between the fine-tune performance is minimal, we observe that both models perform also very similar in the few-shot setting. We hypothesize that this observation could be attributed to the similarity between LAION and Open distributions rather than employing transformer instead of ResNet-50. A controlled study may include replicate Figure 1 but with ViT, and we leave that for future work.
F EFFECT OF PRE-TRAINING DATA DISTRIBUTION: SIMCLR INSTEAD OF CLIP
We extend the set downstream task from three datasets in Figure to six in Figure 10 Findings in Figure 4 still hold true for other downstream tasks. We also observe that similar to using CLIP, Redcaps using SimCL R shows superior performance on PETS.
G DATASETS
G.1 PRE-TRAINING DATASETS
Our study covers 7 pre-training datasets including (YFCC Thomee et al. 2016). LAION Schhhmann et al. 2021), Redcaps Desai et al. (2021). Conceptual captions-3m (Sharma et al. 2018) Conceptual captions- 12m (Changpinyo et al.) 2021), Shutterstock, ImageNet-Caption (Fang et al. 2022 )
Table | shows their main source and total size. We also show some examples of image-caption pairs randomly selected from Shutterstock in Figure [1] Redcaps in Figure 12] YFCC-15m in Figure 13
17

---

Under review as a conference paper at ICLR 2023
100
90
Finetune accuracy [N]
Fansuune accuracy any [W]
50
Finetune accuracy [N]
30
LAOR-400r
D44-00m
LAKO--000
Openn
Opeantation
Openna
Mr (30)
Number shats
Number shots
Number shots
(a) CIFAR 100
(b) DTD
(c) CALTECHIO1
Figure 9: Effect of the pre-training data distribution: ViT instead of ResNet-50 While similar Figure I the difference between the fine-tune performance minimal, we observe that both models perform also very similarly in the few-shot setting. We hypothesize that this observation could be attributed the similarity between LAION and OpenAl distributions rather than employing transformer instead of ResNet-50.
74
90
Finture ancuraey (16)
Finelune rocuracy 1966
Fistune anturayy [96]
60
62
78
10
20
Ai
20
Mi
10
20
Mi
Number of shots
Number of shots
Number of shots
(a) CIFAR100
(b) DTD
(c) CALTECHIOI
75
Freeure MCDIRRIC (
(ncuracy [[6]
Fineune accunscy
56
60
50
25
Number shotss
Number shots
Number of shots (f) CLIPART
(d) PETS
(c) REAL
Figure 10: Effect of the pre-training data distribution when using SimCLR the pre-training method. Using different datasets for pre-training leads to noticeable difference in downstream transfer accuracy. Similar the previous results for CLIP pre-training, the absolute difference in downstream transfer accuracy between different pre-training datasets is smaller when many images available for fine-tuning.
LAION- 11m Figure 14, Conceptual Captions in Figure 13 and WIT in Figure 16 shows the most common words in captions of these pre-training datasets.
Table 2
also
18

---

Under review as a conference paper at ICLR 2023
Looking at Redcaps samples in Figure 12 and also the top 20 captions shows many samples of animals. This showing why Redcaps perform better on PETS. Samples from WIT in Figure 16 and also its top 20 words mostly featuring geographical locations, which is rare in downstream task, hence performing worst compared to other pre-training distributions. Shutterstock top 20 words also include words like "pattem", "texture", "and design" which close to DTD classes, hence showing superior performance in this downstream task.
Dataset
Source
Total size
YFCC LAION CC-12M RedCaps
Flickr
14,826,000
Common Crawl
15,504,742
Unspecified web pages
9,594.338
Reddit
11,882,403
WIT
Wikipedia
5,038,295
ShutterStock
ShutterStock ImageNet
11,800,000
I-
463,622
Table 1: Details on pre-training datasets
Pre-training dataset
Top 20 words in 1M sample of captions
Shutterstock
background, vector, illustration, design, icon, pattern, texture, style, woman, concept, hand, color, flower, view, template, line, business, logo, card, symbol day, today, year, time, cat, plant, friend, anyone, picture, baby, guy, week, dog, home, morning, night, month, way, boy, work photo, day, park, street, city, picture, view, time, world, year, house, state, center, part, garden, shot, image, building, road, museum photo, stock, image, black, woman, design, set, vector, white, print, home, men, blue, dress, art, card, sale, gold, bag, cover illustration, stock, art, design, photo, image, background, room, vector, house, home, woman, wedding, style, photography, royally, car, fashion, girl, world background, actor, artist, player, illustration, view, woman, man, football, leam, tree, premiere, city, vector, day, girl, beach, game, hand, people view, church, station, map, house, building, hall, museum, city, location, street, park, river, state, john, county, town, center, bridge, world
Redcaps
YFCC-15m
LAION-15m
CC-12m
CC-3m
WIT
Table 2: Most common words in captions of pre-training distributions
G.2 DOWNSTREAM TASKS
We measure the transfcrability of learned representations using different methods on Cl- FARI00 (Krizhevsky al. 2009), DTD (Cimpoi al. 2014), Caltech- 101 (Fei-Fei et al. 2004). PETS (Parkhi et al. 2012), REAL and CLIPART from DomainNet (Peng et al. 2019).
19

---

Under review as a conference paper at ICLR 2023
ROME. ITAL, JUNE 13, 2015: 5: Rome hosts popular Pride celebration Rome Gay Pride on Junu 13, 2015, Rome Gay Pride parade takes place on this day, drawing tho...
Abstract Background. Design Tamplote. Modern Pottorn, Vector illustration For Your Design.
Muscular man. Gym characters sport people making exercises bodybullsers posing muscular athlctes
Abstract fractal composition. Magic explosion particles motton Illustration.
Conceptual writing showing Applied, oppased being theoretical applicable.
Baltimore USA silhouette fist design vector illustratiion
BALTIMORE
rocky coast Atlantic ocean in France, Normandy
Happy rich kanelbulle mascot de design carries money bags
Mountain Fuji at top of mountain
Close-up of sliced deli meals with vegetables and beguetle on slate board.
Young woman hitchhiking on countryside road. road. Traveler woman hitchhiking along lonely road Pretty young woman tourist hiking
Infographics with pie chart business presentation trends
glant centiped hicing
Overllakkeeeee Haringvliet
in in the Africa
Green high bain meadow with rocks closeup natural background
Figure 11: Random training samples from Shutterstock
20

---

Under review as a conference paper at ICLR 2023
lofoten archipologo by suur>
foggy night in the vancouver forest
bubba so unbelievably cute when she's sloeping!
duchesse selin wedding guest dress- teaturing bonus pockets!
kids got t-shirts
your condition!
paused the x-men just the right time.
homemaae flammkuchen for dinner...
in field of yellow and green
I'm drunk, and this lucy.
eerie section trail en bong-oorottenn country backroad. long path, calskills park ny
my handsorte new neighbour
up family
from our porch greece
completed small remodel of the half bath. first Limer.
such girl
Figure 12: Random training samples from Redcaps
21

---

Under review as a conference paper at ICLR 2023
Juniper Berries Eastem red ce dar (Juniporus virginlanal laden with berries at the High Line
Kendlesham Forest Suffolk Spider
PISM's analysts On 5 October 2012, in the presence of Council members Directors, inaugurated qu... autumn
Energy Seving 20W CFL bulb mouivalent to incandescent magic.
por Mayer Portugall Thanks your comments
Library. Repairs, www.greatiakesroone
Alaska Trip-Gladie Sitka 1976 Glacier 062 blog here Musings from the Silent Generation Glenn
Nash, North Dakota Nash, North Dakota. From everydot.comv.
Point Mugu State Park On the way to Santa Barbara
Outdbor practice Heikki Karinen teaches how to do makeshift bandaages
dancing monk note the ausience expressions!
Bosts in Porvoo Plus more those cute red storage houses.
paca, was photographed Parama, activated
Paint haunt, Hackney
project You invite...
Chelsea
QLD Police Traffic Branch Commodore with customer!
CHELSEA FOOTBALL CLUB Chelsea Magacine-
bssu 63, November 2009
HOTE
Figure 13: Random training samples from YFCC
22

---

Under review as a conference paper at ICLR 2023
Busy Slaying Vampires Mens T: T-Shirt
Webcam site Stripchat
STRIPCHAT
Yellow sanda for womee poirtly and low Françolse
00200 TIMEPIECES bordeaux croco leather strap
Bedrooms Walton, Liverpool, Merseysi de,
Marriott
Conference Center
Minimum Wage Barbie
Diamond Art Deco 18 Karat White Gold Dangle Drop Earrings
Video editing with laptop. Professional ealtor edding special effects or color grading foolage for co commercial film movie.
BIEE
I'm Rugby Referee - Men's Premiuu Hoodie
Marika Airbrush Yoga Leggings
Julianne Hough: 'Footlocse' Premiere with Kenny Wormald!
JACKEL GAAMESSE
Al MH WD
Collins,
you
Girls Dora Explorer Costume- CiilovennCosuumes-U...k Kids Costumes
SCOMS SHIP SHORE COVERAGE
Figure Random training samples from LAION
23

---

Under review as a conference paper at ICLR 2023
The wolf and the lamb shall feed together, and the lion shall eat straw like the bullork: and dust chall be the serpent's meet. They shell not hurt ne...
PERSONN>
Islamic vector geometric omnaments based on traditional arabic art Oriental seamless pattern. Muslim mosak: Turkish, Arabian
tile on a white backgrourd. Mosque
girl lealthr jacket on black color motorcycle
Light Touch Wall digital markeling activation Canberra Centre.
wedding inspiration brings fabulous gownr crealive designer <PER Affection wedo...
Illustration
collection <PERSON>
Vector Illustration design.
Easy Canbagg Rolls that, <PERSON>, PERSSNN and have no ricel < <PRSSON> budget friendly comfort food recipe adapted from my Russian grandmother!
<PERSON>; <PERSON United States Army first <PERSON> appointed that =ERSONN>
position. first, 6 to serve on joint Chiets of Start Black H...
Wedoing rings on bouquet of roses stock photos
<PERSON= Lattoo, the American number 23 from Akron, United States
Balls Swinging Arm Bearing Kit: for
Search the hidden word. the simple educational kid game. stock illustration
XT Seroo
types photo frames squares background template
army entering Prussia, Photo
Bathmom Makeovers Ideas Small Bathroem Remodel, Modem Bathroom Designs On Budget Minimalist Small Bathrooms, Modem Small Bathrooms, Mo...
The good drinking
Figure 15: Random training samples from Conceptual Captions
24

---

Under review as a conference paper at ICLR 2023
Japanese chamber pot from the Edo period
Rotel
15th
Dedication plaque al Oregon Dures Overlook. Oregon National Recreation
Robinson
Wareham Donald Trump's childhood home
Museum Design Columbus Circle, nearly completed July 2008. piece David Dunlap's the NY Times reveals that the appearance of the letter
Huge der Saint Luke Drawing Virgin, 1470-80. National Museum Ancient Art, Lisbon
Maul Veterans Highway shown just above Kea lia Fond National Willliee Refuge as it anters Kihei
Paul McAuley at Worldcon 2005 Glasgow
Construction restrooms and locker rooms with east side stands and pavilion
McLaren 600LT
praise
performance.
Fiat G 91PAN, service the Frecce Tricolori from 1963 1982
Alexa Stirling, 1919
Figure Random training samples from WIT
25

---

Under review as a conference paper at ICLR 2023
Downstream Task Description
CIFAR100
The task consists in classifying natural images (100 classes, with 500 training images each). Some examples include apples, bottles, dinosaurs, and bicycles. The image size is 32x32.
DTD
The task consists in classifying images of textural patterns (47 classes, with 120 training images each). Some the textures are banded, bubbly, meshed, lined, porous. The image size ranges between 300x300 and 640x640 pixels.
CALTECH-101
The task consists in classifying images objects (9144 images in 101 classes plus a background clutter class), including animals, airplanes, chairs, or scissors. The image size varies, but typically ranges from 200-300 pixels per edge.
PETS
The task consists in classifying images of cat and dog breeds (7000 images in 37 classes), Images dimensions are typically 200 pixels or larger
REAL
The task is a subset of larger DomainNet from six distinct domains, including photos (real), painting, clipart, quickdraw, infograph, and sketch. Total size of 172,000
CLIPART
The task is subset of larger DomainNet from six distinct domains, including photos (real), painting, clipart, quickdraw, infograph, and sketch. Total size of 172,000
Table 3: Details on the downstream datasets used the experiments.
26


Reviews:
[{"review_id": "aiPpgTiGvS2", "reviewer": "Reviewer_qem9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the effect of pre-training dataset on transfer learning, when pretraining by text caption contrastive matching using CLIP.  Several different pre-training datasets are compared, using six transfer task datasets (three in later experiments), and in both few-shot and full-data scenarios.  The experiments are extensive, with a fairly rich set of results from which conclusions might be drawn.  The paper enumerates many interesting points in these comparisons, and has a few main takeaways, including: (a) choice of pre-training dataset matters most for low-data scenarios, (b) smaller but cleaner pre-training data can out-perform noisier captions-based data for datasets up to around 15x smaller, (c) more pretraining data helps, but also mostly for low-data transfer tasks.\n", "summary_of_the_review": "Overall, this paper presents extensive experiments comparing different pretraining data sources.  This is an interesting set of results, but in my opinion under-utilized in the analysis and presentation.  More exact overall conclusions might be drawn than those like \"more data is better\" or \"cleaner data is better\".  The detailed points in each section might be used to help argue for more tangible conclusions, but they aren't clearly stated yet.\n", "strength_and_weaknesses": "The experiments in this paper are extensive, comparing numerous pretraining datasets with different downstream tasks.  This in itself is a useful contribution, particularly if the authors make public the results as a csv file for further analysis, which I would encourage.\n\nHowever, I think the analysis in this paper and its presentation could be clearer and have more material conclusions.  The largest points (a-c) in my summary above are not particularly informative, as they describe general trends, which at best corroborate known behaviors.  The observation that pre-training choices matter most (quite a lot, in fact) for few-shot settings is perhaps the most interesting one of these.\n\nOn the other hand, the paper enumerates many observations describing interesting behaviors in each of the sections --- so some detailed observations are there, but with few conclusions.  For example, at the end of section 4.1, there are 5 enumerated points, which describe many details but few clear conclusions.  Point 2 says \"Pre-training a CLIP model on a 2 times larger yet noisy dataset LAION-1m can outperform the IN1K-Flickr-Captions or yield very similar performance\".  This merely points to the data and restates in words what the plots show, which is fine if supporting a larger argument, but it is left stranded.  What exact conclusion can be drawn from this?  Does this difference support or elaborate on the larger trends observed in this section?\n\nIn addition, most of the figures, while clear, do not always seem the best plots for making the points in the text.  All are organized with number of shots on the x axis, and datasets as point shape/color.  Another view that could work well could be a colored matrix table showing accuracy values as entries, with pretraining datasets on one axis, transfer dataset on the other axis, and transfer training shots as tiled copies of the plot running along the page --- this would make clearer conclusions like 4.1 first sub-section point (2) which says Shutterstock and LAION are best performing, and could also visualize trends in these orderings as transfer shots increase.  There are likely many other figures that could work, and I think more different views could be included to help make more trends apparent.\n\n\n\n\nI also have a few questions on some of the details, enumerated below:\n\n\n* since SimCLR is better than CLIP pretraining on the three transfer tasks, this immediately brings up the question of whether the rest of the pretraining dataset comparisons might be explored with SimCLR as well.  At the very least, this might be mentioned in the future work section, as the findings are still relevant for CLIP training.\n\n* page 7 sec 4.1 point 4:  should this say \"in1k-clip-captions\" instead of flickr?\n\n* fig 3, 4, 5:  why not also compare on pets and domainnet?\n\n* sec 4.3:  Why suddenly switch to LP-FT?  Especially if this makes a difference for simclr vs clip, it would be good to compare with FT only as well.  The rest of the paper used this method, so switching methods appears inconsistent.\n\n* sec 4.3 point 2:  Are there any conclusions or hypotheses about why the best-performing dataset is different between clip and simclr?  Perhaps diversity or relevance of images to the downstream tasks, vs of the captions?\n\n* Figure 4:  only SimCLR results are in these figures, so it's difficult to compare between SimCLR and CLIP --- it would be good to have a figure with both included to facilitate this comparison for sec 4.3\n\n* While the pre-training datasets here do differ in the ways described, such as size or noisiness, they have other differences as well --- for example size of the vocab in the captions, diversity of the objects represented in image data.  Do these affect downstream performance as well, and how might those be controlled for or studied?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LpDwZNQMWt", "reviewer": "Reviewer_bvES", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the effect of the pre-training on transfer learning for image classification. Their study covers 7 pre-training datasets, 6 fine-tuning datasets, 2 pre-training methods for both few-shot and full finetuning. They have made several empirical observations. They find that the pre-training dataset is initially important for low-shot transfer and the importance of a good pre-trained model diminishes when fine-tuning on more data.\n", "summary_of_the_review": "This paper provides empirical observations on the effectiveness of pre-training. However, not much new insights are provided.", "strength_and_weaknesses": "Strength\n- The paper empirically studies the effects of pre-training (data source, pre-training methods, loss function) on downstream tasks. \n- The paper provides fine-tuning results with different pre-training methods (e.g., CLIP based ResNet-50 model), which might be new for practitioners.\n\nWeakness\n- The questions asked in the paper were already well studied and the answer is not surprising. The major take-away conclusion of “the effect of pre-training diminishes as more training data are available” is a well known fact and was shown in literature on transfer learning and model selection such as [1].\n- The downstream tasks are quite limited (only 6) and some of them have quite saturated performance (such as PETS). It is not quite convincing to draw conclusions with these simple datasets.\n- As identified by the authors, there is no explanation or insights about why certain pre-trained models work better than others.\n- Only a single architecture (ResNet-50) is studied. Not clear whether the findings generalize to other architectures as well.\n- [minor] The authors claimed the best performance on the test was picked with variations of learning rates and batch size, however, the appendix said only a single batch size is used.\n \n[1] Deshpande et al, A linearized framework and a new benchmark for model selection for fine-tuning, 2021\n[2] Bolya et al, Scalable Diverse Model Selection for Accessible Transfer Learning, 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "J-AfzD8Xoti", "reviewer": "Reviewer_6HTo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the performance of pre-trained models in downstream classification tasks. ", "summary_of_the_review": "I would suggest a weak rejection in this phase.", "strength_and_weaknesses": "Strength:\n\nThe experiment is extensive and adds more pre-training models and datasets compared to previous papers.\n\n\n\nWeaknesses:\n\nThe paper does experiment with different pre-trained models and datasets in the downtream end, which has already been done in previous papers like [1,2,3]. The paper admits that \"we extend these results to more pre-training datasets and methods\", which is not a substantial contribution to the community according to my criteria. It is better to propose a novel framework to explain the extensive empirical observation. \n\n[1] Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint arXiv:2110.02095, 2021\n\n[2] Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko. A broad study of pre-training for domain generalization and adaptation. arXiv preprint arXiv:2203.11819, 2022.\n\n[3] Kornblith, Simon, Jonathon Shlens, and Quoc V. Le. \"Do better imagenet models transfer better?.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\nThe phenomenon of more downstream data leads to less benefit of pre-training is studies in [4]. Could the author give a discussion on the connection and difference between [4] and this paper?\n\n[4] Ziquan Liu, Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, Xiangyang Ji, Antoni B. Chan, Rong Jin, \"Improved Fine-Tuning by Better Leveraging Pre-Training Data\" Neural Information Processing Systems (NeurIPS), 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9i3WkWhK7w", "reviewer": "Reviewer_thHE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper 5991 presents an empirical study to address the question of what data and method should be used for pre-training. The scope is limited in the pre-training distribution for image classification. To be specific, the paper investigates the impact of dataset size / pre- training method / whether the dataset is curated on transfer learning. The main take-home message is that pre-training dataset is initially important for low-shot transfer but the differences between distributions are diminished as more data is made available for fine-tuning.", "summary_of_the_review": "This paper presents some expected conclusions verified by large-scale experiments.\n\n\n--- Post rebuttal review ---\nI have read reviews and responses from authors and peer reviewers. It seems we are all concerned with the novelty of those results. There are so many suggestions in the review to be incorporated into the paper, which I think is beyond the scope of a revision. I encourage the authors to dive into several specific findings rather than presenting many take-away messages without going into them deeply.", "strength_and_weaknesses": "# Strength:\nA systematic evaluation of the impact of pre-trained data is presented. My educated guess is that this paper requires extensive experiments with massive computation resource. I suggest that authors clearly state or estimate the rough computation resource used for this paper. I suppose curation of those large-scale datasets also contributes largely to the total labor work. In a word, although no novel methods or ideas are presented in this paper, I appreciate the work of verifying common-sense conclusions using large-scale empirical experiments.\n\nSince the authors mention that \"our large-scale experiments yield more than 1000 trained networks\", all the trained networks can be released if possible, which can be considered as another contribution, just like the paper \"Model Zoos: A Dataset of Diverse Populations of Neural Network Models\".\n\n# Weaknesses:\nThe main weaknesses is that conclusions in this paper seem to be known or trivial. It is expected that pre-training dataset is initially important for low-shot transfer but the differences between distributions are diminished as more data is made available for fine-tuning. There are no additional analyses, e.g. on why \"Shutterstock and LAION being the best performing pre-training datasets for different downstream tasks, while WIT yields the worst performance in most cases\".\n\nIn the discussion section, the authors say that \"in the future, a sea of pre-trained models will be available for download from the Internet. Therefore, researchers and practitioners will be faced with the question of where to begin.\" This should be discussed along with the following papers, where they exactly want to address the problem:\nLEEP: A New Measure to Evaluate Transferability of Learned Representations, ICML 2020\nLogME: Practical Assessment of Pre-trained Models for Transfer Learning, ICML 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject