Does progress on ImageNet transfer to real world datasets?
Authors: Alex Fang
Simon Kornblith
Ludwig Schmidt
Venue: Submitted to ICLR 2023 (2023)

Abstract: Does progress on ImageNet transfer to real world datasets? We investigate this question by evaluating ImageNet pre-trained models with varying accuracy (57% - 83%) on six practical image classification datasets. In particular, we study datasets collected with the goal of solving real world tasks (e.g., classifying images from camera traps or satellites), as opposed to web-scraped benchmarks collected for comparing models. On multiple datasets, models with higher ImageNet accuracy do not consistently yield performance improvements. For certain tasks, interventions such as data augmentation improve performance even when architectures do not. We hope that future benchmarks will include more diverse datasets to encourage a more comprehensive approach to improving learning algorithms.

Under review as a conference paper at ICLR 2023
DOES PROGRESS ON IMAGENET TRANSFER TO REAL-WORLD DATASETS?
Anonymous authors Paper under double-bling review
ABSTRACT
Does progress on ImageNet transfer to real-world datasets? We investigate this question by evaluating ImageNet pre-trained models with varying accuracy (57%- 83%) on six practical image classification datasets. In particular, we study datasets collected with the goal of solving real-world tasks (e.g.. classifying images from camera traps or satellites), as opposed to web-scraped benchmarks collected for comparing models. On multiple datasets, models with higher ImageNet accuracy do not consistently yield performance improvements. For certain tasks, interven- tions such as data augmentation improve performance even when architectures do not. We hope that future benchmarks will include more diverse datasets to encourage a more comprehensive approach to improving learning algorithms.
INTRODUCTION
ImageNet is one of the most widely used datasets in machine learning. Initially, the ImageNet com- petition played key role in re- re-popularizing neural networks with the success of AlexNet in 2012. Ten years later, the ImageNet dataset is still one of the main benchmarks for state-of-the-art com-
al.] [2012] SSionnyan & Zisserman, 2015) HH et al.] 2016 Liu iet al. 2018 Howard et al.) 2019: Touvron et al., 2021; Radford et al., 2021). As a result of Ima- geNet's prominence, the machine learning community has invested tremendous effort into develop- ing model architectures, training algorithms, and other methodological innovations with the goal of increasing performance on ImageNet. Comparing methods on a common task has important benefits because it ensures controlled experimental conditions and results in rigorous evaluations. But the singular focus on ImageNet also raises the question whether the community is over-optimizing for this specific dataset.
puter vision models (Krizhevsky
As first approximation, ImageNet has clearly encouraged effective methodological innovation be- yond ImageNet itself. For instance, the key finding from the early years of ImageNet was that large convolution neural networks (CNNs) can succeed on contemporary computer vision datasets by leveraging GPUs for training. This paradigm has led to large improvements in other computer vision tasks, and CNNs are now omnipresent in the field. Nevertheless, this clear example of trans- fer other tasks early in the ImageNet evolution does not necessarily justify the continued focus ImageNet still receives. For instance, it is possible that early methodological innovations transferred more broadly to other tasks, but later innovations have become less generalizable. The goal of our paper is to investigate this possibility specifically for neural network architecture and their transfer real-world data not commonly found the Internet.
When discussing the transfer of techniques developed for ImageNet to other datasets, a key ques- tion what other datasets consider. Currently there is no comprehensive characterization of the many machine learning datasets and transfer between them. Hence we restrict our attention to limited but well-motivated family of datasets. particular, we consider classification tasks derived from image data that were specifically collected with the goal of classification in mind. This is in contrast to many standard computer vision datasets - including ImageNet where the constituent images were originally collected for a different purpose, posted to the web, and later re-purposed for benchmarking computer vision methods. Concretely, we study six datasets ranging from leaf dis- ease classification over melanoma detection to categorizing animals in camera trap images. Since these datasets represent real-worke applications, transfer of methods from ImageNet particularly relevant.

---

Under review as a conference paper at ICLR 2023
Caltech Camera Traps
APTOS 2019 Blindness
Human Protein Atlas
Quadratic weighted kappa
0.930
0.75
76
0.925
Macro scooee
0.70
0.920
0.45
Accuracy
72
70-
0.90
0.50
66-
0.900
0.925
0.40
ImageNet top-1 accuracy
ImageNet top-1 accuracy
ImageNet top-1 accuracy
SIIM-ISIC Melanoma
Cassava Leaf Disease
EuroSAT
0.97
90.4
Area under ROC
0.95
Accuracy
-.7
92.0
0.94
85
00.6
0.83
85-
0.92
40.2
83
70
60
60
15
75
75
80
70
80
ImageNet top-1 accuracy
ImageNet top-1 accuracy
ImageNet top-1 accuracy
AlexNet
ResNet-152
EfficientNet
ConvNext-tiny
MobileNetV3
DelT-smail
EfficientNet
ShuffieNetV2x0.5
VGG
PNASNet-S
DenseNet-121
SqueezeNet
DelT-tiny
Inception-ResNett
ResNexx-50-32xxd
VIT-8116
ResNet-50
VGG-16
SnuffieNetV2x1.0
Figure 1: Overview transfer performance across models from ImageNet to each of datasets we study. Although there seems to be a strong linear trends between ImageNet accuracy and the target metrics (green), these trends become less certain when we restrict the models to those above 70% ImageNet accuracy (blue). Versions with error bars and spline interpolation can be found in Appendix
We find that on four out our six real-world datasets, ImageNet- architecture improve- ments after VGG resulted in little to progress (see Figure D). Specifically, when fit line to downstream model accuracies as a function of ImageNet accuracy, the resulting slope is less than 0.05. The exceptions where VGG architectures yield larger gains are the Caltech Camera Traps-20 (CCT-20) (Beery et al. 2018) dataset (slope 0.11) and the Human Protein Atlas Image Classification (Ouyang al., 2019) dataset (slope 0.29). On multiple other datasets, we find that lask-specific improvements such as data augmentations or extra training data lead to larger gains than using more recent ImageNet architecture. We evaluate on representative testbed of 19 Im- ageNet models, ranging from the seminal AlexNet (Krizhevsky et al. 2012) over VGG (Simonyan Zisserman 2015) and ResNets (He et al. (016) to the more recent and higher-performing Eff- cientNets Tan & Te 2019) and ConvNexts (Liu et al. (022) (ImageNet top-1 accuracies 56.5% to 83.4%). Our testbed includes three Vision Transformer models to cover non-CNN architectures.
Interestingly, our findings stand in contrast to earlier work that investigated the aforementioned image classification benchmarks such as CIFAR-I0 (Krizhevsky Hinton 2009), PASCAL VOC 2007 (Everingham al.) 2010), and Caltech- 101 (Fei-Fei et al. 2004) that scraped from the Internet. On these datasets, Kornblith et al. 2019) found consistent gains in downstream task ac- curacy for similar range of architectures indicate that ImageNet accuracy may be a good predictor for other web-scraped datasets, but less informative for real-world image classification datasets that not sourced through the web. On the other hand, the CCT-20 data point shows that even very recent ImageNet models do help on some downstream tasks that do rely on images from the web. Overall, our results highlight the need for more comprehensive understanding of machine learning datasets to build and evaluate broadly useful data representations.
study in work. Taken together, these findings
RELATED WORK
Transferability of ImageNet architectures. Although there is extensive previous work investigat- ing the effect of architecture upon the transferability of Imaeteet--retrrineed models to different
2

---

Under review as a conference paper at ICLR 2023
datasets, most of this work focuses on performance datasets collected for the purpose of bench- marking. Kormblith et al. (2019) previously showed that ImageNet accuracy of different models is strongly correlated with downstream accuracy on a wide variety of web- web-scraped object-centric com- puter vision benchmark tasks. Later studies have investigated the relationship between ImageNet and transfer accuracy for self-supervised networks (Fricsson al. 2021 Kotar et al. [222]] Nayman it al. 22022), adversarially trained networks (Salman et al.) 2020), or networks trained with different loss functions (Kornblith al. 2021), but still evaluate primarily on web-scraped benchmark tasks. The Visual Task Adaptation Benchmark (VTAB) (Zhai et al. 2019) comprises more diverse set of tasks, including natural and non-nat non-natural classification tasks as well as cooscsassiiination tasks, but nearly all consist of web-scraped or synthetic images. In the medical imaging domain, models have been extensively evaluated on real-world data, with limited gains from newer models that perform better on ImageNet (Raghu et al. 2019 Bressem et al.
2020;
Ke et al. (021).
Most closely related to our work, TTugennrr et al. 2021) investigate performance of 500 CNN archi- tectures on yet another set of datasets, several of which are not web-scraped, and find that accuracy correlates poorly with ImageNet accuracy when training from scratch, but correlations are higher when fine-tuning ImageNet- MmageeNet-pretrainee models. Our work differs from theirs in our focus solely on real-world datasets (e.g., from Kaggle competitions) and in that we perform extensive tuning in order to approach the best single-model performance obtainable on these datasets whereas [Tuggener al.. 2021 instead devote their compute budget to increasing the breadth of architectures investigated.
Transferability of networks trained on other datasets. Other work has evaluated transferability of representations of networks trained on datasets beyond ImageNet. Most notably, Abbar et al. (2022) explore the relationship between upstream and downstream accuracy for models pretrained on JFT and ImageNet-2IK and find that, on many tasks, downstream accuracy saturates with up- stream accuracy. However, they evaluate representational quality using linear transfer rather than end-to-end fine-tuning. Other studies have investigated the impact of relationships between pre- training and fine-tuning tasks (Zamir et al. 2018 Mensink et al. 2021) or the impact of scaling the model and dataset (Goyal et al. 2019 Kolesnikov et al. 2020).
Another direction of related work relates to the effect of pretraining data on transfer learning. Huh eet al.] (2016) look into the factors that make ImageNet good for transfer learning. They find that fine-grained classes are not needed for good transfer performance, and that reducing the dataset size and number of classes only results in slight drops in transfer learning performance. Though there is a common goal of exploring what makes transfer learning work well, our work differs from this line of work by focusing on the fine-tuning aspect of transfer learning.
Other studies of external validity of benchmarks. Our study fits into a broader literature inves- tigating the external validity of image classification benchmarks. Early work in this area identified lack of diversity as a key shortcoming of the benchmarks of the time (Ponce et al.] 2006 Torralba & Efros 2011), problem that was largely resolved with the introduction of the much more di- verse ImageNet benchmark (Deng et al. 2009 Russakovsky et al., 2015). More recent studies have investigated the extent to which ImageNet et classification accuracy correlates with accuracy on out- of-distribution (OOD) data (Recht et al. 2019 Taori et al.] 2020) or accuracy as measured using higher-quality human labels (Shankar et al. 2020 Tsipras et al.] 2020 Beyer et al. 2020).
As in previous studies of OOD generalization, transfer learning involves generalization to test sets that differ in distribution from the (pre-) )training data. However, there are also key differences be- tween transfer learning and OOD generalization. First, in transfer learning, additional training data from the target task is used to adapt the model, while OOD evaluations usually apply trained models to a new distribution without any adaptation. Second, OOD evaluations usually focus on settings with a shared class space so that evaluations without adaptation are possible. In contrast, transfer learning evaluation generally involves downstream tasks with classes different from those in the pre- training dataset. These differences between transfer learning and OOD generalization are not only conceptual but also lead to different empirical phenomena. Miller et al. (2021) has shown that in- distribution accuracy improvements often directly yield out-of -fitt-diistibttion accuracy improvements as well. This is the opposite of our main experimental finding that ImageNet improvements do not directly yield performance improvements on many real- wal-world downstream tasks. Hence our work demonstrates an important difference between OOD generalization and transfer learning.
3

---

Under review as a conference paper at ICLR 2023
3 DATASETS
As mentioned in the introduction, a key choice in any transfer study is the set of target tasks on which to evaluate model performance. Before we introduce our suite of target tasks, we first describe three criteria that guided our dataset selection: (i) diverse data sources, (ii) relevance to an application, and (iii) availability of well-tuned baseline models for comparison.
3.1 SELECTION CRITERIA
Prior work has already investigated transfer of ImageNet architectures to many downstream datasets (Donahue et al.) 2014; Sharif Razavian et al.] 2014] Chathield et al.) 2014 Simonyan & Zisserman 2015). The 12 datasets used by Kornblith et al. (2019) often serve as a standard evalu- ation suite (e.g., in (Salman et al. 2020; Ericsson et al. 2021; Radford et al. (02I)). While these datasets are an informative starting point, they are all object-centric natural image datasets, and do not represent the entire range of image classification problems. There many applications of com- puter vision; the Kaggle website alone lists more than 1,500 datasets as of May 2022. To understand transfer from ImageNet more broadly, we selected six datasets guided by the following criteria.
Diverse data sources. Since collecting data is an expensive process, machine learning researchers often rely on web scraping to gather data when assembling a new benchmark. This practice has led to several image classification datasets with different label spaces such as food dishes, bird species, car models, or other everyday objects. However, the data sources underlying these seemingly different tasks are actually often similar. Specifically, we surveyed the 12 datasets from Kornblith et al.) (2019) and found that all of these datasets were harvested from the web, often via keyword searches in Flickr, Google image search, or other search engines (see Appendix K). This narrow range of data sources limits the external validity of existing transfer learning experiments. To get a broader understanding of transfer from ImageNet, we focus scientific, commercial, and medical image classification datasets that were not originally scraped from the web.
Application relevance. In addition to the data source, the classification task posed on given set of images also affects how relevant the resulting problem is for real-world applications. For instance, it would be possible to start with real-world satellite imagery that shows multiple building types per image, but only label one of the building types for the purpose of benchmarking (c.g.. to avoid high annotation costs). The resulting task may then be of limited value for actual application involving the satellite images that requires all buildings to be annotated. We aim to avoid such pitfalls by limiting our attention to classification tasks that were assembled by domain experts with a specific application in mind.
Availability of baselines. If methodological progress does not transfer from ImageNet given target task, we should expect that, as models perform better on ImageNet. accuracy on the target task saturates. However, observing such trend an experiment is not sufficient reach a conclu- sion regarding transfer because there is an alternative explanation for this empirical phenomenon. Besides lack of transfer, the target task could also simply be easier than the source task so that models with sub-optimal source task accuracy already approach the Bayes error rate. As an illus- trative example, consider MNIST as a target task for ImageNet transfer. A model with mediocre ImageNet accuracy is already sufficient to get 99% accuracy MNIST, but this finding does not mean that better ImageNet models are insufficient to improve MNIST accuracy - the models have already hit the MNIST performance ceiling.
More interesting failures of transfer occur when ImageNet architectures plateau on the target task, but it is still possible improve accuracy beyond what the best ImageNet architecture achieve without target task-specific modifications. In order to make such comparisons, well-tuned base- lines for the target task are essential. If improving ImageNet accuracy alone is insufficient reach these well-tuned baselines, we can indeed conclude that architecture transfer to this target task is limited. In our experiments, we use multiple datasets from Kaggle competitions since the resulting leaderboards offer well-tuned baselines arising from a competitive process.
3.2 DATASETS STUDIED
The datasets studied in this work are practical and cover a variety of applications. We choose four of the most popular image classification competitions on Kaggle, measured by number of com- petitors, teams, and submissions. Each of these competitions is funded by an organization with the goal of advancing performance on that real-world task. Additionally, we supplement these datasets
4

---

Under review as a conference paper at ICLR 2023
Table I: We examine variety of real-world datasets that cover different types of tasks.
Datasel
classes
Truin size
Eval size
Eval metric
Kaggle
Caltech Camera Traps
15
14,071
15,215
Accuracy
APTOS 2019 Blindness
5
2,930
732
Quadratic
weighted kappa
Human Protein Atlas SHIM-ISIC ISIC Melanoma
28
22.582
5,664
Macro score
2
46,372
11,592
Area under ROC
Casuva Leaf Discase EuroSAT
5
17.118
4,279
Accurncy
10
21,600
5,400
Accuracy
Caltech Camera Traps-20
APTOS 2019 Blindness Detection
Human Protein Atlas SII--ISIC Melanoma
Cassava Leaf Discase Classification
EuroSAT
Image Classification
Classification
Figure 2:
Sample images from each of the datasets.
with Caltech Camera Traps (Beery et al. 2018 and EuroSAT (Helber et al. 2019) to broaden the types of applications studied. Details for each dataset can be found in Table
MAIN EXPERIMENTS
We run our experiments across 19 model architectures, including both CNNs and Vision Transform- ers (ViT and DeiT). They range from 57% to 83% ImageNet top-1 accuracy, allowing us to observe the relationship between ImageNet performance and target dataset performance. order get the best performance out of each architecture, we do extensive hyperparameter tuning over learning rate, weight decay, optimizer, and learning schedule. Details about our experiment setup can be found in Appendix [ We now present our results for each of the datasets we investigated. Figure summarizes our results across all datasets, with additional statistics in Table Appendix A contains complete results for all datasets across the hyperparameter grids.
4.1 CALTECH CAMERA TRAPS
We summarize the blue regression lines from Figure [ calculated on models above 70% ImageNet accuracy, with their correlation and slope. Slope is cal- culated that all metrics have range from 0 100.
Table 2:
Beery et al. (0088) created Caltech Camera Traps-20 (CCT-20) using images taken from camera traps deployed to monitor animal pop- ulations. The images contain 15 different ani- mal classes, well
Dataset
Correlation
Slope
empty class that D The dataset con-
Caltech Camera Traps
0.17
0.11
remove for our experiments tains two of validation and test sets which differ by whether they come from locations that are the same as or different from the training set locations. While one of the goals of the dataset to study generalization to new environments, here we only study the sets from the same locations. Although CCT-20 is not Kaggle competition, it is a subset the iWildCam Challenge 2018, whose yearly editions have been hosted on Kaggle.
APTOS 2019 Blindness
0.06
0.01
Human Protein Atlas
0.26
0.29
SII-IIII Melanoma
0.44
0.05
Cassava Leaf Disease EuroSAT
0.12
0.02
0.05
0.00
We see in Figure I (top-left) an overall positive trend between ImageNet performance and CCT- 20 performance. The overall trend is unsurprising, given the number animal classes present in ImageNet. But despite the drastic reduction in the number of classes when compared to ImageNet, CCT-20 has its own set challenges. Animals are often pictured at difficult angles, and sometimes even visible in the image because a sequence of frames triggered by activity all have the same label. Despite these challenges, an even higher performing model still does better on this task train CLIP ViT L/14-336px model (85.4% ImageNet top-1) with additional augmentation achieve 83,4% accuracy on CCT-20.
Dataset download links and PyTorch datasets and splits can be found at REDACTED.
Empty class removed for the classification experiments in Table of Beery et al. 2018
5

---

Under review as a conference paper at ICLR 2023
4.2 APTOS 2019 BLINDNESS DETECTION
This dataset was created for a Kaggle competition run by the Asia Pacific Tele- Oele-Ophthllmoloyy Society (APTOS) with the goal of advancing medical screening for diabetic retinopathy in rural areas (Asia Pacific felc-Ophhhalmoloyy Society 2019). Images are taken using fundus photography and vary in terms of clinic source, camera used, and time taken. Images are labeled by clinicians on a scale of O to 4 for the severity of diabetic retinopathy. Given the scaled nature of the labels, the competition uses quadratic weighted kappa (QWK) as the evaluation metric. We create a local 80% to 20% random dlass-balanced train/valiaation split, as the competition test labels are hidden.
We find that models after VGG do not show significant improvement. Similar to in CCT-20, DeiTT and EfficientNets performs slightly worse, while deeper models from the same architecture slightly help performance. We also find that accuracy has similar trend as QWK, despite it being an inferior metric in the context of this dataset.
When performance stagnates, one might ask whether we have reached a performance limit for our class of models on the dataset. To answer this question, we compare with the Kaggle leaderboard's top submissions. The top Kaggle submission achieves 0.936 QWK on the private leaderboard (85% of the test set) (Xul 2019). They do this by using additional augmentation, using external data, training on I-lss,ss, replacing the final pooling layer with generalized mean pooling, and ensembling variety of models trained with different input sizes. The external data consists of 88,702 images from the 2015 Diabetic Retinopathy Detection Kaggle competition.
Even though performance saturates with architecture, we find that additional data augmentation and other interventions still improve accuracy. We submitted our ResNet-50 and ResNet-152 models with additional interventions, along with Inception ResNeet v2 (Szegedy et al.] (017b) model with hyperparameter tuning. We find that increasing color and affine augmentation by itself can account for 0.03 QWK point improvement. Once we train on 512 input size, additional augmentation, and additional data, our ResNet-50 and Iiception-ReNee v2 both achieve 0.896 QWK on the private leaderboard, while ResNet- 152 achieves 0.890 QWK, once again suggesting that better ImageNet architectures by themselves do not lead to increased performance on this task.
As a comparison, the ensemble from the top leaderboard entry included a single model Inception- ResNet v2 trained with additional interventions that achieves 0.927 QWK. We submitted the original models we trained Kaggle as well, finding that the new models trained with additional interven- tions do at least 0.03 QWK points better. See Appendix F for additional experimental details. Both this result and the gap between our models and the top leaderboard models show that there exist interventions that do improve task performance.
4.3 HUMAN PROTEIN ATLAS IMAGE CLASSIFCATION
The Human Protein Atlas runs the Human Protein Atlas Image Classification competition on Kaggle to build an automated tool for identifying and locating proteins from high-throughput microscopy images (Ouyang al., 2019). Images can contain multiple of the 28 different proteins, so the competition uses the macro Fl score. Given the multi-label nature of the problem, this requires thresholding for prediction. We use 73% / 18% / 9% train/ validation fest-validation split created by a previous competitor (Park) 2019). We report results on the validation split, as we find that the thresholds selected for the larger validation split generalize well to the smaller test-validation split.
We find a slightly positive trend between task performance and ImageNet performance, even when ignoring AlexNet and MobileNet. This is surprising because ImageNet is quite visually distinct from human protein slides. These results suggest that models with more parameters help with downstream performance, especially for tasks that have a lot of room for improvement.
Specific challenges for this dataset are extreme class imbalance, multi-label thresholding, and gen- eralization from the training data the test set. Competitors were able to improve performance beyond the baselines we found by using external data as well as techniques such as data cleaning, additional training augmentation, test time augmentation, ensembling, and oversampling (Dai) 2019] Park] 2019 [Sugaeev] 2019). Additionally, some competitors modified commonly architectures by substituting pooling layers or incorporating attention (Park 2019: ZZheg 2019). Uniquely, the first place solution used metric learning on top of a single DenseNet121 (Dai 2019). These tech- niques may be useful when applied to other datasets, but are rarely used in a typical workflow.

---

Under review as a conference paper at ICLR 2023
4.4 SIIM-ISIC MELANOMA CLASSIFICATION
The Society for Imaging Informatics in Medicine (SIIM) and the International Skin Imaging Collab- oration (ISIC) jointly ran this Kaggle competition for identifying Mclanoma (SIIM & ISIC 2020), a serious type of skin cancer. Competitors use images of skin lesions to predict the probability that each observed image is malignant. Images come from the ISIC Archive, which is publicly available and contains images from a variety of countries. The competition provided 33, 33,126 training images, plus an additional 25,331 images from previous competitions. We split the combined data into an 80% to 20% class-balanced and year-balanced train/validation split. Given the imbalanced nature of the data (8.8% positive), the competition uses area under ROC curve as the evaluation metric.
We find only weak positive correlation (0.44) between ImageNet performance and task perfor- mance, with a regression line with a normalized slope of close to zero (0.05). But if we instead look at classification accuracy, Appendix H shows that there is stronger trend for transfer than that of area under ROC curve, as model task accuracy more closely follows the same order as ImageNet performance. This difference shows that characterizing the relationship between better ImageNet models and better transfer performance is reliant on the evaluation metric as well. We use a rela- tively simple setup to measure the impact of ImageNet models on task performance, but we know we can achieve better results with additional strategies. The top two Kaggle solutions used models with different input size, ensembling, cross- validatiin and significant variety of training augmentation to create a stable model that generalized to the hidden test set (Ha et al. 2020 Pan 2020
4.5
CASSAVA LEAF DISEASE CLASSIFICATION
The Makerere Artificial Intelligence Lab is an academic research group focused on applications that benefit the developing world. Their goal in creating the Cassava Leaf Disease Classification Kaggle competition (Makerere University AI Lab) (021)) was to give farmers access to methods for diagnosing plant diseases, which could allow farmers prevent these diseases from spreading, increasing crop yield. Images were taken with an inexpensive camera and labeled by agricultural experts. Each image was classified as healthy or as one of four different diseases. We report results using a 80% random balsnce train/validation split of the provided training data.
Once we ignore models below 70% ImageNet accuracy. the relationship between the performance on the two datasets has both a weak positive correlation (0.12) and a near-zero normalized slope (0.02). While these are natural images similar to portions of ImageNet, it notable that ImageNet contains very few plant classes (e.g.. buckeye, hip, rapeseed). Yet based on dataset's perceived similarity to ImageNet, it surprising that leaf disease classification is not positively correlated with ImageNet, while the microscopy image based Human Protein Atlas competition is. Our results are supported by Kaggle competitors: the first place solution found that on the private leaderboard, EfficientNet B4 (Tan & Le. 2019), MobileNet, and ViT (Dosovitskiy et al., (022bb) achieve 89.5%, 89.4%, and 88.8% respectively (Hanke 2021). Their ensemble achieves 91.3% on the private leaderboard.
4.6 EUROSAT
Helber et al. (2019) created EuroSAT from Sentinel-2 satellite images to classify land use and land cover. Past work has improved performance the dataset through additional training time tech- niques (Naushad et al. 2021) and using 13 spectral bands (Yassine et al. 2021). We use RGB images and keep our experimental setup consistent compare across a range of models. Since there is no set train/test split, we create a 80% class-balanced split.
All models over 60% ImageNet accuracy achieve over 98.5% EuroSAT accuracy, and the majority of our models achieve over 99.0% EuroSAT accuracy. There are certain tasks where using better ImageNet models does not improve performance, and this would be the extreme case where perfor- mance saturation is close to being achieved. While it is outside the scope of this study, a next step would be to investigate the remaining errors and find other methods reduce this last bit of error.
ADDITIONAL STUDIES
5.1 AUGMENTATION ABLATIONS
In our main experiments, we keep augmentation simple to minimize confounding factors when com- paring models. However, it is possible pre-training and fine-tuning with different combinations of augmentations may have different results. This is an important point because different architectures
7

---

Under review as a conference paper at ICLR 2023
may have different inductive biases and often use different augmentation strategies at pre-training time. To investigate these effects, we run additional experiments on CCT-20 and APTOS to explore the effect of data augmentation transfer. Specifically, we take ResNet-50 models pre- trained with standard crop and flip augmentation, AugMix (Hendrycks et al. 2020), and RandAugment (Cubuk) et al. 2020), and then fine-tune on our default augmentation, AugMix, and RandAugment. We also study DeiT-tiny and Deit-small models by fine-tuning on the same three augmentations mentioned above. We choose examine DeiT models because they are pre-trained using Rugdett and RandErasing (Zhong et al. 2020). We increase the number of epochs we fine-tune on from 30 to 50 to account for augmentation. Our experimental results are found in Appendix G
In our ResNet-50 experiments, both AugMix and RandAugment improve performance on ImageNet, but while pre-training with RandAugment improves performance on downstream tasks, pre-training with Mug does not. Furthermore, fine-tuning with RandAugment usually yields additional per- formance gains when compared to our default fine-tuning augmentation, no matter which pre-trained model is used. For DeiT models, found that additional augmentation did not significantly in- crease performance on the downstream tasks. Thus, as with architectures, augmentation strategies that improve accuracy ImageNet do not always improve accuracy on real-world tasks.
5.2 CLIP MODELS
A natural follow-up to our experiments is to change the source of pre-training data. We exam- ine CLIP models from Radford et al. (202I), which use diverse pre-training data and achieve high performance variety of downstream datasets. We fine-tune CLIP models each of our down- stream datasets by linear probing then fine-tuning (LP-FT) (Kumar et al. 2022 Our results are We see that by using a model that takes larger images we can do better than all previous models, and even without the larger images, ViT-L/14 does better on four out of the six datasets. While across all CLIP models the change in pre-training data increases performance for CCT-20, the effect on the other datasets is more complicated. When controlling for architecture changes by only looking ResNet-50 and ViT/B16, we see that the ad- ditional pre-training data helps for CCT-20, HPA, and Cassava, the former two corresponding to the datasets that empirically benefit most from using better ImageNet models. Additional results can be found in Appendix II while additional fine-tuning details can be found in Appendix J
visualized by the purple stars in Appendix Figure [ppendix[FFiurre] 8
6 DISCUSSION
Alternative explanations for saturation. Whereas Kornblith al. (2019) reported high degree of correlation between ImageNet and transfer accuracy. we find that better ImageNet models do not consistently transfer better on our weal-world tasks. We believe these differences are related to the tasks themselves. Here, we rule out alternative hypotheses for our findings.
Comparison of datasets statistics suggests that the number of classes and dataset size also do not explain the differences from Kornblith et al. (2019). The datasets we study range from two to 28 classes. Although most of the datasets studied in Kornblith et al. (2019) have more classes, CIFAR- 10 has 10. In Appendix F. we replicate CIFAR-10 results from Kornblith et al. (2019) using our experimental setup, finding a strong correlation between ImageNet accuracy and transfer accuracy. Thus, the number of classes is likely not the determining factor. Training set sizes are similar between our study and that of Kornblihh et al. (2019) and thus also do not seem to play a major role. A third hypothesis is that it is parameter count, rather than ImageNet accuracy, that drives trends. We see that VGG BN models appear to outperform their ImageNet accuracy on multiple datasets, and they are among the largest models by parameter count. However, in Appendix L we find that model size is also not a good indicator of improved transfer performance on real world datasets.
Differences between web-scraped datasets and real-world images We conjecture that it is possi- ble to perform well on most, if not all, web-scraped target datasets simply by collecting a very large amount of data from the Internet and training very large model on it. Web-scraped target datasets are by definition within the distribution of data collected from the web, and a sufficiently large model can learn that distribution. In support of this conjecture, recent models such as CLIP (Radford al. 2021). 1, ALIGN (Jia et al. 2021). ViT-G (Zhai et al. 2022), BASIC (Pham et al.) 2021). and CoCa Yuu ct al. 2022) are trained on very large web-scraped datasets and achieve high accuracy on a variety of
We use LP-FT because, in past experiments, we have found that I.-FT makes hyperparameter tuning casier for CLIP models, but does not significantly alter performance when using optimal hyperparameters.
8

---

Under review as a conference paper at ICLR 2023
web-scraped benchmarks. However, this strategy may not be effective for non-web web-scraped datasets, where there is no guarantee that we will train on data that is close in distribution to the target data, even if we train on the entire web. Thus, it makes sense to distinguish these two types of datasets.
There are clear differences in image distribution between the non- web-scraped datasets we consider and web-scraped datasets considered by previous work. In Figure 3 and Appendix M compute Fréchet inception distance (FID) (Heusel et al.] 2017) between ImageNet and each of the datasets we study in this work as well as the ones found in Kornblith et al. (2019). The real-world datasets are further away from ImageNet than those found Ko- rmblith et al. (2019), implying that there is large amount of distribution shift between web-scraped datasets and real-world datasets. However, FID only a proxy measure and may not capture all fac- tors that lead to differences in transferability.
Web (ccunt)
Non web (count)
Number of daravess
0.00
25
50
76
198
125 ng
190
40
175
200 aB
225
250
Whereas web-scraped data is cheap to acquire real- world data can be more expensive. Ideally, progress in computer vision should improve performance not just on web-scraped data, but also on real-world tasks. Our results suggest that the latter has not happened. Gains in ImageNet accuracy over the last decade have primarily come from improving and scaling architectures, and past work has shown that these gains generally transfer other web-scraped datasets, regardless of size (Sun al. 2017 Kornblith et al. 2019; Mahajan et al. 2018 Xie et al.) 20200 Kolesnikov et al. 2020). However, we find that improvements arising from architecture generally do not transfer to non-web- web-scrppe tasks. Nonetheless, data augmentation and other tweaks can provide further gains on these tasks.
FC
Figure 3: FID scores vs ImageNet for the datasets study in this work (red), and the web-scraped datasets studied by Kornblith et al. 2019) (blue).
Recommendations towards better benchmarking. While it is unclear whether researchers have over-optimized for ImageNet, our work suggests that researchers should explicitly search for meth- ods that improve accuracy on real-world -on-web-seraped datasets, rather than assuming that meth- ods that improve accuracy on ImageNet will provide meaningful improvements on real-world datasets as well. Just as there are methods that improve accuracy on ImageNet but not on the tasks investigate, there may be methods that improve accuracy Kaggle community provides some evidence for the existence of such methods; Kaggle submissions often explore architectural improvements that are less common in traditional ImageNet pre-trained models. To measure such improvements on real-world problems, we suggest simply using the aver- age accuracy across our tasks benchmark for future representation learning research.
tasks but not ImageNet. The
Further analysis of our results shows consistencies in the accuracies of different models across the non-web-seraped datasets, suggesting that accuracy improvements on these datasets may translate to other datasets. For each dataset, we use linear regression to predict model accuracies on the target dataset as a linear combination of ImageNet accuracy and accuracy averaged across the other real- world datasets. We perform an F-test to determine whether the average accuracy on other real-world datasets explains significant variance beyond that explained by ImageNet accuracy. We find that this F-test is significant on all datasets except FuroSAT, where accuracy may be very close to ceiling (see further analysis in Appendix N.I.. Additionally, in Appendix [.2 we compare the Spearman rank correlation (i.e., the Pearson correlation between ranks) between each dataset and the accuracy averaged across the other real-world datasets to the Spearman correlation between each dataset and ImageNet. We find that the correlation with the average over real-world datasets is higher than the correlation with ImageNet and statistically significant for CCT-20, APTOS, HPA, and Cassava. Thus, there is some signal in the average accuracy across the datasets that investigate that is not captured by ImageNet top-1 accuracy.
Where do findings leave ImageNet? We suspect that most of the methodological innovations that help on ImageNet are useful for some real-world tasks, and in that sense it has been a successful benchmark. However, the innovations that improve performance on industrial web-scraped datasets such as JFT (Sun et al. 2017) or IG-3.5B-1 17k (Mahajan et al.) 2018) (e.g.. model scaling) may be almost entirely disjoint from the innovations that help with the web-ssaadrd real-world tasks studied here (c.g., data augmentation strategies). We hope that future benchmarks will include more diverse datasets to encourage more comprehensive approach to improving learning algorithms.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of
large scale pre-training. International Conference on Learning Representations, 2022. URL
httppss//openreviee..eee. net. forum (ttp:/ummmiiiccp0ppp
Asia Pacific Tele-Ophthalmologg Society, Aptos 2019 blindness detection, 2019. URL https:
7/www. www.w.agglee. wwt. wwtpc///cooppeetiiooo/ppppeppeee wnw.:ee---iiiiinndndeeeeeeee ion
overview
Sara Beery, Grant Van Hom, and Pietro Perona. Recognition in terra incognita. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference. Munich, Germany, September 8-14, 2018, Proceedings, Part XVI, volume 11220 of Lecture Notes in Computer Science, pp. 472 489. Springer, 2018. doi: 10. 1001/ 3-000-000----00
URL. https://doi.orrg//10 978-3-030-01270-01, 28.
28
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07152 2020.
Keno K Bressem, Lisa C Adams, Christoph Erxleben, Bernd Hamm, Stefan Niehues, and Janis Vahldiek. Comparing different deep learning architectures for classification of chest radiographs. Scientific reports, 10,1) 2020.
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint arXivv 2014.
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, ngs. neur. ipsurrspp...... cc.ppaper/222//hsi/ c85b63ef0ccbli htw.:/oco14400011127720055000000 -tt.aabbbbbbbb0000002a-aattaaat
In Hugo
virtual, 2020. URL https:://proeeee wwstraat..ttta.................. html
Shubin Dai. A cnn classifier and a metric learning model, 1st place so- lution, 2019. human-protein. at wtw.elaasiaaaellllaaaaiiiiaaeiee
URL https://www
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang. Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Inter- national conference on machine learning, pp. 647-655. PMLR, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL/ittpss//ppeneeviee..u aNTTy
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. URL htttp::pp/peeeeeeie...... Dorum?id-Yicse cNTTy.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self supe-super models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 5414-54555 1414,5423, 2021.
10

---

Under review as a conference paper at ICLR 2023
Mark Everingham, Lue Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zis- serman. The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis., 88(2): doi: 10.10077/11123300090000 5 s11263- 009-0275-2
303-338, 2010.
URL
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2004, Washington, DC, USA, June 27- July 2, 2004, pp. 178. IEEE Computer Society, 2004, doi: 10. 10.1109/C 2004.383. URL pettps://dolo htttpp://do..oog/......o 1109/ 1ttp://P....10//....... 2004.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning. In Proceedings of the ieee/cvf International Conference on computer vision, pp. 6391-6400, 2019.
Qishen Ha, Bo Liu, and Fuxu Liu. Identifying melanoma images using efficientnet ensemble: Win- ning solution to the SIIM-ISIC melanoma classification challenge. CoRR, abs/20000001,, 2020. URL peEtps://arxxe wwtpaaiwwiv.irrr/aaaaaaaaaaae wttp://///////oooo///ooo 05351
Jannis Hanke. 1st place solution, 2021. URL hwt.t:ttpps///ctmm www. waw.le.gg......... http://wwww..kgl...oo////////tia wwtp/////woooooccinioooco
cassava-
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10. 10.11099CVPR.2016.990
www.:1119//....... 2016.....01..99
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs Remote. Sens., 12(7):2217 2019. doi:
201. URL
https:// https://dol.or///33 1t0.:/.........1................
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustmess and uncertainty. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. | http:: http::httppppppot /openreviewweeeeee /tt.tttss///ppeeeeeiieee
net/ forumrrmmmmmm
S1 S1gmrxHFvv
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by time-scale update rule converge to local nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6626-6637, 2017. URL https: http:::sss/pppppppeeeeeeeoo neurips cc/ .html
paper http:///ww./..111000000016666710
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoe V. Le, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Search- ing for mobilenetv3. In 2019 JEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 1314-1324. IEEE, 2019. doi: 10. 10.1109//CC2221190000
htRLhtpss: doi. org 10. 1109
201.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 2261- IEEE Computer Society, 2017. doi: 10.1109/CVPR.207..220 201...22.0..244 URL https: //t::///ooo..o org 10 wttp://.....1.///. www...1111..../. 243
Minyoung Huh, Pulkit Agrawal, and Alexci A. Efros. What makes imagenet good for transfer learning? CoRR,
2016. URL httpp::/aarx.w.orr///aa///////// 08614]

---

Under review as a conference paper at ICLR 2023
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <Imb model size. CoRR, abs/1602.07360, 2016.
wwtp:/i/.rg/ab/////// http:/abb/16000.000 07360
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Alexander Ke, William Elisworth, Oishi Banerjee, Andrew Y Ng. and Pranav Rajpurkar. Chextrans- fer: performance and parameter efficiency of imagenet models for chest x-ray interpretation. In Proceedings of the Conference on Health, Inference, and Learning, pp. 116-124, 2021.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European confer- ence on computer vision, pp. 491-507. Springer, 2020.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661- 2671, 2019.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? Advances in Neural Information Processing Systems, 34, 2021.
Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt, Kiana Ehsani, and Roozbeh Mottaghi. Contrasting contrastive self-supervised representation learning pipelines. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9949-9959, 2021.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. In Peter L. Bartleu, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems 25: 26th Annual Conference Neural Information Processing Systems 2012. Proceedings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pp. 1106-1114, 2012. URL https: https::/pproceeedagss.ttt neurips. cctccc//ppee pttpp/ppppppappaeeaapp haahh/ c399862d3b 9dbbb 7e1e666666443666
html
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-disttribttion. In International Con- ference on Learning Representations, 2022. URL htttps://pperrwvicee wwt.
id=UYneFzXSJJu
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Fii-Feee Alan L. Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8- 14, 2018, Proceedings. Part I. volume 11205 of Lecture Notes in Computer Science, pp. 19-35. Springer, 2018 doi: 1007 978-3-030-01246-5 2, URL https: /woi. org 1007 0-30-88---30000000
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. CoRR, abs/2201 103545, 2022. URL htt.ttss//axxxi..rrgg//.oo/////t
0221.03544
Ningning Ma. Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. In Vittorio Ferrari, Martial Hebert, Cristian Sminchis- escu, and Yair Weiss (eds.), Computer Vision - ECCV 2018 15th European Conference, Mu- nich, Germany, September 8-14, 2018, Proceedings, Part XIV. volume 11218 of Lecture Notes in Computer Science, pp. 122-138, 138. Springer, 2018 doi: 10. 111111077777-0000-0000 https:///of .or.lorggg... 1007 0000077978--30000111110-
8. URL
12

---

Under review as a conference paper at ICLR 2023
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181-
196, 2018.
Makerere University Al Lab. Cassava leaf disease classificatton,, 2021. URL https:// Www com/ overview
Thomas Mensink, Jasper Uijlings, Alina Kuznetsova, Michael Gygli, and Vittorio Ferrari. Factors of influence for transfer learning across diverse appearance domains and task types. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 10. 10.109// 3129870.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correla- tion between duttof-distiibution and in-distribution generalization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Re- search, pp. 7721-7735. PMLR, 2021. URL BtlLhttp:/// httt:::/prrceedinne ngs.mm..m........ press/v139/ rtg...l.......s//1133.... mil miller21b. html
Raoof Naushad. Tarunprect Kaur, and Ebrahim Ghaderpour. Deep transfer learning for land use and land cover classification; A comparative study. Sensors, 21(23)88083, 2021. doi: 10.3390/ s21238083. /ttttttss//...ooooo/oooo
3390/ 0000//210 s2l228228
Niv Nayman, Avram Golbert, Asaf Noy, Tan Ping, and Lihi Zelnik-Manor. Diverse imagenet models transfer better. arXiv preprint arXhv:2204.09134, 2022.
Wei Ouyang, Casper F. Winsnes, Martin Hjelmare, Authony Cesnik, Lovisa Akesson, Hao Xu, Devin P. Sullivan, Shubin Dai, Jun Lan, Park Jinmo, Shaikat M. Galib, Christof Henkel, Kevin Hwang, Dmytro Poplavskiy, Bojan Tunguz, Russel D. Wolfinger, Yinzheng Gu, Chuanpeng Li, Jinbin Xie, Dmitry Buslov, Sergei Fironov, Alexander Kiselev, Dmytro Panchenko, Xuan Cao, Runmin Wei, Yuanhao Wu, Xun Zhu, Kuan-Lun Tseng, Zhifeng Gao, Cheng Ju, Xiaohan Yi, Hongdong Zheng, Constantin Kappel, and Emma Lundberg. Analysis of the human protein las image classification competition. Nature Methods, 16 (6(12)1124- 2019. doi: 10.
941592-019-0658 URL https:://ec. //fdei.org orggi.org/ll www.ooo.o......... 10. 1038 0115//119922221
Ian Pan. [2nd place] solution overview, 2020.
URL. https: wwtp://wwww...m www.wwwwww...................w.. com/
aitoma-clasaen wwc.lassiiicaatinnnii....c
175324
Jinmo Park. 3rd place solution with code., 2019. URL
c/
human www.ppooootetttteeeee
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingx- ing Tan, and Quoe V Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.10050, 2021.
Jean Ponce, Tamara L Berg, Mark Everingham, David A Forsyth, Martial Hebert, Svetlana Lazeb- nik, Marcin Marszalek, Cordelia Schmid, Bryan C Russell, Antonio Torralba, et al. Dataset issues in object recognition. Toward category-level object recognition, pp. 29 48. Springer, 2006.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8766 8763. PMLR, 2021.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems, 32, 2019.
13

---

Under review as a conference paper at ICLR 2023
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. $389-5400,
PMLR. 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252) 2015.
Hadi Salman, Andrew llyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33:3533-35455 2020.
Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt. Evaluating machine accuracy on imagenet. International Conference on Machine Learning, pp. 8634-8644. PMLR, 2020.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off- the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and paitern recognition workshops, pp. 806-813, 2014.
Maxim
Shugaev. 2019.
Pretrained
resnet34
with rgby (0.460 public
lb).
34- -tt.aaaceee..tttt--------------- Sttps://ww. ccm/ code/
URL
SIIM and ISIC. Siim-isic melanoma classification, 2020. URL https:///w/ wwwgwe............ com/
competitions ssw.oonssssiiiiiiieeeoe -ssie-iiii-i- me www.em-eeeaaaaaa-aeaaa www.ssaaasssssaaeeeeeeeeeeaaaiae wot.aaaaaaaac.....oooooorrreeeae
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. BRLhttp::0 http::/ttxtii.....tmmm arxiv. www.ewiww.rggga... wwg/abs/////oooooooo.e.....aae http://aab//99..955511 1459.1556
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. CoRR, abs/2106. 10270, 2021. URL htt.hhtsss//aaaii..rrgg.e
httpgg//b//2000.. 10270
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference computer vision, pp. 843-852, 2017.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections learning. In Satinder Singh and Shaul Markovitch (eds.). Proceedings of the Thirty-Firss AAA/ Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 4278 4284.4284. AAAI Press, 2017a. URL CUOP/AAAI/AAN /ttp////AAA///////////e/////////
http:// /aaa/ www./orgg.............o......o.. ocs/
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections learning. In Satinder Singh and Shaul Markovitch (eds.), Proceedings of the Thirty-First AAA/ Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 4278-4284 AAAI Press, 2017b. URL http:/ aaai
ocs/ http:iidxxxxpp /AA.AAAAI/AAA AAAAI/AAAAAA/////....A http://Aa/AAAA11//app/r///////a1 /1880
Mingxing Tan and Quoe V. Le.. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114, PMLR, 2019. URL
mlr wwwpsppsepsee.... v97 1t///////aana 9a. html
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-18599, 2020.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 201 pp. 1521-1528.
IEEE, 201 1.
14

---

Under review as a conference paper at ICLR 2023
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Ma- rina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine mmrr.
Learning Research, pp. 10347-10357. PMLR, 2021. URL htttp://proceedi press/1139/towronn2aaaa.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. From imagenet image classification: Contextualizing progress on benchmarks. In International Con- ference on Machine Learning, pp. 9625-9635. PMLR, 2020.
Lukas Tuggener, Jürgen Schmidhuber. and Thilo Stadelmann. Is it enough to optimize emn architec- tures on imagenet? arXiv preprint arXiv:2103.09108, 2021.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. CoRR, abs/21 10.0046. 2021.
00476
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. Proceedings of the IEEE/CVE conference on computer vision and pallern recognition, pp. 10687-10698, 2020.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated resid- ual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 5987-5995. IEEE Computer Society, 2017.
doi: 10.1109/CVR.22017..66 URL httphttps:///o..co
1109/CVPR.2017.
Guanshuo Xu. 1st place solution summary, 2019. URL https://ww.kagglee com/ Compet it wwiiiiinnsssss 1wtp22019biiiiiiie blindness detect in http://o/ooccccsssoooooo000000
H. Yassine, K. Tout, and M. Jaber. Improving Lule Classification from Satellite Imagery Us- ing Deep Learning Eurosat Dataset. ISPRS International Archives of the Photogram- metry, Remote Sensing and Spatial Information Sciences, 43B3:369-377 376,, June 2021. doi:
1151999/sprs-archin XLIII-B3: B3-2021- 2021-369-2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE con- ference computer vision and pattern recognition, pp. 3712-3722, 2018.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. The visual task adaptation benchmark. 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12104-1218 2022.
Kevin out cv). 2019.
Zheng.
39th solution-attentin
gated resnet18 (single model with- URL https:/ www. wwwgle.co.ooo.ooooooo.ee.ooooooa cttpm//comeetiions///oo
wwwpaallllmmmmmaassseee www.iiaaaaaaaaaaeeeee Songe-classin cat on /tt.in//iii www.ssisssssssssss.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data aug-
mentation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secone Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 13001-13008. AAAI Press, 2020. URL https: 1/ojs. aaai. .or./oorg///ee.ee
15

---

Under review as a conference paper at ICLR 2023
Appendix
A DETAILED EXPERIMENT RESULTS
Table For ImageNet pre-trained model, we provide the best performing model when fine-tuned on each dataset across our hyperparameter grid
Model
ImageNet top-1
CCT20
APTOS
HPA
Melanoma
Cassava
EuroSAT
AlexNet
56.5
63.59
0,8835
0.3846
0.9283
82.58
97.93
SqueezeNet 1.1
58.2
66.36
0,9021
0.3972
0.9073
85.15
98.07
ShuffleNetV2x0.5
60.6
66.37
0,9227
0.5867
0.9289
85.64
98.56
MobileNet V3 small
67.7
66.01
0,9230
0.6108
0.9455
85.81
99.15
Shu1flcNeeV2x1..0
69.4
69.27
0,9202
0.6202
0.9418
87.26
98.91
VGG-13 BN
71.6
75.06
0.9268
0.6794
0,9529
88.73
98.85
DeïT-tiny
72.2
68.77
0.9130
0.5777
0.9510
86.04
99.11
VGG-16 BN
73.4
75.93
0,9287
0.6791
0,9531
88.45
98.93
DenseNet-121
74.4
74.66
0,9287
0.7019
0,9514
87.80
99.06
ResNet-50
76.1
73.96
0.9215
0.6718
0.9524
87.75
99.19
ResNeXt-50-32x4d
77.6
73.73
0.9212
0.6906
0.9588
88.15
99.24
EfficientNet BO
77.7
71.02
0,9195
0.6942
0.9456
87.63
98.80
ResNet-152
78.3
74.05
0.9228
0.6732
0,9562
87.75
99.15
ViT-B/16
78.7
72.07
0.9262
0.5852
0.9600
86.63
99.28
DeiT-small
79.9
71.41
0,9205
0.6148
0.9583
87.12
99.20
Inception
ReceptionRResNet ResNet v2
80.4
70.68
0.9168
0.6882
0.9483
87.57
98.93
Con-Next-tiny
82.5
78.51
0.9297
0.6992
0.9628
88.89
99.11
PNASNet-5 large
82.9
75.21
0.9271
0.6941
0.9584
87.47
99.17
EfficientNet B4
83.4
73.49
0.9211
0.6954
0,9552
88.36
98.70
See attached supplementary material for experiment results across hyperparameters. In the future this will be spreadsheet link.
16

---

Under review as a conference paper at ICLR 2023
MAIN FIGURE VARIATIONS B
Traps
Blindness
weighted kappa
0.70
Macro FT score
0.60
Accuracy
0.40
0..00
0.50
ImageNet top-1 accuracy
imageNat top-1 accuracy
ImageNet top accuracy
SIIM-ISIC Melanoma
Cassava Leaf
EuroSAT
Areo under ROC
0.79
Accuracy
Accuracy
0.34
06.25
9.32
b4
66.00
97.73
97.50
00
82
60
ImageNet top-1 accuracy
ImageNet top-1 accuraty
ImageNet top-1 accuracy
AlexNet
ResNett112
EfficientNet BO
ConvNext-ting -iny
MobileNetV3-amall
EfficientNet
ShufteNetV2x0.5
VGG-13 aN
PNASNet-S
DenseNet-121
SqueezeNet 1.1
DelT4iny
Inception-ResNett
ResNext-50-32xdd
VT-B/16
ResNet-50
VGG-16
ShuffleNetV2x1.0
Figure 4: Figure 1 with error bars. Green is linear trend of all models, while blue is linear trend for models above 70% ImageNet accuracy. use 95% confidence intervals computed with Clopper- Pearson for accuracy metries and bootstrap with 10,000 trials for other metrics.
Caltech Camera Traps
APTOS 2019 Blindness
Human Protein Atlas
Quadratic ginted kappa
0.7
Mecro Fl score 8
72.5
0.00
0.5
0.00
0.0
43.5
03
ImageNet top-1 accuracy
ImageNet top-1 accuracy
ImageNet top-1 accuracy
Melanoma
Leaf
EuroSAT
Area Under ROC
0.95
0.24
Accuracy
Accurai
3.53
02
66.50
06.25
06.00
97.15
07.50
60
ImageNet top-1 accuracy
ImageNit top-1 accuraty
ImageNet top accuraty
AlexNet
ResNet-152
EfficientNet BO
ConvNext-tiny
MobileNatV3-small
DefT-small
EfticientNet B4
ShuttleNetvzx0.5
VGG-13 BN
PNASNet-5
DenseNet-123 121
SqueezeNet 1.1 VIT-8/16
DeiT-tiny
Inception-ResNet v2
ResNeXt-50-32xxt ShuffleNetV2xx..
ResNet--S 50
VGG 16
Figure 5: Figure 4 with spline interpolation fits instead of lincar fits.
17

---

Under review as a conference paper at ICLR 2023
C EXPERIMENT SETUP
C.I
MODELS
Table 4: We examine the effectiveness of transfer learning from a number of models pretrained on ImageNet, including both CNNs and Vision Transformers.
Model
ImageNet top-
# params
Year Released
AlexNet (Kizhevskky al. 2012
56.5
61M
2012
SqueezeNer 1.1 Laaddoia et an 2016
58.2
1.2M
2016
ShuffeNerV22d. [Mia
60.6
MobileNet V3 small [loward a
2019
67.7
2.5M
2019
ShuffeeNe122x et
69.4
2018
VGG-13 BN Simonvan Ziserran
2015
71.6
133M
2014/2015
Deil-tiny [Teavoon ol 2041
72.2
5.7M
2020
VGG-16 BN
2015
73.4
138M
201 4/2015
DenseNet-122 (Hllanng eral 2017
(Filaag
74.4
8.0
2016
ResNet-50 (Hc arat
76.1
26M
2015
ResNeXi.50.522al (Ate clal 2017
77.6
25M
2016
EfficienNee BO Tan Le 2019)
77.7
5.3M
2019
ResNet-152
(H (
A016
78.3
60M
2015
VIT-B/16 (Dasvilski cl 2021a
Steiner et
2021
78.7
304M
2020
DeiT-small Touviio 2021
79.9
22M
2020
DncpptionResNers (Szegedy
2017a
804
56M
2016
Con-Nexx-tiny (Liu cl
82.5
29M
2022
PNASNer-5 large
La al
20118
82.9
86M
2017
EfficientNe B4
Tan Lc
2019
83.4
19M
2019
We examine model architectures this work that cover diverse range of accuracies on ImageNet in order to observe the relationship between ImageNet performance and target dataset performance. In addition to the commonly used CNNs, we also include data-efficient image transformers (DeiT) due to the recent increase in usage of Vision Transformers. Additional model details are in Table 4
C.2 HYPERPARAMETER GRID
Hyperparameter tuning is key part of neural network training, using suboptimal hyperparameters can lead to suboptimal performance. Furthermore, the correct hyperparameters vary across both models and training data. To get the best performance out of each model, we train each model on AdamW with a cosine decay learning rate schedule, SGD with a cosine decay learning rate schedule, and SGD with multi-step decay learning rate schedule. We also grid search for optimal initial learning rate and weight decay combinations, searching logarithmically between 10 to 10 for SGD learning rate, 10 to 10 for AdamW learning rate, and 10 to 10 as well as 0 for weight decay. All models are pretrained on ImageNet and then fine-tuned on the downstream task. Additional training details for each dataset be found in Appendix [ We also hyperparameter grid on CIFAR-10 in Appendix(Ejes verify that we find a strong relationship between ImageNet and CIFAR-10 accuracy as previously reported by Kornblith al. (2019).
D TRAINING DETAILS BY DATASET (IMAGENET MODELS)
Experiments on Cassava Leaf Disease, SIIM-ISIC Melanoma, and EuroSAT datasets were ran on TPU v2-8s, while all other datasets were on NVIDIA A40s.
All experiments were ran with mini-batch size of 128.
For SGD experiments, we use Nesterov momentum, set momentum to 0.9, and try learning rates of le-1, le-2, 1e-3, and le-4, For Adam W experiments, we try learning rates of le-2, 1e-3, le-4, le-5. For all experiments, we try weight decays of 1e-3, Ie-4, 1e-5, 1e-6, and O.
For all experiments, we use weights that are pretrained on ImageNet. AlexNet, DenseNet, Mo- bileNet, ResNet, ResNext, ShuffleNet, SqueezeNet and VGG models from torchvision, while ConvNext. DeiT, EfficientNet. InceptionResNett, and PNASNet models are from timm. Addition- ally, normalize images to ImageNet's 's mean and standard deviation.
For EuroSAT we random resize crop to 224 with area at least 0.65.
For all other datasets, random resize crop with area at least 0.65 to 224 for DeiT models, and 256 for all other models. Additionally, we use horizontal flips. For Human Protein Atlas, Cassava Leaf Disease, and SIIM-ISIC Melanoma, also vertical flips.
18

---

Under review as a conference paper at ICLR 2023
For SIIM-ISIC Melanoma, train for 10 epochs, and for the step scheduler decay with factor 0.1 at epochs.
For all other datasets, train for 30 epochs, and for the step scheduler decay with factor 0.1 at 15, 20, and 25 epochs.
E CIFAR- 10 ON HYPERPARAMETER GRID
CIFAR-10
97
Accuracy
96
ImageNet top-1 accuracy
AlexNet
ResNet-50
Inception-ResNet v2
DenseNet-121
MobileeNetV3-small
ResNet-152
VGG-16 BN
ResNext-50-3244
VGG-13 BN
DeiT-small
EfficientNet BO
ShuftleNetV2x1.0
DeiT-tiny
PNASNet-5
EfficientNNet B4
ConvNext-tiny
Transfer performance across models from ImageNet to CIFAR-10 Green linear trend is computed across all models, while blue linear trend is restricted to models above 70% ImageNet accuracy. We use 95% confidence intervals computed with Clopper-Pearson.
Figure 6:
F APTOS 2019 BLINDNESS DETECTION ABLATIONS
Scores presented are submissions to the Kaggle leaderboard. All scores are evaluated with quadratic weighted kappa. Within each entry, we first present the private leaderboard score, then the pub- lic leaderboard score. The private leaderboard represents 85% of the test data, while the public leaderboard the remaining 15%.
Models used here are trained using Adam with a cosine scheduler. We random resize crop to 512, use random rotations, and use color jitter (brightness=0.2. contrast=0.2, saturation=0.2, hue=0.1), We train on all the available training data, no longer using the local train/validation split mentioned in the main text. This includes both the training data the 2019 competition, as well data from prior 2015 diabetic retinopathy competition.
Table 5: Comparing various models with additional interventions by evaluating on the Kaggle leaderboard.
\wd
1.00E-04
1.00E-05
1.00E-06
1.00E-03
0.8610 0.6317
0.8570 / 0.6180
0.8548 0.6646
5esNet-50
1.00E-04
0.8952
0.7531
0.8918 0.7204
0.8961
0.7547
1.00E-03
0.8658 0.6812
0.6612 0.8686
0.8640 0.6554
ResNet-152
1.00E-04
0.8898 / 0.7164
0.8836/ 0.6946
0.8859 0.6947
1.00E-03
0.8933 0.7748
0.8905 0.756..55
0.8960 0.7585
1.00E-04
0.8897 0.7210
0.8929 0.7420
0.8944 0.7439
19

---

Under review as a conference paper at ICLR 2023
Table 6: Comparing the effect of augmentation Kaggle leaderboard scores. More augmentation is de- scribed earlier this section. Less augmentation only uses random resize crop with at least 0.65 area and horizontal flips.
Ir (d
1.00E-04
1.00E-05
1.00E-06
1.00E-03
0.8669
0.6405
0.8520 0.6013
0.8613 0.6269
ResNet-50 less aug
1,00E-04
0.8525 / 0.6115
0.8570 / 0.6431
0.8483 / 0.6147
1.00E-05
0.8186 / 0.5071
0.8287 0.5647
0.8288 0.5328
1.00E-03
0.8440 0.6432
0.8547 0.6856
0.8524 / 0.7125
ResNet-50
1.00E-04
0.8948 0.7490
0.8972 / 0. 0699
0.8999 0.7758
more aug
1.00E-05
0.8724 / 0.7370
0.8685 / 0.7567
0.8623 0.7376
G AUGMENTATION ABLATION DETAILS
We examine the effect of pre-training augmentation and fine-tuning augmentation on downstream transfer performance. The model specifies the architecture and pre-training augmentation, while each column specifies the downstream ImageNet accuracy do not always improve accuracy on downstream tasks. Pre-trained augmentation models are from Wightman et al. (2021).
Table 7:
fine-tunin tinettning augmentation. We find that augmentation strategies that improve
Model
ImageNet Acc
CCT-20 Basc Aug
CCT-20 AugMix
CCT-20 RandAug
APTOS Base Aug
APTOS AugMix
APTOS RandAug
ResNet-50
72.02 71.63
72.24
73.57 72.39
0.9210 0,9239
0,9212
0.9250
ResNet-50
77.5
71.53
0,9152
0.9222
w/ AugMix
ResNet-50
78.8
72.94
73.54
73.76
0.9190
0.9204
0.9302
w/ RandAug
Deit-tiny
72.2
66.57
66.47
66.95
0.9153
0.9197
0.9172
Deit-small
79.9
70.65
69.72
70.07
0.9293
0.9212
0.9277
H MELANOMA METRIC COMPARISON
SIIM-III Melanoma ROC
SIIM-ISIC Melanoma
0.90
96.0
Area under ROC
0.95
95.5
Accuracy
0.94
0.91
0.90
93.0
70
60
70
80
ImageNet top-1 accuracy
ImageNet top-1 accuracy
AlexNet
ResNet-152
EfficientNet BO
ShuffleNetV2x1.0
MobileNNetV3-smal
DeiT-small
EfficientNet B4
ConvNext-tiny
VGG-13
PNASNet-5
DenseNet-121
ShuffleNetV2x0.5
DeiT-tiny
Inception- ResNett v2
SqueezeNet 1.1
ResNet-50
VGG-16 BN
Figure 7: Comparing transfer performance from ImageNet to Melanoma using different metrics. Green linear trend is computed across models, while blue linear trend is restricted models above 70% ImageNet accuracy. Using accuracy implies that better ImageNet models transfer better; however, ROC is a better metric for this task.
20

---

Under review as a conference paper at ICLR 2023
I
CLIP EXPERIMENT DETAILS
Caltech Camera Traps 20
APTOS 2019 Blindness
Human Protein Atlas
Quadratic weighted kappa
0.75
Madro score
Accuracy
0.32
0.60
0.55
0.30
70-
65-
0.40
0.35
55
63
75
80
65
70
75
80
55
65
70
75
80
ImageNet top-1 accuracy
ImageNet top-1 accuracy
ImageNet top accuracy
SIIM-ISIC Melanoma
Cassava Leaf Disease
EuroSAT
0.97
Area under ROC
0.96
Accuracy
Accuracy
68.0
0.94
0.93
0.92
0.90
85
65
80
15
ImageNet top-1 accuracy
ImageNet top-1 accuracy
ImageNet top accuracy
AlexNet
PNASNet-5
ResNeXt-50-32x4d
CLIP-AN50
MobilleNetV3-small
Inception-ResNet
ShuffleNetV2x1.0
CLIP-RN101
VGG BN
VGG-16 BN
ConvNext-tiny
CLIP-B32
DeiT-tiny
EfficientNet
ShuffleNetV2x0.5
CLIP-B16
ResNet-50
EfficientNet B4
SqueezeNet 1.1
CLIP-L14
1552
DenseNet-121
VIT-B/16
CLIP-L14@336
DefT-small
Figure 8: Figure 4 with CLIP models overlaid (purple stars). The best CLIP models do better than all the ImageNet models, but when looking across all CLIP models, patterns are more complicated.
Table each CLIP pre-trained model, we provide the best performing model when fine-tuned on each dataset across our LP-FT hyperparameter grid
Model
ImageNet top-1
CCT20
APTOS
HPA
Melanoma
Cassava
EuroSAT
CLIP-RN50
73.3
74.45
0.9135
0.7053
0.9350
87.89
98.80
CLIP-RNIOL
75.7
75.19
0.9235
0.6909
0.9378
87.68
99.11
CLIP-B32
76.1
70.57
0.9137
0.5338
0.9546
86.28
99.26
CLIP-B16
80.2
77.81
0.9213
0.6365
0.9619
87.82
99.24
CLIP-L14
83.9
79.99
0.9330
0.6687
0.9717
88.82
99.33
CLIP-L14@336
85.4
83.17
0.9337
0.7131
0.9738
89.24
99,48
Table 9: We directly compare models pre-trained on ImageNet with models pre-trained on OpenAl's CLIP data. Specifically, we look at ResNet 50 and ViT B/16.
Model
ImageNet top-1
CCT20
APTOS
HPA
Melanoma
Cassava
EuroS AT
IN-ResNet-50 76.1
73.96
0.9215
0.6718
0.9524
87.75
99.19
CLIP-RN50
73.3
74.45
0.9135
0.7053
0.9350
87.89
98.80
IN-ViT-B/16
78.7
72.07
0.9262
0,5852
0.9600
86.63
99.28
CLIP-B16
80.2
77.81
0.9213
0.6365
0.9619
87.82
99.24
J CLIP FINE-TUNING DETAILS
We fine-tune by running linear probe, followed by end-to-end fine-tuning on the best model from the first part. We keep total epochs consistent with the previous models, with third of the epochs going toward linear probing. We use AdamW with cosine decay schedule. During the linear probe, we search over
and 10 - learning rates, and during fine-tumine fine-tuning, we search over 10-4 10-3 and 10-6 learning rates. For both parts, search over 10-3 to 10-6 and for weight decay.
21

---

Under review as a conference paper at ICLR 2023
K
CREATION INFORMATION FOR DATASETS STUDIED IN KORNBLITH ET AL. (0019)
Table 10:
We find that the 12 datasets studied in Kornblith al. 2019 from web scraping.
Dataset
Origin
Additional information
Food-101
foodspotting.com
Users upload an image of their food and anno-
tate the type of food; categories chosen by pop- ularity
CIFAR-10
Tinylmages
Web crawl
CIFAR-100
Tinylmages
Web crawl
Birdsnap
Flickr
Also used MTTurk
SUN397
Web search engines
Also used WordNet
Stanford Cars
Flickr,
Google,
Also used MTurk
Bing
FGVC Aireraft
airliners.net
Images taken by 10 photographers N/A
Pascal VOC 2007 Cls.
Flickr
Describable Textures
Google and Flickr Flickr, Google, Catster, Dogster Google
Also used MTTurk
Oxford-IIT Pets
Catster and Dogster are social websites for col-
lecting and discussing pet images
Caltech- 101
97 categories chosen from Webster Collegiate
Dictionary categories associated with drawing small number of images acquired by the pa- per authors taking the pictures
Oxford 102 Flowers
Mostly collected
from web
L RELATIONSHIP BETWEEN MODEL SIZE AND TRANSFER PERFORMANCE
Caltech Camera Traps 20
APTOS 2019 Blindness
Human Protein Atlas
80.0
Quadratic reighted kappa
0.8
0.64
77.5
Macro F1 score
0.7
Accuracy
22.5
0.8
70.0
0.9
0.88
65.0
0.4
62.5
20
60
100
120
20
60
150
120
20
60
300
120
of parameters (In millions)
of parametersss (In millions)
of parameters (in millions)
SIIM-ISIC Melanoma
Cassava
EuroSAT
99.25
Area under ROC
0.90
99.00
0.95
Accuracy
Accurach
0.94
86
98.50
0.93
88.25
0.92
84
0.91
97.75
87.50
129
millions)
of parameters millions)
parameters (in millions)
AlexNet
ResNet-152
ErficientVet BO
ShuftieNetvzx1.0
MobileNetV33smal
DeiT-small
EfficientNet B4 DenseNet-121 ResNeXt-50 50-32x4d
ConvNext-tiny
VGG-13 BN
PNASNet-5
ShuffleNetV2x0.3 SqueezeNet 1.1
DeiT-tiny
Inception-ResNe. ResNet v2
ResNet-50
VGG-16 BN
We compare model size with downstream transfer performance. Again we use separate trend lines for all models (green) and only those above 70% ImageNet accuracy (blue). We use 95% confidence intervals computed with Clopper-Pearson for accuracy metries and bootstrap with 10,000 trials for other metrics.
Figure 9:
22

---

Under review as a conference paper at ICLR 2023
M FID SCORE DETAILS
We calculate FID scores between the ImageNet validation set and each of the datasets we study, as well as between the ImageNet validation set and each of the datasets in Korbblihh et al. (2019). We found that datasct size affects FID score, take 3,662 subset of each downstream dataset. Note that 3,662 is the size of APTOS, which is the smallest dataset.
Table 11:
Dataset
FID
CCT-20
162.69
APTOS
196.24
HPA
230.70
Cassava
179.24
Melanoma
186.34
EuroSAT AT
151.85
Food- -01
108.35
CIFAR-10
132.53
CIFAR-100
120.72
Birdsnap
94.08
SUN397
62.95
Stanford Cars
143.35
FGVC Aircraft
183.35
Pascal VOC 2007 Cls.
39.84
Describable Textures
89.13
Oxford-IIT Pets
77.27
Caltech-101
50.77
Oxford 102 Flowers
140.21
N PREDICTIVE POWER OF ACCURACY ON NON-WEB-SCRAPER DATASETS ON NOVEL DATASETS
We observe that, on many non-web-scraped datasets, accuracy correlates only weakly with Ima- geNet accuracy. It is thus worth asking whether other predictors might correlate better. In this section, we examine the extent to which accuracy on a given non-web-scrape target dataset can be predicted from the accuracy the other non-web-scraped target datasets.
N.1 F-TEST
We can further measure the extent to which the averages of the five other datasets beyond the pre- dictive power provided by ImageNet by using F-tests. For each target task, we fit linear regression model that predicts accuracy either ImageNet accuracy the average accuracy the other five non-web-scraped datasets, and a second linear regression model that predicts accuracy as a func- tion of both ImageNet accuracy and the average accuracy the other five datasets. Since the first model is nested within the second, the second model must explain at least as much variance as the first. The F-test measures whether the increase explained variance is sigmificant. For these ex- periments, we logit-transfomm accuracy values and standardize them to zero mean and unit variance before computing the averages, as in the middle column of Table
Results are shown in Table 12 The average accuracy across the other five datasets explains variance beyond that explained by ImageNet accuracy alone on five of the six datasets. The only exception is EuroSAT, where the range of accuracies is low (most models get -999% and significant fraction of the variance among models may correspond to noise. By contrast, ImageNet accuracy explains variance beyond the average accuracy only two datasets (APTOS and Melanoma). These results indicate that there are patterns in how well different models transfer to non- web-scraped data that captured by ImageNet accuracy alone, but captured by the accuracy other non-web- scraped datasets.
23

---

Under review as a conference paper at ICLR 2023
Results of the F-test described in Section N.1 "+Avg. across datasets" tests whether model that includes both ImageNet accuracy and the average accuracy across the other datasets explains more variance than a model that includes only ImageNet accuracy. "+ImageNet" tests whether model that includes both predictors explains more variance than model that includes only the average accuracy across the other datascts. In addition to F and values, report adjusted R for all models. p-values 0.05 are bold-faced.
Table 12:
+Avg.
+emageNet
Adj. Rt² (ImageNet-onl)
Adj. Nt² (Average only)
Adj. R² (Both predictors)
Dataset
F(1, 16)
p-value
F(1,
p-value
OCT-220
8.2
0.01
0.69
0.42
0.56
0.70
0.69
APTOS
31.0
0.00004
0.047
0.34
0.71
0.76
HPA
11.8
0.003
0.84
0.37
0.60
0.76
0.76
Cassava
13.2
0.0022
0.14
0.71
0.55
0.75
0.74
EuroSAT
0.11
0.72
0.41
0.52
0.49
N.2 SPEARMAN CORRELATION
Table 13: We measure the Spcarman correlation between cach datasct with either the average of the 5 other datascts study, with ImageNet. Normalization is done by logit transforming accuracies, and then dardizing mean and unit variance. The results suggest that using additional datasets predictive model performance than just using ImageNet.
Avg of 5 others (unnormalized)
Avg of 5 others (normalized)
ImageNet
Dataset
p-value
p-value
p-value
CCT-20
0.8684
0.0000
0.9263
0.0000
0.5825
0.0089
APTOS
0.7205
0.0005
0.6950
0.0010
0.3010
0.2105
HPA
0.7351
0.0003
0.6825
0.0013
0.6491
0.0026
Melanoma
0.6561
0.0023
0.7807
0.0000
0.7667
0.0001
Cassava
0.8872
0.0000
0.7442
0.0003
0.5222
0.0218
EuroSAT
0.3030
0.2073
0.3821
0.1065
0.4734
0.0406
24

---

Under review as a conference paper at ICLR 2023
O PRE-TRAINING AUGMENTATION DETAILS
Table 14: time.
For each ImageNet pre-trained model, provide the augmentation strategy used during pre-training
Model
Augmentation
AlexNet
Resize Crop + Flip
SqucezeNet 1.1
Resize Crop Flip
Shu0f
AutoAAgment (TivialAAugmentWide) + RandErasing MMixUp + CutMix
MobileNet V3 small
AutoAAgment (ImageNet/Defaull)> RandErasing
ShuffleNet22x1..
AutoAAument (TrivialAugmeettidd)) + RandErasing MixUp + CutMix
VGG-13 BN
Resize Crop + Flip RandAugment RandErasing
DeïT-tiny
VGG-16 BN
Resize Crop + Flip
DenseNet-121
Resize + Crop + Flip
ResNet-50
Resize Crop + Flip
ResNeXt-50-32x44
Resize Crop + Flip
EfficientNet BO
RandAugment
ResNet-152
Resize Crop Flip
ViT-B/16
RandAugment MixUp
DeiT-small
RandAugment RandErasing
Inception-Rese ResNet v2
Inception Preprocessing (Color Distort Resize + Crop Flip) AutoAugment (Triiaa/AnmmentWide) RandErasing Mix CutMix Whiten + Resize + Crop + Flip RandAugment
ConyNext-tiny
PNASNet-5 large
EfficientNet B4
Caltech Camera Traps 20
APTOS 2019 Blindness
Human Protein Atlas
guadratic velghtted kappa
Macro F1 score
Accuraryy
0.55
0.00
0.05
0.40
0.59
ImageNet accuracy
ImageNet top-1 accuracy
ImageNet top-1 accuuracy
SIIM-ISIC Melanoma
Cassava Leaf Disease
EuroSAT
0.39
10.25
Area Under RO0
Accuracy G
26.75
08.50
0.35
0.82
0.39
ImageNet top-1 accuracy
ImageNet top-1 accuracy
ImageNet top-1
AlexNet
ResNet-152
EfficientNet BO
ConvNext-tiny
MobileNetV3
DelT-small
EfficientNet B4
ShufleNatV2x0..
VGG-13 aN
PNASNet-S
DenseNat-121
SqueezeNet
DeiTitiny
Inception-ResNece ResNet v2
ResNeXt 50 32x4d
VIT.8/16
ResNet-50
VGG-16
ShuffeNeW22k1.0
Figure Figure I with points colored by general pre-training augmentation strategy. Cyan points use simple augmentation (resize, crops, flips, etc.), and red points use automatic augmentation (RandAugment, AutoAug- ment, Wigee
25


Reviews:
[{"review_id": "dUZhQG0d_4", "reviewer": "Reviewer_FJNw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents an extensive study  on the generalization of ImageNet pre-training. Six real-world datasets spanning from camera traps to satellite imagery are taken into account. The work is motivated by the fact that existing work on visual classifier generalization are often analyzing downstream tasks (e.g. detection) on dataset that are collected with the same strategy as imagenet (web scraping), moreover the datasets on which the transfer learning is performed are often object centric.\n\nThe analysis shows that high accuracy on the pre-training dataset not always correlate with high accuracy on real-world tasks for all architectures. Notably some models (E.g. ConvNext) show high correlation.\n\n", "summary_of_the_review": "The paper is interesting and likely useful to the research community.  There are some issues in how experiments are formulated (augmentation) but this can be possibly be clarified in the rebuttal.\n\nSome lower level insights on the learned representation would be interesting to better understand behavior of this models in fine-tuning.", "strength_and_weaknesses": "Strenghts\n\n- The study is well formulated and takes into consideration recent neural architectures and a good variety of target datasets\n- The use of FID to show difference in dataset distribution is interesting, though the value of FID can be hard to interpret. It could be interesting to just train a linear classifier on each pre-trained representation (before fine-tuning) on the target dataset, this should be a good representative of distribution shift and also more understandable.\n\nWeaknesses\n- Unclear if models are downloaded from torchvision with pre-trained weights or re-trained with a fixed set of parameters on Imagenet. This is especially troubling regarding the augmentation strategy that may vary in different pre-trained models (see discussion in ConvNext paper about the pre-training strategy)\n- The analysis  could include some more low-level analysis of the architecture behavior. What are the layers that change the most during fine-tuning? How much feature representation shifts from  the pre-trained network and the fine-tuned one?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N5Heg6hxgI", "reviewer": "Reviewer_ZbHU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides experiments suggesting that there is only a very small correlation between performance of a neural classifier on ImageNet and its performance (after fine-tuning) on datasets that are not web-scraped.\n", "summary_of_the_review": "Some affirmations of the paper are dubious, and the novelty is not very important; it is essentially an empirical work. Besides, the conclusion drawn by the paper feels already like common knowledge. For instance, in deep learning for satellite spectral imagery (like the Sentinel-2 example provided by the author), there are already many papers showing how to adapt the convolution specifically for that type of images that have a radically different distribution with respect to more classical images (scrapped or not scrapped...). The authors present some of their statements as if it is something that the community should change. But the truth is that it is already very clear for the practitioners of the various real-world tasks.", "strength_and_weaknesses": "The numerical experiment seem to be well conducted. Yet, the results of the paper are not very surprising and some claims/conclusions of the authors seem strange or unnecessary. For instance, why this obsession with scrapped VS non-scrapped datasets? Isn't it enough to describe the reason why the 6 chosen datasets have implicitly different underlying distribution?\n\nWhy also not mentioning domain adaptation techniques that are beyond fine-tuning?\n\nSome example of affirmation that are doubtful:\n\n\"In particular, we study datasets collected with the goal of solving real-world tasks (e.g., classifying images from\ncamera traps or satellites), as opposed to web-scraped benchmarks collected for\ncomparing models.\" -> Dataset collected from the web also serve to structuring data in the web. This is a real-world task.\n\n\"In particular, we consider classification tasks derived from image data that were specifically collected with the goal of classification in mind. This is in contrast to many standard computer vision datasets – including ImageNet – where the constituent\nimages were originally collected for a different purpose, posted to the web, and later re-purposed for benchmarking computer vision methods.\" -> There seems to be a confusion between the intent behind collecting the data and the intent behind the existence of the data. And what was the different purpose in the collection of data for ImageNet if not classification? The notion of intent is very subject to discussions...\n\n\"Web-scraped target datasets are by definition within the distribution of data collected from the web, and a sufficiently large model can\nlearn that distribution. This strategy may not be effective for non-web-scraped datasets, where there\nis no guarantee that it is possible to train on data that is close in distribution to the target data, even\nif we train on the entire web. Thus, it makes sense to distinguish these two types of datasets.\" -> While the argument has some logic, I do not find the distinction qualitatively very relevant. There might be a lot to argue. For instance, Camera Traps are a very common object now and many people are already posting it on the web, making the so called web distribution constantly changing.\n\n\"If improving ImageNet accuracy alone is insufficient to reach these well-tuned baselines, we can indeed conclude that architecture transfer to this target task is limited.\" -> Well, this is quite dubious. The improvment the authors perform may lack the power of the training techniques that have been used for these well-tuned baselines. These well-tuned baselines result from specific training approaches that may hypothetically depend on the task at hand, e.g., when employing curriculum learning (which is very much used for real-world applications), and hence not very much fall in the scope of the type of fine-tuning the authors are performing. I would hence be much more careful in their statement.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KT9jQsu-3k", "reviewer": "Reviewer_QVyG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work investigates whether models that perform well on the ImageNet dataset also perform as well on real world datasets. By real world datasets they mean datasets that were not scraped off the internet, their investigation includes datasets such as Caltech Camera Traps-20, the Human Protein Atlas Image Classification, EuroSat, APTOS 2019 Blindness, SIIM-ISIC Melanoma, Cassava leave datasets. They extensively fine tune models pre-trained on ImageNet using these datasets and report  that architectures that do well on ImageNet do not necessarily do well on other real world datasets and that the newer models that do well on ImageNet perform worse on these real world datasets than the older ones like VGG Networks.", "summary_of_the_review": "I think the authors did a good job focusing on transferability of these real-world tasks and consolidating the experiments of diverse datasets into one paper. However, I am concerned about the fact that they do not mention other authors who ran some of their experiments on the same real world datasets that they used on some of the same architectures pretrained on ImageNet also. Overall I think this was a well written paper and all of their experiments are clear and sound easy to follow.", "strength_and_weaknesses": "Strengths: \n--The work digs deeper into making the models work for these real world datasets. I really appreciate that the authors focused their efforts on a few datasets and focused their efforts on different model configurations before coming to their conclusions.\n--The paper is well written and easy to follow.\n--Their findings are some of the things the community has been talking about but had never really been scientifically proven extensively\n\nWeaknesses:\n-- I would have wanted to hear more about how your models compare to pre-existing models pretrained on ImageNet and finetuned with the same datasets that you tested in your experiments. Because it is almost like you are disregarding all the work that exists were other researchers have used ImageNet pretrained models(like how you did) on these same real world datasets. Such as: Naushad, R., Kaur, T. and Ghaderpour, E., 2021. Deep transfer learning for land use and land cover classification: A comparative study. Sensors, 21(23), p.8083.\nWhat are your comments on these previous works? Why did you just not take their published recorded accuracies instead of re-doing some of their experiments?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "26Kt2mQ9RN", "reviewer": "Reviewer_yyk7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates whether models trained on ImageNet with increased accuracy transfer their improved accuracy to multiple datasets. Many models are investigated on a benchmark of six publicly available datasets that try to capture the characteristics practical classification applications. The main finding is that models with higher ImageNet accuracy do not consistently yield performance improvements. ", "summary_of_the_review": "Overall, the problem tackled in this paper is a challenging one as it attempts to find correlations of real-world tasks with ImageNet trained models. This is difficult to accomplish given that different datasets have different characteristics. But such studies are important, especially to understand the effect that improvements on ImageNet can lead to concrete improvements on real world applications. I fully agree with the final sentence of the paper that it is necessary to have a more diverse set of benchmarks that can improve learning algorithms more comprehensively. Such studies are thus important for the community.", "strength_and_weaknesses": "Strengths:\n- This paper demonstrates that ImageNet pretrained models do not always lead to better accuracy for certain applications. Thus it highlights that architectural innovations might be overfitting on the ImageNet data highlighting the need for better benchmarks.\n- A variety of models exhibiting large imagenet accuracy variance are studied. \n- The datasets under investigation can provide a useful initial benchmark for testing models on a variety of classification applications. \n\nWeaknesses:\n- The choice of the six datasets while practical is not inclusive. For instance, there are a number of additional applications such as robotics (I am thinking indoor classification,), sports, or automotive applications that could also have been added. All of which can be classified as real-world applications that may more closely associate with ImageNet. I believe that the methodology of dataset selection can be made more formal while also including some dataset statistics. \n-  The work fails to also highlight another important of training using pretrained weights which is convergence time. Even though imagenet pretraining alone might not lead to better results, would a model trained from scratch also arrive to the same accuracy?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject