SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The Society for Imaging Informatics in Medicine (SIIM) and the International Skin Imaging Collaboration (ISIC) jointly ran this Kaggle competition for identifying Melanoma [60], a serious type of skin cancer. Competitors use images of skin lesions to predict the probability that each observed image is malignant. Images come from the ISIC Archive, which is publicly available and contains images from a variety of countries. The competition provided 33,126 training images, plus an additional 25,331 images from previous competitions. We split the combined data into an  $80\%$  to  $20\%$  class-balanced and year-balanced train/validation split. Given the imbalanced nature of the data  $(8.8\%)$  positive), the competition uses area under ROC curve as the evaluation metric.
We find only a weak positive correlation (0.44) between ImageNet performance and task performance, with a regression line with a normalized slope of close to zero (0.05). But if we instead look at classification accuracy, Appendix G shows that there is a stronger trend for transfer than that of area under ROC curve, as model task accuracy more closely follows the same order as ImageNet performance. This difference shows that characterizing the relationship between better ImageNet models and better transfer performance is reliant on the evaluation metric as well. We use a relatively simple setup to measure the impact of ImageNet models on task performance, but we know we can achieve better results with additional strategies. The top two Kaggle solutions used models with different input size, ensembling, cross-validation and a significant variety of training augmentation to create a stable model that generalized to the hidden test set [17, 48].
4.5 Cassava Leaf Disease Classification
The Makerere Artificial Intelligence Lab is an academic research group focused on applications that benefit the developing world. Their goal in creating the Cassava Leaf Disease Classification Kaggle competition [42] was to give farmers access to methods for diagnosing plant diseases, which could allow farmers to prevent these diseases from spreading, increasing crop yield. Images were taken with an inexpensive camera and labeled by agricultural experts. Each image was classified as healthy or as one of four different diseases. We report results using a  $80\% / 20\%$  random class-balanced train/Validation split of the provided training data.
Once we ignore models below  $70\%$  ImageNet accuracy, the relationship between the performance on the two datasets has both a weak positive correlation (0.12) and a near-zero normalized slope (0.02). While these are natural images similar to portions of ImageNet, it is notable that ImageNet contains very few plant classes (e.g., buckeye, hip, rapeseed). Yet based on a dataset's perceived similarity to ImageNet, it is surprising that leaf disease classification is not positively correlated with ImageNet, while the microscopy image based Human Protein Atlas competition is. Our results are supported by Kaggle competitors: the first place solution found that on the private leaderboard, EfficientNet B4 [66], MobileNet, and ViT [12] achieve  $89.5\%$ ,  $89.4\%$ , and  $88.8\%$  respectively [19]. Their ensemble achieves  $91.3\%$  on the private leaderboard.
Table 3: We examine the effect of pre-training augmentation and fine-tuning augmentation on downstream transfer performance. The model specifies the architecture and pre-training augmentation, while each column specifies the downstream task and fine-tuning augmentation. We find that augmentation strategies that improve ImageNet accuracy do not always improve accuracy on downstream tasks. Pre-trained augmentation models are from Wightman et al. [72].
Model	ImageNet Acc	CCT-20 Base Aug	CCT-20 AugMix	CCT-20 RandAug	APTOS Base Aug	APTOS AugMix	APTOS RandAug
ResNet-50	76.1	72.02	72.24	73.57	0.9210	0.9212	0.9250
ResNet-50 w/ AugMix	77.5	71.63	71.53	72.39	0.9239	0.9152	0.9222
ResNet-50 w/ RandAug	78.8	72.94	73.54	73.76	0.9190	0.9204	0.9302
DeiTtiny	72.2	66.57	66.47	66.95	0.9153	0.9197	0.9172
DeiT-small	79.9	70.65	69.72	70.07	0.9293	0.9212	0.9277
4.6 EuroSAT
Helber et al. [21] created EuroSAT from Sentinel-2 satellite images to classify land use and land cover. Past work has improved performance on the dataset through additional training time techniques [45] and using 13 spectral bands [76]. We use RGB images and keep our experimental setup consistent to compare across a range of models. Since there is no set train/test split, we create a  $80\% / 20\%$  class-balanced split.
All models over  $60\%$  ImageNet accuracy achieve over  $98.5\%$  EuroSAT accuracy, and the majority of our models achieve over  $99.0\%$  EuroSAT accuracy. There are certain tasks where using better ImageNet models does not improve performance, and this would be the extreme case where performance saturation is close to being achieved. While it is outside the scope of this study, a next step would be to investigate the remaining errors and find other methods to reduce this last bit of error.
5 Additional Studies
5.1 Augmentation ablations
In our main experiments, we keep augmentation simple to minimize confounding factors when comparing models. However, it is possible pre-training and fine-tuning with different combinations of augmentations may have different results. This is an important point because different architectures may have different inductive biases and often use different augmentation strategies at pre-training time. To investigate these effects, we run additional experiments on CCT-20 and APTOS to explore the effect of data augmentation on transfer. Specifically, we take ResNet-50 models pre-trained with standard crop and flip augmentation, AugMix [22], and RandAugment [7], and then fine-tune on our default augmentation, AugMix, and RandAugment. We also study DeiT-tiny and Deit-small models by fine-tuning on the same three augmentations mentioned above. We choose to examine DeiT models because they are pre-trained using RandAugment and RandErasing [84]. We increase the number of epochs we fine-tune on from 30 to 50 to account for augmentation. Our experimental results are found in Table 3.
In our ResNet-50 experiments, both AugMix and RandAugment improve performance on ImageNet, but while pre-training with RandAugment improves performance on downstream tasks, pre-training with AugMix does not. Furthermore, fine-tuning with RandAugment usually yields additional performance gains when compared to our default fine-tuning augmentation, no matter which pretrained model is used. For DeiT models, we found that additional augmentation did not significantly increase performance on the downstream tasks. Thus, as with architectures, augmentation strategies that improve accuracy on ImageNet do not always improve accuracy on real-world tasks.
5.2 CLIP models
A natural follow-up to our experiments is to change the source of pre-training data. We examine CLIP models from Radford et al. [52], which use diverse pre-training data and achieve high performance on a variety of downstream datasets. We fine-tune CLIP models on each of our downstream datasets
by linear probing then fine-tuning (LP-FT) [36].<sup>4</sup> Our results are visualized by the purple stars in Appendix H Figure 8. We see that by using a model that takes larger images we can do better than all previous models, and even without the larger images, ViT-L/14 does better on four out of the six datasets. While across all CLIP models the change in pre-training data increases performance for CCT-20, the effect on the other datasets is more complicated. When controlling for architecture changes by only looking at ResNet-50 and ViT/B16, we see that the additional pre-training data helps for CCT-20, HPA, and Cassava, the former two corresponding to the datasets that empirically benefit most from using better ImageNet models. Additional results can be found in Appendix H, while additional fine-tuning details can be found in Appendix I.
Alternative explanations for saturation. Whereas Kornblith et al. [31] reported a high degree of correlation between ImageNet and transfer accuracy, we find that better ImageNet models do not consistently transfer better on our real-world tasks. We believe these differences are related to the tasks themselves. Here, we rule out alternative hypotheses for our findings.
Comparison of datasets statistics suggests that the number of classes and dataset size also do not explain the differences from Kornblith et al. [31]. The datasets we study range from two to 28 classes. Although most of the datasets studied in Kornblith et al. [31] have more classes, CIFAR-10 has 10. In Appendix E, we replicate CIFAR-10 results from Kornblith et al. [31] using our experimental setup, finding a strong correlation between ImageNet accuracy and transfer accuracy. Thus, the number of classes is likely not the determining factor. Training set sizes are similar between our study and that of Kornblith et al. [31] and thus also do not seem to play a major role. A third hypothesis is that it is parameter count, rather than ImageNet accuracy, that drives trends. We see that VGG BN models appear to outperform their ImageNet accuracy on multiple datasets, and they are among the largest models by parameter count. However, in Appendix K, we find that model size is also not a good indicator of improved transfer performance on real world datasets.
Differences between web-scraped datasets and real-world images We conjecture that it is possible to perform well on most, if not all, web-scraped target datasets simply by collecting a very large amount of data from the Internet and training a very large model on it. Web-scraped target datasets are by definition within the distribution of data collected from the web, and a sufficiently large model can learn that distribution. In support of this conjecture, recent models such as CLIP [52], ALIGN [28], ViT-G [82], BASIC [50], and CoCa [78] are trained on such datasets and achieve high accuracy on many web-scraped benchmarks. But this strategy may not be effective for non-web-scraped datasets, as there is no guarantee that we will train on data that is close in distribution to the target data, even if we train on the entire web. Thus, it makes sense to distinguish these two types of datasets.
There are clear differences in image distribution between the non-web-scraped datasets we consider and web-scraped datasets considered by previous work. In Figure 3 and Appendix L, we compute
Frechet inception distance (FID) [23] between ImageNet and each of the datasets we study in this work as well as the ones found in Kornblith et al. [31]. The real-world datasets are further from ImageNet than datasets in Kornblith et al. [31], implying that there is a large amount of distribution shift between web-scraped datasets and real-world datasets. However, FID is only a proxy measure and may not capture all factors that lead to differences in transferability.
Whereas web-scaped data is cheap to acquire, real-world data can be more expensive. Ideally, progress in computer vision architectures should improve performance not just on web-scaped data, but also on real-world tasks. Our results suggest that the latter has not happened. Gains in ImageNet accuracy over proving and scaling architectures, and past work has web-scaped datasets, regardless of size [63, 31, 41,
Figure 3: FID scores vs ImageNet for the datasets we study in this work (red), and the web-scraped datasets studied by Kornblith et al. [31] (blue).
<sup>4</sup>We use LP-FT because, in past experiments, we have found that LP-FT makes hyperparameter tuning easier for CLIP models, but does not significantly alter performance when using optimal hyperparameters.
arising from architecture generally do not transfer to non-web-scraped tasks. Nonetheless, data augmentation and other tweaks can provide further gains on these tasks.
Recommendations towards better benchmarking. While it is unclear whether researchers have over-optimized for ImageNet, our work suggests that researchers should explicitly search for methods that improve accuracy on real-world non-web-scraped datasets, rather than assuming that methods that improve accuracy on ImageNet will provide meaningful improvements on real-world datasets as well. Just as there are methods that improve accuracy on ImageNet but not on the tasks we investigate, there may be methods that improve accuracy on our tasks but not ImageNet. The Kaggle community provides some evidence for the existence of such methods; Kaggle submissions often explore architectural improvements that are less common in traditional ImageNet pre-trained models. To measure such improvements on real-world problems, we suggest simply using the average accuracy across our tasks as a benchmark for future representation learning research.
Further analysis of our results shows consistencies in the accuracies of different models across the non-web-scraped datasets, suggesting that accuracy improvements on these datasets may translate to other datasets. For each dataset, we use linear regression to predict model accuracies on the target dataset as a linear combination of ImageNet accuracy and accuracy averaged across the other real-world datasets. We perform an F-test to determine whether the average accuracy on other real-world datasets explains significant variance beyond that explained by ImageNet accuracy. We find that this F-test is significant on all datasets except EuroSAT, where accuracy may be very close to ceiling (see further analysis in Appendix M.1). Additionally, in Appendix M.2 we compare the Spearman rank correlation (i.e., the Pearson correlation between ranks) between each dataset and the accuracy averaged across the other real-world datasets to the Spearman correlation between each dataset and ImageNet. We find that the correlation with the average over real-world datasets is higher than the correlation with ImageNet and statistically significant for CCT-20, APTOS, HPA, and Cassava. Thus, there is some signal in the average accuracy across the datasets that we investigate that is not captured by ImageNet top-1 accuracy.
Where do our findings leave ImageNet? We suspect that most of the methodological innovations that help on ImageNet are useful for some real-world tasks, and in that sense it has been a successful benchmark. However, the innovations that improve performance on industrial web-scraped datasets such as JFT [63] or IG-3.5B-17k [41] (e.g., model scaling) may be almost entirely disjoint from the innovations that help with the non-web-scraped real-world tasks studied here (e.g., data augmentation strategies). We hope that future benchmarks will include more diverse datasets to encourage a more comprehensive approach to improving learning algorithms.
We would like to thank Samuel Ainsworth, Sara Beery, Gabriel Ilharco, Pieter-Jan Kindermans, Sarah Pratt, Matthew Wallingford, Ross Wightman, and Mitchell Wortsman for valuable conversations while working on this project. We would especially like to thank Sarah Pratt for help with early experimentation and brainstorming.
We would also like to thank Hyak computing cluster at the University of Washington and the Google TPU Research Cloud program for access to compute resources that allowed us to run our experiments.
This work is in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML), Open Philanthropy, Google, and the Allen Institute for AI.
[1] Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=V3C8p78sDa.
[2] Asia Pacific Tele-Ophthalmology Society. Aptos 2019 blindness detection, 2019. URL https://www.kaggle.com/competitions/aptos2019-blindness-detection/overview.
[3] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XVI, volume 11220 of Lecture Notes in Computer Science, pages 472-489.
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
- Did you include the license to the code and datasets? [Yes] See Section ??
- Did you include the license to the code and datasets? [No] The code and the data are proprietary.
- Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss the scope and limitations of this work in the Discussion section
(c) Did you discuss any potential negative societal impacts of your work? [No] However beyond that of generally studying machine learning and image classification, additional societal impact of the work is limited.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Data links and splits (as torch datasets) are provided in a github URL. Training details needed to reproduce the main experiments are contained in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Training details and hyperparameters are specified in various sections in the appendix (e.g. Appendix D). Data splits are reproducible through code provided in a github URL.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We include confidence intervals in figures.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] We mention type of resources used in acknowledgements and Appendix D, but not the total amount of compute used.
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] However, the datasets we use are publicly available and allows usage for academic research
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] However, the data used does not contain either
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
A Detailed experiment results
Table 4: For each ImageNet pre-trained model, we provide the best performing model when fine-tuned on each dataset across our hyperparameter grid
Model	ImageNet top-1	CCT20	APTOS	HPA	Melanoma	Cassava	EuroSAT
AlexNet	56.5	63.59	0.8835	0.3846	0.9283	82.58	97.93
SqueezeNet 1.1	58.2	66.36	0.9021	0.3972	0.9073	85.15	98.07
ShuffleNetV2x0.5	60.6	66.37	0.9227	0.5867	0.9289	85.64	98.56
MobileNet V3 small	67.7	66.01	0.9230	0.6108	0.9455	85.81	99.15
ShuffleNetV2x1.0	69.4	69.27	0.9202	0.6202	0.9418	87.33	98.91
VGG-13 BN	71.6	75.06	0.9268	0.6794	0.9529	88.99	98.85
DeiT-tiny	72.2	68.77	0.9130	0.5777	0.9510	86.25	99.11
VGG-16 BN	73.4	75.93	0.9287	0.6791	0.9531	88.45	98.93
DenseNet-121	74.4	74.66	0.9287	0.7019	0.9514	87.80	99.06
ResNet-50	76.1	73.96	0.9215	0.6718	0.9524	87.75	99.19
ResNeXt-50-32x4d	77.6	73.73	0.9212	0.6906	0.9588	88.15	99.24
EfficientNet B0	77.7	71.02	0.9195	0.6942	0.9456	87.63	98.80
ResNet-152	78.3	74.05	0.9228	0.6732	0.9562	87.75	99.15
ViT-B/16	78.7	72.07	0.9262	0.5852	0.9600	86.63	99.28
DeiT-small	79.9	71.41	0.9205	0.6148	0.9583	87.19	99.20
Inception-ResNet v2	80.4	70.68	0.9168	0.6882	0.9483	87.84	98.93
ConvNext-tiny	82.5	78.51	0.9297	0.6992	0.9628	88.89	99.11
PNASNet-5 large	82.9	75.21	0.9271	0.6941	0.9584	87.77	99.17
EfficientNet B4	83.4	73.49	0.9211	0.6954	0.9552	88.36	98.70
See the following link for experiment results across hyperparameters: https://docs.google.com/spreadsheets/d/1aDeuTHOV1Kid_JMRUt3sF1N76LUCAMDQ007Ykjo3Z4U/edit?usp=sharing.
B Main figure variations
m MobileNetV3-small
VGG-13 BN
DeiT-tiny
ResNet-50
ResNet-152
DeiT-small
PNASNet-5
Inception-ResNet v2
VGG-16 BN
EfficientNet B0
EfficientNet B4
DenseNet-121
ResNeXt-50-32x4d
ShuffleNetV2x1.0
ConvNext-tiny
ShuffleNetV2x0.5
SqueezeNet 1.1
ViT-B/16
Figure 4: Figure 1 with error bars. Green is linear trend of all models, while blue is linear trend for models above  $70\%$  ImageNet accuracy. We use  $95\%$  confidence intervals computed with Clopper-Pearson for accuracy metrics and bootstrap with 10,000 trials for other metrics.
Figure 5: Figure 4 with spline interpolation fits instead of linear fits.
mMobileNetV3-small
VGG-13 BN
DeiT-tiny
ResNet-50
C Experiment setup
C.1 Models
Table 5: We examine the effectiveness of transfer learning from a number of models pretrained on ImageNet, including both CNNs and Vision Transformers.
Model	ImageNet top-1	# params	Year Released
AlexNet [35]	56.5	61M	2012
SqueezeNet 1.1 [27]	58.2	1.2M	2016
ShuffleNetV2x0.5 [40]	60.6	1.4M	2018
MobileNet V3 small [24]	67.7	2.5M	2019
ShuffleNetV2x1.0 [40]	69.4	2.3M	2018
VGG-13 BN [61]	71.6	133M	2014/2015
DeiTtiny [69]	72.2	5.7M	2020
VGG-16 BN [61]	73.4	138M	2014/2015
DenseNet-121 [25]	74.4	8.0M	2016
ResNet-50 [20]	76.1	26M	2015
ResNeXt-50-32x4d [74]	77.6	25M	2016
EfficientNet B0 [66]	77.7	5.3M	2019
ResNet-152 [20]	78.3	60M	2015
ViT-B/16 [11, 62]	78.7	304M	2020
DeiT-small [69]	79.9	22M	2020
Inception-ResNet v2 [64]	80.4	56M	2016
ConvNext-tiny [39]	82.5	29M	2022
PNASNet-5 large [37]	82.9	86M	2017
EfficientNet B4 [66]	83.4	19M	2019
We examine 19 model architectures in this work that cover a diverse range of accuracies on ImageNet in order to observe the relationship between ImageNet performance and target dataset performance. In addition to the commonly used CNNs, we also include data-efficient image transformers (DeiT) due to the recent increase in usage of Vision Transformers. Additional model details are in Table 5.
C.2 Hyperparameter Grid
Hyperparameter tuning is a key part of neural network training, as using suboptimal hyperparameters can lead to suboptimal performance. Furthermore, the correct hyperparameters vary across both models and training data. To get the best performance out of each model, we train each model on AdamW with a cosine decay learning rate schedule, SGD with a cosine decay learning rate schedule, and SGD with a multi-step decay learning rate schedule. We also grid search for optimal initial learning rate and weight decay combinations, searching logarithmically between  $10^{-1}$  to  $10^{-4}$  for SGD learning rate,  $10^{-2}$  to  $10^{-5}$  for AdamW learning rate, and  $10^{-3}$  to  $10^{-6}$  as well as 0 for weight decay. All models are pretrained on ImageNet and then fine-tuned on the downstream task. Additional training details for each dataset can be found in Appendix D. We also run our hyperparameter grid on CIFAR-10 in Appendix E to verify that we find a strong relationship between ImageNet and CIFAR-10 accuracy as previously reported by Kornblith et al. [31].
D Training details by dataset (ImageNet models)