SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alex Fang
University of Washington apf1@cs.washington.edu
Simon Kornblith*
Google Research, Brain Team skornblith@google.com
Ludwig Schmidt*
University of Washington, AI2 schmidt@cs.washington.edu
Does progress on ImageNet transfer to real-world datasets? We investigate this question by evaluating ImageNet pre-trained models with varying accuracy (57% - 83%) on six practical image classification datasets. In particular, we study datasets collected with the goal of solving real-world tasks (e.g., classifying images from camera traps or satellites), as opposed to web-scraped benchmarks collected for comparing models. On multiple datasets, models with higher ImageNet accuracy do not consistently yield performance improvements. For certain tasks, interventions such as data augmentation improve performance even when architectures do not. We hope that future benchmarks will include more diverse datasets to encourage a more comprehensive approach to improving learning algorithms.
ImageNet is one of the most widely used datasets in machine learning. Initially, the ImageNet competition played a key role in re-popularizing neural networks with the success of AlexNet in 2012. Ten years later, the ImageNet dataset is still one of the main benchmarks for state-of-the-art computer vision models [35, 61, 20, 37, 24, 69, 52]. As a result of ImageNet's prominence, the machine learning community has invested tremendous effort into developing model architectures, training algorithms, and other methodological innovations with the goal of increasing performance on ImageNet. Comparing methods on a common task has important benefits because it ensures controlled experimental conditions and results in rigorous evaluations. But the singular focus on ImageNet also raises the question whether the community is over-optimizing for this specific dataset.
As a first approximation, ImageNet has clearly encouraged effective methodological innovation beyond ImageNet itself. For instance, the key finding from the early years of ImageNet was that large convolution neural networks (CNNs) can succeed on contemporary computer vision datasets by leveraging GPUs for training. This paradigm has led to large improvements in other computer vision tasks, and CNNs are now omnipresent in the field. Nevertheless, this clear example of transfer to other tasks early in the ImageNet evolution does not necessarily justify the continued focus ImageNet still receives. For instance, it is possible that early methodological innovations transferred more broadly to other tasks, but later innovations have become less generalizable. The goal of our paper is to investigate this possibility specifically for neural network architecture and their transfer to real-world data not commonly found on the Internet.
When discussing the transfer of techniques developed for ImageNet to other datasets, a key question is what other datasets to consider. Currently there is no comprehensive characterization of the many machine learning datasets and transfer between them. Hence we restrict our attention to a limited but well-motivated family of datasets. In particular, we consider classification tasks derived from image data that were specifically collected with the goal of classification in mind. This is in contrast to many standard computer vision datasets – including ImageNet – where the constituent
Figure 1: Overview of transfer performance across models from ImageNet to each of the datasets we study. Although there seems to be a strong linear trends between ImageNet accuracy and the target metrics (green), these trends become less certain when we restrict the models to those above  $70\%$  ImageNet accuracy (blue). Versions with error bars and spline interpolation can be found in Appendix B.
images were originally collected for a different purpose, posted to the web, and later re-purposed for benchmarking computer vision methods. Concretely, we study six datasets ranging from leaf disease classification over melanoma detection to categorizing animals in camera trap images. Since these datasets represent real-world applications, transfer of methods from ImageNet is particularly relevant.
We find that on four out of our six real-world datasets, ImageNet-motivated architecture improvements after VGG resulted in little to no progress (see Figure 1). Specifically, when we fit a line to downstream model accuracies as a function of ImageNet accuracy, the resulting slope is less than 0.05. The two exceptions where post-VGG architectures yield larger gains are the Caltech Camera Traps-20 (CCT-20) [3] dataset (slope 0.11) and the Human Protein Atlas Image Classification [47] dataset (slope 0.29). On multiple other datasets, we find that task-specific improvements such as data augmentations or extra training data lead to larger gains than using a more recent ImageNet architecture. We evaluate on a representative testbed of 19 ImageNet models, ranging from the seminal AlexNet [35] over VGG [61] and ResNets [20] to the more recent and higher-performing EfficientNets [66] and ConvNexts [39] (ImageNet top-1 accuracies  $56.5\%$  to  $83.4\%$ ). Our testbed includes three Vision Transformer models to cover non-CNN architectures.
Interestingly, our findings stand in contrast to earlier work that investigated image classification benchmarks such as CIFAR-10 [34], PASCAL VOC 2007 [14], and Caltech-101 [15] that were scraped from the Internet. On these datasets, Kornblith et al. [31] found consistent gains in downstream task accuracy for a similar range of architectures as we study in our work. Taken together, these findings indicate that ImageNet accuracy is a good predictor for other web-scaped datasets, but less informative for many real-world image classification datasets that are not sourced through the web. On the other hand, the CCT-20 data point shows that even very recent ImageNet models do help on some downstream tasks that do not rely on images from the web. Overall, our results highlight the need for a more comprehensive understanding of machine learning datasets to build and evaluate broadly useful data representations. We provide sample training code and dataset information at https://github.com/mlfondations/imagenet-applications-transfer to ensure reproducibility and encourage future research in this direction.
Transferability of ImageNet architectures. Although there is extensive previous work investigating the effect of architecture upon the transferability of ImageNet-pretrained models to different datasets, most of this work focuses on performance on datasets collected for the purpose of benchmarking. Kornblith et al. [31] previously showed that ImageNet accuracy of different models is strongly correlated with downstream accuracy on a wide variety of web-scraped object-centric computer vision benchmark tasks. Later studies have investigated the relationship between ImageNet and transfer accuracy for self-supervised networks [13, 33, 46], adversarially trained networks [56], or networks trained with different loss functions [32], but still evaluate primarily on web-scraped benchmark tasks. The Visual Task Adaptation Benchmark (VTAB) [81] comprises a more diverse set of tasks, including natural and non-natural classification tasks as well as non-classification tasks, but nearly all consist of web-scraped or synthetic images. In the medical imaging domain, models have been extensively evaluated on real-world data, with limited gains from newer models that perform better on ImageNet [53, 5, 29].
Most closely related to our work, Tuggener et al. [71] investigate performance of 500 CNN architectures on yet another set of datasets, several of which are not web-scraped, and find that accuracy correlates poorly with ImageNet accuracy when training from scratch, but correlations are higher when fine-tuning ImageNet-pretrained models. Our work differs from theirs in our focus solely on real-world datasets (e.g., from Kaggle competitions) and in that we perform extensive tuning in order to approach the best single-model performance obtainable on these datasets whereas Tuggener et al. [71] instead devote their compute budget to increasing the breadth of architectures investigated.
Transferability of networks trained on other datasets. Other work has evaluated transferability of representations of networks trained on datasets beyond ImageNet. Most notably, Abnar et al. [1] explore the relationship between upstream and downstream accuracy for models pretrained on JFT and ImageNet-21K and find that, on many tasks, downstream accuracy saturates with upstream accuracy. However, they evaluate representational quality using linear transfer rather than end-to-end fine-tuning. Other studies have investigated the impact of relationships between pretraining and fine-tuning tasks [80, 43] or the impact of scaling the model and dataset [16, 30].
Another direction of related work relates to the effect of pretraining data on transfer learning. Huh et al. [26] look into the factors that make ImageNet good for transfer learning. They find that fine-grained classes are not needed for good transfer performance, and that reducing the dataset size and number of classes only results in slight drops in transfer learning performance. Though there is a common goal of exploring what makes transfer learning work well, our work differs from this line of work by focusing on the fine-tuning aspect of transfer learning.
Other studies of external validity of benchmarks. Our study fits into a broader literature investigating the external validity of image classification benchmarks. Early work in this area identified lack of diversity as a key shortcoming of the benchmarks of the time [51, 68], a problem that was largely resolved with the introduction of the much more diverse ImageNet benchmark [9, 55]. More recent studies have investigated the extent to which ImageNet classification accuracy correlates with accuracy on out-of-distribution (OOD) data [54, 67] or accuracy as measured using higher-quality human labels [57, 70, 4].
As in previous studies of OOD generalization, transfer learning involves generalization to test sets that differ in distribution from the (pre-)training data. However, there are also key differences between transfer learning and OOD generalization. First, in transfer learning, additional training data from the target task is used to adapt the model, while OOD evaluations usually apply trained models to a new distribution without any adaptation. Second, OOD evaluations usually focus on settings with a shared class space so that evaluations without adaptation are possible. In contrast, transfer learning evaluation generally involves downstream tasks with classes different from those in the pretraining dataset. These differences between transfer learning and OOD generalization are not only conceptual but also lead to different empirical phenomena. Miller et al. [44] has shown that in-distribution accuracy improvements often directly yield out-of-distribution accuracy improvements as well. This is the opposite of our main experimental finding that ImageNet improvements do not directly yield performance improvements on many real-world downstream tasks. Hence our work demonstrates an important difference between OOD generalization and transfer learning.
As mentioned in the introduction, a key choice in any transfer study is the set of target tasks on which to evaluate model performance. Before we introduce our suite of target tasks, we first describe three criteria that guided our dataset selection: (i) diverse data sources, (ii) relevance to an application, and (iii) availability of well-tuned baseline models for comparison.
3.1 Selection criteria
Prior work has already investigated transfer of ImageNet architectures to many downstream datasets [10, 58, 6, 61]. The 12 datasets used by Kornblith et al. [31] often serve as a standard evaluation suite (e.g., in [56, 13, 52]). While these datasets are an informative starting point, they are all object-centric natural image datasets, and do not represent the entire range of image classification problems. There are many applications of computer vision; the Kaggle website alone lists more than 1,500 datasets as of May 2022. To understand transfer from ImageNet more broadly, we selected six datasets guided by the following criteria.
Diverse data sources. Since collecting data is an expensive process, machine learning researchers often rely on web scraping to gather data when assembling a new benchmark. This practice has led to several image classification datasets with different label spaces such as food dishes, bird species, car models, or other everyday objects. However, the data sources underlying these seemingly different tasks are actually often similar. Specifically, we surveyed the 12 datasets from Kornblith et al. [31] and found that all of these datasets were harvested from the web, often via keyword searches in Flickr, Google image search, or other search engines (see Appendix J). This narrow range of data sources limits the external validity of existing transfer learning experiments. To get a broader understanding of transfer from ImageNet, we focus on scientific, commercial, and medical image classification datasets that were not originally scraped from the web.
Application relevance. In addition to the data source, the classification task posed on a given set of images also affects how relevant the resulting problem is for real-world applications. For instance, it would be possible to start with real-world satellite imagery that shows multiple building types per image, but only label one of the building types for the purpose of benchmarking (e.g., to avoid high annotation costs). The resulting task may then be of limited value for an actual application involving satellite images that requires all buildings to be annotated. We aim to avoid such pitfalls by limiting our attention to tasks that were assembled by domain experts with a specific application in mind.
Availability of baselines. If methodological progress does not transfer from ImageNet to a given target task, we should expect that, as models perform better on ImageNet, accuracy on the target task saturates. However, observing such a trend in an experiment is not sufficient to reach a conclusion regarding transfer because there is an alternative explanation for this empirical phenomenon. Besides a lack of transfer, the target task could also simply be easier than the source task so that models with sub-optimal source task accuracy already approach the Bayes error rate. As an illustrative example, consider MNIST as a target task for ImageNet transfer. A model with mediocre ImageNet accuracy is already sufficient to get  $99\%$  accuracy on MNIST, but this finding does not mean that better ImageNet models are insufficient to improve MNIST accuracy — the models have already hit the MNIST performance ceiling.
More interesting failures of transfer occur when ImageNet architectures plateau on the target task, but it is still possible to improve accuracy beyond what the best ImageNet architecture can achieve without target task-specific modifications. In order to make such comparisons, well-tuned baselines for the target task are essential. If improving ImageNet accuracy alone is insufficient to reach these well-tuned baselines, we can indeed conclude that architecture transfer to this target task is limited. In our experiments, we use multiple datasets from Kaggle competitions since the resulting leaderboards offer well-tuned baselines arising from a competitive process.
3.2 Datasets studied
The datasets studied in this work are practical and cover a variety of applications. We choose four of the most popular image classification competitions on Kaggle, as measured by number of competitors, teams, and submissions. Each of these competitions is funded by an organization with the goal of advancing performance on that real-world task. Additionally, we supplement these datasets with Caltech Camera Traps [3] and EuroSAT [21] to broaden the types of applications studied. The datasets we study are all under 50,000 training images, potentially due to the cost of collecting and
Table 1: We examine a variety of real-world datasets that cover different types of tasks.
Dataset	# of classes	Train size	Eval size	Eval metric	Kaggle
Caltech Camera Traps	15	14,071	15,215	Accuracy
APTOS 2019 Blindness	5	2,930	732	Quadratic weighted kappa	✓
Human Protein Atlas	28	22,582	5,664	Macro F1 score	✓
SIIM-ISIC Melanoma	2	46,372	11,592	Area under ROC	✓
Cassava Leaf Disease	5	17,118	4,279	Accuracy	✓
EuroSAT	10	21,600	5,400	Accuracy
Caltech Camera Traps-20
APTOS 2019 Blindness Detection
SIIM-ISIC Melanoma Classification
Cassava Leaf Disease Classification
Human Protein Atlas Image Classification
EuroSAT
Figure 2: Sample images from each of the datasets.
annotating real-world data, and our focus on transfer from ImageNet limits this study to RGB datasets. Details for each dataset can be found in Table  $1^{2}$
4 Main Experiments
We run our experiments across 19 model architectures, including both CNNs and Vision Transformers (ViT and DeiT). They range from  $57\%$  to  $83\%$  ImageNet top-1 accuracy, allowing us to observe the relationship between ImageNet performance and target dataset performance.
Table 2: We summarize the blue regression lines from Figure 1, calculated on models above  $70\%$  ImageNet accuracy, with their correlation and slope. Slope is calculated so that all metrics have a range from 0 to 100.
Dataset	Correlation	Slope
Caltech Camera Traps	0.17	0.11
APTOS 2019 Blindness	0.06	0.01
Human Protein Atlas	0.26	0.29
SIIM-ISIC Melanoma	0.44	0.05
Cassava Leaf Disease	0.12	0.02
EuroSAT	0.05	0.00
$^{2}$ Dataset download links and PyTorch datasets and splits can be found at https://github.com/mlfoundations/imagenet-applications-transfer.
4.1 Caltech Camera Traps
Beery et al. [3] created Caltech Camera Traps-20 (CCT-20) using images taken from camera traps deployed to monitor animal populations. The images contain 15 animal classes, as well as an empty class that we remove for our experiments<sup>3</sup>. The dataset has two sets of validation and test sets which differ by whether they come from locations that are the same as or different from the training set locations. While one of the goals of the dataset is to study generalization to new environments, here we only study the sets from the same locations. Though CCT-20 is not a Kaggle competition, it is a subset of iWildCam Challenge 2018, whose yearly editions have been hosted on Kaggle.
We see in Figure 1 (top-left) an overall positive trend between ImageNet performance and CCT-20 performance. The overall trend is unsurprising, given the number of animal classes present in ImageNet. But despite the drastic reduction in the number of classes when compared to ImageNet, CCT-20 has its own set of challenges. Animals are often pictured at difficult angles, and sometimes are not even visible in the image because a sequence of frames triggered by activity all have the same label. Despite these challenges, an even higher performing model still does better on this task - we train a CLIP ViT L/14-336px model (85.4% ImageNet top-1) with additional augmentation to achieve 83.4% accuracy on CCT-20.
4.2 APTOS 2019 Blindness Detection
This dataset was created for a Kaggle competition run by the Asia Pacific Tele-Ophthalmology Society (APTOS) with the goal of advancing medical screening for diabetic retinopathy in rural areas [2]. Images are taken using fundus photography and vary in terms of clinic source, camera used, and time taken. Images are labeled by clinicians on a scale of 0 to 4 for the severity of diabetic retinopathy. Given the scaled nature of the labels, the competition uses quadratic weighted kappa (QWK) as the evaluation metric. We create a local  $80\%$  to  $20\%$  random class-balanced train/validation split, as the competition test labels are hidden.
We find that models after VGG do not show significant improvement. Similar to in CCT-20, DeiT and EfficientNets performs slightly worse, while deeper models from the same architecture slightly help performance. We also find that accuracy has a similar trend as QWK, despite it being an inferior metric in the context of this dataset.
When performance stagnates, one might ask whether we have reached a performance limit for our class of models on the dataset. To answer this question, we compare with the Kaggle leaderboard's top submissions. The top Kaggle submission achieves 0.936 QWK on the private leaderboard (85% of the test set) [75]. They do this by using additional augmentation, using external data, training on L1-loss, replacing the final pooling layer with generalized mean pooling, and assembling a variety of models trained with different input sizes. The external data consists of 88,702 images from the 2015 Diabetic Retinopathy Detection Kaggle competition.
Even though performance saturates with architecture, we find that additional data augmentation and other interventions still improve accuracy. We submitted our ResNet-50 and ResNet-152 models with additional interventions, along with an Inception-ResNet v2 [65] model with hyperparameter tuning. We find that increasing color and affine augmentation by itself can account for a 0.03 QWK point improvement. Once we train on 512 input size, additional augmentation, and additional data, our ResNet-50 and Inception-ResNet v2 both achieve 0.896 QWK on the private leaderboard, while ResNet-152 achieves 0.890 QWK, once again suggesting that better ImageNet architectures by themselves do not lead to increased performance on this task.
As a comparison, the ensemble from the top leaderboard entry included a single model Inception-ResNet v2 trained with additional interventions that achieves 0.927 QWK. We submitted the original models we trained to Kaggle as well, finding that the new models trained with additional interventions do at least 0.03 QWK points better. See Appendix F for additional experimental details. Both this result and the gap between our models and the top leaderboard models show that there exist interventions that do improve task performance.
4.3 Human Protein Atlas Image Classification
The Human Protein Atlas runs the Human Protein Atlas Image Classification competition on Kaggle to build an automated tool for identifying and locating proteins from high-throughput microscopy
${}^{3}$  Empty class is removed for the classification experiments in Table 1 of Beery et al. [3]
images [47]. Images can contain multiple of the 28 different proteins, so the competition uses the macro F1 score. Given the multi-label nature of the problem, this requires thresholding for prediction. We use a  $73\% / 18\% / 9\%$  train/ validation/test-validation split created by a previous competitor [49]. We report results on the validation split, as we find that the thresholds selected for the larger validation split generalize well to the smaller test-validation split.
We find a slightly positive trend between task performance and ImageNet performance, even when ignoring AlexNet and MobileNet. This is surprising because ImageNet is quite visually distinct from human protein slides. These results suggest that models with more parameters help with downstream performance, especially for tasks that have a lot of room for improvement.
Specific challenges for this dataset are extreme class imbalance, multi-label thresholding, and generalization from the training data to the test set. Competitors were able to improve performance beyond the baselines we found by using external data as well as techniques such as data cleaning, additional training augmentation, test time augmentation, ensembling, and oversampling [8, 49, 59]. Additionally, some competitors modified commonly-used architectures by substituting pooling layers or incorporating attention [49, 83]. Uniquely, the first place solution used metric learning on top of a single DenseNet121 [8]. These techniques may be useful when applied to other datasets, but are rarely used in a typical workflow.
4.4 SIIM-ISIC Melanoma Classification