SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Micah Goldblum $^{1}$  * Hossein Souri $^{2}$  * Renkun Ni $^{3}$  * Manli Shu $^{3}$  * Viraj Prabhu $^{4}$
Growthami Somepalli<sup>3</sup> Prithvijit Chattopadhyay<sup>4</sup> Mark Ibrahim<sup>6</sup> Adrien Bardes<sup>5,6</sup>
Judy Hoffman $^{4}$  Rama Chellappa $^{2}$  Andrew Gordon Wilson $^{1}$  Tom Goldstein $^{3}$
Neural network based computer vision systems are typically built on a backbone, a pretrained or randomly initialized feature extractor. Several years ago, the default option was an ImageNet-trained convolutional neural network. However, the recent past has seen the emergence of countless backbones pretrained using various algorithms and datasets. While this abundance of choice has led to performance increases for a range of systems, it is difficult for practitioners to make informed decisions about which backbone to choose. Battle of the Backbones (BoB) makes this choice easier by benchmarking a diverse suite of pretrained models, including vision-language models, those trained via self-supervised learning, and the Stable Diffusion backbone, across a diverse set of computer vision tasks ranging from classification to object detection to OOD generalization and more. Furthermore, BoB sheds light on promising directions for the research community to advance computer vision by illuminating strengths and weakness of existing approaches through a comprehensive analysis conducted on more than 1500 training runs. While vision transformers (ViTs) and self-supervised learning (SSL) are increasingly popular, we find that convolutional neural networks pretrained in a supervised fashion on large training sets still perform best on most tasks among the models we consider. Moreover, in apples-to-apples comparisons on the same architectures and similarly sized pretraining datasets, we find that SSL backbones are highly competitive, indicating that future works should perform SSL pretraining with advanced architectures and larger pretraining datasets. We release the raw results of our experiments along with code that allows researchers to put their own backbones through the gauntlet here: https://github.com/hsouri/Battle-of-the-Backbones.
The dominant paradigm for building machine vision systems involves a feature extractor network, also known as a backbone, which feeds into a task-specific head. The backbone might output a dense array of features for object detection and localization, or a single feature vector for classification or image retrieval. While backbones can be trained from scratch on task-specific data, many off-the-shelf backbones are pretrained on large benchmark datasets and then fine-tuned for the task at hand. This transfer learning approach has several advantages. First, it dramatically reduces the application-specific data requirements of deep learning and has led to improved performance on a wide range of
*Authors contributed equally. Correspondence to goldblum@nyu.edu and hsouri1@jhu.edu. This work was conducted at New York University<sup>1</sup>, Johns Hopkins University<sup>2</sup>, University of Maryland<sup>3</sup>, Georgia Institute of Technology<sup>4</sup>, Inria<sup>5</sup>, and Meta AI Research<sup>6</sup>.
applications. Second, it can speed up training and reduce compute costs even when large amounts of task-specific data are available [29]. Finally, pretraining datasets often contain images from many disparate domains, resulting in model robustness that can be transferred to downstream tasks.
Early deep learning based vision systems relied heavily on ImageNet pretraining [23, 59]. In contrast, today's practitioners have access to a cornucopia of choices, with different pretrained models resulting in significant performance differences. There are three primary factors that influence the performance of such a model: its architecture, the pretraining algorithm, and the pretraining dataset. Each of these design dimensions presents many options, resulting in a dizzying array of choices for practitioners building a computer vision system. Despite this wide variety of choices, practitioners have no resource to turn to and instead are left piecing together results from method papers or testing out the backbones themselves.
We pit these backbones against each other in a Battle of the Backbones (BoB). BoB compares many popular publicly available pretrained checkpoints, as well as randomly initialized baselines, on a wide variety of downstream tasks including image classification on natural, medical, and satellite images (Section 3.1), object detection and segmentation (Section 3.2), out-of-distribution generalization (Section 3.3), and image retrieval (Section 3.4).
Aside from assisting practitioners building computer vision systems, another central goal of this benchmark is to help guide the research community towards fruitful research directions in their quest for designing better backbones. BoB sheds light on the strengths and weaknesses of pretraining routines and architectures, revealing popular misconceptions and fundamental limitations, as well as promising directions for improvement. Below, we summarize several of our primary findings and discuss previous efforts for comparing backbones.
1.1 Battle of the Backbones: The TLDR
The subsequent sections in this paper contain numerous experimental details. Therefore, we distill several key findings below:
$\triangleright$  Across the suite of comprehensive evaluations in BoB, spanning tasks, datasets, and settings (including ID and OOD), supervised ConvNeXt-Base, supervised SwinV2-Base trained using ImageNet-21k, and CLIP ViT-Base come out on top. The same winners also win at smaller scales. Among smaller backbones, ConvNeXt-Tiny and SwinV2-Tiny emerge victorious, followed by DINO ViT-Small.
Figure 1: Performance is correlated across tasks. Performance for each model is reported in terms of standard deviations above/below the mean averages across datasets. Left: Comparison between classification and detection. Right: Comparison between classification and OOD classification.
$\triangleright$  Despite the recent attention paid to transformer-based architectures and self-supervised learning, high-performance convolutional networks pretrained via supervised learning outperform transformers on the majority of tasks we consider.
> The observed superiority of supervised pretraining occurs because such models are often trained on larger datasets. In apples-to-apples comparisons on the same dataset scale, SSL models outperform their supervised counterparts.
$\triangleright$  ViTs are more sensitive to the amount of pretraining data and the number of parameters than CNNs.
> Performance across tasks is strongly correlated – the top-performing backbones in BoB tend to be universally good across tasks and settings. See Figure 1.
1.2 Previous Benchmarks
Throughout much of the last decade, the most popular backbones were pretrained on ImageNet [17]. Since 2020, SimCLR [10] and CLIP [73] have popularized self-supervised backbones and spawned much new research. While method papers that propose a new pretraining routine typically compare to similar competitors on several downstream tasks, we focus in this section on works that specifically benchmark large collections of backbones on diverse tasks.
In 2019, Goyal et al. [25] compared AlexNet [47] and ResNet-50 [28] models pretrained using colorization and jigsaw pretext tasks to supervised learning models, finding that supervised learning massively outperformed SSL at the time. Kolesnikov et al. [44] similarly compared several pretext tasks and convolutional neural network architectures, showing that architectural advances on supervised learning do not always translate to improved self-supervised learning. Kornblith et al. [45] instead benchmarked the transferability of ImageNet-trained supervised learning models on downstream classification tasks, varying the architecture and finding that the correlation between downstream performance and ImageNet test accuracy is nearly perfect across architectures. In the same year, Zhai et al. [107] built the Visual Task Adaptation Benchmark (VTAB) and tested various self-supervised learning methods including VAEs and GAN discriminators, also exhibiting the dominant performance of supervised learning models. In 2020, Ericsson et al. [21] evaluated ResNet-50 models trained on ImageNet using various SSL algorithms, finding that the performance of then-existing SSL algorithms on a richer set of downstream tasks were strongly correlated with their ImageNet-1k test accuracy and finding improved performance of the newer SSL algorithms compared to previous studies.
Since the above works, pretraining algorithms along with their training sets and architectures have made tremendous progress, and whereas supervised learning was previously the default approach to pretraining, the options now are endless. Therefore, benchmarking backbones deserves renewed attention. See Appendix A for an additional survey of task-specific benchmarks.
2 A Guide to BoB
Among the distinguishing features of the diverse backbones competing in our battle are their architectures, pretraining routines, and the datasets on which they were pretrained. Table 1 contains an overview of the backbones we benchmark including their pretraining algorithms, pretraining datasets, and architectures. We also provide a more detailed description of these features and the precise pretrained checkpoints we use in Appendix B.
A Note on Scale and Apples-to-Apples Comparison. Many practitioners have limited compute and moreover will need to tune hyperparameters on their own datasets without exceeding their compute budget. To simulate this scenario, we perform moderate hyperparameter sweeps, we preclude particularly long training schedules, and we do not consider architectures bigger than ConvNeXt-Base, except for the Stable Diffusion backbone which does not come in a smaller size. Specific hyperparameter grids are detailed in subsequent sections. Moreover, we only use publicly available checkpoints that would also be accessible to practitioners. Available checkpoints were pretrained with varying amounts of hyperparameter tuning, and different pretraining algorithms were trained on different datasets and architectures making a precise apples-to-apples comparison infeasible. Nevertheless, this comparison of existing checkpoints is the relevant one for practitioners, as it represents realistic conditions, and we use identically sized hyperparameter sweeps for each backbone on downstream tasks.
Table 1: A synopsis of the backbones we benchmark. Columns correspond to the pretraining algorithm, a coarse categorization, the pretraining dataset, and the architectures we include. A detailed description of each algorithm, pretraining dataset, and architecture can be found in Appendix B.
Pretraining	Style	Dataset	Architecture(s)
MoCo v3 [12]	SSL	ImageNet-1k [17]	ViT [18]
VICReg [3]	SSL	ImageNet-1k	ResNet [28]
VICRegL [4]	SSL	ImageNet-21k	ConvNeXt [58]
DINO [8]	SSL	ImageNet-1k	ResNet, ViT
MAE [30]	SSL	ImageNet-1k	ViT
Stable Diffusion [77]	Vision-Language	LAION-2B [81]	Stable Diffusion encoder
CLIP [73]	Vision-Language	LAION-2B, CLIP	ResNet, ViT
MiDaS [75]	Supervised	12 × Depth Datasets	SwinV2 [57]
Image classification	Supervised	ImageNet-21k,-1k	All above architectures
Random initialization	None	N/A	All above architectures
2.1 The Tasks
In order to comprehensively probe the capabilities of the backbones, we evaluate their performance both fine-tuned and frozen on a number of downstream tasks belonging to the following categories:
- Classification: We measure both fine-tuned and linear probe performance of backbones on various downstream classification tasks including natural, medical, or satellite image datasets in Section 3.1. Image classification tasks require that a backbone extract features which identify the content of an image's foreground but not necessarily how many of an object there are or where they are located within an image.
- Object detection and segmentation: Unlike image classification, dense prediction tasks require backbones to extract features containing the precise locations of objects, on a pixel basis for segmentation and in enough fidelity to draw bounding boxes for object detection. We evaluate backbones on both of these tasks in Section 3.2.
- Out-of-distribution generalization: In real-world applications, computer vision systems are often deployed on data which does not reflect their training set distribution. Even high-performing models are known to fail under domain shifts [71, 32]. Therefore, we evaluate the abilities of models both to generalize to new downstream domains in Section 3.3.
- Image retrieval: Image retrieval requires a backbone to match like images via proximity in feature space. We explore tasks that require matching the images with respect to various criteria such as semantic content and visual similarity in Section 3.4.
We now describe our experimental setup for each task. Specifically, we list learning protocols, datasets, and evaluation metrics. Find complete experimental and implementation details in Appendix C.
3.1 Classification
Learning protocols. We evaluate pretrained backbones on various datasets under two fine-tuning protocols, following previous works [12, 30, 8, 10]: end-to-end fine-tuning (including experiments with only a small number of labeled samples) and linear probing. In the former scenario, we fine-tune the full model end-to-end on a given dataset or on a fraction of it, and we measure the accuracy on the test split. In the linear probing scenario, we extract features from the frozen pretrained backbone, and only learn a linear classifier on top of these pretrained representations. These two protocols are widely used in previous work to evaluate the quality of pretraining methods such as in self-supervised learning [12, 30, 8, 10] and vision-language pretraining [1, 106].
Datasets and evaluation metrics. We conduct experiments on 6 common image classification datasets, covering multiple domains such as natural images (ImageNet-1K [17], CIFAR-100 [46],
Flowers-102 [65], Aircraft [61]), satellite images (EuroSAT [31]), and medical X-ray data (CheXpert [37]) showing the generalization and transferability of the pretrained backbones. All datasets we use are publicly available, and we list their details including size and the number of classes in Appendix C. For experiments with only a fraction of the training set, we randomly sample  $1\%$  and  $10\%$  of the training samples and fine-tune the pretrained backbones on these subsets. When sampling the subsets, we maintain the original dataset's label distribution. Note that we only consider in-domain generalization here, where the training and testing splits are from the same source.
To evaluate, we measure classification accuracy and Area Under the ROC Curve (AUC) on the test split as performance metrics for single-label and multi-label classification tasks, respectively. In addition to the best score among hyperparameter vectors, we also plot the accuracy for the first several epochs to show the convergence rate of different pretrained backbones. Moreover, we benchmark the latency and the memory usage of each backbone on the same device.
3.2 Object Detection and Segmentation
Learning protocols. For evaluations on object detection and instance segmentation, we employ the Cascade Mask R-CNN framework [5]. We conduct experiments with three protocols: (1) end-to-end training from random initialization, (2) end-to-end finetuning using pretrained backbones, and (3) finetuning with frozen backbones. Whereas finetuning with a frozen backbone is atypical in segmentation and detection, this latter protocol allows us to probe localization within features extracted by pretrained models and complements linear probing classification experiments. See Appendix C.1 for a discussion on the potential for ViTs, especially large ones, to exceed the performance of other models under more expensive training protocols.
Datasets and evaluation metrics. We conduct object detection and instance segmentation evaluations on the popular COCO dataset [54]. We follow the COCO-style average precision (AP) metric, which calculates the average across various Intersection over Union (IoU) thresholds. We report the box Average Precision (box AP), box AP@50, and AP@75 for object detection and mask Average Precision (mask AP), mask AP@50, and mask AP@75 for instance segmentation [55].
3.3 Out-of-Distribution Generalization
While modern networks may exhibit strong performance on data distributions they are trained on, a wide body of prior work [71, 32] has found that the performance of such models can degrade significantly under distribution shifts. In addition to evaluating the in-distribution performance of backbones across a diverse set of downstream tasks, we also consider how this performance translates to out-of-distribution (OOD) settings.
Learning protocols. Several task-specific datasets and benchmarks have been proposed to evaluate the robustness of models to deviations from their training distributions. Concretely, we study the generalization of the trained backbones on two tasks, (1) image classification and (2) object detection, and on two types of distribution shifts, (A) structure and style variations within ImageNet and (B) synthetic-to-real generalization.
Datasets and evaluation metrics. We consider the following broad benchmarks for OOD evaluation:
(A) Robustness to changes in structure and style. We measure OOD generalization of ImageNet-trained or fine-tuned models on the following benchmarks: (i) ImageNet-A [34]. ImageNet-A(dversarial) contains a curated subset of ImageNet test images spanning 200 categories that are especially challenging for trained deep models. (ii) ImageNet-V2 [76]. ImageNet-V2 is an additional test set of ImageNet-like images collected a decade after the original dataset following an identical collection protocol. (iii) ImageNet-R [33]. ImageNet-R(conditional) contains artistic renditions for 200 categories from ImageNet, including cartoons, graffiti, embroidery, origami, sculptures, etc. (iv) ImageNet-S [93]. ImageNet-S(ketch) is a web-crawled and manually cleaned collection of black and white sketch images from ImageNet categories.
(B) Syn-to-real generalization. We also measure the performance of models trained on synthetic data and tested on real data. Synthetic data has emerged as a popular alternative in settings where it may be hard or expensive to curate reliably annotated real-world data. We measure syn-to-real generalization for image classification and object detection on the two following popular benchmarks: (i) VisDA Syn  $\rightarrow$  Real. The VisDA classification benchmark consists of  $\sim 152\mathrm{k}$  synthetic images and
$\sim 55\mathrm{k}$  real images across 12 classes. The synthetic images in VisDA are 3D renderings of objects from multiple viewpoints and under different lighting conditions. The real counterparts are crops of the 12 classes obtained from the COCO dataset. (2) Sim10k  $\rightarrow$  Cityscapes. For object detection, we use Sim10k as the synthetic training dataset and Cityscapes as the real evaluation dataset. Sim10k consists of  $\sim 10\mathrm{k}$  street view images (drawn from GTAV). Cityscapes consists of  $\sim 5\mathrm{k}$  densely annotated street view images curated from vehicular viewpoints in the real world. Following prior work [13], we train on the entirety of Sim10k to detect instances of "car" and measure detection performance on the validation split of Cityscapes.
We report generalization performance using classification accuracy on the OOD test set for image classification and mean average precision or mAP@50 for object detection.
3.4 Image Retrieval
We conduct evaluations on a diverse set of retrieval datasets encompassing content-based image retrieval and classification datasets that we repurpose for semantic retrieval tasks. For geographic landmark retrieval, we utilize the Oxford dataset [69] and the Paris dataset [70]. To ensure accuracy, we employ the cleaned-up versions of these datasets with corrected labels [72]. The INSTRE dataset [95] consists of objects such as toys and irregularly-shaped products placed in different locations and conditions. To examine fine-grained retrieval, we employ the Caltech-UCSD Birds-200 dataset (CUB-200) [91], which contains various bird classes captured under different backgrounds, poses, and lighting conditions. For a diverse set of natural images, we use the iNaturalist dataset [88]. This dataset offers a wide range of fine-grained categories classified into 13 super-categories, including Plant, Insect, Bird, and Mammal. To evaluate retrieval performance in real-world scenarios, we employ the Objectnet dataset [2]. This dataset consists of 313 object classes with randomly varying backgrounds, rotations, and imaging viewpoints. For large-scale landmark recognition, we utilize the Google Landmarks v2 dataset [99], which includes approximately 200,000 unique landmarks. Lastly, we employ the INRIA Copydays dataset [19], which comprises a small collection of holiday photos. Among the datasets mentioned, iNaturalist, Objectnet, and CUB-200 can be categorized as semantic retrieval datasets, while the remaining datasets fall under content-based retrieval datasets.
To evaluate, we measure model performance using mean-Average-Precision or mAP [68]. We first compute the average precision for a given query image, and then compute the mean over all queries to find the mAP. We also measure Recall@k, which measures the proportion of correct matches among the top  $k$ , and MRR (Mean Reciprocal Rank), which records the number of results returned before the first correct match and computes the mean of the reciprocal of these misses. Higher is better for all metrics.
4 I'm a Practitioner. Which Backbone Should I Choose?
Practitioners today can choose from a large catalogue of backbones of varying sizes, training methods, and pretraining data: which backbone should a practitioner select for a particular task or in general? To answer this question, in BoB, we systematically compare publicly available backbones (see Table 1) across multiple tasks, datasets and settings. To make these comparisons, we use the following ranking protocol:
(1) Setting-specific Z-Scores. For a particular task and setting (e.g., top-1 classification accuracy on ImageNet), we first compute z-scores for all the backbones being evaluated – i.e., for setting specific performance (e.g., accuracy) values  $\{x_{i}\}_{i = 1}^{N}$ , z-scores are computed as  $\left\{\frac{x_i - \mu}{\sigma}\right\}_{i = 1}^N$  where  $\mu$  and  $\sigma$  are the mean and standard deviation of the sample. This allows us to measure how good a specific backbone is (stds above or below) compared to "mean" performance of all backbones in that setting.
(2) Cross-setting Comparisons. To compare backbones across different tasks and settings, we simply aggregate and compare the previously obtained z-scores to obtain a relatively (coarse) ranking of backbones.
Using rankings, we can report not only the best performing backbones for each task but also the best backbone in terms of overall performance across tasks, datasets and settings (see Table 2 for a summary).
Table 2: Which backbone should I choose? We list the top 3 most performant backbones (left to right) for various tasks and settings. Red corresponds to OOD evaluations and Green indicates overall comparisons.
Task	Good	Better	Best
1 Cls	ConvNeXt-B (IN-21k)	CLIP ViT-B (LAION-2B)	Sup. SwinV2-B (IN-21k,1k)
2 Det	Sup. ConvNeXt-B (IN-1k)	Sup. SwinV2-B (IN-21k,1k)	Sup. ConvNeXt-B (IN-21k)
3 Seg	Sup. ConvNeXt-B (IN-1k)	Sup. SwinV2-B (IN-21k,1k)	Sup. ConvNeXt-B (IN-21k)
4 Ret	CLIP ViT-B (LAION-2B)	Sup. SwinV2-B (IN-21k,1k)	Sup. ConvNeXt-B (IN-21k)
5 (OOD) Cls	CLIP ViT-B (LAION-2B)	Sup. SwinV2-B (IN-21k,1k)	Sup. ConvNeXt-B (IN-21k)
6 (OOD) Det	Sup. ConvNeXt-B (IN-21k)	Sup. ConvNeXt-T (IN-1k)	Sup. ConvNeXt-B (IN-1k)
7 All	CLIP ViT-B (LAION-2B)	Sup. SwinV2-B (IN-21k,1k)	Sup. ConvNeXt-B (IN-21k)
4.1 Task-Specific Backbones