SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Classification. For classification, across multiple datasets and experimental settings (fine-tuning, linear probing, full and low-shot training), we find "Supervised SwinV2-Base trained on IN-21k (finetuned on IN-1k)" to be the best performing backbone, followed by "CLIP ViT-Base" and "Supervised ConvNeXt-Base trained on IN-21k" (see row 1, Table 2).<sup>2</sup>
Object Detection & Segmentation. For object detection and instance segmentation, we find "Supervised ConvNeXt-Base trained on IN-21K" > "Supervised SwinV2-Base trained on IN-21k (finetuned on IN-1k)" > "Supervised ConvNeXt-Base trained on IN-1k".
Image Retrieval. For image retrieval, we find "Supervised ConvNeXt-Base trained on IN-21k" to be the best choice, with "Supervised SwinV2-Base trained on IN-21k (finetuned on IN-1k)" and "CLIP ViT-B trained on LAION-2B" being second and third.
(OOD) Classification. Across OOD evaluations for classification, we find "Supervised ConvNeXt-Base trained on IN-21k" > "Supervised SwinV2-B trained on IN-21k (finetuned on IN-1k)" > "CLIP ViT-Base trained on LAION-2B".
(OOD) Object Detection. For  $\mathrm{Syn} \rightarrow$  Real object detection, we find "Supervised ConvNeXt-Base trained on IN-1k" to be the best backbone, followed by "Supervised ConvNeXt-Tiny trained on IN-1k" and "Supervised ConvNeXt-Base trained on IN-21k".
4.2 Best Backbones Overall
For practitioners with no specific task in mind, the best performing models in terms of aggregate performance are "Supervised ConvNeXt-Base trained on IN-21k" followed by "Supervised SwinV2-Base trained on IN-21k (finetuned on IN-1k)" and "CLIP ViT-Base trained on LAION-2B". Overall, we note that backbones trained in a supervised fashion (SwinV2-Base, ConvNeXt-Base) or with vision and language supervision (CLIP ViT-Base) outperform the rest. Furthermore, we find that CLIP ViT-Base is closely followed by Supervised ViT-Base trained on IN-21k (finetuned on IN-1k). We more precisely compare approaches and analyze trends in Section 5.
4.3 Backbones on a Tight Budget
Many computer vision applications demand efficient backbones for fast or on-device inference. In this section, we benchmark three small backbones: RegNetX-400F [74], EfficientNet-B0 [84] and ResNet-18 [28] all pretrained in a supervised fashion on ImageNet-1k. We rank the performance of these small backbones on the set of tasks in Table 3. We find that EfficientNet-B0 performs best overall and across classification, retrieval, and OOD classification, followed by RegNetX-400MF and then ResNet-18. Interestingly, ResNets still outperform newer efficient architectures for detection and segmentation.
To ensure fair comparisons across backbones, we exclude MiDaS variants evaluated on ImageNet for this comparison.
Table 3: Which tiny backbone should I choose? We rank the most performant very lightweight backbones (left to right) for various tasks and settings. Red correspond to OOD evaluations and Green indicates overall comparisons.
Task	Good	Better	Best
1 Cls	ResNet-18	RegNetX-400MF	EfficientNet-B0
2 Det	RegNetX-400MF	EfficientNet-B0	ResNet-18
3 Seg	RegNetX-400MF	EfficientNet-B0	ResNet-18
4 Ret	ResNet-18	RegNetX-400MF	EfficientNet-B0
5 (OOD) Cls	ResNet-18	RegNetX-400MF	EfficientNet-B0
6 (OOD) Det	EfficientNet-B0	ResNet-18	RegNetX-400MF
7 All	ResNet-18	RegNetX-400MF	EfficientNet-B0
5 Observations and Trends
Figure 2: Transformers benefit significantly more from end-to-end fine-tuning than CNNs on dense prediction tasks. We visualize the difference in performance between end-to-end fine-tuning and only training the head atop a frozen feature extractor on different tasks. The x-axis is the difference in relative performance (fine-tuning z-score minus fixed backbone z-score). Across panels, the performance differences correlate between tasks.
strongly correlated with accuracy ( $r = -0.8278$ ), ECE exhibits a weaker correlation ( $r = -0.4876$ ). In both cases, we observe p-values under 0.05. We also note that self-supervised pretraining typically leads to inferior calibration.
$\triangleright$  CNNs and SSL are more adversarially robust. We additionally measure the adversarial robustness of each backbone on the ImageNet test set using an  $\ell_{\infty}$ -constrained PGD attack with multiple radii (see Appendix Table 19). For each architecture where we possess self-supervised learning versions, we see that supervised pretraining always yields inferior robustness. Moreover, ViTs are more vulnerable to adversarial examples than convolutional networks. Notably, ConvNeXt is more adversarially robust even when trained in a supervised fashion.
6 Where Are Things Going From Here?
At the core of every computer vision model is a backbone. In our battle of the backbones, we compared more than 1,500 training runs to surface insights for computer vision practitioners and researchers. To guide practitioners, we analyzed the performance of publicly available vision backbones across a broad range of tasks from segmentation and detection to classification and retrieval. We found supervised ConvNext, supervised SwinV2, and CLIP models performed well across this broad range of tasks. For computationally constrained settings, in our battle of the "small" backbones we found smaller counterparts to the same architectures supervised ConvNext-T and SwinV2, followed by DINO with a small ViT performed quite well. BoB offers practitioners a guide to select sensible backbones from the dizzying array of choices.
For researchers looking ahead, we also observed several notable trends. First, we found performance across tasks is strongly correlated, suggesting a shift away from specialized vision backbones to universal backbones that work well across a range of tasks. Next, we found throughput and performance are inverse related, suggesting scaling remains a promising avenue to improve backbones. Finally, we found that while our practical recommendations include many supervised models, in apple-to-apples comparisons to standard supervised training, self-supervised learning holds promise. By releasing all our experimental results along with code to put new backbones to the test, we hope BoB serves as a useful guide to both practitioners today and researchers looking ahead at tomorrow.
Limitations. We note that insights obtained from BoB are contingent on the vocabulary of tasks, backbones, and settings considered in this work. We intend for takeaways from this study to provide practical considerations useful for computer vision researchers, recognizing that such insights need to continuously evolve as more backbones are introduced and more tasks and settings are taken into account. Lastly, we note that studies in BoB focus mostly primarily on aspects related to performance, and exploration along other axes of importance (biases in models, etc.) remain.
Our benchmark does not include backbones larger than ConvNext-Base, aside from Stable Diffusion, and some rankings may change at a large scale. For instance, while we find that modern convolutional architectures pretrained via supervised learning perform best on most tasks, we also find that transformers benefit more from scale, both in terms of pretraining data and architecture size. It is possible that transformer backbones will pull ahead of convolutional backbones at very large scales.
7 Computation Cost and Carbon Footprint
The experiments in this paper took a cumulative 127k GPU hours on NVIDIA RTX A100 cards. Assuming the GPUs were running with an average carbon efficiency of  $0.37\mathrm{kgCO_2eq / kWh}$ , the total emissions are estimated to be  $11792.36\mathrm{kgCO_2eq}$  [48].
MG and AGW were supported in part by NSF CAREER IIS-2145492, NSF I-DISRE 193471, NIH R01DA048764-01A1, NSF IIS-1910266, BigHat Biosciences, Capital One, and an Amazon Research Award. HS and RC were supported in part by the ONR MURI grant N00014-20-1-2787. VP, PC, and JH were supported in part by ARL, NASA ULI, Google, and NSF #2144194. RN, MS, GS, and TG were supported by the ONR MURI program, the Office of Naval Research (N000142112557), the AFOSR MURI program, and the National Science Foundation (IIS-2212182 & 2229885).
Classification benchmarks. Image classification is the most common task in computer vision, and we have multiple benchmarks. For example, the timm library [100] benchmarks ImageNet classification performance across loads of backbones trained with different methods and on different datasets. In addition, we have dataset-specific benchmarks, such as "paperwithcode"3. The latter contains multiple datasets and methods, but it lacks systematic analysis among these results. Almost all the self-supervised learning method papers perform their own evaluation for image classification. To accelerate the research cycle in self-supervised learning, VISSL [26] provides a library for implementing SSL methods and evaluations. In this work, we evaluate various backbones trained in both self-supervised and supervised fashion, and on multiple datasets on different domains (natural images, satellite maps, and medical images). Moreover, we benchmark these backbones and datasets with exactly the same learning setting and conduct a thorough analysis of the collected results, which we believe is essential and helpful to practitioners.
Object detection and segmentation benchmarks. Benchmarking backbones for object detection and segmentation has been an active area of research. Several works have focused on evaluating and comparing the performance of various backbone architectures for these tasks [51, 52, 29, 9]. Popular backbone networks such as supervised pretrained ResNet have been extensively utilized and compared, while modern feature extractors such as vision-language models and self-supervised learning models have not been extensively studied. These studies either focus on a limited subset of backbones or incorporate diverse detectors with varying backbones, thereby hindering an accurate comparison. To the best of our knowledge, we are the first to present a comprehensive study of the various backbones with various architectures and pretraining methods for object detection and instance segmentation.
Out-of-distribution generalization benchmarks. Several prior works have benchmarked the out-of-distribution performance of visual models. For image classification, these have included variants of ImageNet [32-34, 93, 76, 63], synthetic-to-real adaptation benchmarks [66], and benchmarks with images belonging to varied domains including paintings, clipart, etc. [50, 67, 90, 87], sometimes even spanning multiple modalities and applications [43]. Similarly for object detection, several OOD generalization benchmarks have been fashioned from sim-to-real, cross-weather, and cross-city self-driving datasets [38, 15, 105, 80, 64]. Recently, [41] conducted a broad study of pretraining strategies for domain adaptation and generalization. In this work, we perform a similar study but on a larger scale and also include a diverse suite of backbones designed for varied tasks.
Image retrieval benchmarks. To the best of our knowledge, our study represents the first comprehensive examination of multiple pretrained deep learning backbones for image retrieval task. While previous survey papers [11, 108, 20, 39] have explanations of various types of deep learning methods, such as off-the-shelf models trained in an unsupervised or supervised fashion, single pass, and multiple pass methods, none have quantitatively analyzed these backbones. Therefore, our work fills this crucial gap in the existing literature.
B An Extended Guide to BoB
B.1 The Architectures
Below is a list of all architectures we compare. As different neural network architectures are believed to have distinct properties, from invariances to a reliance on different Fourier frequencies, evaluating a variety of architectures will allow us examine potential benefits of architectural differences. Many of the pretraining strategies we evaluate are accompanied by multiple checkpoints with different architectures or sizes, so we include multiple versions of each. We describe architectural modifications to these backbones for object detection and segmentation in Section 3.2.
- ResNet [28]: These are the staple convolutional neural networks we all know and love, complete with skip connections and batch normalization [36]. We include experiments on ResNet-50 and ResNet-101 backbones.
<sup>3</sup>https://paperswithcode.com/
- ConvNeXt [58]: ConvNeXt is a purely convolutional network with numerous modern architectural improvements including depthwise convolutions, inverted bottleneck blocks, large convolutional kernels, and a larger proportion of layers allocated to the third stage. This architecture improves performance of convolutional architectures at scale while maintaining their strong object detection and segmentation capabilities. We include experiments on ConvNeXt-Tiny and ConvNeXt-Base.
- Vision Transformer [18]: Vision transformers (ViTs) were derived from transformer language models [89] and inherit their multi-headed self-attention (MHSA) and position-wise feed-forward network (FFN) components. Unlike ResNets, ViTs do not encode translation equivariance, and they only encode locality by embedding images on a patch-by-patch basis. We include experiments on ViT-Small and ViT-Base.
- Swin Transformer V2 [57]: Swin Transformer [56] is a transformer architecture which incorporates hierarchical representations, translation invariance, and increased locality and efficiency into ViTs by only performing attention within spatial windows and merging these windows iteratively. SwinV2 is equipped with several modifications which improve scalability and transferability across input resolutions. We include experiments on SwinV2-Tiny and SwinV2-Base. For SwinV2-Base, unless otherwise stated, we use the model with a window size of 24.
- Stable Diffusion encoder [77]: Stable Diffusion is a text-to-image generative diffusion model which conducts diffusion in a latent space. We include experiments with a backbone formed by the learned encoder that converts images from pixel-space into the latent space where diffusion is performed followed by Stable Diffusion's U-Net, and we freeze the text encoder, using its frozen embedding. The encoder uses a convolutional architecture with added attention mechanisms. More details can be found in Rombach et al. [77].
B.2 The Pretraining Algorithms
The primary source of diversity amongst the backbones we consider stems from their different pretraining algorithms. We choose prototypical examples of categories including supervised learning, self-supervised learning (SSL), and vision-language since such types of pretraining routines are widely believed to confer their own unique properties. For instance, SSL backbones are thought to extract more transferable features [86], while vision-language models are thought to resist domain shifts [78].
- Classification: We include image classifiers pretrained on ImageNet-1k and -21k [17]. ImageNet pretraining has long been the de facto choice for computer vision systems.
- MiDaS [75]: MiDaS is trained in a supervised fashion for monocular depth estimation. In this task, the model accepts a natural image and outputs a dense 2D array of depth values representing the distance of the scene from the image plane.
- MoCo v3 [12]: Contrastive learning is a popular approach to SSL in computer vision which encourages a model extract similar features corresponding to different augmentations of the same image, called positive pairs and dissimilar features corresponding to different images, called negative pairs. MoCo v3 is a high-performing variant which employs a momentum encoder and multi-crop training as well as prediction and projection heads.
- VICReg [3]: Instead of adopting contrastive learning and negative pairs, VICReg avoids feature vectors collapsing towards a constant during SSL by regularizing their variance and covariance. VICRegL is a version which also applies the VICReg criterion to local features to teach the model to extract localization information in its features for downstream dense prediction tasks [4].
- DINO [8]: Much like MoCo v3, DINO uses a momentum encoder and multi-crop SSL, but DINO swaps out contrastive learning for self-distillation, demonstrating strong performance on ViTs with a small patch size.
- MAE [30]: Masked Autoencoders (MAE) use a distinct style of SSL adapted from masked language modeling [40]. MAE models are trained to reconstruct masked out input patches, unlike the above 3 models which instead match the features of augmented images.
- CLIP [73]: CLIP also uses contrastive learning, but on image-caption pairs instead of augmented image views. Language supervision endows CLIP features with information relating to the semantic meaning of image components, compared to models trained solely on image data [22]. We only use CLIP's image feature extractor in our experiments.
- Stable Diffusion [77]: Text-to-image diffusion models are an entirely different type of vision-language backbone, trained for image generation. The Stable Diffusion encoder, which we benchmark, maps images to a highly compressed latent space where diffusion is performed.
- Random initialization: In experiments where we fine-tune backbones on downstream tasks, we also evaluate baselines trained on the downstream training sets from random initialization.
B.3 The Pretraining Datasets
The backbones we benchmark are pretrained on datasets across a wide range of scales including image classification, image-text pairs, and images with depth annotations:
- ImageNet-1k and -21k [17]: ImageNet-21k contains over 14 million training images in 21,841 classes. ImageNet-1k is a subset of the aforementioned dataset containing almost 1.3 million training images in 1000 classes. These popular web-scraped image classification datasets are used for supervised pretraining with the labels, or self-supervised pretraining without them, among numerous backbones we benchmark. We denote pretraining on ImageNet-21k followed by fine-tuning on ImageNet-1k by "ImageNet-21k-1k".
- LAION-2B [81]: LAION-2B is a subset of the larger LAION-5B, which contains 5.85 billion web-scraped image-text pairs filtered by CLIP. LAION-2B specifically contains those 2.32 billion pairs with English language captions. Despite being by far the largest dataset amongst those we consider, LAION-2B is known to contain a large number of duplicates [97]. Stable Diffusion is further fine-tuned on LAION-Aesthetic, a subset of LAION-2B containing 120 million images filtered by a model trained to rate images by aesthetics.
- CLIP [73]: Since there is no OpenCLIP ResNet checkpoint available, we use the original CLIP checkpoint trained on OpenAI's diverse proprietary captioned-image dataset containing 400 million images scraped from publicly available internet sources.
- MiDaS [75]: MiDaS was trained on a combination of 12 image datasets with various types of depth annotations, and objectives: ReDWeb [101], DIML [42], Movies [75], MegaDepth [53], WSVD [92], TartanAir [96], HRWSI [102], ApolloScape [35], BlendedMVS [104], IRS [94], KITTI [62], NYU Depth V2 [82]. These models are therefore trained with multi-objective optimization. Collectively, the MiDaS training set contains more than 3.7 million images. An earlier version of MiDaS was trained on a smaller collection of 5 datasets, but we use the most recent version trained on the largest training set.
Evaluation datasets and licenses. In Table 4, Table 5, Table 6, and Table 7 we summarize the datasets we use for evaluating classification, object detection, segmentation, out-of-domain generalization and retrieval performance. We include the number of classes as well as the number of test samples for each dataset. To be noticed, we only use  $10\%$  of the labeled dataset for EuroSAT and Chexpert to distinguish the performance among all the backbones. Object detection and instance segmentation experiments are conducted on COCO datasets [54]. COCO is released under the Creative Commons Attribution 4.0 License<sup>4</sup>. This license permits users to share and adapt the dataset for any purpose, as long as the original creators are appropriately credited. For OOD classification, we use the ImageNet-Adversarial [34], ImageNet-Sketch [93], ImageNet-Renditions [33], ImageNet-V2 [76], and VisDA [66] datasets, all of which are freely available for research use. For OOD detection, we use the Sim10k [38] and Cityscapes [15] datasets. Densely annotated images for Sim10k are available freely<sup>5</sup> and can only be used for non-commercial applications. The license agreement for the Cityscapes dataset dictates that the dataset is made freely available to academic and non-academic entities for non-commercial purposes such as academic research, teaching, scientific publications, or personal experimentation and that permission to use the data is granted under certain
4https://cocodataset.org/#termsofuse
<sup>5</sup>https://fcav.engin.umich.edu/projects/driving-in-the-matrix
Table 4: Image Classification Datasets
Dataset	Description	Size	Classes
ImageNet-1k [17]	Natural images of versatile categories	1.3M	1,000
CIFAR-100 [46]	Natural images of versatile categories	50K	100
EuroSAT [31]	Satellite images (RGB) of land use and land cover	13.5K	10
Flowers-102 [65]	Images of flowers categories	1K	102
Aircraft [61]	Images of aircraft model variant, family, manufacturer	3K	100
Chexpert [37]	Medical images	191K	5
Table 5: Object Detection and Instance Segmentation Datasets
Dataset	Description	Size	Classes
COCO [54]	Large-scale object detection and segmentation dataset	330K	80
conditions. $^{6}$ . All datasets used in benchmarking retrieval tasks (except for Objectnet) are restricted to non-commercial research and educational purposes. Objectnet is free to use for both research and commercial applications. $^{7}$
Classification. For fine-tuning, we train the backbones for 100 epochs using AdamW [60] and weight decay  $\{5e^{-2}, 1e^{-3}\}$ . We use a cosine annealing learning rate scheduler with 5 warmup epochs. We run grid searches for learning rates with the default grid range being  $\{1e^{-3}, 5e^{-4}, 1e^{-4}\}$  as we observe peaking performance on multiple models with these learning rates. We expand the search range for learning rate when training models from scratch (i.e., fine-tuning from randomly initialized weights) to  $\{1e^{-2}, 5e^{-3}, 1e^{-3}, 5e^{-4}, 1e^{-4}\}$ . We keep the batch size of 1024 the same for all experiments and use gradient accumulation when Out-of-Memory occurs for large models (such as the Stable Diffusion encoder). For data augmentation, we follow He et al. [30], including random horizontal flips and crops, mixup, CutMix, and a RandAug [16] policy with hyperparameter (9, 0.5) corresponding to the number of transformations and magnitude. For regularization strategies, we apply the stochastic depths with a drop rate of 0.1, layer-wise learning rate decay of 0.75, and label smoothing with a rate of 0.1. For linear probing, again we follow He et al. [30], where we set weight decay to zero and disable Mix-Up, Cut-Mix, stochastic depth, or color jitter. We train the model with LARS optimizer with a batch size of 4096 for 90 epochs. For fine-tuning on low-shot ImageNet, we follow Cai et al. [6], where we use AdamW for all the transformer-based backbones and SGD for Convolution-only backbones. For transformer-based backbones, we use grid search among three peak learning rates of  $\{1e^{-4}, 2.5e^{-4}, 5e^{-4}\}$  and two layer-decay rates of 0.65, 0.75 for AdamW. We use grid search among three peak learning rates of  $\{1e^{-1}, 5e^{-2}\}$  for SGD. We fix the
<sup>6</sup>https://www.cityscapes-dataset.com/licenses/
<sup>7</sup>https://objectnet.dev/download.html
Table 6: OOD Generalization Datasets
Task	Train Dataset	Test Dataset	Test Size	Classes
Image Classification	ImageNet-1K [79]	ImageNet-A [34]	7.5K	0.2K
Image Classification	ImageNet-1K [79]	ImageNet-v2 [76]	10K	1K
Image Classification	ImageNet-1K [79]	ImageNet-R [33]	30K	0.2K
Image Classification	ImageNet-1K [79]	ImageNet-S [93]	50K	1K
Image Classification	VisDA-Synthetic [66]	VisDA-Real [66]	55.4K	12
Object Detection	Sim10K [38]	Cityscapes [15]	0.5K	1