Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks
Authors: Micah Goldblum
Hossein Souri
Renkun Ni
Manli Shu
Viraj Uday Prabhu
Gowthami Somepalli
Prithvijit Chattopadhyay
Mark Ibrahim
Adrien Bardes
Judy Hoffman
Rama Chellappa
Andrew Gordon Wilson
Tom Goldstein
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: self-supervised learning
object detection
segmentation
out-of-distribution
domain shift
image retrieval
transfer learning
TL;DR: We benchmark a diverse suite of pretrained models across a diverse set of computer vision tasks ranging from classification to object detection to OOD generalization and more.

Abstract: Neural network based computer vision systems are typically built on a backbone, a pretrained or randomly initialized feature extractor.  Several years ago, the default option was an ImageNet-trained convolutional neural network.  However, the recent past has seen the emergence of countless backbones pretrained using various algorithms and datasets. While this abundance of choice has led to performance increases for a range of systems, it is difficult for practitioners to make informed decisions about which backbone to choose.  Battle of the Backbones (BoB) makes this choice easier by benchmarking a diverse suite of pretrained models, including vision-language models, those trained via self-supervised learning, and the Stable Diffusion backbone, across a diverse set of computer vision tasks ranging from classification to object detection to OOD generalization and more.  Furthermore, BoB sheds light on promising directions for the research community to advance computer vision by illuminating strengths and weakness of existing approaches through a comprehensive analysis conducted on more than 1500 training runs.  While vision transformers (ViTs) and self-supervised learning (SSL) are increasingly popular, we find that convolutional neural networks pretrained in a supervised fashion on large training sets still perform best on most tasks among the models we consider. Moreover, in apples-to-apples comparisons on the same architectures and similarly sized pretraining datasets, we find that SSL backbones are highly competitive, indicating that future works should perform SSL pretraining with advanced architectures and larger pretraining datasets.  We release the raw results of our experiments along with code that allows researchers to put their own backbones through the gauntlet here: https://github.com/hsouri/Battle-of-the-Backbones.

Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks
Micah Goldblum'
Hossein Souri?
Renkun Ni
Manli Shu
Viraj Prabhu
Gowthami Somepalli
Prithvijit Chattopadhyay
Mark Ibrahim
Adrien Bardes
Judy Hoffman
Rama Chellappa"
Andrew Gordon Wilson'
Tom Goldstein
Abstract
Neural network based computer vision systems are typically built on a backbone, a pretrained randomly initialized feature extractor. Several years ago, the default option was an ImageNet-traine cent past has seen the emergence of countless backbones pretrained using various algorithms and datasets. While this abundance of choice has led to performance increases for range of systems, it is difficult for practitioners to make informed decisions about which backbone to choose. Battle of the Backbones (BoB) makes this choice easier by benchmarking diverse suite of pretrained models, including vision-language models, those trained via self-supervised learning, and the Stable Diffusion backbone, across diverse of computer vision tasks ranging from clas- sification to object detection to OOD generalization and more. Furthermore, BoB sheds light on promising directions for the research community to advance com- puter vision by illuminating strengths and weakness of existing approaches through a comprehensive analysis conducted on more than 1500 training runs. While vision transformers (ViTs) and self-superviser learning (SSL) are increasingly popular, we find that convolutional neural networks pretrained training sets still perform best on most tasks among the models consider. More- over, in apples-to-apples comparisons on the same architectures and similarly sized pretraining datasets, we find that SSL backbones are highly competitive, indicating that future works should perform SSL pretraining with advanced architectures and larger pretraining datasets. We release the raw results of our experiments along with code that allows researchers to put their own backbones through the gauntlet here: https: // /ttgtttttt///////////////t com www/ http:/sourrr//////ee/e.. wesoorr/Baattteeeee Bat/tBttttllaeeee teeethe-BBee Bahe-Backoones
convolutional neural network. However, the re-
supervised fashion on large
Introduction
The dominant paradigm for building machine vision systems involves a feature extractor network. also known as backbone, which feeds into task-specific head. The backbone might output dense array of features for object detection and localization, or a single feature vector for classification or image retrieval. While backbones backbones are pretrained on large benchmark datasets and then fine-tuned for the task at hand. This transfer learning approach has several ad advantages. First, it dramatically reduces the application- specific data requirements of deep learning and has led to improved performance on a wide range of
be trained from scratch task-specific data, many off -he-l
Authhrrs contributed equally. Correspondence to goldblum@nyu edu and hsouri 10jhu. edu. This work was conducted at New York University! Johns Hopkins University University of Maryland', Georgia Institute of Technology Inria and Meta AI Research
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

applications. Second, it can speed up training and reduce compute costs even when large amounts of task-specific data are available [29]. Finally, pretraining datasets often contain images from many disparate domains, resulting in model robustness that can be transferred to downstream tasks.
Early deep learning based vision systems relied heavily ImageNet pretraining [2). In contrast, today's 's practitioners have access to a coru of choices, with different pretrained models resulting in significant performance differences. There are three primary factors that influence the performance of such a model: its architecture, the pretraining algorithm, and the pretraining dataset. Each of these design dimensions presents many options, resulting dizzying array of choices for practitioners building a computer vision system. Despite this wide variety of choices, practitioners have no resource turn and instead are left piecing together results from method papers or testing out the backbones themselves.
We pit these backbones against each other in Battle of the Backbones (BoB). BoB compares many popular publicly available pretrained checkpoints, as well as randomly initialized baselines, on a wide variety of downstream tasks including image classification on natural, medical, and satellite images (Section 3.1), object detection and segmentation (Section 3.2), out-of-distribution generalization (Section 3.3), and image retrieval (Section 3.4).
Aside from assisting practitioners building computer vision systems, another central goal of this benchmark is to help guide the research community towards fruitful research directions in their quest for designing better backbones. BoB sheds light on the strengths and weaknesses of pretraining routines and architectures, revealing popular misconceptions and fundamental limitations, well as promising directions for improvement. Below, we summarize several of our primary findings and discuss previous efforts for comparing backbones.
1.1 Battle of the Backbones: The TLDR
The subsequent sections in this paper contain numerous experimental details. Therefore, we distill several key findings below:
> Across the suite of comprehensive evaluations in BoB, spanning tasks, datasets, and settings (includ- ing ID and OOD), supervised ConwNe supervised SwinV2-Base trained using ImageNet-21K, and CLIP ViT-Base come out on top. The same winners also win at smaller scales. Among smaller backbones, ConyNeXt-Timg and SwinV2-Tiny emerge victorious, followed by DINO ViT-Small.
Preteaining
Supervizel
MiDa8
4e-08
ConvNeXi.
Random luit
VidReg
Pananttrss
3.2e-08
ResNet
CLIP
Stable Diffusion
8.0e+407 1.6e+-08
Architecture VIT
SwinV2
MAE
DINO
Stable Dillissin
MOCO V3
Dotection Performange (Relative)
1.3-
IN-salk IN-1k IN-21k
O013 Closiiication Reriemance (Reletive)
1.5-
IN-akk
21
03-
15-21k
0.5-
IN IN
N-1k
0.3-
-0.5
0.0
+
-1.3-
-1.0
1kk
-044
-1.2
0.0 0.2 00.
0.6
0.5
1.0
Clessification Performance (Relative)
Classification Performance (Relacive)
Figure 1: Performance is correlated across tasks. Performance for each model is reported in terms of standard deviations above/below the mean averages across datasets. Left: Comparison between classification and detection. Right: Comparison between classification and OOD classification.

---

> Despite the recent attention paid to transformer- based architectures and self- self-super learning, high-performance convolutional networks pretrained via supervised learning outperform transformers on the majority of tasks we consider.
The observed superiority of supervised pretraining occurs because such models are often trained on larger datasets. In apples-to-apples comparisons on the same dataset scale, SSL models outperform their supervised counterparts.
D. ViTs are more sensitive to the amount of pretraining data and the number of parameters than CNNs.
> Performance across tasks is strongly correlated - the uop-erformning backbones in BoB tend to be universally good across tasks and settings. See Figure 1.
1.2 Previous Benchmarks
Throughout much of the last decade, the most popular backbones were pretrained on ImageNet [17]. Since 2020, SimCLR [10] and CLIP [73] have popularized self-supervised backbones and spawned much new research. While method papers that propose a new pretraining routine typically compare to similar competitors on several downstream tasks, we focus in this section on works that specifically benchmark large collections of backbones on diverse tasks.
In 2019, Goyal et al. [25] compared AlexNet [47] and ResNet-50 [28] models pretrained using colorization and jigsaw pretext tasks to supervised learning models, finding that supervised learning massively outperformed SSL at the time. Kolesnikov et al. [44] similarly compared several pretext tasks and convolutional neural network architectures, showing that architectural advances on supervised learning do not always translate to improved self-supervised learning. Kornblith et al. [45] instead benchmarked the transferability of ImageNet-trainee downstream classification tasks, varying the architecture and finding that the correlation between downstream performance and ImageNet test accuracy is nearly perfect across architectures. In the same year, Zhai et al. [107] built the Visual Task Adaptation Benchmark (VTABB and tested various self-super learning methods including VAEs and GAN discriminators, also exhibiting the dominant performance of supervised learning models. In 2020, Ericsson al. [21] evaluated ResNet-50 models trained on ImageNet using various SSL algorithms, finding that the performance of then-existing SSL algorithms their ImageNet- 1k test accuracy and finding improved performance of the newer SSL algorithms compared to previous studies.
supervised learning models on
richer set of downstream tasks were strongly correlated with
Since the above works, pretraining algorithms along with their training sets and architectures have made tremendous progress, and whereas supervised learning was previously the default approach to pretraining, the options now are endless. Therefore, benchmarking backbones deserves renewed attention. See Appendix A for an additional survey of task-specific benchmarks.
2 A Guide to BoB
Among the distinguishing features of the diverse backbones competing in our battle are their archi- tectures, pretraining routines, and the datasets on which they were pretrained. Table I contains an overview of the backbones we benchmark including their pretraining algorithms, pretraining datasets, and architectures. We also provide a more detailed description of these features and the precise pretrained checkpoints we use in Appendix B.
Note on Seale and Apples-to-Apples Comparison. Many practitioners have limited compute and moreover will need to tune hyperparameters compute budget. To simulate this scenario, we perform moderate hyperparameter sweeps, we preclude particularly long training schedules, and we do not consider architectures bigger than ComNNett-Bor Base, except for the Stable Diffussion backbone which does not come in a smaller size. Specific hyperparameter grids are detailed in subsequent sections. Moreover, only use publicly available checkpoints that would also be accessible to practitioners. Available checkpoints were pretrained with varying amounts of hyperparameter tuning, and different pretraining algorithms were trained on different datasets and architectures making a precise apples-to-applle comparison infeasible. Nevertheless, this comparison of existing checkpoints is the relevant one for practitioners, as it represents realistic conditions, and we use identically sized hyperparameter sweeps for each backbone on downstream tasks.
their own datasets without exceeding their
3

---

Table 1: A synopsis of the backbones we benchmark. Columns correspond the pretraining algorithm, a coarse categorization, the pretraining dataset, and the architectures we include. A detailed description of each algorithm, pretraining dataset, and architecture can found Appendix
Pretraining
Style
Dataset
Architecture(s)
MoCo v3 [12]
SSL
ImageNet- [17]
ViT [18]
VICReg [3]
SSL
ImageNet- lk
ResNet [28]
VICRegI. [4]
SSL.
ImageNet-211
ConvNeXt [58]
DINO [8]
SSL
ImageNet- 1k
ResNet, VIT
MAE [30]
SSL
ImageNet-
VIT
Stable Diffusion [77]
Vision-Language
LAION-2B [81]
Stable Diffusion encoder
CLIP [73]
Vision-Language
LAION-2B, CLIP
ResNet, VIT
MiDaS [75]
Supervised
12 x Depth Datasets
SwinV2 [57]
Image classification
Supervised
ImageNet-2Ik,IIK
All above architectures
Random initialization
None
N/A
All above architectures
2.1 The Tasks
In order to comprehensively probe the capabilities the backbones, we evaluate their performance both fine-tuned and frozen number of downstream tasks belonging the following categories:
Classification: We measure both fine-tuned and linear probe performance of backbones on various downstream classification tasks including natural, medical, or satellite image datasets in Section 3.1. Image classification tasks require that backbone extract features which identify the content of an image's foreground but not necessarily how many of an object there are or where they are located within image.
Object detection and segmentation: Unlike image classification, dense prediction tasks require backbones to extract features containing the precise locations of objects, on pixel basis for segmentation and in enough fidelity draw bounding boxes for object detection. We evaluate backbones on both of these tasks in Section 3.2.
generalization: In real-world applications, computer vision systems are often deployed on data which does not reflect their training set distribution. Even high- performing models are known to fail under domain shifts 32]. Therefore, we evaluate the abilities of models both to generalize to new downstream domains in Section 3.3.
Image retrieval: Image retrieval requires backbone to match like images via proximity in feature space. We explore tasks that require matching the images with respect various criteria such as semantic content and visual similarity in Section 3.4.
3
Experimental Setup
We now describe experimental setup for each task. Specifically, we list learning protocols, datasets, and evaluation metrics. Find complete experimental and implementation details in Appendix C.
3.1 Classification
Learning protocols. We evaluate pretrained backbones on various datasets under two fine-tuning protocols, following previous works [12, 30, 8, 10]: end-to-end fine-tuning (including experiments with only a small number of labeled samples) and linear probing. In the former scenario, we given dataset
fine-tune the full model end-to-end accuracy on the test split. In the linear probing scenario, we extract features from the frozen pretrained backbone, and only learn linear classifier top of these pretrained representations. These two protocols are widely used in previous work to evaluate the quality of pretraining methods such as in self-supervised learning [12, 30, 8, 10] and vision-language pretraining [1, 106).
fraction of it, and measure the
Datasets and evaluation metries. We conduct experiments on common image classification datascts, covering multiple domains such natural images (ImageNet- IK [17], CIFAR- 100 [46].

---

Flowers- 102 [65], Aircraft [61]), satellite images (EuroSAT [31]). and medical X-ray data (CheX- pert [37]) showing the generalization and transferability of the pretrained backbones. All datasets we use are publicly available, and we list their details including size and the number of classes in Appendix C. For experiments with only a fraction of the training set, we randomly sample I 1% and 10% of the training samples and fine-tune the pretrained backbones on these subsets. When sampling the subsets, we maintain the original dataset's s label distribution. Note that we only consider in-domain generalization here, where the training and testing splits are from the same source.
To evaluate, we measure classification accuracy and Area Under the ROC Curve (AUC) on the test split as performance metries for single-label and muti-label classification tasks, respectively. In addition to the best score among hyperparameter vectors, we also plot the accuracy for the first several epochs to show the convergence rate of different pretrained backbones. Moreover, we benchmark the latency and the memory usage of each backbone on the same device.
3.2 Object Detection and Segmentation
Learning protocols. For evaluations on object detection and instance segmentation, we employ the Cascade Mask R-CNN framework [5]. We conduct experiments with three protocols: (1) end-to-end training from random initialization, (2) end-to-end finetuning using pretrained backbones, and (3) finetuning with frozen backbones. Whereas finetuning with frozen backbone is atypical in segmen- tation and detection, this latter protocol allows us to probe localization within features extracted by pretrained models and complements linear probing classification experiments. See Appendix C. for a discussion on the potential for ViTs, especially large ones, to exceed the performance of other models under more expensive training protocols.
Datasets and evaluation metrics. We conduct object detection and instance segmentation evaluations on the popular COCO dataset [54]. We follow the COCO-style average precision (AP) metric, which calculates the average across various Intersection over Union (IoU) thresholds. We report the box Average Precision (box AP), box AP@50, and AP@75 75 for object detection and mask Average Precision (mask AP), mask AP@50, and mask AP@75 75 for instance segmentation [55].
3.3 Out-of-Distributioo Generalization
While modern networks may exhibit strong performance on data distributions they are trained on, a wide body of prior work [71, 32] has found that the performance of such models can degrade significantly under distribution shifts. In addition to evaluating the in-distribution performance of backbones across a diverse set of downstream tasks, we also consider how this performance translates to out-of-distribution (OOD) settings.
Learning protocols. Several task-specific datasets and benchmarks have been proposed to evaluate the robustness of models deviations from their training distributions. Concretely, we study the generalization of the trained backbones on two tasks, (1) image classification and (2) object detection, and on two types of distribution shifts, (A) structure and style variations within ImageNet and (B) synthetic-to-real generalization.
Datasets and evaluation metrics. We consider the following broad benchmarks for OOD evaluation:
(A) Robustness changes in structure and style. We measure OOD generalization of ImageNet-
trained or fine-tuned models on the following benchmarks: (i) ImageNet-A [34]. ImageNet- A(dversarial) contains a curated subset of ImageNet test images spanning 200 categories that are especially challenging for trained deep models. (ii) ImageNet-V?: V2 [76]. ImageNet-V2 is additional test set of ImageNet- like images collected decade after the original dataset following an identical collection protocol. (iii) ImageNet-R [33]. Reennditio)))) contains artistic renditions for 200 categories from ImageNet, including cartoons, graffiti, embroidery, origami, sculptures, etc. (iv) ImageNet-S [93]. ImageNet-S(ketch) is web-crawled and manually cleaned collection of black and white sketch images from ImageNet categories.
(B) Syn-to-real generalization. We also measure the performance of models trained on synthetic data and tested on real data. Synthetic data has emerged as a popular alternative settings where it may be hard or expensive to curate reliably annotated real-world data. We measure syn-to-real generalization for image classification and object detection on the two following popular benchmarks: (i) VisDA Syn->Real. The VisDA classification benchmark consists of ~ 152k synthetic images and
5

---

~ 55k real images across 12 classes. The synthetic images in VisDA are 3D renderings of objects from multiple viewpoints and under different lighting conditions. The real counterparts are crops of the 12 classes obtained from the COCO dataset. (2) Sim 10k (ik---itssappes.. For object detection, we use Sim10k as the synthetie training dataset and Cityscapes as the real evaluation dataset. Sim10k consists of ~ 10k street view images (drawn from GTAV). Cityscapes consists of ~ 5k densely annotated street view images curated from vehicular viewpoints in the real world. Following prior work [13], we train the entirety of Sim to detect instances of "car" and measure detection performance on the validation split of Cityscapes.
We report generalization performance using classification accuracy on the OOD test set for image classification and mean average precision or mAP@ 50 for object detection.
3.4 Image Retrieval
We conduct evaluations diverse set of retrieval datasets encompassing content-based image retrieval and classification datasets that we repurpose for semantic retrieval tasks. For geographic landmark retrieval, utilize the Oxford dataset [69] and the Paris dataset [70]. To ensure accuracy, we employ the cleaned-up versions of these datasets with corrected labels [72]. The INSTRE dataset [95] consists of objects such toys and irregulary-shaped products placed in different locations and conditions. To examine fine- fine-grained retrieval, we employ the Caltech-UCSD Birds-200 dataset (CUB-200) [91]. which contains various bird classes captured under different backgrounds. poses, and lighting conditions. For a diverse set of natural images, we use the iNaturalist dataset [88]. This dataset offers wide range of fine-grate tineigraieed categories classified into 13 super-categories, including Plant, Insect, Bird, and Mammal. To evaluate retrieval performance in real-world scenarios, we employ the Objectnet dataset [2]. This dataset consists of 313 object classes with randomly varying backgrounds, rotations, and imaging viewpoints. For large-scale landmark recognition, we utilize the Google Landmarks v2 dataset [99], which includes approximately 200,000 unique landmarks. Lastly, we employ the INRIA Copydays dataset [19], which comprises a small collection of holiday photos. Among the datasets mentioned, iNaturalist, Objectnet, and CUB-200 can be categorized as semantic retrieval datasets, while the remaining datasets fall under content-based retrieval datasets.
To evaluate, we measure model performance using compute the average precision for given query image, and then compute the mean over all queries to find the mAP. We also measure Recall@ k, which measures the proportion of correct matches among the top k, and MRR (Mean Reciprocal Rank), which records the number of results returned before the first correct match and computes the mean of the reciprocal of these misses. Higher better for all metrics.
Average-Preeist or mAP [68]. We first
I'm a Practitioner. Which Backbone Should Choose?
Practitioners today can choose from a large catalogue of backbones of varying sizes, training methods, and pretraining data: which backbone should a practitioner select for particular task in general? To answer this question, in BoB, we systematically compare publicly available backbones (see Table I) across multiple tasks, datasets and settings. To make these comparisons, we use the following ranking protocol:
(1) Setting-specific Z-Scores. For a particular task and setting (e.g, top-1 classification accuracy on ImageNet), we first compute z-scores for all the backbones being evaluated - i.e., for setting specific performance (c.g., accuracy) values J. z-scores are computed as { شسس NN where μ and are the mean and standard deviation of the sample. This allows us to measure how good a specific backbone (stds above or below) compared "mean" performance of all backbones in that setting.
(2) Cross-setting Comparisons. To compare backbones across different tasks and settings, we simply aggregate and compare the previously obtained z-scores to obtain relatively (coarse) ranking of backbones.
Using rankings, we can report not only the best performing backbones for each task but also the best backbone in terms of overall performance across tasks, datasets and settings (see Table for a summary).

---

Table Which backbone should I choose? We list the top 3 most performant backbones (left to
right) for various tasks and settings. overall comparisons.
Red corresponds to OOD evaluations and Green indicates
Task
Good
Better
Best
Cls
ConvNeXI-B (IN-21k)
CLIP ViT-B (LAION-2B)
Sup. SwinV2-B (IN-2IK.
Det
Sup. ConvNeXt-B (IN-Ik)
Sup. SwinV2-B (IN-21k,1k)
Sup. ConvNeXt-B (IN-2Ik)
Seg
Sup. ConvNeXI-B (IN-1k)
Sup. SwinV2-B (IN-2I
Sup. ConvNeXt-B (IN-21k)
4 Ret
CLIP ViT-B (LAION-2B)
Sup. SwinV2-B (IN-2IK,IK)
Sup. ConvNeXt-B B (IN-2IK)
5 (OOD) Cls
CLIP ViT-B (LATON-2B)
Sup. SwinV2-B (IN-21k,1k)
Sup. ConvNeXI-B (IN-21k)
6 (OOD) Det
Sup. ConvNeXt-B (IN-2IK)
Sup. ConvNeXi-T (IN-IK)
Sup. ConvNeXt-B (IN-IK)
All
CLIP ViT-B (LAION-2B)
Sup, SwinV2-B (IN-2IK,1k)
Sup. ConvNeXt-B (IN-21K)
4.1 Task-Specific Backbones
Classification, For classification, across multiple datasets and experimental settings tine-tuuing linear probing, full and low-shot training), we find "Suppevised Swin V2-Base trained on IN-21k (finetuned on IN-1k)" be the best performing backbone, followed by "CLIP ViT-Base" and "Supervised ConvNeXt-Baln trained on IN-21K" (see row 1, Table 2).²
Object Detection & Segmentation. For object detection and instance segmentation, we find "Supervised ConvNeXt-Baan trained on IN-2IK" > "Supervised SwinV2-Base trained on IN-2Ik (finetuned on IN-IK) "Supervised ConvNeXI-Base trained on IN-IK".
Image Retrieval. For image retrieval, we find "Supeevised ConyNeXt-Base trained on IN-21K" to be the best choice, with "Supervised V2-Baa2BBsss trained on IN-21k (finetuned on IN-IK)" and "CLIP ViT-B trained on LAION-2B" being second and third.
(OOD) Classification. Across OOD evaluations for classification, we find "Supervised ConvNeXt- Base trained on IN-21K" > "Supervised V2-B trained on IN-21k (finetuned on IN-Ik)" > "CLIP ViT-Base trained on LAION-2B".
(OOD) Object Detection. For Syn-+Real object detection, we find "Supervised ConvNeXI-Baas trained on IN-IK" to be the best backbone, followed by "Supervised ConvNeXt-Tiny trained on IN-IK" and "Supervised ConyNeXt-Bass trained IN-2IK".
4.2 Best Backbones Overall
For practitioners with no specific task in mind, the best performing models in terms of aggregate performance are "Supervvsed ConyNeXt-Barr trained on IN-21k" followed by "Supervised SwinV2. Base trained on IN-21k (finetuned IN-1K)" and "CLIP ViT-Base trained on LAION-2B". Overall, we note that backbones trained in a supervised fashion SiwinV2-Basss ConvNeXt-Base or with vision and language supervision (CLIP ViT-Base) outperform the rest. Furthermore, we find that CLIP ViT-Base is closely followed by Supervised ViT-Base trained on IN-21k (finetuned on IN-Ik). We more preciscly compare approaches and analyze trends in Section 5.
4.3 Backbones on a Tight Budget
Many computer vision applications demand efficient backbones for fast or on-device inference. In this section, we benchmark three small backbones: RegNetX-4000 [74], EfficientNet-BO [84] and ResNet-18 18 [28] all pretrained in a supervised fashion on ImageNet- We rank the performance of these small backbones the of tasks in Table 3. We find that EfficientNet-BoO performs best overall and across classification, retrieval, and OOD classification, followed by RegNetX-440MF and then 18. Interestingly, ResNets still outperform newer efficient architectures for detection and segmentation.
2To ensure fair comparisons across backbones, we exclude MiDaS variants evaluated on ImageNet for this comparison.
7

---

Table 3: Which tiny backbone should I choose? We rank the most performant very lightweight
backbones (left to right) for various tasks and settings. Green indicates overall comparisons.
Red correspond to OOD evaluations and
Task
Good
Better
Best
1 Cls
ResNet-18
RegNetX-400MF
EfficientNet-B0
2 Det
Ro0NetX-400MM
EfficientNNet-D
ResNet-18
3 Seg
RegNetX-400MF
EfficientNNe-BOO
ResNet-18
4 Ret
ResNet-18
RegNerX-400MF
EfficientNet-B0
5 (OOD) Cls
ResNet-18
RegNetX-400MF
EfficientNet-BO
6 (OOD) Det
EfficientNet-B0
ResNet-18
RegNetX-400MF
7 All
ResNet- 18
RegNetX-400MF
EfficientNet-BO
5 Observations and Trends
> A performance comparison of VīTs and CNNs. Modern architectures strongly outperform vanilla ViTs. We see in Table that the best performing backbone (ConvNeXt-Basso is convolutional, with hierarchical transformer (Swin V2-Base) being close second. The latter transformer architec- ture incorporates a strong spatial inductive bias. These findings suggest that the community should move past vanilla ViTs which are still used frequently. As caveat, we do not evaluate very large models, and it possible that ViTs might outperform their more advanced variants or convolutional networks at larger scales.
VITs benefit more from scale than CNNs. For the suite of backbones considered in BoB, we find that relative performance (z-scores) for both CNNs and ViTs correlates positively with parameter count but more so for ViTs (spearman p 0.58) than for CNNs (spearman p 0.35). Similarly, while overall relative performance correlates with the size of pretraining data, the correlation is again significantly higher for ViTs (p- 0.72) than for CNNs (p = 0.33). This observation indicates that benchmarking much larger backbones might yield different winners, possibly ones with transformer- based architectures.
> Supervised or not? Supervised learning backbones dominate, but primarily because they are available pretrained on larger datasets. SSL backbones can outperform supervised pre- training with similar sized pre-training datasets. We obtain the average score of the top 3 backbones within different pretraining styles, namely self-supervised, supervised with ImageNet- 1K, and supervised with ImageNet-2iK, for each task (see Appendix D). ConvNeXt and SwinV2 pretrained with supervision on ImageNet-21K outperform the SSL backbones on all tasks. The results suggest that we should try using advanced architectures, either convolutional or transformers, when applying SSL methods, and we should train on large datasets to compete with supervised learning. In these experiments, supervised pretraining checkpoints are often available trained on much larger datasets (ImageNet-21k)) When comparing models pretrained on similarly sized datasets, SSL or -uttof-diittibution) and retrieval tasks, which heavily rely on the learned representations. However, supervised learning backbones maintain a decisive edge for detection and segmentation. We can also compare backbones which use the same ViT-Base architecture and find that SSL methods do outperform ImageNet- Ik supervised backbones but are worse than ImageNet-21k trained backbones.
pretraining methods achieve better performance on classification (both in- and
Performance across tasks is highly correlated. Across tasks examined. we find a strong positive Spearman correlation between performance on task pairs (typically p >> 0.8). This finding supports the current trend of general purpose foundation models for computer vision. Moreover, this finding also supports recent work which argues that a single inductive bias can solve wide range of seemingly different problems [24]. However, noteworthy that the retrieval task exhibited comparatively lower but still statistically significant correlation (p = 0.49) with respect to classification and retrieval ranking. This lower correlation can be attributed to the performance limitations of the MiDaS and MAE pretrained models in the context of retrieval. Upon removing these two backbones, the correlation coefficient y increased to 0.8, reinforcing the influence the aforementioned models on the observed results.
8

---

Supercised
Supariiso
Supervisd
VieRog
Vielle,
VidReg
DINO-
DINO
DINO
M---mai
MOCO V--
MOCO
M000 Va-
SeleV2-ilny
MAE
MAR
SvinV2 lose
MAB
CLIP
CLIP
Stabie bittudon-hase
MEDaS
MiDsS
CUP
Stable Diffusion
Stable Diffasioo
Mina8
2255
0.0
2.5
-2.5
0.0
2.5
Detection Porf THE
Seg.
Perf. DIE
Sim10k-- -Citsnages Por. Diif.
Figure 2: Transformers benefit significantly more from end-to-end fine-tuning than CNNs on dense prediction tasks. We visualize the difference in performance between end-to-end fine-tuning and only training the head atop a frozen feature extractor on different tasks. The x-axis is the difference in relative performance (fine-tuning z-score minus fixed backbone z-score). Across panels, the performance differences correlate between tasks.
D Transformers excel under end-to-end fine-tuning while convolutional networks excel under linear probing. For "linear probing" experiments, we freeze a pretrained backbone and only learn the head. Note that for detection and segmentation, the head is more than a linear layer. By inspecting the performance difference between the two fine-tuning strategies (Figure 2), we find that ViTs benefit significantly more from end-to-end fine-tuning compared to CNNs, both for supervised and self-supervised pretraining. See Figure 2 for a comparison on dense prediction tasks.
CLIP models and the promise of advanced architectures in vision-language modeling. For almost all the tasks (except OOD detection), CLIP pretraining is the best among the vanilla vision transformers, even compared to ImageNet-21k supervised trained backbones. Among all the back- bones, CLIP is only worse than ImageNet-21k trained Swin V2 and ConvNeXt, which shows the power of vision- languaee pretraining and again, suggests that we should consider more backbones other than plain ViTs when conducting self- or weakly-supervised learning.
What about generative backbones? In contrast models trained using supervised or self- supervised approaches with contrastive loss, backbones trained with a generative objective, such as MAE or Stable Diffusion, had comparatively inferior performance. We recommend caution when interpreting this result, as the evaluation of Stable Diffusion is currently limited to select tasks. Nonetheless, Stable Diffusion is a larger backbone than others considered in this benchmark and is trained on a very large dataset, yet it exhibits inferior performance.
Battle of the "small" backbones. Keeping limited resources in mind, we also compare the "small" subset of backbones in BoB (< 30M parameters) with ViT-Small, ConvNeet-Tinn Tiny, Swin-Tin Tiny and ResNet-50 architectures. Overall, we find Supervised CenvNeXI-T trained on IN- 1k to be the best, followed by Supervised V2-T trained on IN- Ik and DINO ViT-S trained on IN-IK. Interestingly. supervised learning again dominates, and backbones pretrained on just IN- 1k outperform ones trained on a considerably more diverse and larger dataset (MiDaS).
> Performance vs. Speed? Our analysis reveals strong negative correlation (p -0.41) between throughput (computed on NVIDIA RTX A5000) and average performance z-scores across all tasks when considering each backbone. This finding aligns with our previous observation that larger models tend to exhibit superior performance. Consequently, in order to achieve enhanced performance, one may need to sacrifice speed.
> Monocular depth-estimation as a general purpose pretraining strategy. In our experiments, MiDaS achieves performance competitive with that of top conventional supervised and SSL backbones at classification, object detection, and segmentation, even outside of the natural image domain, for example satellite images. This observation suggests that depth-estimation may serve powerful and generalizable primary or auxiliary pretraining task for foundation models, supporting findings of Lao al. [49].
> Calibration and test likelihood are correlated with accuracy. We measure expected calibration error (ECE) as well as test cross-entropy loss on the ImageNet test set. Whereas test likelihood is

---

strongly correlated with accuracy (r = -0.8278), ECE exhibits a weaker correlation (r = -0.4876) In both cases, we observe p-values under 0.05. We also note that self-supervised pretraining typically leads to inferior calibration.
CNNs and SSL are more adversarially robust. We additionally measure the adversarial robustness of each backbone on the ImageNet test set using an lx -c-noostrained PGD attack with multiple radii (see Appendix Table 19). For each architecture where we possess self-supervised learning versions, we see that supervised pretraining always yields inferior robustness. Moreover, ViTs are more vulnerable to adversarial examples than convolutional networks. Notably, ConvNeXt is more adversarially robust even when trained in supervised fashion.
Where Are Things Going From Here?
At the core of every computer vision model is backbone. In our battle of the backbones, compared more than 1,500 training runs to surface insights for computer vision practitioners and researchers. To guide practitioners, a broad range of tasks from segmentation and detection to classification and retrieval. We found supervised ConvNext, supervised V2, and CLIP models performed well across this broad range of tasks. For computationally constrained settings, in our battle of the "small" backbones we found smaller counterparts to the same archiectures supervised ConvNext-T and S2 followed by DINO with a small ViT performed quite well. BoB offers practitioners a guide to select sensible backbones from the dizzying array of choices.
analyzed the performance of publicly available vision backbones across
For researchers looking ahead, we also observed several notable trends. First, we found performance across tasks is strongly correlated, suggesting shift away from specialized vision backbones to universal backbones that work well across a range of tasks. Next, we found throughput and performance are inverse related, suggesting scaling remains promising avenue to improve backbones. Finally, we found that while our practical recommendations include many supervised models, in apple-to-apples comparisons to standard supervised training, self-supervised learning holds promise. By releasing all our experimental results along with code to put new backbones to the test, we hope BoB serves as a useful guide both practitioners today and researchers looking ahead at tomorrow.
Limitations. We note that insights obtained from BoB are contingent on the vocabulary of tasks, backbones, and settings considered in this work. We intend for takeaways from this study to provide practical considerations useful for computer vision researchers, recognizing that such insights need to continuously evolve more backbones account. Lastly, we note that studies in BoB focus mostly primarily on aspects related to performance, and exploration along other axes of importance (biases in models, etc.) remain.
introduced and more tasks and settings are taken into
Our benchmark does not include backbones larger than ConyNext- Base, aside from Stable Diffusion, and some rankings may change at large scale. For instance, while we find that modern convolu- tional architectures pretrained via supervised learning perform best on most tasks, we also find that transformers benefit more from scale, both in terms of pretraining data and architecture size. It is possible that transformer backbones will pull ahead of convolutional backbones at very large scales.
7 Computation Cost and Carbon Footprint
The experiments in this paper took a cumulative 127k GPU hours on NVIDIA RIX A100 cards. Assuming the GPUs were running with an average carbon efficiency of 0.37 kgCOzeq/kWh,, the total emissions are estimated to be 11792.36 kgCOzeq [48].
Acknowledgements
MG and AGW supported in part by NSF CAREER IIS-2145492, NSF I-DISRE 193471, NIH R01DA048764-01A NSF IIS- 1910266, BigHat Biosciences, Capital One, and an Amazon Research Award. HS and RC were supported in part by the ONR MURI grant N00014-20-1-2877. VP, PC, and JH were supported in part by ARL, NASA ULI, Google, and NSF #2144194. RN MS, GS, and TG were supported by the ONR MURI program, the Office of Naval Research (N0001421 12557), the AFOSR MURI program, and the National Science Foundation (IS-2212182 & 2229885).
10

---

References
[1] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint
2021.
[2] Andrei Barbu, David Mayo, Julian Alverio. William Luo, Christopher Wang. Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale hias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
[3] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg:
inwariancce self self-supervised learning. In International Conference Learning Representations. 2022.
regularization for
[4] Adrien Bardes, Jean Ponce, and Yann LeCun. Vieregl: supervisd learning of local visual features.
In Advances in Neural Information Processing Systems, 2022.
[5] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cmm: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and puitern recognition, pages 6154-6162, 2018.
[6] Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang, Davide Modolo, Rahul Bhotika, Zhuowen Tu. and Stefano Soatto. ceni-supervised vision transformers at seale. arXiv preprint arXiv:2208.05683 05688, 2022.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part I 16, pages 213-229. Springer, 2020.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-965000 2021.
[9] Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint 1906:/9006077
2019.
[10] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton, A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[11] Wei Chen, Yu Liu, Weiping Wang, Erwin Bakker, Theodoros Georgiou, Paul Fieguth, Li Liu, and Michael S Lew. Deep learning for instance retrieval: A survey. arXiv preprint arXiv:2001.1 /282, 2021.
[12] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training sele-supe vision transformers. Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[13] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Van Gool. Domain adaptive faster r-cnn for object detection in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3339-3348, 2018.
[14] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In The Eleventh International Conference on Learning Representations, 2023.
[15] Marius Cordts. Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler. Rodrigo Be- nenson, Uwe Franke, Stefan Roth, and Bert Schicle. The cityscapes dataset for semantic urban seene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5213-3223. 2016.
[16] Ekin D Cubuk, Barret Zoph. Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with a reduced scarch space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255 Icee, 2009.
[18) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 2011112010 2020.
11

---

[19] Mathijs Douze, Hervé Jégou, Harsimrat Sandhawalia, Laurent Amsaleg, and Cordelia Schmid. Evaluation of gist descriptors for web-scale image search. In Proceedings of the ACM International Conference on Image and Video Retrieval, pages 1-8, 2009.
[20] Shiv Ram Dubey, A decade survey of content based image retrieval using deep learning. IEEE Transac- tions on Circuits and Systems for Video Technology. 32(5):2687-22 2704444444 2021.
[21] Linus Eriesson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 5414- 5423, 2021.
[22] Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. What do vision transformers learn? a visual exploration. arXiv preprint arXiv:2212.06727, 2022.
[23] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hicrarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and patter recogmition, pages 580-587, 2014.
[24] Micah Goldblum, Mare Finzi, Keefer Rowan, and Andrew Gordon Wilson, The free lunch the- orem, kolmogorov complexity, and the role of inductive biases in machine learning. arXiv preprint arXiv:2304:05366, 2023.
[25] Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self-supervised visual representation learning. In Proceedings of the ieee/cvf International Conference on computer vision, pages 6391-6400, 2019.
[26] Priya Goyal, Benjamin Lefaudeux, Mannat Singh, Jeremy Reizenstein, Vinicius Reis, Min Xu,, Matthew Leavitt, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra, Vissl. httpa://githut com/facebo http///ebboooeeeerrrrrrreeeeeeee
2021.
[27] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A datasct for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 5356- 5364, 2019.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[29] Kaiming Hc, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 491 1--4777, 2019.
[30] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[31] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):221722266 2019.
[32] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: /2261, 2019.
[33] Dan Hendrycks. Steven Basart, Norman Mu. Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribation generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[34] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 1.2222152771 2021.
[35] Xinyu Huang, Xinjing Cheng, Qichuan Geng, Binbin Cao, Dingfu Zhou, Peng Wang. Yuanqing Lin, and Ruigang Yang. The apolloscape dataset for autonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 954-960, 2018.
[36] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. pmlr, 2015.
12

---

[37] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo. Robyn Ball, Katie Shpanskaya, al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artifiiaal intelligence, volume 33, pages 590-597, 2019.
[38] Matthew Johnson- Robbesoon, Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human- genaannee annotations for real world tasks? In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 746-753. IEEE, 2017.
[39] Rajiv Kapoor, Deepak Sharma, and Tarun Gulati. State of the art content based image retrieval techniques using deep learning: survey. Multimedia Tools and Applications, 80119):29561-22583 2021.
[40] Jacob Devlin Ming-Wes Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[41] Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko. broad study of pre-training for domain generalization and adaptation. In Computer Vision-ECCC ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXXIII, pages 621 638..38. Springer. 2022.
[42] Youngjung Kim, Hyungjoo Jung, Dongbo Min, and Kwanghoon Sohn. Deep monocular depth estimation via integration of global and local predictions. IEEE transactions Image Processing, 27(8):4131-4144 2018.
[43] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. International Conference on Machine Learming, pages 5637-5664.
PMLR, 2021.
[44] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting supe-supe visual representation learning. In Proceedings of the IEFF/CVF conference on computer vision and pattern recogmition, pages 1920-1929, 2019.
[45] Simon Kornblith, Jonathon Shlens, and Quoe Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2661-2677 2019.
[46] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[47] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84 90, 2017.
[48] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700, 2019.
[49] Dong Lao, Alex Wong, and Stefano Soatto. Depth estimation vs classification as pre-training for semantic segmentation. arXiv preprint arXiv:2203.1396 2022.
[50] Da Li. Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542-5550, 2017.
[51] Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429 2021.
[52] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Isruel, October 23-27, 2022, Proceedings, Part IX, pages 280-296. Springer, 2022.
[53] Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2041-2050, 2018.
[54] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
13

---

[55] Tsung-Y Yi Lin Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Coco delection evaluation. http://cocodataset. org/ #detection- eval, 2018.
[56] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012 100222 2021.
[57] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Dong, et al. Swin transformer v2: Scaling up capacity and resolution. Proceedings of the I.FF//VF conference on computer vision and pattern recognition, pages 12009-12019, 2022.
[58] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11976-11988 2022.
[59] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic seg- mentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[60] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,220 2017.
[61] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1rXiv:1306.5151, 2013.
[62] Moritz Menze and Andreas Geiger. Object scene flow autonomous vehicles. Proceedings of the I.F.F. conference on computer vision and pattern recognition, pages 3061-3070, 2015.
[63] Mazda Moayeri, Sahil Singla, and Sohcil Feizi. Hard imagenet: Segmentations for objects with strong spurious cues. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmanks Track, 2022.
[64] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference computer vision, pages 4990-4999, 2017.
[65] Maria-Flena Nilsback and Andrew Zässerman. Automated flower classification over large numher of classes. In 2008 Sixth Indian Conference on Computer Vision. Graphics & Image Processing, pages 722-729. IEEE, 2008.
[66] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 2017.
[67] Xingchao Peng. Qinxun Bai, Xide Xia. Zijun Huang. Kate Saenko, and Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
[68] Florent Perronnin, Yan Liu, and Jean-Michel Renders. A family of contextual measures of similarity between distributions with application image retrieval. In 2009 JEEE Conference on computer vision and pattern recognition, pages 2358-2288 2265. IEEE, 2009.
[69] James Philbin, Ondrej Chum, Michael Isard, Josef Sivie, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. In 2007 IEEE conference on computer vision and pattern recognition, pages 1-8. IEEF, 2007.
[70] James Philbin, Ondrej Chum, Michael Isard, Josef Sivie, and Andrew Zisserman. Lost quantization: Improving particular object retrieval in large scale image databases. In 2008 IEEE conference on computer vision and pattern recognition. pages 1-8. IEEE, 2008
[71] Joaquin Quinonero-Candela. Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008.
[72] Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondtej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5706-5715 2018.
14

---

[73] Alec Radford, Jong Wook Kim. Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR. 2021.
[74] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEFF//VV conference on computer vision and pattern recognition, pages 10428-10436, 2020.
[75] René Ranftl, Katrin Lasinger. David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross dross-d transfer. JEEE transactions on pattern analysis and machine intelligence, 44(3):1623-166) 2020.
[76] Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[77] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjömn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[78] Yangjun Ruan, Yann Dubois, and Chris J Maddison, Optimal representations for covariate shift. arXiv preprint 220.::::0010000557 2021.
[79] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-22 2522122222222 2015.
[80] Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 126:973 992, 2018.
[81] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[82] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rghd images. ECCV (5), 7576:746-760, 2012.
[83] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang. al. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14454-1444 14463, 2021.
[84] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105 611 PMLR, 2019.
[85] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sahlayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. International conference on muchine learning, pages 10347-10357. PMLR, 2021.
[86] Tuan Truong, Sadegh Mohammadi, and Matthias Lenga. How transferable are self supervised features in medical image classification tasks? In Machine Learning for Health, pages 54-74. PMIRR 2021.
[87] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adap- tation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.
[88] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongic. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages $769-8778, 2018
[89] Ashish Vaswani, Noam Shazcer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
[90] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018-5025 502,720222.. 2017.
15

---

[91] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200 dataset. 2011.
[92] Chaoyang Wang, Simon Lucey, Federico Perazzi, and Oliver Wang. Web stereo video supervision for depth prediction from dynamic scenes. In 2019 International Conference on 3D Vision (3DV), pages 348-357. IEEE, 2019.
[93] Haohan Wang. Songwei Ge. Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power, In Advances in Neural Information Processing Systems, pages 10506- 10518, 2019.
[94] Qiang Wang, Shizhen Zheng. Qingsong Yan, Deng, Kaiyong Zhao, and Xiaowen Chu. Irs: A large naturalistic indoor robotics stereo dataset to train deep models for disparity and surface normal estimation. arXiv preprint arXiv:/912.099 2019.
[95] Shuang Wang and Shuqiang Jiang. Instre: a new benchmark for instance-level object retrieval and recognition. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 11(3)1122, 2015.
[96] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafci Hu, Ashish Kapoor, and Sebastian Scherer. Tartanair: A dataset to push the limits of visual slam. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4909 4916. IEEE, 2020.
1971 Ryan Webster, Julien Rabin, Loic Simon, and Frederie Jurie. On the de-duplication of laion-2b, arXiv preprint arXiv:2303. /2733, 2023.
[98] T. Weyand, A. Araujo, B. Cao, and J. Sim. Google Landmarks Dataset v2 A Large-Scale Benchmark for Instance-Level Recognition and Retrieval. Proc. CVPR, 2020.
[99] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Googlc landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2575-2584, 2020.
[100] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2 2021.
[101] Ke Xian, Chunhua Shen, Zhiguo Cao, Hao Lu, Yang Xiao, Ruibo Li, and Zhenbo Luo. Monocular relative depth perception with web stereo data supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 31 1-320, 2018.
[102] Ke Xian, Jianming Zhang, Oliver Wang, Long Mai, Zhe Lin, and Zhiguo Cao. Structure- guiided ranking loss for single image depth prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 611-620, 2020.
[103] Junfei Xiao, Yutong Bai, Alan Yuille, and Zongwei Zhou. Delving into masked autoencoders for multi- label thorax disease classification. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 3588-3600, 2023.
[104] Yao Yao, Zixin Luo, Shiwei Li. Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEFF//CV Conference on Computer Vision and Pattern Recognition, pages 1790-1799, 2020.
[105] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangehen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636-2645, 2020.
[106] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Scyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[107] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario I Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint 190.0:1910.04867. 2019.
[108] Wengang Zhou, Houqiang Li, and Qi Tian. Recent advance in content-based image retrieval: A literature survey. arXiv preprint arxiv: 0eee:776606666 2017.
16

---

A Additional Related Work
Classification benchmarks. Image classification is the most common task in computer vision, and we have multiple benchmarks. For example, the timm library [100] benchmarks ImageNet classification performance across loads of backbones trained with different methods and on different datasets. In addition, we have dataset-specific benchmarks, such as "paperwithcode", The latter contains multiple datasets and methods, but it lacks systematic analysis among these results. Almost all the superviee learning method papers perform their own evaluation for image classification. To accelerate the research cycle in self-supervised learning, VISSL [26] provides a library for implementing SSL methods and evaluations. In this work, we evaluate various backhones trained in both self-supervised and supervised fashion, and on multiple datasets on different domains (natural images, satellite maps, and medical images). Moreover, we benchmark these backbones and datasets with exactly the same learning setting and conduct a thorough analysis of the collected results, which we believe is essential and helpful to practitioners.
Object detection and segmentation benchmarks. Benchmarking backbones for object detection and segmentation has been active area of research. Several works have focused evaluating and comparing the performance of various backbone architectures for these tasks [51, 52, 29, 9]. Popular backbone networks such supervised pretrained ResNet have been extensively utilized and compared, while modern feature extractors such as vision-language models and self-supervised learning models have not been extensively studied. These studies either focus limited subset of backbones or incorporate diverse detectors with varying backbones, thereby hindering an accurate comparison. To the best of our knowledge, we are the first present a comprehensive study of the various backbones with various architectures and pretraining methods for object detection and instance segmentation.
diiti--iisttiuution generalization benchmarks. Several prior works have benchmarked the out- of-distribution performance of visual models. For image classification, these have included variants of ImageNet [32-34, 93, 76, 63], synthetic-t--real adaptation benchmarks [66]. and benchmarks with images belonging to varied domains including paintings, clipart, etc. [50, 67, 90, 87], sometimes even spanning multiple modalities and applications [43]. Similarly for object detection, several OOD generalization benchmarks have been fashioned from sim-to-real, cross-weather and cross-city self-driving datasets [38, 15, 105, 80, 64]. Recently, [41] conducted a broad study of pretraining strategies for domain adaptation and generalization. In this work, we perform a similar study but on a larger scale and also include diverse suite of backbones designed for varied tasks.
Image retrieval benchmarks. To the best of our knowledge, our study represents the first com- prehensive examination of multiple pretrained deep learning backbones for image retrieval task. While previous survey papers [11, 108, 20, 33] have explanations of various types of deep learning methods, such as off-the-shelf models trained in an unsupervised or supervised fashion, single pass, and multiple pass methods, none have quantitatively analyzed these backbones. Therefore, our work fills this crucial gap in the existing literature.
B
An Extended Guide to BoB
B.1 The Architectures
Below is a list of all architectures compare. As differrnt neural network architectures are believed to have distinct properties, from invariances to a reliance on different Fourier frequencies, evaluating variety of architectures will allow us examine potential benefits of architectural differences. Many of the pretraining strategies we evaluate are accompanied by multiple checkpoints with different architectures sizes, include multiple versions of each. We describe architectural modifications to these backbones for object detection and segmentation in Section 3.2.
These are the staple convolutional neural networks we all know and love, complete with skip connections and batch normalization [36]. We include experiments on ResNet-50 and ResNet-101 backbones.
ResNet [28]:
hhtps://ppperswinhoodd.com/
17

---

ConvNeXt [58]: ConvNeXt is a purely convolutional network with numerous modern architectural improvements including depthwise convolutions, inverted bottleneck blocks, large convolutional kernels, and larger proportion of layers allocated to the third stage. This architecture improves performance of convolutional architectures scale while maintaining their strong object detection and segmentation capabilities. We include experiments on ConvNeXI-Ting and ConyNeXI-Base.
Vision Transformer [18]: Vision transformers (ViTs) were derived from transformer language models [89] and inherit their nulti-headed self-attention (MHSA) and position- wise feed-forward network (FFN) components. Unlike ResNets, VITs do not encode translation equivariance, and they only encode locality by embedding images on a patch-by- patch basis. We include experiments on ViT-Small and ViT-Base.
Swin Transformer V2 [57]: Swin Transformer [56] transformer architecture which incorporates hierarchical representations, translation invariance, and increased locality and efficiency into VITs by only performing attention within spatial windows and merging these windows iteratively. SwinV2 is equipped with several modifications which improve scalability and transferability across input resolutions. We include experiments on Swin V2- Tiny and Swin V2-Base. For Swin WwinV2-Bass unless otherwise stated, we use the model with a window size of 24.
Stable Diffusion encoder [77]: Stable Diffusion is a text-to-image generative diffusion model which conducts diffusion in latent space. We include experiments with backbone formed by the learned encoder that converts images from pixel-space into the latent space where diffusion is performed followed by Stable Diffusion's U-Net, and freeze the text encoder, using its frozen embedding. The encoder uses a convolutional architecture with added attention mechanisms. More details can be found in Rombach al. [77].
B.2
The Pretraining Algorithms
The primary source of diversity amongst the backbones we consider stems from their different pretraining algorithms. We choose prototypical examples of categories including supervised learning, self-supervised learning (SSL), and vision- Assion-anguuag since such types of pretraining routines are widely believed to confer their own unique properties. For instance, SSL backbones thought to extract more transferable features [86], while vision-language models are thought to resist domain shifts [78].
Classification: We include image classifiers pretrained on ImageNet-Il Ik and -21k [17]. ImageNet pretraining has long been the de facto choice for computer vision systems.
MiDaS [75]: MiDaS is trained in a supervised fashion for monocular depth estimation. In this task, the model accepts natural image and outputs dense 2D array of depth values representing the distance of the scene from the image plane.
MoCo v3 [12]: Contrastive learning is popular approach to SSL in computer vision which encourages model extract similar features corresponding to different augmentations of the same image, called positive pairs and dissimilar features corresponding to different images, called negative pairs. MoCo v3 is nigh-performing variant which employs a momentum encoder and multi-crop training as well as prediction and projection heads.
VICReg [3]: Instead of adopting contrastive learning and negative pairs, VICReg avoids feature vectors collapsing towards a constant during SSL by regularizing their variance and covariance. VICRegl. is version which also applies the VICReg criterion local features to teach the model to extract localization information in its features for downstream dense prediction tasks [4].
DINO [8]: Much like MoCo v3, DINO uses a momentum encoder and multi-crop SSL, but DINO swaps out contrastive learning for self-distillation, demonstrating strong performance on ViT's with a small patch size.
MAE [30]: Masked Autoencoders (MAE) use a distinct style of SSL adapted from masked language modeling [40]. MAE models are trained to reconstruct masked out input patches, unlike the above 3 models which instead match the features of augmented images.
18

---

CLIP [73]: CLIP also uses contrastive learning, but on image-caption pairs instead of augmented image views. Language supervision endows CLIP features with information relating to the semantic meaning of image components, compared to models trained solely on image data [22]. We only use CLIPP's image feature extractor our experiments.
Stable Diffusion [77]: Text-to-image diffusion models an entirely different type of vision-language backbone, trained for image generation. The Stable Diffusion encoder, which we benchmark, maps images to a highly compressed latent space where diffusion performed.
Random initialization: In experiments where we fine-tune backbones on downstream tasks, we also evaluate basclines trained on the downstream training sets from random initialization.
B.3 The Pretraining Datasets
The backbones we benchmark are pretrained on datasets across wide range of scales including image classification, image-text pairs, and images with depth annotations:
ImageNet-1k and -21k [17]: ImageNet-21K 2kk contains over 14 million training images in 31.841 classes. ImageNet- is a subset of the aforementioned dataset containing almost 1.3 million training images in 1000 classes. These popular web-scraped image classification datasets are used for supervised pretraining with the labels, or self-supervised pretraining without them, among numerous backbones we benchmark. We denote pretraining ImageNet-211 Ik followed by fine-tuning on ImageNet- Ik by "ImageNet-2lk-IIX
LAION-2B [81]: LAION-2B is subset of the larger LAION-5B, which contains 5.85 billion web-scraped image-text pairs filtered by CLIP. LAION-2B specifically contains those 2.32 billion pairs with English language captions. Despite being by far the largest dataset amongst those we consider, LAION-2B is known to contain a large number of duplicates [97]. Stable Diffusion is further fine-tuned on LAION-Aesthetic a subset of LAION-2B containing 120 million images filtered by model trained to rate images by aesthetics.
CLIP [73]; Since there is no OpenCLIP ResNet checkpoint available, we use the orig- inal CLIP checkpoint trained OpenAT's diverse proprietary captioned-image dataset containing 400 million images scraped from publicly available internet sources.
MiDaS [75]: MiDaS was trained on a combination 12 image datasets with various types of depth annotations, and objectives: ReDWeb [101], DIML [42]. Movies [75], MegaDepth [53]. WSVD [92]. TartanAir [96], HRWSI [102], ApolloScape [35], BlendedMVS [104], IRS [94], KITTI [62], NYU Depth V2 [82]. These models are therefore trained with multi- objective optimization. Collectively, the MiDaS training set contains more than 3.7 million images. An earlier version of MiDaS trained on smaller collection of 5 datasets, but we use the most recent version trained on the largest training set.
Evaluation datasets and licenses. In Table 4, Table 5, Table 6, and Table 7 we summarize the datasets we use for evaluating classification, object detection, segmentation, out-of-domain generalization and retrieval performance. We include the number of classes well the number of test samples for each dataset. To be noticed, we only use 10% of the labeled dataset for EuroS and Chexpert distinguish the performance among all the backbones. Object detection and instance segmentation experiments are conducted on COCO dateset [54]. COCO is released under the Creative Commons Attribution 4.0 License This license permits users to share and adapt the dataset for any purpose, as long as the original creators are appropriately credited. For OOD classification, we use the ImageNet- ImageNet-Adversariil [34], ImageNet-Skee Skeech [93]. ImageNet-Renditonn
[33], ImageNet- V2 [76], and VisDA [66] datasets, all of which are freely available for research use. For OOD detection, we use the Sim10k [38] and Cityscapes [15] datasets. Densely annotated images for Sim10k are available freely" and can only be used for non commereial applications. The license agreement for the Cityscapes dataset dictates that the dataset made freely available to academic and non-academic entities for con-commeccaal purposes such as academic research, teaching, scientific publications, or personal experimentation and that permission to use the data is granted under certain
hhtps://ffcaveenggn wwwch. wtt.::/www...iciccccccoopee/e//r in th-thhhmatix
19

---

Table Image Classification Datasets
Dataset
Description
Size
Classes
ImageNet-Ile |k [17]
Natrrll images of versatile categories
1.3M
1,000
CIFAR- 100 [46]
Natural images of versatile categories
50K
100
EuroSAT [31]
Satellite images (RGB) land use and land cover
13.5K
10
Flowers- 102 [65]
Images of flowers categories
IK
102
Aircraft [61]
Images of aireraft model variant, family, manufacturer
3K
100
Chexpert [37]
Medical images
191K
5
Table 5: Object Detection and Instance Segmentation Datasets
Dataset
Description
Size
Classes
COCO [54]
Large-scale object detection and segmentation dataset
330K
80
conditions. 6, All datasets used in benchmarking retrieval tasks (except for Objectnet) are restricted to non-commercial research and educational purposes. Objectnet is free to use for both research and commercial applications."
C Experimental Details
C.1 Implementation Details
Classification. For fine-tuning, train the backbones for 100 epochs using AdamW [60] and weight decay (5e 2 1)_ We use a cosine annealing learning rate scheduler with 5 warmup epochs. We run grid searches for learning rates with the default grid range being {le-3 5e-4, le 4] as we observe peaking performance on multiple models with these learning rates. We expand the search range for learning rate when training models from scratch (i.e., fine-tuning from randomly initialized weights) to f1e-2, 5e 1e-3, 5e le 1e_4). We keep the batch size of 1024 the same for all experiments and use gradient accumulation when Out-of -eimo-ry--rr occurs for large models (such as the Stable Diffusion encoder). For data augmentation, we follow He et al. [30], including random horizontal flips and crops, mixup, CutMix, and RandAug [16] policy with hyperparameter (9, 0.5) corresponding to the number of transformations and magnitude. For regularization strategies, we apply the stochastic depths with a drop rate of 0.1, layer-wise learning rate decay of 0.75, and label smoothing with rate of 0.1. For linear probing, again follow He al. [30], where set weight decay to zero and disable Mix-Up, Cut-Mix, stochastic depth, or color jitter. We train the model with LARS optimizer with batch size of 4096 for 90 epochs. For fine-tuning on low-shot ImageNet, we follow Cai et al. [6], where we use AdamW for all the transformer based backbones and SGD for Convolution-oni only backbones. For transformer baaedd backbones, among three peak learning rates of {le 2.5e 4, 5e 4] and two decay rates of 0.65, 0.75 for AdamWi W. We use grid search among three peak learning rates of {le_t 1 5e 2) for SGD. We fix the
grid search
7 http::/loiicceee.ddvvddoooooooad
Table 6: OOD Generalization Datasets
Task
Train Dataset
Test Dataset
Test Size
Classes
Image Classification
ImageNet- IK [79]
ImageNet-A [34]
7.5K
0.2K
Image Classification
ImageNet- 1K [79]
ImageNet-v2 [76]
10K
1K
Image Classification
ImageNet- 1K [79]
ImageNet-RR [33]
30K
0.2K
Image Classification
ImageNet- 1K [79]
ImageNet-S [93]
50K
1K
Image Classification
MiDDA-Sytheiis [66]
VisDA-Reat [66]
55.4K
12
Object Detection
Sim [38]
Cityscapes [15]
0.5K
1
20

---

Table 7: Image Retrieval Datasets:
Kindly note that the provided numbers are approximate refer- ences. For precise details, please consult the original research papers.
Dataset
Description
Size
Classes
CUB-200 [91]
Fine-grained bird images
12k
200
i-Naturalist [88]
Fine-grained species classification
450k
>1k
Object Net [2]
Object recognition dataset with uncommon poses
50k
313
INSTRE [95]
Fine-grained recognition data set toys and house-hold objects
28k
200
Google Landmarks V2 [98]
Landmarks across the world
mil
>1k
Oxford [69]
Specific buildings from Oxford Specifie landmarks and buildings from Paris Copy detection dataset of landscapes, buildings etc
6400
12
Paris [70]
5000
11
Copy Days [19]
2,286
n/a
weight decay to be 0.05 and use the same data augmentation the regular fine-tuning for 10% but without strong data augmentations, such as mix-up, cut-mix, and random erasing for the 1% setup.
Object Detection and Segmentation. For the training of Cascade Mask R-CNN, we adopt the standard training settings previously employed in ConvNext [58], Swin [56], and ViTDet [52]. For all experiments, we utilize the AdamW optimizer [60] with weight decay of 0.05, batch size of 16, and we conduct grid search for the learning rate, considering values of {8e 1e_1, 2e- -,3e *,3e-t). "}- Additionally, we employ 3x schedule, spanning total of 36 epochs, and the learning rate decayed by a factor of 10 at epochs 27 and 33. In addition, we apply multi-scale training [7, 83), excluding ViT-based backbones and Stable Diffusion. For ViT-based backbones, inspired by the approach employed VITDet along with the fact that ViT backbones perform poorly on detection without specially tailored training strategies, we use large-scale jitter (LSJ) with the image resolution of 1024 x 1024 and scale range of [0.1, 2.0]. In order to maintain fair comparisons across backbones, we minimize architecture- specific modifications. Thus, for our VIT backbones, we avoid implementing some ViTDet modifications such as "simple feature pyramid" instead of FPN. Instead, we employ an FPN that utilizes the final feature map, without employing stage division. It is worth noting, as highlighted in the ViTDet paper, the performance of "FPN, last-map" is inferior to the "simple feature pyramid".. Additionally, we use the supervised training results from DeiT [85] for supervised ViT-S and ViT-B pretrained backbones. For Stable Diffusion, we use a single resolution of 512 x 512 (resizing the image such that the longer side is 512) overcome the significantly larger computational cost of this backbone compared to its competitors in our benchmark.
A note on ViIDet and scale: architectural modifications and very long training schedules can benefit ViTs in particular, as was found in Li et al. [52]. Similarly, Chen et al. [14] point out that ViTDet achieves stronger performance than their own work due to long and expensive training routines, behavior which stems from ViTs weak vision inductive bias. Additionally, in our analysis, we found that ViTs benefit more from scale, ViTs might overtake other models at larger scales. Including large models in our benchmark, which includes many tasks and pretraining methods, would be prohibitively expensive, yet practitioners with sufficient computational resources for larger models, longer training routines, and architectural modifications may consider ViT backbones.
Syn-to-Res Real Generalization. For VisDA [66] Syn-Reea, we report accuracy the target domain (real) using models trained on the source domain (synthetic) for 12-way classification. For training, use a learning rate of le on 4 A40 GPUs with a batch size of 64 and report accuracy after 10 epochs. For object detection, we train the backbones outlined in Appendix B. with the Cascade- RCNN architecture. For training and fine-tuning backbones on Siml0k [38], we use learning rate of le 44 on 4 A40 or RTX 6000 GPUs To enable consistent evaluation across all backbones (CNNs and ViTs), we downsample Cityscapes [15] images to 1024 512 during evaluation. We train all models on the entirety of Sim 10k and evaluate on the validation split of Cityscapes.
Image Retrieval. We have only evaluated pretrained models for this task. Dataset and metrics are discussed in the main body. Refer to table Table 7 for a brief summary of all the retrieval datasets.
D Results
Image Classification, We present the ImageNet Top-1 and Top-5 classification accuracy for back- bones pretrained with different methods and on various datasets in Table 8. We adopt the ImageNet results for supervised learning with random initialization from the timm library [100]. We omit
21

---

ImageNet results for ImageNet-pretrained pretrained backbones since those coincide with the randomly ini- tialized backbones on ImageNet. We also present top-1 classification accuracy for finetuning on various datasets in Table 9, and we include ImageNet calibration and uncertainty estimation results in Table 10.
Table 8: Classification accuracy (%) ) for ImageNet-related tasks. "Ip" denotes linear probing, "I "1%"
and " "10%" denote the percentage of labeled training images used during the fine-tuning.
ImageNet (Ip)
ImageNet
ImageNet (10%)
Image Net
Backbone
Method
Pretrain
Top-1
Top-3
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Supervised
ImageNet. Ik
94.60
ResNet-50
VicReg
ImageNet-Il lk
72.15
90.22
78.77
94.29
69,54
89.07
55.04
79.34
CLIP
LAION-2B
65.98
87.84
80.55
95.26
69.26
89,91
43.78
70.67
DINO
ImageNett Ik
74.17
91.56
79.08
94.60
68.18
89.35
51.38
77.82
Supervised
ImageNet- lk
78.34
94.29
ViT-Small
MoCoV3
InageNet-1k
73.11
90,94
79.65
94.96
70,27
90.11
$4.14
79.15
DINO
ImageNet-11 1k
76.08
92.63
81.33
95.71
73.83
91.89 89
58.15
80.07
Supervised
IeageNet-11 1k
79.15
94.09
McCoV3
ImageNet- 1k
75.96
92.69
82.85
96.31
74.80
92.54
62.88
85,31
ViT-Base
MAE
ImageNet-
67.67
87.49
83.41
96.50
72.87
91.54
56.02
81.07
DINO
ImageNet-Ik
77.31
93.43
83,40
96,42
75.92
93.30
63.92
84,52
CLIP
LAION-2B
79.74
95.53
85.19
97.46
78.67
95.00
66.44
89.11
SwinV2-Tiny
Supervised
IageNet-
81.82
99.99
MiDaS
MiDaS
76.44
92.66
82.55
95.92
79.92
94.57
75.44
90.83
SwinV2 Base
Supervised
ImageNet-2Ik-IK
87.10
98.23
MiDaS
MiDaS
81.09
95.47
86.48
98.00
84.14
96.91
79.26
93.66
Stable Diffusion
Stable Diflusion
LAION-2B
79.90
95.10
71.30
89.07
38.02
65.78
Table 9: Top-1 classification accuracy (%) for fine-tuning pretrained (and randomly initialized) backbones with different methods on various datasets. We omit the ImageNet performance for the backbones trained on ImageNet supervised fashion that setup the same as backbones trained from random initialization.
Backbone
Method
Pretrain Data
ImageNet 80.38
EuroSAT
Flower
CIFAR-100
Chexpert
Aireraft
Rand Tnit
89.61
11.36
72.33
86.67
20.73
Supervised
ImageNet-Ik
98.26
86.52
84.71
86.82
55.31
ResNet-50
Viekeg
ImageNet-1k
78.77
95.11
92.68
87.56
86.55
67.51
CLIP
LAION
98.72
88,92
73.50
DINO
ImageNet- Ik
79.08
98.74
94.04
86.49
87.49
74.13
ResNet-101
Rand
81.93
62.07
87.41
Supervised
ImageNet-1k
97.19
83.59
83.07
86.51
40.47
Rand Init
78.84
42.61
38.73
56.08
77.36
5.52
Supervised
ImageNet-Ik
95.54
87.60
61.69
VIT-Small
Supervised
ImageNet-21k
81.39
95.30
98.73
92.51
87.39
59.89
MoCoV3
ImageNet-Ik
79.65
96.02
83,59
88.10
54.68
DINO
ImageNet-1k
81.33
89.09
73.73
90.00
87.75
48.20
Rand Init
79.15
48.33
40,20
$4,30
79.61
6.99
Supervised
ImageNet-1k
96.22
94.04
90.62
87.27
62.02
Supervised
ImageNet-21k
84.53
94.93
99.41
93.89
87.72
70.2
VTT-Base
McCoV3
ImageNet-Ik
82.85
96.74
94.53
90.89
86.82
71.55
MAE
InageNet 1k
83.41
95.54
94.63
89 99.96
88.01
72.27
DINO
ImagaNet-IK
83.40
95.59
97.17
91.22
87.05
71.82
CLIP
LAION-2B
85.19
94,37
96.78
91.22
87.74
76.38
Init
87.76
SwinV2-Tiny
Supervised
InageNet- 1k
98.91
96.58
89.50
88.39
68.71
MiDaN
MiDaS
69.54
SwinV2-Bis
Supervised
ImageNet-222 221-e 1lk-lk
87.10
95.94
99.61
93.09
87.73
79.46
MiDaS
MiDaS
86.48
96.53
99.32
93.08
88.08
79.62
ConNNeXi-Ting
Rand Init
82.10
73,80
18.24
75.31
83.25
4.35
Supervised
ImageNet-Ik
95.70
96.00
89.89
87.56
69.99
Rand Init Supervised
30.39
19.41
85.26
ConvNeXt-Base
ImageNet-1k
93.06
94.92
88.98
88.98
64.72
Supervised
ImageNet-21k
85.87
96.19
99.61
92.84
87,80
69.39
Stable Dillusion
Stable Dillusion
LBION-2B
79.90
92.22
91.89
90.50
87.49
72.45
Object Detection and Segmentation. In our experiment, we utilize the train2017 and val2017 splits of the COCO dataset for training and evaluation, respectively. We report results on bounding box object detection (APhor) and instance segmentation (Apmask)) In Table 1, Table 12 and Table 13, we present the comprehensive results of our experiment. Table reflects various backbone architectures when the detector is fine-tuned while the backbone remains frozen. Table 12 contains both training from scratch with randomly initialized backbones as well as end-to- end fine-tuning using pretrained backbones. In Table 13, additionally present results the harder
results obtained with
22

---

Table 10: Calibration and uncertainty estimation on ImageNet. We measure Top-1 accuracy (%),
cross-entropy test loss (CE), and expected calibration error (ECE).
Backbone
Method
Pretrain Data
Accuracy
CE
ECE
Supervised
ImageNet-1k
80.38
0.94
0.09
VicReg
ImageNet-1k
78.77
1.11
0.21
ResNet-50
CLIP
LAION-2B
80.55
1.02
0.19
DINO
ImageNet-1k
79.08
1.11
0.22
ResNet-101
Supervised
ImageNet-1K
81.93
0.92
0.16
Supervised
ImageNet-1k
78.84
0.84
0.03
Supervised
ImageNet-21k
81.39
0.68
0.01
ViT-Small
MoCoV3
ImageNet-1k
79.65
0.90
0.11
DINO
ImageNet-1k
81.33
0.83
0.10
Supervised
ImageNet-1k
79.15
0.86
0.05
Supervised
ImageNet-21k
84.53
0.56
0.01
MoCoV3
ImageNet-Ik
82.85
0.77
0.08
ViT-Base
MAE
ImageNet-1k
83.41
0.75
0.09
DINO
ImageNet-1k
83.40
0.76
0.07
CLIP
LAION-2B
85.19
0.66
0.08
Supervised
ImageNet-lk
81.82
0.83
0.09
SWinV2-Tiny
MiDaS
MiDaS
82.55
0.83
0.07
ConvNeXt-Tiny
Supervised
ImageNet-Ik
82.10
0.79
0.06
Supervised
ImageNet-1k
83.88
0.69
0.04
ConvNeXt-Base
Supervised
ImageNet-21k
85.87
0.56
0.03
LVIS dataset [27] using the best transformer-based (SwinV2) and convolutional-bsse (ConvNeXt) architectures from our experiments at several sizes. We again see benefit of scale here as well as the slightly superior performance of modern convolutional architectures. These tables provide a comprehensive overview of the performance achieved across various backbones and scenarios, enabling a thorough analysis and comparison of the different backbones utilized in our study.
Table 11: Object detection and instance segmentation using Cascade Mask-RCNI on COCO with frozen backbones.
Pretrain
Input Size
Aplo
APio-
APlcs
Apmusk
APmusk
APmusk
Supervised VicReg
ImageNes-1k -1k
92M
1333 x 800
42.5
61.0
46.2
37.1
38.1
39.5
ResNet-50
ImageNer-Ik
82M
1333 x 800
44.1
62.3
48.1
38.8
50.7
42.0
DINO
ImageNet-1k
60.3
ResNet-101
Supervised
ImageNes- lk
101M
1333 x 800
43.4
62.1
47.1
38.0
39.3
40.7
Supervised
ImagoNet-1k 1k ImageNet-Ik
84M
1024
1024
27.3
44.3
22..1
23.6
40.8
23.9
NiT-Small
MoCoV3
SIM
1024
27.8
DINO
ImageNer- 1k
94M
1024 x 1024
33.6
52.5
35.9
20.1
49.0
30.0
Supervised
ImngoNet-11 lk
155M
1024 1021
31.1
51.0
36.1
29.6
50066
30.0
MoCoV3
ImageNet-
155M
1024
ViT-Base
MAE
InageNet- 1k
155M
1021 x 1021
35.6
54.0
38.7
31.5
51.1
33.7
DINO
ImageNet-1 1kk
155M
1024
1024
36.2
55.6
39.
31.7
52.2
32.9
LAION-2R
156M
1024
SwinV2-Tiny
Supervised
MiDaS
ImageNet-11 1k MiDuS
SIM
9IM
1333 x 800
1533
36.9
31..3
51.5
55.4
37.2
39.9
30.2
32.55
32.6
48.9
32.0
34.4
Supervised
ImageNes-1k
37.1
34.4
SwinV2-Base-we
Supervised
MiDaS
ImageNet-21k-11
MiDaS
145M
145M
1303 x 800
1533 500
44.6
42.2
64.7
61.55
45.9
48.6
37.1
38.8
61.6
58.7
39.3
41.5
ConvNeXi-Tiny
Supervised
ImageNet-1k
60.7
Supervised
ImageNer-1k
146M
1333 x 800
44.8
69.1
48.6
381.2
61.4
42.0
Supervised
ImageNet-21k
146
1333 x 800
46.2
66.0
5002
40.1
62.9
43.0
Stable
LAION-2B
35.4
23

---

Table 12: Object defection and instance segmentation results using Cascade Mask-RCNN on
COCO.
Hackhone
Method
Pretrain Data
Params
Input Size
A
AP
AP
Aprusak
AP
A
Random fnir Supervised VicReg
92M
41.3
57.4
45.1
36.0
35.1
ImageNer-11 Ik ImageNet-1k
61.9
RosNer-50
82M
1333 x 800
38.2
55.2
41.5
33.5
52.9
35.9
DINO
ImageNet-11
82M
1533
800
45.4
63.5
19.2
39.1
61.1
12.4
ResNet-101
Superviscd
ImageNet-1k
101M
101M
1333 800
45.7
47.7
63.0
65.6
50.0
52.0
41.3
00.5
63.2
44.6
Rundom Init
S4M
1024
1024
43.2
62.2
47.2
38.0
59.3
40.7
Supervised
ImageNet-11 Ik ImageNes-11
SIM
1024
NiT-Small
MrCoV3
94M
1024 x 1024
47.6
67.5
51.7
41.6
64.3
44.4
DINO
ImageNet-1k
84M
1024 1024
48.2
67.6
62.4
422.3
64.9
44.9
Supervised
ImageNes-14 1k
155M
1024 x 1024
49.4
60.3
533.5
42.9
66.1
46.2
ViT-Base
MoCoV3
ImageNer-11 Ik
ImageNet-14
155M
7024
1024 1024
49.7
63.4
633.0
43.2
06.6
48.5
DINO
ImageNet-11 1k
155M
1021 x 1024
29.5
69.0
53.7
42.8
66.1
16.0
CLIP
LAION-2B
155M
1024 x 1024
50.0
69.3
54.4
4.3.3
66.33
16.6
Rundom Init Supervised
S6M
47.0
65.3
51.2
40.8
62.7
44.1
SwinV2-Tiny
ImageNet-1k MiDuS
96M
50:2
00.1
64.6
43.4
06.3
46.9
MiDaS
86M
1533
800
50.2
69.3
54.5
43.5
66.1
17.0
Swin W2BBBs-wwBBB
Rundom Init
Supervised
ImageNet-1k
146M
145M
1333
48.1
52.4
66.6
71.0
52.3
67.1
41.5
45.2
64.1
68.6
49.1
44.7
2eiiaV2-Bse--24
Supervised MiDaS
ImageNet-2166 21k-
MilaS
145M
145M
1533 x 800
52.7
52.9
71.4
71.4
57.2
67.5
45.7
69.0
63.0
49.6
ConvNeXi: Tiny
Random Init Supervised
88M
1333
47.5
65.5
51.7
41.2
63.0
11.3
ImageNet-1K
8GM
1353
900
49.9
68.1
54.3
43.2
66.0
46.8
Rundom Init Supervised Supervised
146M
48.3
66.4
52.7
41.9
63.8
45.3
ConvNeXI-Bass
ImageNet-1k
140M
51.7
70.2
56.0
44.6
67.7
48.3
ImageNot-21k
146M
1553
800
52.9
71.7
57.3
45.8
69.2
19.9
Rundom Init Stable Diffusion
442M
37.1
51.4
40.1
31.9
43.7
31.2
Stable Diffusion
LAION-2B
442M
1333
43.4
38.1
46.3
38.
36.9
40.2
Table 13: Object detection and instance segmentation using Cascade Mask-RCNN on LVIS v1.
Backhone
Method
Pretrain Data
Params
Input Size
Aphar
APb
APto
Apmest
APm
AP
VWinV2-Tiny
Supervised MiDDaS
ImageNet. Ik
8GM
1333 x 800
33.0
46.3
35.3
29.9
44.5
31.9
Swin MaeVV2--Basa B-Basc-w8
Supervised
ImageNet- Ik
145M
1333
800
36.7
48.7
38.0
32.0
47.0
34.4
CoavNeXi-Tinny
Supervised
ImageNet- 1k
86M
1333 x 800
33.2
46.1
35.4
29.9
41.3
32.2
Com-NeXttBase
Supervised
ImageNet-
32.0
46.9
OOD Generalization. We include results for OOD generalization for image classification in Table 14 and for object detection in Table 15.
Table 14: Top-1 OOD classification accuracy (%).
Test Dataset
Test Dataset
Backbome
Method
Pretrain
Train Dataset
ImageNet
ImageNet ImageNet-K ImageNet
ViSDA (real)
Init Supervised
VisDA (syn)
22.07
ImageNet- Ik
10.03
29.63
40.25
68.75
VisDA (syn)
47.77
ResNet-50
VicReg
ImageNet-11 1k
39.98
VisDA (syn)
44.19
C1.IP
LAION-2B
15.53
30.10
42.44
69
VisDA (syn)
21.98
ImageNet-Ik
28.38
39.74
67.55
VisDA (syn)
Rand Init
VisDA
21.81
ResNet-101
Supervised
ImageNet-Ik
19.04
34.73
45.10
70.88
VisDA (syn)
56.56
VisDA (syn)
22,16
Supervised
ImageNer- Ik
14.85
27.99
38.02
66.45
Vi<DA (syn)
63.57
Supervised
ImageNet-2Ik-II
43.21
56.79
74.01
VisDA (syn)
65.62
ViT-Base
McCoV3
ImageNet-Ik 1k
32.57
37.33
34.85
72.71
VisDA (syn)
57.21
MAB ImageNet- 1k
36.2
34.75
48.77
73.47
VisDA (syn)
48.90
DINO
ImageNet-Ik
35.32
36.52
49.15
72.71
VisDA (syn)
59.96
CLIP
LAION-25 2B
46.59
63.33
50.17
75.39
VisDA (syn)
46.05
SwinV2-Tiny
MiDaS
MiDaS
29.41
41.26
VisDA (syn)
61.62
Supervised
ImageNet-21k-16
45.97
37.00
49.69
74.42
VisDA (syn)
68.84
SwinV2-Base
MiDaSS
MiDaS
63.05
48.77
63.12
77.06
VisDA (syn)
68.66
ConvNeXi-Be Base
Supervised
ImageNet-Ik
37.29
$1.07
74.09
VisDA (syn)
71.12
Supervised
ImageNet 21k
60.8
48.92
62.53
76.96
VisDA (syn)
74.96
24

---

Table 15: OOD object detection on Sim10k- Citysscppss.. Out-of-distribution generalization across backbones for object detection Bascade-RCNNY models trained on Sim 10k and evaluated on Cityscapes to detect instances "car". (Frozen) column corresponds settings where the backbone has been frozen.
Backbone
Method
Pretrain Data
mAP@50
mAP@50 (Frozen)
Random Init Supervised VicReg
30.6
ImageNet-1k
46.5
52.4
ResNct-50
ImageNet-IK
44.4
49.8
DINO
ImageNet-Ik
50.0
52.2
Random Init Supervised
25.2
ResNet-
ImageNet-1k
46.9
52.6
Random Init
8.7
Supervised
ImageNet-Il Ik
33.7
15.9
ViT-Small
MoCoV3
ImageNet-Ik
32.8
19.4
DINO
ImageNet-Ik
43.0
31.3
Random Init
10.3
Supervised
ImageNet-Ik
38.2
24.6
MoCoV3
ImageNet--I
36.7
27.9
ViT-Base
MAE
ImageNet-Ik
44.2
32.3
DINO
ImageNet-Ik
41.7
35.1
CLIP
LAION-2B
38.1
9.7
Random Init
40.9
SwinV2-Tiny
Supervised
ImageNet-Ik MiDaS
48.0
48.2
MiDaS
49.5
45.5
Random Init
38.0
SwinV2-Base-w8
Supervised
ImageNet-
51.1
49.3
Supervised MiDaS
ImageNet-21kk-I
51.0
50.2
SwinV2-Base-w22
MiDas
52.5
50.3
Random Init Supervised
36.2
ConvNeXt-Tiny
ImageNet-Ik
54.5
50.3
Random Init Supervised Supervised
25.5
ConvNeXt-Bass
ImageNet- lk ImageNet-21k
55.5
52. 1
52.4
53.9
Retrieval. We present the mAP, MRR and Recall @5 values for various backbones across different datasets in Table 16, Table 17, and Table 18, respectively.
Analysis. In Table 20, compare the performance of backbones pretrained with SSL (including CLIP) and supervised learning on ImageNet- Ik and ImageNet-21k. We pick the top-3 backbones in each category and calculate the mean z-scores for all the tasks.
Adversarial robustness. In Table 19, we show adversarial robustness on the ImageNet test set against untargeted PGD adversarial attack with &oo constraints of and 2. For attack hyperparameters, we use 20 steps and step size &.
255
E
Assets and Licenses
The assets used in this work can be categorized as Code Repositories, Backbones and Dependencies (licenses for datasets are included in Appendix B.3).
25

---

Table 16: MAP scores for image refrieval experiments.
Method
Data
Obj
INSTRE
GLM
Copyl)
Con-Next-Base
Supervised Supervised
ImageNet-Ik ImageNet-2Ik-Ik
20.55
5.53
11.42
52.13
12.71
26.73
55.84
79.32
62.51
19.43
19.57
69.4
21.34
42.42
73.33
86.18
Supervised
ImageNet- Ik
23.96
5.66
10.35
49.33
12.6
28.1
56.03
77.86
Con-Next-Small
Supervised
InageNet-21k-I4
61.09
17.06
17.12
62.79
19.53
42.19
71.03
83.56
Com-Next-Xl.arg
Supervised
ImageNet-2Ik-Ik
65.31
21.98
21.37
69.28
21.67
45.33
74.76
86.81
ResNet-101
CLIP
OpenAl
21.75
4.50
5.4
56.71
15.35
23.93
52.1
79.07
Supervised
ImageNet-
22.86
5.52
8.9
47.37
13.83
31.27
58.51
78.52
ResNet-50
CLIP
OpenAI
15.37
3:45
3.81
48.10
14.20
24.09
52.14
79.61
Supervised
ImageNet-Ik lk
18.75
4.28
5.31
41.59
11.98
25.59
52.37
79.78
ResNet-50
VieReg
ImageNet-1k 1k
8.84
3.15
1.82
42.45
13.76
31,71
58.24
82.48
CeoNNe--S0x6a
CLIP
OpenAl
42,30
10.02
15.11
85,38
26.91
41,93
70.27
91.36
V2-Bae
Supervised
MiDaS
ImageNet-2Ik-Ik
MiDaS
57.57
10.44
17.95
3.84
17.81
1.93
66.87
24.91
20.33
11.52
44.35
29.76
71.28
56.34
84.03
87.57
W16
Supervised
InageNet-21k-Ik
58.77
19.93
18.6
69.48
20.97
44.91
71.7
88.2
Swin
Supervised
InageNet-2Ik-Ik
57,55
17.74
16.36
71
21.64
44.26
75.58
88.59
SwinV2-Tiny
MiDaS
MiDaS
10.44
3.84
1.93
24.91
11.52
29.76
56.34
84.03
SwinV2-Tiny
Supervised
Supervised
ImageNet-Ik
ImageNet-Ik
22.91
22.91
5.71
5.71
6.94
6.94
54.31
54.31
13.76
13.76
27.35
27.35
56.29
56.29
82.93
82.93
CLIP
LAION-2B
40.27
8.17
12.98
81.13
23.83
44.84
73.62
88.79
ViT-Base
MAE
ImageNet-Ik lk
1.26
0.22
0.43
4.86
1.43
5.19
11.12
48.12
ViT-Base
McCoV3
ImageNet-
12.97
4.16
2.29
40.53
12.11
30.25
51.6
84.75
Supervised
ImageNet-Il
17.09
3.81
4.59
39.75
9.64
21.97
50.03
77.7
ViT-Base
Supervised
DINO
ImageNet-2Ik.I1
ImageNet-1k 1k
22.21
52.18
11.99
7.43
4.57
7.56
46.58
53.01
15.35
14.3
37.03
30.96
59,82
62.22
86.18
83.65
CLIP
LAION-2B
47.77
9.62
12.87
80,29
25.45
39.19
90.32
ViT-Large
MAE
ImageNet-Ik
2.53
0.84
0.53
12.24
3.78
13.26
24.92
70.34
ViT-Large
Supervised
ImageNet-21k-14
62.44
18.19
16
55.18
18.49
37.68
67.07
87.04
Vit-Small
MoCoV3
ImageNet- 1k ImageNet-
12,99
3.7
1.86
35.28
11.39
24.83
501,44
82.67
Supervised
19.44
4.05
4.61
40.45
10.51
21.69
49.08
79.54
Vit-Small
Supervised
DINO
ImageNet-21k.I
ImageNet-1k
31.38
7.35
6.34
3.99
52.64
48.17
14.37
14.79
37.98
61.01
81.92
85,3
Code Repositories. We provide supporting code for all our experiments here. For image classification experiments, we build on top of the timm library [100]*, the original MAE repo" and the medical dataset pretrain repo [103]'' timm is distributed under Apache 2.0 License and MAE under the 4.0 International License. For object detection, instance segmentation, and OOD detection experiments, we build on top of the MMDetection framework " MMDetection is distributed under Apache License 2.0.
Backbones. We use publicily available pretrained backbones. The full list is provided in Appendix
Dependencies. Key dependencies for all our experiments include pytorch, timm, HuggingFace utilities and MMCV. Please see our repo README for comprehensive list of all dependencies to reproduce the experiments.
F Observations about Hyperparameters
For hyperparameter tuning, find that the learning rate strategy is highly method- and dataset- dependent. For example, on ImageNet classification, the best learning rate we tried for CLIP was le-4 while the best learning rate for MAE was 1e-3, which similar to the best learning fate for training from scratch. We speculate that this learning rate sensitivity occurs because different pretraining algorithms lead to parameter vectors of very different magnitudes. For image classification, a shorter
"I whtrps//ggiiih.co///mmmeeeeer// MAR
26

---

Table 17: MRR scores for image retrieval experiments.
Backhone
Method
Pretrain
CUB
iNat
Obj
INSTRE
GLM
rOxf
Copy
Supervised
ImageNet-Ik lk
0.63
0.15
0.38
0.88
0.37
0.63
0.97
0.87
ConyNeext-Base
Supervised
ImugeNet-21k-I1
0.9
0.41
0.52
0,94
0.53
0.84
0.99
0.92
Supervised
ImageNet-1K
0.66
0.16
0.38
0.87
0.36
0.68
0.97
0.86
SonvNext-Smmil
Supervised
ImageNet-21k:
0.9
0.49
0.92
Conv/Nexi-Xl.arg
Supervised
ImageNet-222 21k-1k
0.9
0.53
0.93
ResNet-101
CLIP
OpenAl
0.67
0.15
0.31
0.93
0.44
0.65
0.97
0.86
Supervised
ImageNet- 1k
0.66
0.16
0.35
0.86
0.4
0.67
0.97
0.86
CLIP
OpenAI
0.59
0.12
0.24
0.88
0.41
0.67
0.97
0.86
ResNet-50
Supervised
ImageNet-Ik
0.6
0.28
0.84
0.37
0.87
VieReg
ImageNet-1k
0.45
0.11
0.18
0.86
0.42
0.97
0.89
ResNet-50x64
CLIP
OpenAl
0.83
0.27
0.53
0.99
0.60
0.85
0.99
0.95
SwinV2-Base
MiDaS
MiDaS
0.54
0.13
0.17
0.70
0.36
0.72
0.97
0.90
SwinV2-Base-wer
Supervised
ImageNet-2Ik-H1 1k
0.89
0.4
0.51
0.93
0.51
0.87
0.99
0.92
B2s
Supervised
ImageNet-2Ik-11 1k
0.89
0.43
0.51
0.94
0.52
0.87
0.99
0.93
Swin SwinV2-Large-w16
Supervised
ImageNet-2lk-IH
0.88
0.39
0.47
0.94
0.52
0.84
0.98
0.93
Swin L2Lar22-
Supervised
ImageNet-2Ik-1k
0.88
0.42
0.49
0.94
0.52
0.84
0.98
0.94
MiDuS
MiDaS
0,51
0.09
0.16
0.64
0.35
0.58
0.96
0.88
SwinV2-Tiny
Supervised
ImageNet-11 1k
0.68
0.17
0.32
0.9
0.4
0.67
0.99
0.89
CLIP
LAION-2B
0.82
0.23
0.46
0.98
0.55
0.87
0.97
0.93
MAE
ImageNet-IK
0.14
0.01
0.04
0.34
0.07
0.61
MoCoV3
ImageNet- lk
0.57
0.13
0.2
0.84
0.38
0.78
0.95
0.91
ViT-Base
Supervised
ImageNet-1k
0.56
0.12
0.23
0.84
0.3
0.56
0.95
0.86
Supervised
11ageNee-21k-16
0.33
DINO
ImageNet-
0.72
0.21
0.29
0.91
0.43
0.85
0.99
0.91
MAE
ImageNet-Ik 1k
0.24
0.04
0.05
0.56
0.16
0.53
0.92
0.81
ViT-Large
CLIP
LAION-2B
0.85
0.26
0.47
0.98
0.58
0.82
0.97
0.94
Supervised
ImageNet-21k
MoCoV3
ImageNet-1k 1k
0.57
0.12
0.18
0.82
0.36
0.89
Supervised
ImageNet- 1k
0.61
0.13
0.25
0.85
0.32
0.61
0.95
0.87
ViT-Small
Supervised
ImageNet-21ke 1kkk
0.84
0.27
0.3
0.88
0.43
0.74
0.96
0.88
DINO
ImageNet-1k
0.79
0.27
0.42
training period is enough for finetuning, where we only train the model for 100 epochs which is 1/3
as many epochs as we use for training from scratch. Also on smaller datasets, such as Flowers- 102 and aircraft datasets, finetuning obtains much higher accuracy compared to training from scratch. In contrast, finetuning does not save quite as many epochs for detection and segmentation where detection systems contain lots of new parameters that are randomly initialized for downstream training.
27

---

Table 18: Recall@5 scores for image retrieval experiments.
Backbone
Method
Pretrain Data
CUB
INat
Obj
INSTRE
GLM
rOxt
rPar
CopyD
ConyNext-Base
Supervised
ImageNet-Ik
0.045
0.069
0.03
0.097
0.099
0.087
0.026
0.854
Supervised
InageNet-21k-I6 1k
0.085
0.221
0.046
0.111
0.163
0.174
0.027
0.902
Supervised
ImageNet-Ik
0.05
0.069
0.029
0.094
0.099
0.092
0.026
0.831
Con-Next-Small
Supervised
InageNet-21k-IB
0.084
0.042
0.106
0.151
0.027
0.888
Con-Next-XLange
Supervised
ImageNet-2lk-Ik
0.086
0.247
0.048
0.111
0.164
0.172
0.027
0.933
CLIP
OpenAI
0.05
0.06
0.02
0.106
0.117
0.082
0.026
0.848
ResNet-101
Supervised
ImageNet-1k
0.049
0.07
0.026
0.092
0.107
0.085
0.026
0.857
CLIP
OpenAI
0.04
0.046
0.015
0.097
0.107
0.092
0.026
0.852
ResNet-50
Supervised
ImageNet-Il Ik
0.043
0.056
0.018
0,089
0.099
0.084
0.026
0.836
VieReg
ImageNet-1k
0.026
0.042
0.009
0.09
0.116
0.106
0.027
0.868
ResNet-50x64
CLIP
OpenAl
0.073
0.124
0.044
0.122
0.191
0.181
0.027
0.948
SwinV2-Base
0.032
0.008
0.063
0.095
0.027
SwinW2-Ba
Supervised
ImageNet-2Ik-Ik
0.083
0.207
0.043
0.109
0.154
0.178
0.027
0.917
SwinV2-Base-w24
Supervised
ImageNet-2lk-Ik
0.084
0.226
0.045
0,111
0.158
0.177
0.027
0.923
SwinV2-Large-116
Supervised
ImageNNt-2Ik-Ik
0.082
0.204
0.04
0.112
0.16
0.167
0.027
0.93
SwinV2-Lange-w24
Supervised
ImageNet-21k-Ik
0.083
0,223
0.041
0.113
0.161
0.17
0.027
0.931
SwinV2-Tiny
MiDaS
MiDaS
0.031
0.034
0.008
0.055
0.095
0.059
0.026
0.849
Supervised
ImageNet-1B
0.114
0.027
CLIP
LAION-2B
0.071
0.104
0.037
0.12
0.172
0.175
0.027
0.92
MAE
ImageNet- lk
0.005
0.003
0.001
0.02
0,012
0.02
0.015
0.542
VIT-Base
Supervised
McCoV3
ImageNet- Ik
ImageNet-Ik
0.039
0.036
0.049
0.015
0.01
0.088
0.088
0.078
0.1
0.053
0.114
0.025
0.026
0.887
0.82
Supervised
ImageNet-2Ik-Ik
0.077
0.146
0.024
0.094
0.111
0.088
0.026
0.883
DINO
ImageNet-Ik
0.055
0.093
0.018
0.102
0.125
0.158
0.027
0.894
CLIIP
LAION-2B
0.077
0.122
0.038
0.12
0.183
0.127
0.027
0.934
ViT-Large
MAE
ImageNet-Ik
0.01
0.012
0.002
0.042
0,036
0.038
0.023
0.753
Supervised
ImageNet-2144
0.141
McCoV3
ImageNet-1k
0.037
0.05
0.009
0,082
0.097
0.094
0.026
0.877
Supervised
ImageNet: Ik
0.044
0.053
0.016
0.088
0.085
0.062
0.025
0.843
Vit-Small
Supervised
ImageNet-21k-Ik
0.126
0.097
0.118
0.026
DINO
ImageNet- 1k
0.065
0.093
0.016
0.101
0.123
0.16
0.027
0.888
Table 19: Top-1 classification accuracy (%) for ImageNet against adversarial attacks with foc constraint radii 1/255 and 2/255.
Backbone
Method
Pretrain Data
Clean
é 1/255
= 2/255
Z-scores
Supervised
ImageNet- lk
80.38
28.79
13.25
-0.99
VicReg
ImageNet-Il lk
78.77
36.60
22.01
-0.26
ResNet-50
CLIP
LAION-2B
80.55
32.78
18.55
0.58
DINO
ImageNet-Ik
79.08
35.80
20.75
-0.35
ResNet-101
Supervised
ImageNet-
81.93
44.23
32.43
0.53
Supervised
ImageNet-Ik
78.84
21.21
6.42
-1.62
Supervised
ImageNet-21k
81.39
16.50
3.83
-1.44
ViT-Small
MoCoV3
ImageNet-1k
79.65
48.62
32.16
0.71
DINO
ImageNet-
81.33
47.87
30.04
0.57
Supervised
ImageNet-
79.15
27.08
9.54
-1.23
Supervised
ImageNet-21k
84.53
23.04
6.92
-1.52
MoCoV3
ImageNet-lk
82.85
55.44
39.49
1.32
ViT-Base
MAE
ImageNet-Ik
83.41
50.85
31.69
0.77
DINO
ImageNet-Ik
83.40
53.59
36.61
1.11
CLIP
LAION-2B
85.19
47.91
28.35
0.50
Supervised
ImageNet-
81.82
40.91
23.15
-0.03
Swin V2-Tiny
MiDaS
MiDaS
82.55
41.44
25.20
0.08
ConvNeXt- Tiny
Supervised
ImageNet-
82.10
49.74
31.42
0.72
Supervised
ImageNet-Il kk
83.88
55.31
37.19
1.21
Cot-NeXt
Supervised
ImageNet-21k
85.87
53.78
34.05
1.00
28

---

Table 20: Z-scores for best-performing SSL and supervised learning backhones. Mean z-scores for each task averaged across the 3 top performing backbones dividing models into self (weakly- >-supervised learning (SSL. on ImageNet-1k, supervised learning on ImageNet-Kk (Sup-1k), and ImageNet-21K (Sup-2Ik).
Task
SSL
Sup-1k
Sup-21k
Cls
0.573
0.527
0.936
Det
0.298
0.743
1.076
Seg
0.314
0.717
1.071
Ret
0.489
-0.079
0.708
(OOD) Cls
0.419
0.287
1.271
(OOD) Det
0.414
0.923
0.853
29


Reviews:
[{"review_id": "miRoYoLt7Z", "reviewer": "Reviewer_eLJh", "summary": "", "questions": "", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper presents a comprehensive comparative analysis of the most recent and high-performing models in the literature. \nThe comparison presented confirms existing observations that larger models perform better at tasks and Vits are more sensitive to the amount of training data available, and show that performance across tasks is strongly correlated.\nThe ablations presented use (pre-existent) publicly available checkpoints that can accessible to practitioners interested follow up with in the models ablated.\nThese checkpoints cover models pretrained with supervised, vision-language and SSL.  \nIn conclusion, a one-to-one comparison of public checkpoints can be highly beneficial for practitioners who seek off-the-shelf solutions and do not have the resources or interest in developing their own solutions.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "TKuB1K1b7e", "reviewer": "Reviewer_SBvp", "summary": "", "questions": "", "limitations": "- This paper provides good information about comparing backbone models. However, it fails to explain why Sup. ConvNeXt is best in Det, Seg, Ret, and OOD or Sup. Swin V2-B is the best in Cls. Extensive experiments are a good trial, but I think many researchers or practitioners want to know why the backbone models are good for the task. If this is included, it will be a better paper.\n\n- The authors selected several tasks for experiments, such as classification, object detection & segmentation, image retrieval, and OOD tasks. Why did you select only those tasks? There are many other tasks in computer vision, such as human pose estimation, depth estimation, super-resolution, deblurring, and so on. What about these tasks? Would ConvNext and Swin also be the best?\n\n- For the classification task, several datasets are used for experiments Furthermore, medical and satellite datasets are used which implies a large domain gap between pre trained dataset and the target dataset. However, only COCO dataset is used for object detection and segmentation tasks. Similar to the classification experiments, how about adding large domain gap datasets of medical or satellite for object detection and segmentation tasks? (medical: [1], satellite: [2], [3])\n\n[1] https://paperswithcode.com/datasets?mod=medical\n\n[2] Ke Li, Gang Wan, Gong Cheng, Liqiu Meng, and Junwei Han. Object detection in optical remote sensing images: A survey and a new benchmark. ISPRS Journal of Photogrammetry and Remote Sensing, 159:296–307, 2020\n\n[3] Jian Ding, Nan Xue, Gui-Song Xia, Xiang Bai, Wen Yang, Michael Yang, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Object detection in aerial images: A large-scale benchmark and challenges. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- It is easy to follow.\n- It can be useful information when selecting backbone models for some parts of computer vision tasks.\n- Extensive experiments.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "tXz5pYrXr1", "reviewer": "Reviewer_fJFu", "summary": "", "questions": "", "limitations": "The authors partially addressed the comparison of smaller models by providing an analysis for models with <30M parameters. However, there is still a lack of comparisons for even smaller models that are more commonly used in practical applications.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper is well-written with a clear organization, and the authors provide very detailed implementation specifics.\n2. The authors conduct comprehensive experiments to compare the performances of various backbones across different downstream tasks. To ensure a fair comparison, they meticulously adjust hyperparameters in downstream tasks to minimize their influence.\n3. The motivation for this study is compelling, and I am convinced that this benchmark can serve as a valuable empirical reference for the community.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "7kp431PvtZ", "reviewer": "Reviewer_YjWA", "summary": "", "questions": "", "limitations": "Yes, the authors write the limitations and potential negative societal impact such as exploring biases in models.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1) Transfer learning from the pretrained backbones has emerged as an important topic in computer vision and language tasks. The paper handles representative backbone architectures with recently popular pretraining methods. Their pretrained checkpoints are available in public, which provides high accessibility to practitioners.\n\n2) The clear and meaningful message is given: among backbones evaluated in this benchmark study, three models, named Sup. ConvNeXt-B (In-21k), Sup. SwinV2-B (IN-21k,1k), and CLIP ViT-B (LAION-2B), are recommended to be utilized by practitioners and researchers in 2D computer vision tasks, as a backbone.\n\n3) Through the extensive experiments in the four computer vision tasks, the authors provide a unified conclusion in the backbone selection. This may have an impact to practitioners who consider backbones with model parameters of ~150M or less.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "BWyt4OmAaQ", "reviewer": "Reviewer_eVj3", "summary": "", "questions": "", "limitations": "As the authors have pointed out, this paper mainly focuses on the performance. But when choosing the model, fairness is another important aspect to consider. For example, different biases introduced by vision-language pertaining (e.g., CLIP) and vision-only pretraining (e.g., ConvNeXt) are not studied.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- This paper provides an extensive benchmark of different pretrained models for a series of computer vision tasks. It reveals the strengths and weaknesses of various pretraining routines and architectures, providing valuable insights for practitioners and researchers alike.\n- The experiments were conducted with many different datasets for diverse tasks and thus the meta-analysis is more likely to be generalizable.\n- The open-source codebase can be a valuable resource for the community.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper received a majority of acceptance votes from the reviewers, with a consensus praising its comprehensive study and clear, informative takeaways. One reviewer offered a borderline evaluation due to a lack of independent one-to-one training and insufficient exploration of new combinations, alongside missing object detection and segmentation analysis; this AC concurs that the raised point is valid. Although the authors attempted to address this with some out-of-distribution (OOD) object detection experiments, this AC recommends further experiments on challenging object detection benchmarks and offering more analyses, as suggested by reviewer eLJh. Furthermore, expanding comparisons as advised by the reviewer, would strengthen the paper. 

Lastly, this AC advises the authors to revise the paper by addressing the reviewers' outstanding concerns in the final version, encompassing 1) the narrow range of utilized backbones and pretraining methods; 2) the limited scope of tasks and employed datasets, some of which are misaligned due to domain gaps; 3) the absence of a detailed rationale elucidating why a specific backbone demonstrates superior performance. Moreover, please improve the readability of some figures (e.g., specifically address the legends in Figure 1). Overall, this paper merits acceptance.
Decision: Accept (Poster)