SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 7: Image Retrieval Datasets: Kindly note that the provided numbers are approximate references. For precise details, please consult the original research papers.
Dataset	Description	Size	Classes
CUB-200 [91]	Fine-grained bird images	12k	200
i-Naturalist [88]	Fine-grained species classification	450k	>1k
Object Net [2]	Object recognition dataset with uncommon poses	50k	313
INSTRE [95]	Fine-grained recognition data set of toys and house-hold objects	28k	200
Google Landmarks V2 [98]	Landmarks across the world	>1 mil	>1k
Oxford [69]	Specific buildings from Oxford	6400	12
Paris [70]	Specific landmarks and buildings from Paris	5000	11
Copy Days [19]	Copy detection dataset of landscapes, buildings etc	2,286	n/a
weight decay to be 0.05 and use the same data augmentation as the regular fine-tuning for  $10\%$  but without strong data augmentations, such as mix-up, cut-mix, and random erasing for the  $1\%$  setup.
Object Detection and Segmentation. For the training of Cascade Mask R-CNN, we adopt the standard training settings previously employed in ConvNext [58], Swin [56], and ViTDet [52]. For all experiments, we utilize the AdamW optimizer [60] with weight decay of 0.05, batch size of 16, and we conduct a grid search for the learning rate, considering values of  $\{8e^{-5}, 1e^{-4}, 2e^{-4}, 3e^{-4}\}$ . Additionally, we employ a  $3 \times$  schedule, spanning a total of 36 epochs, and the learning rate decayed by a factor of 10 at epochs 27 and 33. In addition, we apply multi-scale training [7, 83], excluding ViT-based backbones and Stable Diffusion. For ViT-based backbones, inspired by the approach employed in ViTDet along with the fact that ViT backbones perform poorly on detection without specially tailored training strategies, we use large-scale jitter (LSJ) with the image resolution of  $1024 \times 1024$  and scale range of  $[0.1, 2.0]$ . In order to maintain fair comparisons across backbones, we minimize architecture-specific modifications. Thus, for our ViT backbones, we avoid implementing some ViTDet modifications such as "simple feature pyramid" instead of FPN. Instead, we employ an FPN that utilizes the final feature map, without employing stage division. It is worth noting, as highlighted in the ViTDet paper, the performance of "FPN, last-map" is inferior to the "simple feature pyramid". Additionally, we use the supervised training results from DeiT [85] for supervised ViT-S and ViT-B pretrained backbones. For Stable Diffusion, we use a single resolution of  $512 \times 512$  (resizing the image such that the longer side is 512) to overcome the significantly larger computational cost of this backbone compared to its competitors in our benchmark.
A note on ViTDet and scale: architectural modifications and very long training schedules can benefit ViTs in particular, as was found in Li et al. [52]. Similarly, Chen et al. [14] point out that ViTDet achieves stronger performance than their own work due to long and expensive training routines, behavior which stems from ViTs weak vision inductive bias. Additionally, in our analysis, we found that ViTs benefit more from scale, so ViTs might overtake other models at larger scales. Including large models in our benchmark, which includes many tasks and pretraining methods, would be prohibitively expensive, yet practitioners with sufficient computational resources for larger models, longer training routines, and architectural modifications may consider ViT backbones.
Syn-to-Real Generalization. For VisDA [66]  $\mathrm{Syn} \rightarrow \mathrm{Real}$ , we report accuracy on the target domain (real) using models trained on the source domain (synthetic) for 12-way classification. For training, we use a learning rate of  $1\mathrm{e}^{-3}$  on 4 A40 GPUs with a batch size of 64 and report accuracy after 10 epochs. For object detection, we train the backbones outlined in Appendix B.1 with the Cascade-RCNN architecture. For training and fine-tuning backbones on Sim10k [38], we use a learning rate of  $1\mathrm{e}^{-4}$  on 4 A40 or RTX 6000 GPUs. To enable consistent evaluation across all backbones (CNNs and ViTs), we downsample Cityscapes [15] images to  $1024 \times 512$  during evaluation. We train all models on the entirety of Sim10k and evaluate on the validation split of Cityscapes.
Image Retrieval. We have only evaluated pretrained models for this task. Dataset and metrics are discussed in the main body. Refer to table Table 7 for a brief summary of all the retrieval datasets.
D Results
Image Classification. We present the ImageNet Top-1 and Top-5 classification accuracy for backbones pretrained with different methods and on various datasets in Table 8. We adopt the ImageNet results for supervised learning with random initialization from the timm library [100]. We omit
ImageNet results for ImageNet-pretrained backbones since those coincide with the randomly initialized backbones on ImageNet. We also present top-1 classification accuracy for finetuning on various datasets in Table 9, and we include ImageNet calibration and uncertainty estimation results in Table 10.
Table 8: Classification accuracy (%) for ImageNet-related tasks. "lp" denotes linear probing, "1%" and "10%" denote the percentage of labeled training images used during the fine-tuning.
Backbone	Method	Pretrain Data	ImageNet (lp)	ImageNet	ImageNet (10%)	ImageNet (1%)
Top-1	Top-5	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
ResNet-50	Supervised	ImageNet-1k	-	-	80.38	94.60	-	-	-	-
VicReg	ImageNet-1k	72.15	90.22	78.77	94.29	69.54	89.07	55.04	79.34
CLIP	LAION-2B	65.98	87.84	80.55	95.26	69.26	89.91	43.78	70.67
DINO	ImageNet-1k	74.17	91.56	79.08	94.60	68.18	89.35	51.38	77.82
ViT-Small	Supervised	ImageNet-1k	-	-	78.84	94.29	-	-	-	-
MoCoV3	ImageNet-1k	73.11	90.94	79.65	94.96	70.27	90.11	54.14	79.15
DINO	ImageNet-1k	76.08	92.63	81.33	95.71	73.83	91.89	58.15	80.07
ViT-Base	Supervised	ImageNet-1k	-	-	79.15	94.09	-	-	-	-
MoCoV3	ImageNet-1k	75.96	92.69	82.85	96.31	74.80	92.54	62.88	85.31
MAE	ImageNet-1k	67.67	87.49	83.41	96.50	72.87	91.54	56.02	81.07
DINO	ImageNet-1k	77.31	93.43	83.40	96.42	75.92	93.30	63.92	84.52
CLIP	LAION-2B	79.74	95.53	85.19	97.46	78.67	95.00	66.44	89.11
SwinV2-Tiny	Supervised	ImageNet-1k	-	-	81.82	95.99	-	-	-	-
MiDaS	MiDaS	76.44	92.66	82.55	95.92	79.92	94.57	75.44	90.83
SwinV2-Base	Supervised	ImageNet-21k-1k	-	-	87.10	98.23	-	-	-	-
MiDaS	MiDaS	81.09	95.47	86.48	98.00	84.14	96.91	79.26	93.66
Stable Diffusion	Stable Diffusion	LAION-2B	-	-	79.90	95.10	71.50	89.07	38.02	65.78
Table 9: Top-1 classification accuracy (%) for fine-tuning pretrained (and randomly initialized) backbones with different methods on various datasets. We omit the ImageNet performance for the backbones trained on ImageNet in a supervised fashion that setup is the same as backbones trained from random initialization.
Backbone	Method	Pretrain Data	ImageNet	EuroSAT	Flower	CIFAR-100	Chexpert	Aircraft
ResNet-50	Rand Init	-	80.38	89.61	41.96	72.33	86.67	20.73
Supervised	ImageNet-1k	-	98.26	86.52	84.71	86.82	55.31
VicReg	ImageNet-1k	78.77	95.11	92.68	87.56	86.55	67.51
CLIP	LAION-2B	80.55	98.72	90.62	84.35	88.92	73.50
DINO	ImageNet-1k	79.08	98.74	94.04	86.49	87.49	74.13
ResNet-101	Rand Init	-	81.93	62.07	44.51	67.94	87.41	13.08
Supervised	ImageNet-1k	-	97.19	83.59	83.07	86.51	40.47
ViT-Small	Rand Init	-	78.84	42.61	38.73	56.08	77.36	5.52
Supervised	ImageNet-1k	-	95.54	96.17	89.48	87.60	61.69
Supervised	ImageNet-21k	81.39	95.30	98.73	92.51	87.39	59.89
MoCoV3	ImageNet-1k	79.65	96.02	83.59	89.38	88.10	54.68
DINO	ImageNet-1k	81.33	89.09	73.73	90.00	87.75	48.20
ViT-Base	Rand Init	-	79.15	48.33	40.20	54.30	79.61	6.99
Supervised	ImageNet-1k	-	96.22	94.04	90.62	87.27	62.02
Supervised	ImageNet-21k	84.53	94.93	99.41	93.89	87.72	70.2
MoCoV3	ImageNet-1k	82.85	96.74	94.53	90.89	86.82	71.55
MAE	ImageNet-1k	83.41	95.54	94.63	89.96	88.01	72.27
DINO	ImageNet-1k	83.40	95.59	97.17	91.22	87.05	71.82
CLIP	LAION-2B	85.19	94.37	96.78	91.29	87.74	76.38
SwinV2-Tiny	Rand Init	-	81.82	89.33	29.90	66.49	87.76	5.31
Supervised	ImageNet-1k	-	98.91	96.58	89.50	88.39	68.71
MiDaS	MiDaS	82.55	96.33	96.48	90.53	87.69	69.54
SwinV2-Base	Supervised	ImageNet-21k-1k	87.10	95.94	99.61	93.09	87.73	79.46
MiDaS	MiDaS	86.48	96.53	99.32	93.08	88.08	79.62
ConvNeXt-Tiny	Rand Init	-	82.10	73.80	18.24	75.31	83.25	4.35
Supervised	ImageNet-1k	-	95.70	96.00	89.89	87.56	69.99
ConvNeXt-Base	Rand Init	-	83.88	30.39	19.41	73.04	85.26	4.02
Supervised	ImageNet-1k	-	93.06	94.92	88.98	88.98	64.72
Supervised	ImageNet-21k	85.87	96.19	99.61	92.84	87.80	69.39
Stable Diffusion	Stable Diffusion	LAION-2B	79.90	92.22	91.89	90.50	87.49	72.45
Object Detection and Segmentation. In our experiment, we utilize the train2017 and val2017 splits of the COCO dataset for training and evaluation, respectively. We report results on bounding box object detection  $(\mathrm{AP}^{box})$  and instance segmentation  $(\mathrm{AP}^{mask})$ . In Table 11, Table 12 and Table 13, we present the comprehensive results of our experiment. Table 11 reflects the results obtained with various backbone architectures when the detector is fine-tuned while the backbone remains frozen. Table 12 contains both training from scratch with randomly initialized backbones as well as end-to-end fine-tuning using pretrained backbones. In Table 13, we additionally present results on the harder
Table 10: Calibration and uncertainty estimation on ImageNet. We measure Top-1 accuracy (\%), cross-entropy test loss (CE), and expected calibration error (ECE).
Backbone	Method	Pretrain Data	Accuracy	CE	ECE
ResNet-50	Supervised	ImageNet-1k	80.38	0.94	0.09
VicReg	ImageNet-1k	78.77	1.11	0.21
CLIP	LAION-2B	80.55	1.02	0.19
DINO	ImageNet-1k	79.08	1.11	0.22
ResNet-101	Supervised	ImageNet-1k	81.93	0.92	0.16
ViT-Small	Supervised	ImageNet-1k	78.84	0.84	0.03
Supervised	ImageNet-21k	81.39	0.68	0.01
MoCoV3	ImageNet-1k	79.65	0.90	0.11
DINO	ImageNet-1k	81.33	0.83	0.10
ViT-Base	Supervised	ImageNet-1k	79.15	0.86	0.05
Supervised	ImageNet-21k	84.53	0.56	0.01
MoCoV3	ImageNet-1k	82.85	0.77	0.08
MAE	ImageNet-1k	83.41	0.75	0.09
DINO	ImageNet-1k	83.40	0.76	0.07
CLIP	LAION-2B	85.19	0.66	0.08
SwinV2-Tiny	Supervised	ImageNet-1k	81.82	0.83	0.09
MiDaS	MiDaS	82.55	0.83	0.07
ConvNeXt-Tiny	Supervised	ImageNet-1k	82.10	0.79	0.06
ConvNeXt-Base	Supervised	ImageNet-1k	83.88	0.69	0.04
Supervised	ImageNet-21k	85.87	0.56	0.03
LVIS dataset [27] using the best transformer-based (SwinV2) and convolutional-based (ConvNeXt) architectures from our experiments at several sizes. We again see a benefit of scale here as well as the slightly superior performance of modern convolutional architectures. These tables provide a comprehensive overview of the performance achieved across various backbones and scenarios, enabling a thorough analysis and comparison of the different backbones utilized in our study.
Table 11: Object detection and instance segmentation using Cascade Mask-RCNN on COCO with frozen backbones.
Backbone	Method	Pretrain Data	Params	Input Size	\( \mathbf{A}\mathbf{P}^{box} \)	\( \mathbf{A}\mathbf{P}^{box} \) 50	\( \mathbf{A}\mathbf{P}^{box} \) 75	\( \mathbf{A}\mathbf{P}^{mask} \)	\( \mathbf{A}\mathbf{P}^{mask} \) 50	\( \mathbf{A}\mathbf{P}^{mask} \) 75
ResNet-50	Supervised	ImageNet-1k	82M	1333 × 800	42.5	61.0	46.2	37.1	58.1	39.5
VicReg	ImageNet-1k	82M	1333 × 800	44.1	62.3	48.1	38.8	59.7	42.0
DINO	ImageNet-1k	82M	1333 × 800	44.6	62.9	48.8	39.0	60.3	42.0
ResNet-101	Supervised	ImageNet-1k	101M	1333 × 800	43.4	62.1	47.1	38.0	59.3	40.7
ViT-Small	Supervised	ImageNet-1k	84M	1024 × 1024	27.3	44.3	29.1	23.6	40.8	23.9
MoCoV3	ImageNet-1k	84M	1024 × 1024	27.8	44.6	29.7	24.2	41.5	24.9
DINO	ImageNet-1k	84M	1024 × 1024	33.6	52.5	35.9	29.1	49.0	30.0
ViT-Base	Supervised	ImageNet-1k	155M	1024 × 1024	34.1	54.0	36.1	29.6	50.6	30.0
MoCoV3	ImageNet-1k	155M	1024 × 1024	32.1	50.4	34.6	28.2	47.1	29.3
MAE	ImageNet-1k	155M	1024 × 1024	35.6	54.0	38.7	31.8	51.1	33.7
DINO	ImageNet-1k	155M	1024 × 1024	36.2	55.6	39.1	31.7	52.2	32.9
CLIP	LAION-2B	155M	1024 × 1024	21.9	36.7	22.5	18.8	33.6	18.8
SwinV2-Tiny	Supervised	ImageNet-1k	86M	1333 × 800	36.9	55.4	39.9	32.5	52.6	34.4
MiDaS	MiDaS	86M	1333 × 800	34.3	51.5	37.2	30.2	48.9	32.0
SwinV2-Base-w8	Supervised	ImageNet-1k	145M	1333 × 800	38.6	57.4	41.7	33.5	54.4	35.3
SwinV2-Base-w24	Supervised	ImageNet-21k-1k	145M	1333 × 800	44.6	64.7	48.6	38.8	61.6	41.5
MiDaS	MiDaS	145M	1333 × 800	42.2	61.5	45.9	37.1	58.7	39.3
ConvNeXt-Tiny	Supervised	ImageNet-1k	86M	1333 × 800	44.4	63.1	48.7	38.7	60.7	41.6
ConvNeXt-Base	Supervised	ImageNet-1k	146M	1333 × 800	44.8	64.1	48.6	39.2	61.4	42.0
Supervised	ImageNet-21k	146M	1333 × 800	46.2	66.0	50.2	40.1	62.9	43.0
Stable Diffusion	Stable Diffusion	LAION-2B	442M	1333 × 800	38.2	55.4	41.6	34.0	52.7	36.4
Table 12: Object detection and instance segmentation results using Cascade Mask-RCNN on COCO.
Backbone	Method	Pretrain Data	Params	Input Size	\( \mathbf{A}\mathbf{P}^{box} \)	\( \mathbf{A}\mathbf{P}^{box} \)50	\( \mathbf{A}\mathbf{P}^{box} \)75	\( \mathbf{A}\mathbf{P}^{mask} \)	\( \mathbf{A}\mathbf{P}^{mask} \)50	\( \mathbf{A}\mathbf{P}^{mask} \)75
ResNet-50	Random Init	-	82M	1333 × 800	41.3	57.4	45.1	36.0	55.1	38.9
Supervised	ImageNet-1k	82M	1333 × 800	46.6	64.6	50.6	40.2	61.9	43.5
VicReg	ImageNet-1k	82M	1333 × 800	38.2	55.2	41.5	33.5	52.9	35.9
DINO	ImageNet-1k	82M	1333 × 800	45.4	63.5	49.2	39.4	61.1	42.4
ResNet-101	Random Init	-	101M	1333 × 800	45.7	63.0	50.0	39.5	60.5	43.1
Supervised	ImageNet-1k	101M	1333 × 800	47.7	65.6	52.0	41.3	63.2	44.6
ViT-Small	Random Init	-	84M	1024 × 1024	43.2	62.2	47.2	38.0	59.3	40.7
Supervised	ImageNet-1k	84M	1024 × 1024	48.2	68.1	51.8	41.7	65.0	44.7
MoCoV3	ImageNet-1k	84M	1024 × 1024	47.6	67.5	51.7	41.6	64.3	44.4
DINO	ImageNet-1k	84M	1024 × 1024	48.2	67.6	52.4	42.3	64.9	44.9
ViT-Base	Random Init	-	155M	1024 × 1024	46.0	65.0	50.7	40.1	62.3	43.1
Supervised	ImageNet-1k	155M	1024 × 1024	49.4	69.3	53.5	42.9	66.1	46.2
MoCoV3	ImageNet-1k	155M	1024 × 1024	49.7	69.4	53.9	43.2	66.6	46.5
MAE	ImageNet-1k	155M	1024 × 1024	51.3	70.3	55.9	44.5	67.7	48.1
DINO	ImageNet-1k	155M	1024 × 1024	49.5	69.0	53.7	42.8	66.1	46.0
CLIP	LAION-2B	155M	1024 × 1024	50.0	69.3	54.4	43.3	66.3	46.6
SwinV2-Tiny	Random Init	-	86M	1333 × 800	47.0	65.3	51.2	40.8	62.7	44.1
Supervised	ImageNet-1k	86M	1333 × 800	50.2	69.1	54.6	43.4	66.3	46.9
MiDaS	MiDaS	86M	1333 × 800	50.2	69.3	54.5	43.5	66.4	47.0
SwinV2-Base-w8	Random Init	-	145M	1333 × 800	48.1	66.6	52.3	41.5	64.1	44.7
Supervised	ImageNet-1k	145M	1333 × 800	52.4	71.0	57.1	45.2	68.6	49.1
SwinV2-Base-w24	Supervised	ImageNet-21k-1k	145M	1333 × 800	52.9	71.4	57.5	45.7	69.0	49.6
MiDaS	MiDaS	145M	1333 × 800	52.7	71.4	57.2	45.7	69.0	49.7
ConvNeXt-Tiny	Random Init	-	86M	1333 × 800	47.5	65.5	51.7	41.2	63.0	44.3
Supervised	ImageNet-1k	86M	1333 × 800	49.9	68.4	54.3	43.2	66.0	46.8
ConvNeXt-Base	Random Init	-	146M	1333 × 800	48.3	66.4	52.7	41.9	63.8	45.3
Supervised	ImageNet-1k	146M	1333 × 800	51.7	70.2	56.0	44.6	67.7	48.3
Supervised	ImageNet-21k	146M	1333 × 800	52.9	71.7	57.3	45.8	69.2	49.9
Stable Diffusion	Random Init	-	442M	1333 × 800	37.1	51.4	40.1	31.9	43.7	31.2
Stable Diffusion	LAION-2B	442M	1333 × 800	43.4	59.1	46.3	38.1	56.9	40.2
Table 13: Object detection and instance segmentation using Cascade Mask-RCNN on LVIS v1.
Backbone	Method	Pretrain Data	Params	Input Size	\( \mathbf{A P^{box}} \)	\( \mathbf{A P^{box}}_{50} \)	\( \mathbf{A P^{box}}_{75} \)	\( \mathbf{A P^{mask}} \)	\( \mathbf{A P^{mask}}_{50} \)	\( \mathbf{A P^{mask}}_{75} \)
SwinV2-Tiny	Supervised	ImageNet-1k	86M	\( 1333 \times 800 \)	33.0	46.3	35.3	29.9	44.5	31.9
MiDaS	MiDaS	86M	\( 1333 \times 800 \)	32.6	45.7	34.9	29.6	43.9	32.0
SwinV2-Base-w8	Supervised	ImageNet-1k	145M	\( 1333 \times 800 \)	35.7	48.7	38.0	32.0	47.0	34.4
ConvNeXt-Tiny	Supervised	ImageNet-1k	86M	\( 1333 \times 800 \)	33.2	46.1	35.4	29.9	44.3	32.2
ConvNeXt-Base	Supervised	ImageNet-1k	146M	\( 1333 \times 800 \)	35.8	48.8	38.0	32.0	46.9	34.5
OOD Generalization. We include results for OOD generalization for image classification in Table 14 and for object detection in Table 15.
Table 14: Top-1 OOD classification accuracy  $(\%)$
Backbone	Method	Pretrain Data	Test Dataset	Train Dataset	Test Dataset
ImageNet-A	ImageNet-S	ImageNet-R	ImageNet-V2	VisDA (real)
ResNet-50	Rand Init	-	-	-	-	-	VisDA (syn)	22.07
Supervised	ImageNet-1k	10.03	29.63	40.25	68.75	VisDA (syn)	47.77
VicReg	ImageNet-1k	10.29	28.63	39.98	67.56	VisDA (syn)	44.19
CLIP	LAION-2B	15.53	30.10	42.44	69	VisDA (syn)	21.98
DINO	ImageNet-1k	10.13	28.38	39.74	67.55	VisDA (syn)	-
ResNet-101	Rand Init	-	-	-	-	-	VisDA (syn)	21.81
Supervised	ImageNet-1k	19.04	34.73	45.10	70.88	VisDA (syn)	56.56
ViT-Base	Rand Init	-	-	-	-	-	VisDA (syn)	22.16
Supervised	ImageNet-1k	14.85	27.99	38.02	66.45	VisDA (syn)	63.57
Supervised	ImageNet-21k-1k	43.24	43.21	56.79	74.01	VisDA (syn)	65.62
MoCoV3	ImageNet-1k	32.57	37.33	34.85	72.71	VisDA (syn)	57.21
MAE	ImageNet-1k	36.2	34.75	48.77	73.47	VisDA (syn)	48.90
DINO	ImageNet-1k	35.32	36.52	49.15	72.71	VisDA (syn)	59.96
CLIP	LAION-2B	46.59	63.33	50.17	75.39	VisDA (syn)	46.05
SwinV2-Tiny	MiDaS	MiDaS	30.87	29.41	41.26	71.27	VisDA (syn)	61.62
SwinV2-Base	Supervised	ImageNet-21k-1k	45.97	37.00	49.69	74.42	VisDA (syn)	68.84
MiDaS	MiDaS	63.05	48.77	63.12	77.06	VisDA (syn)	68.66
ConvNeXt-Base	Supervised	ImageNet-1k	41.92	37.29	51.07	74.09	VisDA (syn)	71.12
Supervised	ImageNet-21k	60.8	48.92	62.53	76.96	VisDA (syn)	74.96
Table 15: OOD object detection on Sim10k→Cityscapes. Out-of-distribution generalization across backbones for object detection (Cascade-RCNN) models trained on Sim10k and evaluated on Cityscapes to detect instances of "car". (Frozen) column corresponds to settings where the backbone has been frozen.
Backbone	Method	Pretrain Data	mAP@50	mAP@50 (Frozen)
ResNet-50	Random Init	-	30.6	-
Supervised	ImageNet-1k	46.5	52.4
VicReg	ImageNet-1k	44.4	49.8
DINO	ImageNet-1k	50.0	52.2
ResNet-101	Random Init	-	25.2	-
Supervised	ImageNet-1k	46.9	52.6
ViT-Small	Random Init	-	8.7	-
Supervised	ImageNet-1k	33.7	15.9
MoCoV3	ImageNet-1k	32.8	19.4
DINO	ImageNet-1k	43.0	31.3
ViT-Base	Random Init	-	10.3	-
Supervised	ImageNet-1k	38.2	24.6
MoCoV3	ImageNet-1k	36.7	27.9
MAE	ImageNet-1k	44.2	32.3
DINO	ImageNet-1k	41.7	35.1
CLIP	LAION-2B	38.1	9.7
SwinV2-Tiny	Random Init	-	40.9	-
Supervised	ImageNet-1k	48.0	48.2
MiDaS	MiDaS	49.5	45.5
SwinV2-Base-w8	Random Init	-	38.0	-
Supervised	ImageNet-1k	51.1	49.3
SwinV2-Base-w24	Supervised	ImageNet-21k-1k	51.0	50.2
MiDaS	MiDas	52.5	50.3
ConvNeXt-Tiny	Random Init	-	36.2	-
Supervised	ImageNet-1k	54.5	50.3
ConvNeXt-Base	Random Init	-	25.5	-
Supervised	ImageNet-1k	55.5	52.1
Supervised	ImageNet-21k	52.4	53.9
Retrieval. We present the mAP, MRR and Recall@5 values for various backbones across different datasets in Table 16, Table 17, and Table 18, respectively.
Analysis. In Table 20, we compare the performance of backbones pretrained with SSL (including CLIP) and supervised learning on ImageNet-1k and ImageNet-21k. We pick the top-3 backbones in each category and calculate the mean z-scores for all the tasks.
Adversarial robustness. In Table 19, we show adversarial robustness on the ImageNet test set against an untargeted PGD adversarial attack  $\epsilon$  with  $\ell_{\infty}$  constraints of  $\frac{1}{255}$  and  $\frac{2}{255}$ . For attack hyperparameters, we use 20 steps and step size  $\frac{\epsilon}{5}$ .
E Assets and Licenses
The assets used in this work can be categorized as - Code Repositories, Backbones and Dependencies (licenses for datasets are included in Appendix B.3).
Table 16: MAP scores for image retrieval experiments.
Backbone	Method	Pretrain Data	CUB	iNat	Obj	INSTRE	GLM	rOxf	rPar	CopyD
ConvNext-Base	Supervised	ImageNet-1k	20.55	5.53	11.42	52.13	12.71	26.73	55.84	79.32
Supervised	ImageNet-21k-1k	62.51	19.43	19.57	69.4	21.34	42.42	73.33	86.18
ConvNext-Small	Supervised	ImageNet-1k	23.96	5.66	10.35	49.33	12.6	28.1	56.03	77.86
Supervised	ImageNet-21k-1k	61.09	17.06	17.12	62.79	19.53	42.19	71.03	83.56
ConvNext-XLarge	Supervised	ImageNet-21k-1k	65.31	21.98	21.37	69.28	21.67	45.33	74.76	86.81
ResNet-101	CLIP	OpenAI	21.75	4.50	5.4	56.71	15.35	23.93	52.1	79.07
Supervised	ImageNet-1k	22.86	5.52	8.9	47.37	13.83	31.27	58.51	78.52
ResNet-50	CLIP	OpenAI	15.37	3.45	3.81	48.10	14.20	24.09	52.14	79.61
Supervised	ImageNet-1k	18.75	4.28	5.31	41.59	11.98	25.59	52.37	79.78
ResNet-50	VicReg	ImageNet-1k	8.84	3.15	1.82	42.45	13.76	31.71	58.24	82.48
ResNet-50x64	CLIP	OpenAI	42.30	10.02	15.11	85.38	26.91	41.93	70.27	91.36
SwinV2-Base	MiDaS	MiDaS	10.44	3.84	1.93	24.91	11.52	29.76	56.34	84.03
Supervised	ImageNet-21k-1k	57.57	17.95	17.81	66.87	20.33	44.35	71.28	87.57
SwinV2-Base-w16	Supervised	ImageNet-21k-1k	58.77	19.93	18.6	69.48	20.97	44.91	71.7	88.2
SwinV2-Large-w16	Supervised	ImageNet-21k-1k	57.55	17.74	16.36	71	21.64	44.26	75.58	88.59
SwinV2-Tiny	MiDaS	MiDaS	10.44	3.84	1.93	24.91	11.52	29.76	56.34	84.03
SwinV2-Tiny	Supervised	ImageNet-1k	22.91	5.71	6.94	54.31	13.76	27.35	56.29	82.93
Supervised	ImageNet-1k	22.91	5.71	6.94	54.31	13.76	27.35	56.29	82.93
ViT-Base	CLIP	LAION-2B	40.27	8.17	12.98	81.13	23.83	44.84	73.62	88.79
MAE	ImageNet-1k	1.26	0.22	0.43	4.86	1.43	5.19	11.12	48.12
ViT-Base	MoCoV3	ImageNet-1k	12.97	4.16	2.29	40.53	12.11	30.25	51.6	84.75
Supervised	ImageNet-1k	17.09	3.81	4.59	39.75	9.64	21.97	50.03	77.7
ViT-Base	Supervised	ImageNet-21k-1k	52.18	11.99	7.56	46.58	14.3	30.96	59.82	83.65
DINO	ImageNet-1k	22.21	7.43	4.57	53.01	15.35	37.03	62.22	86.18
ViT-Large	CLIP	LAION-2B	47.77	9.62	12.87	80.29	25.45	39.19	70	90.32
MAE	ImageNet-1k	2.53	0.84	0.53	12.24	3.78	13.26	24.92	70.34
ViT-Large	Supervised	ImageNet-21k-1k	62.44	18.19	16	55.18	18.49	37.68	67.07	87.04
ViT-Small	MoCoV3	ImageNet-1k	12.99	3.7	1.86	35.28	11.39	24.83	50.24	82.67
Supervised	ImageNet-1k	19.44	4.05	4.61	40.45	10.51	21.69	49.08	79.54
ViT-Small	Supervised	ImageNet-21k-1k	49.6	10.22	6.34	48.17	14.37	31.24	61.5	81.92
DINO	ImageNet-1k	31.38	7.35	3.99	52.64	14.79	37.98	61.01	85.3
Code Repositories. We provide supporting code for all our experiments here. For image classification experiments, we build on top of the timm library  $[100]^8$ , the original MAE repo and the medical dataset pretrain repo  $[103]^{10}$ . timm is distributed under Apache 2.0 License and MAE under the Attribution-NonCommercial 4.0 International License. For object detection, instance segmentation, and OOD detection experiments, we build on top of the MMDetection framework  $[9]^{11}$ . MMDetection is distributed under Apache License 2.0.
Backbones. We use publicly available pretrained backbones. The full list is provided in Appendix B.
Dependencies. Key dependencies for all our experiments include pytorch, timm, HuggingFace utilities and MMCV. Please see our repo README for a comprehensive list of all dependencies to reproduce the experiments.
F Observations about Hyperparameters
For hyperparameter tuning, we find that the learning rate strategy is highly method- and dataset-dependent. For example, on ImageNet classification, the best learning rate we tried for CLIP was 1e-4 while the best learning rate for MAE was 1e-3, which is similar to the best learning fate for training from scratch. We speculate that this learning rate sensitivity occurs because different pretraining algorithms lead to parameter vectors of very different magnitudes. For image classification, a shorter
$^{8}$ https://github.com/huggingface/pytorch-image-models
<sup>9</sup>https://github.com/facebookresearch/mae
10 https://github.com/lambert-x/Medical_MAe
11 https://github.com/open-mmlab/mmdetection