$\triangleright$  Transformers excel under end-to-end fine-tuning while convolutional networks excel under linear probing. For "linear probing" experiments, we freeze a pretrained backbone and only learn the head. Note that for detection and segmentation, the head is more than a linear layer. By inspecting the performance difference between the two fine-tuning strategies (Figure 2), we find that ViTs benefit significantly more from end-to-end fine-tuning compared to CNNs, both for supervised and self-supervised pretraining. See Figure 2 for a comparison on dense prediction tasks.
$\triangleright$  CLIP models and the promise of advanced architectures in vision-language modeling. For almost all the tasks (except OOD detection), CLIP pretraining is the best among the vanilla vision transformers, even compared to ImageNet-21k supervised trained backbones. Among all the backbones, CLIP is only worse than ImageNet-21k trained SwinV2 and ConvNeXt, which shows the power of vision-language pretraining and again, suggests that we should consider more backbones other than plain ViTs when conducting self- or weakly-supervised learning.
$\triangleright$  What about generative backbones? In contrast to models trained using supervised or self-supervised approaches with contrastive loss, backbones trained with a generative objective, such as MAE or Stable Diffusion, had comparatively inferior performance. We recommend caution when interpreting this result, as the evaluation of Stable Diffusion is currently limited to select tasks. Nonetheless, Stable Diffusion is a larger backbone than others considered in this benchmark and is trained on a very large dataset, yet it exhibits inferior performance.
$\triangleright$  Battle of the "small" backbones. Keeping limited resources in mind, we also compare the "small" subset of backbones in BoB ( $< 30\mathrm{M}$  parameters) - with ViT-Small, ConvNeXt-Tiny, Swin-Tiny and ResNet-50 architectures. Overall, we find Supervised ConvNeXt-T trained on IN-1k to be the best, followed by Supervised SwinV2-T trained on IN-1k and DINO ViT-S trained on IN-1k. Interestingly, supervised learning again dominates, and backbones pretrained on just IN-1k outperform ones trained on a considerably more diverse and larger dataset (MiDaS).
$\triangleright$  Performance vs. Speed? Our analysis reveals a strong negative correlation  $(\rho = -0.41)$  between throughput (computed on NVIDIA RTX A5000) and average performance z-scores across all tasks when considering each backbone. This finding aligns with our previous observation that larger models tend to exhibit superior performance. Consequently, in order to achieve enhanced performance, one may need to sacrifice speed.
$\triangleright$  Monocular depth-estimation as a general purpose pretraining strategy. In our experiments, MiDaS achieves performance competitive with that of top conventional supervised and SSL backbones at classification, object detection, and segmentation, even outside of the natural image domain, for example on satellite images. This observation suggests that depth-estimation may serve as a powerful and generalizable primary or auxiliary pretraining task for foundation models, supporting findings of Lao et al. [49].
$\triangleright$  Calibration and test likelihood are correlated with accuracy. We measure expected calibration error (ECE) as well as test cross-entropy loss on the ImageNet test set. Whereas test likelihood is