$\triangleright$  A performance comparison of ViTs and CNNs. Modern architectures strongly outperform vanilla ViTs. We see in Table 2 that the best performing backbone (ConvNeXt-Base) is convolutional, with a hierarchical transformer (SwinV2-Base) being a close second. The latter transformer architecture incorporates a strong spatial inductive bias. These findings suggest that the community should move past vanilla ViTs which are still used frequently. As a caveat, we do not evaluate very large models, and it is possible that ViTs might outperform their more advanced variants or convolutional networks at larger scales.
$\triangleright$  ViTs benefit more from scale than CNNs. For the suite of backbones considered in BoB, we find that relative performance (z-scores) for both CNNs and ViTs correlates positively with parameter count but more so for ViTs (spearman  $\rho = 0.58$ ) than for CNNs (spearman  $\rho = 0.35$ ). Similarly, while overall relative performance correlates with the size of pretraining data, the correlation is again significantly higher for ViTs ( $\rho = 0.72$ ) than for CNNs ( $\rho = 0.33$ ). This observation indicates that benchmarking much larger backbones might yield different winners, possibly ones with transformer-based architectures.
Supervised or not? Supervised learning backbones dominate, but primarily because they are available pretrained on larger datasets. SSL backbones can outperform supervised pretraining with similar sized pre-training datasets. We obtain the average score of the top 3 backbones within different pretraining styles, namely self-supervised, supervised with ImageNet-1K, and supervised with ImageNet-21K, for each task (see Appendix D). ConvNeXt and SwinV2 pretrained with supervision on ImageNet-21K outperform the SSL backbones on all tasks. The results suggest that we should try using advanced architectures, either convolutional or transformers, when applying SSL methods, and we should train on large datasets to compete with supervised learning. In these experiments, supervised pretraining checkpoints are often available trained on much larger datasets (ImageNet-21k). When comparing models pretrained on similarly sized datasets, SSL or vision-language pretraining methods achieve better performance on classification (both in- and out-of-distribution) and retrieval tasks, which heavily rely on the learned representations. However, supervised learning backbones maintain a decisive edge for detection and segmentation. We can also compare backbones which use the same ViT-Base architecture and find that SSL methods do outperform ImageNet-1k supervised backbones but are worse than ImageNet-21k trained backbones.
$\triangleright$  Performance across tasks is highly correlated. Across tasks examined, we find a strong positive Spearman correlation between performance on task pairs (typically  $\rho > 0.8$ ). This finding supports the current trend of general purpose foundation models for computer vision. Moreover, this finding also supports recent work which argues that a single inductive bias can solve a wide range of seemingly different problems [24]. However, it is noteworthy that the retrieval task exhibited a comparatively lower but still statistically significant correlation ( $\rho = 0.49$ ) with respect to classification and retrieval ranking. This lower correlation can be attributed to the performance limitations of the MiDaS and MAE pretrained models in the context of retrieval. Upon removing these two backbones, the correlation coefficient  $\rho$  increased to 0.8, reinforcing the influence of the aforementioned models on the observed results.