SlowGuess/NeurIPS_2023_OCR huggingface dataset:
swin_smallPatch4_window7_224, swin_tinyPatch4_window7_224, swsl_resnet18, swsl_resnet50, swsl_resnext101_32x16d, swsl_resnext101_32x4d, swsl_resnext101_32x8d, swsl_resnext50_32x4d, tf_efficientnet_b0, tf_efficientnet_b0_ap, tf_efficientnet_b0_ns, tf_efficientnet_b1, tf_efficientnet_b1_ap, tf_efficientnet_b1_ns, tf_efficientnet_b2, tf_efficientnet_b2_ap, tf_efficientnet_b2_ns, tf_efficientnet_b3, tf_efficientnet_b3_ap, tf_efficientnet_b3_ns, tf_efficientnet_b5, tf_efficientnet_b5_ap, tf_efficientnet_b5_ns, tf_efficientnet_b6, tf_efficientnet_b6_ap, tf_efficientnet_b6_ns, tf_efficientnet_b7, tf_efficientnet_b7_ap, tf_efficientnet_b7_ns, tf_efficientnetCc_b0_4e, tf_efficientnetCc_b0_8e, tf_efficientnetCc_b1_8e, tf_efficientnet_el, tf_efficientnet_em, tf_efficientnet es, tf_efficientnet lite0, tf_efficientnet lite1, tf_efficientnet lite2, tf_efficientnet lite3, tf_efficientnet lite4, tf_efficientnetv2_b0, tf_efficientnetv2_b1, tf_efficientnetv2_b2, tf_efficientnetv2_b3, tf_efficientnetv2_1, tf_efficientnetv2_1_in21ft1k, tf_efficientnetv2_m, tf_efficientnetv2_m_in21ft1k, tf_efficientnetv2_s_in21ft1k, tf_inception_v3, tf_mixnet_l, tf_mixnet_m, tf_mixnet_s, tf_mobilenetv3_large_075, tf_mobilenetv3_large_100, tf_mobilenetv3_large_minimal_100, tf_mobilenetv3_small_075, tf_mobilenetv3_small_100, tf_mobilenetv3_small_minimal_100, tinynet_a, tinynet_b, tinynet_c, tinynet_d, tinynet_e, tint_spatch16_224, tv_densenet121, tv_resnet101, tv_resnet152, tv_resnet34, tv_resnet50, tv_resnext50_32x4d, twins_pcpvt_base, twins_pcpvt_large, twins_pcpvt_small, twins_svt_base, twins_svt_large, twins_svt_small, vgg11, vgg11_dn, vgg13, vgg13_dn, vgg16, vgg16_dn, vgg19, vgg19_dn, visformer_small, vit_basepatch16_224, vit_basepatch16_224_miil, vit_basepatch16_384, vit_basepatch32_224, vit_basepatch32_384, vit_basepatch8_224, vit_base_r50_s16_384, vit_smallPatch16_224, vit_smallPatch16_384, vit_smallPatch32_224, vit_smallPatch32_384, vit_small_r26_s32_224, vit_small_r26_s32_384, vit TinyPatch16_224, vit TinyPatch16_384, vit Tiny_r_s16_p8_224, vit Tiny_r_s16_p8_384, wide_resnet101_2, wide_resnet50_2,xception, xception41,xception65,xception71,xcit_large_24_p16_224,xcit_large_24_p16_224_dist,xcit_large_24_p16_384_dist,xcit_large_24_p8_224,xcit_large_24_p8_224_dist,xcit_large_24_p8_384_dist,xcit_large_24_p8_384_dist,xcit_large_24_p8_384_dist,xcit_large_24_p8_384_dist,xcit_large_24_p8_384_dist,xcit_large_24_p8_384_dist,xcit_large_24_p8_384_dist,xcit_large_24_p8_384_dist,xcit_large_24 p8 224,xcittinyp8 224
C Correlations with ImageNet validation accuracy
We note that alignment metrics also have a (weak) U-shaped relationship with ImageNet validation accuracy (see Figure 5). We also note the following correlations between ImageNet validation accuracy and other properties which show that validation accuracy does not explain all the variance in downstream properties:
- ImageNet-A: 0.81
ImageNet-S: 0.86
ImageNet-R: 0.82
CIFAR10 1-shot learning: 0.68
FMNIST 1-shot learning:0.10
MNIST 1-shot learning: 0.07
Figure 5: Comparing Pearson pairwise alignment and ImageNet validation accuracy.
D Outlier examples
We compared performance on the original 'noisy' ImageNet and relabeled ImageNet-ReaL [10]. All models had better performance on ReaL, but the improvement has a weak inverted-U relationship with alignment. This may seem surprising, but our understanding is that the ReaL dataset was collected by relabelling images on which models (pre-2020) disagreed with the original labels. Models with low performance by today's standards (and alignment near 0.5) were used to select which images to relabel, so it is unsurprising that models with alignment near 0.5 benefit most from this relabeling. This suggests that ImageNet labels may need to be revisited once again as ReaL relabelling may have been biased.
E Reproducibility: Code and results data
All code and full results data are provided as part of the supplemental information. We will share them publicly after the anonymity period is over. All experiments were conducted on an AWS "x1.16xlarge" instance (no GPUs).