SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Experiments on Cassava Leaf Disease, SIIM-ISIC Melanoma, and EuroSAT datasets were ran on TPU v2-8s, while all other datasets were ran on NVIDIA A40s.
All experiments were ran with mini-batch size of 128.
For SGD experiments, we use Nesterov momentum, set momentum to 0.9, and try learning rates of 1e-1, 1e-2, 1e-3, and 1e-4. For AdamW experiments, we try learning rates of 1e-2, 1e-3, 1e-4, 1e-5. For all experiments, we try weight decays of 1e-3, 1e-4, 1e-5, 1e-6, and 0.
For all experiments, we use weights that are pretrained on ImageNet. AlexNet, DenseNet, MobileNet, ResNet, ResNext, ShuffleNet, SqueezeNet and VGG models are from torchvision, while ConvNext, DeiT, EfficientNet, InceptionResNet, and PNASNet models are from timm. Additionally, we normalize images to ImageNet's mean and standard deviation.
For EuroSAT we random resize crop to 224 with area at least 0.65.
For all other datasets, we random resize crop with area at least 0.65 to 224 for DeiT models, and 256 for all other models. Additionally, we use horizontal flips. For Human Protein Atlas, Cassava Leaf Disease, and SIIM-ISIC Melanoma, we also use vertical flips.
For SIIM-ISIC Melanoma, we train for 10 epochs, and for the step scheduler decay with factor 0.1 at 5 epochs.
For all other datasets, we train for 30 epochs, and for the step scheduler decay with factor 0.1 at 15, 20, and 25 epochs.
E CIFAR-10 on hyperparameter grid
Figure 6: Transfer performance across models from ImageNet to CIFAR-10. Green linear trend is computed across all models, while blue linear trend is restricted to models above  $70\%$  ImageNet accuracy. We use  $95\%$  confidence intervals computed with Clopper-Pearson.
●AlexNet	■ResNet-50	●Inception-ResNet v2	✓DenseNet-121
mMobileNetV3-small	◆ResNet-152	*VGG-16 BN	▲ResNeXt-50-32x4d
◆VGG-13 BN	▷DeiT-small	●EfficientNet B0	◁ShuffleNetV2x1.0
◁DeiT-tiny	●PNASNet-5	●EfficientNet B4	▼ConvNext-tiny
F APTOS 2019 Blindness Detection ablations
Scores presented are submissions to the Kaggle leaderboard. All scores are evaluated with quadratic weighted kappa. Within each entry, we first present the private leaderboard score, then the public leaderboard score. The private leaderboard represents  $85\%$  of the test data, while the public leaderboard is the remaining  $15\%$ .
Models used here are trained using AdamW with a cosine scheduler. We random resize crop to 512, use random rotations, and use color jitter (brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1). We train on all the available training data, no longer using the local train/Validation split mentioned in the main text. This includes both the training data in the 2019 competition, as well as data from a prior 2015 diabetic retinopathy competition.
Table 6: Comparing various models with additional interventions by evaluating on the Kaggle leaderboard.
lr \wd	1.00E-04	1.00E-05	1.00E-06
ResNet-50	1.00E-03	0.8610 / 0.6317	0.8570 / 0.6180	0.8548 / 0.6646
1.00E-04	0.8952 / 0.7531	0.8918 / 0.7204	0.8961 / 0.7547
ResNet-152	1.00E-03	0.8658 / 0.6812	0.8686 / 0.6612	0.8640 / 0.6554
1.00E-04	0.8898 / 0.7164	0.8836 / 0.6946	0.8859 / 0.6947
Inception-Resnet-v2	1.00E-03	0.8933 / 0.7748	0.8905 / 0.7565	0.8960 / 0.7585
1.00E-04	0.8897 / 0.7210	0.8929 / 0.7420	0.8944 / 0.7439
Table 7: Comparing the effect of augmentation on Kaggle leaderboard scores. More augmentation is as described earlier in this section. Less augmentation only uses random resize crop with at least 0.65 area and horizontal flips.
lr \wd	1.00E-04	1.00E-05	1.00E-06
ResNet-50	1.00E-03	0.8669 / 0.6405	0.8520 / 0.6013	0.8613 / 0.6269
less aug	1.00E-04	0.8525 / 0.6115	0.8570 / 0.6431	0.8483 / 0.6147
1.00E-05	0.8186 / 0.5071	0.8287 / 0.5647	0.8288 / 0.5328
ResNet-50	1.00E-03	0.8440 / 0.6432	0.8547 / 0.6856	0.8524 / 0.7125
more aug	1.00E-04	0.8948 / 0.7490	0.8972 / 0.7693	0.8999 / 0.7758
1.00E-05	0.8724 / 0.7370	0.8685 / 0.7567	0.8623 / 0.7376
G Melanoma Metric Comparison
Figure 7: Comparing transfer performance from ImageNet to Melanoma when using different metrics. Green linear trend is computed across all models, while blue linear trend is restricted to models above  $70\%$  ImageNet accuracy. Using accuracy implies that better ImageNet models transfer better; however, ROC is a better metric for this task.
ResNet-152
EfficientNet B0
ShuffleNetV2x1.0
m MobileNetV3-small
DeiT-small
EfficientNet B4
ConvNext-tiny
VGG-13 BN
PNASNet-5
DenseNet-121
ShuffleNetV2x0.5
DeiT-tiny
Inception-ResNet v2
ResNeXt-50-32x4d
SqueezeNet 1.1
ResNet-50
VGG-16 BN
H CLIP experiment details
Figure 8: Figure 4 with CLIP models overlaid (purple stars). The best CLIP models do better than all the ImageNet models, but when looking across all CLIP models, the patterns are more complicated.
● AlexNet	● PNASNet-5	▲ ResNeXt-50-32x4d	★ CLIP-RN50
m MobileNetV3-small	● Inception-ResNet v2	▲ ShuffleNetV2x1.0	★ CLIP-RN101
◆ VGG-13 BN	■ VGG-16 BN	▼ ConvNext-tiny	★ CLIP-B32
← DeiT-tiny	● EfficientNet B0	▲ ShuffleNetV2x0.5	★ CLIP-B16
■ ResNet-50	● EfficientNet B4	q SqueezeNet 1.1	★ CLIP-L14
◆ ResNet-152	▼ DenseNet-121	▲ ViT-B/16	★ CLIP-L14@336
→ DeiT-small
Table 8: For each CLIP pre-trained model, we provide the best performing model when fine-tuned on each dataset across our LP-FT hyperparameter grid
Model	ImageNet top-1	CCT20	APTOS	HPA	Melanoma	Cassava	EuroSAT
CLIP-RN50	73.3	74.45	0.9135	0.7053	0.9350	87.89	98.80
CLIP-RN101	75.7	75.19	0.9235	0.6909	0.9378	87.68	99.11
CLIP-B32	76.1	70.57	0.9137	0.5338	0.9546	86.28	99.26
CLIP-B16	80.2	77.81	0.9213	0.6365	0.9619	87.82	99.24
CLIP-L14	83.9	79.99	0.9330	0.6687	0.9717	88.82	99.33
CLIP-L14@336	85.4	83.17	0.9337	0.7131	0.9738	89.24	99.48
Table 9: We directly compare models pre-trained on ImageNet with models pre-trained on OpenAI's CLIP data. Specifically, we look at ResNet 50 and ViT B/16.
Model	ImageNet top-1	CCT20	APTOS	HPA	Melanoma	Cassava	EuroSAT
IN-ResNet-50	76.1	73.96	0.9215	0.6718	0.9524	87.75	99.19
CLIP-RN50	73.3	74.45	0.9135	0.7053	0.9350	87.89	98.80
IN-ViT-B/16	78.7	72.07	0.9262	0.5852	0.9600	86.63	99.28
CLIP-B16	80.2	77.81	0.9213	0.6365	0.9619	87.82	99.24
I CLIP fine-tuning details
We fine-tune by running a linear probe, followed by end-to-end fine-tuning on the best model from the first part. We keep total epochs consistent with the previous models, with a third of the epochs going toward linear probing. We use AdamW with a cosine decay schedule. During the linear probe, we search over  $10^{-1}$ ,  $10^{-2}$ , and  $10^{-3}$  learning rates, and during fine-tuning, we search over  $10^{-4}$ ,  $10^{-5}$ , and  $10^{-6}$  learning rates. For both parts, we search over  $10^{-3}$  to  $10^{-6}$  and 0 for weight decay.
J Creation information for datasets studied in Kornblith et al. [31]
Table 10: We find that the 12 datasets studied in Kornblith et al. [31] come from web scraping.
Dataset	Origin	Additional information
Food-101	foodspotting.com	Users upload an image of their food and annotate the type of food; categories chosen by popularity
CIFAR-10	TinyImages	Web crawl
CIFAR-100	TinyImages	Web crawl
Birdsnap	Flickr	Also used MTurk
SUN397	Web search engines	Also used WordNet
Stanford Cars	Flickr, Google, Bing	Also used MTurk
FGVC Aircraft	airliners.net	Images taken by 10 photographers
Pascal VOC 2007 Cls.	Flickr	N/A
Describable Textures	Google and Flickr	Also used MTurk
Oxford-IIT Pets	Flickr, Google, Catster, Dogster	Catster and Dogster are social websites for col- lecting and discussing pet images
Caltech-101	Google	97 categories chosen from Webster Collegiate Dictionary categories associated with a drawing
Oxford 102 Flowers	Mostly collected from web	A small number of images acquired by the paper authors taking the pictures
K Relationship between model size and transfer performance
Figure 9: We compare model size with downstream transfer performance. Again we use separate trend lines for all models (green) and only those above  $70\%$  ImageNet accuracy (blue). We use  $95\%$  confidence intervals computed with Clopper-Pearson for accuracy metrics and bootstrap with 10,000 trials for other metrics.
L FID score details
Table 11: We calculate FID scores between the ImageNet validation set and each of the datasets we study, as well as between the ImageNet validation set and each of the datasets in Kornblith et al. [31]. We found that dataset size affects FID score, so we take a 3,662 subset of each downstream dataset. Note that 3,662 is the size of APTOS, which is the smallest dataset.
Dataset	FID
CCT-20	162.69
APTOS	196.24
HPA	230.70
Cassava	179.24
Melanoma	186.34
EuroSAT	151.85
Food-101	108.35
CIFAR-10	132.53
CIFAR-100	120.72
Birdsnap	94.08
SUN397	62.95
Stanford Cars	143.35
FGVC Aircraft	183.35
Pascal VOC 2007 Cls.	39.84
Describable Textures	89.13
Oxford-IIT Pets	77.27
Caltech-101	50.77
Oxford 102 Flowers	140.21
M Predictive power of accuracy on non-web-scraped datasets on novel datasets
We observe that, on many non-web-scraped datasets, accuracy correlates only weakly with ImageNet accuracy. It is thus worth asking whether other predictors might correlate better. In this section, we examine the extent to which accuracy on a given non-web-scraped target dataset can be predicted from the accuracy on the other non-web-scraped target datasets.
M.1 F-test
We can further measure the extent to which the averages of the five other datasets beyond the predictive power provided by ImageNet by using F-tests. For each target task, we fit a linear regression model that predicts accuracy as either ImageNet accuracy or the average accuracy on the other five non-web-scraped datasets, and a second linear regression model that predicts accuracy as a function of both ImageNet accuracy and the average accuracy on the other five datasets. Since the first model is nested within the second, the second model must explain at least as much variance as the first. The F-test measures whether the increase in explained variance is significant. For these experiments, we logit-transform accuracy values and standardize them to zero mean and unit variance before computing the averages, as in the middle column of Table 13.
Results are shown in Table 12. The average accuracy across the other five datasets explains variance beyond that explained by ImageNet accuracy alone on five of the six datasets. The only exception is EuroSAT, where the range of accuracies is low (most models get  $\sim 99\%$ ) and a significant fraction of the variance among models may correspond to noise. By contrast, ImageNet accuracy explains variance beyond the average accuracy only on two datasets (APTOS and Melanoma). These results indicate that there are patterns in how well different models transfer to non-web-scraped data that are not captured by ImageNet accuracy alone, but are captured by the accuracy on other non-web-scraped datasets.
Table 12: Results of the F-test described in Section M.1. “+Avg. across datasets” tests whether a model that includes both ImageNet accuracy and the average accuracy across the 5 other datasets explains more variance than a model that includes only ImageNet accuracy. “+ImageNet” tests whether a model that includes both predictors explains more variance than a model that includes only the average accuracy across the 5 other datasets. In addition to  $F$  and  $p$  values, we report adjusted  $R^2$  for all models.  $p$ -values  $< 0.05$  are bold-faced.
Dataset	+Avg. across datasets	+ImageNet	Adj. R2(ImageNet-only)	Adj. R2(Average-only)	Adj. R2(Both predictors)
F(1, 16)	p-value	F(1, 16)	p-value
CCT-20	8.2	0.01	0.69	0.42	0.56	0.70	0.69
APTOS	31.0	0.00004	4.6	0.047	0.34	0.71	0.76
HPA	11.8	0.003	0.84	0.37	0.60	0.76	0.76
Melanoma	5.8	0.03	7.8	0.01	0.74	0.71	0.79
Cassava	13.2	0.002	0.14	0.71	0.55	0.75	0.74
EuroSAT	2.9	0.11	0.72	0.41	0.43	0.52	0.49
M.2 Spearman correlation
Table 13: We measure the Spearman correlation between each dataset with either the average of the 5 other datasets we study, or with ImageNet. Normalization is done by logit transforming accuracies, and then standardizing to zero mean and unit variance. The results suggest that using additional datasets is more predictive of model performance than just using ImageNet.
Dataset	Avg of 5 others (unnormized)	Avg of 5 others (normalized)	ImageNet
ρ	p-value	ρ	p-value	ρ	p-value
CCT-20	0.8684	0.0000	0.9263	0.0000	0.5825	0.0089
APTOS	0.7205	0.0005	0.6950	0.0010	0.3010	0.2105
HPA	0.7351	0.0003	0.6825	0.0013	0.6491	0.0026
Melanoma	0.6561	0.0023	0.7807	0.0000	0.7667	0.0001
Cassava	0.8872	0.0000	0.7442	0.0003	0.5222	0.0218
EuroSAT	0.3030	0.2073	0.3821	0.1065	0.4734	0.0406
N Pre-training augmentation details
Table 14: For each ImageNet pre-trained model, we provide the augmentation strategy used during pre-training time.
Model	Augmentation
AlexNet	Resize + Crop + Flip
SqueezeNet 1.1	Resize + Crop + Flip
ShuffleNetV2x0.5	AutoAugment (TrivialAugmentWide) + RandErasing + MixUp + CutMix
MobileNet V3 small	AutoAugment (ImageNet/Default)+ RandErasing
ShuffleNetV2x1.0	AutoAugment (TrivialAugmentWide) + RandErasing + MixUp + CutMix
VGG-13 BN	Resize + Crop + Flip
DeiT-tiny	RandAugment + RandErasing
VGG-16 BN	Resize + Crop + Flip
DenseNet-121	Resize + Crop + Flip
ResNet-50	Resize + Crop + Flip
ResNeXt-50-32x4d	Resize + Crop + Flip
EfficientNet B0	RandAugment
ResNet-152	Resize + Crop + Flip
ViT-B/16	RandAugment + MixUp
DeiT-small	RandAugment + RandErasing
Inception-ResNet v2	Inception Preprocessing (Color Distort + Resize + Crop + Flip)
ConvNext-tiny	AutoAugment (TrivialAugmentWide) + RandErasing + MixUp + CutMix
PNASNet-5 large	Whiten + Resize + Crop + Flip
EfficientNet B4	RandAugment
Figure 10: Figure 1 with points colored by general pre-training augmentation strategy. Cyan points use simple augmentation (resize, crops, flips, etc.), and red points use automatic augmentation (RandAugment, AutoAugment, TrivialAugmentWide).
● AlexNet	◆ ResNet-152	● EfficientNet B0	▼ ConvNext-tiny
m MobileNetV3-small	◆ DeiT-small	● EfficientNet B4	► ShuffleNetV2x0.5
VGG-13 BN	PNASNet-5	► DenseNet-121	q SqueezeNet 1.1
DeiT-tiny	◆ Inception-ResNet v2	► ResNeXt-50-32x4d	▲ ViT-B/16
■ ResNet-50	■ VGG-16 BN	► ShuffleNetV2x1.0
O Additional Models
Here we include four additional low accuracy models and two newer high accuracy models that were not originally included in our study. While the inclusion of the newer models causes the linear fit slopes to go up slightly, progress on downstream tasks still significantly lags behind progress on ImageNet for some of the datasets.
O.1 Models
Model	ImageNet top-1	# params	Year Released
TinyNet e [18]	59.9	2.0M	2020
DLA46 c [77]	64.9	1.3M	2018
DLA46x c [77]	66.0	1.1M	2018
TinyNet d [18]	67.0	2.3M	2020
SWIN Base [38]	83.6	88M	2021
CAFormer B36 [79]	85.5	99M	2022
Figure 11: Figure 1 with additional models. Green is linear trend of all models, while blue is linear trend for models above  $70\%$  ImageNet accuracy.
Table 15: We summarize the blue regression lines from Figure 11, calculated on models above  $70\%$  ImageNet accuracy, with their correlation and slope. Slope is calculated so that all metrics have a range from 0 to 100.
Dataset	Correlation	Slope
Caltech Camera Traps	0.43	0.28
APTOS 2019 Blindness	0.23	0.03
Human Protein Atlas	0.39	0.39
SIIM-ISIC Melanoma	0.63	0.08
Cassava Leaf Disease	0.31	0.06
EuroSAT	0.36	0.02
Table 16: For each ImageNet pre-trained model, we provide the best performing model when fine-tuned on each dataset across our hyperparameter grid
Model	ImageNet top-1	CCT20	APTOS	HPA	Melanoma	Cassava	EuroSAT
TinyNet e	59.9	66.05	0.9096	0.6189	0.9310	86.67	99.09
DLA46 c	64.9	71.99	0.9123	0.6265	0.9265	88.29	99.04
DLA46x c	66.0	70.07	0.9088	0.6306	0.9278	88.22	98.94
TinyNet d	67.0	69.99	0.9063	0.6208	0.9365	86.35	99.07
SWIN Base	83.6	76.79	0.9317	0.7369	0.9647	88.43	99.44
CAFormer B36	85.5	79.36	0.9261	0.6939	0.9669	88.92	99.41