SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 17: MRR scores for image retrieval experiments.
Backbone	Method	Pretrain Data	CUB	iNat	Obj	INSTRE	GLM	rOxf	rPar	CopyD
ConvNext-Base	Supervised	ImageNet-1k	0.63	0.15	0.38	0.88	0.37	0.63	0.97	0.87
Supervised	ImageNet-21k-1k	0.9	0.41	0.52	0.94	0.53	0.84	0.99	0.92
ConvNext-Small	Supervised	ImageNet-1k	0.66	0.16	0.38	0.87	0.36	0.68	0.97	0.86
Supervised	ImageNet-21k-1k	0.9	0.38	0.49	0.92	0.49	0.85	0.98	0.9
ConvNext-XLarge	Supervised	ImageNet-21k-1k	0.9	0.43	0.53	0.93	0.52	0.87	0.99	0.92
ResNet-101	CLIP	OpenAI	0.67	0.15	0.31	0.93	0.44	0.65	0.97	0.86
Supervised	ImageNet-1k	0.66	0.16	0.35	0.86	0.4	0.67	0.97	0.86
ResNet-50	CLIP	OpenAI	0.59	0.12	0.24	0.88	0.41	0.67	0.97	0.86
Supervised	ImageNet-1k	0.6	0.13	0.28	0.84	0.37	0.65	0.97	0.87
VicReg	ImageNet-1k	0.45	0.11	0.18	0.86	0.42	0.8	0.97	0.89
ResNet-50x64	CLIP	OpenAI	0.83	0.27	0.53	0.99	0.60	0.85	0.99	0.95
SwinV2-Base	MiDaS	MiDaS	0.54	0.13	0.17	0.70	0.36	0.72	0.97	0.90
SwinV2-Base-w16	Supervised	ImageNet-21k-1k	0.89	0.4	0.51	0.93	0.51	0.87	0.99	0.92
SwinV2-Base-w24	Supervised	ImageNet-21k-1k	0.89	0.43	0.51	0.94	0.52	0.87	0.99	0.93
SwinV2-Large-w16	Supervised	ImageNet-21k-1k	0.88	0.39	0.47	0.94	0.52	0.84	0.98	0.93
SwinV2-Large-w24	Supervised	ImageNet-21k-1k	0.88	0.42	0.49	0.94	0.52	0.84	0.98	0.94
SwinV2-Tiny	MiDaS	MiDaS	0.51	0.09	0.16	0.64	0.35	0.58	0.96	0.88
Supervised	ImageNet-1k	0.68	0.17	0.32	0.9	0.4	0.67	0.99	0.89
ViT-Base	CLIP	LAION-2B	0.82	0.23	0.46	0.98	0.55	0.87	0.97	0.93
MAE	ImageNet-1k	0.14	0.01	0.04	0.34	0.07	0.33	0.81	0.61
MoCoV3	ImageNet-1k	0.57	0.13	0.2	0.84	0.38	0.78	0.95	0.91
Supervised	ImageNet-1k	0.56	0.12	0.23	0.84	0.3	0.56	0.95	0.86
Supervised	ImageNet-21k-1k	0.85	0.29	0.33	0.88	0.41	0.74	0.96	0.9
DINO	ImageNet-1k	0.72	0.21	0.29	0.91	0.43	0.85	0.99	0.91
ViT-Large	MAE	ImageNet-1k	0.24	0.04	0.05	0.56	0.16	0.53	0.92	0.81
CLIP	LAION-2B	0.85	0.26	0.47	0.98	0.58	0.82	0.97	0.94
Supervised	ImageNet-21k-1k	0.88	0.38	0.47	0.9	0.47	0.83	0.97	0.92
ViT-Small	MoCoV3	ImageNet-1k	0.57	0.12	0.18	0.82	0.36	0.7	0.97	0.89
Supervised	ImageNet-1k	0.61	0.13	0.25	0.85	0.32	0.61	0.95	0.87
Supervised	ImageNet-21k-1k	0.84	0.27	0.3	0.88	0.43	0.74	0.96	0.88
DINO	ImageNet-1k	0.79	0.21	0.27	0.9	0.42	0.87	0.99	0.9
training period is enough for finetuning, where we only train the model for 100 epochs which is 1/3 as many epochs as we use for training from scratch. Also on smaller datasets, such as Flowers-102 and aircraft datasets, finetuning obtains much higher accuracy compared to training from scratch. In contrast, finetuning does not save quite as many epochs for detection and segmentation where detection systems contain lots of new parameters that are randomly initialized for downstream training.
Table 18: Recall@5 scores for image retrieval experiments.
Backbone	Method	Pretrain Data	CUB	iNat	Obj	INSTRE	GLM	rOxf	rPar	CopyD
ConvNext-Base	Supervised	ImageNet-1k	0.045	0.069	0.03	0.097	0.099	0.087	0.026	0.854
Supervised	ImageNet-21k-1k	0.085	0.221	0.046	0.111	0.163	0.174	0.027	0.902
ConvNext-Small	Supervised	ImageNet-1k	0.05	0.069	0.029	0.094	0.099	0.092	0.026	0.851
Supervised	ImageNet-21k-1k	0.084	0.196	0.042	0.106	0.151	0.169	0.027	0.888
ConvNext-XLarge	Supervised	ImageNet-21k-1k	0.086	0.247	0.048	0.111	0.164	0.172	0.027	0.903
ResNet-101	CLIP	OpenAI	0.05	0.06	0.02	0.106	0.117	0.082	0.026	0.848
Supervised	ImageNet-1k	0.049	0.07	0.026	0.092	0.107	0.085	0.026	0.857
ResNet-50	CLIP	OpenAI	0.04	0.046	0.015	0.097	0.107	0.092	0.026	0.852
Supervised	ImageNet-1k	0.043	0.056	0.018	0.089	0.099	0.084	0.026	0.836
VicReg	ImageNet-1k	0.026	0.042	0.009	0.09	0.116	0.106	0.027	0.868
ResNet-50x64	CLIP	OpenAI	0.073	0.124	0.044	0.122	0.191	0.181	0.027	0.948
SwinV2-Base	MiDaS	MiDaS	0.032	0.051	0.008	0.063	0.095	0.113	0.027	0.890
SwinV2-Base-w16	Supervised	ImageNet-21k-1k	0.083	0.207	0.043	0.109	0.154	0.178	0.027	0.917
SwinV2-Base-w24	Supervised	ImageNet-21k-1k	0.084	0.226	0.045	0.111	0.158	0.177	0.027	0.923
SwinV2-Large-w16	Supervised	ImageNet-21k-1k	0.082	0.204	0.04	0.112	0.16	0.167	0.027	0.93
SwinV2-Large-w24	Supervised	ImageNet-21k-1k	0.083	0.223	0.041	0.113	0.161	0.17	0.027	0.931
SwinV2-Tiny	MiDaS	MiDaS	0.031	0.034	0.008	0.055	0.095	0.059	0.026	0.849
Supervised	ImageNet-1k	0.052	0.072	0.021	0.101	0.114	0.096	0.027	0.868
ViT-Base	CLIP	LAION-2B	0.071	0.104	0.037	0.12	0.172	0.175	0.027	0.92
MAE	ImageNet-1k	0.005	0.003	0.001	0.02	0.012	0.02	0.015	0.542
MoCoV3	ImageNet-1k	0.036	0.055	0.01	0.088	0.1	0.114	0.026	0.887
Supervised	ImageNet-1k	0.039	0.049	0.015	0.088	0.078	0.053	0.025	0.82
Supervised	ImageNet-21k-1k	0.077	0.146	0.024	0.094	0.111	0.088	0.026	0.883
DINO	ImageNet-1k	0.055	0.093	0.018	0.102	0.125	0.158	0.027	0.894
ViT-Large	CLIP	LAION-2B	0.077	0.122	0.038	0.12	0.183	0.127	0.027	0.934
MAE	ImageNet-1k	0.01	0.012	0.002	0.042	0.036	0.038	0.023	0.753
Supervised	ImageNet-21k-1k	0.082	0.207	0.04	0.099	0.141	0.15	0.027	0.9
ViT-Small	MoCoV3	ImageNet-1k	0.037	0.05	0.009	0.082	0.097	0.094	0.026	0.877
Supervised	ImageNet-1k	0.044	0.053	0.016	0.088	0.085	0.062	0.025	0.843
Supervised	ImageNet-21k-1k	0.076	0.126	0.021	0.097	0.118	0.097	0.026	0.869
DINO	ImageNet-1k	0.065	0.093	0.016	0.101	0.123	0.16	0.027	0.888
Table 19: Top-1 classification accuracy (%) for ImageNet against adversarial attacks with  $\ell_{\infty}$  constraint radii 1/255 and 2/255.
Backbone	Method	Pretrain Data	Clean	ε = 1/255	ε = 2/255	Z-scores
ResNet-50	Supervised	ImageNet-1k	80.38	28.79	13.25	-0.99
VicReg	ImageNet-1k	78.77	36.60	22.01	-0.26
CLIP	LAION-2B	80.55	32.78	18.55	0.58
DINO	ImageNet-1k	79.08	35.80	20.75	-0.35
ResNet-101	Supervised	ImageNet-1k	81.93	44.23	32.43	0.53
ViT-Small	Supervised	ImageNet-1k	78.84	21.21	6.42	-1.62
Supervised	ImageNet-21k	81.39	16.50	3.83	-1.94
MoCoV3	ImageNet-1k	79.65	48.62	32.16	0.71
DINO	ImageNet-1k	81.33	47.87	30.04	0.57
ViT-Base	Supervised	ImageNet-1k	79.15	27.08	9.54	-1.23
Supervised	ImageNet-21k	84.53	23.04	6.92	-1.52
MoCoV3	ImageNet-1k	82.85	55.44	39.49	1.32
MAE	ImageNet-1k	83.41	50.85	31.69	0.77
DINO	ImageNet-1k	83.40	53.59	36.61	1.11
CLIP	LAION-2B	85.19	47.91	28.35	0.50
SwinV2-Tiny	Supervised	ImageNet-1k	81.82	40.91	23.15	-0.03
MiDaS	MiDaS	82.55	41.44	25.20	0.08
ConvNeXt-Tiny	Supervised	ImageNet-1k	82.10	49.74	31.42	0.72
ConvNeXt-Base	Supervised	ImageNet-1k	83.88	55.31	37.19	1.21
Supervised	ImageNet-21k	85.87	53.78	34.05	1.00
Table 20: Z-scores for best-performing SSL and supervised learning backbones. Mean z-scores for each task averaged across the 3 top performing backbones dividing models into self (weakly)-supervised learning (SSL) on ImageNet-1k, supervised learning on ImageNet-1k (Sup-1k), and ImageNet-21k (Sup-21k).
Task	SSL	Sup-1k	Sup-21k
Cls	0.573	0.527	0.936
Det	0.298	0.743	1.076
Seg	0.314	0.717	1.071
Ret	0.489	-0.079	0.708
(OOD) Cls	0.419	0.287	1.271
(OOD) Det	0.414	0.923	0.853