SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Figure 3a, we visualize the multi-ID effective robustness on CIFAR-10.2, where the accuracies of all the models approximately lie on a plane (the baseline function) on the logit scale, and thus these models have similar effective robustness as the OOD accuracy of all the models can be approximately predicted using a simple plane. We also show projected views of Figure 3a in Figure 4, where Figure 4a and Figure 4b correspond to the single-ID evaluation taking different ID test sets with contradictory conclusions. In contrast, our new evaluation provides a more holistic view.
5.3 Evaluation on ImageNet-like OOD Test Sets
We then conduct evaluation on ImageNet-like OOD test sets, and we compare ImageNet models with models trained on YFCC and LAION, respectively. We show the fitting quality in Table 2 and the effective robustness in Tables 3a and 3b. Consistent with results in Section 5.2, our multi-ID evaluation improves the fitting quality over the single-ID evaluation to better predict and understand the OOD accuracies from ID accuracies. On effective robustness, single-ID evaluation leads to a perception of an effective robustness gain when there is mismatch between the training data and the single ID test set. Our multi-ID evaluation enjoys a holistic view of all the training distributions and suggests that all the models evaluated here have similar effective robustness.
Specifically, the improvement of fitting quality is particularly significant for models involving LAION on ImageNet-R ( $R^2$  improved from 0.216 to 0.982 and MAE reduced from  $9.23\%$  to  $1.32\%$ ) and ImageNet-Sketch ( $R^2$  improved from 0.281 to 0.937 and MAE reduced from  $7.90\%$  to  $2.10\%$ ). On the effective robustness values, under the single-ID evaluation, YFCC and LAION models have positive effective robustness values ( $2.59 \pm 2.43\%$ ) on average for YFCC models and  $5.96 \pm 4.96\%$ ) on average for LAION models), which is consistent with the findings in Fang et al. (2022); Nguyen et al. (2022). In contrast, under our multi-ID evaluation, the average effective robustness becomes  $0.77 \pm 0.85\%$  for YFCC models, and  $-0.00 \pm 0.52\%$  for LAION models, much closer to 0. While single-ID evaluation used by prior works (Fang et al., 2022; Nguyen et al., 2022) suggests effective
Table 2: Fitting quality of single-ID and multi-ID effective robustness, respectively, on ImageNet-like OOD test sets. For ImageNet v.s. YFCC, models involved include ImageNet, YFCC, and ImageNet+YFCC models. For ImageNet v.s. LAION, models involved include ImageNet, LAION, and ImageNet+LAION models. Multi-ID evaluation improves the fitting quality.
ImageNet v.s.	Test set	R² (↑)	MAE (%, ↓)
Single-ID	Multi-ID	Single-ID	Multi-ID
YFCC (103 models)	ImageNet-V2	0.990	0.999	1.44	0.54
ImageNet-R	0.879	0.965	2.55	1.30
ImageNet-Sketch	0.928	0.945	1.56	1.31
ObjectNet	0.903	0.936	2.60	1.98
LAION (107 models)	ImageNet-V2	0.992	0.999	1.33	0.51
ImageNet-R	0.216	0.982	9.23	1.32
ImageNet-Sketch	0.281	0.937	7.90	2.10
ObjectNet	0.849	0.906	2.88	2.38
Table 3: Single-ID and multi-ID effective robustness (%) of the models on variants of ImageNet OOD test sets. The effective robustness of all the models becomes close to 0 under the multi-ID evaluation.
(a) Models involving ImageNet and YFCC.
Models	Test set	Single-ID	Multi-ID
ImageNet (36 models)	ImageNet-V2	-1.23±0.46	-0.19±0.50
ImageNet-R	-2.80±1.34	-0.41±1.83
ImageNet-Sketch	-1.25±1.90	0.14±2.57
ObjectNet	-0.99±4.23	0.74±4.14
Average	-1.57±1.20	0.07±1.68
YFCC (13 models)	ImageNet-V2	1.69±1.84	-0.16±0.57
ImageNet-R	3.44±3.25	1.07±1.17
ImageNet-Sketch	1.90±2.25	0.89±1.29
ObjectNet	3.32±2.55	1.27±0.85
Average	2.59±2.43	0.77±0.85
ImageNet+YFCC (54 models)	ImageNet-V2	0.70±1.75	0.13±0.79
ImageNet-R	1.31±2.75	0.16±1.76
ImageNet-Sketch	0.69±1.89	0.21±1.52
ObjectNet	0.21±1.98	-0.60±1.01
Average	0.73±1.96	-0.03±1.06
(b) Models involving ImageNet and LAION.
Models	Test set	Single-ID	Multi-ID
ImageNet (37 models)	ImageNet-V2	-1.21±0.56	0.05±0.65
ImageNet-R	-9.45±2.79	-0.54±1.90
ImageNet-Sketch	-7.63±3.40	-0.72±3.03
ObjectNet	-1.90±4.48	1.14±4.18
Average	-5.05±2.21	-0.02±1.53
LAION (14 models)	ImageNet-V2	1.42±1.73	-0.03±0.57
ImageNet-R	9.48±8.84	-0.65±1.05
ImageNet-Sketch	8.71±7.15	-1.10±1.98
ObjectNet	4.24±2.39	1.77±1.20
Average	5.96±4.96	-0.00±0.52
ImageNet+LAION (56 models)	ImageNet-V2	0.65±1.43	0.07±0.61
ImageNet-R	6.01±8.43	0.56±1.32
ImageNet-Sketch	5.99±7.39	1.04±2.46
ObjectNet	0.63±2.20	-1.01±1.44
Average	3.32±4.63	0.16±0.65
robustness gains of YFCC models compared to ImageNet models (Figure 5a), all the models have similar effective robustness under our multi-ID evaluation. Additionally, we provide an ablation study on using models with mixed training data in Appendix A.3 and additional interactive visualization on our website at https://shizhouxing.github.io/effective-robustness.
5.4 Evaluation on Additional Models
We also evaluate additional models that are not used in fitting the baseline functions. We download models pre-trained by existing works, including OpenCLIP (Ilharco et al., 2021) and SLIP (Mu et al., 2022). OpenCLIP provides CLIP models trained on YFCC and LAION, and SLIP provides YFCC models trained using CLIP and also a combination of self-supervised learning (Chen et al., 2020a,b) and CLIP (SimCLR+CLIP namely SLIP). And we also fine-tune CLIP models on ImageNet for models we pre-train on YFCC and LAION. We use both vanilla fine-tuning and also WiSE-FT (Wortsman et al., 2022b) which aims to improve the robustness after fine-tuning, using a weight-space ensembling of the pre-trained model and the fine-tuned model. Details are in Appendix B.1.
In Table 4, we show results involving YFCC and LAION, respectively. Since these models are not used in the fitting, we do not use  $R^2$ , but we use MAE to measure the fitting quality. Our multi-ID evaluation generally reduces MAE compared to the single-ID evaluation, and thus the multi-ID evaluation can still more accurately predict the OOD accuracy from the ID accuracies for these models that are not used in the fitting. The effective robustness values of the models also generally become closer to 0, especially for the zero-shot CLIP models. The results further validate that zero-shot CLIP models, although may achieve high OOD performance if pre-trained with large-scale data (Radford et al., 2021), generally do not improve effective robustness if we control for all the ID accuracies. Among the models evaluated here, SLIP models on YFCC and WiSE-FT models from LAION achieve relatively higher average effective robustness compared to other models, under our multi-ID evaluation, although the gains are not consistently significant on all the test sets and become much smaller than those reflected in the single-ID evaluation. However, we are not certain
Table 4: Fitting quality and effective robustness for downloaded and fine-tuned models. The models are not used in the fitting and directly evaluated. Note that MAE and effective robustness are different, where MAE takes absolute values but not effective robustness. For CLIP by Mu et al. (2022) and SLIP, only models pre-trained on YFCC are available.
Model	Test set	Models with pre-training on YFCC	Models with pre-training on LAION
MAE (%, ↓)	Effective Robustness (%)	MAE (%, ↓)	Effective Robustness (%)
Single-ID	Multi-ID	Single-ID	Multi-ID	Single-ID	Multi-ID	Single-ID	Multi-ID
OpenCLIP	ImageNetN-V2	3.95	0.45	3.95±0.70	-0.33±0.45	4.70	0.38	4.70±0.01	0.38±0.01
ImageNetN-R	8.98	3.12	8.98±1.24	3.12±1.27	39.80	7.49	39.80±0.03	7.49±0.09
ImageNetN-Sketch	5.32	2.49	5.32±1.07	2.49±0.81	38.92	1.31	38.92±0.00	-1.31±0.23
ObjectNet	5.68	1.04	5.68±0.81	-0.22±1.04	6.94	1.35	6.94±0.09	-1.35±0.04
Average	5.98	1.77	5.98±0.96	1.26±0.89	22.59	2.63	22.59±0.02	1.30±0.07
Vanilla FT	ImageNetN-V2	1.14	0.85	0.16±1.23	0.77±0.70	0.82	0.91	-0.12±0.89	0.51±0.96
ImageNetN-R	2.90	1.82	-2.90±1.86	-1.82±0.92	4.63	2.38	-4.45±2.89	2.27±2.20
ImageNetN-Sketch	2.26	3.16	2.20±1.86	3.16±1.19	4.78	7.50	3.47±4.67	7.50±3.29
ObjectNet	3.46	3.11	-3.42±2.36	-3.11±1.43	4.07	2.27	-4.07±2.10	-2.26±1.95
Average	2.44	2.23	-0.99±1.76	-0.25±0.96	3.57	3.27	-1.29±2.19	2.01±1.38
WiSE-FT	ImageNetN-V2	1.97	1.05	1.97±0.76	1.05±0.59	1.72	0.81	1.72±0.45	0.81±0.48
ImageNetN-R	3.64	1.76	3.64±0.61	1.76±0.60	13.08	7.40	13.08±1.85	7.40±1.10
ImageNetN-Sketch	5.47	4.41	5.47±1.20	4.41±0.99	16.84	10.21	16.84±1.58	10.21±0.49
ObjectNet	2.12	1.30	2.12±1.38	-0.68±1.34	1.65	1.65	1.52±1.64	0.31±1.67
Average	3.30	2.13	3.30±0.96	1.63±0.80	8.32	5.02	8.29±1.04	4.68±0.55
CLIP by Mu et al. (2022)	ImageNetN-V2	4.95	0.83	4.95±0.29	-0.83±0.45
ImageNetN-R	6.54	1.86	6.54±2.66	-1.86±1.15
ImageNetN-Sketch	1.49	4.16	0.58±1.68	-4.16±0.44
ObjectNet	9.32	1.49	9.32±1.45	1.49±0.44
Average	5.57	2.09	5.35±1.49	-1.34±0.28
SLIP	ImageNetN-V2	5.25	0.64	5.25±0.57	-0.12±0.84
ImageNetN-R	14.47	6.13	14.47±5.98	5.75±4.77
ImageNetN-Sketch	7.71	2.90	7.71±4.05	2.11±2.79
ObjectNet	14.79	5.02	14.79±2.77	5.02±1.53
Average	10.56	3.67	10.56±3.11	3.19±2.05
on whether SLIP and WiSE-FT can alter the underlying training distribution, because SLIP uses SimCLR that introduces different training images, and WiSE-FT essentially edits the weights of the models by weight-space ensembling. Thus, we do not draw a definite conclusion for the effective robustness of SLIP and WiSE-FT models and leave further validation for future work.
For natural distribution shifts, the linear correlations between the ID and OOD performance ("accuracy-on-the-line" (Miller et al., 2021) in the single-ID effective robustness) have earlier been observed in dataset reproduction works (Recht et al., 2018, 2019; Yadav & Bottou, 2019; Miller et al., 2020). Taori et al. (2020) evaluated many ImageNet models on several ImageNet-like OOD test sets, and given the widely held linear correlations, they proposed to evaluate effective robustness by controlling for ID accuracy. Miller et al. (2021) further validated accuracy-on-the-line with a broader scope. Nevertheless, accuracy-on-the-line may not hold on some distribution shifts, such as some corruption shifts (Hendrycks & Dietterich, 2018) and shifts in the wild (Miller et al., 2021), and sometimes ID accuracy and OOD accuracy can inversely correlate (Miller et al., 2021; Teney et al., 2022). Baek et al. (2022) also observed a linear correlation between ID agreement and OOD agreement for a pair of neural networks (namely "agreement-on-the-line") for testing whether accuracy-on-the-line holds by agreement-on-the-line which does not require labeled data. We focus on distribution shifts where at least accuracy-on-the-line holds for models from each of the training datasets, and we further propose "accuracy-on-the-plane" using multiple ID test sets.
Recently, CLIP-like models with language image pre-training which has been studied earlier in works such as Sariyildiz et al. (2020); Zhang et al. (2022); Desai & Johnson (2021), were shown to achieve exceptional effective robustness in the single-ID evaluation (Radford et al., 2021; Jia et al., 2021). Fang et al. (2022) analyzed the cause of the effective robustness gain of CLIP and concluded that the pre-training data determined the robustness. Nguyen et al. (2022) experimented on more pre-training data, and they observed difference in the single-ID effective robustness of models trained on different data. While Fang et al. (2022); Nguyen et al. (2022) both suggested that the pre-training data could determine the effective robustness gains, our evaluation suggests that zero-shot CLIP models do not have effective robustness gains. Besides, Kumar et al. (2021); Andreassen et al. (2022); Wortsman
et al. (2022b) studied the robustness of fine-tuned CLIP models. Moreover, Devillers et al. (2021); Santurkar et al. (2022) studied the transfer performance of CLIP models, which is out of our scope on the robustness against natural distribution shifts.
To conclude, we propose a new and more precise effective robustness evaluation for models with different training data. In our evaluation, the OOD accuracy can generally be better predicted from multiple ID accuracies compared to previous effective robustness evaluation with a single ID test. We find that zero-shot CLIP models pre-trained on language-image data do not have better effective robustness compared to standard image classifiers, and we provide a new understanding of the apparently significant effective robustness gains observed by prior works.
There remain several limitations that may be addressed in future works:
Acknowledgments & Funding Disclosure
We thank Alex Fang and Jindong Gu for helpful discussions and the reviewers for constructive feedback. This work was supported in part by NSF 2008173, 2048280, 2325121, 2331966, ONR N00014-23-1-2300:P00001.
Andreassen, A. J., Bahri, Y., Neyshabur, B., and Roelofs, R. The evolution of out-of-distribution robustness throughout fine-tuning. Transactions on Machine Learning Research, 2022.
Awadalla, A., Wortman, M., Ilharco, G., Min, S., Magnusson, I., Hajishirzi, H., and Schmidt, L. Exploring the landscape of distributional robustness for question answering models. In *Findings of the Association for Computational Linguistics: EMNLP* 2022, pp. 5971-5987, 2022.
Baek, C., Jiang, Y., Raghunathan, A., and Kolter, J. Z. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35:19274-19289, 2022.
Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33: 22243-22255, 2020b.
Darlow, L. N., Crowley, E. J., Antoniou, A., and Storkey, A. J. Cinc-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505, 2018.
Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Li, F. Imagenet: A large-scale hierarchical image database. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Desai, K. and Johnson, J. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11162-11173, 2021.
Devillers, B., Choksi, B., Bielawski, R., and VanRullen, R. Does language help generalization in vision models? In Proceedings of the 25th Conference on Computational Natural Language Learning, pp. 171-182, 2021.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Fang, A., Ilharco, G., Wortsman, M., Wan, Y., Shankar, V., Dave, A., and Schmidt, L. Data determines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning, pp. 6216-6234. PMLR, 2022.
Goyal, P., Duval, Q., Seessel, I., Caron, M., Singh, M., Misra, I., Sagun, L., Joulin, A., and Bojanowski, P. Vision models are more robust and fair when pretrained on uncurated images without supervision. arXiv preprint arXiv:2202.08360, 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.1109/CVPR.2016.90.
A.1 Projected Views
In Figure 5, we show projected views of Figure 3b.
(a) ImageNet-R accuracy against ImageNet accuracy. YFCC models have higher ImageNet-R accuracy compared to ImageNet models when controlling for ImageNet accuracy only.
(b) ImageNet-R accuracy against YFCC accuracy. YFCC models have similar ImageNet-R accuracy compared to ImageNet models when controlling for YFCC accuracy only.
Figure 5: Projected views of Figure 3b. Figure 5a and Figure 5b correspond to single-ID evaluation using different ID test sets, Figure 5a suggests effective robustness gains of YFCC models but the gains diminish in Figure 5b. Our multi-ID evaluation shows a holistic view where all the models have a similar effective robustness.
A.2 Agreement between Single-ID and Multi-ID Evaluation
We conduct an experiment to check the correlation between the single-ID evaluation and our new multi-ID evaluation, in terms of the relative ranking between different models trained on the same data. We use Kendall's rank correlation test (Kendall, 1948) and we report the  $\tau$  statistic computed by scipy.stats.kendalltau in Tables 5 to 7. Results show that the rankings on the single-ID effective robustness and multi-ID effective robustness are positively correlated for CIFAR-10 and ImageNet models. There is also a weaker positive correlation for YFCC models. For LAION models, there is sometimes a negative correlation. Overall, while there is often a positive correlation between the rankings provided by the single-ID evaluation and the multi-ID evaluation, respectively, it is not necessarily consistent on all the datasets. Thus, when comparing models trained on the same data, our multi-ID evaluation is not intended to replace the single-ID evaluation. Our multi-ID evaluation is mainly for comparing models trained on different data, and may be used as a supplementary evaluation if all the models are trained on a single dataset.
Table 5:  $\tau$  statistics in the Kendall's rank correlation test for evaluating the correlation between the rankings provided by the single-ID evaluation and the multi-ID evaluation, respectively, for models trained on the same data. We consider CIFAR-10 models and ImageNet models, respectively, on CIFAR-like OOD test sets. For ImageNet models, ImageNet instead of CIFAR-10 is used as the ID test set in the single-ID evaluation.
Test set	CIFAR-10.1	CIFAR-10.2	CINIC-10
CIFAR-10 models	0.9619	0.6667	0.8095
ImageNet models	0.1664	0.1522	0.4907
A.3 Models with Mixed Training Data in the Fitting
As mentioned in Section 5.1, we train models with mixed data to obtain models with diverse accuracies. In Tables 8 and 9, we show that if we do not include models with mixed training data in the fitting, the MAE for these models can become higher, although the difference is not large.
Table 6:  $\tau$  statistics (similar to Table 5) for ImageNet models and YFCC models on ImageNet-like OOD test sets. For YFCC models, YFCC instead of ImageNet is used as the ID test set in the single-ID evaluation.
Test set	ImageNet-V2	ImageNet-R	ImageNet-Sketch	ObjectNet
ImageNet models	0.5142	0.4412	0.8031	0.8063
YFCC models	0.0256	0.8461	0.7179	0.2564
Table 7:  $\tau$  statistics (similar to Table 5) for ImageNet models and LAION models on ImageNet-like OOD test sets. For LAION models, LAION instead of ImageNet is used as the ID test set in the single-ID evaluation.
Test set	ImageNet-V2	ImageNet-R	ImageNet-Sketch	ObjectNet
ImageNet models	0.3123	0.4384	0.6216	0.9729
LAION models	-0.4945	0.6483	-0.2747	0.6483
In this work, we train models with data mixed at various ratios and consider models with diverse combinations of accuracies, to obtain more convincing conclusions on the effective robustness.
Table 8: MAE (%) for ImageNet+YFCC models when they are excluded and included in the fitting, respectively, where comparing the effective robustness of ImageNet models and YFCC models.
Test set	ImageNet+YFCC models in the fitting
Excluded	Included
ImageNet-V2	0.71	0.64
ImageNet-R	1.30	1.21
ImageNet-Sketch	0.98	0.94
ObjectNet	1.73	0.95
B.1 Details of Models
We use TF-Vision<sup>3</sup> under the Apache License Version 2.0 to train standard classifiers on CIFAR-10 and ImageNet. We follow the configurations provided in TF-Vision for vanilla ResNet training on ImageNet and we train ResNet-18, ResNet-50 and ResNet-101 models. We reuse the configurations to train models on CIFAR-10, where we only change the dataset, number of classes, and image size, without tuning hyperparameters for the training. And we load checkpoints of ViT-S/16, ViT-B/16, and ViT-L/16 models pre-trained on ImageNet, provided by TF-Vision.
For training CLIP models, we mostly follow hyperparameters provided in Fang et al. (2022) and the implementation in Open-CLIP (Ilharco et al., 2021). While Fang et al. (2022) used a batch size of 1024, we use 2048 for more parallelism. We use YFCC-15M in Radford et al. (2021), which is a subset of YFCC-100M (Thomee et al., 2016). And we use LAION-15M which we uniformly sample from LAION-400M (Schuhmann et al., 2021). For fine-tuning CLIP models, we fine-tune for 50,000 steps, using learning rates  $3 \times 10^{-5}$  and  $1 \times 10^{-4}$ , respectively. For WiSE-FT, we take  $\alpha = 0.5$  which is the coefficient for weight-space ensembling. For OpenCLIP models $^{4}$ , we use ViT-B/32 models trained on LAION-400M. For SLIP $^{5}$ , we use all the CLIP and SLIP models trained on YFCC-15M.
For data subsampling, we uniformly sample a proportion of training examples from the entire dataset, at ratios of  $\{5\%, 10\%, 20\%, 30\%, 40\%, 50\}$ , respectively. For combining two training datasets at various ratios, given a coefficient  $\lambda$  ( $0 < \lambda < 1$ ), we uniformly sample a proportion of data from the two datasets at ratios of  $\lambda$  and  $(1 - \lambda)$ , respectively, and then we combine the two subsets. When combining ImageNet and CIFAR-10, we take  $\lambda \in \{0.001, 0.01, 0.1, 0.5, 0.9, 0.99, 0.995\}$ ; when combining ImageNet with YFCC and LAION, respectively, we take  $\lambda \in \{0.01, 0.1, 0.25, 0.5\}$ .
<sup>3</sup>https://github.com/tensorflow/models/tree/master/official/vision
<sup>4</sup>https://github.com/mlfoundations/openclip
<sup>5</sup>https://github.com/facebookresearch/SLIP
Table 9: MAE (%) for ImageNet+LAION models when they are excluded and included in the fitting, respectively, where comparing the effective robustness of ImageNet models and LAION models.
Test set	ImageNet+LAION models in the fitting
Excluded	Included
ImageNet-V2	0.54	0.51
ImageNet-R	1.56	1.19
ImageNet-Sketch	2.62	2.03
ObjectNet	2.91	1.47
Models are trained using  $4 \times 4$  or  $4 \times 8$  TPU v2 Pods, and they are evaluated using NVIDIA V100 GPUs, on the cloud.
B.2 Details of ID Test Sets
We construct an ID test set from YFCC-15M and LAION-15M, respectively, and we automatically generate classification labels by matching text with ImageNet class names, which has also been similarly performed in Fang et al. (2022) for training classifiers using caption data. On YFCC-15M which contains metadata, we use tags for matching. On LAION-15M which does not provide metadata such as tags, we simply use the entire text for matching. We adopt a label only if a unique ImageNet class can be determined by matching for the involved image. We then construct a balanced and labelled test set by keeping 50 examples for each class that has at least 100 examples in the labelled images. The test examples are then held out from the training data. For the YFCC test set, there are 22550 examples for 451 classes; and for the LAION test set, there are 20400 examples 408 classes.
B.3 Licenses of the Datasets
We have used the following datasets:
- CIFAR-10 (Krizhevsky et al., 2009). License is not clearly known.
- YFCC $^6$  under various Creative Commons licenses.
- LAION<sup>7</sup> under the Creative Common CC-BY 4.0 license for the metadata, and the images are under the copyright of the original authors.
CIFAR-10.18 under the MIT license.
- CIFAR-10.2 $^{9}$ . License is not clearly known.
- CINIC-  $10^{10}$  under the MIT license.
- ImageNet-V2 $^{11}$  under the MIT license.
- ImageNet- $\mathbf{R}^{12}$  under the MIT license.
- ImageNet-Sketch<sup>13</sup> under the MIT license.
- ObjectNet $^{14}$  with a license provided on the webpage.
$^{6}$ https://multimediacommons.wordpress.com/yfcc100m-core-dataset
<sup>7</sup>https://laion.ai/blog/laion-400-open-dataset
<sup>8</sup>https://github.com/modestyachts/CIFAR-10.1