SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results Zooming in consistently helps ResNet-50 networks, with improvements varying from +13 to +24 points. The best-performing network is torchvision-V2 which uses the max aggregator and achieves 29.65%. These results suggest that simple aggregation over the proposed zoom transform is effective for datasets that have dominant center bias.
Table A10: The results of different aggregation functions on four ResNet-50 variants when tested on ImageNet-A (\%). Each model has been trained using different training-time augmentation techniques. Improvements values in parentheses are with respect to the 1-crop baseline.
ResNet-50	Baseline	Max	Mean
torchvision V1	0.21	16.11 (+15.90)	6.23
MoEx+CutMix [40]	8.60	24.72 (+16.12)	15.32
DeepAug+AugMix [26]	3.94	27.93 (+23.99)	13.16
torchvision V2	16.62	29.65 (+13.03)	22.08
3ResNet50_Weights.IMAGENET1K_V2
4ResNet50_Weights.IMAGENET1K_V1
In this section, we provide several visualizations of zooming transforms.
D.1 Visualizations for 36 top performing zoom transforms
Figure A28: Different framing of an image of a lorikeet according to 36 high-performing transforms of a ResNet-50 model
nail 24.03
lorikeet 32.33
nail 18.32
lorikeet 85.56
lorikeet 84.60
brambling 20.48
lorikeet 84.89
junco 20.68
american chameleon 35.06
lorikeet 89.01
fox squirrel 4.87
matchstick 1.91
lorikeet 63.08
junco 13.49
african chameleon 22.09
lorikeet 84.10
hummingbird 4.90
fox squirrel 14.09
mantis 5.92
lorikeet 89.44
walking stick 42.57
brambling 7.15
lorikeet 66.01
lorikeet 83.27
jacamar 46.12
lorikeet 79.72
nail 26.69
lorikeet 53.44
lorikeet 40.54
american chameleon 7.24
Figure A29: Different framing of an image of a 1orikeet according to 36 high-performing transforms of a ViT/B-32 model
stethoscope 84.86
maraca 9.43
golden_retriever 25.41
golden_retriever 53.12
Chihuahua 83.43
stethoscope 39.02
Chihuahua 27.77
stethoscope 90.66
stethoscope 27.31
golden_retriever 63.55
hand_blower 46.10
golden_retriever 20.16
stethoscope 92.62
Chihuahua 45.02
hand_blower 24.99
stethoscope 74.40
stethoscope 52.77
Chihuahua 96.98
stethoscope 92.74
handblower 38.98
studio_couch 26.20
stethoscope 94.44
maraca 10.31
envelope 39.45
golden_retever 51.76
stethoscope 49.90
stethoscope 97.62
Chihuahua 38.00
stethoscope 94.04
stethoscope 64.81
Figure A30: Different framing of an image of a stethoscope according to 36 high-performing transforms of a ResNet-50 model
Figure A31: Different framing of an image of a stethoscope according to 36 high-performing transforms of a ViT/B-32 model
D.2 Overview of 324 transforms
The visualizations below illustrate the transforms that result in the correct prediction of the query image, using ViT-B/32 [17] and CLIP-ViT-L/14 [53]. Each circle represents a transform, with the initial zoom scale indicated in the accompanying text. The green circles represent the transformations that lead to correct classification, while the red circles indicate incorrect ones.
Figure A32: Visualization of effective transforms that lead to the correct classification of an image containing scorpion, using a ViT-B/32 model.
Figure A33: Visualization of effective transforms that lead to the correct classification of an image containing scorpion, using a CLIP-ViT-L/14 model.
Figure A34: Visualization of effective transforms that lead to the correct classification of an image containing bubble, using a ViT-B/32 model.
Figure A35: Visualization of effective transforms that lead to the correct classification of an image containing bubble, using a CLIP-ViT-L/14 model.
Figure A36: Visualization of effective transforms that lead to the correct classification of an image containing agama, using a ViT-B/32 model.
Figure A37: Visualization of effective transforms that lead to the correct classification of an image containing agama, using a CLIP-ViT-L/14 model.
Figure A38: Visualization of effective transforms that lead to the correct classification of an image containing acorn, using a ViT-B/32 model.
Figure A39: Visualization of effective transforms that lead to the correct classification of an image containing acorn, using a CLIP-ViT-L/14 model.
D.3 Only zoom-out solves
Sample images that required zooming out to be classified correctly.
D.3.1 ImageNet-Sketch
Figure A40: ImageNet-Sketch images that can only be solved using zoom-out. Predictions are from a ResNet-50 classifier.
D.3.2 ImageNet-R
Figure A41: ImageNet-R images that can only be solved using zoom-out. Predictions are from a ResNet-50 classifier.
D.3.3 ObjectNet
Figure A42: ObjectNet images that can only be solved using zoom-out. Predictions are from a ResNet-50 classifier.
D.4 Only zoom-in solves
Sample images that required zooming in to be classified correctly.
D.4.1 ObjectNet
Figure A43: ObjectNet images that can only be solved using zoom-in. Predictions are from a ResNet-50 classifier.
D.5 AugMix and RandomResizedCrop
Figure A44:  $K = 16$  sample outputs from AugMix [25] (which yields the results of random sampling from 13 transformations that include both spatial and color distortions).
Figure A45:  $K = 16$  sample outputs from RandomResizedCrop (RRC), which basically randomly zooms into an arbitrary region in the input image.
Figure A46:  $K = 16$  sample outputs from AugMix [25] (which yields the results of random sampling from 13 transformations that include both spatial and color distortions).
Figure A47:  $K = 16$  sample outputs from RandomResizedCrop (RRC), which basically randomly zooms into an arbitrary region in the input image.
E ImageNet-Hard
In this section, we provide details about the ImageNet-hard dataset.
E.1 Distribution
Figure A48: The distribution of the dataset within the ImageNet-Hard Dataset.
E.2 Samples images
Figure A49: Sample images from ImageNet-Hard dataset with groundtruth labels.
E.3 Analysis of wrong predictions
We used gpt-3.5-turbo to categorize each misprediction made by EfficientNet-L2 into two classes: plausible and implausible, based on the semantic distance between the groundtruth label and the predicted label.
Figure A50: Sample prompt and response of gpt-3.5-turbo for a plausible classification. The text in the Assistant block is the generated response.
Figure A51: Sample prompt and response of gpt-3.5-turbo for an implausible classification. The text in the Assistant block is the generated response.
E.4 Confusing classes
In this section, we present a selection of examples highlighting the errors made by our highest-performing model, EfficientNet-L2.
Figure A52: Images misclassified into coffemaker by EfficientNet-L2
Figure A53: Images misclassified into strainer by EfficientNet-L2
Figure A54: Images misclassified into space shuttle by EfficientNet-L2
Figure A55: Images misclassified into safety pin by EfficientNet-L2
E.5 Common and rare misclassification
This section shows some sample misclassification for EfficientNet-L2 and OpenCLIP's ViT-bigG-14 classifiers.
Figure A56: Examples of misclassifications by EfficientNet-L2 under the Common category.
Figure A57: Examples of misclassifications by EfficientNet-L2 under the Rare category.
Figure A58: Examples of misclassifications by OpenCLIP under the Common category.
Figure A59: Examples of misclassifications by OpenCLIP under the Rare category.
E.6 Evaluating OpenCLIP models' performance on ImageNet-Hard
All the models in this section are downloaded and used from the OpenCLIP library version 2.20.0.
Table A11: Zero-shot performance of OpenCLIP on ImageNet-Hard (%)
Model	Pre-trained Dataset	Top-1 Accuracy
RN50	yfcc15m	0.80
RN50	cc12m	1.18
RN50-quickgelu	yfcc15m	0.75
RN50-quickgelu	cc12m	1.08
RN101	yfcc15m	0.65
RN101-quickgelu	yfcc15m	0.62
ViT-B/32	laion400m_e31	5.34
ViT-B/32	laion400m_e32	5.41
ViT-B/32	laion2b_e16	5.66
ViT-B/32	laion2b_s34b_b79k	6.13
ViT-B/32	datacomp_m_s128m_b4k	2.79
ViT-B/32	commonpool_m Clip_s128m_b4k	2.50
ViT-B/32	commonpool_m laion_s128m_b4k	2.41
ViT-B/32	commonpool_m image_s128m_b4k	2.72
ViT-B/32	commonpool_m text_s128m_b4k	2.46
ViT-B/32	commonpool_m basic_s128m_b4k	2.23
ViT-B/32	commonpool_m s128m_b4k	1.73
ViT-B/32	datacomp_s_s13m_b4k	0.61
ViT-B/32	commonpool_s clip_s13m_b4k	0.84
ViT-B/32	commonpool_s laion_s13m_b4k	0.66
ViT-B/32	commonpool_s image_s13m_b4k	0.61
ViT-B/32	commonpool_s text_s13m_b4k	0.77
ViT-B/32	commonpool_s basic_s13m_b4k	0.75
ViT-B/32	commonpool_s s13m_b4k	0.43
ViT-B/32-quickgelu	laion400m_e31	5.34
ViT-B/32-quickgelu	laion400m_e32	5.28
ViT-B/16	laion400m_e31	6.31
ViT-B/16	laion400m_e32	6.46
ViT-B/16	laion2b_s34b_b88k	7.18
ViT-B/16	datacomp_l_s1b_b8k	5.98
ViT-B/16	commonpool_l clip_s1b_b8k	4.92
ViT-B/16	commonpool_l laion_s1b_b8k	4.44
ViT-B/16	commonpool_l image_s1b_b8k	4.75
ViT-B/16	commonpool_l text_s1b_b8k	5.63
ViT-B/16	commonpool_l basic_s1b_b8k	4.44
ViT-B/16	commonpool_l s1b_b8k	3.83
ViT-B/16-plus-240	laion400m_e31	6.65
ViT-B/16-plus-240	laion400m_e32	6.69
ViT-L/14	laion400m_e31	8.83
ViT-L/14	laion400m_e32	8.72
ViT-L/14	laion2b_s32b_b82k	10.13
ViT-L/14	datacomp_xl_s13b_b90k	15.60
ViT-L/14	commonpool_xl clip_s13b_b90k	11.58
ViT-L/14	commonpool_xl laion_s13b_b90k	11.42
ViT-L/14	commonpool_xl_s13b_b90k	12.44
ViT-H/14	laion2b_s32b_b79k	13.01
ViT-g/14	laion2b_s12b_b42k	11.47
ViT-g/14	laion2b_s34b_b88k	14.03
ViT-bigG-14	laion2b_s39b_b160k	15.93
roberta-ViT-B/32	laion2b_s12b_b32k	5.21
xlm-roberta-base-ViT-B/32	laion5b_s13b_b90k	5.72
xlm-roberta-large-ViT-H/14	frozen_laion5b_s13b_b90k	12.95
convnext_base	laion400m_s13b_b51k	4.74
convnext_base_w	laion2b_s13b_b82k	6.09
convnext_base_w	laion2b_s13b_b82k_augreg	7.25
convnext_base_w	laion_aesthetic_s13b_b82k	5.57
convnext_base_w_320	laion_aesthetic_s13b_b82k	5.50
convnext_base_w_320	laion_aesthetic_s13b_b82k_augreg	7.14
convnext_large_d	laion2b_s26b_b102k_augreg	10.39
convnext_large_d_320	laion2b_s29b_b131k_ft	10.69
convnext_large_d_320	laion2b_s29b_b131k_ft_soup	11.20
convnext_xxlarge	laion2b_s34b_b82k_augreg	14.27
convnext_xxlarge	laion2b_s34b_b82k_augregRewind	14.23
convnext_xxlarge	laion2b_s34b_b82k_augreg_soup	14.68
coca_ViT-B/32	laion2b_s13b_b90k	5.83
coca_ViT-B/32	mscoco_finetuned_laion2b_s13b_b90k	0.20
coca_ViT-L/14	laion2b_s13b_b90k	10.79
coca_ViT-L/14	mscoco_finetuned_laion2b_s13b_b90k	9.28
Table A12: Zero-shot performance of CommonPool and DataComp models on ImageNet-Hard (%)
Scale	Model	Pretrained	Top-1 Accuracy
xlarge	ViT-L/14	datacomp_xl_s13b_b90k	15.60
ViT-L/14	commonpool_xl Clip_s13b_b90k	11.58
ViT-L/14	commonpool_xl laion_s13b_b90k	11.42
ViT-L/14	commonpool_xl_s13b_b90k	12.44
large	ViT-B/16	datacomp_l_s1b_b8k	5.98
ViT-B/16	commonpool_l Clip_s1b_b8k	4.92
ViT-B/16	commonpool_l laion_s1b_b8k	4.44
ViT-B/16	commonpool_l image_s1b_b8k	4.75
ViT-B/16	commonpool_l text_s1b_b8k	5.63
ViT-B/16	commonpool_l basic_s1b_b8k	4.44
ViT-B/16	commonpool_l s1b_b8k	3.83
medium	ViT-B/32	datacomp_m_s128m_b4k	2.79
ViT-B/32	commonpool_m Clip_s128m_b4k	2.50
ViT-B/32	commonpool_m laion_s128m_b4k	2.41
ViT-B/32	commonpool_m image_s128m_b4k	2.72
ViT-B/32	commonpool_m text_s128m_b4k	2.46
ViT-B/32	commonpool_m basic_s128m_b4k	2.23
ViT-B/32	commonpool_m s128m_b4k	1.73
small	ViT-B/32	datacomp_s_s13m_b4k	0.61
ViT-B/32	commonpool_s clip_s13m_b4k	0.84
ViT-B/32	commonpool_s laion_s13m_b4k	0.66
ViT-B/32	commonpool_s image_s13m_b4k	0.61
ViT-B/32	commonpool_s text_s13m_b4k	0.77
ViT-B/32	commonpool_s basic_s13m_b4k	0.75
ViT-B/32	commonpool_s s13m_b4k	0.43
E.7 Evaluating classifiers on ImageNet-Hard-4K
Table A13: Top-1 accuracy (%) on ImageNet-Hard-4K. Most models obtain a lower accuracy compared to their corresponding accuracy on ImageNet-Hard.
Classifier	Accuracy	Classifier	Accuracy	Classifier	Accuracy
AlexNet	7.08 (-0.16)	ViT-B/32	18.12 (-0.40)	CLIP-ViT-L/14@224px	1.81 (-0.05)
VGG-16	11.32 (-0.68)	EfficientNet-B0@224px	12.94 (-3.63)	CLIP-ViT-L/14@336px	1.88 (-0.14)
ResNet-18	10.42 (-0.44)	EfficientNet-B7@600px	18.67 (-4.53)	OpenCLIP-ViT-bigG-14	14.33 (-1.60)
ResNet-50	13.93 (-0.81)	EfficientNet-L2@800px	28.42 (-10.58)	OpenCLIP-ViT-L-14	13.04 (-2.56)
E.8 Obviously ill-posed samples from ImageNet-Sketch
Figure A60: Sample images from ImageNet-Sketch that are completely black.
F Additional Details
F.1 Additional results for performance of classifiers using maximum possible accuracy
In this section, we present the delta values for Tab. 1, which represent the difference in relation to the 1-crop accuracy for each cell.
Table A14: On in-distribution data (IN & ReaL) there exists a substantial improvement when models are provided with an optimal zoom, either selected from 36 (b) or 324 pre-defined zoom crops (c). In contrast, OOD benchmarks still pose a significant challenge to IN-trained models even with optimal zooming (i.e., all upper-bound accuracy scores  $< 80\%$ ). Improvements are respected to the standard 1-crop accuracy.
IN	RealL	IN+RealL	IN-A	IN-R	IN-S	ON
(a) Standard top-1 accuracy based on N = 1 crop
AlexNet	56.16	62.67	61.76	1.75	21.10	10.05	14.23
VGG-16	71.37	78.90	78.52	2.69	26.98	16.78	28.32
ResNet-18	69.45	76.94	76.47	1.37	32.14	19.41	27.59
ResNet-50	75.75	82.63	82.97	0.21	35.39	22.91	36.18
ViT-B/32	75.75	81.89	82.59	9.64	41.29	26.83	30.89
CLIP-ViT-L/14	75.03	80.68	81.95	71.28	87.74	58.23	66.32
(b) Upper-bound accuracy using N = 36 crops
AlexNet	85.19 (+29.03)	90.30 (+27.63)	89.74 (+27.98)	31.37 (+29.62)	47.04 (+25.94)	24.40 (+14.35)	49.17 (+34.94)
VGG-16	92.30 (+20.93)	96.08 (+17.18)	95.81 (+17.29)	46.69 (+44.00)	52.86 (+25.88)	34.34 (+17.56)	62.94 (+34.62)
ResNet-18	92.08 (+22.63)	95.97 (+19.03)	95.73 (+19.26)	47.48 (+46.11)	58.85 (+26.71)	37.91 (+18.50)	63.08 (+35.49)
ResNet-50	94.46 (+18.71)	97.36 (+14.73)	97.40 (+14.43)	55.68 (+55.47)	61.42 (+26.03)	41.71 (+18.80)	69.60 (+33.42)
ViT-B/32	95.05 (+19.30)	97.61 (+15.72)	97.88 (+15.29)	68.43 (+58.79)	68.77 (+27.48)	49.10 (+22.27)	70.30 (+39.41)
CLIP-ViT-L/14	94.19 (+19.16)	97.32 (+16.64)	97.56 (+15.61)	97.16 (+25.88)	98.60 (+10.86)	83.77 (+25.54)	89.59 (+23.27)
(c) Upper-bound accuracy using N = 324 crops
AlexNet	90.03 (+33.87)	93.85 (+31.18)	93.48 (+31.72)	42.23 (+40.48)	55.52 (+34.42)	29.53 (+19.48)	59.65 (+45.42)
VGG-16	95.30 (+23.93)	97.90 (+19.00)	97.66 (+19.14)	58.27 (+55.58)	60.88 (+33.90)	39.90 (+23.12)	71.85 (+43.53)
ResNet-18	95.15 (+25.70)	97.76 (+20.82)	97.55 (+21.08)	58.87 (+57.50)	66.89 (+34.75)	43.68 (+24.27)	71.44 (+43.85)
ResNet-50	96.78 (+21.03)	98.62 (+15.99)	98.57 (+15.60)	66.68 (+66.47)	68.84 (+33.45)	47.64 (+24.73)	76.83 (+40.65)
ViT-B/32	97.19 (+21.44)	98.75 (+16.86)	98.91 (+16.32)	78.03 (+68.39)	75.58 (+34.29)	55.99 (+29.16)	79.28 (+48.39)
CLIP-ViT-L/14	96.78 (+21.75)	98.69 (+18.01)	98.80 (+16.85)	98.45 (+27.17)	99.20 (+11.46)	89.00 (+30.77)	93.13 (+26.81)
F.2 Comparing modern architectures with regard to their maximum possible accuracy.
In this section, we repeated the experiment presented in Tab. 1 in order to include a broader range of architectures and conduct a rank analysis. Specifically, we added MaxViT [72], Swin Transformer [43], ConvNext [44], EfficientNet [69], and MobileNetV3 [28].
Table A15: Repeating the experiment in Tab. 1 for various architectures, please note that the small variations of ResNet-50 and ViT-B/32 models arise from the use of different CUDA and PyTorch versions.
(a) Standard top-1 accuracy based on N = 1 crop
Classifier	ImageNet	ImageNet-ReAL	ImageNet-A
MobileNetV3-Small	67.42	74.47	2.73
ResNet-50	75.74	82.63	0.17
EfficientNet-B0	77.69	84.13	7.01
ViT-B/32	75.75	81.89	9.51
ConvNext-Base	83.49	88.19	33.99
Swin-B	83.37	87.80	34.87
MaxViT-T	83.49	88.05	36.00
(b) Upper-bound accuracy using N = 36 crops
MobileNetV3-Small	91.79	95.63	44.45
ResNet-50	94.47	97.35	54.97
EfficientNet-B0	94.87	97.49	59.33
ViT-B/32	95.04	97.60	68.33
ConvNext-Base	95.78	97.68	82.25
Swin-B	95.95	97.75	82.81
MaxViT-T	96.11	97.99	83.33
(c) Upper-bound accuracy using N = 324 crops
MobileNetV3-Small	95.20	97.66	57.53
ResNet-50	96.77	98.63	66.15
EfficientNet-B0	96.87	98.60	69.20
ViT-B/32	97.19	98.75	77.77
ConvNext-Base	97.16	98.53	87.16
Swin-B	97.39	98.69	88.15
MaxViT-T	97.45	98.83	88.24
(a) ImageNet
(b) ImageNet-ReaL
(c) ImageNet-A
Figure A61: Comparing modern architectures using maximum possible accuracy
Figure A62: Average rank analysis of various architectures on ImageNet, ImageNet-ReaL, and ImageNet-A using different metrics. Max-ViT consistently emerges as the top performer across all experiments. Despite its high 1-crop accuracy, ConvNext's rank declined when more crops were included. Conversely, ViT-B/32 exhibited a reverse trend; while its 1-crop rank was not robust, the inclusion of more crops elevated its accuracy beyond that of ConvNext.
F.3 Error-analysis for MEMO results
In this section, we repeated the MEMO experiment using various random seeds to evaluate the consistency of the results. Our findings demonstrate a consistent trend between MEMO and RRC, indicating that this relationship holds steady regardless of variations in individual runs.
(a) ResNet-50 [23]
(b) DeepAug+AugMix [26]
(c) MoEx+CutMix [40]
Figure A63: Error bars representing the outcomes of the MEMO experiment on ImageNet, conducted with three distinct random seeds.
(c) MoEx+CutMix [40]
Figure A64: Error bars representing the outcomes of the MEMO experiment on ImageNet-A, conducted with three distinct random seeds.
F.4 Grad-CAM visualizations for MEMO + RRC
chameleon
Figure A65: Additional Grad-CAM visualization for the final convolutional layer of a ResNet-50 before and after MEMO + RRC update.
stingray
chameleon
small white
Figure A66: Additional Grad-CAM visualization for the final convolutional layer of a ResNet-50 before and after MEMO + RRC update.
G Datasheet for ImageNet-Hard
G.1 Motivation
The questions in this section are primarily intended to encourage dataset creators to clearly articulate their reasons for creating the dataset and to promote transparency about funding interests. The latter may be particularly relevant for datasets created for research purposes.
- For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.
The ImageNet-Hard is a new benchmark to test the robustness of state-of-the-art image classifiers. It comprises an array of challenging images collected from six validation datasets of ImageNet. This dataset challenges state-of-the-art image classification models because even by perfectly localizing the key objects, the state-of-the-art classifiers still fail to correctly recognize.
- Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
The dataset was created in collaboration efforts between the University of Alberta, Canada, and Auburn University, USA; mostly by, Mohammad Reza Taesiri and Anh Nguyen.
- Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
Anh Nguyen was supported by NSF Grant No. 2145767, and donations from NaphCare Foundation, and Adobe Research
- Any other comments?
G.2 Composition
Dataset creators should read through these questions prior to any data collection and then provide answers once data collection is complete. Most of the questions in this section are intended to provide dataset consumers with the information they need to make informed decisions about using the dataset for their chosen tasks. Some of the questions are designed to elicit information about compliance with the EU's General Data Protection Regulation (GDPR) or comparable regulations in other jurisdictions.
Questions that apply only to datasets that relate to people are grouped together at the end of the section. We recommend taking a broad interpretation of whether a dataset relates to people. For example, any dataset containing text that was written by people relates to people.
- What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
Each instance of the ImageNet-Hard dataset corresponds to an image and at least one groundtruth label that will be used to assess image classifiers.
- How many instances are there in total (of each type, if appropriate)?
There are 10,980 images in this dataset.
- Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
ImageNet-Hard is a combination of various publicly available datasets. We tried multiple refinement steps to make sure to get the best possible samples for the intended purpose. Then, it is not representative of any larger sets but a selective combination of multiple sets.
- What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) or features? In either case, please provide a description.
The dataset contains both raw and processed images. The processed images come from ImageNet-C. Details can be found in Sec. 4.4.
- Is there a label or target associated with each instance? If so, please provide a description. Yes. Each sample has the label that is the folder name the image belongs to. Basically, we follow the structure of the ImageNet paper [59].
- Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text. No.
- Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit. No. The individual instances has no relationships.
- Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them. No. This dataset is created for the testing purposes.
- Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description. To the best of our knowledge, No. We tried our best efforts to filter any errors, sources of noise, or redundancies to create the ImageNet-Hard dataset.
- Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
Yes. It does link and inherits from existing image datasets and was detailed in Sec. 4.4.
- Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? If so, please provide a description. No.
- Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why. No.
If the dataset does not relate to people, you may skip the remaining questions in this section.