SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On average, over all three models and five datasets, our RRC outperforms AugMix by  $+0.28$  points, with a larger impact on IN-A, where it achieves a mean improvement of  $+1.10$  points (Tab. 2). Our results show that zoom-in alone can be a useful inductive bias, helping improve downstream image classification. In contrast, some of the transformations among the 13 transform functions in AugMix may not be essential to the results of Zhang et al. [83] (no ablation studies of transformations were provided in [83]) and are less effective than our zoom-in.
Table 2: MEMO + RRC (i.e. random zoom-in transforms) outperforms baselines and MEMO [83].
Baseline (1-crop)	MEMO + AugMix [83]	MEMO + RRC (Ours)
IN	IN-A	IN-R	IN-S	ON	IN	IN-A	IN-R	IN-S	ON	IN	IN-A	IN-R	IN-S	ON
ResNet-50 [23]	76.13	0.00	36.17	24.09	35.92	77.27	0.83	41.28	27.63	38.38	77.50	1.31	40.81	27.53	38.85
DeepAug+AugMix [26]	75.82	3.87	46.77	32.62	34.81	76.27	5.35	50.79	35.70	36.42	76.38	5.76	50.88	35.65	36.64
MoEx+CutMix [40]	79.04	7.97	35.52	23.96	38.59	79.38	11.21	40.65	27.07	40.62	79.49	13.61	40.41	26.80	41.43
mean ± std	36.75 ± 24.75	39.26 (+2.51) ± 24.32	39.54 (+2.79) ± 24.10
Figure 4 shows Grad-CAM visualizations for three samples, providing evidence of how the network's behavior changes before and after the MEMO update. For an image of a pig, the network initially focused on a kitchen appliance, failing to detect the object correctly. After applying the MEMO modification, it refocused on the dog, classifying it accurately. Similarly, in an image of a fox squirrel, the network initially had a diffuse focus but refocused on the fox squirrel after the update. These results demonstrate the effectiveness of the MEMO modification in guiding the network's attention or encouraging the model to perform an implicit zoom on the regions of interest, thereby improving its classification performance.
Figure 4: Grad-CAM for the activation of the last convolutional layer of a ResNet-50 before and after the MEMO update suggests that the network attends to the object of interest after the update.
4.4 ImageNet-Hard: A benchmark with images that remain unclassifiable, even after 324 zoom attempts
Existing ImageNet-scale benchmarks followed one of the following three construction methods: (1) perturbing real images with the aim of making them harder for models to classify (e.g., ImageNet-C [24] and DImageNet [13]); (2) collecting the real images that models misclassify (e.g., IN-A, ImageNet-O [27]); or (3) setting up a highly-controlled data collection process (e.g., IN-S and ON). Yet, none of such benchmarks explicitly challenge models on the ability to recognize a well-framed object in an image (i.e., no zooming required). For example, ON is supposed to test the recognition of objects in unusual poses but the cluttered background in ON images is actually a major reason for misclassification (Sec. 4.2). Furthermore, the results in Tab. 1 suggest that given optimal zooming, these existing benchmarks only challenge IN-trained models but not the Internet-scale vision-language models (e.g. CLIP) anymore. We propose ImageNet-Hard, a novel ImageNet-scale benchmark that challenges existing and future SOTA models. ImageNet-Hard is a collection of images that the SOTA CLIP-ViT-L/14 fails to correctly classify even when 324 zooming attempts are provided.
4.4.1 ImageNet-Hard construction
Initial data collection We take CLIP-ViT-L/14 (the highest-performing model in Tab. 1) and run the zooming procedure to find "Unclassifiable images" (defined in Sec. 4.1) from the following six datasets: IN-V2 [56], IN-ReaL, IN-A, IN-R, IN-S, and ON. That is, for each image  $x$ , we generate 324 zoomed versions of  $x$  and feed them into CLIP-ViT-L/14. We add  $x$  to ImageNet-Hard only if none of the 324 versions are correctly classified.
Adding ImageNet-C The original IN-C [24] are the original IN images but center-cropped to 224  $\times$  224 px, which significantly makes the classification task unnecessarily more ill-posed (e.g., by adding Gaussian noise to a crop where the main object is already removed).
To find a subset of IN-C images for adding into ImageNet-Hard, we first re-generate ImageNet-C by adding the 19 types of corruption noise to IN without resizing the original IN images. Second, we run CLIP-ViT-L/14 on all 19 corruption types and manually select a subset of six diverse and lowest-accuracy corruption groups: Impulse Noise, Frost, Fog, Snow, Brightness, and Zoom Blur. We repeat the initial data collection process for these 6 image sets of IN-C.
Groundtruth labels After the above procedure, our dataset contains 13,925 images collected from IN+ReaL, IN-V2, IN-A, IN-C, IN-R, IN-S, and ON (see the distribution in Appendix E.1). ImageNet-Hard presents a 1000-way classification task where the 1000 classes are from ImageNet. We manually inspect all images and remove 295 samples that are obviously ill-posed (e.g. an entirely black image but labeled great white shark in IN-S Fig. A60), arriving at a total of 13,630 ImageNet-Hard images. A sample contains only one groundtruth label from its original datasets except for IN and IN-C images, which have a set of IN+ReaL labels. Each IN or IN-C image is considered correctly labeled by a model if its top-1 predicted label is among the groundtruth labels.
Refining groundtruth labels via human feedback Label noise is still present in IN and OOD benchmarks despite cleaning efforts [10, 56, 81]. Since ImageNet-Hard contains images misclassified by CLIP-ViT-L/14, our manual inspection confirms many misclassified images have debatable labels.
To ameliorate the issue, we orchestrate a human feedback study for eliminating images with inaccurate labels. First, the first author examine every image and flag 3,133 images as ambiguous and needs verification. Then, we have two groups of annotators to help verify the labels (by choosing Accept, Reject, or Not Sure). Group A is composed of three students, each examine all 3,133 images where Group B is composed of 38 students, each examine 50 randomly-selected images. Our inter-annotator aggregation procedure merges labels from both groups and results in 2,280 images removed (out of 3,133 originally flagged), leaving ImageNet-Hard at a total of 11,350 images.
That is, we accept an image  $x$  if one of the two conditions is satisfied: (1) when all 3/3 group-A annotators accept  $x$ ; or (2) when 2/3 group-A annotators accept  $x$  and all group-B reviewers of  $x$  accept  $x$  (assuming at least 1 group-B annotator reviews  $x$ ; otherwise  $x$  will be rejected).
Inspired by IN-ReaL [10], we further clean up the labels by eliminating 370 images associated with the labels sunglass, sunglasses, tub, bathtub, cradle, bassinet, projectile, and missile, i.e., the classes that often contain similar images that belong to more than one class. After this refinement, the final ImageNet-Hard dataset contains a total of 10,980 images.
4K version We utilize GigaGAN [33] to upscale every image in our final dataset and construct ImageNet-Hard-4K, which is aimed to facilitate future research into how a super-resolution step may improve image classification results (e.g., to classify an object when the image is blurry).
Release ImageNet-Hard and ImageNet-Hard-4K are released on HuggingFace (see samples in Fig. A49) under MIT License. Code for evaluating models on ImageNet-Hard is on GitHub.
4.4.2 ImageNet-Hard challenges SOTA classifiers, especially those operating at  $224 \times 224$
Here, we evaluate the standard 1-crop, top-1 accuracy of SOTA classifiers on ImageNet-Hard. We use the image pre-processing function defined by each classifier. In addition to the 6 models in Sec. 4.1, we also test CLIP-ViT-L/14@336px [53], EfficientNet (B0@224px and B7@600px) [69], and EfficientNet-L2@800px [1]. CLIP-ViT-L/14@336px, EfficientNet-B7@600px, and EfficientNet-L2@800px are state-of-the-art models that operate at high resolutions of  $336 \times 336$ ,  $600 \times 600$ , and  $800 \times 800$  respectively. In addition, our evaluation includes models from the OpenCLIP family [30].
Results Tab. 3 shows fairly low top-1 accuracy by various classifiers on ImageNet-Hard. First, all well-known IN-trained classifiers that operate at  $224 \times 224$  perform poorly between  $7.34\%$  (AlexNet) and  $18.52\%$  accuracy (ViT-B/32).
Since ImageNet-Hard is based on a collection of images that OpenAI's CLIP ViT-L/14@224px mislabels, this classifier's accuracy on our dataset is only  $1.86\%$ . Yet, interestingly, CLIP-ViT-L/14@336px also performs poorly at  $2.02\%$  (Tab. 3). Furthermore, all 68 tested OpenCLIP models perform poorly, with an accuracy below  $16\%$  (see details in Appendix E.6).
Separately, we observe a trend that models operating at a higher resolution tend to perform better on ImageNet-Hard with EfficientNet-L2@800px scoring highest at  $39.00\%$  (compared to  $88.40\%$  [79] on the original ImageNet). Overall, all models perform substantially worse on ImageNet-Hard (Tab. 3) than on other ImageNet-scale datasets (see Tab. A6; 1-crop). This result is expected because ImageNet-Hard is a set of hard cases collected from those OOD benchmarks.
ImageNet-Hard-4K We find that when upsampling images to 4K using GigaGAN [33] and downsampling them back to the resolution of each classifier does not help but even hurt the accuracy slightly (Tab. A13). Given that GigaGAN performs remarkably well, this result suggests ImageNet-Hard is different from typical fine-grained animal classification where improving the texture details increases classification accuracy [76]. The next section (Sec. 4.5) sheds light on model failures on ImageNet-Hard, revealing challenges posed to future SOTA models.
Table 3: Top-1 accuracy (\%) on ImageNet-Hard of IN-trained models and those trained on larger, non-ImageNet datasets (black). All models operate at  $224 \times 224$  unless otherwise specify.
Classifier	Accuracy	Classifier	Accuracy	Classifier	Accuracy
AlexNet	7.34	ViT-B/32	18.52	CLIP-ViT-L/14@224px	1.86
VGG-16	12.00	EfficientNet-B0@224px	16.57	CLIP-ViT-L/14@336px	2.02
ResNet-18	10.86	EfficientNet-B7@600px	23.20	OpenCLIP-ViT-bigG-14	15.93
ResNet-50	14.74	EfficientNet-L2@800px	39.00	OpenCLIP-ViT-L-14	15.60
4.5 Analysis of Image Classification Errors
Motivated by the fact that EfficientNet-L2 is the best classifier on ImageNet-Hard, we qualitatively analyze its failure cases to characterize the challenge posed by our benchmark. Specifically, we provide gpt-3.5-turbo [51] with a pair of EfficientNet-L2's top-1 (incorrect) label and the groundtruth label and ask it to categorize the error into "common" or "rare" based on the labels' semantic similarity (see Appendix E.3 for full details). For instance, mislabeling bucket into barrel is common (as two objects are quite related) while mislabeling cloak into jigsaw puzzle is rare.
(a) Common
(b) Rare
Figure 5: ImageNet-Hard samples misclassified by EfficientNet-L2@800px can be categorized into two groups: (a) Common: the top-1 label is related to the groundtruth label; and (b) Rare: the top-1 label is semantically far from the groundtruth label. See Figs. A56 and A57 for more samples.
Results See Appendix E.4 for samples of wrong labels that EfficientNet-L2 most frequently misclassifies into. We find that  $39.4\%$  of EfficientNet-L2's misclassifications on the ImageNet-Hard dataset are "common", while  $60.6\%$  are "rare".
A. Common group captures model confusion between two related classes (e.g. two fish species: clownfish and rock beauty; Fig. 5a). Yet, another source of problem for these "errors" is the debatable groundtruth labels, which may require domain-expert annotators to verify and rectify [45].
B. Rare group captures errors where the model confusion is between two semantically distant classes (e.g.,  $11\lambda \mathrm{ma}\rightarrow$  plectrum; Fig. 5b). This often happens with abstract images or objects in unusual
poses [5] or forms [15]. Classifying this group of images is challenging and sometimes requires a strong understanding of context and reasoning capabilities.
Limitations By manual inspection, we estimate  $14.7\%$  of labeling noise, which ImageNet-Hard inherits from the source datasets.
Our study rigorously analyzed the zooming effect on six known classifiers and image classification benchmarks. We first demonstrate that previous state-of-the-art classifiers, as old as AlexNet [38], could potentially achieve near  $90\%$  accuracy with optimal zooming. This sparks the intriguing question of whether image classifiers' evolution over the past ten years is about mastering where and at what scale to zoom (instead of enhancing feature extractors, a.k.a. representation learning [3]). Through another lens, we probe the evolution by analyzing the implicit zooming mechanisms that deep classifiers apply to input images. This perspective diverges from [54], which studied the progression of representation learning from CNNs to ViTs.
We are the first to document the spatial biases of existing benchmarks. Notably, IN-A and ON contain a large center bias and simply zooming to the center will de-clutter the scene and yield a high accuracy (24.69% for ViT-B/32 on IN-A; Tab. A5), which is competitive with state-of-the-art trained models (e.g. 24.1% of Robust ViT [11]) and much higher than state-of-the-art TTA techniques (e.g. 11.21% of MEMO [83]; Tab. 2). Our simple, but strong zoom-in baselines on IN-A and ON motivate future research into better-controlled benchmarks that more explicitly test models on a set of pre-defined properties. Our proposed TTA method with zoom-in transforms (MEMO + RRC) is not only more accurate but also more interpretable and faster to run (Tab. A7) than the original MEMO.
Finally, we introduce ImageNet-Hard (Sec. 4.4), a new challenging dataset for SOTA IN-trained and vision-language classifiers.
GN is supported by Auburn University PGRF Fellowship. AN was supported by a NSF CAREER award No. 2145767, and donations from NaphCare Foundation, and Adobe Research. We greatly appreciate David Seunghyun Yoon's help in generating the ImageNet-Hard-4K (Sec. 4.4.2) version using GigaGAN. We also thank those students at Auburn University and Alberta University who participated in our experiment for cleaning up the labels of ImageNet-Hard images.
[81] Yun, S., Oh, S. J., Heo, B., Han, D., Choe, J., and Chun, S. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2340-2350, 2021. 8
[82] Zagoruyko, S. and Komodakis, N. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016. 27
[83] Zhang, M., Levine, S., and Finn, C. Memo: Test time robustness via adaptation and augmentation. arXiv preprint arXiv:2110.09506, 2021. 2, 3, 6, 7, 10, 27
[84] Zheng, H., Fu, J., Zha, Z.-J., and Luo, J. Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5012-5021, 2019. 2, 3
[85] Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16793-16803, 2022. 6
[86] Zhu, Z., Xie, L., and Yuille, A. L. Object recognition with and without objects. arXiv preprint arXiv:1611.06596, 2016. 2, 34
Appendix for: ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification
In this section, we provide a detailed description of our experimental setup, including the Python code for our zoom transform, the classifiers we employed, and the setup we used for zero-shot classification.
A.1 Sample Python code for zoom-based transform
from PIL import Image   
import torchvisiontransformsfunctional as fv   
import torchvisiontransforms as transforms   
from functools import partial   
def crop_at(size, slice_x, slice_y): def slicecrop(image,size,slice_x,slice_y): width,height  $=$  image.size tile_size_x  $=$  width//3 tile_size_y  $=$  height//3 anchor_x  $=$  (slice_y \*tile_size_x)  $^+$  (tile_size_x // 2) anchor_y  $=$  (slice_x \*tile_size_y)  $^+$  (tile_size_y // 2) return fv.crop( image, anchor_y-(size//2), anchor_x-(size//2), size, size,) return partial(sliceCrop,size  $\equiv$  size,slice_x,slice_y  $=$  slice_y)   
zoom_scale  $= 255$    
zoom_transform  $=$  transformsCompose( [ transforms Resize( zoom_scale, interpolation  $\equiv$  transforms.InterpolationMode .BICUBIC, max_size  $\equiv$  None, antialias  $\equiv$  None, ), crop_at(224,i,j), ]
Figure A1: Sample python code.
A.2 Datasets' licenses
Dataset Name	License
ImageNet	Custom license, non-commercial
ImageNet-A	License
ImageNet-R	MIT License
ImageNet-Sketch	MIT License
ImageNet-C	MIT License
ObjectNet	Custom license derived from Creative Commons Attribution 4.0
ImageNet-V2	MIT License
Table A1: Dataset Licenses
A.3 Zoom Scales used
In our experiments, we tried the following zoom scales:
10, 16, 32, 48, 64, 96, 122, 128, 192, 224, 235, 240, 256, 288, 320, 348, 384, 448, 460, 512, 573, 576, 640, 664, 672, 680, 686, 690, 700, 720, 768, 798, 832, 896, 911, 1024.
A.4 Model selection
We use the official OpenAI's official CLIP for all CLIP-related experiments. All IN-trained models are retrieved from the torchvision [48] library. For models from the OpenCLIP family, we utilize the OpenCLIP library version 2.20.0. In the case of the EfficientNet-B family, we use the Hugging Face Transformers library. Lastly, for EfficientNet-L2, we use the implementation from the timm library.
A.5 Zero-shot classification using CLIP
For CLIP, we follow the standard zero-shot classification. This involves creating a text template for each class in the dataset, which contains a generic description of an image featuring an object from that class. Then, we use CLIP's text encoder to obtain embeddings for these templates and then average them to obtain a final vector that represents the class. To classify an image, we calculate the cosine similarity between its embedding and the text vectors for each class and then select the class with the highest value.
A.6 Zoom-based transform
(b)
Figure A2: (a) Making a 3-by-3 uniform grid out of the image. We pick the center point in each region as the anchor. (b) Sample image showing how our zoom transform is applied to an image.
In this section, we provide additional results for our experiments.
B.1 Zooming out is needed for a small portion of the datasets
In our approach, we leverage the power of both zoom-in and zoom-out transforms, and Tab. 1 results indicate that this combined zooming approach can be effective in classifying images from diverse datasets. Zooming in enhances texture patterns while zooming out provides a better perspective of the object's shape. The question we aim to answer is which dataset and model pairs require which type of zoom, and whether zooming is always necessary. Additionally, we investigate which types of networks are less reliant on explicit zooming, as they implicitly focus on the main object in the image.
Experiment We separate zoom transforms into three groups and report the maximum possible accuracy as defined in Sec. 3. We use transforms in the minimum set covers (as shown in Fig. A10) for each dataset and classifier pair. We then report the number of images that can only be classified using transforms in each group separately.
Figure A3: A sample image from the ImageNet-Sketch dataset that can only be solved by zooming out. For this image, with the standard ImageNet transform, the entire body of the animal is not visible. Instead, zooming out of the image helps you see the whole body of the animal. More samples can be found in Appendix D.3.
Results In general, we find that zooming in is more effective than zooming out. Zooming in provides two benefits: (1) it helps the model to focus on the key region where the target object is located, and (2) the model can extract features from the target object at a higher resolution. Across all methods and datasets, we can see a certain percentage of images are only classifiable using transforms of the zoom-out group. In particular, for ImageNet-R and ImageNet-Sketch, between  $1.2\% - 3\%$  (Table A2) of the entire dataset can only be solved using a transform in the zoom-out group. This is especially true for drawings, where the texture may lack distinguishable features, and zooming out allows us to better perceive the shape.
Table A2: Breakdown of maximum possible accuracy by different zoom groups. In each dataset, certain images necessitate a specific zoom group for correct classification regardless of the model being used. However, CLIP performs well overall without depending heavily on a particular zoom level. On average, the percentage of datasets that can only be solved with a specific zoom group is very small for this model.
Dataset	Model	zoom-in Solve	zoom-out Solves	zoom-224 Solves	Only zoom-in Solves	Only zoom-out Solves	Only zoom-224 Solves
ImageNet	ResNet-18	94.57	79.49	81.16	10.59	0.43	0.08
ResNet-50	96.30	85.84	86.39	7.59	0.40	0.04
ViT-B/32	96.83	86.18	85.12	7.59	0.30	0.02
VGG-16	94.60	82.11	83.08	8.92	0.58	0.07
AlexNet	89.17	62.92	67.98	18.01	0.65	0.18
CLIP-ViT-L/14	95.82	90.80	87.04	4.81	0.83	0.05
ImageNet RealL	ResNet-18	97.37	86.10	87.62	7.38	0.27	0.07
ResNet-50	98.22	91.07	91.87	4.65	0.25	0.04
ViT-B/32	98.50	90.79	88.06	4.92	0.18	0.03
VGG-16	97.38	88.43	89.40	6.02	0.38	0.07
AlexNet	93.15	69.58	74.85	15.47	0.45	0.19
CLIP-ViT-L/14	98.05	94.44	91.69	3.20	0.55	0.04
ImageNet+RealL	ResNet-18	97.16	85.51	86.77	7.72	0.28	0.05
ResNet-50	98.25	91.10	91.77	4.60	0.24	0.03
ViT-B/32	98.70	91.00	90.95	4.92	0.14	0.02
VGG-16	97.12	87.88	89.09	6.25	0.42	0.06
AlexNet	92.79	68.65	73.93	16.25	0.47	0.16
CLIP-ViT-L/14	98.24	95.09	92.41	2.75	0.47	0.04
ImageNet-A	ResNet-18	63.66	47.95	45.37	13.97	2.75	0.21
ResNet-50	65.28	52.36	48.59	12.05	3.13	0.22
ViT-B/32	73.07	56.34	54.84	14.20	2.04	0.27
VGG-16	56.67	44.95	39.35	11.80	3.85	0.24
AlexNet	52.69	32.86	31.95	17.15	2.34	0.30
CLIP-ViT-L/14	98.35	96.71	93.57	1.70	0.69	0.04
ImageNet-R	ResNet-18	57.07	12.19	10.07	40.67	0.92	0.19
ResNet-50	64.52	12.95	10.36	48.72	1.00	0.23
ViT-B/32	76.71	18.57	21.92	51.75	0.85	0.15
VGG-16	56.59	13.15	13.27	38.24	0.93	0.29
AlexNet	39.91	10.39	9.11	26.27	1.08	0.36
CLIP-ViT-L/14	97.99	81.32	77.03	12.01	0.44	0.05
ImageNet-Sketch	ResNet-18	41.14	27.06	27.41	11.83	1.77	0.36
ResNet-50	44.72	32.80	31.45	10.99	2.23	0.24
ViT-B/32	53.45	37.43	37.38	13.11	1.83	0.36
VGG-16	36.20	27.20	24.59	9.47	2.97	0.28
AlexNet	27.71	13.84	15.11	11.26	1.22	0.33
CLIP-ViT-L/14	86.20	80.67	73.94	6.64	2.38	0.12
ObjectNet	ResNet-18	68.98	38.52	37.23	25.76	1.93	0.25
ResNet-50	74.16	51.56	47.79	19.68	2.16	0.30
ViT-B/32	77.66	44.49	42.65	27.43	1.34	0.20
VGG-16	69.19	41.72	39.49	23.34	2.27	0.31
AlexNet	56.76	23.45	22.59	28.85	2.27	0.33
CLIP-ViT-L/14	91.28	82.22	77.60	8.37	1.38	0.15
Average	ResNet-18	74.28	53.83	53.66	16.85	1.19	0.17
ResNet-50	77.35	59.67	58.32	15.47	1.34	0.16
ViT-B/32	82.13	60.69	60.13	17.70	0.95	0.15
VGG-16	72.54	55.06	54.04	14.86	1.63	0.19
AlexNet	64.60	40.24	42.22	19.04	1.21	0.26
CLIP-ViT-L/14	95.13	88.75	84.75	5.64	0.95	0.07
B.2 Anchor-based analysis of Center bias in ImageNet and OOD datasets
(c) ImageNet-R
Figure A4: AlexNet
(d) ImageNet-Sketch
(e) ObjectNet
(a) ImageNet-RealL
Figure A5: VGG16
(b) ImageNet-A
Figure A6: ResNet-18
(b) ImageNet-A
Figure A7: ResNet-50
B.3 Distribution of the minimum set cover per classifier and dataset
In this section, we provide details on the distribution of minimum set cover size.
B.4 Only  $70\%$  of all transforms are needed to reach maximum possible accuracy
In Sec. 4.1, we first pre-define all 324 zoom transforms and then compute the maximum possible accuracy to ensure the predicted labels were the results of models looking at a controlled zoomed region (i.e. not because a model was given 324 arbitrary trials per image). Here, we aim to compute the minimum number of zoom settings required for a model to reach the same upper-bound accuracy.
(b) ImageNet-A
Figure A9: CLIP-ViT-L/14
Figure A8:ViT-B/32
(c) ImageNet-R