SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mohammad Reza Taesiri
mtaesiri@gmail.com
Giang Nguyen
nguyengiangbkhn@gmail.com
Sarra Habchi
sarra.habchi@ubisoft.com
Cor-Paul Bezemer
bezemer@ualberta.ca
Anh Nguyen
anh.ng8@gmail.com
Image classifiers are information-discarding machines, by design. Yet, how these models discard information remains mysterious. We hypothesize that one way for image classifiers to reach high accuracy is to zoom to the most discriminative region in the image and then extract features from there to predict image labels, discarding the rest of the image. Studying six popular networks ranging from AlexNet to CLIP, we find that proper framing of the input image can lead to the correct classification of  $98.91\%$  of ImageNet images. Furthermore, we uncover positional biases in various datasets, especially a strong center bias in two popular datasets: ImageNet-A and ObjectNet. Finally, leveraging our insights into the potential of zooming, we propose a test-time augmentation (TTA) technique that improves classification accuracy by forcing models to explicitly perform zoom-in operations before making predictions. Our method is more interpretable, accurate, and faster than MEMO, a state-of-the-art (SOTA) TTA method. We introduce ImageNet-Hard, a new benchmark that challenges SOTA classifiers including large vision-language models even when optimal zooming is allowed.
Since the release of AlexNet in 2012 [38], deep neural networks have set many ImageNet (IN) [59] accuracy records [38, 23]. While many papers reported improved learning algorithms or architectures, little is known about how the inner workings of image classifiers actually evolve. The success is often attributed to a network's ability to detect more objects [9] and a variety of facets of each object (i.e., invariance to style, pose, and form changes) [50, 21]. By aggregating the information from all the visual cues in a scene, a classifier somehow chooses a better label for the image. For example, Figs. 13-14 in [58] show that a model detects both dogs and cats in the same image and only discards the dog features right before the classification layer to arrive at a tiger cat prediction.
When processing an image, a network may implicitly zoom in or out (defined in Sec. 3) to the most discriminative image region ignoring the rest of the image (Fig. 1a), and then extract that localized region's features to predict image labels. We hypothesize that the improved image classification may largely be due to the networks accurately zooming to the discriminative areas (e.g., junco and magpie birds in Fig. 1a) rather than more accurately describing them (i.e. generating better features of these two birds). In this work, we present supporting evidence for our zooming hypothesis.
(b)
Figure 1: (a) Each subfigure shows an input image, predicted label, and confidence score from an ImageNet classifier (top and middle: ResNet-50 [23]; bottom: ViT-B/32 [17]). With the standard center-crop transform, all 3 samples were misclassified (left-most column). Adjusting framing via zooming yields correct predictions. (b) The zooming process correctly classifies a snail1 ImageNet-A image. We uniformly adjust the input query image's smaller dimension to match the target scale  $S$ . We then partition the image into a  $3 \times 3$  grid, generating 9 crops centered at grid-cell centers (i.e., anchor points) and feed each crop to the original image classifier.
We conduct a thorough study to test the effects of zooming in and out on the classification accuracy of six network architectures on six ImageNet-scale benchmarks. Our main findings also include:
1. A major, surprising finding is that state-of-the-art, IN-trained models can accurately predict up to  $98.91\%$  of ImageNet samples when an optimally zoomed image is provided. The remaining few hundred IN images  $(0.39\%)$  that are never correctly labeled by any model (despite optimal zooming) include mostly ill-posed and rare images (Sec. 4.1).
2. ImageNet-A [27] and ObjectNet [86] both exhibit a substantial center bias. For example, by only upsampling and center-cropping each ImageNet-A image, ResNet-50's accuracy increases dramatically from  $0.09\%$  to  $14.58\%$  (Sec. 4.2).
3. Zooming can be leveraged as an inductive bias at test time to improve ImageNet classification accuracy. That is, integrating zoom transformations into MEMO [83], a leading test-time augmentation method, yields consistently higher accuracy than the baseline ResNet-50 models and also MEMO with default transformations on multiple datasets (Sec. 4.3).
Our findings show that the accuracy of image classifiers can be improved by finding an optimal zoom setting first and then classifying that crop alone (Fig. 1a). Motivated by this insight, we build ImageNet-Hard<sup>1</sup>, a new 1000-way classification benchmark that challenges state-of-the-art (SOTA) classifiers despite the application of optimal zooming (Sec. 4.4). In other words, we collect images from seven existing ImageNet-scale benchmarks where OpenAI's CLIP ViT-L/14 [53] misclassifies even when allowed to try 324 zooming settings. Interestingly, SOTA classifiers that operate at  $224 \times 224$  resolution perform poorly on ImageNet-Hard (below  $19\%$  accuracy). Analyzing misclassifications on ImageNet-Hard reveals a major remaining challenge in the era of SOTA classifiers of Transformers [17], EfficientNets [69, 36], and large vision-language models (Sec. 4.5).
Learning to Zoom in image classification Leveraging zoom-in or crops of an image has a long history of improving fine-grained image classification with approaches varying from combining multiple crops at different resolutions [18, 73], using multiple crops of the object (i.e., part-based classification) [16, 37, 84, 68, 84, 36] to warping the input image [55, 31, 32]. We note that a common
$^{1}$ Code and data are available on https://taesiri.github.io/ZoomIsAllYouNeed.
prior definition of "zoom" [55, 31, 32, 70] is to first divide an input image into a grid and then warp the image, distorting the aspect ratio of the objects in the image. In contrast, our zoom procedure utilizes only two functions: resize and crop, maintaining the original aspect ratio.
Furthermore, to our knowledge, we are the first to perform a zoom study on ImageNet-scale datasets (ImageNet [59], ImageNet-A [27], ObjectNet [8], etc) while prior zoom approaches [16, 37, 84, 68, 84, 36, 32] exclusively focus on non-ImageNet, fine-grained classification (e.g. classifying birds or dogs). Due to such differences in the image distribution of interests, prior works mostly study zooming in (which benefits fine-grained classification) while we study both zooming in and out.
Test-time data augmentation (TTA) is a versatile technique that could help estimate uncertainty [65, 7, 6] and improve classification accuracy [38, 67, 23, 52, 46, 62, 34, 14]. When test inputs are sampled from unseen, non-training distributions, augmenting the data often improves a model's generalization to new domains [74, 83]. A simple TTA method is 10-crop evaluation [38] in which 5 patches of  $224 \times 224$  px along with their horizontal reflections (resulting in 10 patches) are extracted from the original image. An alternative way to leverage the marginal output distributions over augmented data is to use them as gradient signals to update the classifier's parameters [75, 83]. We employ this approach to update the model during test time, with patches being zoom-based augmentations.
Biases in ImageNet and datasets Resize-then-center-crop has been a pre-processing standard for ImageNet classification since AlexNet [38]. This pre-processing exploits the known center bias of ImageNet. While ImageNet has been shown to contain a variety of biases in image labels [10, 71], object poses [5], image quality [39], we are the first to examine the positional biases of the out-of-distribution (OOD) benchmarks for ImageNet classifiers and find a strong center bias in ImageNet-A and ObjectNet that could affect how the community interprets progress on these OOD benchmarks.
Perhaps the closest to us is a preprint by Li et al. [42] that shows that cropping to the main object can improve model accuracy on ImageNet-A [27]. Yet, unlike [42], we study six ImageNet-scale datasets, both zooming in and zooming out, and we propose a new dataset of ImageNet-Hard.
Zoom definition To zoom in or out of the image, we only use resize and crop operations. Initially, we uniformly resize the test image so that the smaller dimension matches the target scale of  $S$ . Then, we define a  $3 \times 3$  grid on the image to divide it into 9 patches. We perform a CenterCrop operation at the center (  $\bullet$  in Fig. 1b) of each patch to extract a  $224 \times 224$  px crop from each of the nine locations (see Python code in Appendix A.1). In the CenterCrop step, zero-padding is used when the content to be cropped is smaller than  $224 \times 224$ . Overall, at each target scale  $S$ , we generate 9 crops (Fig. 1b).
We test 36 different values of  $S$  ranging from 10 to  $1024\mathrm{px}$ , resulting in a total of  $36\times 9 = 324$  different zoomed versions for each image. Based on initial scale factor  $S$ , we define three groups: (1) zoom-out group contains all augmented crops where  $S < 224$ ; (2) zoom-in group contains all augmented crops where  $S > 224$ ; and (3) zoom-224 group contains the 9 patches where  $S = 224$ .
Benchmark datasets We use the ImageNet (IN) [59] dataset with both the original and ImageNet-ReaL [10] (ReaL) labels. For each IN image, we use the union of the IN and ReaL labels (IN+ReaL) to complement each other to reduce noise in IN labels. We further examine the effects of zoom-based transformations on four popular OOD benchmarks: (a) natural adversaries (ImageNet-A [27]), (b) image renditions (ImageNet-R [26]), (c) black-and-white sketches (ImageNet-Sketch [75]), and (d) viewpoint-and-background-controlled samples (ObjectNet [8]). We refer to these as benchmarks as IN-A, IN-R, IN-S, and ON, respectively, in the rest of the paper.
Classifiers We study the effects of zoom-based transformations on six popular image classifiers in the last decade: AlexNet [38], VGG-16 [63], ResNet-18 & ResNet-50 [23], ViT-B/32 [17], and OpenAI's CLIP-ViT-L/14 [53]. The inclusion of the 11-year-old AlexNet provides a baseline for the power of deep features (when given the right region to look at). Predicted labels from CLIP-ViT-L/14 are acquired using its standard zero-shot classification setup (Appendix A.5).
4.1 Zooming has the potential to substantially improve image classification accuracy
To understand the potential of zooming in improving image classification accuracy, first, we establish an upper-bound accuracy (i.e. when an "optimal" zoom is given). That is, we apply 36 scales  $\times 9$  anchors  $= 324$  zoom transformations (Sec. 3) to each image to generate 324 zoomed versions of the input. We then feed all  $N = 324$  versions to each network and label an image "correctly classified given the optimal zoom" if at least 1 of the 324 is correctly labeled. We call such maximum possible top-1 accuracy "upper-bound accuracy" in Tab. 1. Our experiment also informs the community of the type of image that cannot be correctly labeled even with an optimal zooming strategy.
Table 1: On in-distribution data (IN & ReaL) there exists a substantial improvement when models are provided with an optimal zoom, either selected from 36 (b) or 324 pre-defined zoom crops (c). In contrast, OOD benchmarks still pose a significant challenge to IN-trained models even with optimal zooming (i.e., all upper-bound accuracy scores  $< 80\%$ ).
IN	ReAL	IN+ReAL	IN-A	IN-R	IN-S	ON
(a) Standard top-1 accuracy based on N = 1 crop
AlexNet	56.16	62.67	61.76	1.75	21.10	10.05	14.23
VGG-16	71.37	78.90	78.52	2.69	26.98	16.78	28.32
ResNet-18	69.45	76.94	76.47	1.37	32.14	19.41	27.59
ResNet-50	75.75	82.63	82.97	0.21	35.39	22.91	36.18
ViT-B/32	75.75	81.89	82.59	9.64	41.29	26.83	30.89
CLIP-ViT-L/14	75.03	80.68	81.95	71.28	87.74	58.23	66.32
(b) Upper-bound accuracy using N = 36 crops
Random	3.60	3.60	3.60	18.00	18.00	3.60	31.85
AlexNet	85.19	90.30	89.74	31.37	47.04	24.40	49.17
VGG-16	92.30	96.08	95.81	46.69	52.86	34.34	62.94
ResNet-18	92.08	95.97	95.73	47.48	58.85	37.91	63.08
ResNet-50	94.46	97.36	97.40	55.68	61.42	41.71	69.60
ViT-B/32	95.05	97.61	97.88	68.43	68.77	49.10	70.30
CLIP-ViT-L/14	94.19	97.32	97.56	97.16	98.60	83.77	89.59
(c) Upper-bound accuracy using N = 324 crops
Random	32.40	32.40	32.40	100.00	100.00	32.40	100.00
AlexNet	90.03	93.85	93.48	42.23	55.52	29.53	59.65
VGG-16	95.30	97.90	97.66	58.27	60.88	39.90	71.85
ResNet-18	95.15	97.76	97.55	58.87	66.89	43.68	71.44
ResNet-50	96.78	98.62	98.57	66.68	68.84	47.64	76.83
ViT-B/32	97.19	98.75	98.91	78.03	75.58	55.99	79.28
CLIP-ViT-L/14	96.78	98.69	98.80	98.45	99.20	89.00	93.13
Figure 2: IN+Real samples that are not correctly classifiable by IN-trained models using any of the 324 zoom transforms.
Results Table 1 shows upper-bound accuracy over different values of  $N = \{1,36,324\}$ . First, the random baselines (given  $N = 324$  attempts per image) are at  $32.4\%$  for 1,000 classes (IN, ReAL, IN+ReAL, and IN-Sketch), and  $100\%$  for 200 and 313 classes (IN-A and ON, respectively). Yet, the accuracy of models with optimal zooming is far from random—e.g. ResNet-50 largely outperforms not only the random baseline but also the 1-crop baseline on IN (96.78% vs. 75.75%; Tab. 1a vs. c).
On the IN, ReAL, and IN+ReAL datasets, there is a substantial gap for all models (around +20 to +35 points) between the 1-crop and the optimal zooming setting (Tab. 1a vs. c). Surprisingly, given optimal zooming, the 11-year-old AlexNet actually can correctly label over  $90\%$  of IN images, which is roughly the 1-crop accuracy  $(87.8\%)$  of the 2022 state-of-the-art ConvNexts [44]. This result is consistent with our hypothesis: One way for state-of-the-art classifiers to obtain their current accuracy is to simply learn how to zoom on top of the same, old feature extractors (e.g. that of AlexNet).
Unclassifiable IN images Interestingly, even with optimal zooming, no model reaches  $100\%$  on IN images. We find that  $0.39\%$  of the  $\mathrm{IN} + \mathrm{RealL}$  images were not classified correctly by any of the IN-trained classifiers and these images are similar to natural adversarial images (Fig. 2) and can be categorized into four groups:
1. Lack of information (Fig. 2a): Images lack adequate signals for classification due to low light, occlusion, blurriness, or noise.
2. Rare cases (Fig. 2b): Images depict the primary object but in an uncommon form, pose, or rendition.
(a) ImageNet-ReAL
(b) ImageNet-A
(c) ImageNet-R
Figure 3: Upper-bound accuracy (\%) of ResNet-50 at each of the 9 zoom locations. The large gaps between the center and eight off-center locations on IN-A and ON demonstrate a center bias, which is much smaller in IN (Appendix B.2) and IN-R (b). The values in parentheses indicate the delta with respect to the center crop.
(d) ImageNet-Sketch
(e) ObjectNet
3. Illusions (Fig. 2c): Images have misleading elements, like a shadow appearing as a staircase, leading to misclassification.
4. Many objects (Fig. 2d): Images displaying several classes of objects but not all classes are listed in the set of groundtruth labels.
OOD datasets pose a significant challenge to IN-trained models despite optimal zooming. Across IN-A, IN-R, and ON, all IN-trained models perform far below the 324-crop random baseline (100%) with the highest score being  $79.28\%$  (Tab. 1). In contrast, CLIP reaches far better scores than IN-trained models (98.45% on IN-A and 99.20% on IN-R; Tab. 1). Our result suggests that OOD images (e.g. objects in unusual poses or renditions) require a more robust feature extractor to recognize besides zooming. And that CLIP was trained on an Internet-scale dataset [53] and thus is much more familiar with a variety of poses, styles, and shapes of objects [21].
Among the 324 zoom transformations, for each (classifier, dataset) pair, we initially construct a bipartite graph connecting transforms to images based on their correct classification. With this graph, we employ the iterative, greedy minimum-set cover algorithm [64, 35] to compute the minimum set of transforms required to achieve the upper-bound accuracy detailed in Sec. 4.1. Through this process, we discover that, on average, only  $70\%$  of the transforms are essential. Furthermore, we identify the top-36 zoom transforms most important to classification (see visualizations in Appendix D.1). More details on this process can be found in Appendix B.4.
The upper-bound accuracy using 36 crops (Tab. 1b) is only slightly lower than that when using all 324 crops but is substantially higher than (1) the standard 1-crop, e.g.  $85.19\%$  vs.  $56.16\%$  for AlexNet on IN (Tab. 1b); and (2) the random baseline (i.e.  $3.6\%$  for IN). Our result confirms that these 36 zoom transforms are indeed important to classification (not because models are given 36 random trials per image) and that studying them might reveal interesting insights into the datasets.
As our 324 transforms include both zoom-in and zoom-out, we further analyze the contribution of each zoom type to each dataset. We find that, across 7 datasets, zoom-in is more useful than zoom-out. And that zoom-out is the most important to abstract images i.e., of IN-R and IN-S (Appendix B.6).
4.2 ImageNet-A and ObjectNet suffer from a severe center bias
The standard image pre-processing for IN-trained models involves resizing the image so its smaller dimension is 256, then taking the center  $224 \times 224$  crop of the resized image [2, 38]. While suitable for ImageNet, this pre-processing may not be optimal for every OOD dataset, not allowing a model to fully utilize off-center visual cues (which optimal zooming could). Leveraging the minimum set of transforms obtained in Appendix B.4, we quantify which spatial locations (out of 9 anchors; Fig. 1b) contain the most discriminative features in each dataset. That is, we compute the upper-bound accuracy for each of the 9 anchor points per dataset and discover biases in some benchmarks.
Experiment For each image, we have 9 anchors (Fig. 1b) and the originally  $K = 36$  zoomed versions per anchor as defined in Sec. 3. Yet, after reducing to the minimum set (Appendix B.4),  $K$  averages at 25, over all datasets, and  $10 \leq K \leq 31$ . Here, we count the probability that the  $K$  zoomed versions per anchor lead to a correct prediction. In other words, we compute the upper-bound accuracy as in Sec. 4.1 but for each anchor separately.
Results First, as expected, the upper-bound accuracy for each anchor (Fig. 3) is consistently lower than when all 9 anchors are allowed (Tab. 1c). Second, across all 6 datasets, the center anchor consistently achieves the highest upper-bound accuracy versus the other 8 locations (Fig. 3 and Appendix B.2), indicating a center bias in all datasets. However, we find this bias is small in IN, IN-R, and IN-S but large in IN-A and ON (i.e., the largest difference between center accuracy and the lowest off-center accuracy is around -25 and -23 points, respectively; whereas for other datasets, it is around (-1) to (-5) points, as shown in Fig. 3).
The center bias in ObjectNet can be explained by the fact that the images were captured using smartphones with aspect ratios of 2:3 or 9:16 (Appendix D.3.3). Overall, such strong center bias in IN-A and ON may not be desirable since improvements on these two benchmarks may be attributed to learning to zoom to the center as opposed to the intended quest of recognizing objects in unusual forms (IN-A) or poses (ON). By merely upscaling the image and center cropping, we can achieve higher accuracy using nearly all the same models on these two datasets (Figs. A14 and A17).
We also find that, during test time, center-zooming (Appendix B.5) increases the top-1 accuracy of all IN-trained models but not CLIP, even on IN-A and ON images. This observation is intriguing considering these OOD datasets contain more distracting objects than ImageNet images (Appendix C.2) and therefore, center-zooming should de-clutter the scene for more accurate classification. However, CLIP prefers a specific zoom scale that provides sufficient background for object recognition—it struggles to identify a single object in a tightly-cropped image [85]. Future research should examine whether this "zoom bias" of CLIP is due to its image- or text-encoder, or both.
4.3 Test-time augmentation of MEMO with only zoom-in transforms improves accuracy
Aggregating model predictions over zoom-in versions of the input image during test time leads to higher top-1 accuracy on IN, IN-ReaL, IN-A and ON, but lower accuracy on IN-R and IN-S (Appendix B.7). However, interestingly, always zooming out on IN-R and IN-S abstract images also hurts accuracy, suggesting that an adaptive zooming strategy might be a better approach.
Here, we test building such an adaptive test-time zooming strategy by modifying MEMO [83], a SOTA test-time augmentation method that finetunes a pre-trained classifier at test time to achieve a more accurate prediction. Specifically, MEMO finds a network that produces a low-entropy predicted label over a set of  $K = 16$  augmented versions of the test image  $I$  and then runs this finetuned model on  $I$  again to produce the final prediction. It does this by applying different augmentations to the test point  $I$  to get augmented points  $I_1, \ldots, I_K$ , passing these through the model to obtain predictive distributions, and updating the model parameters by minimizing the entropy of the averaged marginal distribution over predictions. While improving accuracy, MEMO requires a pre-defined set of diverse augmentation transforms (e.g. sheer, rotate, and solarize in AugMix [26]). Yet, the hyperparameters for each type of transform are hard-coded, and the contribution of each transform to improved classification accuracy is unknown.
We improve MEMO's accuracy and interpretability by replacing AugMix transforms with only zoom-in functions. Intuitively, a model first looks at all zoomed-in frames of the input image (at different zoom scales and locations) and then decides to achieve the most confident prediction.
Experiment MEMO relies on AugMix [25], which applies a set of 13 image transforms, such as translation, rotation, and color distortion, to an original image at varying intensities, and then chains them together to create  $K = 16$  new augmented images (examples in Appendix D.5).
We replace AugMix with RandomResizedCrop [4] (RRC), which takes a random crop of the input image (i.e. at a random location, random rectangular area, and a random aspect ratio) and then resize it to the fixed  $224 \times 224$  (i.e. the network input size). RRC basically implements a random zoom-in function (examples in Appendix D.5).
We compare the original MEMO [83] (which uses AugMix) and our version that uses RRC on five benchmarks (IN, IN-A, IN-R, IN-S, and ON). We follow the same experimental setup as in [83] (e.g.  $K = 16$ ). Specifically, we test three ResNet-50 variants that were pre-trained using distinct augmentation techniques.
2The ResNet-50 model used as a baseline in this Sec. 4.3 is different from that in our other (non-MEMO) sections of the paper.
We utilize Grad-Cam [58] to understand the impact of MIMO on the network's attentions within the final layer, both before and after modification. Specifically, our investigation seeks test our hypothesis concerning the model's focus on the regions of interest within an image.
Results Both our MEMO + RRC and the original MEMO + AugMix [83] consistently outperform the baseline models, which do not use MEMO, on all five datasets (Tab. 2). That is, when combined with MEMO, zoom-in transforms implemented via RRC are also helpful in classifying IN-S and IN-R images—where we previously find zoom-in to not help in mean/max aggregation (Appendix B.7).