SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table A3: Distribution of the minimum set cover per classifier and dataset. (ZI: zoom-in, ZO: zoom-out, ZL: zoom-224)
RealL	IN-A	IN-R	IN-Sketch	ON
ZI	ZO	ZL	Total	ZI	ZO	ZL	Total	ZI	ZO	ZL	Total	ZI	ZO	ZL	Total	ZI	ZO	ZL	Total
ResNet-18	160	33	8	201	174	31	6	211	204	65	9	278	209	51	9	269	191	54	9	254
ResNet-50	136	33	9	178	165	42	7	214	200	62	9	271	216	56	9	281	187	63	9	259
ViT-B/32	134	30	4	168	167	19	7	193	196	52	9	257	218	46	9	273	206	58	9	273
VGG-16	158	34	9	201	181	33	8	222	214	66	9	289	210	54	9	273	198	52	9	259
AlexNet	191	40	8	239	170	33	9	212	212	51	9	272	217	49	9	275	201	58	9	268
CLIP-ViT-L/14	141	48	8	197	75	14	4	93	76	33	5	114	142	61	9	212	205	66	9	280
Evaluating this minimum set may reveal spatial biases of a dataset (Sec. 4.2) as well as the implicit zoom operation that a state-of-the-art model (e.g. CLIP) may have learned.
Experiment Given a (dataset, classifier) pair, we constructed a bipartite graph  $G = (N, E)$ , where  $N = A \cup B$ ,  $A$  represents the set of transforms, and  $B$  represents the set of images. The edges  $E$  are defined as follows:
$E = \{(n_{i},n_{j})\mid n_{i}\in A,n_{j}\in B,$  and transform  $n_i$  leads to the correct classification of image  $n_j\}$
We aim to find a minimum set cover [64, 35] in this graph, synonymous with finding a minimum subset of transforms among the 324 that lead to the correct prediction for all classifiable images in Sec. 4 (i.e. those that make up the accuracy scores in Tab. 1c), without unnecessary transforms.
The resulting subset of transforms from the process leads to the correct prediction for all classifiable images without sacrificing accuracy. During each iteration of the greedy minimum set cover algorithm, the transform that yields the highest number of correct classifications for the remaining images is selected. This process continues until all of the images have been "covered," i.e. all images have connected to a transform with at least one edge. The result aligns with our initial goal to remove unnecessary zoom transforms while maintaining the maximum possible accuracy, as outlined in Sec. 4 (i.e., those that make up the accuracy scores in Tab. 1c). The outline of the algorithm can be seen in Algorithm 1.
Results Fig. A10 shows the minimum number of transforms per dataset required to reach the maximum possible accuracy. Although this number varies depending on the dataset and classifier, on average, the size of the minimum cover is 229, which is  $\sim 70\%$  of all 324 pre-defined transforms.
We evaluate the maximum possible accuracy using the top 36 transforms, the same number as the number of zoom scales and report the results in Tab. 1b. This set of transforms is achieved by stopping the algorithm after 36 iterations, which provided us with 36 high-performing transforms. The maximum possible accuracy using only 36 crops is only slightly lower than that when using all 324 crops but is substantially higher than the standard 1-crop, e.g.  $85.19\%$  vs.  $56.16\%$  for AlexNet on IN (Tab. 1b). Also, the upper-bound accuracy for 36 crops being much higher than the
Algorithm 1 Greedy Minimum Set Cover for Transforms
1: Initialization:  $C = \emptyset$  (Covered set of images),  $T = \emptyset$  (Selected transforms)
2: while  $C \neq B$  do
3: Find  $n_i \in A \setminus T$  that maximizes  $|n_j \in B \setminus C|$  ( $n_i, n_j \in E$ )
4:  $C = C \cup \{n_j \mid (n_i, n_j) \in E\}$
5:  $T = T\cup \{n_i\}$
6: end while
7: Result: The subset of transforms corresponding to  $T$  can classify images without sacrificing accuracy.
Figure A10: The minimum number of zoom transforms (out of 324) required to achieve the maximum possible accuracy scores reported in Tab. 1c.
IN	ReaL	IN+ReaL	IN-A	IN-R	IN-S	ON	μ
AlexNet	255	239	246	212	272	275	268	252
VGG-16	242	201	201	222	289	273	259	241
ResNet-18	250	201	208	211	278	269	254	239
ResNet-50	234	178	183	214	271	281	259	231
ViT-B/32	233	168	173	193	257	273	273	224
CLIP-ViT-L/14	251	197	186	93	114	280	212	190
random baseline (i.e.  $3.6\%$  for IN) confirms that the pre-defined zoom transforms are important to classification (not because models are given 36 random trials per image). The top-36 zoom transforms for ResNet-50 on ImageNet contain zooms at various locations in the image (see the visualizations in Appendix D.1).
Remarkably, CLIP requires 190 transforms on average, which is fewer than every other model (Fig. A10;  $\mu$  column). This can be attributed to either the implicit zoom power of CLIP or the fact it has a stronger feature extractor.
B.5 Center-zooming increases the accuracy of all ImageNet-trained models but not CLIP
Previously, we have found that CLIP obtains the best accuracy on all six datasets (Tab. 1a) and also requires the smallest minimum set of zoom transforms to obtain the upper-bound accuracy (Appendix B.4). It is important to understand what classification strategy a CLIP classifier internally performs to classify better. Here, we test the hypothesis that the state-of-the-art CLIP is already performing an implicit zoom on images. If that is true, directly zooming to the center, exploiting the strong center bias of ImageNet-A and ObjectNet, will not improve CLIP accuracy.
Experiment We evaluate the accuracy of all models when center-zooming on IN-A and ON images at 11 different scales  $S \in \{128, 160, 192, \dots, 448\}$  (Fig. A11). That is, center-zooming at  $S$  first resizes the input image so that the smaller dimension becomes  $S$  and then takes a  $224 \times 224$  center crop (zero-padding is applied when necessary).
Results In Fig. A11, we show the changes in the top-1 accuracy (1-crop) when varying the centerzoom scales away from the default ImageNet transform scale ( $S = 256$ ) for both ImageNet-A and ObjectNet. While IN-trained networks exhibit consistent improvement as the zoom scale increases, CLIP shows a monotonic decrease in performance (Fig. A11; yellow curves decreasing on both sides of  $S = 256$ ). This result is surprising but consistent with our hypothesis that CLIP internally performs implicit zooming to reach its peak accuracy and therefore manually zooming (either in or out) at the center mostly ruins its performance.
B.6 Zoom-in is more useful than zoom-out, which is most important to abstract images
Zooming in enhances texture patterns while zooming out provides a better perspective of the object's shape, which is known to be useful to image classification [12, 19]. Results in Sec. 4.1 and Appendix B.4 indicate that this combined zooming approach can be effective in classifying images from diverse datasets. Here, we test which dataset and model pairs require which type of zoom, and whether zooming in or out is always necessary.
Experiment To better understand the effectiveness of each zoom group, we calculate the maximum possible accuracy using all nine locations and different zoom scales  $S$  to show per-dataset trends. Additionally, we examined the percentage of images within each dataset that required a specific zoom group to be accurately classified. This analysis allowed us to gain a more comprehensive
(a) ImageNet-A
Figure A11: Absolute changes in the top-1 accuracy  $(\%)$  of 6 models on ImageNet-A (a) and ObjectNet (b) when center-zooming images at various scales. Interestingly, center-zooming helps IN-trained networks but hurts CLIP.
(b) ObjectNet
understanding of the role that each zoom group played in reaching the maximum possible accuracy reported in Tab. 1.
Results The maximum possible accuracy for different zoom scales reveals a clear trend for each dataset. For instance, a slight zoom-out enhances accuracy for abstract image datasets like IN-Sketch (Fig. A12a). Conversely, for adversarial image datasets such as IN-A, zooming in improves accuracy (Fig. A12b) This pattern is also evident in evaluations using standard 1-crop accuracy (Appendix B.9). Furthermore, the percentage of images that are exclusively classifiable with the zoom-in group is consistently higher than the other two groups, i.e. using ViT-B/32  $51.75\%$  on IN-A, and  $13.11\%$  on IN-S (Tab. A4a). This shows that most datasets necessitate focusing on the object of interest in the image to both see texture patterns better and reduce background clutter (see Tab. A2 for full results). However, we also find that the zoom-out group is also necessary for the correct classification of a small portion of each dataset. For instance,  $1.22\% - 2.97\%$  of IN-S images (Tab. A4b) require a zoom-out transform to be correctly labeled (i.e. zoom-in does not help at all).
Table A4: % of images in the entire dataset that require a particular zoom group to be classified correctly. See Tab. A2 for full results.
zoom-in (a)	zoom-out (b)	zoom-224 (d)
IN-A	IN-S	IN-A	IN-S	IN-A	IN-S
ResNet-18	40.67	11.83	0.92	1.77	0.19	0.36
ResNet-50	48.72	10.99	1.00	2.23	0.23	0.24
ViT-B/32	51.75	13.11	0.85	1.83	0.15	0.36
VGG-16	38.24	9.47	0.93	2.97	0.29	0.28
AlexNet	26.27	11.26	1.08	1.22	0.36	0.33
CLIP-ViT-L/14	12.01	6.64	0.44	2.38	0.05	0.12
B.7 Simple aggregation of the zoom transforms can improve accuracy on some datasets but not all
Sec. 4.1 and Appendix B.5 show that using the same feature extractors (even as old as AlexNet), it is possible to achieve higher image classification accuracy if we know where to zoom and at which scale. A practical follow-up question is: How to build a classifier that knows how to zoom given
(a) ImageNet-Sketch
Figure A12: Maximum possible accuracy using nine crops at varying scales. The vertical line represents the standard ImageNet zoom scale ( $S = 256$ ). While for ImageNet-Sketch (a), zooming out marginally improves the accuracy, for scale factors larger than 256, ImageNet-A (b) exhibits an increase in accuracy. See Appendix B.9 for details.
a test image? In this section, we establish simple baselines that aggregate predictions over a set of zoom transforms.
Experiment We employ the mean method from prior work [61, 46], and the max method to aggregate output marginal distributions. For a given image, we get  $N$  output distributions over classes from a classifier, in which  $N$  is the total number of used transforms. The aggregation process combines these  $N$  distributions and outputs a final prediction for the given image. In the aggregation step, we use the mean or max method to infer the final confidence for each class along  $N$  distributions. Finally, we select the class that has the highest confidence score. Additionally, we test 5-crop and 10-crop evaluation [38, 63, 23] and compare them with our methods. We use the transforms in the minimum set found for IN-ReaL to evaluate the remaining datasets. The purpose is to reduce the number of augmentations and prevent training on OOD benchmarks.
Results max aggregation of zoom-in transforms results in the largest improvements on ImageNet-A. That is, on IN-A, ViT-B/32 reaches a top-1 accuracy of  $24.69\%$  (+15.05) (Tabs. A5 and A6) and a ResNet-50 accuracy increases by +13.03 points from 16.62% to 29.65% (Appendix C.3)—a surprisingly strong baseline for future studies. On ObjectNet, max aggregation of zoom-in transforms also yields +1.99 improvement over the 1-crop ViT-B/32 baseline.
On the other hand, mean aggregation results in smaller but more consistent improvements over the 1-crop baseline for many datasets (+3.56 on IN, +4.08 on ReaL, +4.65 on IN-A, and +3.03 on ON; Tab. A5). mean aggregation (Tab. A5b) also outperforms the standard 5-crop and 10-crop [38, 23] aggregation on these four datasets (Tab. A5e-f).
In contrast, for all 6 datasets, aggregating zoom-out and zoom-224 transforms consistently worsen the performance over the 1-crop baseline (Tab. A5c-d). That is, we find that for a few dozen images (e.g. sketches and abstract visuals; Fig. 1ac), interestingly, only zooming out can lead to a correct classification (Appendix B.6), yet for most images in these 6 benchmarks, zooming out hurts the accuracy.
In summary, based on the insights from Sec. 4.1, showing that zooming could help classification, we find that simple methods for aggregating zoom-in transforms at test-time can directly improve model accuracy over the 1-crop and zoom-224 baselines on four benchmarks, i.e. all except IN-R and IN-S, which contain abstract images.
Table A5: Top-1 accuracy (\%) of aggregation methods on an IN-trained ViT-B/32 model. Compared to the 1-crop baseline, aggregating zoom-in transforms consistently yields improved accuracy on IN-A, ON but worse accuracy on IN-R and IN-S. zoom-224 refers to the set of zoom transforms at  $S = 224$ . See Tab. A6 for more results.
(a)	(b) zoom-in	(c) zoom-out	(d) zoom-224	(e) 5-crop	(f) 10-crop [38]
Dataset	1-crop	max	mean	max	mean	max	mean	max	mean	max	mean
IN	75.75	74.35 (-1.40)	79.31 (+3.56)	71.48	69.47	72.66	73.67	77.33	77.73	77.30	77.87
ReaL	81.89	80.22 (-1.67)	85.97 (+4.08)	77.95	76.28	79.25	80.31	83.24	83.80	83.17	83.87
IN-A	9.64	24.69 (+15.05)	14.29 (+4.65)	7.79	5.48	8.12	7.39	12.19	9.88	12.32	9.67
IN-R	41.29	39.90 (-1.39)	40.06 (-1.23)	39.05	36.21	39.52	39.28	43.90	43.17	44.31	43.28
IN-S	26.83	19.74 (-7.09)	20.89 (-5.94)	22.37	19.25	25.06	25.21	28.72	28.66	28.94	28.76
ON	30.89	32.88 (+1.99)	33.92 (+3.03)	22.56	19.51	22.75	22.72	26.96	24.98	27.14	24.97
B.8 Runtime analysis of MEMO
Another benefit of RRC compared to AugMix is faster inference time. Table A7 shows the runtime analysis of MEMO. Typically, TTA methods suffer from slow runtime due to augmentation and test-time training processes. We find that MEMO + RRC consistently leads to an average  $1.6 \times$  speed-up compared to MEMO + AugMix (Tab. A7; 0.65s / image vs. 1.15s / image), providing more evidence to support this transformation as a viable option for test-time augmentations.
B.9 1-crop accuracy with different zoom scales
In this section, we demonstrate the performance of various models when zooming in or out of an image. In other words, we utilize the standard 1-crop ImageNet transform while altering the initial scale of the image.
Table A7: Average runtime per query image (in seconds). Using RandomResizedCrop in MEMO speed ups the runtime by an average factor of  $1.6 \times$ .
Runtime (in seconds)	IN	IN-A	IN-R	IN-S	ON
MEMO + AugMix [83]
ResNet-50 [23]	1.24	1.12	1.12	1.32	1.51
DeepAug+AugMix [26]	1.19	1.07	1.12	1.23	1.55
MoEx+CutMix [40]	1.15	1.16	1.11	1.31	1.53
MEMO + RRC (Ours)
ResNet-50 [23]	0.64	0.60	0.65	0.88	1.19
DeepAug+AugMix [26]	0.62	0.62	0.64	0.87	1.18
MoEx+CutMix [40]	0.65	0.62	0.66	0.88	1.19
In this section, we are conducting experiments using the following models: AlexNet [38], ConvNet (Base, Large, Small, Tiny) [44], DenseNet-161 [29], EfficientNet-B7 [69], MobileNet (V2, V3 Large) [60, 28], ResNet (50, 101) [23], ResNeXt-50 (32x4d) [80], ShuffleNet V2 x1.0 [47], VGG-19 [63], Vision Transformer (ViT-B/16, ViT-B/32, ViT-L/16, ViT-L/32) [17], and Wide ResNet-50-2 [82].
Figure A13: ImageNet accuracy using a 1-crop transform (the vertical line represents the standard ImageNet transform scale factor).
Figure A14: ImageNet-A accuracy using a 1-crop transform (the vertical line represents the standard ImageNet transform scale factor).
Figure A15: ImageNet-R accuracy using a 1-crop transform (the vertical line represents the standard ImageNet transform scale factor).
Figure A16: ImageNet-Sketch accuracy using a 1-crop transform (the vertical line represents the standard ImageNet transform scale factor).
Figure A17: Accuracy using a 1-crop transform on 5K random images of the ObjectNet dataset (the vertical line represents the standard ImageNet transform scale factor).
Figure A18: Breakdown of the accuracy of IN-trained models at different crop locations and scale size - Analysis of accuracy across various crop locations and scale sizes reveals that different datasets exhibit distinct optimal conditions. For instance, the IN-A dataset experiences a considerable increase in accuracy when zoomed in, while ImageNet-R yields better results when zoomed out.
B.10 Distribution of the Top 36 performing transforms.
In this section, we provide more details about the distribution of the top-36 performing transforms. Our results suggest that, on average,  $26.65\%$  of all top-36 performing transforms belong to the center at varying scales.
Figure A19: Distribution of Top-36 performing transforms for ImageNet-ReaL
Figure A20: Distribution of Top-36 performing transforms for ImageNet-A
Figure A21: Distribution of Top-36 performing transforms for ImageNet-R
Figure A22: Distribution of Top-36 performing transforms for ImageNet-Sketch
Figure A23: Distribution of Top-36 performing transforms for ObjectNet
B.11 Background occlusion in ImageNet dataset
Sample images for images with and without occlusion.
(a) A sample image of the Tank class without occlusion.
(b) Image with heavy background occlusion.
(c) A clean sample image of the Four Poster class.
Figure A24: Background occlusion examples.
(d) A low-quality image with background occlusion.
In this section, we provide additional experiments with the proposed zoom-based transform.
C.1 Zooming is similarly important to the foreground and background contents
Background pixels, despite often being neglected in image classification, can contain predictive signals [86, 78, 20, 57]. It has remained largely unknown how much the image context (background) could contribute to the model performance. While Zhu et al. [86] disentangle the predictiveness of background (BG) and foreground (FG) via model training, we directly measure how pretrained models perceive these two signals.
Experiment Using bounding-box annotations provided by Russakovsky et al. [59], we create two dataset variations of ImageNet: FGSet and BGSet, following Zhu et al. [86]. We mask all the background for FGSet as in Fig. A25b, and for BGSet we mask all the main objects, as depicted in Fig. A25d & Fig. A25f. After that, we compute the accuracy of these two sets with all tested classifiers using ImageNet and ImageNet-ReaL labels as in Tab. A8.
Results Our results suggest that zooming is important to ImageNet regardless of whether foreground or background features are used, with the difference for FGSet and BGSet on average being similar (Tab. A8). Additionally, when only the background features were available, almost half of ImageNet images  $(45.23\%)$  could be correctly classified if optimal Zoom was used. Finally, we found that with only foreground information, ViT-B/32 could achieve a maximum possible accuracy of  $95.50\%$  given an optimal zooming method, suggesting that only  $98.75\% - 95.50\% = 3.25\%$  of images (Tab. 1) required the background information. These findings suggest that both foreground and background features are important for ImageNet classification, but that an optimal zooming method can considerably improve performance even in the absence of one of these feature sets.
(c)
(d)
Figure A25: The foreground and the background both contain predictive signals. A ResNet-50 classifier can detect axolot1 (a), even when the main object is masked (b). Removing the background from images of 'goose' (c) and 'parachute' (e) causes misclassification (d, f).
C.2 Adversarial datasets contain more objects compared to ImageNet
So far, our findings indicate that if we apply the zoom-in operation to the two datasets of ImageNet-A and ObjectNet, the performance of conventional vision models improves consistently up to a certain threshold (Sec. 4 and Appendix B.5). This suggests that the initial images contain distracting elements that impede the model from correctly identifying the object of interest. Both ImageNet-A and ObjectNet are considered out-of-distribution datasets, which are specifically designed to evaluate a vision model's ability to withstand natural adversarial and pose attacks. We hypothesize that the primary reason that these datasets are hard can be attributed to background clutter, multiple objects, and the presence of a positional bias in these images.
Experiment We use OWL-ViT [49], an open vocabulary object detection model, to quantify the number of objects present in three datasets of ImageNet, ImageNet-A, and ObjecNet. The OWL-ViT expects an input image with a set of object names and will determine if any object instances are present in the image. To specify object names, we use LVIS vocabulary [22], which encompasses a
Table A8: ImageNet classification from object-only and background-only signals. Numbers show the maximum possible top-1 accuracy (\%) using zoom-based transforms for minimum set covers in Appendix B.4. We discover that background signals potentially hold significance for image classification. The bold numbers show the highest possible accuracy per dataset and group.
1-crop	Max possible using zooming
FGSet	BGSet	FGSet	BGSet
IN	ReAL	IN	ReAL	IN	ReAL	IN	ReAL
ResNet-18	59.77	64.97	4.91	7.84	89.89(+30.12)	92.04(+27.07)	25.81(+20.90)	31.33(+23.49)
ResNet-50	68.02	72.90	6.18	9.83	93.45(+25.43)	94.89(+21.99)	30.30(+24.12)	35.98(+26.15)
ViT-B/32	67.46	71.78	9.72	13.38	94.40(+26.94)	95.50(+23.72)	39.70(+29.98)	45.23(+31.85)
VGG-16	63.78	69.09	5.36	8.59	91.01(+27.23)	92.91(+23.82)	26.98(+21.62)	32.62(+24.03)
AlexNet	42.38	46.54	3.66	5.46	80.20(+37.82)	83.25(+36.71)	22.02(+18.36)	27.04(+21.58)
CLIP-ViT-L/14	74.46	78.62	9.49	13.80	96.14(+21.68)	97.35(+18.73)	36.85(+27.36)	42.51(+28.71)
mean	62.65	67.32	6.55	9.82	90.85 (+28.20)	92.66 (+25.34)	30.28 (+23.73)	35.79 (+25.97)
comprehensive list of 1203 distinct objects. The OWL-ViT model includes a threshold parameter that reflects its confidence level in its predictions. To assess whether different threshold values would affect our results, we conducted our experiment using both 0.1 and 0.05 as threshold values.
After calculating the distribution of the number of objects in images, we perform a Mann-Whitney U test to determine whether there is a statistically significant difference in this distribution between datasets. As each dataset has a different number of classes, we limited our analysis to shared classes between any two datasets.
Results The results of our study reveal a contrast between ImageNet and ImageNet-A, as well as ImageNet and ObjectNet. This finding implies a dissimilarity between the images in the original ImageNet dataset and its OOD datasets that might arise from the presence of background clutter. Specifically, on average, images in ImageNet-A and ObjectNet datasets tend to feature more objects, which can pose more significant distractions for image classification models.
The results of the Mann-Whitney U test also reflect this finding, the p-value for both thresholds was found to be less than 0.05, which is statistically significant at the  $95\%$  confidence level (Tab. A9).
Figure A26: Comparison of the number of objects in two datasets of ImageNet and ImageNet-A using OWL-ViT [49] -  $T$  denotes the classification's threshold
C.2.1  $p$ -values for Mann Whitney U test
Table A9: The result of the Mann-Whitney U test to compare ImageNet with ImageNet-A and ObjectNet
T = 0.05	T = 0.01
ImageNet-A	6.27E-265	1.71E-235
ObjectNet	1.80E-02	3.66E-02
Figure A27: Comparison of the number of objects in two datasets of ImageNet and ObjectNet using OWL-ViT [49] -  $T$  denotes the classification's threshold
C.3 Zooming further improves robustified models on ImageNet-A
Intensive data augmentations have been proven to significantly boost CNNs' performance [77, 66] on ImageNet. Motivated by these previous successes and the fact that neural networks trained on diverse augmentations are able to learn robust representations [41], we want to know if robustified pretrained models (i.e. trained with intensive augmentations) could reach higher accuracy on ImageNet-A using zooming in.
Experiment We test 4 different ResNet-50 classifier versions that have been trained with different data augmentation procedures. From the the torchvision library, we select two sets of model weights; trained with  $(\mathrm{V}2^{3})$  and without  $(\mathrm{V}1^{4})$  data augmentations. We also take two other models trained with DeepAugmentation+AugMix [26] and MoEx+CutMix [40]. The second column in Tab. A10 represents the accuracy of models using 1-crop.