SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(as defined in §3.3), we conclude that an entity similar to our prediction is shown in the image and proceed with our analysis. Otherwise, we conclude that the analyzed error is not fine-grained. To confirm that the shown entity is indeed out-of-vocabulary, we first collect a set of proposal labels from the following (illustrated in Fig. 10): all IMAGENET labels in the same superclass as the model's prediction (IV), all direct WORDNET siblings of the model's prediction (IV and OOV), and all WORDNET ancestors of the model's prediction, up to but excluding the first common ancestor with the IMAGENET label (IV and OOV). Finally, we use CLIP as an open-world classifier to score each of the proposed labels. If the highest scoring class is not included in the IMAGENET label set, we consider the shown entity to be OOV and conclude that this error was indeed fine-grained OOV.
We visualize the portion and number of multi-label errors categorized as fine-grained OOV in Fig. 11. Interestingly, we observe that fine-grained OOV errors are not only much more prevalent in artifacts but also that their portion increases much more quickly with MLA for artifacts. We hypothesize that this is due to the IMAGENET labels covering organisms occurring in the validation set much more comprehensively than artifacts, and many images of artifacts being cluttered and full of
other (OOV) objects. We note that this might be a reason why trained humans still outperform even state-of-the-art models on artifacts but not organisms (Shankar et al., 2020). Interestingly, we observe that, across pretraining datasets and model sizes, the number of fine-grained OOV errors for artifact drops much quicker with MLA above around  $93\%$  MLA (see confidence intervals in Fig. 11 right).
Figure 11: Portion (left) and number (right) of MLA errors identified as fine-grained  $OOV$  by group - organisms (green) and artifacts (red).  $95\%$  confidence interval linear fit is shown on the right. For artifacts, models are divided at  $93\%$  MLA.
3.5 Non-Prototypical Instances
Many concepts described by any one IMAGENET class label are broad, depend on social and geographic factors, and change over time. Combined with the search-based data collection process, this has led to a biased dataset with skewed representations of the concepts described by the contained classes. Therefore it is unsurprising that models perform worse on non-prototypical instances or pictures of any given object, i.e., instances that, while clearly belonging to a given class, can be considered outliers in the IMAGENET distribution of that class.
Figure 12: Image with label whistle, but predcition perfume.
For example, in Fig. 12 we see a non-prototypical whistle. We believe it is interesting to track progress on these hard, non-prototypical instances as they are likely to be a good indicator to what extent a model has learned to recognize the underlying concepts of a class. However, defining what constitutes a non-prototypical instance of any class is hard and to a large extent subjective. Fortunately, this error
type is independent of the (incorrect) prediction made by the model, allowing us to directly leverage the manual categorization of non-prototypical images by Vasudevan et al. (2022). We thus implicitly decide that all images that are classified correctly by the state-of-the-art models (ViT-3B and Greedy Soups) they considered are not sufficiently non-prototypical to explain an error by another model.
Portion of Group's Multi-Label Errors  $[\%]$
Figure 13: Portion (left) and number (right) of MLA errors identified as non-prototypical sample.
We visualize the number of errors caused by non-prototypical instances in Fig. 13. Interestingly and in contrast to all other error types, there is no strong correlation between performance on non-prototypical examples and overall MLA. This suggests that these non-prototypical examples are not inherently hard to classify for all models. Further surprisingly, non-prototypical examples account for a very similar portion of errors for artifacts and organisms, despite the appearance of artifacts of the same class varying significantly more, which we expected would lead to a larger portion of artifact errors being explained by their non-prototypical appearance.
3.6 Spurious Correlations
Entities of many classes frequently co-occur with features that have no causal relationship to that class. For example, oxen are frequently observed on green pastures (such as in Fig. 4). We call these correlated features (Neuhaus et al., 2022). Inherently this is not a problem, however, it has been observed that models frequently rely on these correlated features to make predictions (Choi et al., 2012; Beery et al., 2018; Neuhaus et al., 2022; Moayeri et al., 2022), for example, by predicting ox when shown only a green pasture or camel when shown
Figure 14: An image with label ski mask (and also multi-class label alp), but prediction ski.
a cow in the desert. When correlated features lead to prediction errors, we call these spurious correlations. Here, we focus on the case where the presence of a correlated spurious feature causes a misclassification to the correlated class, despite no corresponding entity being shown. In Fig. 14, we show an example of such a spurious correlation where the presence of a ski mask and alps causes a model to predict ski, despite the image containing no skis. We do not consider the error mode, where the absence of a correlated feature causes the model to not predict the correlated class, despite a corresponding entity being shown, investigated by Singla and Feizi (2022); Moayeri et al. (2022).
To identify errors caused by spurious correlations, we identify pairs of commonly co-occurring classes and then categorize errors as spurious correlations if an incorrect model prediction and a multi-label form such a co-occurrence pair. More concretely, we first extract all pairs of co-occurring labels from the RealL multi-label annotations (Beyer et al., 2020), excluding samples we evaluate on. We then filter out pairs that either belong to the same superclass as defined in §3.3 or only co-occur once. Using
Figure 15: Portion (left) and number (right) of MLA errors caused by spurious correlations by group. A  $95\%$  confidence interval linear fit is shown on the right. For artifacts, models are divided into those with more and less than  $93\%$  MLA.
this process, we extract 13090 label pairs from 6622 images with more than one label, yielding 1019 unique co-occurrence pairs after filtering, which indicate a spurious correlation if they occur.
We visualize the portion and number of errors caused by spurious correlation in Fig. 15. We observe that artifacts and organisms follow notably different trends. Not only is the portion of errors caused by spurious correlations much larger for artifacts than organisms, but it also increases with MLA for artifacts (at an increased rate for higher MLA), while it stays roughly constant for organisms. For state-of-the-art models, spurious correlations explain up to  $15\%$  of errors on artifacts making them the second largest error source we identify.
4 Analysis of Model Errors on ImageNet
In this section, we discuss global trends, analyze their interaction with architecture and training choices, and validate our automatic pipeline against the manual analysis of Vasudevan et al. (2022).
Model Failures After removing all minor (see §3.3 and §3.4) and explainable (see §3.5 and §3.6) errors, from the multi-label errors (MLE), we are left with a set of particularly severe, unexplained model failures (MLF). In Fig. 16 we visualize the portion of these unexplained model failures over multi-label accuracy (MLA) and standard top-1 accuracy, again split by artifact (red) and organism (green). A ratio of 1 corresponds to none of the MLEs being explainable by our pipeline, while a ratio of 0 corresponds to all MLEs being explainable,
Figure 16: Portion of MLA (left) and top-1 (right) errors that can not be explained and are thus considered model failures - organisms (green) and artifacts (red).  $95\%$  confidence interval linear fit is shown shaded. Models are divided by MLA (at  $93\%$ ) and top-1 accuracy (at  $80\%$ ) for linear fits.
thus consisting only of less severe error types. Surprisingly, both top-1 accuracy and MLA are pessimistic when it comes to reporting model progress, with unexplained model failures decreasing at a much quicker rate than both top-1 errors and MLE.
Further, we observe different error distributions for organisms and artifacts. Firstly, the portion of model failures is much higher for artifacts than for organisms. Secondly, while the portion of unexplained errors decreases for both organisms and artifacts with MLA and top-1 accuracy, indicating that as models get more accurate they make not only less but also less severe errors, the rate of this change differs. For artifacts, this decrease is initially slower but then at around  $93\%$  MLA or  $80\%$  top-1 error, the portion of severe model failures starts to drop rapidly (roughly three times as fast as before), while the decrease is roughly linear for organisms. This phase change becomes particularly apparent when viewing the trend over MLA, where it is even visible for organisms.
4.1 (Pre-)training Dataset
Throughout §3, we have illustrated the pretraining dataset size with the marker hue. Generally, we observe that conditioned on identical MLA, the effect of pre-training dataset size is rather modest. Here, we divide the 12 datasets we consider into 4 categories from "small" (<5M) to "xlarge" (>500M) depending on the number of included images, illustrating separate fits in Fig. 17, with a darker shade corresponding to a larger dataset. We observe that across a wide range of accuracies, larger pretraining datasets lead to a faster reduction of model failures with MLA for both artifacts and organisms. While this effect is partially explained by larger datasets more frequently leading to higher MLA, it is also observed for narrow MLA ranges. Considering individual error types, we observe that in particular fine-grained errors are more frequent for larger pretraining datasets. We hypothesize that this is due to these larger datasets leading to better underlying representations, allowing the model to succeed in the coarse classification for harder samples, while still failing
Figure 17: Portion of model failures (MLF) over MLA depending on pretraining dataset size with  $95\%$  confidence intervals for models with at least  $82\%$  MLA. Confidence intervals are shown darker for larger pretraining dataset sizes.
on the fine-grained distinctions.
4.2 Model Architecture
In Fig. 18, we again show the portion of unexplained model failures over MLA, this time broken down by architecture type and colored by the number of parameters (larger models have darker hues). To investigate whether modern convolutional architectures (ConvNeXts (Liu et al., 2022)) and vision transformers (ViT (Dosovitskiy et al., 2021), Swin (Liu et al., 2021), EVA (Fang et al., 2023)) exhibit different trends in terms of model failures, we focus on state-of-the-art models with at least  $92\%$  MLA that were trained using a dataset with at least 50M images, leaving 28 CNN-based and 79 transformer-based models. While we observe clear trends of larger models performing better, the rate of model failures seems to be largely independent of model size when conditioned on MLA. We do observe a slight trend where the model failure rate of transformers decreases faster with MLA for organisms but slower for artifacts when compared to
Figure 18: Portion of model failures (MLF) over MLA depending on model architecture.  $95\%$  confidence interval for transformer-based models is shown darker and for CNN-based models lighter.
ConvNeXts. We speculate that this might be due to the ConvNeXts leveraging textures more heavily (Geirhos et al., 2019), which could be particularly helpful for fine-grained class distinctions on organisms leading to a larger portion of multi-label errors being model failures at the same MLA.
4.3 Comparison to Vasudevan et al. (2022)
To evaluate the alignment of our automated error analysis with a panel of human experts, we compare its results to those of Vasudevan et al. (2022) on both networks they consider, here for ViT-3B and in App. A for Greedy Soups.
Comparing the results in Table 1, we confirm that our pipeline is conservative, classifying  $16.4\%$  (62 / 378) of the errors assigned an explanation by Vasudevan et al. (2022) as model failures. On the remaining errors, our pipeline agrees with the panel of human experts in  $73.1\%$  of the cases. By manually inspecting the 85 differently classified errors, we determine that for only 32 of these  $(8.4\%)$  of all errors), the categorization by Vasudevan et al. (2022) is clearly preferable to ours. Furthermore, our pipeline encodes a minimal-severity bias, absent in Vasudevan et al. (2022)'s categorization. That is, when multiple error explanations would be valid, our pipeline consistently chooses the least severe one. This highlights a further advantage of our approach. While any human judgment is inherently subjective and thus prone to differ between expert panels or even for the same panel over time, as acknowledged by Vasudevan et al. (2022), our approach is consistent and repeatable.
Table 1: Comparison of our automated analysis of the errors of a ViT-3B model with the manual annotations from (Vasudevan et al., 2022). Rows correspond to the error categories assigned by a human expert while columns correspond to the error types assigned by our automated pipeline.
Error categories	FG	FG OOV	Non-prot.	Spur. Corr.	Model failures	Total (row)
Fine-grained	192	15	0	10	25	242
Fine-grained OOV	9	20	0	11	14	54
Non-prototypical	13	2	12	3	0	30
Spurious Correlation	10	12	0	7	23	52
Total (col)	224	49	12	31	62	378
Observing similar trends for Greedy Soups in App. A, we confirm that on all models for which human expert error categorizations are available, our automated pipeline is well aligned with their judgment while providing consistent, repeatable, and conservative error categorizations.
In this section, we briefly reflect on the limitations of our work.
Personal Bias Our choices in the implementation of the error analysis pipeline reflect our personal biases and decisions, such as the superclasses we picked in §3.3. This is not only limited to our personal biases, but, as our pipeline relies on prior work in several places, it also encodes the biases of their authors. For example, we rely on the class overlap mappings from Vasudevan et al. (2022) and multi-class labels by Shankar et al. (2020) and Vasudevan et al. (2022).
Extension to New Datasets To be applied to a new dataset, our pipeline requires multi-label and non-prototypical annotations and superclass definitions. While less precise than using a human-annotated gold standard, multi-label annotations could be sourced using a Re-Label (Yun et al., 2021) approach, allowing (given sufficient scale) spurious correlation pairs to be extracted from co-occurrence frequencies. After defining superclasses manually as required by their very definition, this would allow all error types except for the rare non-prototypical instances to be categorized. With the rest of the pipeline in place, non-prototypical instances could be annotated by reviewing the un categorized errors shared by multiple well-performing models. However, while feasible these steps still require a significant amount of manual work. We showcase a (partial) adaption to the similar IMAGENET-A dataset in App. C.
As state-of-the-art models come close to and exceed human performance on IMAGENET, focus is increasingly shifting towards understanding the last remaining errors. Towards this goal, we propose an automated error categorization pipeline that we use to study the distribution of different error types across 962 models, of different scales, architectures, training methods, and pre-training datasets. We distinguish between minor errors, constituting failures on fine-grained class distinctions both when the ground truth was in and out-of-vocabulary, explainable errors, constituting failures on non-prototypical examples and due to spurious correlations, and unexplained model failures, constituting all remaining errors. We find that even MLA is a pessimistic measure of model progress with the portion of severe errors quickly decreasing with multi-label-accuracy. Further, we find that organism and artifact classes exhibit very different trends and prevalences of error types. For example, we observe that fine-grained class distinctions are a much bigger challenge for organisms than for artifacts, while artifacts suffer much more from spurious correlations and out-of-vocabulary errors, with these trends becoming increasingly pronounced as models become more accurate. We believe that such an analysis of a new method's effects on model error distributions can become an important part of the evaluation pipeline and lead to insights guiding future research in computer vision.
We thank our anonymous reviewers for their constructive comments and insightful feedback.
This work has been done as part of the EU grant ELSA (European Lighthouse on Secure and Safe AI, grant agreement no. 101070617) and the SERI grant SAFEAI (Certified Safe, Fair and Robust Artificial Intelligence, contract no. MB22.00088). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or European Commission. Neither the European Union nor the European Commission can be held responsible for them.
The work has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERI).
J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li, "Imagenet: A large-scale hierarchical image database," in Proc. of CVPR, 2009.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and L. Fei-Fei, "Imagenet large scale visual recognition challenge," Int. J. Comput. Vis., vol. 115, no. 3, 2015.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Proc. of NeurIPS, 2012.
M. Tan, R. Pang, and Q. V. Le, "Efficientdet: Scalable and efficient object detection," in Proc. of CVPR, 2020.
S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, "Image segmentation using deep learning: A survey," IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 7, 2022.
L. Beyer, O. J. Henaff, A. Kolesnikov, X. Zhai, and A. van den Oord, "Are we done with imagenet?" ArXiv preprint, vol. abs/2006.07159, 2020.
D. Tsipras, S. Santurkar, L. Engstrom, A. Ilyas, and A. Madry, "FromImagenet to image classification: Contextualizing progress on benchmarks," in Proc. of ICML, vol. 119, 2020.
V. Shankar, R. Roelofs, H. Mania, A. Fang, B. Recht, and L. Schmidt, "Evaluating machine accuracy on imagenet," in Proc. of ICML, vol. 119, 2020.
G. V. Horn, S. Branson, R. Farrell, S. Haber, J. Barry, P. Ipeirotis, P. Perona, and S. J. Belongie, "Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection," in Proc. of CVPR, 2015.
C. G. Northcutt, A. Athalye, and J. Mueller, "Pervasive label errors in test sets destabilize machine learning benchmarks," in Proc. of NeurIPS Datasets and Benchmarks, 2021.
H. S. Lee, A. A. Agarwal, and J. Kim, "Why do deep neural networks still not recognize these images?: A qualitative analysis on failure cases of imagenet classification," ArXiv preprint, vol. abs/1709.03439, 2017.
V. Vasudevan, B. Caine, R. G. Lopes, S. Fridovich-Keil, and R. Roelofs, "When does dough become a bagel? analyzing the remaining mistakes on imagenet," in Proc. of NeurIPS, 2022.
S. Yun, S. J. Oh, B. Heo, D. Han, J. Choe, and S. Chun, "Re-labeling imagenet: From single to multi-labels, from global to localized labels," in Proc. of CVPR, 2021.
H. Mania, J. Miller, L. Schmidt, M. Hardt, and B. Recht, "Model similarity mitigates test set overuse," in Proc. of NeurIPS, 2019.
R. Geirhos, K. Meding, and F. A. Wichmann, "Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency," in Proc. of NeurIPS, 2020.
A Extended Comparison to Vasudevan et al. (2022)
Similarly to the analysis in Sec. 4.3, we now evaluate our automated error classification pipeline on Greedy Soups, the only other model for which manual error categorizations exist (Vasudevan et al., 2022), showing the results in Table 2. We do not assign an explanation to  $11.6\%$  (29 / 249) of the errors, classifying them as model failures. On the remaining errors, our pipeline agrees with the human expert annotation in  $74.1\%$  (163 / 220) of the cases. After a manual review of the remaining 57 differently classified errors, we determine that the categorization provided by Vasudevan et al. (2022) is clearly preferable to ours for only 26 samples ( $10.4\%$  of all errors). Overall, the results for Greedy Soups are very similar to those for ViT-3B in §4.3, thus further validating our modeling choices. Therefore, we believe that our automatic categorization is indeed aligned with the opinion of human experts and note that an efficient and automatic classification pipeline is valuable even if it is imperfect, as it enables the study of models and trends at scale.
Table 2: Comparison of our automated analysis of the errors of a Greedy Soups model with the manual annotations from (Vasudevan et al., 2022). Rows correspond to the error categories assigned by a human expert while columns correspond to the error types assigned by our automated pipeline.
Error categories	FG	FG OOV	Non-prot.	Spur. Corr.	Model failures	Total (row)
Fine-grained	139	14	1	7	11	172
Fine-grained OOV	7	8	0	5	6	26
Non-prototypical	8	1	8	2	0	19
Spurious Correlation	7	5	0	8	12	32
Total (col)	161	28	9	22	29	249
In addition to the analysis presented in the main paper, which focused on the portion of errors due to the different error categories for artifacts and organisms, we provide more detailed results in this section, considering absolute error counts as well as portions for artifacts, organisms, the remaining classes, and all classes jointly.
Figures 19-24 contain these statistics grouped by error type:
- Fig. 19 - top-1 errors caused by class overlap (§3.1);
- Fig. 20 - top-1 errors due to missing multi-label annotations (§3.2);
Fig. 21 - fine-grained multi-label errors (§3.3);
- Fig. 22 - fine-grained out-of-vocabulary multi-label errors (§3.4);
- Fig. 23 - multi-label errors due to non-prototypical instances (§3.5);
- Fig. 24 - multi-label errors due to various spurious correlations (§3.6).
We show the number of errors in relative and absolute terms. In each figure, subfigures (a) and (b) show the results for each group of classes separated in "artifacts", "organisms" and "others" (the rest of the labels), where for (a) the relative portions of top-1 or multi-label errors are computed per group, i.e., by diving by the number of top-1 or multi-label errors in the respective group. Red hues again correspond to artifacts, green hues – to organisms, and blue hues – to the remaining classes (others). Subfigures (c) and (d) present the relative and absolute statistics for all classes jointly.
(a) Relative portion per group.
(b) Abs. number per group.
Figure 19: Top-1 errors caused by class overlap. The relative portion of these errors on "others" is the smallest and the trend is noisier compared to "artifacts" and "organisms". This is possibly due to the low absolute number of classes in this group (68 out of 1000). In absolute terms, the number of this kind of error remains relatively constant.
(c) Relative portion for all samples.
(d) Abs. number for all samples.