SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Momchil Peychev,*Mark Niklas Muller,*Marc Fischer, Martin Vechev
ETH Zurich, Switzerland
{momchil.peychev, mark.mueller, marc.fischer, martin.vechev}@inf.ethz.ch
While the ImageNet dataset has been driving computer vision research over the past decade, significant label noise and ambiguity have made top-1 accuracy an insufficient measure of further progress. To address this, new label-sets and evaluation protocols have been proposed for ImageNet showing that state-of-the-art models already achieve over  $95\%$  accuracy and shifting the focus on investigating why the remaining errors persist. Recent work in this direction employed a panel of experts to manually categorize all remaining classification errors for two selected models. However, this process is time-consuming, prone to inconsistencies, and requires trained experts, making it unsuitable for regular model evaluation thus limiting its utility. To overcome these limitations, we propose the first automated error classification framework, a valuable tool to study how modeling choices affect error distributions. We use our framework to comprehensively evaluate the error distribution of over 900 models. Perhaps surprisingly, we find that across model architectures, scales, and pre-training corpora, top-1 accuracy is a strong predictor for the portion of all error types. In particular, we observe that the portion of severe errors drops significantly with top-1 accuracy indicating that, while it underreports a model's true performance, it remains a valuable performance metric. We release all our code at https://github.com/eth-sri/automated-error-analysis.
IMAGENET (Deng et al., 2009; Russakovsky et al., 2015) has established itself as one of the most influential and widely used datasets in computer vision, driving progress in object recognition (Krizhevsky et al., 2012), object detection (Tan et al., 2020), and image segmentation (Minaee et al., 2022). As state-of-the-art models have come close to, and by some metrics exceeded, human performance, the significance of further progress in top-1 and top-5 accuracy has, been questioned in the face of label errors and systematic biases (Beyer et al., 2020; Tsipras et al., 2020).
The most severe such bias is the lack of multi-label annotations in the original IMAGENET dataset, with recent studies finding that roughly a fifth of images show multiple entities. While state-of-the-art models have learned to exploit labeling biases on these images (Tsipras et al., 2020), best practices have shifted to reporting multi-label accuracy (MLA) computed using new multi-label annotations (Beyer et al., 2020; Shankar et al., 2020). Further, many IMAGENET and especially organism classes, are hard to distinguish even by trained humans (Horn et al., 2015; Northcutt et al., 2021a; Lee et al., 2017), leading to persistent labeling errors.
In the face of these challenges and with state-of-the-art models exceeding  $95\%$  MLA, the focus has increasingly shifted towards analyzing and understanding the remaining model errors instead of blindly pursuing improvements in headline accuracy numbers. To this end, Vasudevan et al. (2022) review all remaining errors of two state-of-the-art models using a panel of experts and classify them
with regards to both error category and severity. While they find that many of the remaining errors are minor or can be attributed to fine-grained class distinctions, they also find major classification errors that are not so easily explainable. We believe that tracking and analyzing the distribution of these error types over a large number of models can not only help us to better understand the impact of novel training techniques and architectures but also identify where the biggest challenges lie and thus how to address them. However, the manual review process employed by Vasudevan et al. (2022) has several issues, preventing it from being employed for a large scale or repeated study of model errors: (i) it is time-consuming even for precise models, making it infeasible to repeat for a large number of potentially less precise models, (ii) it requires (a panel of) expert reviewers which need to be trained on the fine-grained class distinctions, and (iii) it is inconsistent as different reviewers or even the same reviewers at different times might classify the same error differently.
This Work To overcome these challenges, we propose an automated error classification pipeline that we use to study the distribution of different types of errors across 962 models of different scales, architectures, training methods, and pre-training datasets. Our pipeline allows us to automatically detect all four error categories identified by Vasudevan et al. (2022): (i) fine-grained classification errors are detected using a set of 161 manually defined superclasses, (ii) fine-grained out-of-vocabulary errors are detected using a visual similarity based criterion for prediction quality and confirmation of their out-of-vocabulary nature using an open-world classifier, (iii) non-prototypical examples are identified using the exhaustive annotations by Vasudevan et al. (2022), and (iv) spurious correlations are detected using a co-occurrence-frequency based criterion.
Main Findings Our automated error classification pipeline allows us to, for the first time, study the distribution of different error types across a large number of models, leading to the following insights: (i) even MLA is a pessimistic measure of model progress with the portion of severe model failures quickly decreasing with MLA, (ii) this reduction of model failure rate with MLA is more pronounced for larger (pre-)training corpora, i.e., models trained on more data make less severe errors even at the same top-1 or multilabel accuracy, and (iii) organism and artifact classes exhibit very different trends and prevalences of error types, e.g., fine-grained classification errors are much more frequent for organisms than for artifacts, while artifacts suffer much more from spurious correlations and out-of-vocabulary errors. We believe that these insights can help guide future research in computer vision and that studying the effects of new methods on the resulting error distribution can become an important part of the evaluation pipeline.
Multi-Label Annotations While the IMAGENET dataset is annotated with a single label per image, many images have been found to contain multiple entities (Beyer et al., 2020; Shankar et al., 2020; Tsipras et al., 2020; Yun et al., 2021). Thus, multi-label accuracy (MLA) with respect to new multi-label annotations has been established as a more meaningful metric. Yun et al. (2021) generate pixel-wise labels for IMAGENET by directly applying a classification layer to image embeddings before spatial pooling. Beyer et al. (2020) collect Reassessed Labels (ReaL) for the whole IMAGENET validation set, by first identifying 6 models with high prediction coverage and accuracy, before manually annotating all images where these 6 models disagree using non-expert labelers. They discard 3163 images where these labelers disagreed. Tsipras et al. (2020) collect multi-label annotations for 10000 validation set images, and find that top-1 accuracy is  $10\%$  lower for multi- compared to single-label images while MLA accuracy is identical. Shankar et al. (2020) collect multi-label annotation for 40000 IMAGENET and IMAGENETv2 validation images by manually reviewing all predictions made by a diverse set of 72 models, using human expert labelers.
Label Errors Northcutt et al. (2021a) study label errors across 10 commonly used datasets, including IMAGENET, using a confident learning framework to identify potential errors before validating them with Mechanical Turk (MTurk). Studying the whole validation set, they report an error rate of over  $5.8\%$ . Lee et al. (2017) manually review 400 randomly selected classification errors of an ensemble model and find the IMAGENET label for a substantial portion to either be incorrect or not describe the main entity in the image. Vasudevan et al. (2022) reviewed all remaining errors for two state-of-the-art models with a panel of experts and found that of 676 reviewed model mistakes, 298 were either correct, ambiguous, or the original ground truth incorrect or problematic.
Error Analysis Recent work has focused on understanding the types of errors models still make on IMAGENET. To this end, one strand of work analyses the differences in errors between models. Mania et al. (2019) find that independently trained models make errors that correlate well beyond what can be expected from their accuracy alone. Geirhos et al. (2020) analyze the consistency between errors made by humans and CNNs on a 16 class version of IMAGENET. They observe that while CNNs make remarkably similar errors, the consistency between humans and CNNs barely goes beyond chance. Nguyen et al. (2021) analyze wide and deep ResNets and find that these exhibit different error patterns. Mania and Sra (2020) find that more precise models typically dominate less precise ones, i.e., their error set is a subset of that of less precise models. Lopes et al. (2022), in contrast, find that different pretraining corpora and training objectives can significantly increase error diversity with Andreassen et al. (2021) showing that this diversity reduces significantly during finetuning.
Vasudevan et al. (2022) focus on the errors current state-of-the-art models make, identifying four categories: (i) fine-grained errors describe a model's failure to distinguish between two very similar classes, (ii) fine-grained out-of-vocabulary errors occur when an image shows an entity not contained in the IMAGENET vocabulary and the model instead predicts a similar class, (iii) spurious correlations cause the model to predict a class that is not shown in the image in the presence of strongly correlated features, and (iv) non-prototypical instantiations of a class are not recognized by the model.
Datasets for Error Analysis In addition to work analyzing model errors on fixed datasets, there has been a growing interest in datasets specifically designed to highlight and thus analyze specific error types. Singla and Feizi (2022) and Moayeri et al. (2022) find that, for some classes, models rely heavily on correlated or spurious features suffering severely reduced accuracy if these are absent or removed. To study this effect, they introduce SALIENT IMAGENET (Singla and Feizi, 2022) and HARD IMAGENET (Moayeri et al., 2022), providing soft masks for causal and correlated features and segmentation masks, respectively. Hendrycks et al. (2021) propose IMAGENET-A, a dataset of 7500 single-entity, natural adversarial examples, which induce high-confidence misclassifications in a set of ResNet50 and belong to 200 IMAGENET classes that were chosen to minimize class overlap and the potential for fine-grained errors (see App. C for an analysis). Vasudevan et al. (2022) collect images where state-of-the-art models fail in an unexplained manner in IMAGENET-MAJOR. Taesiri et al. (2023) investigate the effect of zoom and crop on classifier performance and introduce IMAGENET-HARD as the set of images (from a range of IMAGENET-like datasets) that none of their considered models classified correctly for any crop. Idrissi et al. (2023) annotate IMAGENET images with respect to how they differ from what is prototypical for their class, introducing IMAGENET-X. This allows them to study the effect of these image-specific variations such as atypical pose, background, or lighting situation, on accuracy over a large number of models. In contrast, our (and Vasudevan et al. (2022)'s) work focuses more on systematic errors caused (partially) by labeling (set) choices rather than on what makes an individual image hard to classify.
3 Categorizing ImageNet Errors
In this section, we introduce our automated error classification pipeline, which aims to explain model errors by assigning it one of six error types. We consider errors that can not be explained in this way to be particularly severe model failures. While the definitions of our error types are heavily inspired by Vasudevan et al. (2022), we address three main issues of their manual approach with our automated pipeline: (i) it is time-consuming even for precise models and intractable for imprecise ones, (ii) it requires a panel of expert reviewers, typically not available, and (iii) it introduces inconsistencies due to human expert error, disagreement, or ambiguity.
Below, we first provide a brief overview of our pipeline (illustrated in Fig. 1), before discussing the different error types and how we identify them in more detail.
We consider errors due to overlapping class definitions (discussed in §3.1) and missing multi-label annotations (§3.2) to be the least severe, as they can be interpreted as labeling errors rather than classification errors. When a model predicts a class that is closely related to one of the labels, we call this a fine-grained classification error (§3.3), as the model succeeds in recognizing the type of entity but fails during the fine-grained distinction. When an image contains an entity that does not belong to any of the IMAGENET classes and a model predicts a label that is closely related to this (out-of-vocabulary) class, we call this a fine-grained out-of-vocabulary error (§3.4). We consider both types of fine-grained errors to be minor. If a model fails on an image that shows a very
Figure 1: We first remove errors w.r.t. the original IMAGENET labels caused by overlapping class definitions or missing multi-label annotations, yielding multi-label accuracy (MLA). We then, in this order, identify fine-grained misclassifications, fine-grained misclassifications where the true label of the main entity is not included in the IMAGENET labelset, non-prototypical examples of a given class, and spurious correlations. This leaves us with severe model failures that are unexplained by our categorization.
non-prototypical instance of a given class, we call this a non-prototypical error (§3.5). If a model predicts a class that commonly co-occurs with a ground-truth class but is not shown in the image, we attribute this error to spurious correlation and discuss it in §3.6. We denote both of these errors as explainable errors. If an error can be attributed to multiple causes, we assign the least severe category and thus design our pipeline to consider error types in order of increasing severity (see Fig. 1).
Throughout this section, we include examples and trends for the kinds of errors we discuss. We generally describe these separately for images with a ground-truth class describing an organism (410 of 1000 classes) and artifacts (522 classes) (in line with previous work (Shankar et al., 2020)), as we observe that these two groups of classes exhibit different error patterns. These two groups account for almost all IMAGENET classes, with the remaining 68 classes being assigned the group "other". Where we find interesting trends, we further distinguish models by their (pre-)training dataset ranging in size from one million (IMAGENET) to multiple billion images (INSTAGRAM, LAION-2B), their architecture, including a broad range of MLPs, CNNs, transformer-based, and hybrid models, and their (pre-) training method. For the full details on all 962 models we consider, please refer to App. F. We provide more examples of every error type in App. E and more detailed trends in App. B.
3.1 Class Overlap
Prior work has established that a small number of IMAGENET classes exhibit extreme overlap (Northcutt et al., 2021b; Vasudevan et al., 2022). We illustrate one such example in Fig. 2: tusker is defined as "an animal with tusks", which describes a strict superset of african elephant and has a significant overlap with indian elephant (females have short and sometimes no tusks). To avoid penalizing a model for correct predictions that do not match the ground truth, we consider all predictions describing a superset or equivalent of the ground-truth class to be correct. For example, we accept tusker for an image labeled african
elephant, but not vice-versa, as the latter might actually show a boar. We follow the mappings of equivalence and containment from Vasudevan et al. (2022) and refer to their App. C for full details.
Figure 2: Venn-Diagram of the tusker, indian elephant, and african elephant classes.
In Fig. 3, we visualize the portion and number of top-1 errors identified to be correct, separating images with a ground-truth class describing an organism (green hues) and artifacts (red hues) and encoding the pre-training dataset size by assigning darker colors to larger datasets. While trends for artifacts and organisms seem very similar for portions of top-1 errors, we observe a clear difference when looking at their absolute numbers. There, two
Figure 3: Portion (left) and number (right) of top-1 errors caused by class overlap by group - organisms (green) and artifacts (red). A  $95\%$  confidence interval linear fit is shown on the right.
competing effects are at play. With increasing accuracy, more samples get classified correctly, in-
2According to the Merriam-Webster Dictionary.
cluding as overlapping classes, thus increasing the number of such errors. On the other hand, more accurate models leverage labeling biases to pick the "correct" among equivalent classes (Tsipras et al., 2020), thus reducing the number of such errors. While the former effect seems to dominate for artifacts, the latter dominates for organisms.
3.2 Missing Multi-Label Annotations
While the IMAGENET dataset is annotated with a single label per image, many images contain multiple entities (Beyer et al., 2020; Shankar et al., 2020; Tsipras et al., 2020). Consequently, a model predicting the class of any such entity, different from the original IMAGENET label, is considered incorrect when computing top-1 accuracy. For example, the image shown in Fig. 4 contains ox, barn, and fence while only being labeled ox. To remedy this issue, multi-label accuracy (MLA) considers all shown classes, according to some multi-label annotation, to be correct. In this work, we use the annotations collected by Shankar et al. (2020) and improved by Vasudevan et al. (2022) combined with the mean class-wise accuracy definition of MLA (Shankar et al., 2020).
Figure 4: Image with label ox, but also showing the classes barn and fence. Example from Yun et al. (2021)
We visualize the portion and number of top-1 errors that turn out to be correct under multi-label evaluation in Fig. 5. When looking at the portion of errors (Fig. 5 left), we observe a similar trend for organisms and artifacts with missing multi-label annotations accounting for an increasing portion of top-1 errors as MLA increases and artifacts consistently more affected than organisms. When looking at the absolute numbers, we again observe the number of errors explained by multi-label annotations to first increase with ac
Figure 5: Portion (left) and number (right) of top-1 errors caused by missing multi-label annotations by group - organisms (green) and artifacts (red).
curacy as models become more precise, before decreasing again as models start to leverage labeling biases (Tsipras et al., 2020). As missing multi-label annotations explain up to  $60\%$  of model errors, we henceforth use MLA instead of top-1 accuracy as the reference for model comparison.
3.3 Fine-Grained Classification Errors
Many of the IMAGENET classes and especially the organism classes are very similar, making their fine-grained distinction challenging even for trained humans. For example, Fig. 6 is classified to cornet while actually showing a french horn, both brass wind instruments. While these errors can be corrected for the relatively small validation set using expert human judgment (Horn et al., 2015; Vasudevan et al., 2022), the much larger training set remains uncorrected. In this light, we believe a comprehensive model evaluation and comparison, should consider the failure to distinguish between very similar classes to be less severe than the failure to distinguish very different classes.
Figure 6: Image labeled french horn, but predicted to show a cornet.
To automatically detect such fine-grained classification errors, we manually review all 1000 IMAGENET classes and define semantically similar superclasses guided by whether an untrained human could reasonably confuse them. We obtain 161 superclasses, containing between 2 and 31 classes with an average of 6.7 and a median of 4. An additional 74 classes are not part of any superclass. 50 superclasses contain organisms (9.8 on average) and 101 contain artifacts (5.3 on average). We visualize the portion and num
Figure 7: Portion (left) and number (right) of MLA errors caused by fine-grained misclassifications by group - organisms (green) and artifacts (red).
ber of fine-grained errors in Fig. 7. As expected, we observe significantly more fine-grained errors for organisms than for artifacts, explaining up to  $88\%$  and  $50\%$  of multi-label errors, respectively.
Figure 8: To confirm a fine-grained out-of-vocabulary error we proceed as follows. Given a misclassified input image, we first retrieve the 10 most visually similar images from the IMAGENET train set. If the superclass of any of these images matches the model prediction, we conclude that an entity with a class similar to our prediction is shown in the image. To confirm that this entity is indeed out-of-vocabulary, we first collect a set of proposal labels from WordNet (Miller, 1992) which are partially in and out of vocabulary. Then, we use an Open World Classifier to score each of the proposed labels. If the highest scored among these is not included in the IMAGENET labelset, we consider the analyzed error to be OOV.
3.4 Fine-Grained Out-of-Vocabulary Errors
Often images contain entities that do not belong to any IMAGENET class, we call these out-of-vocabulary. For example, Fig. 9 shows an image of two blue-cheeked butterflyfish, which is a class not part of the IMAGENET labelset. Instead, the target label is coral reef, describing the background of the image. The classifier predicts rock beauty, which is a WORDNET child of butterflyfish (despite being part of the angelfish and not the butterflyfish family) and the closest class in the IMAGENET labelset to the blue-cheeked butterflyfish. While an optimal classifier could be expected to always predict a class from the intersection of the contained entities and the IMAGENET labelset, the
following is often observed in practice (Vasudevan et al., 2022). The classifier tries to classify a prominent entity in the image, but as it is not part of the IMAGENET labelset, it predicts a closely related class instead. We argue, that these cases should be treated at least as leniently as a fine-grained classification error, but they are harder to detect. As the entity the model tried to classify can not be assigned any IMAGENET label, it will not be included in any multi-label annotation and can thus not be captured by defining groups of closely related classes.
Figure 9: Image labeled coral reef, but prediction rock beauty (fish).
To still detect fine-grained out-of-vocabulary (OOV) errors, we propose the following approach, illustrated in Fig. 8 for the above example. We first retrieve the 10 most visually similar images from the IMAGENET training set using cosine similarity in the CLIP embedding space (Radford et al., 2021), in this case all labeled rock beauty. If a label of any of these images belongs to the same superclass as the model's prediction
Figure 10: Illustration of proposal labels. In-vocabulary labels are shown in bold. Classes in the same superclass as the prediction  $\square$  are shown in blue boxes, direct WORDNET siblings in orange  $\square$ , and ancestors not shared with the label  $\square$  in green  $\square$ .