Title: Automated Classification of Model Errors on ImageNet
Venue: NeurIPS 2023
Accepted: yes
Keywords: ImageNet

evaluation

error classification

error analysis
Abstract: While the ImageNet dataset has been driving computer vision research over the past decade, significant label noise and ambiguity have made top-1 accuracy an insufficient measure of further progress. To address this, new label-sets and evaluation protocols have been proposed for ImageNet showing that state-of-the-art models already achieve over 95% accuracy and shifting the focus on investigating why the remaining errors persist.

Recent work in this direction employed a panel of experts to manually categorize all remaining classification errors for two selected models. However, this process is time-consuming, prone to inconsistencies, and requires trained experts, making it unsuitable for regular model evaluation thus limiting its utility. To overcome these limitations, we propose the first automated error classification framework, a valuable tool to study how modeling choices affect error distributions. We use our framework to comprehensively evaluate the error distribution of over 900 models. Perhaps surprisingly, we find that across model architectures, scales, and pre-training corpora, top-1 accuracy is a strong predictor for the *portion* of all error types. In particular, we observe that the portion of severe errors drops significantly with top-1 accuracy indicating that, while it underreports a model's true performance, it remains a valuable performance metric.

We release all our code at https://github.com/eth-sri/automated-error-analysis.

Automated Classification of Model Errors on ImageNet

Abstract

While the ImageNet dataset has been driving computer vision research over the past decade, significant label noise and ambiguity have made top-1 accuracy an insufficient measure of further progress. To address this, new label-sets and evaluation protocols have been proposed for ImageNet showing that state-of-the-art models already achieve over 95% accuracy and shifting the focus on investigating why the remaining errors persist. Recent work in this direction employed a panel of experts to manually categorize all remaining classification errors for two selected models. However, this process is time-consuming, prone to inconsistencies, and requires trained experts, making it unsuitable for regular model evaluation thus limiting its utility. To overcome these limitations, we propose the first automated error classification framework, a valuable tool to study how modeling choices affect error distributions. We use our framework to comprehensively evaluate the error distribution of over 900 models. Perhaps surprisingly, we find that across model architectures, scales, and pre-training corpora, top-1 accuracy is a strong predictor for the portion of all error types. In particular, we observe that the portion of severe errors drops significantly with top-1 accuracy indicating that, while it underreports a model's true performance, it remains a valuable performance metric. We release all our code at https://github.com/eth-sri/automated-error-analysis .

1 Introduction

IMAGENET (Deng et al., 2009; Russakovsky et al., 2015) has established itself as one of the most influential and widely used datasets in computer vision, driving progress in object recognition (Krizhevsky et al., 2012), object detection (Tan et al., 2020), and image segmentation (Minaee et al., 2022). As state-of-the-art models have come close to, and by some metrics exceeded, human performance, the significance of further progress in top-1 and top-5 accuracy has, been questioned in the face of label errors and systematic biases (Beyer et al., 2020; Tsipras et al., 2020).

The most severe such bias is the lack of multi-label annotations in the original IMAGENET dataset, with recent studies finding that roughly a fifth of images show multiple entities. While state-of-the-art models have learned to exploit labeling biasses on these images (Tsipras et al., 2020), best practices have shifted to reporting multi-label accuracy (MLA) computed using new multi-label annotations (Beyer et al., 2020; Shankar et al., 2020). Further, many IMAGENET and especially organism classes, are hard to distinguish even by trained humans (Horn et al., 2015; Northcutt et al., 2021a; Lee et al., 2017), leading to persistent labeling errors.

In the face of these challenges and with state-of-the-art models exceeding 95% MLA, the focus has increasingly shifted towards analyzing and understanding the remaining model errors instead of blindly pursuing improvements in headline accuracy numbers. To this end, Vasudevan et al. (2022) review all remaining errors of two state-of-the-art models using a panel of experts and classify them

with regards to both error category and severity. While they find that many of the remaining errors are minor or can be attributed to fine-grained class distinctions, they also find major classification errors that are not so easily explainable. We believe that tracking and analyzing the distribution of these error types over a large number of models can not only help us to better understand the impact of novel training techniques and architectures but also identify where the biggest challenges lie and thus how to address them. However, the manual review process employed by Vasudevan et al. (2022) has several issues, preventing it from being employed for a large scale or repeated study of model errors: (i) it is time-consuming even for precise models, making it infeasible to repeat for a large number of potentially less precise models, (ii) it requires (a panel of) expert reviewers which need to be trained on the fine-grained class distinctions, and (iii) it is inconsistent as different reviewers or even the same reviewers at different times might classify the same error differently.

This Work To overcome these challenges, we propose an automated error classification pipeline that we use to study the distribution of different types of errors across 962 models of different scales, architectures, training methods, and pre-training datasets. Our pipeline allows us to automatically detect all four error categories identified by Vasudevan et al. (2022): (i) fine-grained classification errors are detected using a set of 161 manually defined superclasses, (ii) fine-grained out-of-vocabulary errors are detected using a visual similarity based criterion for prediction quality and confirmation of their out-of-vocabulary nature using an open-world classifier, (iii) non-prototypical examples are identified using the exhaustive annotations by Vasudevan et al. (2022), and (iv) spurious correlations are detected using a co-occurrence-frequency based criterion.

Main Findings Our automated error classification pipeline allows us to, for the first time, study the distribution of different error types across a large number of models, leading to the following insights: (i) even MLA is a pessimistic measure of model progress with the portion of severe model failures quickly decreasing with MLA, (ii) this reduction of model failure rate with MLA is more pronounced for larger (pre-)training corpora, i . e ., models trained on more data make less severe errors even at the same top-1 or multilabel accuracy, and (iii) organism and artifact classes exhibit very different trends and prevalences of error types, e . g ., fine-grained classification errors are much more frequent for organisms than for artifacts, while artifacts suffer much more from spurious correlations and out-of-vocabulary errors. We believe that these insights can help guide future research in computer vision and that studying the effects of new methods on the resulting error distribution can become an important part of the evaluation pipeline.

2 Related Work

Multi-Label Annotations While the IMAGENET dataset is annotated with a single label per image, many images have been found to contain multiple entities (Beyer et al., 2020; Shankar et al., 2020; Tsipras et al., 2020; Yun et al., 2021). Thus, multi-label accuracy (MLA) with respect to new multi-label annotations has been established as a more meaningful metric. Yun et al. (2021) generate pixel-wise labels for IMAGENET by directly applying a classification layer to image embeddings before spatial pooling. Beyer et al. (2020) collect Reassessed Labels (ReaL) for the whole IMAGENET validation set, by first identifying 6 models with high prediction coverage and accuracy, before manually annotating all images where these 6 models disagree using non-expert labelers. They discard 3 163 images where these labelers disagreed. Tsipras et al. (2020) collect multi-label annotations for 10 000 validation set images, and find that top-1 accuracy is 10% lower for multi- compared to single-label images while MLA accuracy is identical. Shankar et al. (2020) collect multi-label annotation for 40 000 IMAGENET and IMAGENETV2 validation images by manually reviewing all predictions made by a diverse set of 72 models, using human expert labelers.

Label Errors Northcutt et al. (2021a) study label errors across 10 commonly used datasets, including IMAGENET, using a confident learning framework to identify potential errors before validating them with Mechanical Turk (MTurk). Studying the whole validation set, they report an error rate of over 5 . 8% . Lee et al. (2017) manually review 400 randomly selected classification errors of an ensemble model and find the IMAGENET label for a substantial portion to either be incorrect or not describe the main entity in the image. Vasudevan et al. (2022) reviewed all remaining errors for two state-of-the-art models with a panel of experts and found that of 676 reviewed model mistakes, 298 were either correct, ambiguous, or the original ground truth incorrect or problematic.

Error Analysis Recent work has focused on understanding the types of errors models still make on IMAGENET. To this end, one strand of work analyses the differences in errors between models. Mania et al. (2019) find that independently trained models make errors that correlate well beyond what can be expected from their accuracy alone. Geirhos et al. (2020) analyze the consistency between errors made by humans and CNNs on a 16 class version of IMAGENET. They observe that while CNNs make remarkably similar errors, the consistency between humans and CNNs barely goes beyond chance. Nguyen et al. (2021) analyze wide and deep ResNets and find that these exhibit different error patterns. Mania and Sra (2020) find that more precise models typically dominate less precise ones, i . e ., their error set is a subset of that of less precise models. Lopes et al. (2022), in contrast, find that different pretraining corpora and training objectives can significantly increase error diversity with Andreassen et al. (2021) showing that this diversity reduces significantly during finetuning.

Vasudevan et al. (2022) focus on the errors current state-of-the-art models make, identifying four categories: (i) fine-grained errors describe a model's failure to distinguish between two very similar classes, (ii) fine-grained out-of-vocablary errors occur when an image shows an entity not contained in the IMAGENET vocabulary and the model instead predicts a similar class, (iii) spurious correlations cause the model to predict a class that is not shown in the image in the presence of strongly correlated features, and (iv) non-prototypical instantiations of a class are not recognized by the model.

Datasets for Error Analysis In addition to work analyzing model errors on fixed datasets, there has been a growing interest in datasets specifically designed to highlight and thus analyze specific error types. Singla and Feizi (2022) and Moayeri et al. (2022) find that, for some classes, models rely heavily on correlated or spurious features suffering severely reduced accuracy if these are absent or removed. To study this effect, they introduce SALIENT IMAGENET (Singla and Feizi, 2022) and HARD IMAGENET (Moayeri et al., 2022), providing soft masks for causal and correlated features and segmentation masks, respectively. Hendrycks et al. (2021) propose IMAGENET-A, a dataset of 7 500 single-entity, natural adversarial examples, which induce high-confidence misclassifications in a set of ResNet50 and belong to 200 IMAGENET classes that were chosen to minimize class overlap and the potential for fine-grained errors (see App. C for an analysis). Vasudevan et al. (2022) collect images where state-of-the-art models fail in an unexplained manner in IMAGENET-MAJOR. Taesiri et al. (2023) investigate the effect of zoom and crop on classifier performance and introduce IMAGENETHARD as the set of images (from a range of IMAGENET-like datasets) that none of their considered models classified correctly for any crop. Idrissi et al. (2023) annotate IMAGENET images with respect to how they differ from what is prototypical for their class, introducing IMAGENET-X. This allows them to study the effect of these image-specific variations such as atypical pose, background, or lighting situation, on accuracy over a large number of models. In contrast, our (and Vasudevan et al. (2022)'s) work focuses more on systematic errors caused (partially) by labeling (set) choices rather than on what makes an individual image hard to classify.

3 Categorizing ImageNet Errors

In this section, we introduce our automated error classification pipeline, which aims to explain model errors by assigning it one of six error types. We consider errors that can not be explained in this way to be particularly severe model failures . While the definitions of our error types are heavily inspired by Vasudevan et al. (2022), we address three main issues of their manual approach with our automated pipeline: (i) it is time-consuming even for precise models and intractable for imprecise ones, (ii) it requires a panel of expert reviewers, typically not available, and (iii) it introduces inconsistencies due to human expert error, disagreement, or ambiguity.

Below, we first provide a brief overview of our pipeline (illustrated in Fig. 1), before discussing the different error types and how we identify them in more detail.

We consider errors due to overlapping class definitions (discussed in §3.1) and missing multi-label annotations (§3.2) to be the least severe, as they can be interpreted as labeling errors rather than classification errors. When a model predicts a class that is closely related to one of the labels, we call this a fine-grained classification error (§3.3), as the model succeeds in recognizing the type of entity but fails during the fine-grained distinction. When an image contains an entity that does not belong to any of the IMAGENET classes and a model predicts a label that is closely related to this (out-of-vocabulary) class, we call this a fine-grained out-of-vocabulary error (§3.4). We consider both types of fine-grained errors to be minor. If a model fails on an image that shows a very

All Errors

Overlapping

Classes

§3.1

Multi

Label

§3.2

Fine

Grained

§3.3

Fine Grained

Out Of Vocabulary

§3.4

Non-Prototypical

§3.5

Spurious

Correlation

§3.6

No Error

Minor Error

Explainable Error

non-prototypical instance of a given class, we call this a non-prototypical error (§3.5). If a model predicts a class that commonly co-occurs with a ground-truth class but is not shown in the image, we attribute this error to spurious correlation and discuss it in §3.6. We denote both of these errors as explainable errors. If an error can be attributed to multiple causes, we assign the least severe category and thus design our pipeline to consider error types in order of increasing severity (see Fig. 1).

Throughout this section, we include examples and trends for the kinds of errors we discuss. We generally describe these separately for images with a ground-truth class describing an organism (410 of 1000 classes) and artifacts (522 classes) (in line with previous work (Shankar et al., 2020)), as we observe that these two groups of classes exhibit different error patterns. These two groups account for almost all IMAGENET classes, with the remaining 68 classes being assigned the group 'other'. Where we find interesting trends, we further distinguish models by their (pre-)training dataset ranging in size from one million (IMAGENET) to multiple billion images (INSTAGRAM, LAION-2B), their architecture, including a broad range of MLPs, CNNs, transformer-based, and hybrid models, and their (pre-) training method. For the full details on all 962 models we consider, please refer to App. F. We provide more examples of every error type in App. E and more detailed trends in App. B.

3.1 Class Overlap

Prior work has established that a small number of IMAGENET classes exhibit extreme overlap (Northcutt et al., 2021b; Vasudevan et al., 2022). We illustrate one such example in Fig. 2: tusker is defined as 'an animal with tusks' 2 , which describes a strict superset of african elephant and has a significant overlap with indian elephant (females have short and sometimes no tusks). To avoid penalizing a model for correct predictions that do not match the ground truth, we consider all predictions describing a superset or equivalent of the ground-truth class to be correct. For example, we accept tusker for an image labeled african

tusker

african

elephant

indian

elephant

elephant , but not vice-versa, as the latter might actually show a boar. We follow the mappings of equivalence and containment from Vasudevan et al. (2022) and refer to their App. C for full details.

In Fig. 3, we visualize the portion and number of top-1 errors identified to be correct, separating images with a groundtruth class describing an organism (green hues) and artifacts (red hues) and encoding the pre-training dataset size by assigning darker colors to larger datasets. While trends for artifacts and organisms seem very similar for portions of top-1 errors, we observe a clear difference when looking at their absolute numbers. There, two

50

65

80

95

Top-1 Accuracy [%]

0

5

10

Portion of Group's Top-1 Errors [%]

mlp

cnn

transformer

hybrid

50

65

80

95

Top-1 Accuracy [%]

0

50

100

# Correct Class Overlap Top-1 Errors

organisms

artifacts

competing effects are at play. With increasing accuracy, more samples get classified correctly, in-

cluding as overlapping classes, thus increasing the number of such errors. On the other hand, more accurate models leverage labeling biases to pick the 'correct' among equivalent classes (Tsipras et al., 2020), thus reducing the number of such errors. While the former effect seems to dominate for artifacts, the latter dominates for organisms.

3.2 Missing Multi-Label Annotations

While the IMAGENET dataset is annotated with a single label per image, many images contain multiple entities (Beyer et al., 2020; Shankar et al., 2020; Tsipras et al., 2020). Consequently, a model predicting the class of any such entity, different from the original IMAGENET label, is considered incorrect when computing top-1 accuracy. For example, the image shown in Fig. 4 contains ox , barn , and fence while only being labeled ox . To remedy this issue, multi-label accuracy (MLA) considers all shown classes, according to some multi-label annotation, to be correct. In this work, we use the annotations collected by Shankar et al. (2020) and improved by Vasudevan et al. (2022) combined with the mean class-wise accuracy definition of MLA (Shankar et al., 2020).

We visualize the portion and number of top-1 errors that turn out to be correct under multi-label evaluation in Fig. 5. When looking at the portion of errors (Fig. 5 left), we observe a similar trend for organisms and artifacts with missing multilabel annotations accounting for an increasing portion of top-1 errors as MLA increases and artifacts consistently more affected than organisms. When looking at the absolute numbers, we again observe the number of errors explained by multi-label annotations to first increase with ac-

50

65

80

95

Top-1 Accuracy [%]

0

20

40

60

80

Portion of Group's Top-1 Errors [%]

organisms

artifacts

mlp

cnn

transformer

hybrid

大

50

65

80

95

Top-1 Accuracy [%]

0

500

1000

# Top-1 Errors

organisms

artifacts

curacy as models become more precise, before decreasing again as models start to leverage labeling biases (Tsipras et al., 2020). As missing multi-label annotations explain up to 60% of model errors, we henceforth use MLA instead of top-1 accuracy as the reference for model comparison.

3.3 Fine-Grained Classification Errors

Many of the IMAGENET classes and especially the organism classes are very similar, making their fine-grained distinction challenging even for trained humans. For example, Fig. 6 is classified to cornet while actually showing a french horn , both brass wind instruments. While these errors can be corrected for the relatively small validation set using expert human judgment (Horn et al., 2015; Vasudevan et al., 2022), the much larger training set remains uncorrected. In this light, we believe a comprehensive model evaluation and comparison, should consider the failure to distinguish between very similar classes to be less severe than the failure to distinguish very different classes.

To automatically detect such fine-grained classification errors, we manually review all 1000 IMAGENET classes and define semantically similar superclasses guided by whether an untrained human could reasonably confuse them. We obtain 161 superclasses, containing between 2 and 31 classes with an average of 6 . 7 and a median of 4 . An additional 74 classes are not part of any superclass. 50 superclasses contain organisms ( 9 . 8 on average) and 101 contain artifacts ( 5 . 3 on average). We visualize the portion and num-

60

80

100

MLA [%]

0

30

60

90

Portion of Group's Multi-Label Errors [%]

organisms

artifacts

mlp

cnn

transformer

hybrid

#Multi-Label Errors

800

600

mlp

artifacts

cnn

400

4

transformer

口

hybrid

organisms

200

0

09

80

100

MLA [%]

ber of fine-grained errors in Fig. 7. As expected, we observe significantly more fine-grained errors for organisms than for artifacts, explaining up to 88% and 50% of multi-label errors, respectively.

Image

Embedding

Reference Dataset

Lab:

coral reef

Pred:

rock

beauty

Similar Images

rock

beauty

rock

beauty

rock

beauty

fish_rest

✓

fish_rest

✓

fish_rest

✓

Label

Super

Class

Open

World

Classifier

Label Proposals

OOV

angelfish

rock beauty

butterfly fish

anemone fish

tench

✓

✓

✗

✓

✗

✗

✓

Decision

Alternative Labels

Visually

Similar

Images

Fine-Grained

Error?

OOV?

3.4 Fine-Grained Out-of-Vocabulary Errors

Often images contain entities that do not belong to any IMAGENET class, we call these out-of-vocabulary. For example, Fig. 9 shows an image of two blue-cheeked butterflyfish, which is a class not part of the IMAGENET labelset. Instead, the target label is coral reef , describing the background of the image. The classifier predicts rock beauty , which is a WORDNET child of butterflyfish (despite being part of the angelfish and not the butterflyfish family) and the closest class in the IMAGENET labelset to the blue-cheeked butterflyfish. While an optimal classifier could be expected to always predict a class from the intersection of the contained entities and the IMAGENET labelset, the

following is often observed in practice (Vasudevan et al., 2022). The classifier tries to classify a prominent entity in the image, but as it is not part of the IMAGENET labelset, it predicts a closely related class instead. We argue, that these cases should be treated at least as leniently as a fine-grained classification error, but they are harder to detect. As the entity the model tried to classify can not be assigned any IMAGENET label, it will not be included in any multi-label annotation and can thus not be captured by defining groups of closely related classes.

To still detect fine-grained out-ofvocabulary (OOV) errors, we propose the following approach, illustrated in Fig. 8 for the above example. We first retrieve the 10 most visually similar images from the IMAGENET training set using cosine similarity in the CLIP embedding space (Radford et al., 2021), in this case all labeled rock beauty . If a label of any of these images belongs to the same superclass as the model's prediction

physical object

whole

unit

percoid

fish

butterfly fish

rock

beauty

angelfish

damselfish

anemone

fish

geological

formation

ridge

reef

coral

reef

(as defined in §3.3), we conclude that an entity similar to our prediction is shown in the image and proceed with our analysis. Otherwise, we conclude that the analyzed error is not fine-grained. To confirm that the shown entity is indeed out-of-vocabulary, we first collect a set of proposal labels from the following (illustrated in Fig. 10): all IMAGENET labels in the same superclass as the model's prediction (IV), all direct WORDNET siblings of the model's prediction (IV and OOV), and all WORDNET ancestors of the model's prediction, up to but excluding the first common ancestor with the IMAGENET label (IV and OOV). Finally, we use CLIP as an open-world classifier to score each of the proposed labels. If the highest scoring class is not included in the IMAGENET label set, we consider the shown entity to be OOV and conclude that this error was indeed fine-grained OOV.

We visualize the portion and number of multi-label errors categorized as finegrained OOV in Fig. 11. Interestingly, we observe that fine-grained OOV errors are not only much more prevalent in artifacts but also that their portion increases much more quickly with MLA for artifacts. We hypothesize that this is due to the IMAGENET labels covering organisms occurring in the validation set much more comprehensively than artifacts, and many images of artifacts being cluttered and full of

60

80

100

MLA [%]

0

5

10

15

20

25

30

Portion of Group's Multi-Label Errors [%]

organisms

artifacts

mlp

cnn

transformer

hybrid

60

80

100

MLA [%]

0

60

120

180

240

# Multi-Label Errors

organisms

artifacts

mlp

cnn

transformer

hybrid

other (OOV) objects. We note that this might be a reason why trained humans still outperform even state-of-the-art models on artifacts but not organisms (Shankar et al., 2020). Interestingly, we observe that, across pretraining datasets and model sizes, the number of fine-grained OOV errors for artifact drops much quicker with MLA above around 93% MLA (see confidence intervals in Fig. 11 right).

3.5 Non-Prototypical Instances

Many concepts described by any one IMAGENET class label are broad, depend on social and geographic factors, and change over time. Combined with the search-based data collection process, this has led to a biased dataset with skewed representations of the concepts described by the contained classes. Therefore it is unsurprising that models perform worse on non-prototypical instances or pictures of any given object, i . e ., instances that, while clearly belonging to a given class, can be considered outliers in the IMAGENET distribution of that class.

For example, in Fig. 12 we see a nonprototypical whistle . Webelieve it is interesting to track progress on these hard, nonprototypical instances as they are likely to be a good indicator to what extent a model has learned to recognize the underlying concepts of a class. However, defining what constitutes a non-prototypical instance of any class is hard and to a large extent subjective. Fortunately, this error

60

80

100

MLA [%]

0

1

2

Portion of Group's Multi-Label Errors [%]

mlp

cnn

transformer

hybrid

60

80

100

MLA [%]

0

5

10

15

# Multi-Label Errors

organisms

artifacts

type is independent of the (incorrect) prediction made by the model, allowing us to directly leverage the manual categorization of non-prototypical images by Vasudevan et al. (2022). We thus implicitly decide that all images that are classified correctly by the state-of-the-art models ( ViT-3B and Greedy Soups ) they considered are not sufficiently non-prototypical to explain an error by another model.

We visualize the number of errors caused by non-prototypical instances in Fig. 13. Interestingly and in contrast to all other error types, there is no strong correlation between performance on nonprototypical examples and overall MLA. This suggests that these non-prototypical examples are not inherently hard to classify for all models. Further surprisingly, non-prototypical examples account for a very similar portion of errors for artifacts and organisms, despite the appearance of artifacts of the same class varying significantly more, which we expected would lead to a larger portion of artifact errors being explained by their non-prototypical appearance.

3.6 Spurious Correlations

Entities of many classes frequently co-occur with features that have no causal relationship to that class. For example, oxen are frequently observed on green pastures (such as in Fig. 4). We call these correlated features (Neuhaus et al., 2022). Inherently this is not a problem, however, it has been observed that models frequently rely on these correlated features to make predictions (Choi et al., 2012; Beery et al., 2018; Neuhaus et al., 2022; Moayeri et al., 2022), for example, by predicting ox when shown only a green pasture or camel when shown

a cow in the desert. When correlated features lead to prediction errors, we call these spurious correlations. Here, we focus on the case where the presence of a correlated spurious feature causes a misclassification to the correlated class, despite no corresponding entity being shown. In Fig. 14, we show an example of such a spurious correlation where the presence of a ski mask and alp s causes a model to predict ski , despite the image containing no skis. We do not consider the error mode, where the absence of a correlated feature causes the model to not predict the correlated class, despite a corresponding entity being shown, investigated by Singla and Feizi (2022); Moayeri et al. (2022).

To identify errors caused by spurious correlations, we identify pairs of commonly co-occurring classes and then categorize errors as spurious correlations if an incorrect model prediction and a multi-label form such a co-occurrence pair. More concretely, we first extract all pairs of co-occurring labels from the ReaL multi-label annotations (Beyer et al., 2020), excluding samples we evaluate on. We then filter out pairs that either belong to the same superclass as defined in §3.3 or only co-occur once. Using

60

80

100

MLA [%]

0

5

10

15

20

Portion of Group's Multi-Label Errors [%]

organisms

artifacts

mlp

cnn

transformer

hybrid

60

80

100

MLA [%]

0

50

100

150

200

# Multi-Label Errors

organisms

artifacts

mlp

cnn

transformer

hybrid

this process, we extract 13 090 label pairs from 6 622 images with more than one label, yielding 1019 unique co-occurrence pairs after filtering, which indicate a spurious correlation if they occur.

We visualize the portion and number of errors caused by spurious correlation in Fig. 15. We observe that artifacts and organisms follow notably different trends. Not only is the portion of errors caused by spurious correlations much larger for artifacts than organisms, but it also increases with MLA for artifacts (at an increased rate for higher MLA), while it stays roughly constant for organisms. For state-of-the-art models, spurious correlations explain up to 15% of errors on artifacts making them the second largest error source we identify.

4 Analysis of Model Errors on ImageNet

In this section, we discuss global trends, analyze their interaction with architecture and training choices, and validate our automatic pipeline against the manual analysis of Vasudevan et al. (2022).

Model Failures After removing all minor (see §3.3 and §3.4) and explainable (see §3.5 and §3.6) errors, from the multilabel errors (MLE), we are left with a set of particularly severe, unexplained model failures (MLF). In Fig. 16 we visualize the portion of these unexplained model failures over multi-label accuracy (MLA) and standard top-1 accuracy, again split by artifact (red) and organism (green). A ratio of 1 corresponds to none of the MLEs being explainable by our pipeline, while a ratio of 0 corresponds to all MLEs being explainable,

60

80

100

MLA [%]

0

25

50

75

Portion of Group's Multi-Label Errors [%]

organisms

artifacts

mlp

cnn

transformer

hybrid

50

65

80

95

Top-1 Accuracy [%]

0

25

50

75

Portion of Group's Top-1 Errors

organisms

artifacts

thus consisting only of less severe error types. Surprisingly, both top-1 accuracy and MLA are pessimistic when it comes to reporting model progress, with unexplained model failures decreasing at a much quicker rate than both top-1 errors and MLE.

Further, we observe different error distributions for organisms and artifacts. Firstly, the portion of model failures is much higher for artifacts than for organisms. Secondly, while the portion of unexplained errors decreases for both organisms and artifacts with MLA and top-1 accuracy, indicating that as models get more accurate they make not only less but also less severe errors, the rate of this change differs. For artifacts, this decrease is initially slower but then at around 93% MLA or 80% top-1 error, the portion of severe model failures starts to drop rapidly (roughly three times as fast as before), while the decrease is roughly linear for organisms. This phase change becomes particularly apparent when viewing the trend over MLA, where it is even visible for organisms.

4.1 (Pre-)training Dataset

Throughout §3, we have illustrated the pretraining dataset size with the marker hue. Generally, we observe that conditioned on identical MLA, the effect of pre-training dataset size is rather modest. Here, we divide the 12 datasets we consider into 4 categories from 'small' ( < 5M) to 'xlarge' ( > 500M) depending on the number of included images, illustrating separate fits in Fig. 17, with a darker shade corresponding to a larger dataset. We observe that across a wide range of accuracies, larger pretraining datasets lead to a faster reduction of model failures with MLA for both artifacts and organisms. While this effect is partially explained by larger datasets more frequently leading to higher MLA, it is also observed for narrow MLA ranges. Considering individual error types, we observe that in particular fine-grained errors are more frequent for larger pretraining datasets. We hypothesize that this is due to these larger datasets leading to better underlying representations, allowing the model to succeed

60

80

100

MLA [%]

0

25

50

75

Portion of Group's Multi-Label Errors [%]

organisms

artifacts

small (

<

5M)

medium (

<

50M)

large (

<

500M)

xlarge (

>

500M)

in the coarse classification for harder samples, while still failing on the fine-grained distinctions.

4.2 Model Architecture

In Fig. 18, we again show the portion of unexplained model failures over MLA, this time broken down by architecture type and colored by the number of parameters (larger models have darker hues). To investigate whether modern convolutional architectures (ConvNeXts (Liu et al., 2022)) and vision transformers (ViT (Dosovitskiy et al., 2021), Swin (Liu et al., 2021), EVA (Fang et al., 2023)) exhibit different trends in terms of model failures, we focus on state-of-the-art models with at least 92% MLA that were trained using a dataset with at least 50M images, leaving 28 CNN-based and 79 transformer-based models. While we observe clear trends of larger models performing better, the rate of model failures seems to be largely independent of model size when conditioned on MLA. We do observe a slight trend where the model failure rate of transformers decreases faster with MLA for organisms but slower for artifacts when compared to

60

80

100

MLA [%]

0

25

50

75

Portion of Group's Multi-Label Errors [%]

organisms

artifacts

ConvNeXt

ViT

EVA

ConvNeXts. We speculate that this might be due to the ConvNeXts leveraging textures more heavily (Geirhos et al., 2019), which could be particularly helpful for fine-grained class distinctions on organisms leading to a larger portion of multi-label errors being model failures at the same MLA.

4.3 Comparison to Vasudevan et al. (2022)

To evaluate the alignment of our automated error analysis with a panel of human experts, we compare its results to those of Vasudevan et al. (2022) on both networks they consider, here for ViT-3B and in App. A for Greedy Soups .

Comparing the results in Table 1, we confirm that our pipeline is conservative, classifying 16 . 4% ( 62 / 378 ) of the errors assigned an explanation by Vasudevan et al. (2022) as model failures. On the remaining errors, our pipeline agrees with the panel of human experts in 73 . 1% of the cases. By manually inspecting the 85 differently classified errors, we determine that for only 32 of these ( 8 . 4% of all errors), the categorization by Vasudevan et al. (2022) is clearly preferable to ours. Furthermore, our pipeline encodes a minimal-severity bias, absent in Vasudevan et al. (2022)'s categorization. That is, when multiple error explanations would be valid, our pipeline consistently chooses the least severe one. This highlights a further advantage of our approach. While any human judgment is inherently subjective and thus prone to differ between expert panels or even for the same panel over time, as acknowledged by Vasudevan et al. (2022), our approach is consistent and repeatable.

Observing similar trends for Greedy Soups in App. A, we confirm that on all models for which human expert error categorizations are available, our automated pipeline is well aligned with their judgment while providing consistent, repeatable, and conservative error categorizations.

5 Limitations

In this section, we briefly reflect on the limitations of our work.

Personal Bias Our choices in the implementation of the error analysis pipeline reflect our personal biases and decisions, such as the superclasses we picked in §3.3. This is not only limited to our personal biases, but, as our pipeline relies on prior work in several places, it also encodes the biases of their authors. For example, we rely on the class overlap mappings from Vasudevan et al. (2022) and multi-class labels by Shankar et al. (2020) and Vasudevan et al. (2022).

Extension to New Datasets To be applied to a new dataset, our pipeline requires multi-label and non-prototypicality annotations and superclass definitions. While less precise than using a human-annotated gold standard, multi-label annotations could be sourced using a Re-Label (Yun et al., 2021) approach, allowing (given sufficient scale) spurious correlation pairs to be extracted from co-occurrence frequencies. After defining superclasses manually as required by their very definition, this would allow all error types except for the rare non-prototypical instances to be categorized. With the rest of the pipeline in place, non-prototypical instances could be annotated by reviewing the uncategorized errors shared by multiple well-performing models. However, while feasible these steps still require a significant amount of manual work. We showcase a (partial) adaption to the similar IMAGENET-A dataset in App. C.

6 Conclusion

As state-of-the-art models come close to and exceed human performance on IMAGENET, focus is increasingly shifting towards understanding the last remaining errors. Towards this goal, we propose an automated error categorization pipeline that we use to study the distribution of different error types across 962 models, of different scales, architectures, training methods, and pre-training datasets. We distinguish between minor errors, constituting failures on fine-grained class distinctions both when the ground truth was in and out-of-vocabulary, explainable errors, constituting failures on non-prototypical examples and due to spurious correlations, and unexplained model failures, constituting all remaining errors. We find that even MLA is a pessimistic measure of model progress with the portion of severe errors quickly decreasing with multi-label-accuracy. Further, we find that organism and artifact classes exhibit very different trends and prevalences of error types. For example, we observe that fine-grained class distinctions are a much bigger challenge for organisms than for artifacts, while artifacts suffer much more from spurious correlations and out-of-vocabulary errors, with these trends becoming increasingly pronounced as models become more accurate. We believe that such an analysis of a new method's effects on model error distributions can become an important part of the evaluation pipeline and lead to insights guiding future research in computer vision.

References

Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li, 'Imagenet: A large-scale hierarchical image database,' in Proc. of CVPR , 2009.

Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and L. Fei-Fei, 'Imagenet large scale visual recognition challenge,' Int. J. Comput. Vis. , vol. 115, no. 3, 2015.

Krizhevsky, I. Sutskever, and G. E. Hinton, 'Imagenet classification with deep convolutional neural networks,' in Proc. of NeurIPS , 2012.

Tan, R. Pang, and Q. V. Le, 'Efficientdet: Scalable and efficient object detection,' in Proc. of CVPR , 2020.

Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, 'Image segmentation using deep learning: A survey,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 44, no. 7, 2022.

Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, and A. van den Oord, 'Are we done with imagenet?' ArXiv preprint , vol. abs/2006.07159, 2020.

Tsipras, S. Santurkar, L. Engstrom, A. Ilyas, and A. Madry, 'From imagenet to image classification: Contextualizing progress on benchmarks,' in Proc. of ICML , vol. 119, 2020.

Shankar, R. Roelofs, H. Mania, A. Fang, B. Recht, and L. Schmidt, 'Evaluating machine accuracy on imagenet,' in Proc. of ICML , vol. 119, 2020.

V. Horn, S. Branson, R. Farrell, S. Haber, J. Barry, P. Ipeirotis, P. Perona, and S. J. Belongie, 'Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection,' in Proc. of CVPR , 2015.

G. Northcutt, A. Athalye, and J. Mueller, 'Pervasive label errors in test sets destabilize machine learning benchmarks,' in Proc. of NeurIPS Datasets and Benchmarks , 2021.

S. Lee, A. A. Agarwal, and J. Kim, 'Why do deep neural networks still not recognize these images?: A qualitative analysis on failure cases of imagenet classification,' ArXiv preprint , vol. abs/1709.03439, 2017.

Vasudevan, B. Caine, R. G. Lopes, S. Fridovich-Keil, and R. Roelofs, 'When does dough become a bagel? analyzing the remaining mistakes on imagenet,' in Proc. of NeurIPS , 2022.

Yun, S. J. Oh, B. Heo, D. Han, J. Choe, and S. Chun, 'Re-labeling imagenet: From single to multi-labels, from global to localized labels,' in Proc. of CVPR , 2021.

Mania, J. Miller, L. Schmidt, M. Hardt, and B. Recht, 'Model similarity mitigates test set overuse,' in Proc. of NeurIPS , 2019.

Geirhos, K. Meding, and F. A. Wichmann, 'Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency,' in Proc. of NeurIPS , 2020.

Nguyen, M. Raghu, and S. Kornblith, 'Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth,' in Proc. of ICLR , 2021.

Mania and S. Sra, 'Why do classifier accuracies show linear trends under distribution shift?' ArXiv preprint , vol. abs/2012.15483, 2020.

G. Lopes, Y. Dauphin, and E. D. Cubuk, 'No one representation to rule them all: Overlapping features of training methods,' in Proc. of ICLR , 2022.

Andreassen, Y. Bahri, B. Neyshabur, and R. Roelofs, 'The evolution of out-of-distribution robustness throughout fine-tuning,' ArXiv preprint , vol. abs/2106.15831, 2021.

Singla and S. Feizi, 'Salient imagenet: How to discover spurious features in deep learning?' in Proc. of ICLR , 2022.

Moayeri, S. Singla, and S. Feizi, 'Hard imagenet: Segmentations for objects with strong spurious cues,' in Proc. of NeurIPS , 2022.

Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, 'Natural adversarial examples,' in Proc. of CVPR , 2021.

R. Taesiri, G. Nguyen, S. Habchi, C. Bezemer, and A. Nguyen, 'Imagenet-hard: The hardest images remaining from a study of the power of zoom and spatial biases in image classification,' CoRR , vol. abs/2304.05538, 2023.

Y. Idrissi, D. Bouchacourt, R. Balestriero, I. Evtimov, C. Hazirbas, N. Ballas, P. Vincent, M. Drozdzal, D. Lopez-Paz, and M. Ibrahim, 'Imagenet-x: Understanding model mistakes with factor of variation annotations,' in Proc. of ICLR , 2023.

G. Northcutt, L. Jiang, and I. L. Chuang, 'Confident learning: Estimating uncertainty in dataset labels,' J. Artif. Intell. Res. , vol. 70, 2021.

A. Miller, 'WordNet: A lexical database for English,' in Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992 , 1992.

Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, 'Learning transferable visual models from natural language supervision,' in Proc. of ICML , vol. 139, 2021.

Neuhaus, M. Augustin, V. Boreiko, and M. Hein, 'Spurious features everywhere - large-scale detection of harmful spurious features in imagenet,' CoRR , vol. abs/2212.04871, 2022.

J. Choi, A. Torralba, and A. S. Willsky, 'Context models and out-of-context objects,' Pattern Recognit. Lett. , vol. 33, no. 7, 2012.

Beery, G. V. Horn, and P. Perona, 'Recognition in terra incognita,' in Proc. of ECCV , vol. 11220, 2018.

Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, 'A convnet for the 2020s,' in Proc. of CVPR , 2022.

Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, 'An image is worth 16x16 words: Transformers for image recognition at scale,' in Proc. of ICLR , 2021.

Liu, Y. Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, 'Swin transformer: Hierarchical vision transformer using shifted windows,' in Proc. of ICCV , 2021.

Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao, 'EVA: exploring the limits of masked visual representation learning at scale,' in Proc. of CVPR , 2023.

Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, 'Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness,' in Proc. of ICLR , 2019.

A Extended Comparison to Vasudevan et al. (2022)

Similarly to the analysis in Sec. 4.3, we now evaluate our automated error classification pipeline on Greedy Soups , the only other model for which manual error categorizations exist (Vasudevan et al., 2022), showing the results in Table 2. We do not assign an explanation to 11 . 6% ( 29 / 249 ) of the errors, classifying them as model failures. On the remaining errors, our pipeline agrees with the human expert annotation in 74 . 1% ( 163 / 220 ) of the cases. After a manual review of the remaining 57 differently classified errors, we determine that the categorization provided by Vasudevan et al. (2022) is clearly preferable to ours for only 26 samples ( 10 . 4% of all errors). Overall, the results for Greedy Soups are very similar to those for ViT-3B in §4.3, thus further validating our modeling choices. Therefore, we believe that our automatic categorization is indeed aligned with the opinion of human experts and note that an efficient and automatic classification pipeline is valuable even if it is imperfect, as it enables the study of models and trends at scale.

B Additional Results

In addition to the analysis presented in the main paper, which focused on the portion of errors due to the different error categories for artifacts and organisms, we provide more detailed results in this section, considering absolute error counts as well as portions for artifacts, organisms, the remaining classes, and all classes jointly.

Figures 19-24 contain these statistics grouped by error type:

Fig. 19 - top-1 errors caused by class overlap (§3.1);

Fig. 20 - top-1 errors due to missing multi-label annotations (§3.2);

Fig. 21 - fine-grained multi-label errors (§3.3);

Fig. 22 - fine-grained out-of-vocabulary multi-label errors (§3.4);

Fig. 23 - multi-label errors due to non-prototypical instances (§3.5);

Fig. 24 - multi-label errors due to various spurious correlations (§3.6).

We show the number of errors in relative and absolute terms. In each figure, subfigures (a) and (b) show the results for each group of classes separated in 'artifacts', 'organisms' and 'others' (the rest of the labels), where for (a) the relative portions of top-1 or multi-label errors are computed per group , i . e ., by diving by the number of top-1 or multi-label errors in the respective group. Red hues again correspond to artifacts, green hues - to organisms, and blue hues- to the remaining classes (others). Subfigures (c) and (d) present the relative and absolute statistics for all classes jointly.

50

65

80

95

Top-1 Accuracy [%]

0

5

10

Portion of Group's Top-1 Errors [%]

mlp

cnn

transformer

hybrid

50

65

80

95

Top-1 Accuracy [%]

0

50

100

# Top-1 Errors

organisms

artifacts

others

50

65

80

95

Top-1 Accuracy [%]

0

5

10

Portion of All Top-1 Errors [%]

mlp

cnn

transformer

hybrid

50

65

80

95

Top-1 Accuracy [%]

0

50

100

150

# Top-1 Errors

(a) Relative portion per group.

(b) Abs. number per group.

Figure 19: Top-1 errors caused by class overlap. The relative portion of these errors on 'others' is the smallest and the trend is noisier compared to 'artifacts' and 'organisms'. This is possibly due to the low absolute number of classes in this group (68 out of 1000). In absolute terms, the number of this kind of error remains relatively constant.

50

65

80

95

Top-1 Accuracy [%]

0

20

40

60

80

Portion of Group's Top-1 Errors [%]

organisms

artifacts

others

mlp

cnn

transformer

hybrid

50

65

80

95

Top-1 Accuracy [%]

0

500

1000

# Top-1 Errors

organisms

artifacts

others

50

65

80

95

Top-1 Accuracy [%]

0

20

40

60

Portion of All Top-1 Errors [%]

mlp

cnn

transformer

hybrid

50

65

80

95

Top-1 Accuracy [%]

0

250

500

750

1000

# Top-1 Errors

mlp

cnn

transformer

hybrid

(b) Abs. number per group.

(c) Relative portion for all samples.

(d) Abs. number for all samples.

60

80

100

MLA [%]

0

30

60

90

Portion of Group's Multi-Label Errors [%]

organisms

artifacts

others

mlp

cnn

transformer

hybrid

60

80

100

MLA [%]

0

200

400

600

800

# Multi-Label Errors

organisms

artifacts

others

mlp

cnn

transformer

hybrid

60

80

100

MLA [%]

0

20

40

60

80

Portion of All Multi-Label Errors [%]

mlp

cnn

transformer

hybrid

60

80

100

MLA [%]

0

500

1000

1500

# Multi-Label Errors

mlp

cnn

transformer

hybrid

(a) Relative portion per group.

(b) Abs. number per group.

(c) Relative portion for all sam- ples.

60

80

100

MLA [%]

0

5

10

15

20

25

30

Portion of Group's Multi-Label Errors [%]

organisms

artifacts+others

mlp

cnn

transformer

hybrid

(a) Relative portion per group.

60

80

100

MLA [%]

0

60

120

180

240

# Multi-Label Errors

organisms

artifacts

others

mlp

cnn

transformer

hybrid

(b) Abs. number per group.

60

80

100

MLA [%]

0

5

10

15

Portion of All Multi-Label Errors [%]

mlp

cnn

transformer

hybrid

(c) Relative portion for all sam-

ples.

60

80

100

MLA [%]

0

50

100

150

200

250

300

# Multi-Label Errors

mlp

cnn

transformer

hybrid

(d) Abs. number for all samples.

60

80

100

MLA [%]

0

1

2

Portion of Group's Multi-Label Errors [%]

mlp

cnn

transformer

hybrid

(a) Relative portion per group.

60

80

100

MLA [%]

0

5

10

15

# Multi-Label Errors

organisms

artifacts

others

(b) Abs. number per group.

60

80

100

MLA [%]

0

1

2

Portion of All Multi-Label Errors [%]

mlp

cnn

transformer

hybrid

(c) Relative portion for all sam-

60

80

100

MLA [%]

0

5

10

15

20

# Multi-Label Errors

mlp

cnn

transformer

hybrid

(d) Abs. number for all samples.

ples.

60

80

100

MLA [%]

0

5

10

15

20

Portion of Group's Multi-Label Errors [%]

organisms

artifacts

others

mlp

cnn

transformer

hybrid

(a) Relative portion per group.

60

80

100

MLA [%]

0

50

100

150

200

# Multi-Label Errors

organisms

artifacts

others

mlp

cnn

transformer

hybrid

(b) Abs. number per group.

60

80

100

MLA [%]

0

5

10

Portion of All Multi-Label Errors [%]

mlp

cnn

transformer

hybrid

(c) Relative portion for all sam-

60

80

100

MLA [%]

0

50

100

150

200

250

# Top-1 Errors

mlp

cnn

transformer

hybrid

(d) Abs. number for all samples.

ples.

Finally, in Fig. 25 we present the ratio of unexplained model failures (MLF) over multi-label accuracy (MLA) and standard top-1 accuracy. Again, subfigures (a) and (b) are organized per group, while (c) and (d) show the results for all samples.

(d) Abs. number for all samples.

60

80

100

MLA [%]

0

25

50

75

Portion of Group's Multi-Label Errors [%]

organisms

artifacts

others

mlp

cnn

transformer

hybrid

50

65

80

95

Top-1 Accuracy [%]

0

25

50

75

Portion of Group's Top-1 Errors

organisms

artifacts

others

60

80

100

MLA [%]

0

20

40

60

Portion of All Multi-Label Errors [%]

mlp

cnn

transformer

hybrid

50

65

80

95

Top-1 Accuracy [%]

0

25

50

75

Portion of Top-1 Errors

C Analysis of IMAGENET-A

To demonstrate the applicability of our automated error analysis pipeline to other datasets, we now apply it to IMAGENET-A. IMAGENET-A was introduced by Hendrycks et al. (2021) as an IMAGENET-like test set of 'Natural Adversarial Examples'. It contains 7500 images from 200 IMAGENET classes, picked to fool a set of ResNet-50 classifiers, while still constituting high-quality, single-entity images, clearly depicting an entity of

50

65

80

95

ImageNet Top-1 Accuracy [%]

0

20

40

60

80

ImageNet-A Top-1 Acc [%]

mlp

cnn

transformer

hybrid

50

65

80

95

ImageNet Top-1 Accuracy [%]

0

20

40

60

80

ImageNet-A Top-1 Acc [%]

small (

<

5M)

medium (

<

50M)

large (

<

500M)

xlarge (

>

500M)

the target class. A comparison of IMAGENET and IMAGENET-A accuracy is shown in Fig. 26. In this section, marker color encodes model parameter counts, i.e., larger models have darker markers.

As IMAGENET-A images were selected to be challenging for IMAGENET classifiers at the time, we see that many classifiers with IMAGENET accuracies below 80% have almost 0 accuracy on IMAGENET-A. In fact, 801 of the 962 models we consider have an accuracy of less than 30% on IMAGENET-A. As classifiers improved ( > 80% top-1 IMAGENET accuracy) IMAGENET-A accuracy catches up rapidly. Among other improvements, the main driver seems to be the model parameter count. Note that due to the much lower IMAGENET-A accuracy (of some models), the x-axis of all further plots starts at 0% rather than 60% , as before.

As IMAGENET-A uses IMAGENET classes, we keep the split into organisms (77 out of 200 classes, 38 . 5% ), artifacts (104 / 200, 52% ), and other classes (19 / 200, 9 . 5% ).

Class Overlap Due to the shared class structure, we can use the same analysis as for IMAGENET and show the results in Fig. 27. The 200 classes in IMAGENET-A were selected from the 1000 IMAGENET classes to avoid class overlap and label ambiguity. Thus, as expected, we see a much lower overall amount of such errors (below 0 . 8% instead of up to 8 . 5% for IMAGENET). In particular, for artifacts and other classes, we see no

0

10

20

30

40

50

60

70

ImageNet-A Top-1 Accuracy [%]

0

1

2

Portion of Group's Top-1 Errors [%]

mlp

cnn

transformer

hybrid

0

10

20

30

40

50

60

70

ImageNet-A Top-1 Accuracy [%]

0

5

10

15

20

25

# Correct Class Overlap Top-1 Errors

class overlap errors at all. Interestingly, for organisms, we still observe a small number ( ≤ 22 ) of errors caused by class overlap. We observe that errors become more frequent (albeit still rare) for models with higher accuracy. As with IMAGENET this is due to more accurate models classifying more images correctly, including to overlapping classes.

Multi-Label Annotations In contrast to IMAGENET, IMAGENET-A was designed to only include single-entity images. Thus, we skip the multi-label analysis for IMAGENET-A and use top-1 accuracy/error rather than MLA accuracy/error for all further analyses.

Fine-Grained Classification Errors

Due to the shared class structure, we can apply the same analysis including superclasses for fine-grained classification errors as on IMAGENET. We show results in Fig. 28.

As with IMAGENET, the group for which we observe most of these errors are organisms (explaining up to 47% of the group's errors). However,

0

10

20

30

40

50

60

70

ImageNet-A Top-1 Accuracy [%]

0

10

20

30

40

50

Portion of Group's Top-1 Errors [%]

organisms

artifacts

others

mlp

cnn

transformer

hybrid

0

10

20

30

40

50

60

70

ImageNet-A Top-1 Accuracy [%]

0

200

400

600

800

# Fine-Grained Top-1 Errors

organisms

artifacts

others

the portion of explained errors is much lower than for IMAGENET, where up to 88% of MLA errors for organisms can be explained by fine-grained classification errors. We see the percentage of errors increase linearly with accuracy. Looking at absolute counts, especially for organisms, we see that fine-grained errors initially become more prevalent as model accuracy increases before dropping off as models more frequently also succeed in the fine-grained class distinction.

Fine-Grained Out-of-Vocabulary Errors Again, we can use the same superclass definition, CLIP embedding and open world classifier as for IMAGENET to identify fine-grained OOV errors. We generally observe slightly larger portions of OOV errors than on IMAGENET and a much clearer linear dependence on accuracy (see Fig. 29). Notably, the superlinear increase observed on IMAGENET (see Fig. 11) is absent here. Looking at absolute counts, after an initial increase and stagnation, we observe a change.

0

10

20

30

40

50

60

70

ImageNet-A Top-1 Accuracy [%]

0

10

20

30

40

Portion of Group's Top-1 Errors [%]

0

10

20

30

40

50

60

70

ImageNet-A Top-1 Accuracy [%]

0

100

200

300

400

# Fine-Grained OOV Top-1 Errors

linear decrease with model accuracy. In contrast to IMAGENET, we again do not observe a slope

Non-Prototypical Instances Our analysis pipeline for IMAGENET uses the manual annotation of non-prototypical instances from (Vasudevan et al., 2022). As we do not have these available for IMAGENET-A we skip this analysis step here. However, for standard IMAGENET we only observe a small number of such errors ( ≤ 2% of MLE) and note that the IMAGENET-A images were manually selected to only include high-quality images (Hendrycks et al., 2021). Thus, we expect their portion to be even smaller here.

Spurious Correlations Again following our IMAGENET approach for spurious correlation errors we obtain the results in Fig. 30. At first glance, we observe much less spurious correlations than for IMAGENET ( 6 . 2% compared to 15% for artifacts). However, we observe a more linear instead of super-linear increase in the portion of errors explained by spurious correlations, leading to this reduction being smaller for less accurate models.

Model Failures Looking at the remaining failures, we see that we can only explain up to half of the errors for the best-performing models. Not only are top-1 and MLA error rates much higher than for IMAGENET but also a much larger portion of the remaining errors constitutes severe model failures, highlighting that IMAGENET-A is a much harder dataset.

0

10

20

30

40

50

60

70

ImageNet-A Top-1 Accuracy [%]

0

2

4

6

Portion of Group's Top-1 Errors [%]

mlp

cnn

transformer

hybrid

0

10

20

30

40

50

60

70

ImageNet-A Top-1 Accuracy [%]

0

25

50

75

# Spurious Correlation Top-1 Errors

0

10

20

30

40

50

60

70

ImageNet-A Top-1 Accuracy [%]

0

30

60

90

Portion of All Top-1 Errors [%]

mlp

cnn

transformer

hybrid

60

80

100

ImageNet MLA [%]

0

30

60

90

Portion of All Top-1 Errors [%]

mlp

cnn

transformer

hybrid

However, analyzing Fig. 31, we observe that both the IMAGENET-A top-1 and IMAGENET MLA accuracy are good predictors for the portion of model failures which decreases with increasing model accuracy. This highlights again and perhaps surprisingly that IMAGENET MLA underreports progress in model performance, even on particularly challenging subdistributions like IMAGENET-A.

D Extended Examples on Fine-Grained OOV Error Classification

In this section we provide further step by step examples of the procedure that determines if a given model mistake is categorized as a fine-grained OOV error. The sample in Fig. 32 is confirmed to be fine-grained OOV, the one in Fig. 33 is not fine-grained and the one in Fig. 34 is rejected as OOV.

PORAND

TC3ACI

A

SOPTTSRC

Pred: shopping basket

Pred superclass: box

E Mistake Examples by Error Type

研

Choeslate

Pretzels.50

Cavered,

UWM LI6

TR.25

OMAJR

MTERMEDIA

Hari Viswanathan

www.naturfoto.cz

LubosMraz

INDIANS

and

COWBOYS

ToUSEY

ARANDMINALLY

BO

CARTI

Figure 38: Fine-grained OOV : examples of common prediction errors (Pred) due to fine-grained OOV errors. The correct (individual) multi-labels (GT) are separated by commas; the original IMAGENET label is listed first.

3”

CLIPPER FLYING CLOUD

LY-8C

alibaba.com.cn

POLICE

PN57MSX

@ Jon-999

sanital

el

MODEL:W2712161

SAT

Subject Test

MATHEMAT

ANGELS

WOLFF

ANNADEAVERE SMITH TWILIGHT

Fition

TRIEND!

CHOMSKY

TRIGONOMETRYfor

LadeM

TUVAorBUST

Eenesl

Socond

Edltion

PROFIT

REEDON

&

IN

TABACOS

PRiMITIVA

Celstra Smort

PhonecardAcoin

MNU

KIRIN

花國

F Experimental Setup

In this section, we provide further details on our evaluation.

We publish our code and detailed instructions on how to reproduce our results at https://github.com/eth-sri/automated-error-analysis .

F.1 Computing the Multi-Label Accuracy

When computing the models' multi-label accuracies, we closely follow the recommendations by Shankar et al. (2020) 3 . In particular, we discard the set of problematic images, then compute per-class accuracies and finally average them to obtain the multi-label accuracy. We consider a model prediction to be correct if it is marked as correct or unclear in the multi-label dataset.

F.2 Datasets and Labels

We consider the validation set of the ILSVRC-2012 subset of IMAGENET (Deng et al., 2009; Russakovsky et al., 2015), available under a non-commercial research license 4 . More concretely, we use the subset of this validation set labeled by Shankar et al. (2020) and then Vasudevan et al. (2022), with the labels 3 being available under Apache License 2.0. We further evaluate our pipeline on the ImageNet-A dataset (Hendrycks et al., 2021) available under MIT License 5 .

F.3 Summary of Evaluated Models

Table 3 contains a list of all models we considered in this study and a subset of their metadata. The models were obtained from multiple sources: Torchvision 6 , torch.hub 7 , HuggingFace 8 , and timm 9 . They can all be automatically downloaded and evaluated using our open-sourced code. After collecting all model outputs (6 days for ImageNet and 1 day for ImageNet-A on a single GeForce RTX 2080 Ti GPU), running our error analysis pipeline on all models takes 12 to 24 hours using a single GeForce RTX 2080 Ti GPU for ImageNet and ImageNet-A respectively.

Table continues onto next page

Table continues onto next page

Table continues onto next page

Table continues onto next page

Table continues onto next page

Table continues onto next page

Table continues onto next page

F.4 Fine-grained Superclasses

In this section, we detail the superclasses we use for detecting fine-grained and fine-grained OOV errors and their constituting IMAGENET classes. When defining these superclasses, we focused on visual and semantic similarity rather than positions in the WordNet hierarchy. We believe that the latter is fundamentally unsuitable to obtain high-quality groupings, as visually similar or closely related classes often end up having a large WordNet distance, while very different classes are close. Further, we ensured that superclass definitions are not overly generic, providing a much more finegrained split than prior works. The superclasses are defined programmatically in our code and can be easily reused and adapted by others.

List of superclass definitions:

42. beagle _ hound _ pointer _ spaniel _ setter - 27 classes:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a pipeline to automatically categorize ImageNet classification misclassifications. Previous work [1] defined and categorized those misclassifications into 4 types manually and this is the main motivation of automating the process. The results show that their pipeline has an accuracy of 60% based on human annotations of model misclassifications.\n\n\n[1] analyzing the remaining mistakes on imagenet\n\nI have read the author’s rebuttal and adjust the rating accordingly!\n\nSTRENGTHS:\nThe paper is easy to follow as they mostly based on the previous work [1].\n\nWEAKNESSES:\n- The novelty of this paper is minimal and the motivation is not strongly supported. For example, the definitions and categorization of misclassifications have been done before in [1]. In writing, I do not see the reason why we need such categorization of misclassifications (i.e. the motivation). In Related work where I expect to see the motivation, the authors did not contrast their work with the literature then it is hard for me to see the contribution. I believe Related work can be greatly improved. Automating [1] is not considered novel at NeurIPS standard in my opinion.\n- The authors only tested on 1 model and 1 dataset. It does not make the findings generalize. Another problem is the number of samples being examined. Based on Table 1, only 378 were tested!!!\n- It looks like the authors are trying to do 4-way classification. In this case, it makes more sense to build a model to do that rather than having different methods for different misclassifications. The four types of misclassifications can be easily collected or synthesized.\n- The authors only compared with [1]. The reviewer encourage them to extend the comparisons to others (e.g. [2]) and with other datasets and classifiers.\n\n[2] ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
The paper introduces an automated analysis on the model errors in ImageNet.
The analysis sequentially categorizes into six types of errors with increasing
criticality, unveiling the true model failures. The results are contextualized
with previous work.

STRENGTHS:
- This work is of high interest to the computer vision and machine learning
  community, given the extensive use of ImageNet as a benchmark.

- The quality and presentation of the analysis is excellent.

- The full code to conduct the pipeline presented in the paper is provided with
  the paper, which is extremely value considering the reproducability under the
  complexity of the analysis (wrt. the number of models, etc.). Although the
  analysis may seem very large-scale, it is surprisingly affordable and, with
  the code, is very easily reproducable.

- The analysis builds upon and refines the previous NeurIPS's work by Vasudevan
  et al..

- The sequential categorization, starting with the least critical category of
  error, is a very good idea.

WEAKNESSES:
- I could not find any major weakness of the paper that would
  impede its acceptance in the current state.

QUESTIONS:
- While not critical, I am wondering why the paper did mention to use the
  WordNet parent classes for the super classes. A single sentence on the
  fitness of the parent classes as super classes would we interesting.

LIMITATIONS:
The paper addresses its major limitations well.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an automated pipeline to classify ImageNet's errors from 4 categories: (i) fine-grained categories; (ii) fine-grained OOV; (iii) Non-prototypical instances; and (iv) Spurious correlations. The main contribution of the paper is this automated pipeline with which various errors can be categorized.  While there exists a plethora of recent works in classifying ImageNet errors, the paper does a good job of summarizing all these errors and also designing an automated pipeline to detect them for ImageNet.\n\nSTRENGTHS:\n- Good overview of ImageNet errors and a simple, but effective automated pipeline for error detection.\n- A wide-range of models are tested which is a strong point for the paper.\n\nWEAKNESSES:\n- The analysis is restricted to only analysing ImageNet errors. While it is a good direction to inform vision research -- it will be beneficial to extend the framework to understand error types in other ImageNet variants such as ImageNet-A. What are the error distributions in such sets? The current analysis though good, is restrictive.\n- The paper does not discuss recent works such as ImageNet-X (https://facebookresearch.github.io/imagenetx/site/home)  which also looks at understanding model failures. While the categories considered in this paper is a subset of ImageNet-X error types, can the authors compute an overlap metric between some of the error types detected in this paper and ImageNet-X? \n- While a range of models are tested, can the authors distinguish between the pre-training strategies more in the plots? E.g., highlight SSL methods, Vision-Language pre-training methods and Supervised Pre-training methods using different colors / notations. This would make the paper stronger.\n- Can the authors give some intuitions on the unexplained failures? What are the potential causes? \nHow can the error analysis learnings be useful in improving model training? Can the authors provide some intuition on this?\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\n- While the paper provides a good overview of ImageNet errors and provides an automated pipeline to explain a subset of those errors, the analysis is still restricted to ImageNet and is not extended to other ImageNet test sets where the drop in accuracy is more severe. \n- Although the paper tests a large number of models, each component (e.g., pre-training strategy, augmentations) is not well dissected deeply.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper aims to create an automated pipeline for defining the types of errors made by ML models on ImageNet. Error categories are from previous work, and the paper compares the results between the previous human based pipeline and the newly automated version. The authors use the pipeline to evaluate over 100 models, and provide some analysis on the findings.

STRENGTHS:
- Paper structure is clear
- Design decisions for automating each step of the pipeline make sense
- Results are validated and put in context with prior work

WEAKNESSES:
- Analysis of results seems limited. I would like to see additional analysis that builds upon the benefits of such pipelines. For example, if we can now easily get to severe model errors, can we do human analysis on these to create categories or better understand them?
- I would like to better understand the motivation behind this work, such as potential use cases for the pipeline to improve machine learning research

QUESTIONS:
See weaknesses

LIMITATIONS:
I believe that limitations are addressed adequately
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an automatic error classification pipeline for ImageNet models. In particular, they categorize mistakes into overlapping classes, multi-object images, fine-grained and fine-grained with out-of-vocabulary (OOV), non-prototypical, the examples influenced by spurious correlations and unexplained mistakes. To detect fine-grained OOV errors the authors used WordNet and an open world CLIP classifier. They evaluated 121 models and showed that better performing models (in terms top-1 accuracy or multi-label accuracy) have a lower _portion_ of unexplained errors, and higher portions of errors which are explained by class ambiguity and fine-grained cases. They analyzed error portion trends separately for objects and organisms ImageNet classes, and analyzed the effect of training dataset and model architecture.\n\nSTRENGTHS:\nWhile prior works studied questions similar to this paper such as [1] analyzed error types of one or a few high performing ImageNet classifiers such as ViT-3B, and [2] looked at how multi-label accuracy improves with model scale, this paper studies a more detailed question of how the error type distribution changes as we scale up the models in terms of the architecture size and pre-training dataset.\nThey also propose an automatic way to identify fine-rained out-of-vocabulary mistakes which relies on an external open world classifier model CLIP (assuming that we filtered out overlapping and fine-grained classes).\n\n**References**\n\n[1] Vasudevan, Vijay, et al. \"When does dough become a bagel? analyzing the remaining mistakes on imagenet.\" \n\n[2] Beyer, Lucas, et al. \"Are we done with imagenet?.\"\n\nWEAKNESSES:\n1. **Limited novelty.**\n\nGiven the results from the prior works that study label mistakes and ambiguity in ImageNet dataset like [1] and [2], the novel insights from this paper seem limited.\nI would suggest that the authors explicitly emphasize that the error categorization and definitions as in Figure 1 and most subsections in Section 3 (except for novel approach for fine-grained OOV) were proposed in prior works, and that they separate the background and results from prior works and their novel approaches and insights. \n\nIn particular, both [1] and [2] categorize mistakes in a similar fashion, and the categorization in [2] is the one this paper relies on. [1] focus on analyzing mistakes of a few large-scale models like ViT-3B and report the percentage of different error types and their severity in their Section 4.1 Table 2. Similarly to this paper they report mistakes separately for objects and organisms classes from ImageNet. They also propose ImageNet-M — a subset of only 68 examples where high-performing models still consistently make major mistakes.\n[2] also discuss a similar error categorization but they focus on analyzing how scaling up models affects top-1 and multi-label accuracy. \nThis paper combines two types of analysis from [1] and [2] and studies how scaling up models affects the error types distribution, and concludes that for stronger performing models the percentage of major mistakes significantly decreases. While this experiment and analysis is new, the conclusions are somewhat straightforward given the results from [1] and [2]? \n\nCould the authors please list the new observations and conclusions that were made in their analysis and that were not known in prior work?\n\n\n\n2. **The extent to which the pipeline of error type classification is automatic seems to be overclaimed.**\nThe introduction and abstract of the work make an impression that the paper proposes a fully automatic way to classify model’s mistakes, without relying on manual annotation unlike prior works [1, 2].\nHowever, after reading Section 3 it turns out that class overlap (Sec 3.1), multi-object cases (Sec 3.2), and spurious correlations (Sec 3.6) rely on manually obtained labels from [1] and [2], the definition of non-prototypical examples (Sec 3.5) relies on the ViT model’s mistakes which were analyzed in [1], and fine-grained errors (Sec 3.3) rely on manual grouping of classes into super-classes (see another comment re this case in Questions).\nThe only case of automatic error categorization is the fine-grained out-of-vocabulary error in Section 3.4 which involves WordNet tree and open world CLIP classifier. \n I suggest that the authors adjust their claims accordingly since such a pipeline can’t be called automatic.\n\nIn particular, if the proposed error classification pipeline was indeed automatic and didn’t have limitations discussed in lines 45-47 in terms of relying on human labellers, how would one apply the proposed pipeline to another large-scale dataset where we don’t have these labels from [1] and [2]? If it is possible, it would be helpful to see the application of such error classification to another dataset.\n\n\n3. **The design choices for detection of fine-grained out-of-vocabulary mistakes is not fully clear.**\nThe only automatic and novel part of the error classification pipeline is detection of fine-grained OOV mistakes: (1) first we find 10 most visually similar images in ImageNet train set in terms of the CLIP embedding, (2) check that at least one label belongs to the same superclass as model’s prediction, and (3) measure CLIP similarity with the correct class and WordNet in-vocabulary and OOV sibling classes related to both correct class and model's prediction.\n\nFirst, for the last step the description of the label proposal set is a bit confusing, and it would be helpful if the authors could clarify the exact set of proposals they are considering.\nSecond, it would be helpful to have more examples like in Figure 10 to illustrate step-by-step fine-grained OOV mistakes identification and show for each example what were the 10 closest images in embeddings space and which labels they had, and then which label proposal set CLIP scored (in particular, where those proposals were taken from and what were the CLIP scores exactly). It would be helpful to have more intuition for each step of this process.  \n\n4. [Minor] While the paper is generally easy to follow, I highlight a few points below which were not as clear:\n\n-  For Figures like 3, 5 and other trends, please clearly define what exactly is shown on y axis. I am assuming it is (# of mistakes of type X ) / (# of all mistakes) for each model.\n- On the same Figures, does each marker correspond to one model? Is it correct that each model is shown twice: once for objects and once for organisms?\n\n- Please fix captions for Figures 7, 12 and 14.\n\n\nTypos:\n\n- Line 54: and -> an\n- Line 99: find\n- Line 113: need to\n- Line 189: the\n\n**References**\n\n[1] Vasudevan, Vijay, et al. \"When does dough become a bagel? analyzing the remaining mistakes on imagenet.\" \n\n[2] Beyer, Lucas, et al. \"Are we done with imagenet?.\"\n\nQUESTIONS:\n1. Line 210: “we manually reviewed all 1000 ImageNet classes and defined semantically similar superclasses guided by the question of whether an untrained human could reasonably confuse them”\nWhy did you choose to manually review and group classes instead of using WordNet tree?\nThis also doesn’t match well with the claims about the error classification pipeline being automatic.\n\n2. What are the examples of spurious correlations that you identified in Section 3.6? How are they related to the findings from prior works studying spurious correlations in ImageNet like e.g. Salient ImageNet [3] and Hard ImageNet [4]?\n\n3. Line 370: “Arguably, we observe vision transformers achieving lower model failure rates on artifacts at very high MLA when compared to ConvNeXts.”\nDoes this observation hold even when we condition on the size of pre-training dataset?\n\n**References**\n\n\n[3] Singla, Sahil, and Soheil Feizi. \"Salient ImageNet: How to discover spurious features in Deep Learning?.\" \n\n[4] Moayeri, Mazda, Sahil Singla, and Soheil Feizi. \"Hard imagenet: Segmentations for objects with strong spurious cues.\"\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}