Automated Classification of Model Errors on ImageNet
Authors: Momchil Peychev
Mark Niklas Mueller
Marc Fischer
Martin Vechev
Venue: NeurIPS 2023 poster (2023)
Keywords: ImageNet
evaluation
error classification
error analysis
TL;DR: We present an automated pipeline for classification and analysis of model errors on ImageNet.

Abstract: While the ImageNet dataset has been driving computer vision research over the past decade, significant label noise and ambiguity have made top-1 accuracy an insufficient measure of further progress. To address this, new label-sets and evaluation protocols have been proposed for ImageNet showing that state-of-the-art models already achieve over 95% accuracy and shifting the focus on investigating why the remaining errors persist.

Recent work in this direction employed a panel of experts to manually categorize all remaining classification errors for two selected models. However, this process is time-consuming, prone to inconsistencies, and requires trained experts, making it unsuitable for regular model evaluation thus limiting its utility. To overcome these limitations, we propose the first automated error classification framework, a valuable tool to study how modeling choices affect error distributions. We use our framework to comprehensively evaluate the error distribution of over 900 models. Perhaps surprisingly, we find that across model architectures, scales, and pre-training corpora, top-1 accuracy is a strong predictor for the *portion* of all error types. In particular, we observe that the portion of severe errors drops significantly with top-1 accuracy indicating that, while it underreports a model's true performance, it remains a valuable performance metric.

We release all our code at https://github.com/eth-sri/automated-error-analysis.

Automated Classification of Model Errors on ImageNet
Momchil Peychev; Mark Niklas Müller, Mare Fischer, Martin Vechev
Department of Computer Science ETH Zurich, Switzerland
(momchil L. peychev, mark. muellerrrrrr marc. fischer, martin. verhe)..ecceeeeiifff wtw....feeeecc....
Abstract
While the ImageNet dataset has been driving computer vision research over the past decade, significant label noise and ambiguity have made top-1 accuracy an insufficient measure of further progress. To address this, new label-sets and evaluation protocols have been proposed for ImageNet showing that state-of-the-ard models already achieve over 95% accuracy and shifting the focus on investigating why the remaining errors persist. Recent work in this direction employed panel of experts to manually categorize all remaining classification errors for two selected models. However, this process is time-consuming prone to inconsistencies. and requires trained experts, making unsuitable for regular model evaluation thus limiting its utility. To overcome these limitations, we propose the first automated error classification framework, a valuable tool to study how modeling choices affect error distributions We our framework to comprehensively evaluate the error distribution of over 900 models. Perhaps surprisingly. we find that across model architectures, scales, and pre-training corpora, top-1 accuracy a strong predictor for the portion of all error types. In particular, we observe that the portion of severe errors drops significantly with top-1 accuracy indicating that, while it underreports a model's 's true performance, it remains a valuable performance metric. We release all our code at httpps///iithup wtmm/eth-sri/automttettee -erorr--r--r-ee erroor-nalyyiis
Introduction
IMAGENET (Deng et al.) 2009 Russakovsky et al., (015)) has established itself as one of the most influential and widely used datasets in computer vision, driving progress in object recognition (Krizhevsky et al.) 2012), object detection (Tan et al., 2020). and image segmentation (Minace) (022).. As state-of-the-art models have come close to, and by some metrics exceeded, human performance, the significance of further progress in top-1 and top-5 accuracy has, been questioned in the face of label errors and systematic biases (Beyer al. 2020 Tsipras et al. 2020).
The most severe such bias is the lack of multi-labed annotations in the original IMAGENET dataset, with recent studies finding that roughly fifth of images show multiple entities. While state-of-the-art models have learned to exploit labeling biasses on these images (Tsipras et al., 2020), best practices have shifted to reporting multi-labed accuracy (MI.) computed using new multi-label annotations
(Beyer et al. 2020; Shankar et al. 2020). Further, many IMAGENET and especially organism classes, hard to distinguish even by trained humans (Horn et al. 2015 Northcutt et al. 202 Lee et al. 2017). leading to persistent labeling errors.
In the face of these challenges and with sfate-of-the-a aht models exceeding 95% MLA, the focus has
increasingly shifted towards analyzing and understanding the remaining model errors instead of blindly pursuing improvements in headline accuracy numbers. To this end, Vasudevan et al. (2022) review all remaining errors of two state-of-the-art models using a panel of experts and classify them
"Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

with regards to both error category and severity. While they find that many of the remaining enrors are minor or can be attributed fine- ane-grained class distinctions, they also find major classification errors that are not so easily explainable. We believe that tracking and analyzing the distribution of these error types over large number of models can not only help us to better understand the impact of novel training techniques and architectures but also identify where the biggest challenges lie and thus how to address them. However, the manual review process employed by Vasudevan et al. (2022) has several issues, preventing it from being employed for a large scale repeated study of model errors: (i) it is time-consuming even for precise models, making it infeasible to repeat for a large number of potentially less precise models, (ii) it requires (a panel of) expert reviewers which need to be trained on the fine- theegrained class distinctions, and (ii) it is inconsistent as different reviewers or even the same reviewers at different times might classify the same error differently.
This Work To overcome these challenges, we propose an automated error classification pipeline that we use to study the distribution of different types of errors across 962 models of different scales, architectures, training methods, and pre-training datasets. Our pipeline allows us to automatically detect all four error categories identified by Vasudevan et al. (2022): (i) fine-grained classification errors are detected using set of 161 manually defined superclasses, (ii) fine-grained out-of-vocabuse errors are detected using a visual similarity based criterion for prediction quality and confirmation of their out-of- vocabulary nature using open-world classifier. (iii) non-prototypical examples are
identified using the exhaustive annotations by Vasudevan et al. (2022), and (iv) spurious correlations detected using co-occurrence- -freuurrnneeeereuuennyy based criterion.
Main Findings Our automated error classification pipeline allows us to, for the first time, study the distribution of different error types across a large number of models, leading to the following insights: (i) even MLA is pessimistic measure of model progress with the portion of severe model failures quickly decreasing with MLA, (ii) this reduction of model failure rate with MLA is more pronounced for larger (pre-)training corpora, i.e., models trained more data make less severe errors even at the same top-1 or multilabel accuracy, and (iii) organism and artifact classes exhibit very different trends and prevalences of error types, e.g., fine-grained classification errors are much more frequent for organisms than for artifacts, while artifacts suffer much more from spurious correlations and out-of-vocee vocabulary errors. We believe that these insights can help guide future research in computer vision and that studying the effects of new methods on the resulting error distribution can become an important part of the evaluation pipeline.
2 Related Work
While the IMAGENET dataset is annotated with a single label per image. many images have been found to contain multiple entities (Beyer et al, 2220: Shankar et al. 2020
Multi-Label Annotations
Tsipras et al., 2020 Yun et al. 202T). Thus, multi-label accuracy (MLA) with respect to new multi-label annotations has been established as more meaningful metric. Yun et al. (02TT) generate pixel-wise labels for IMAGENET by directly applying a classification layer to image embeddings before spatial pooling. Beyer et al. (2020) collect Reassessed Labels (ReaL) for the whole IMAGENET validation set, by first identifying 6 models with high prediction coverage and accuracy, before manually annotating all images where these 6 models disagree using non-expert labelers. They discard 3 163 images where these labelers disagreed. 'Tsipras et al.] (2020) collect multi-label annotations for 10 000 validation set images, and find that top-1 accuracy is 10% lower for multi- compared to single-label images while MLA accuracy is identical. (Shanaar et al. (2020) collect multi-label annotation for 40 000 IMAGENET and IMAGENETV2 V2 validation images by manually reviewing all predictions made by a diverse set of 72 models, using human expert labelers.
Label Errors Northeutt et al. 202 study label errors across 10 commonly used datasets, including IMAGENET, using a confident learning framework to identify potential errors before validating them with Mechanical Turk (MTurk). Studying the whole validation set, they report an error rate of over 5.8%. Lee et al. 2017) manually review 400 randomly selected classification errors of an ensemble model and find the IMAGENET label for a substantial portion to either be incorrect or not describe the main entity in the image. Vasudevan et al. (2022) reviewed all remaining errors for two state-of-ther thr models with panel of experts and found that of 676 reviewed model mistakes, 298 were either correct, ambiguous, or the original ground truth incorrect or problematic.

---

Recent work has focused on understanding the types of errors models still make on IMAGENET. To this end, one strand of work analyses the differences in errors between models. Mania et al. (019) find that independently trained models make errors that correlate well beyond what can be expected from their accuracy alone. Geirhos et al. (2020) analyze the consistency between errors made by humans and CNNs on a 16 class version of IMAGENET. They observe that while CNNs make remarkably similar errors, the consistency between humans and CNNs barely goes beyond chance. Nguyen et al. (2021 1) analyze wide and deep ResNets and find that these exhibit different error patterns. Mania and Sra (020) find that more precise models typically dominate less precise ones, i.e., their error set is subset of that of less precise models. Lopee et al. 2022). in contrast, find that different pretraining corpora and training objectives can significantly increase error diversity with Andreassen et al. (2021) showing that this diversity reduces significantly during finetuning.
Error Analysis
Vasudevan et al. (022)) focus on the errors current state-of-the-a models make, identifying four categories: (i) fine-grained errors describe a model's failure to distinguish between two very similar classes, (ii) -ieeggaieed out-of-vocablary errors occur when an image shows an entity not contained in the IMAGENET vocabulary and the model instead predicts similar class, (iii) spurious correlations cause the model to predict a class that is not shown in the image in the presence of strongly correlated features, and (iv) non-prototypical instantiations of class are not recognized by the model.
Datasets for Error Analysis In addition to work analyzing model errors on fixed datasets, there has been growing interest in datasets specifically designed to highlight and thus analyze specific error types. Singla and Feizi (2022) Moayeri et al. (2022) find that, for some classes, models rely heavily on correlated or spurious features suffering severely reduced accuracy if these are absent or removed. To study this effect, they introduce SALIENT IMAGENET (Singla and Feizi 2022) and
HARD IMAGENET (Moayeri et al. 2022). providing soft masks for causal and correlated features and segmentation masks, respectively. Hendrycks et al. (022I) propose IMAGENET-A, a dataset of 7 500 single-entity, natural adversarial examples, which induce high-confidence misclassiffcations in a set of ResNet50 and belong to 200 IMAGENE: classes that were chosen to minimize class overlap and the potential for fine-grained errors (see App. C for an analysis). Vasudevan et al. (2022) collect images where -fate-of-the-arr models fail in an unexplained manner in IMAGENET MEENNEETMAAE MAJOR. TTesiri et al. (2023) investigate the effect of zoom and crop on classifier performance and introduce IMAGENET- HARD as the set of images (from range of IMAGENET-like datasets) that none of their considered models classified correctly for any crop. Idrissi et al. (2023) annotate IMAGENET images with respect to how they differ from what is prototypical for their class, introducing IMAGENET-X. This allows them to study the effect of these image-specifee variations such as atypical pose, background, or lighting situation, (2022)ss s) work focuses more on systematic errors caused (partially) by labeling (set) choices rather than what makes individual image hard to classify.
accuracy over large number of models. In contrast, our (and Vasudevan et al.)
3 Categorizing ImageNet Errors
In this section, we introduce our automated error classification pipeline, which aims to explain model errors by assigning it one of six error types. We consider errors that can not be explained in this way to be particularly severe model failures. While the definitions of our error types are heavily inspired by Vasudevan et al. (2022). ) we address three main issues of their manual approach with our automated pipeline: (i) it moneccosuningg even for precise models and intractable for imprecise ones, (ii) it requires a panel of expert reviewers, typically not available, and (iii) it introduces inconsistencies due to human expert error, disagreement, or ambiguity.
Below, we first provide a brief overview of our pipeline (illustrated in Fig. before discussing the different error types and how we identify them in more detail.
We consider errors due overlapping class definitions (discussed in $ $3.1 and missing multi-label annotations (3.2) to be the least severe, as they can be interpreted as labeling errors rather than classification errors. When model predicts a class that is closely related to one of the labels, we call this a fine-grand grained classification error ($3.3), as the model succeeds in recognizing the type of entity but fails during the fine- greine distinction. When an image contains an entity that does not belong to any of the IMAGENET classes and model predicts label that is closely related to this ofut-of--ocabule consider both types of fine-grained errors to be minor. If model fails on image that shows very
class, we call this fine-grained voccbuuuyya error ($3.4). We
3

---

No Error
Minor Error
Overlapping Classes 3.1
Multi Label 3.2
Fine Grained 3.3
Fine Grained Out Of Vocabulary
All Errors
3.4
Spurious Correlation
Non-Prototy Propotypiica
Model Failures
$
$3.6
Explainahle Firor
Figure 1: We first remove errors w.r.t. the original IMAGENET labels caused by overlapping class definitions or missing multi-label annotations, yielding multi- label accuracy (MLA). We then, in this order, identify fine- grained misclassifications, fine-grained misclassifications where the true label of the main entity is not included the IMAGENET labelset, non- prototypical examples a given class, and spurious correlations. This leaves us with severe model failures that are unexplained by our categorization.
non-prototypical instance of a given class, we call this non-prototypical error ($3.5) If model predicts a class that commonly co-occurs with a ground-trute class but not shown the image, we attribute this error to spurious correlation and discuss it in (3.6 We denote both of these errors as explainable errors. If an error can be attributed multiple causes, we assign the least severe category and thus design our pipeline to consider error types in order of increasing severity (see Fig. D
Throughout this section, include examples and trends for the kinds of errors discuss. We generally describe these separately for images with a ground-trunt class describing an organism (410 of 1000 classes) and artifacts (522 classes) (in line with previous work (Shankar al. 2020), we observe that these two groups of classes exhibit different error patterns. These two groups account for almost all IMAGENET classes, with the remaining 68 classes being assigned the group "other". Where we find interesting trends, we further distinguish models by their (pre- Itraining dataset ranging in size from one million (IMAGENET) multiple billion images (INSTAGRAM, LAION-2B), their architecture, including broad range of MLPs, CNNs, transformer-basee based, and hybrid models, and their (pre-) training method. For the full details on all 962 models we consider, please refer App. We provide more examples of every error type in App. E and more detailed trends in App. B
3.1 Class Overlap
Prior work has established that small number of IMAGENET classes exhibit extreme overlap (Northcutt et al. 2021b] Vasudevan et al.) 2022). We illustrate one such example in Fig- 2 tusker is defined as "an animal with tusks') D which describes a strict superset of african elephant and has significant overlap with indian elephant (females have short and sometimes no tusks). To avoid penalizing model for correct predictions that do not match the ground truth, we consider all predictions describing a superset or equivalent of the ground-truie -ruth class be correct. For example. we accept tusker for an image labeled african elephant, not versa, as the latter might actually show boar. We follow the mappings of equivalence and containment from Vasudevan et al. (2022) and refer to their App. C for full details.
indian elephant
african elephant
Venn-Diagram of the tusker. indian elephant, and african elephant classes.
Figure 2:
In Fig. visualize the portion and number of top-1 errors identified to be correct, separating images with a ground- truth class describing an organism (green hues) and artifacts (red hues) and encod- ing the pre- pre-training dataset size by assign- ing darker colors larger datasets. While trends for artifacts and organisms seem very similar for portions of top-1 errors, we observe clear difference when look- ing at their absolute numbers. There, two competing effects are at play. With increasing accuracy, more samples get classified correctly, in-
Portion Geoupls Topp Eaoos
Corret Clas Overaa Erves
100
arttfctt
reguniinns
65
56
65
Too 1 Accuney [X]
Top Aecuray (%)
Figure 3: Portion (left) and number (right) of top-1 errors caused by claxs overlap by group organisms (green) and artifacts (red). A 95% confidence interval linear fit is shown on the right.
AAcording to the Merriam Webster Dictionary

---

cluding as overlapping classes, thus increasing the number of such errors. On the other hand, more accurate models leverage labeling biases pick the "correct" among equivalent classes (Tsipras et al., 2020), thus reducing the number of such emrrs. While the former effect seems to dominate for artifacts, the latter dominates for organisms.
3.2 Missing Multi-Label Annotations
While the IMAGENET dataset is annotated with a single label per image, many images contain multiple entities (Beyer et al., 2020: (Shankar al. 2020, [Tspras et al. (020). Consequently, a model predicting the class of any such entity, different from the original IMAGENET label, considered incorrect when computing top-1 accuracy. For example, barn, and fence while only being labeled ox. To remedy this issue, multi-label accuracy (MLA) considers all shown classes, according to some multi-label annotation, to be correct. this work, we use the annotations collected by Shankar ct al. (2020) and improved by Vasudevan al. (2022) combined with the mean class-wise accuracy definition of ML. (Shankar et al. 2020
the image shown in Fig. 4 contains
Figure 4: Image with la-
bel ox, but also showing the classes barn and fence. Ex- ample from Yun et al. 2221
We visualize the portion and number of top-1 1 er- rors that turn out to be correct under multi-label evaluation in Fig. 5 When looking at the portion of errors (Fig. Ileet), we observe a similar trend for organisms and artifacts with missing multi-
Porton of Cooups Top Eroar [S]
Top Ernes
SAM
ootiliets
Organimn
label annotations accounting for an increasing portion of top-1 errors MLA increases and artifacts consistently more affected than organ- isms. When looking the absolute numbers, again observe the number of errors explained by multi-label annotations to first increase with ac- curacy as models become more precise, before decreasing again as models start to leverage labeling biases TTippas al., 2020). As missing multi-label annotations explain up 60% of model errors, we henceforth use MLA instead of top-1 accuracy as the reference for model comparison.
Tagel Accnmey X4
Figure 5: Portion (left) and number (right) of top-1 emors caused by missing multi-label annotations by group organisms (green) and artifacts (red).
3.3 Fine-Grained Classification Errors
Many of the IMAGENET classes and especially the organism classes are very similar, making their fine-grained distinction challenging even for trained humans. For example, Fig. B classified to cornet while actually showing a french horn, both brass wind instruments. While these errors can be corrected for the relatively small validation set using expert human judgment (Horn et al. 2015 Vasudevan et al.) 20022), the much larger training set remains uncorrected. In this light, we believe a comprehensive model evaluation and comparison, should consider the failure to distinguish between very similar classes to be less severe than the failure distinguish very different classes.
Figure 6: Image laheled french horn, but predicted to show a cornet.
To automatically detect such fine-grained clas- sification errors, we manually review all 1000 IMAGENET classes and define semantically sim- ilar superclasses guided by whether an untrained human could reasonably confuse them. We ob- tain 161 superclasses, containing between 2 and 31 classes with an average of 6.7 and a median of 4. An additional 74 classes are not part of any superclass. 50 superclasses contain organisms (9.8 on average) and 101 contain artifacts (5.3 on average). We visualize the portion and num- ber of fine-grained errors in Fig. 7 As expected, we observe significantly more fine- errors for organisms than for artifacts, explaining up to 88% and 50% of mubti-label errors, respectively.
Portara Group's Malh-Lahel Emar [5]
Mutti Labe Erross
oeger
1000
600
actifacts
4X0
arganiane
800
100
MLA
MLA
[4]
Figure 7: Portion (left) and number (right) of MLA errors caused by fine-grained misclassification by group organisms (green) and artifacts (red).
5

---

Lab: reat
Image Embedding
Prod: rock beauty
Altemsatvve Labels
Visually
Input Image
Reference
Similar
Images
Label Proposals
OOV
chaetodon
Fine Geni-Gr Eroo?
Open World
angelfish
OOV?
rock beauty
Label
rack
rock
rock
Decision
Classifier
butterfly fish
beauty
beauty
beauty
fi tench
fish
fisturest
Eisurest
Similar Images
Figure 8: To confirm a fine-grained out- vocaaullayy error we proceed as follows. Given misclassified input image, we first retrieve the 10 most visually similar images from the IMAGENET train set. If the superclass of any of these images matches the model prediction, we conclude that entity with class similar prediction is shown in the image. To confirm that this entity is indeed out-of-vacabulary. we first collect a set of proposal lahels from WordNet (Miller 1992) which are partially in and out of vocabulary. Then, we use an Open World Classifier to score each of the proposed labels. If the highest scored among these is not included in the IMAGENET labelset, consider the analyzed error to be OOV.
3.4 Fine-Grained Out-of-Vocabulle Vocabulary Errors
Often images contain entities that do not belong to any IMAGENET class, call these out-of-vocabulary For example, Fig- 9 shows image of two blue-cheeked butterflyfish, which is class not part of the IMAGENET labelset. Instead, the target label is coral reef, describing the background of the image. The classifier predicts rock beauty, which is a W WORDNET child of butterflyfist (despite being part of the angelfish and not the butterflyfish family) and the closest class in the IMAGENET labelset the blue-cheeked butterflyfish. While an optimal classifier could be expected to always predict a class from the intersection of the contained entities and the IMAGENET labelset, the following is often observed in practice (Vasudevan et al., (022). The classifier tries to classify prominent entity the image, but as not part of the IMAGENET labelset, it predicts closely related class instead. We argue, that these cases should be treated at least as leniently as fine-grained classification error, but they are harder to detect. the entity the model tried classify can not be assigned any IMAGENET label, it will not be included in any multi-label annotation and can thus not be captured by defining groups closely related classes.
Figure 9: Image labeled coral reef, but prediction rock beauty (fish).
To still detect fine-grained out-of- vocabulary (OOV) errors, we pro- pose the following approach, illus- trated in Fig. [ for the above exam- ple. We first retrieve the 10 most vi- sually similar images from the IMA- GENET training set using cosine sim- ilarity in the CLIP embedding space (Radford et al. 2021). in this case all labeled rock beauty. If label of any of these images belongs to the same
physical object
whole unit
geological fornation
percold fish
ridge
buttorfly
danselfish
reef
chactodon
rock beauty
angel fish
anezone
fish
coral
reef
Figure 10: Illustration of proposal labels C. In-ooabbulary labels
shown in bold. Classes in the same superclass the prediction are shown in blue boxes direct WORDNET siblings orange superclass as the model's prediction C and ancestors not shared with the label green W.
(as defined in {3.3). we conclude that entity similar to prediction is shown in the image and proceed with our analysis. Otherwise, we conclude that the analyzed error is not fine aeee-graindd To confirm that the shown entity is indeed out-of-vocabulary, we first collect set of proposal labels from the following (illustrated in Fig. TO): all IMAGENET labels in the same superclass as the model's prediction (IV), all direct WORDNET siblings of the model's prediction (IV and OOV), and all WORDNET ancestors of the model's prediction, the IMAGENET label (IV and OOV). Finally, we use CLIP as an open-world classifier to score each of the proposed labels. If the highest scoring class is not included in the IMAGENET label set, we consider the shown entity be OOV and conclude that this error was indeed fine- mraiird OOV.
to but excluding the first common ancestor with

---

We visualize the portion and number of multi-label errors categorized as fine-
Portion of Grorpls Motti Lebel Eroos [SC
MaaliCaaidd Binne
240
grained OOV in Fig. 11 Interestingly, we observe that fine-grained OOV errors are not only much more prevalent in artifacts but also that their portion increases much more quickly with MLA for artifacts. We hypothesize that this is due to the IMA- GENET labels covering organisms occur- ring in the validation set much more com- prehensively than artifacts, and many im- ages of artifacts being cluttered and full of other (OOV) objects. We note that this might be a reason why trained humans still outperform even ofte-offthhe-rr models on artifacts but not organisms (Shankar et al., (020). Interestingly, we observe that, across pretraining datasets and model sizes, the number of fine-grained OOV errors for artifact drops much quicker with MLA above around 93% MLA (see confidence intervals in Fig. [1 right).
180
artificts
artdlctss
120
biol
80
100 80
50
103
MLA [S]
MLA PS
Figure 11: Portion (left) and number (right) of MLA errors identified as fine-grained OOV by group organisms (green) and artifacts (red). 95% confidence interval linear fit is shown the right. For artifacts, models are divided at 93% MLA.
3.5 Non-Prototypical Instances
Many concepts described by any one IMAGENET class label are broad, depend on social and geographic factors, and change over time. Com- bined with the search-based data collection process, this has led to a biased dataset with skewed representations of the concepts described by the contained classes. Therefore it is unsurprising that models perform worse on non con-proootypial instances or pictures of any given object, ie.. instances that, while clearly belonging to given class, can be considered outliers in the IMAGENET distribution of that class.
Figure 12: Image with la- bel whistle, but predcition perfume.
we see a non- prototypical whistle. We believe is inter- esting to track progress on these hard, non- prototypical instances as they are likely to be good indicator to what extent a model has learned to recognize the under- lying concepts of class. However, defin- ing what constitutes a non-prototypical in- stance of any class is hard and to large extent subjective. Fortunately, this error type is independent of the (incorrect) prediction made by the model, allowing to directly leverage the manual categorization of non- non-prototypical images by Vasudevan et al. (2022). We thus implicitly decide that all images that classified correctly by the state-of -he--e--rt models (ViT-3B and G Greedy Soups) they considered are not sufficiently non-prototypical to explain an error by another model.
For example, in Fig. 12
Portur of Grorzle Muits-aall Label Fonoss [S]]
Muutt Label Enoes
10
100
60
100
MLA 1X
MLA [X]
Figure 13: Portion (left) and number (right) of MLA errors identified as poopooootllical sample.
We visualize the number of errors caused by non-prototypical instances in Fig. 13 Interestingly and in contrast to all other error types, there is no strong correlation between performance on non- prototypical examples and overall MLA. This suggests that these non con-rototyppcal examples not inherently hard to classify for all models. Further surprisingly, non- non-prototypica examples account for a very similar portion of errors for artifacts and organisms, despite the appearance of artifacts of the same class varying significantly more, which we expected would lead to a larger portion of artifact errors being explained by their non- non-prototypical appearance.
3.6 Spurious Correlations
Entities of many classes frequently co-occur with features that have no causal relationship to that class. For example, oxen are frequently observed on green pastures (such as in Fig. 4.. We call these correlated features (Neuhaus et al.) 2022). Inherently this is not a problem, however, it has been observed that models frequently rely on these correlated features to make predictions (Choi et al. 2012 [Beery et al. 2018, Neuhaus et al., 2022; Moayeri et al., )022), for example, by predicting ox when shown only a green pasture or camel when shown
Figure 14: An image with label ski mask (and also multi-class label alp), but prediction ski.
7

---

a cow in the desert. When correlated features lead to prediction errors, we call these spurious correlations. Here, we focus on the case where the presence of a correlated spurious feature causes a misclassification to the correlated class, despite no corresponding entity being shown. In Fig. [4] we show an example of such a spurious correlation where the presence of a ski mask and a alps causes a model to predict ski, despite the image containing no skis. We do not consider the error mode, where the absence of a correlated feature causes the model to not predict the correlated class, despite a corresponding entity being shown, investigated by Singla and Feizi (2022); Moayeri et al. (2022)
To identify errors caused by spurious cor- relations, we identify pairs of commonly co-occurring classes and then categorize er- rors as spurious correlations if an incorrect model prediction and a multi-label form such co-occurrence pair. More concretely, we first extract all pairs of co-occurring la- bels from the ReaL multi-label annotations (Beyer et al. 2020), excluding samples we evaluate on. We then filter out pairs that either belong to the same superclass as de- fined in $3.3 or only co-occur once. Using this process, we extract 13 090 label pairs from 6 622 images with more than one label, yielding 1019 unique co-occurrence pairs after filtering, which indicate a spurious correlation if they occur.
Portion of Grouphe Milli-Laaall [%.
200
mondiare Intex
150
11
arrdlcts
100
bites
80
80
2003
102
MLA [S]
MLA 1%
Figure 15: Portion (left) and number (right) of MLA errors caused hy spurious correlations by group. A 95% confidence interval linear fit is shown on the right. For artifacts, models are divided into those with more and less than 93% MLA.
We visualize the portion and number of errors caused by spurious correlation in Fig. [5] We observe that artifacts and organisms follow notably different trends. Not only is the portion of errors caused by spurious correlations much larger for artifacts than organisms, but it also increases with MLA for artifacts (at an increased rate for higher MLA), while it stays roughly constant for organisms. For state-of-the-art models, spurious correlations explain up to 15% of errors on artifacts making them the second largest error source we identify.
4 Analysis of Model Errors on ImageNet
In this section, we discuss global trends, analyze their interaction with architecture and training choices, and validate our automatic pipeline against the manual analysis of Vasudevan et al. (002))
Model Failures
After removing all mi-
Portion of Georple Malti- Eroos [%c
75
Portion Group's is Top Ermes
nor (see $3.3 and $3.4) and explainable (see $3.5 and (3.6) errors, from the multi- label errors (MLE), we are left with a set of particularly severe, unexplained model failures (MLF). In Fig. 16 we visualize the portion of these unexplained model failures over multi-label accuracy (MLA) and stan- dard top-1 accuracy, again split by artifact (red) and organism (green). A ratio of corresponds to none of the MLEs being ex- plainable by our pipeline, while a ratio of 0 corresponds to all MLEs being explainable, thus consisting only of less severe error types. Surprisingly, both top-1 accuracy and MIA are pessimistic when it comes to reporting model progress, with unexplained model failures decreasing at much quicker rate than both top-1 errors and MLE.
attiiotss
aniilurbe
25
organimm
20
30
[4]
Top-1 Aerrray
Figure 16: Portion of MLA (left) and top-1 (right) errors that can not be explained and are thus considered model failures- organisms (green) and artifacts (red). 95% confidence interval lincar fit is shown shaded. Models are divided by MLA (at 93%) and top-1 accuracy (at 80%) for linear fits.
Further, we observe different error distributions for organisms and artifacts. Firstly, the portion of model failures is much higher for artifacts than for organisms. Secondly, while the portion of unexplained errors decreases for both organisms and artifacts with MLA and top-1 accuracy, indicating that as models get more accurate they make not only less but also less severe errors, the rate of this change differs. For artifacts, this decrease initially slower but then at around 93% MLA or 80% top-1 error, the portion of severe model failures starts to drop rapidly (roughly three times as fast as before), while the decrease roughly linear for organisms. This phase change becomes particularly apparent when viewing the trend over MLA, where it is even visible for organisms.
8

---

4.1 (Pre-)training Dataset
Throughout [ we have illustrated the pretraining dataset size with the marker hue. Generally, we observe that conditioned on identical MLA, the effect of pre-training dataset size is rather modest. Here, we divide the 12 datasets we consider into categories from "small" (<5M) to "xlarge" (>> 500M) depending on the number of included images, illustrating separate fits in Fig. 17 with a darker shade corresponding to larger dataset. We observe that across a wide range of accuracies, larger pretraining datasets lead to a faster reduction of model failures with MLA for both artifacts and organisms. While this effect is partially explained by larger datasets more frequently leading to higher MLA, it is also observed for narrow MLA ranges. Considering individual error types, we observe that in particular fine-grained errors are more frequent for larger pretraining datasets. We hypothesize that this is due to these larger datasets leading to better underlying representations, allowing the model to succeed in the coarse classification for harder samples, while still failing on the fine-grained distinctions.
Portion of Group's 's Multi- Label Errors [X]
artifacts
smail
mxdium ansty
25
begn 300M)
stargn (o 500M)
organisma
80
100
MLA [%6]
Figure 17: Portion model failures (MLF) over MLA depending on pre- training dataset size with 95% con- fidence intervals for models with at least 82% MLA. Confidence intervals are shown darker for larger pretrain- ing dataset sizes.
4.2 Model Architecture
we again show the portion of unexplained model failures over MLA, this time broken down by architecture type and colored by the number of parameters (larger models have darker hues). To investigate whether modern convolutional archi-
In Fig. 18
Pretion Groe's Multi-Label Errors
75
artilacte
ChenNNltt
tectures (ConyNeXts (Liu et al. 2022 and vision transformers
VIF
(ViT (Dosovitskiy et al. (021). D. Swin (Liu et al., 2021). EVA
Swin
(Fang et al.] )023) exhibit different trends in terms of model failures, we focus on state-of-the-ar models with at least 92% MLA that were trained using a dataset with least 50M images, leaving 28 CNN-based and 79 transformer-ba basedd models. While we observe clear trends of larger models performing better, the rate of model failures seems to be largely independent of model size when conditioned on MLA. We do observe slight trend where the model failure rate of transformers decreases faster with MLA for organisms but slower for artifacts when compared to ConvNeXts. We speculate that this might be due to the ConvNeXts leveraging textures more heavily (Geirhos et al.] 22019), which could be particularly helpful for fine-grained class distinctions on organisms leading to a larger portion of multi-label errors being model failures at the same MLA.
organisus
0
MLA [3]]
Figure 18: Portion of model fail- ures (MLF) over MLA depending on model architecture. 95% confidence interval transformer-basee models is shown darker and for CNN-based based models lighter.
4.3 Comparison to Vasudevan et al. (0022)
To evaluate the alignment of our automated error analysis with panel of human experts, we compare its results to those of Vasudevan et al. (0022) on both networks they consider, here for ViT-3B and in App. for Greedy Soups.
Comparing the results in Table we confirm that our pipeline is conservative, classifying 16.4%
(62 / 378) of the errors assigned an explanation by Vasudevan et al. (022) as model failures. On the remaining errors, our pipeline agrees with the panel of human experts 73.1% of the cases. By manually inspecting the 85 differently classified errors, we determine that for only 32 of these (8.4% of all errors), the categorization by Vaasudvvan et al. (2022) is clearly preferable to ours. Furthermore, our pipeline encodes a minimal-severity bias, absent in Vasudevan et al. (2022)'s 's categorization. That is, when multiple error explanations would be valid, our pipeline consistently chooses the least severe one. This highlights a further advantage of our approach. While any human judgment is inherently subjective and thus prone to differ between expert panels or even for the same panel over time, as acknowledged by Vasudevan al. 2022), our approach is consistent and repeatable.

---

Table 1: Comparison of automated analysis of the of ViT-3B model with the manual annotations from (Vasudevan al. 2022). Rows correspond to the error categories assigned by a human expert while
columns correspond to the error types assigned by our automated pipeline.
Error categories
FG
FG OOV
Non-prot.
Spur. Corr.
Model failures
Total (row)
Fine-grained
192
15
0
10
25
242
Fine-grained OOV
20
0
14
Non-prototypical
13
2
12
0
30
Spurious Correlation
10
12
0
23
52
Total (col)
224
49
12
31
62
378
Observing similar trends for Greedy Soups in App. A confirm that on all models for which human expert error categorizations are available, our automated pipeline is well aligned with their judgment while providing consistent, repeatable, and conservative categorizations.
5 Limitations
In this section, we briefly reflect on the limitations of our work.
Personal Bias Our choices in the implementation of the error analysis pipeline reflect our personal biases and decisions, such the superclasses we picked in $3.3 This is not only limited to our personal biases, but, as our pipeline relies on prior work in several places, it also encodes the biases of their authors. For example, we rely the class overlap mappings from Vasudevan et al. (2022) and multi-class labels by (Shankar et al. (2020) and Vasudevan et al. (2022).
Extension to New Datasets To be applied new dataset, pipeline requires multi-labed label and non-prototypicality annotations and superclass definitions. While less precise than using a gold standard, multi-label annotations could be sourced using Re-Label (Yun et al. 202T) approach, allowing (given sufficient scale) spurious correlation pairs to be extracted from co-occurrence frequencies. After defining superclasses manually required by their very definition, this would allow all error types except for the rare non-prototypical instances to be categorized. With the rest of the pipeline in place, non-prototypical instances could be annotated by reviewing the uncategorized errors shared by multiple well-performing models. However, while feasible these steps still require a significant amount of manual work. We showcase a (partial) adaption the similar IMAGENET-A A dataset in App.
6 Conclusion
As state-of-the-an models come close to and exceed human performance on IMAGENET, focus increasingly shifting towards understanding the last remaining errors. Towards this goal, propose an automated error categorization pipeline that we use to study the distribution of different error types across 962 models, of different scales, architectures, training methods, and pre-training datasets. We distinguish between minor errors, constituting failures on fine-grained class distinctions both when the ground truth was in and out-ol-vocabularr, explainable errors, constituting failures on non-pon non-prototyppcal examples and due to spurious correlations, and unexplained model failures, constituting all remaining errors. We find that even MLA is pessimistic measure of model progress with the portion of severe errors quickly decreasing with multi-label-aae lalel-aaccaacccy Further, we find that organism and artifact classes exhibit very different trends and prevalences of error types. For example, we observe that fine-graine class distinctions are a much bigger challenge for organisms than for artifacts, while artifacts suffer much more from spurious correlations and vot-ol-vocabulary errors, with these trends becoming increasingly pronounced as models become more accurate. We believe that such an analysis of new method's effects on model error distributions can become an important part of the evaluation pipeline and lead to insights guiding future research in computer vision.
10

---

Acknowledgements
We thank our anonymous reviewers for their constructive comments and insightful feedback.
This work has been done as part of the EU grant ELSA (European Lighthouse on Secure and Safe AI, grant agreement no. 101070617) and the SERI grant SAFEAI (Certified Safe, Fair and Robust Artificial Intelligence, contract no. MB22.
Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or European Commission. Neither the European Union nor the European Commission can be held responsible for them.
The work has received funding from the Swiss State Secretariat for Education, Research and Innova- tion (SERI).
References
J. Deng. W. Dong, R. Socher, L. Li, K. Li, and F. Li, "Imagenet: A large-scale hierarchical image database," " in Proc. of CVPR, 2009.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and L. Fei-Fei, "Imagenet large scale visual recognition challenge," Int. J. Comput. Vis., vol. 115, no. 3, 2015.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Proc. of NeurlPS, 2012.
M. Tan, R. Pang, and Q. V. Le, "Efficientdet: Scalable and efficient object detection," in Proc. of CVPR, 2020.
S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, "Image segmentation using deep learning: A survey," IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 7, 2022.
L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, and A. van den Oord, "Are we done with imagenet?" ArXiv preprint, vol. abs/2006.07152 2020.
D. Tsipras, S. Santurkar, L. Engstrom, A. Ilyas, and A. Madry, "From imagenet to image classification: Contextualizing progress on benchmarks," in Proc. of ICML, vol. 119, 2020.
V. Shankar, R. Roelofs, H. Mania, A. Fang, B. Recht, and L. Schmidt, "Evaluating machine accuracy on imagenet," Proc. of ICML, vol. 119, 2020.
G. V. Horn, S. Branson, R. Farrell, S. Haber, J. Barry, P. Ipeirotis, P. Perona, and S. J. Belongie, "Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-gra fine-grained dataset collection." in Proc. of CVPR, 2015.
C. G. Northcutt, A. Athalye, and Mueller, "Pervasive label errors in test sets destabilize machine learning benchmarks," in Proc. of NeurlPS Datasets and Benchmarks, 2021.
H. S. Lee, A. A. Agarwal, and J. Kim, "Why do deep neural networks still not recognize these images?: qualitative analysis on failure cases of imagenet classification," ArXiv preprini, vol. abs/1709,,3439, 2017.
V. Vasudevan, Caine, R. G. Lopes, S. Fridovich- and Roelofs, "When does dough become bagel? analyzing the remaining mistakes on imagenet," in Proc. of NewrlPS, 2022.
S. Yun, J. Oh. B. Heo, D. Han, Choe, and Chun, "Re-labeling imagenet: From single to lalti- from global to localized labels." in Proc. of CVPR, 2021.
H. Mania, J. Miller, L.. Schmidt, M. Hardt, and Recht, "Model similarity mitigates test set overuse," in Proc. of NeurlPS, 2019.
R. Geirhos, Meding, and A. Wichmann, "Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency," in Proc. of NeurlPS, 2020.
11

---

T. Nguyen, M. Raghu, and S. Kornblith, "Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth." " in Proc. of ICLR. 2021.
H. Mania and S. Sra, "Why do classifier accuracies show linear trends under distribution shift?" ArXiv preprint, vol. abs/2012. 154833, 2020.
R. G. Lopes, Y. Dauphin, and E. D. Cubuk, "No one representation to rule them all: Overlapping features of training methods," in Proc. of ICLR, 2022.
A. Andreassen, Y. Bahri. B. Neyshabur, and R. Roclofs, "The evolution of out-of-distribution robustness throughout fine-tuning," " ArXiv preprint, vol. abs/2106. 15831, 2021.
S. Singla and S. Feizi, "Salient imagenet: How to discover spurious features in deep learning?" in Proc. of ICLR, 2022.
M. Moayeri, S. Singla, and S. Feizi, "Hard imagenet: Segmentations for objects with strong spurious cues," in Proc. of NeurlPS, 2022.
D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, "Natural adversarial examples," " in Proc. of CVPR, 2021.
M. R. Taesiri, G. Nguyen, S. Habchi, C. Bezemer, and A. Nguyen, "Imagenet-hard: The hardest images remaining from a study of the power of zoom and spatial biases in image classification," CoRR, vol. abs/2304.05533. 2023.
B. Y. Idrissi, D. Bouchacourt, Balestriero, I. Evtimov, C. Hazirbas, N. Ballas, P. Vincent, M. Drozdzal, D. Lopez- Paz, and M. Ibrahim, "Imagenet-x: Understanding model mistakes with factor of variation annotations." in Proc. ICLR, 2023.
C. G. Northeutt, L. Jiang, and I. L. Chuang. "Confident learning: Estimating uncertainty in dataset labels," J. Artif. Intell. Res., vol. 70, 2021.
G. A. Miller, "WordNet: A lexical database for English," in Speech and Natural Language: Proceed- ings of a Workshop Held at Harriman, New York, February 23-26, 1992, 1992.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin,
J. Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision." in Proc. of ICML, vol. 139, 2021.
Y. Neuhaus, M. Augustin, V. Boreiko, and M. Hein, "Spurious features everywhere large-scale detection of harmful spurious features in imagenet," CoRR, vol. abs/2212.04871, 2022.
J. Choi, A. Torralba, and A. S. Willsky, "Context models and out-of-context objects," Pattern Recognit. Lett. vol. 33, no. 7, 2012.
S. Beery, G. V. Hor,, and P. Perona, "Recognition in terra incognita," in Proc. of ECCV, vol. 11220, 2018.
Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, "A convnet for the 2020s," in Proc. of CVPR, 2022.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, Uszkoreit, and N. Houlsby, "An image worth 16x16 words: Transformers for image recognition at scale," in Proc. of ICLR. 2021.
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang. S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows." " in Proc. of ICCV, 2021.
Y. Fang, W. Wang, B. Xie, Q Sun, L. Wu, X. Wang, Huang, X. Wang, and Y. Cao, "EVA: exploring the limits of masked visual representation learning at scale," in Proc. of CVPR, 2023.
R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, "Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness," in Proc. of ICLR, 2019.
12

---

A Extended Comparison to Vasudevan et al. (2022)
Similarly the analysis in Sec. 4.3 now evaluate our automated error classification pipeline on Greedy Soups, the only other model for which manual error categorizations exist (Vasudevan et al. 2022), showing the results in Table 2 We do not assign an explanation 11.6% (29 249) of the errors, classifying them as model fallures. On the remaining errors, our pipeline agrees with the human expert annotation 74.1% (163 / 220) of cases. After a manual review the remaining 57 differently classified errors, we determine that the categorization provided by Vasudevan et al. (2022) is clearly preferable to ours for only 26 samples (10.4% of all errors). Overall, the results for Greedy Soups very similar to those for ViT-3B in 4.3 thus further validating choices. Therefore, we believe that our automatic categorization is indeed aligned with the opinion of human experts and note that efficient and automatic classification pipeline is valuable even if is imperfect, as it enables the study of models and trends at scale.
modeling
Table 2: Comparison of our automated analysis of the errors of a Greedy Soups model with the manual annotations from Vasudevan et al. 2022). Rows correspond to the error categories assigned by human expert while columns correspond to the error types assigned by automated pipeline.
Error categories
FG
FG OOV
Non-prot.
Spur. Corr.
Model failures
Total (row)
Fine-grained
139
11
172
Fine-grained OOV
7
8
5
6
26
Non-prototypical
8
2
0
19
Spurious Correlation
Total (col)
161
22
29
249
13

---

B
Additional Results
In addition to the analysis presented in the main paper, which focused on the portion of errors due to the different error categories for artifacts and organisms, provide more detailed results in this section, considering absolute error counts as well as portions for artifacts, organisms, the remaining classes, and all classes jointly.
Figures T9
24
contain these statistics grouped by error type:
.
Fig. 19
top-1 errors caused by class overlap ($3.1):
Fig. 20
top-1 errors due to missing multi-label annotations 33.2
. Fig. 21
fine-grained multi-label errors ($.3);
Fig.
fine-grained aut-of-vocabulary multi- errors ($3.4)
.
Fig. 23
multi-label errors due to non non-prototypical instances ($3.5):
Fig. 24
multi-label errors due various spurious correlations ($3.6).
We show the number of errors in relative and absolute terms. In each figure, subfigures (a) and (b) show the results for each group of classes separated in "artifacts", "organisms"" and "others" (the rest of the labels), where for (a) the relative portions of top-1 or multi-label errors are computed per group, i.e., by diving by the number of top-1 or multi-label errors the respective group. Red hues again correspond to artifacts, green hues to organisms, and blue hues- to the remaining classes (others). Subfigures (c) and (d) present the relative and absolute statistics for all classes jointly.
Partian Ceocaale Tro-1 Erross (K)
* Top- Envor
Portson of ATl Topel Broes Ni Dep-1 Ences
celp
utlacs
90
15 Top-1
90
Tope
Arcesayy
Top-1 Accumay
Avcrrway [X]
Typ-1 Aconnyy
(n) Relative portion per group.
(b) Abs. number per group.
(c) Relative portion samples. (d) Abs. number all samples.
Figure 19: Top-1 errors caused by class overlap. The relative portion of these errors on "others" is smallest and the trend is noisier compared to "artifacts" and "organisms". This is possibly due to the low absolute numher of classes in this group (68 out of 1000). In absolute terms, the number of this kind of error remains relatively constant.
Portion Crorp's Top-1 Eroos (S))
Tapel
Portion of Top-l Droes Xi
Top-1
100
caly
1000
730
900
500
clacks
30
Tep: Accrrncy 166
Topo Acouncy
Top-1 Acorcwcy [4]
(a) Relative portion per group.
(b) Abs. number per group.
(c) Relative portion all samples. (d) Abs. number for all samples.
Figure 20: Top-1 errors caused by missing multi-lahel annotations. Surprisingly, the portion of these errors the highest on the "others" group. absolute terms, "organisms" and "others" result in roughly the same constant amount of errors, despite the much larger number of organism classes (410 vs. 68). For "artifacts", it first increases and then starts dropping, possibly due to more accurate models learning to leverage IMAGENET labeling biases.
14

---

of Ghoup's Mitisaahd Brron 1%
Fortacs of Mitistabe Blmoss [S]
3aah--abe Errves
orgasbere
Hays
1500
01
artifacts
40
1000
100
200
10 101
100
192
100
MILA []]
MLLA [X]
MLA [S]
MLA
[S]
(a) Relative portion per group.
(h) Abs. number per group.
(e) Relative portion all sam- (d) Abs. number samples.
ples.
Figure 21: Multi-label errors due fine-grained misclassifications, The portion of fine- fine-grained errors "others" larger than those on "artifacts" but smaller than "organisms".
Perticn of Gooppl Malts-aied
Laied Emasi [5]
201
Xaatis Label Emoas
Fortine All Calee Erso [S]
Medl Labed Emore
15
150
artlbote
15
MIAA
MLA 35
MILA
[X]
MEA
(3) Relative portion group.
(h) Abs. number per group.
(c) Relative portion for all sam- (d) Abs. number for all samples. ples.
Figure 22: Fine-grained voocaulooca multi-label errors. The portion of errors follows a similar trend for "artifacts" and "others", while the sudden sharp drop in absolute numbers for "artifacts" is absent for "others".
Portion Groppe Maatelaaee Brrce [Q]
Laid Bimos
Poerios of Malli Labd Emer (8]
20" Lalis Ernes
aetiffate
10
10
Iylval
crgaallima
XLA
MLX
MLX
MLA
(a) Relative portion per group.
(b) Abs. number per group.
(c) Relative portion for all (d) Abs. number for all samples. ples.
Figure 23: Multi label errors identified as non- con-proootypical instances. The number of samples identified as non-protolyppcal by /Vasudevan al. (2022) is inherently small (36) and the majority of them are artifacts, leading noisy results. As the models get better other types of errors, the relative portion of multi-label errors due to conppooootypiaal images increases.
Pertice of Groppl Malti-akee Frox [X]
Laie Emara
Portioe All Milest abel Froee Bi
Tipe Fmox
150
200
16
10
oegintiine
XLA
MLA
MLA [X]
MLA
(a) Relative portion per group.
(b) Abs. number per group.
(c) Rolative portion for all sam- (d) Abs. number for all samples. ples.
Figure 24: Multi-lahel errors caused hy spurious correlations. In relative terms, "artifacts" and "others" follow a similar trend which is noisier for the "others" group.
Finally, in Fig. 25 we present the ratio of unexplained model failures (MLF) over multi-label accuracy (MLA) and standard top-1 accuracy. Again, subfigures (a) and (b) are organized per group, while (c) and (d) show the results for all samples.
15

---

70
Pietion Group's Emurx
15
Grappi Top-J
Strves
25
Portion Top-l
aetioatt
10
20
urilitt
20
26
25
alacs
WI MILA [5]
300
16
300
Tap-1 Avumay [X]
MILA [8]
Topel Areraay [3]
(a) Portion of unexplained model fail-(b) Portion of unexplained model (c) Portion of unexplained model (d) Portion of uncxplained model ures per group.
failures per group.
failures for all samples.
failures samples.
Figure 25: Ratio unexplained model failures (MLF) over multi label accuracy (MLA (a) and (c)) and standard top-1 accuracy ((b) and (d)). In general, our pipeline explains the biggest portion of emror made on "organisms" and the lowest portion of errors on "artifacts". However, better models quickly close this gap. For the best models, our pipeline explains up 85% of all multi label erors and 95% of all top-1 errors.
16

---

C Analysis of IMAGENET-A
To demonstrate the applicability of automated error analysis pipeline to other datasets, we now apply it to
Imsgg Top-1 Ace [3]]
ImageNett-A Top-1 Ane (36
up
saxill (cr (XC)
monifnn (< D0M))
5
6U-
largy (<r tocks)
IMAGENET-A. IMAGENET-A was in-
sdanpe 5008d)
troduced by Hendrycks et al. (202T) as NEAGEeETTlike test set of "Nat- ural Adversarial Examples". It con- tains 7500 images from 200 IMA- GENET classes, picked to fool a set of ResNet-50 classifiers, while still constituting high-quality, single-entity images, clearly depicting an entity of the target class. A comparison of IMAGENET and IMAGENET-A accuracy is shown in Fig. 26 In this section, marker color encodes model parameter counts, i.e., larger models have darker markers.
40
40-
30
05
80
15
50
65
80
96
IneegeNee "Top-1 Aecuracy [%]]
ImagpeNet Top-1 Aucuracy [34]
Figure 26: IMAGENET-A A top-1 Accuracy IMAGENET top-1 Ac- curacy grouped by architecture (left) and training data size (right).
As IMAGENET-A images were selected to be challenging for IMAGENET classifiers at the time, we see that many classifiers with IMAGENET accuracies below 80% have almost 0 accuracy on IMAGENET-A. A. In fact, 801 of the 962 models we consider have an accuracy of less than 30% on IMAGENET-A, As classifiers improved (> 80% top-1 IMAGENET accuracy) IMAGENET-A accuracy catches up rapidly. Among other improvements, the main driver seems to be the model parameter count. Note that due to the much lower IMAGENET-A accuracy (of some models), the x- x-axis of all further plots starts at 0% rather than 60%, as before.
As IMAGENET-A A uses IMAGENET classes, keep the split into organisms (77 out of 200 classes, 38.5%), artifacts (104/ / 200, 52%), and other classes (19 / 200, 9.5%).
Class Overlap Due to the shared class structure, we can use the same analysis as for IMAGENET and show
Portion of Group's Tsp-1 Ervos [X]
Coottet Class Overlap Top Erros
mlp
25"
20
the results in Fig. 27 The 200 classes in IMAGENET-A were selected from the 1000 IMAGENET classes avoid class overlap and label ambiguity, Thus, as expected, we see a much lower overall amount of such errors (below 0.8% instead of up to 8.5% for IMAGENET). In particular, for ar- tifacts and other classes, we see no class overlap errors at all. Interestingly, for organisms, we still observe small number (S 222 of errors caused by class overlap. We observe that errors become more frequent (albeit still rare) for models with higher accuracy. As with IMAGENET this is due to more accurate models classifying more images correctly, including to overlapping classes.
betest
15.
10
10
20
30
40
50
60
70
To
AmaggeNei-A Tup-1 Axcuracy
[X]
ImagaNNet-A
Rep-1 Ausuray [4]
Figure 27: Portion (left) and number (right) of top-1 errors caused by class overlap for organisms (green). For the artifacts and other groups we observe no errors.
Multi-Label Annotations In contrast to IMAGENET, IMAGENET-A A was designed to only in- clude single-entity images. Thus, skip the multi-label analysis for IMAGENET-A A and use top-1 I accuracy/crror rather than MLA accuracy/crror for all further analyses.
Fine-Grained Classification Errors
50
Portice of Group's Top-1 Brcoss [%]
mip
Fine-Craimol Tog-1 Errots
Due to the shared class structure, we
40-
con
800-
can apply the same analysis including superclasses for fine-grained classifi-
orgomienze
60C-
organista
others
10C-
cation errors as on IMAGENET. We
show results in Fig. 28
200
artifucts
attilacte
othing 2om
As with IMAGENET, the group for which we observe most of these er- rors are organisms (explaining up to 47% of the group's 's errors). However,
1 10
30
60
40
60 70
IanggeNet-A Top-1 Accuncy [%]]
IeangeNet-A Top-1
Aceunay [X]
Figure 28: Portion (left) and number (right) of top-1 errors caused by fine-grained misclassifications by group organisms (green), artifacts (red), and others (blue).
17

---

the portion of explained errors is much lower than for IMAGENET, where up to 88% of MLA errors for organisms can be explained by finegrainedd classification errors. We see the percentage of errors increase linearly with accuracy. Looking at absolute counts, especially for organisms, we see that fine-grained errors initially become more prevalent as model accuracy increases before dropping off as models more frequently also succeed in the fine-grained class distinction.
GrainG Out-of- Vocabulary Errors Again, we can use the same superclass definition, CLIP embed- ding and open world classifier for IMAGENET to identify fine-grained OOV errors. We generally observe slightly larger portions of OOV er- rors than IMAGENET and much clearer linear dependence on accuracy (see Fig. (9). Notably, the superlin- ear increase observed on IMAGENET (see Fig. IT) is absent here. Looking at absolute counts, after an initial in- crease and stagnation, we observe a linear decrease with model accuracy. change.
Poction Group's Top.1 Ertons
[9]
Fine-Grained GOV Top-1 Brross
300
200
10-
100
1/ 10
ImageNett A Top Accuracy [%6
Imugge Not-A
Tog-1 Ancurary [X9]
Figure 29: Portion (left) and number (right) of top-1 errors identified as fine- O0V by group organisms (green), artifacts (red) and other (blue).
In contrast to IMAGENET, we again do not observe a slope
Non- Non-Prototypical Instances Our analysis pipeline for IMAGENET uses the manual annotation of protottyotyllll instances from (Vasudevan al. 2022). As we do not have these available for IMAGENET-A A we skip this analysis step here. However, for standard IMAGENET we only observe a small number of such errors (S 2% of MLE) and note that the IMAGENET-A images were manually selected to only include high-quality images (Hendrycks et al. 2021). Thus, we expect their portion to be even smaller here.
Spurious Correlations Again fol- lowing our IMAGENET approach for
Portion of Group's Top-1 Errors [X]]
# Spurions Complation By-1 Rrive
75
spurious correlation errors we obtain the results in Fig. 30 At first glance, we observe much less spurious cor- relations than for IMAGENET (6.2% compared to 15% for artifacts). How- ever, we observe a more linear instead of super-linear increase in the portion of errors explained by spurious corre- lations, leading to this reduction being smaller for less accurate models.
traedoraee
Asteil
50-
25
10
40
50
60 70
50
ImagoNNtttA
Top-1
Acruncyy
[X]
ImageNett Top Accuracy [%]
Figure 30: Portion (left) and number (right) of top-1 errors identified
as spurious correlations by group organissms (green), artifacts (red) and other (blue).
Model Failures Looking at the re-
Portion of All Top-1 Erors (%)
Portion All Top-1 Erroes [X]
maining failures, we see that we can only explain up to half of the errors for the best-performing models. Not only top-1 and MIAA error rates much higher than for IMAGENET but also a much larger portion of the remaining errors constitutes severe model fail- ures, highlighting that IMAGENET-A is a much harder dataset.
90
90
60-
zipy
>>
30-
10
20
50
SD
100
ImageNNtt.J Top-1 Accuracy [S]
Irsuge/eet MLA [X]]
Figure 31: Portion for remaining failures over IMAGENET-A A top-1 (left) and IMAGENET MLA (right).
However, analyzing Fig. 31 ob- serve that both the IMAGENET-A A top-1 and IMAGENET MLA accuracy are good predictors for the portion of model failures which decreases with increasing model accuracy. This highlights again and perhaps surprisingly that IMAGENET MLA underreports progress in model performance, even on particularly challenging subdistributions like IMAGENET-A.
18

---

D Extended Examples on Fine-Grained OOV Error Classification
In this section we provide further step by step examples of the procedure that determines a given model mistake is categorized as fine-grained OOV error. The sample in Fig. 32 is confirmed to be
fine-grained OOV, the one in Fig. 33 is not fine- grained and the one in Fig. 34 is rejected as OOV.
GT: suit
Pred: bicycle-buil serlle-built--ee
Pred superclass: motor, eycle
(1) GT: suit suits
(2) GT: suit suits
(3) GT: unicycle motor_eyde _ecleeeyylle__
(4) GT: backpack bag
(5) GT: sutt suits
GT: noped motor_eycle /
(7) traffic Light No superclass
(8) plastic bag No superclass
(9) GT: bicyclo-buile built-for two motorr_cycle /
(10) GT: bicycle- built-for. tuo cootr__cycle-.
Figure 32: Fine-grained 00V confirmed. evaluated validation sample (top)
ground-truth IMAGENET
label (GT) suit, hut the model predicts bicycle-bute bu f-rrowtwo which falls in the "motor_cycle" superclass. Below, we show the top- 10 training images that are the most visually similar the validation sample (according CLIP similarity) together with their IMAGENET labels and the superelasses of these labels. The labels of images 3, 6, and are
same superclass as predicted label (motor cycle). hence we proceed with procedure. After obtaining the in-vocabulary and OOV label proposals, as described in Sec. [3.4 the proposals with the highest probability according to CLIP are: safety bicycle. ordinary bicycle, ve velocipede and bicycle, all of which are OOV, confirming that the error is indeed fine- tine-grained 0OV.
19

---

JHHN
GT: rship, analog clock Pred: jinrikisha
Pred superclass: open_cart
clock
gergions_castte_piiee
car bus_bbsssnn track
GT: tran train
GT: tran train
GT: tran train
GT: tran train
GT: tran Irain
GT: unbrella No superclass
GT: tran train
Figure 33: Fine-grained OOV rejected. The evaluated validation sample (top) has ground-truth IMAGENET multi-labels (GT) ai rirship and analog clock, but the model predicts jinrikisha, arguably attempting to classify the carriages the street. The predicted label is in the "open_ cart" superclass. Next, show the top-10 training images are most visually similar the validation sample (according to CLIP similarity) together with their IMAGENET labels and the superclasses of these labels. None of the labels is in the "open cart" superclass, thus we terminate the procedure and reject the mistake as possible fine-grained OOV crror.
20

---

GT:
Pred: shopping basket
Pred superclass:
())
(2) lawn nower work_com
shopping box /
upent_cart
GT: shopping box /
GT)
tricycle motor_ motor__yycle
open_cart
open_cart
(8) GT: bassinet baby_hed
(9) GT: tricycle cestor_cyclee
(10) GT: cradle haby hed
Figure 34: Fine-grained OOV rejected. The evaluated validation sample (top) has a ground-truth IMAGENET label (GT) barrow, but the model predicts shopping basket, arguably attempting to classify the basket in the cart. The predicted label is in the "box" superclass. Next, we show the top- 10 training images that are the most visually similar the validation sample (according CLIP similarity) together with their IMAGENET labels and the superclasses these labels. The labels images 3 4 are in same superclass as predicted label (box), hence we proceed with the procedure. After obtaining the in-vocabulary (IV) and OOV label proposals, as described Sec. Scc.[3.4] the proposal with the highest probability according to CLIP is shopping cart, which is in-vocabulary, therefore, the mistake is not categorized as fine-grained OOV error. The other most likely proposals are: shopping basket (IV), bushel basket (OOV), basket (OOV) and handbasket (OOV).
21

---

E
Mistake Examples by Error Type
GT. weasel, polecat, nink, black-fostad ferret Prod: polecat
GT: screen, nonitor Pred: nonitor
GT: laptop, notebook
GT. projectill,, nissile Pred: missite
Pred: natebook
UWW US
TA. 25
GT: elephant,
nug, cup
cassette player. tape player Pred: tape player
husky. Eskino dog Pred: Eskiro dog
Pred: tusker
Pred: cup
GTT naillot, Pred: tank suit
sungless, glasses Pred: dark glasses
bathtub, weshbasin Pred: tub
Figure 35: Class overlap: examples of common prediction errors (Pred) due to overlapping classes. The correct (individual) multi-labels (GT) separated by commas; the original IMAGENET label is listed first.
22

---

GT: monastery, altar
GT: nedicine chest, xashbasin, GT: English foxhound, beagle shounr curtain Prod: shower curtain
GT: paper towel, dack Pred: dock
Pred: bragle
GT: green snake. green nanba
GT: partridge, ptamrigan ptarmigan
GT: ground beetle, Pred: tiger beetle
GT: Lumbernill, chainlink fence Pred: chainlink fence
Pred: green
GT: minibus, school Pred: school bus
GT: Pred: nousetrap
GT computer desktop computer Prod: desktop computer
GT: chacolate
Pred: chocolate sauce
GT: cucumber, zurchini
GT: alp, ibex Pred: 1bex
GT. honeyconb, pollow millaag Pred: pillow
GT: burrito Pred: burrito
Pred: zucchini
GT: container ship, drilling platfors drilling platfom
GT: computer desk, desktop computer, screen conputer keyboard desktop computer
space
GT: shoji, sliding door Pred: sliding
Pred: conputer keyboard
Figure 36: Multi-label: examples common prediction errors (Pred) multi-label errors. The correct
(individual) multi-labels (GT) are separated by commas; the original IMAGENET label is listed first.
23

---

www.taariii frrheez Lallod Midda
GT: china cabinet
GT: water jug pitcher
GT: hare
GT: nonastery. stone wall Pred:
tabby
noving van Pred: tow truck
Pred: coyote
Pred: Egyptian cat
Pred: broccol1
aaa Twwn
GT: bloodhound
GT: triraran
GT: tobacco shop
stoth bear Pred:
INDIANS COWBOYS
GT: squirrel monkey Pred: titl nonkey
GT: Pred: coffeepot
GT: boak jacket Prod: conic book
GT: whippt Pred: Saluki
GT: grey fox Prod:
GT: crate apiary
GT. window screen, cup Pred: egenog
GT: Norwich terrier Pted:
Figure Fine-grained: examples common prediction errors (Pred) fine-grained error. The correct (individual) multi-labels (GT) are separated by commas; the original IMAGENET label is listed first.
24

---

GT: sunglass, dark glasses schnauzer 00V: Other dog breed
GT: suit bicycle-built OOV: Single-person bicycle
GT: velvet cardigan OOV: Ganment
GT: china cabinet
Pred: toyshop
OOV: Other shop store
GT: nicrovave
GT: wardrobe stiding doer OOV: Wardrobe door
GT: car wheel, grille beach wagon OOV: Other car model
GT: cannon aircaatt carrier OOV: Military vessel
wardrobe
O0V: Cupboard cabinct
GT: lumbermill
GT: dock
GT: wheel Pred: sports car OOV: Coupe other model
GT: bear
Pred: crane
Pred: rcraft carrier
Pred: toyshop
00V: Lifting device facility
0OV: Guard ship facility
OOV: Gift shop other
GT:
GT: tiger
paddle Pred: canoe
GT:
Prod: ice crean
Prod: Lynx
Pred: pufferfish
Baby ocelot
OOV:
GT. hammer, Pred: shavel
GT: strainer Pred: cucunber
GT:
Pred: altar
Pred: revalver Semiautomatic pistol
type
vegetable
Altar
Figure Fine-grained 0OV: examples of common prediction errors (Pred) due to fine-grained OOV errors. The correct (individual) multi-lahels (GT) are separated hy commas; the original IMAGENET label is listed first.
25

---

GT: pencil. box, purse pillow
GT: padlock
GT. breastplate Pred: water jug
GT: flute Pred: plunger
339
GT: namrsset Pred:
GT. badger
GT: water jug Pred: espresso
GT: green Lizard Pred: buckeye
GT:
foxhound Pred: Staffordshire terrier
nipple Pred: candle
Pred: pinwheel
Pred: radio
LY
GT: sirliner
GT: lisstick
GT: bonnet, lipstick. saltshaker
GT: jack-o'-tentern Lentern Pred: traffic Light
Pred: bucket
Figure 39: Non-prototypical: examples of common prediction emrors (Pred) due non-prototypice instances. The correct (individual) multi lalti-labels (GT) are separated by commas; the original IMAGENET label listed first.
26

---

sanita
MODELI M772-3189
GT. vestnentt
GT: washbasin Pred: nedicine chest
tank suit, naillot Pred: bathing
chain Pred: hook
Pred: altar
SAT
Subject
100
GT: grille
GT. atterhound Pred: briard
GT: ski nask, elpp Pred: ski
GT: bookcase Pred: bookshap
Pred: police van
GT: strawberry
GT: flowerpot
GT: wool Pred: bonnet
GT: chain, suspension bridge
Pred: trifle
Pred: greenhouse
Pred:
GT. traffic Light, crash helnet
GT: kuvasz
GT. Windsor tie Pred: suit
GT: water jug Pred: water bottle
Pred: Sanoyed
Pred: bicycle-buti buil--fforttttt
G
TABACOS
Partrik
GT: bulletproof police van Pred: nilitary uniforn
GT: warplane, pole, chain Pred: ai reraft carrier
GT: pole Pred: traffic Light
GT: tobacco
Pred: street sign
Figure 40: Spurious correlations: examples of common prediction (Pred) due to spurious correlations. correct (individual) lalti- (GT) are separated by commas; the original IMAGENET label listed first.
27

---

KIRIN
wardrobe
backpack Pred: unicycle
Loggerhead Pred: scuba diver
Pred: refrigerator
Pred: vending nachine
GT: saccer ball Pred: potter's wheel
GT. missite, projectile obelisk
GT. overskirt Pred: apron
GT: durbbell
Pred:
GT: pay-phone Pred: cash nachine
hatchet, Pred: pedestal
GT: Pred: lab coat
Pred: confectionery
GT. letter opener Pred: pencil sharpener
GT: suit. unbrella Pred: swing
GT: bannister Pred: crib
GT: passenger car Pred: studio couch
GT: pig Pred: greenhouse
GT: frying pan
GT: dark glasses, sunglass Pred: sunscreen
GT: thinble
Pred:
Pred: microphone
Figure 41: Model failures: examples of common prediction errors (Pred) classified model failures. The correct (individual) multi-labels (GT) are separated by commas; the original IMAGENET is listed first.
28

---

Experimental Setup
In this section, we provide further details on our evaluation.
We publish our code and detailed instructions on how to reproduce our results at htt.://ww....teettsss////tttttte -tt.rommtttdd-errrr---aallssssss analysiss
F.1 Computing the Multi-Label Accuracy
When computing the models' multi-label accuracies, we closely follow the recommendations by Shankar et al. (220 In particular, we discard the set of problematic images, then compute per-class accuracies and finally average them to obtain the multi-label accuracy. We consider a model prediction be correct if it is marked correct unclear in the multi-label dataset.
F.2 Datasets and Labels
We consider the validation set of the ILSVRC-2012 subset of IMAGENET (Deng et al. 2009
Russakovsky et al. 2015), available under non-commercial research license" More concretely y. we
use the subset of this validation set labeled by Shhnnar et al. (2020) and then Vasudevan et al. (2022)
with the labels being available under Apache License 2.0. We further evaluate our pipeline on the ImageNet-A dataset (Hendrycks et al. 2021) available under MIT License
F.3 Summary Evaluated Models
Table contains list of all models we considered in this study and suhset of their metadata.
The models were obtained from multiple sources: Torchvisior torch. hut?] HuggingFace and timm [ collecting all model outputs (6 days for ImageNet and I day for ImageNet-A on a single GeForce RTX 2080 Ti GPU), running our error analysis pipeline on all models takes 12 to 24 hours using single GeForce RTX 2080 GPU for ImageNet and ImageNet-A respectively.
They can all be automatically downloaded and evaluated using our open-sourced code. After
Table 3: List of evaluated models and a subset of their metadata.
No
Model ID
Source
Arch Dataset
alexnet
torch
ImageNet-
timm enn ImageNet-Ik
convformer b36. inlk
timm cnn ImageNet-Ik
4.
canyformer_b36.co Lnnailliikkk111 in1k_384
timm cnn ImageNet-1k
timm
5.
nmwformer__b36.co
b33.saillin22k f_inkk
ImageNet-21k
6.
convformer_b33 bai. i_____222__________ innkkk33__
timm
ImageNet-21k
convformer_mm36.aal
cnn
ImageNet-Ik
8.
naavformer_m36.ssa
inlk_38 384
timm can ImageNet-1k
9.
convformer_m366.al fiaill________iii__
timm
cnn
ImageNet-21k
10.
convformer_m36.sail Laaaaail__i2_______e
convformer_s18.sai__mm frikkkff_innk3e_
timm
ImageNet-21k
11.
cnn
ImageNet-Ik
convformer sl8.sail inlk 384
timm
can ImageNet-Ik
13.
convformer_slB8 in22k inlk
timm
can ImageNet-21k
14.
convformer_sIB3 sl8.saii in22k ft inlk
timm
can ImageNet-21K
15.
convfommer_36..cal. saik6.ail_inkkkk
timm
ImageNet-Ik
convformer_s36 sail sail_inlk_388
enn ImageNet-Ik
17.
convformer_s36.smm _t____n22_______ inlk
timm
cnn
ImageNet-21k
Table continues onto next page
https://ww.
more
https::/ppytorc..orgghbbbb
http:://luggingaaaeccc//oodettt

---

Model summaries (continued from previous page)
No
Model ID
Source
Arch
Dataset
18.
convformer manffrrmer_s36.sal s33..sail (el.xall_iiii _el__k_22_____ in4
timm
cnn ImageNet-211
19.
convmixer 1024 20 ks9 1s99_p14.inlk
timm
can
ImageNet-
20.
convmixer 1536 20.inlk
timm
can ImageNet-1K
21.
convmixer 768 32.inlk
timm enn
ImageNet-1k
22.
convnext-base-222.
huggingfacc
ImageNet-Ik
huggingface
cnn ImagoNet-21k
24.
convnext-base-384
huggingface
enn ImageNet-
25.
convnext-base-384-222.
huggingface
ImageNet-21K
26.
convnext-large2224
huggingface
ImageNet-Ik
convnext-large-384
huggingface
can ImageNet-Ik
28.
convnext-large-388
huggingface
can
ImageNet-21K
29.
conanext-small-224
huggingface
can ImageNet-1k
30.
convnext-tiny-224
huggingface
ImageNet-Ik
31.
convnext-llrrge-224-22----l
huggingface
ImageNet-21k
convnext-llagee-884222-
huggingface timm cnn
cnn
ImageNet-21k
33.
convnext_atto.d2_2 dt_inddd__nkkk
ImageNet-1k
34.
convnext atto ols.a2 inlk
timm can ImageNet-1K
35.
convnext base
torch
ImageNet-1k
convnext base.clip laion2b augreg in12 inlk
timm
can
LAION-2B
37.
timm
cmn
LAION-2B
38.
timm
cnn
LAION-2B
39.
fonnnex
timm
ImageNet-1k
40.
convnext fewtofelltsseeeee inlk
timm
ImageNet-Ik
convnext large
torch
cnn
ImageNet-Ik
42.
convnext_ large..biin22k_fttt ft22k_ffff____ ft_ikkkk
timm
cnn
ImageNet-21k
43.
larbeext_arrr__aeaa mwp.cciiplaiinnb Jaion2b, _ugrre inlk
timm
can
LAION-2B
44.
convnext wwtpe/emll.llpp.ppm laion2b Luuooaauuee___ ft iiiiiink3330 384
tmm
LAION-2B
convnext large mwp.clip laion2b laion2b_soup inlkk inlk
convnext_large large_____r______eeeeee soanext_lrrge_mmppcln wawp.cliplliionnbbbbbbppp.
timm
can
LAION-2B
timm
cnn
LAION-2B
47.
timm
cnn
ImageNet-lk
48.
convnext_nano caraanoo_ossds_iiiii.
timm
ImageNet-1k
49.
convnext pico.dl inlk
timm
ImageNet-Ik
convnext_ pico ols.d1 inlk
timm
can ImageNet-Ik
51.
convnext small
torch
can ImageNet-Ik
52.
convnext small.fb in22k ft inlk
timm
can ImageNet-21k
53.
fownext_small.fb. 2ic_ffbiin22_ttt_ 1n22k_ft_nnkkk3666 ___inlk3844
tim
ImageNet-21k
convnext tiny
cnn ImageNet-Ik
55.
convnext_ tin...f.btnn.bbbbb.o... in22k ft_ iniikk
timm
can
ImageNet-21k
56.
perneex_tinnyfb.. in22k 11222__ft_inile ft_inik_384
timm
cnn
ImageNet-21k
57.
convnext_tiny tiny www.hnnffahhiinkk conveex__in__bnf.22__lllll
timm
ImageNet-1k
58.
convnext_ munnexx_xxlarge.clin httpluneeeeliplllinnn_oooooo
Jaalllliiooooonaeeaeee foo__sou__ft_ikk inlk
timm
cnn
LAION-2B
convnextv2 femae inlk
can ImageNet-Ik
60.
convnextv2 base.femace inlk
timm can
ImageNet-1k
61.
convnextv2 base.femae ft in22k inlk
timm can ImageNet-21k
62.
convnextv2 www.ffma_fftin222_iint
timm
ImageNet-21k
convnextv2 femttofcmma__fe. fffmc_fft_nikk
enn
ImageNet-Ik
64.
convnextv2 tange.femae_ftee ft_inkk
timm cnn ImageNet-Ik
65.
convnextv2 huge.femae ft i__ii_n2______ inlk_384
timm
can ImageNet-21k
66.
2onvnextv2 huge.femae f__in2kkinil__
timm cnn ImageNet-21k
67.
convnextv2 large.femae ft inlk
timm
ImageNet-Ik
convnextv2 Jarge. Swwvnextv__Jare.ffomm. ft_taa______nk______ ____in22__iig_ inlk
enn ImageNet-211
69.
convnextv2
wwwp:/ttty__aaee.oooooo lerge.fcmae _t_in22k_nnlk_3k 384
timm can ImageNet-21k
70.
convnextv2_name htw.:/222nnnoffmmaa..m wtw.oooccme.f...mmm in
timm
cnn ImageNet-Ik
71.
1itrmextt2__ano.femmte faae_____n22___l_____1__
timm
ImageNet-21k
72.
convnextv2 cen.nextt2_nnno.ffmmee ft__n22k_iinn_____ 3883
timm
ImageNet-21k
convnextv2 convnextv2_pico.fcmat fico.femac. ft. inli
timm
cnn ImageNet-Ik
74.
convnextv2 tiny.femae ft inlk
timm
can ImageNet-1k
75.
convnexty2 finy.femae ft in22k inlk
timm
can ImageNet-21k
76.
convnextv2 fenafe ft in22k inlk 384
timm
ImageNet-211
Table continues onto next page
30

---

Model summaries (continued from previous page)
No
Model ID
Source
Arch
Dataset
ex3darknet focus_l.c2nnn Las____nn________eee inlk
timm cnn
ImageNet-11 1k
78.
cs3darknet focus m.c2ns inlk
timm
can ImageNet- 1k
79.
cs3darknet_ Lc2ns inlk
tim can ImageNet-Ik
80.
cs3darknet m.c2ns inlk
ImageNet-Ik
81.
esădarknet_x.c2ns,
enn ImageNet-Ik
es3sedarkmet_.ee2n, inlk
timm
cnn ImagoNet-Ik
83.
csisedarkmet_x.c2mms inlk
timm
enn ImageNet-1K
84.
cspdarknet53.ra_ing
timm can ImageNet-Ik
85.
cspresnet50.raa _an_5.kkka
timm
ImageNet-Ik
espresnext50.ra. inlk
timm
cnn ImageNet-1k
87.
darknet53.e2ns inlk
timm can ImageNet-Ik
88.
darknetaa53.c2ns. inlk
timm can ImageNet-
89.
densenet121
torch
ImageNet-Ik
90.
densenet121.ra in1k
timm
ImageNet-Ik
16
cnn ImageNet-1k
92.
169
torch
cnn ImageNet-1k
93.
densenet201
torch
can ImageNet-Ik
94.
densenetblur121d.r inlk
timm
ImageNet-Ik
dla102.inlk
timm can ImageNet-Ik
96.
dla102x.inIk
timm
cnn ImageNet-lk
97.
dla102x2.in1k
timm cnn ImageNet-1k
98.
dla169.inIk
timm
ImageNet-Ik
99.
dla34.in1k
timm
ImageNet-Ik
100.
dla46_ccinlk
cnn ImageNet-Ik
101.
dla46x_c.inl
timm cnn ImageNet-lk
102.
dla60.in1k
timm can ImageNet-Ik
103.
dla60 res2net.inlk
tim can
ImageNet-Ik
104.
dla60 res2next.inlk
ImageNet-Ik
105.
dla60x.in1k
cnn ImageNet-Ik
106.
dla60x_c.inlk
timm cnn ImageNet-lk
107.
dmn_nfeetfO.dmm_inl
ImageNet-1k
108.
dm nfnet fl.dm inlk
ImageNet-Ik
109.
dm nfnet f2.dm inlk
can ImageNet-Ik
110.
dm nfnenfnett f3.dm inlk
can ImageNet-1k
111.
iin_nfnet [4.dm inlk
timm can ImageNet-Ik
112.
tim cnn ImageNet-Ik
ImageNet-Ik
113.
dm dm_nfnet_ffo..ommm faaeet_ffddm_mmmm
114.
apn107.mxx inlk
timm can ImageNet-lk
115.
dpn131.mx
mmni1..xxjnnkk
timm cnn ImageNet-1k
1 116.
dpn68.mx_in1k
timm
ImageNet-1k
117.
dpn68b.mx inlk
cnn ImageNet-Ik
118.
dpn68b.ra inlk
can ImageNet-Ik
119.
dpn92.mx inlk
timm can ImageNet-1k
120.
dpn98.mx inlk
can ImageNet-Ik
121.
cca_halonex226sscl__in
ImageNet-1k
122.
cea_nfnet_1O.ra__in
enn ImageNet-Ik
123.
[ca_nfine_ll.ra2_111
cnn ImageNet-Ik
124.
eca 12a_n
12.ra3 inlk
timm can ImageNet-1k
125.
eca eca_ressme res.eeemt33..sa2....11113 inlk
timm cnn ImageNet-1k
126.
eca oca_resnex126ss.hh inlk
ImageNet-Ik
127.
ccaresnetl01d.miil..
enn
ImageNet-Ik
128.
ecaresnet101d Bociid_puneddddee miil inlk
timm can ImageNet-
129.
ccaresnet269d.raa.1 in
timm cnn
ImageNet-
130.
ecaresmet26.rra_inle
timm
ImageNet-1k
131.
ecaresnet50d..mil... _n
timm
cnn ImageNet-1k
132.
ccaresnet50d_ maatiieddddd__rmmmmmmimmm..
timm
cnn ImageNet-1k
133.
ecaresnet50Lal inlk
timm
can ImageNet-1k
134.
ecaresnet50t.a2 inlk
timm
can ImageNet-
135.
ecaresnet50La3 inlk
timm
ImageNet-Ik
Table continues onto next page
31

---

Model summaries (continued from previous page)
No Model ID)
Source
Arch
Dataset
136.
ccaresmet50t.ral inlk
timm cnn
ImageNet-11 1k
137.
ecaresnetlight.mi inlk
timm can ImageNet- 1k
138.
efficientnet b0.ra_ inlk
can ImageNet-Ik
139.
efficientnet, bl.ft inlk
ImageNet-Ik
140.
enn ImageNet-Ik
141.
lfficientnet_bb.ra..
timm
cnn ImagoNet-Ik
142.
#ffiieetnet_b2__ruundd.
timm
cnn ImageNet-1k
143.
efficientnet b3.r__3_r______a inlk
timm
can ImageNet-Ik
144.
efficientnet b3 prunedd.nnkk b
timm
ImageNet-Ik
145.
efficientnet b4.ra2. in
timm
cnn ImageNet-1k
146.
afficientmet_el.t lkk
timm
can ImageNet-Ik
147.
efficientnet pww.epprreeiiiiieeee Piiicientte_ee__prnnediii in
timm can ImageNet-
148.
efficientnet_emt em.ra2_ inlk
tim
ImageNet-Ik
149.
efficientmet_es.ra. inlk
ImageNet-Ik
150.
setncettet_es_prunddde
timm
cnn ImageNet-1k
151.
Efficientnt_llitc..e __jnnll k
timm
cnn ImageNet-1k
152.
efficientnetv2 rw maagc inlk
timm
can ImageNet-Ik
153.
efficientnetv22 s.ra2. inlk
Gimm
ImageNet-Ik
154.
efficienttetv2 rw tra2 inlk
can ImageNet-Ik
155.
fbnete_
@onete_10.rmsppimmg
timm cnn
ImageNet-lk
156.
Dpnetv3_b.ra2_mm
timm cnn ImageNet-1k
157.
seetv3dd.raa_ing
timm
ImageNet-Ik
158.
fbnetv3_g.Fd Baatw33ggra2_imm in_k
timm
ImageNet-Ik
159.
focalnet base base_lrf.ms._..... inlk
cnn ImageNet-Ik
160.
docalnet_base
inlk
timm cnn ImageNet-lk
161.
feaannt_smalll [nall_Irf.ms ms _nikk inlk focalnet small srf.ms inlk
timm can ImageNet-Ik
162.
timm
ImageNet-Ik
163.
focalnet tiny Irf.ms inlk
timm
can ImageNet-Ik
164.
focalnet tiny_ simy_srf..ss_im _nlk
timm
cnn ImageNet-Ik
165.
ge_ re__efficiettnet2_rww jw_tagc_innk
timm cnn
ImageNet-lk
166.
geresnet33ts.ra__n ____k
timm
ImageNet-1k
167.
gcresmet50t.ra2 inlk
timm
ImageNet-Ik
168.
geresnext26is.ch inlk
timm
can ImageNet-Ik
169.
gcresnext5Ots.ch inlk
timm
can ImageNet-Ik
170.
googlenet
torch
can ImageNet-Ik
171.
aaonet26t.alhiiit
timm cnn
ImageNet-1k
172.
halonet50ts.alh inlk
ImageNet-Ik
173.
hardcorenas. _amilll__am__a___.._ arreei_lgeeeeenn___a_eee green_inlk inlk
timm can ImageNet-lk
174.
hardcorenas, b_miil________.. Bealll_ggeee___eee green_inkk
timm cnn ImageNet-1k
175.
hardcorenas_c.nm Laaereeaaascmmmlm____1 Saclll_greeennie in
timm
ImageNet-1k
176.
hardcorenas_ sa.iereaanadmdlmi____ee Balmmill_eeeeiiml green_inlk
cnn
ImageNet-Ik
177.
hardcorenas e.miil green inlk
can
ImageNet-Ik
178.
hardcorenas. f.mii green inlk
can
ImageNet-1k
179.
hrnet w18.ms inlk
can ImageNet-Ik
180.
mine_ww8.mmsjin
ImageNet-1k
181.
hret_w18_smaa www.11_smmllls_iiiii mamall.s_inkkk ms_innk
enn ImageNet-Ik
182.
hrne_w18__sm
@_w1__smal_____ simall_v..ma _il___2___i_mmmmm inlk
cnn ImageNet-Ik
183.
hrnet_ w18 _=s__ssdd. inlk
timm can ImageNet-1k
184.
hrnet_w.30.ms.
timm cnn ImageNet-1k
185.
krmet_w32.ms. inlk
ImageNet-Ik
186.
hrnet_w40.ms. inlk
enn
ImageNet-Ik
187.
wss.t_w44.ms. inlk
timm can ImageNet-
188.
hrnet_w48.ms bioo_w4mmms__kk inlk
timm cnn
ImageNet-
189.
timm
ImageNet-1k
190.
hrnet_w64.ms, in
timm
cnn ImageNet-1k
191.
cncpttionresnnt. -ii___2.f__ees... fif_ens_adv_ink
timm
cnn ImageNet-1k
192.
inception resnet v2.tf inlk
timm
can ImageNet-1k
193.
inception v3
torch
can ImageNet-
194.
inception v3.gluon inlk
timm
ImageNet-Ik
Table continues onto next page
32

---

Model summaries (continued from previous page)
No
Model ID
Source
Arch
Dataset
195.
inception v3.tf _d.tf_ady__i_ ady__nik inlk
timm cnn
ImageNet-11 1k
196.
inception_ v3.tf inlk
timm
can ImageNet- 1k
197.
inception_ v4.tt inlk
tim can ImageNet-Ik
198.
lambda_resnetee resnet26rpt 256.cl inlk
ImageNet-Ik
199.
lambda_resnot26f.c1___
enn ImageNet-Ik
200.
lambda_resnot50tsaall..
timm
cnn ImagoNet-Ik
201.
timm
cnn ImageNet-1k
202.
lenet Omne__05.ra2__11 inlk
timm
can ImageNet-Ik
203.
Enee_100.nz j__
timm
ImageNet-Ik
204.
mixnet_.ff I.ft inlk
timm
cnn ImageNet-1k
205.
mixnet_m.ft inlk
timm can ImageNet-Ik
206.
mixnet_s.ft_nnl l
timm can ImageNet-
207.
mixnet pootet_x_rr__la xl.ra_inkk
timm
ImageNet-Ik
208.
mnasnet(_5
torch
ImageNet-Ik
209.
mnasneto)
torch
cnn ImageNet-1k
210.
mnasnet1 0
torch
cnn ImageNet-1k
211.
mnasnetl 3
torch can ImageNet-Ik
212.
mmasnet 100.rmsp inlk
timm
ImageNet-Ik
213.
mmnssne samb inlk
timm
can ImageNet-Ik
214.
mobilenet_v2
torch
cnn ImageNet-lk
215.
mobilenet_v3_lare
torch
cnn ImageNet-1k
216.
mobilenet_v3_ssme
torch
ImageNet-Ik
217.
mobilenetv2
2a0e2_0000lmabbbbb inlk
timm
ImageNet-Ik
218.
mobilenetv2 101.10000iinll. inlk
timm
cnn ImageNet-Ik
219.
mobilenetv2_11000 111d111ddrrammmm inlk
timm
cnn ImageNet-lk
220.
mobilenetv2_12U01 120d.ra_innk
timm can ImageNet-1K
221.
mobilenetv2_ 1ere2__11rrr__a__e_ inkk
timm
ImageNet-Ik
222.
mobilenetv3 Iarge_Ir000 mii.:ei00mmmll........ in21k inlk
timm
can ImageNet-21k
223.
mobilenatv3_large_10000 Laa.aarr__1.Orrnninlll.. _nnkk
timm
cnn ImageNet-Ik
224.
mobilcnetv:_rr.rmsp_..
timm
cnn ImageNet-lk
225.
mobilenetv3_small (050.llmb_jnn
timm
ImageNet-1k
226.
mobilenetv3_smme _mm33___mll____1 la5.lamb inlk
timm
ImageNet-Ik
227.
mobilenetv3 3mmalll__lll_mm la0. bncibbinlkkh inlk
timm
can
ImageNet-Ik
228.
af regnet bl.ra2, inlk
timm can ImageNet-Ik
229.
nf resmet550ra2__nn 2_ikk
timm
can ImageNet-Ik
230.
ffnet_10.ra2___ in
timm
ImageNet-Ik
231.
regnet_x_16g 16gf
torch
cnn ImageNet-Ik
232.
regnet_x_1 1_6gf
torch
can ImageNet-lk
233.
regnet_x_ x x_32gf
torch cnn ImageNet-1k
234.
regnet_x x x_3____3______ x_3_2gf
torch
ImageNet-1k
235.
regnet_* 400mf
torch
cnn ImageNet-Ik
236.
regnet 800mf
torch can ImageNet-Ik
237.
regnet x 8gf
torch
can ImageNet-Ik
238.
regnet 128gf swag
torch
can Instagram
239.
regnet_y_16gt
torch
ImageNet-1k
240.
gonnt_y_llff_swee
torch
enn Instagram
241.
regnet_y_ porctt_yl_66pe '6f
torch
cnn ImageNet-Ik
242.
regnet_ y ___2
torch
can ImageNet-Ik
243.
regnet_ y "_32gf swag
torch
cnn Instagram
244.
regnet_ y y_3 y_3_2gf
torch
ImageNet-Ik
245.
regnet_y4400mf 400mf
torch
enn ImageNet-Ik
246.
regnet_y. y y_800mf
torch
can ImageNet-Il lk
247.
regnet_y_8ge
torch
cnn ImageNet-
248.
regnetv regnety(04Orra3 00_0000000r_0.
_nlk
timm
ImageNet-1k
249.
regnetv_ regnetv_064.ra33 000.0.04.33_immm rikk_
timm
cnn ImageNet-
250.
regnetx_002.pp
timm
cnn ImageNet-1k
251.
regnetx 004lp inlk
timm
can ImageNet-1k
252.
regnetx 00clp inlk
timm
can ImageNet-
253.
regnetx 0ycls inlk
timm
ImageNet-Ik
Table continues onto next page
33

---

Model summaries (continued from previous page)
No
Model ID)
Source
Arch
Dataset
254.
pennexx006.pyyeee ln
timm cnn
ImageNet-11 1k
255.
regnetx 032.pycls inlk
timm can ImageNet- 1k
256.
regnetx 040,,ycls inlk
can ImageNet-Ik
257.
regnetx 064.pycls inlk
ImageNet-Ik
258.
gggnetx_80.pycls_.in
enn ImageNet-Ik
259.
regnetx_120.pycls_a
timm
cnn ImagoNet-Ik
260.
regnetx_
rggeet__6d..pcclsinikk
timm
enn ImageNet-1k
261.
regneix_
Nenett_320.pycls_inll inkk
timm
can ImageNet-Ik
262.
regnety_
Sinetty_022.yclssiin inlk
timm
ImageNet-Ik
263.
pegnet__04..pye inlk
timm
cnn ImageNet-1k
264.
Sonett_006.pycls..
timm
can ImageNet-Ik
265.
seneet_008,pyclss
timm can ImageNet-
266.
regnety_016.p
inlk
tim
ImageNet-Ik
267.
regnety_
(ately_o22.yycl_iigg inlk
ImageNet-Ik
268.
regnety_0322 (acety__32rrr_iimi inri
timm
cnn ImageNet-Ik
269.
regnety_ regnety_(40.pyeee pycl. in
timm
cnn ImageNet-1k
270.
regnety 040.ra3 inlk
timm
can ImageNet-Ik
271.
regnety 064.pycls inlk
timm
ImageNet-Ik
272.
regnety 064.ra3 inlk
can ImageNet-Ik
273.
pyyeety000..yyls_fii inlk
timm cnn
ImageNet-1k
274.
(anett_000.ra3....
timm cnn
ImageNet-Ik
275.
rcgnety_120.pycls. in
timm
ImageNet-Ik
276.
regnety_1280.scer inlk
timm cnn
RI
277.
regnety_I6dl bandy__66..dcitinnkk
timm cnn
ImageNet- 1k
278.
regnety_160) 1e0pplls inlk
timm cnn
ImageNet-1K
279.
regnety_2560.see. 2560.seerr inlk
cnn
RI
iu
280.
regnety_ 3atoly_3300ppyssssse inlk
timm cnn
ImageNet-Ik
281.
regnety_320.n se0..seer_____ inlk
RI
282.
regnety 640.seer.. inlk
timm can
RI
283.
regnetz 040.ra3 inlk
timm can ImageNet-IK
284.
regnetz 040 h.ra3 inlk
ImageNet-Ik
285.
regnetz_bb6.raa
cnn ImageNet-Ik
286.
regnetz_el6.ra3 inlk
cnn ImageNet-1k
287.
regnetz_cl6_ewee whllleevoocchh..ooeeee cnlk
timm can ImageNet-
288.
regnetz_d32.raf inlk
timm
ImageNet-1k
289.
sggnet__8.ral inlk
ImageNet-Ik
290.
regnetz_d8 _etss_heecchhhhhee inik
cnn ImageNet-Ik
291.
regnetz_e8..o ra.. inlk
can ImageNet-1k
292.
repvge a2.rvge inlk
timm can ImageNet-1k
293.
repvge. b0.vvgg inlk
ImageNet-Ik
294.
repvgg_bl.ree bl.gggll..ggiiii.... inlk
cnn
ImageNet-Ik
295.
repvgg_bl rlg..vgggg.. inlk
timm cnn ImageNet-1k
296.
timm cnn ImageNet-1k
297.
repvge_ b2g4.rvgg inlk
timm cnn ImageNet-Ik
298.
repyge_ repvgg_b3.rvge b3.rvgg_ inlk
timm
ImageNet-Ik
299.
repvgg_ repvgg_b3g4.rvg inlk
cnn ImageNet-Ik
300.
res2net101
timm can ImageNet-
301.
res2net10ld.img
timm cnn ImageNet-
302.
res2net50 ww__8s.inkk
timm
ImageNet-Ik
303.
res2net50 26w_4s.inlk
timm
ImageNet-Ik
304.
res2net50 26.n6k.
timm
cnn ImageNet-1k
305.
res2net50
1aii50026ww_000 26w__s.wnl
timm
cnn ImageNet-1k
306.
res2net50
1kinkk
timm
ImageNet-1k
307.
res2net50d.inlk
timm
ImageNet-Ik
308.
res2next50.nnlk
timm
cnn ImageNet-Ik
309.
resnestlOle.nnlk
timm can ImageNet-Ik
310.
resnest14d 2e.111d.ggluoo.
timm cnn
ImageNet-
Table continues onto next page
RRandommnneemeemmgess
34

---

Model summaries (continued from previous page)
No Model ID)
Source
Arch
Dataset
311.
resnes220e.inlk
timm cnn
ImageNet-11 1k
312.
resnest269e.inlk
timm
can ImageNet- 1k
313.
resnes(26d, inlk
timm can ImageNet-Ik
314.
resnest50d.inlk
timm
ImageNet-Ik
315.
Sanest50d_ls4224d.mm
enn
ImageNet-Ik
316.
cesnest50d_42x440..ntt
timm
cnn ImagoNet-Ik
317.
resnet101
torch
enn ImageNet-1k
318.
resne1101.aliin inlk
timm
can ImageNet-Ik
319.
resnet101.alh_ inlk
timm
ImageNet-Ik
320.
resnet101.a2 in
timm
cnn ImageNet-1k
321.
resnet101.a3
timm
can ImageNet-Ik
322.
resnet101.glus 1aaee100.gguuonnnee ll
timm can ImageNet-
323.
resnet101c.gluon. inlk
tim
ImageNet-Ik
324.
resnet/01d.g/uon inlk
ImageNet-Ik
325.
resner101d.ra2_ine
timm
cnn ImageNet-1k
326.
resnct101s.glunn_ing _i
timm
cnn ImageNet-1k
327.
resnet10r.c3 inlk
timm
can ImageNet-Ik
328.
resnet14t.c3 inlk
timm
ImageNet-Ik
329.
resnet152
torch
can ImageNet-Ik
330.
52.net152.al
timm
cnn ImageNet-lk
331.
timm cnn ImageNet-1k
332.
resnet152.a2_im _n_
timm
ImageNet-Ik
333.
resnet152.a3_ing
timm
ImageNet-Ik
334.
1esnet152.gluon in
timm
cnn ImageNet-Ik
335.
(2c.ll152.. k
timm
cnn ImageNet-lk
336.
lnlk
timm can ImageNet-IK
337.
resnet152d.ra2 inlk
timm
ImageNet-Ik
338.
resnet152s.gluon inlk
timm
can ImageNet-Ik
339.
resnet18
torch
cnn ImageNet-Ik
340.
resnet18.al
timm
cnn ImageNet-lk
341.
essne18..a2_im
timm
ImageNet-1k
342.
resnet18.a3 inlk
timm
ImageNet-Ik
343.
resnet18.gluoo inlk
timm
can ImageNet-Ik
344.
resnet18 ssl
torch
can
Flickr YFCC
345.
(annell_sswss
torch
can Instagram
346.
essne118d.ra_in _dkkka
timm
ImageNet-Il
347.
resnet200d.ra2. in
cnn ImageNet-Ik
348.
resnet26.bt_inle
timm can ImageNet-1k
349.
resnet26d.bt inlk
timm cnn ImageNet-1k
350.
resnet26t.ra2_im kn1k
timm
ImageNet-1k
351.
resnet32ts.ra2.1 in1k
timm
cnn
ImageNet-Ik
352.
resnet33ts.ra2 inlk
timm
can ImageNet-Ik
353.
resnet34
torch
can ImageNet-1k
354.
resnet34.al inlk
timm can ImageNet-Ik
355.
resnet34.a2_in ln
timm enn
ImageNet-1k
356.
resnet34.a3_in ___k
ImageNet-Ik
357.
resnet34.bt_nld
timm cnn ImageNet-Ik
358.
resnet34.gluon inlk
timm can
ImageNet-1k
359.
resnet34d.ra2_inlll
timm cnn ImageNet-1k
360.
resne150
torch
ImageNet-Ik
361.
resnet50.al
enn ImageNet-Ik
362.
resnet50.alh_in inlk
timm can ImageNet-
363.
resnet50.a2.in) inlk
timm
cnn ImageNet-
364.
resnet50..3_imm in
timm
ImageNet-1k
365.
resnet50.am l
timm
ImageNet-1k
366.
resnet50.blk.. jinkkkk
timm
cnn ImageNet-1k
367.
resnet50.b2% inlk
timm
can ImageNet-1k
368.
resnet50.bt inlk
timm
can ImageNet-
369.
resnet50.c1 inlk
timm
ImageNet-Ik
Table continues onto next page
35

---

Model summaries (continued from previous page)
No Model ID)
Source
Arch
Dataset
370.
resnet50.c2_ inlk
timm cnn
ImageNet-14 k
371.
resnet50.d inlk
timm
can ImageNet- 1k
372.
resnet50.gluo0 inlk
timm can ImageNet-Ik
373.
resnet50.ra inlk
timm
ImageNet-Ik
374.
resnet50.ram_
enn
ImageNet-Ik
375.
resnet50_gn waw.asooggg.hhhhhmc aii_illkk
timm
cnn
ImageNet-1k
376.
resnet50_ssl
torch
enn
Flickr YFCC
377.
resnet50_ssv
torch
Instagram
378.
resnet50c.glunn inlk
timm
ImageNet-Ik
379.
resnet50d.al in
timm
cnn ImageNet-1k
380.
resnet50d.a2. inlk
timm
can ImageNet-IK
381.
resnet50d.a3 inlk
timm can ImageNet-
382.
resnet50d.glus .loionn.gnoonn... in
tim
ImageNet-Ik
383.
resnet50d.ra2 ijink
ImageNet-Ik
384.
eesnet5Os.gluon, in
timm
cnn ImageNet-1k
385.
1esnet51q.ra2..1 _aa__inkkk
timm
cnn ImageNet-1k
386.
resnet6lq.ras inlk
timm
can ImageNet-Ik
387.
resnetaa50.alh inlk
timm
ImageNet-Ik
388.
resnetblur50.br inlk
can ImageNet-Ik
389.
1fsnetrs101.tf. _nlk
timm cnn ImageNet-lk
390.
resnetrs152.rr tf
timm cnn ImageNet-1k
391.
resnetrs200.tf_fim _fnk
ImageNet-Ik
392.
resnetrs270.tf. inlk
timm cnn ImageNet-Ik
ImageNet-Ik
393.
resnetrs350.ff inlk
394.
resnetrs420.tf_im inlk
timm cnn ImageNet-lk
395.
resnetrs50.tf _nlk
timm can ImageNet-1k
396.
resnetv2 aatpee2_11lalhhh__a 101.alb_inkk
timm
ImageNet-Ik
397.
resnetv2 101x1 bit___iiigooo_ing.
inokg_nn2kk..1 inlk
can
ImageNet-21k
398.
resnetv2_101s1
hatbb/__iiggoo_in21141 fnlk
timm cnn ImageNet-21k
399.
resnctv2_152x2_
http://1_2222iiggognnnna.....___ ffn__inn21kfttinlk
timm
cnn ImageNet-21k
400.
resnetv2_1522 2atpe22_11222_h11_1 355222_bitggoog_ttaccee_____ _ia__inn22ffflll__
timm
ImageNet-21k
401.
resnetv2. 152x2 bit.goog teacher ik21k ft inlk 384
Gimm
ImageNet-21k
402.
resnetv2 152x4 bit.goog in21k inlk
can ImageNet-21K
403.
resnetv2 aesaetv2_50.ahhh inlk
timm
can ImageNet-Ik
404.
resnetv2 _50d__eos.ahhh inlk
timm
can ImageNet-1k
405.
resnetv2_50d___ 50d_gn.ah_inlk
tim
ImageNet-1k
406.
resnetv2_500 10____00__biii___ L___bbtggoo_ddstiiiii
inlk
cnn ImageNet-21k
407.
resnetv2
1esnetv2_50xl__c 50x1
bit. in2kk 122k ft inlk
timm can
ImageNet-21k
408.
resnetv2 3a5netv2_50x3____ brtt:3bbtgooo_nn211111 in2k inlk
timm cnn
ImageNet-21k
409.
mesnext101_3244 12d2x6dd_sss
torch
Flickr YFCC
410.
resnext101__55 16d__ swsl
torch
Instagram
411.
resnext101 32x16d. wsl
torch
cnn
Instagram
412.
resmext101 32x32d wsl
torch
can
Instagram
413.
resnext101 32x48d wsl
torch
can
Instagram
414.
(amnex/00_33x4d.ghhee
timm
ImageNet-1k
415.
1amext001332x4d
torch
enn
Flickr YFCC
416.
resnext101 11_00011_______ee swsl
torch
cnn
Instagram
417.
resnext101 32x8d
torch
can
ImageNet-Ik
418.
resnext101 33x8dd ssl
torch
cnn
Flickr YFCC
419.
resnext101 1_xxxxx000_2xdd___ swsl
torch
Instagram
420.
resnext101 1_100011______ww____
torch
enn Instagram
421.
resnext101 1_x44
torch
can ImageNet- 1k
422.
@esnext101_64x44
timm
cnn
ImageNet- 1k
423.
resnext101 1amext101_64xx4.ggll
__f_x_4...ggoooiimm. inlk
timm
ImageNet-1k
424.
resnext26ts.ra2_is
timm
enn ImageNet-1k
425.
1reext50_32x444
torch
cnn ImageNet-1k
426.
resnext50 32x4d.ll inlk
timm
can ImageNet-1k
427.
resnext50 32x4d.alh inlk
timm
can ImageNet-Ik
428.
resnext50 32x4d.a2 inlk
timm
ImageNet-Ik
Table continues onto next page
36

---

Model summaries (continued from previous page)
No
Model ID)
Source
Arch
Dataset
429.
resnext50_322eee _3xx4dd.3 inlk
timm cnn
ImageNet-Ik 1k
430.
resmext50 32x4d.gluon inlk
timm
can ImageNet-Ii 1k
431.
resnext50 32x4d.ra inlk
timm can ImageNet-Il
432.
resnext50 32x4d ssl
torch
Flickr YFCC
433.
resnext50_32xde 32x4d_swsl
torch
enn Instagram
434.
resnext50d (aixxx55ddd3xdddbbbb _33x4d.bt_imm
timm
cnn ImageNet-Ik k
435.
rexnet_100.nat mak_nnkkk
timm
enn ImageNet- lk
436.
rexnet_ mmeet_130.nav __ink
timm
can ImageNet-IK
437.
Sxnnet_10.nne ma__kilkkk inlk
timm
ImageNet-Ik
438.
rexnet_ rexnet_200.nav n00.nav __ill
timm
cnn ImageNet-Ik
439.
300.._.
timm
can ImageNet-Ik
440.
schalonet3ttsrra__ing
timm can ImageNet-
441.
semnasnet_ cemnasnet_075.mmm 075.mmmppppmmmmmmmmmmmmmmm inlk
tim
ImageNet-Ik
442.
semnasnet_ matpusseee_000mmmpp_h1..... inlk
ImageNet-Ik
443.
1edesnet152d.ra2. __2__nlkkk
timm
cnn ImageNet-1k
444.
seresnet33ts.ra__ing
timm
cnn ImageNet-1k
445.
seresmet50.al inlk
timm
can ImageNet-Ik
446.
seresmet50.a2 inlk
timm
ImageNet-Ik
447.
seresmet50.a3 inlk
can ImageNet-Ik
448.
seresnet50.ra2
timm cnn ImageNet-lk
449.
scremext101_32xxddggle
_3xx4d.gluon_nnl _nink
timm cnn ImageNet-1k
450.
cresnext101332x666 ___xxddddh__nnll.. inlk
timm
ImageNet-Ik
451.
seresnext101 64x4d.gluon inlk
timm
ImageNet-Ik
452.
seresmext101d 332xddah in
cnn ImageNet-Ik
453.
2ressexx26dd332x44.bb 3t inlk
timm cnn ImageNet-lk
454.
timm can ImageNet-IK
455.
seresnext26ts.ch inlk
timm
ImageNet-Ik
456.
seresmext50 32x4d.gluon _nkk
timm can
ImageNet-Ik
457.
seresnext50 30x4500032xxdddmmmmmm inlk
timm cnn ImageNet-Ik
458.
seresnexiaa 101d_ 32..d.xxxhhhh__. aat inlk
timm
cnn ImageNet-lk
459.
shufflenct v2__0_5
torch
ImageNet-1k
460.
shufflenet v2 xl_0
torch
ImageNet-Ik
461.
shufflenet x1__
torch
can ImageNet-Ik
462.
shufflenet v2 x2_0
torch
can ImageNet-Ik
463.
skresnet18, 18.ra inlk
timm
can ImageNet-Ik
464.
skresnet34.ra rnkk
timm cnn ImageNet-Ik
ImageNet-1k
465.
466.
squeezenet1_0
torch
can ImageNet-lk
467.
squeezenet1_1
torch
timm cnn ImageNet-1k
468.
if paaerienttet_bhlaa_inn
ImageNet-1k
469.
If_ wwwefffciettntthh..co lttpitttttthhhppi_l_1___
cnn
Ima
470.
efficientnet b0.inlk
can
ImageNet-Ik
471.
If efficientnet b0.ns. jitssssttttte inlk
timm can
JFT
472.
If efficientnet bl.aa inlk
timm can
ImageNet-Ik
473.
tf f__cfficientmet_ll.a.....
ImageNet-1k
474.
#f_efficientnet_.l batttnntttbbiiinll.
enn
ImageNet-Ik
475.
ff_efficientnet_b bl. joljjjjnn_l inlk
cnn
JFT
476.
efficcenttet b2.aa inlk
timm can
ImageNet-Ik
477.
If efficientnet b2.ap_in11 in
timm cnn
ImageNet-Ik
478.
wwfficientnet__22222 b2.inlk
ImageNet-Ik
479.
wwf.ccienteet_..... ___.n___ft_________._________ _n__jft_inkk
enn
JFT
480.
efficientnet ba..aattbbbaaaan.ee.e inkkka_inkk
timm can
ImageNet- lk
481.
tf effficientnet_.333
timm cnn
ImageNet- 1k
482.
If ff_efficientnet_ha3 __3.ink
timm
ImageNet-1k JFT
483.
[__efficientmet_ht henneett_hhh______eee_______ js__fttiinkk inlk
timm
484.
timm
cnn ImageNet-1k
485.
efficientnet b4.ap inlk
timm
can ImageNet-
486.
If efficientnet b4.inlk
timm
can ImageNet-
487.
If efficientnet b4.ns jft inlk
timm
JFT
Table continues onto next page
37

---

Model summaries (continued from previous page)
No
Model ID
Source
Arch
Dataset
488.
Lefficientnet_ii aaaaaett_hhaaaaa_aaeeaaa _akk__inkkk
timm cnn
ImageNet-Ik
489.
If efficientnet bS.ap_ inkk inlk
timm can
ImageNet-
490.
If efficientnet b5.inlk
can
ImageNet-Ik
491. If efficientnet bS.us inlk
JFT
492.
enn ImageNet-1K
493.
cfffficientet__bbaa___a
timm
cnn ImagoNet-1K
494.
if wwwffccienteet_bb.ppp....
timm
enn
ImageNet-lk
495.
If efficienttet__boe b6.ns_jft ____s_jf___ inlk
timm
can JFT
496.
efficientnet_
_a_pe___________aa inlk
timm
ImageNet-Ik
497.
efficientnet__b//
_ inlk
timm
cnn
ImageNet-Ik
498.
efficientmet_ b_.ns_____________pp__ _t_.i__jft___ii.i inlk
timm
can
JFT
499.
if efffcienttett_b..... wttpeeet__rrrallllell
timm can
ImageNet- lk
500.
efficientnet_ bt..eeet____pppp_pa__a____ sp_inlk
tim
ImageNet-IK
501.
ImageNet-Ik
502.
Lefffccientnet_c__@@ catpttt____hhh__iiiii_ee
timm
cnn ImageNet-1k
503. If efficiinttnt_cc__ll fa__ttt___hhh___i__i__
timm
cnn ImageNet-1k
504.
If efficientnet cc bl inkk
timm
can ImageNet-Ik
505.
efficientnet.elle. el.inlk
timm
ImageNet-Ik
506.
efficienttet em.inlk
can
ImageNet-Ik
507.
if _f____fffffe cefffccientntttesnnnkkk
timm cnn
ImageNet-1k
508.
if efffccientnee22.nnsjjfiiilll
JFT timm cnn
509.
if wtttcciemmmet_..nnnsiiiii______.
timm
JFT
510.
Baticiientne_llito..nnll
timm
ImageNet-Ik
511.
effficientne__litel...oo
linkl
cnn ImageNet-Ik
512.
If cittiienttne_lit22.ili
timm cnn ImageNet-1k
513.
If 3ettriennttt_tte33.....
timm can ImageNet-Ik
514.
If efffcienttet
timm
ImageNet-Ik
515.
efficientteet2
10.inkk
can ImageNet-Ik
516.
If f__efficientnet2__llii
timm cnn ImageNet-Ik
517.
if [Lefficientnett2_22.inl
timm cnn
ImageNet-lk
518.
If f__ffficeetteevv_bb.... bb3inkkk
timm
ImageNet-1k
519.
efficiinttetv22 b3.in21k ft.kkft... inlk
ImageNet-21k
520.
efficientnetv2 l.ink
can ImageNet-Ik
521.
efficientnetv2 1in21k inlk
timm can ImageNet-21K
522.
If efficientnetv2 mminlk
timm can ImageNet-1k
523.
if
timm cnn ImageNet-Ik
ImageNet-21k
524.
offffciienttett_ssiinkkk
525.
If wtfficieentnetv_ssinnltt in inlk
timm can ImageNet-21k
526.
If L_eeffciiennetvv_llliiee inlk
timm cnn ImageNet-21k
527.
ff_mixnetl.inle
timm
ImageNet-1k
528.
f__mixnet_minnkk
cnn
ImageNet-Ik
529.
mixxeet_s.inlk
can ImageNet-Ik
530.
mobilenetv3
large_075.inll. 075.inlk
timm can ImageNet-1k
531.
if mobilenetv3
100.inlk large_100.inlk
timm can ImageNet-Ik
532.
tf ___mobilnnttv__lrre__miinie Lattrrr__mmimmall1011..
100.inkk
timm enn
ImageNet-1k
533.
ImageNet-Ik
534.
If @f_mobilcncv3__mmll1100000
timm cnn ImageNet-Ik
535.
If [__mbillentt3__smate minimal 100.in
timm can
ImageNet-1k
536.
tinynet_a.in)
timm cnn ImageNet-1k
537.
tinynet_b.inl biink
timm
ImageNet-Ik
538.
dinyne_ccinlk
enn
ImageNet-Ik
539.
tinyne_ddinnlk
timm can ImageNet-
540.
tinynet_e.nnkk
timm
cnn ImageNet-
541.
tresnet_.mmillimgg inlk
timm
ImageNet-1k
542.
besnet__miillinkk...
timm
ImageNet-1k
543.
tresnet_m.mill_ne inlk
timm
cnn ImageNet-Ik
544.
tresnet m.mil inlk 448
timm
can ImageNet-1k
545.
tresnet m.miil in21k ft inlk
timm
can ImageNet-211
S46.
tresnet v2 Lmiil in21k ft inlk
timm
ImageNet-211
Table continues onto next page
38

---

Model summaries (continued from previous page)
No
Model ID
Source
Arch
Dataset
547.
tresnet_xl.an beanet_xl.miil. inlk
timm cnn ImageNet-11
548.
tresnet xlmiil inlk 448
timm can
ImageNet- 1k
549.
vgel1
torch can ImageNet-Ik
550.
vgg bn
torch
ImageNet-Ik
551.
1g
torch
enn ImageNet-Ik
552.
vgg 1e1___n
torch cnn ImagoNet-Ik
553.
1g
torch enn ImageNet-1k
554.
vgg16 bn
torch can ImageNet-Ik
555.
1g
torch
ImageNet-Ik
556.
1g
torch
cnn ImageNet-1k
557.
side_resnet1011
torch
can ImageNet-Ik
558.
wide_resnet5002
torch
can ImageNet-
559.
20r0000_2_aammm__ in
timm
ImageNet-Ik
560.
xception41..f__mg
timm
ImageNet-Ik
561.
Iception41p.ra3__m __iik
timm
cnn ImageNet-1k
562.
xception655raa 3rajinnn ra3__nik
timm
cnn ImageNet-1k
563.
xception65.tf inlk
timm
can
ImageNet-Ik
S64.
xception66p.raa3 inlk
timm
ImageNet-Ik
565.
xception71.1ff inlk
timm
can
ImageNet-Ik
566.
caformer_b3..sil_
timm
hybrid
ImageNet-lk
567.
timm
hybrid
ImageNet-Ik
568.
100ommer_b36.sii_n22222
fi_kkkl___kkkkk
timm
hybrid
ImageNet-21k
569.
naformer_b36.sa Bal_sail_nn2kk___ in22k folkkfft_aakp ______ik_____nk_______
384
timm
hybrid
ImageNet-21k
570.
caformer_m336... inlk
timm
hybrid
ImageNet-Ik
571.
caformer_m336.st in1k_384
timm
hybrid
ImageNet-1k
572.
caformer_m366 1ac1:ossii__in2k_____ ___________________ ft_inkk
timm
hybrid
ImageNet-21K
573.
caformer_m36.sai _al____n22________ inlk_384
timm
hybrid
ImageNet-21k
574.
caformer_s18.sai inlk
hybrid
ImageNet-Ik
575.
caformer_s18.sai__mg 1i1_k_3ai_iikk__44444444
timm
hybrid
ImageNet-1k
576.
cafommer_118.sailin222.
timm
hybrid
ImageNet-21k
577.
cafommer_s18.sail_n22222 ____22k____iin_33__
timm
hybrid
ImageNet-21k
578.
eaformer_s36.san inlk
timm
hybrid
ImageNet-Ik
579.
eaformer_s36..s. inlk 384
hybrid
ImageNet-Ik
580.
aaformer_s36.sse in22k inlk
timm
hybrid
ImageNet-21k
581.
caformer_s366
in22k ft inlk_n3
timm
hybrid
ImageNet-21K
582.
coatnet_0_rw__000 st__r__222ss_______
22_.sk_
timm
hybrid
ImageNet-Ik
583.
coatnet_l_rw_2444
hybrid
ImageNet-Ik
584.
coatnet_bn_0_nt SDw__224.sw_ii
timm
hybrid
ImageNet-1k
585.
coatnet_nano__
2e_.-2222__wmme_ w_inlk
timm
hybrid
ImageNet-1k
586.
1aatee__mmpp_____220 2e__22222__sml__
timm
hybrid
ImageNet-1k
587.
coatnet_mmp______ _2_.222222nn sw_inlk
hybrid
ImageNet-1k
588.
goatnet_mmlp nano IW 224.sw inlk
hybrid
ImageNet-Ik
589.
coatnext nano rw 224.sw inlk
timm
hybrid
ImageNet-1k
$90.
edgenext base.in211 1k ft inlk
timm
hybrid
ImageNet-21K
591.
jisennet_baac.uui_ie
timm
hybrid
ImageNet-1k
592.
edgenext_small.us. lnk_
hybrid
ImageNet-Ik
593.
perenext__mmal_pgg haclll_w.www_ing
timm
hybrid
ImageNet-Ik
594.
edgenext_x__ i_
timm
hybrid
ImageNet-1k
595.
edgenext xx_ sx_small.inkk
timm
hybrid
ImageNet-1k
596.
elficientformer 11. snapsmp_ddaa dist in
timm
hybrid
ImageNet-Ik
597.
efficienttfrmmrr_] 13. snnapppppp_ppppp_...ppp dist_inIl in
hybrid
ImageNet-Ik
598.
efficieentfrrmrr
snap www.smapdddd. dist inlk
timm
hybrid
ImageNet-
599.
efticientforrery2__1.sss -nnap_______p_s_ssss_____d_____s dist inlk
timm
hybrid
ImageNet-
600.
inlk
timm
hybrid
ImageNet-1k
601.
efficientformerr2, 2es___ssss____iiie inlt_
timm
hybrid
ImageNet-1k
602.
efficientformerv22 sewpenerrrrsssapa____e_e s22.ssaa_dist_init
timm
hybrid
ImageNet-1k
603.
levit 128.fb dist inlk
timm
hybrid
ImageNet-Ik
604.
levit fb dist inlk
timm
hybrid
ImageNet-
605.
levit 192.fb dist inlk
timm
hybrid
ImageNet-1k
Table continues onto next page
39

---

Model summaries (continued from previous page)
No
Model ID)
Source
Arch
Dataset
606.
levit_226... 226.fb__6iii diss_inlk
timm
hybrid
ImageNet-Ile
607.
levit 384.fb dist inlk
timm
hybrid
ImageNet-Ii
608.
levit conv 128.fb dist inlk
tim
hybrid
ImageNet-Ik
609.
12 conv_128s.fb levit conv
dist inlk
hybrid
ImageNet-Ik
610.
levit_conv_192.11 1192.fb_dist_iim
hybrid
ImageNet-Ik
611.
___iit__inl_
timm
hybrid
ImagoNet-Ik
612.
levit_conv_36e 3o0.fw_.4.fb...... do__ist_inlk
timm
hybrid
ImageNet-1k
613.
maxvit base if If_222kk
timm
hybrid
ImageNet-Ik
614.
maxvit base if_384.inlk
timm
hybrid
ImageNet-Ik
615.
maxvit base 284.in21k_fttimg
timm
hybrid
ImageNet-21k
616.
maxvit base _f_____.inikk
timm
hybrid
ImageNet-1k
617.
base 1f__512.in2kkkffmmm
timm
hybrid
ImageNet-21K
618.
maxvit_ Jarge_rrr_____ tf 224.inlk
timm
hybrid
ImageNet-1k
619.
maxvit_large rf_384.inlk
hybrid
ImageNet-Ik
620.
maxvit_large_It htt_______.nnnnnf_ee__m
timm
hybrid
ImageNet-21k
621.
maxvit_large___ k jarge_tf_512.nnlk
timm
hybrid
ImageNet-1k
622.
maxvit large If 512.n21k ft inlk
timm
hybrid
ImageNet-21K
623.
maxvit nano 256.sw inlk
timm
hybrid
ImageNet-Ik
624.
maxvit ralp nano rw s5 inlk
hybrid
ImageNet-Ik
625.
navvt__mlp__pco_rw__2.. 2__256.sw_inlk
timm
hybrid
ImageNet-lk
626.
maxvi__mlp_small Bacpaal____2200 rw_224.sv 222.sw_inll w_inlk
timm
hybrid
ImageNet-1k
627.
maxvit_rmlp_tiny s___56.sw.a... sw_inkk
timm
hybrid
ImageNet-Ik
628.
maxvit_ 224.inkk
timm
hybrid
ImageNet-Ik
629.
maxvit_smale
33..inkkkkk
timm
hybrid
ImageNet-Ik
630.
maxvit_small |f_ 512.inkk
timm
hybrid
ImageNet-Ik
631.
maxvit_ _i___ttn____________ Biny_w______ www_224.sww_rr sw_inkk
timm
hybrid
ImageNet-Ik
632.
maxvit tiny_ntyyy___ 224.inlk
timm
hybrid
ImageNet-Ik
633.
maxvit_timg f8_.8kknkk
hybrid
ImageNet-Ik
634.
maxvit_tiny 1inny__f____ 512.inkk
timm
hybrid
ImageNet-Ik
635.
maxvit_xlarge__f__B. 3n2.44.421k_ff__m_
timm
hybrid
ImageNet-21k
636.
sawvi_xtarge_f__522.n2221 ffnn222fft_inkk inkk
timm
hybrid
ImageNet-21k
637.
maxxvit rmlp nano 256.sw inlk
timm
hybrid
ImageNet-Ik
638.
maxxvit rmlp small rw 256.sw inlk
timm
hybrid
ImageNet-Ik
639.
maxxvitv2 nano rw 256.sw inlk
timm
hybrid
ImageNet-Ik
640.
mobilevit-s.ccve s.cvnets inlk
timm
hybrid
ImageNet-Ik
641.
mobilevit_xs.co
inlk
timm
hybrid
ImageNet-1k
642.
mobilevit_xxs. evnetssssssssss.....eeeesssseess jnlk
hybrid
ImageNet-Ik
643.
wst.tveessssssstsssssooooosmm j_ikk
timm
hybrid
ImageNet-1k
644.
mobilevity2_00e 20w.:2200775sc55s.5 cvnetssinnkkk
timm
hybrid
ImageNet-1k
645.
mobilevitv2_1000ccvees___. jnlk
timm
hybrid
ImageNet-1k
646.
mobilevinv2_112.eeenesss 1et___12eeeeessinii...m
timm
hybrid
ImageNet-1k
647.
mobilevitv2 150.evnets inlk
hybrid
ImageNet-Ik
648.
mobilevitv2 150.cvnets. in22k ft inlk
timm
hybrid
ImageNet-21k
649.
mubDlevvity__S0.cceeeess in22k ft inlk 1ilk... 384
150.cvnets.
timm
hybrid
ImageNet-21k
650.
mubilevitv2__175.cveett..
timm
hybrid
ImageNet-1k
651.
mobilevitv2_175.cvnets. 2rtSSccnets_in22__ftt... in2ik_fttinkkk
hybrid
ImageNet-21k
652.
timm
hybrid
ImageNet-21k
653.
mobilevitv2 200.cvnetsssss cink
timm
hybrid
ImageNet-Ik
654.
200.cvnets. mubillvvit222000cvmeess
in22k 300.cvnets_in22k
fn_ikk
timm
hybrid
ImageNet-21k
655.
mobilevitv2 2tttlevvii2200..eeeettstssssss 200.evnets in22k ft, inlk_nnk
timm
hybrid
ImageNet-21k
656.
WVL_v__00.nI..
hybrid
ImageNet-Ik
657.
pvt_v2_br inlk
timm
hybrid
ImageNet-
658.
pvt_v2_ WwL_v_bb2.nlk
timm
hybrid
ImageNet-
659.
pvt_v2_b2 li.inlk
timm
hybrid
ImageNet-1k
660.
pvt_v2_ks ______b3.inkk
timm
hybrid
ImageNet-1k
661.
pvt_v2_b pvt_v2_b4.in1k
timm
hybrid
ImageNet-1k
662.
pvt v2. b5.inlk
timm
hybrid
ImageNet-1k
663.
gmixer 24 224.ra3 inlk
timm
mlp
ImageNet-
664.
gmlp s16 224.ra3 inlk
timm
mlp
ImageNet-Ik
Table continues onto next page
40

---

Model summaries (continued from previous page)
No
Model ID)
Source
Arch
Dataset
665.
mixer bel__24._e___ 224.goog. goog_in2ik. f__t____t inlk
timm mlp
ImageNet-211
666.
mixer b16 22i.miil in2 k inlk
timm
mlp
ImageNet-21K
667.
mixer 116 (166_224.gooo 224.goog in21k Ik ft inlk
tim
mlp
ImageNet-21K
668.
poolformer m36.sail inlk
mip
ImageNet-Ik
669.
poolformer_m48..cm _animmmer__m..al_______ _aik_
mlp
ImageNet-Ik
670.
poolformer_s12.sul
timm
mlp
ImageNet-Ik
671.
poofforee__s24.sil_.in
timm
mip
ImageNet-1k
672.
poolfomer-_366.... inlk
timm
mlp
ImageNet-IK
673.
poolfomeerv2_m366 mmn.sssillmmmm_l.m.__..m.m. inlk
timm
mlp
ImageNet-Ik
674.
poolformerv2_mmg 1112__44_ssa
timm
mlp
ImageNet-Ik
675.
poolfomerr2_si22.stii inlk
timm
mlp
ImageNet-Ik
676.
2olformerv2_s24.00m innk
timm
mlp
ImageNet-IK
677.
poolformerv2 1aaloomerv2_s33.salll _nnk
inlk
timm
mip
ImageNet-Ik
678.
resmip_ 10_11222444 20..224.fbb_b cil.ffdddttilled_inn
mip
ImageNet-Ik
679.
resmlp_ 12__24.fbb______ ffn_kkf
timm
mip
ImageNet-Ik
680.
resmlp_24. 114_224.fb__ 22244ff_distilled_nt
timm
mip
ImageNet-1k
681.
resmlp 24. 224.fb inlk
timm
mlp
ImageNet-Ik
682.
resmip 36 224.fb distilled inlk
timm
mip
ImageNet-Ik
683.
resmlp 224.fb
mlp
ImageNet-Ik
684.
resmlp_bigg hipp_bi__244 2e__224_22.ff...
1144.f_disttillcd_i11
timm
mlp
ImageNet-lk
685.
resmlp_by waib_b__________1 24_224.fb_inlk
timm
mlp
ImageNet-Ik
686.
resmlp_ 2amnll_big_22__2444_ 22_.22.2222.._22__. 2ttp:___222.____22_________
timm
mip
ImageNet-21k
687.
beit base_patch16 22f6__2222222,
in22k in1k
timm
vit
ImageNet-21k
688.
beit base
3824.n222k ft. in22k inlk
timm
ImageNet-21k
689.
beit laii_larr__pacch66 (ttpaachh6_222..222222 224.in22k ft ____n22k_inll
timm
vit ImageNet-21k
690.
beit__argee Jarge_attchh66 Baauchhl6334.in222 384.in22k 11122/ff_iiiii__ ft_in22k_nnl
timm
vit ImageNet-21k
691.
beit large_patchl6. 512.in22k ft ft_in22____ inlk
timm
vit ImageNet-21k
692.
beitv2 base batchp 224.inlk It inlk
ImageNet-Ik
693.
beitv2 base hawn2__bse_pptcch6_6. 22_.22222_2__________ _fft_in222_inkk
timm
ImageNet-21k
694.
beitv__larg_ppacchh_2222 2ttp1/_2222.inn_fiiill
timm
vit
ImageNet-lk
695.
beitv2_large Las__larre______
timm
vit ImageNet-21k
696.
cait m36 384.Ib dist inlk
timm
vit ImageNet-Ik
697.
cait
448.fb dist inlk
ImageNet-Ik
698.
cait s24 224.fb dist inlk
timm
vit
ImageNet-1k
699.
cait s24 224_384.f0 384.fb dist inkk
timm
vii
ImageNet-1k
700.
cait_s36. hitt_s36_34.fff 384.fb fec.ff_dis_iitiie dist_inkk
dmm
vit ImageNet-Ik
701.
xs22_xs24 384.fb fil_ff_diss dist inlk
ImageNet-Ik
702.
xxi2_xxs24
22..22222200 dist inlk
timm vit
ImageNet-1k
703.
cait xiit_xxs2____ _0_33.44___ii_ _ikk
timm vit ImageNet-1k
704.
cait_ xuit_xx336__
224.f22222022 224.fb_distjinll
timm vit ImageNet-1k
705.
cait_ xxs_x_xx34_......
inlk
vit ImageNet-
706.
coat medium.inll
ImageNet-Ik
707.
coat lite medium 384.inlk
vit
ImageNet-1k
708.
coat lite mini.nnk
vit
ImageNet-Ik
709.
coat_lite_att coat_lite_smallinnk
vit
ImageNet-1k
710.
coat_lite_tioe coat_lite_tiny.inkk
ImageNet-Ik
711.
coat coat_mine poat_mini..nl in
vit
ImageNet-1k
712.
coat smal_mmll.
timm vit ImageNet-1k
713.
coat tiny.inkk
timm vit ImageNet-1k
714.
convit_ base.ft inlk
vit ImageNet-1k
715.
convit_ convit_small.fb jn_i
ImageNet-Ik
716.
convit tiny.fb in
timm vit ImageNet-
717.
crossvit_1s 115_240.inlk
timm vit ImageNet-
718.
crossvit 15 dagger_240.in
timm vit ImageNet-1k
719.
crossvit_ dagger_408.inlt
vit ImageNet-
720.
crossvit ink
timm
ImageNet-1k
721.
crossvit 18 dagger 240.
timm
vit ImageNet-1k
722.
crossvit 18 dagger 408.inlk
timm
vit ImageNet-
723.
crossvit 240.inlk
timm vit
ImageNet-Ik
Table continues onto next page
41

---

Model summaries (continued from previous page)
Model ID)
Source
Arch
Dataset
724.
crossvit 9_ dagger flagger240..nnl 240.inlk
timm vit
ImageNet-1k
725.
crossvit base 240.inlk
timm
ImageNet-Ik
726.
crossvit small 240.inlk
tmm
vii
ImageNet-Ik
727.
crossvit tiny. 240.inlk
vit
ImageNet-1k
728.
Devit_base.msft_.. _nkk
ImageNet-Ik
729.
davit povit_small.sstt_in
timm
vit
ImagoNet-Ik
730.
davit davit_tiny.msff_in
timm
vit
ImageNet-1k
731.
deit3 base_
pase_paich16 224.fb__. _b_inlk
timm
vii
ImageNet-1k
732.
deit3 _as__bae_______________ base_paich16 224.fb in22k foikk_t_____ inlk
timm
vit
ImageNet-21k
733.
deit3 base_patchh6....mm 384.fb inlk
timm
ImageNet-Ik
734.
deit3 base_
case__atcchl__3101 384.fb__B.___ f____3____in22__ff________ it
timm
vit
ImageNet-21k
735.
deit3 wwge__atccl4_22444
timm
vit
ImageNet-Ik
736.
deit3
Weir3_huge__pate
auge_patchi4 224.ff
in22k fnlk
Gim
vit
ImageNet-21k
737.
deit3
vit
ImageNet-Ik
738.
deit3
Lutt_aaare__atchl__22 2aape16__22..hh_222e1_22
timm
ImageNet-21k
739.
deit3
Large_patchl6_338.ff fl
timm
vit
ImageNet-1k
740.
deit3 suge_patchl66 384.fb in22k ft inlk
timm
vii
ImageNet-21K
741.
deit3 16tchl6 224.fb inlk
timm
vit
ImageNet-Ik
742.
deit3 medium p 224.fb in22k inlk
ImageNet-21K
743.
deirs_small_patchl6_2222 ft..4.ttchll_222.fbbbbbb __ink
timm
ImageNet-lk
744.
deit3 deit__smallpaatch66_22 f24.fb ft____224f___222________ _in22k_ft_inlk
timm
vit
ImageNet-21k
745.
deit3 deirs_small_patch1__38844 fb_inkk
timm
vit
ImageNet-Ik
746.
deit3_ iis__smlll_pateee _run__smal__pachhl____ f__4.ffbb6___bbb_b.. in22k inlk
timm
vit ImageNet-21k
747.
base base_distilled Destclled_patchh6. 224.fb inlk
timm
ImageNet-Ik
748.
deit_ base bhase_dsttilled_ppl
f_8.f_ _nlk
timm
vit
ImageNet-1k
749.
deit_base patchll__22.fbbb___e_ep 224.fb_inlk
timm
vit
ImageNet-1K
750.
deit_base base bat__patcchhhl..__c wttp_al__chcaallla__b_a inlk
tmm
vit
ImageNet-Ik
751.
deit small distilled patch16 224.fb_inkk
ImageNet-Ik
752.
deit ceit_small_patchhe 224.fb__inn fb_inlk
timm
ImageNet-Ik
753.
deir_tiny_diitillcd__aaa httpppaccccc2222__ihimaaaaa
timm
vit
ImageNet-lk
754.
deit hei__tnn__pacchh6_22._.___1
timm
vit
ImageNet-1k
755.
eva02 base baise_patchl___.1 448.mim
saicch1_444..mmm...m in22k inlk
timm
vit
ImageNet-21k
756.
eva02 base_ base_patch14 448.mim in22k in22k inlk
timm
ImageNet-211
757.
eva02 large large_patchl4 448.mim in22k ft inlk
timm
vit
ImageNet-21k
758.
eva02 large_ patrse__pachll___1 1aicch444444 448.mim in22k ft [__n22kkinll inlk
timm
vii
ImageNet-21K
759.
aeo2__aarge_patch4 4444..4..m_m.... ftttmmmm8m__f___m_ ft_ikf_kkkkkkkk
dmm
vit Merged
760.
eva02_large__atchll
mwt.ahhh11444.mmmmmmm.... m00mmm3mmmm_fmm ftw.fffti2222111 mn22k_inlk
Merged
761.
eva02
htt_smmlll_path11_3366666 in2ii___ii22____1eee inik
timm
vit ImageNet-21k
762.
eva02_ it___inn_________ Siim___aach14_336.mmm
in22k_ft_innli
timm
vit
ImageNet-21k
763.
eva
timm
vit
LAION-400M
764.
ww__giant_pachh4433366
ft_inkkkkk
timm
vit
LA1ON-400M
765.
eva giant patch14 336.m30m in22k inlk
Merged
766.
eva giant patch14 maich14_560.330m 5600m330 in22k inlk
timm
vit
Merged
767.
eva large patch14 1atch14_19..i2222 196.in22k inlk
timm
vit ImageNet-21K
vit
768.
wwwlargee_aachl411996 1wt.hh11191111________ 11tp:iin222____in222____222_
ft_in22k_innk
timm
ImageNet-21k
769.
vaL_arge__aachh443366
ImageNet-21k
770.
eva eva_large_patche wappacch1__3555 Baaphchh4_36.in2222
21166in222_ff_222222221 ft_in22k_inll
timm
ImageNet-21k
771.
flexivit base. 1200ep _nnk
timm
vit
ImageNet-lk
772.
_exivit_base_30000 base.300ep_img inlk
timm
vit ImageNet-1k
773.
flexivit_base b00ep. inlk
timm vit ImageNet-1k
774.
ImageNet-Ik
775.
Soxivit_large.30000 _nlk
timm vit
ImageNet-
776.
Dexivit_large.60000 Lackep_nnkkk
timm vit ImageNet-
777.
flexivit_small, 1200pp_ in
timm vit ImageNet-1k
778.
flexivit_small.300
timm
vit ImageNet-
779.
flexivit_small
sativt_small,600ep_.i
timm
ImageNet-1k
780.
gevit base.inlk
timm
vit ImageNet-1k
781.
gevit small.ink
timm vit ImageNet-
782.
gevit. tiny.inkk
timm vit
ImageNet-Ik
Table continues onto next page
42

---

Model summaries (continued from previous page)
No
Model ID
Source
Arch
Dataset
783.
gevit_ Xttn_xxtiniinkkkkk
timm vit
ImageNet-Ik
784.
gevit xxtiny.nnkk
timm
ImageNet-lk
785.
mvitv2 base.fb inlk
vii
ImageNet-Ik
786.
mvity2 large.fb inlk
vit
ImageNet-Ik
787.
mvity2_smalll
ImageNet-Ik
788.
wwity2_tiny.fb..
timm
vit
ImageNet-1k
789.
nest_base baase__i_ooog_inl
timm
vit
ImageNet-1k
790.
nest rest_small
_xs__smalljxxgoogg.u sogoooinnllo inlk
timm
vii
ImageNet-1k
791.
nest Lin_tiny_uu jot__iiy_jx.goog... jx.goog_goog inlk
timm
vit
ImageNet-1k
792.
pit_b 224.in
timm
ImageNet-Ik
793.
pit_b_ distilledd224.inlk
224.inlk
timm
vit
ImageNet-Ik
794.
mi_s_222.inlk
timm
vit
ImageNet-
795.
d_ditilldlleddddeeeeeeee 224.inlk
timm
vit
ImageNet-Ik
796.
pit_ f1__224.nlkk
vit
ImageNet-Ik
797.
timm
ImageNet-Ik
798.
pit_xs_ wi_xs_ 2s__22,,nkkk
timm
vit
ImageNet-1k
799.
pit xs distilled 224.inlk
timm
vii ImageNet-Ik
800.
swin
torch
vit ImageNet-Ik
801. swin base patch4 window12 3s4.ms inlk
timm
ImageNet-Ik
802. wwwp_uaas__pach__iinddol.......
timm
ImageNet-21k
803. swin_base_pateeee
timm
vit
ImageNet-Ik
804. swin Lrrp_aasc_patch__wiidee 2017___22.mm_...
______2_222222______22__________ j___j_n22_____ ft_ikkk
timm
vit
ImageNet-21k
805. swin lain__larg__pach_____ window ms4.ms..mm..mmm in22k ft inlk
timm
vit ImageNet-21k
806. swin_large paci__pacch4_4_ patthh4_wnndww7 Berddow7_222.mmm swin_s
window7
224.ms in22k inlk
timm
ImageNet-21K
807.
torch
vit ImageNet-1k
808. swin_s3 base base_224.ms _nlk
timm
vit ImageNet-1k
809. swin_s3 _0a__s3__all___________ 224 224.ms.ae inlk
timm
vit
ImageNet-Ik
810. wwi__s3_tiny
Se__iiny_22..ms 224.ms inlk
timm
ImageNet-Ik
811. wwin_small_patehh #eratchh_windww7__1
handow7_224.ms memmssnnk in_inkk
timm
ImageNet-Ik
812.
wws_in2kkfttimmmmmmmm in
timm
vit
ImageNet-21K
813. swin_t
torch
vit
ImageNet-1k
814.
swin tiny patch4 7 224.ms inlk
timm
vit
ImageNet-Ik
815.
swin tiny patch4 window7 224.ms in22k inlk
timm
ImageNet-21K
816.
swin v2.b
torch
vit
ImageNet-1k
817.
swin_v2 v
torch
vii ImageNet-1k
818.
swin_v2. v2_t
torch
vit ImageNet-1k
819.
swinv2 base mase_window// 16o 2tteel22111111126l1oeoeo in22k inlk
ImageNet-21k
820.
swinv2 base wawe_windowww/e22eaeaeaaa 1eeo024__99to38.mmmmm_
ft inlk
timm
vit ImageNet-21k
821.
Saanw__ase____ mase_windowwle 16 m56_mmmmmmmmmmmmm _nn_
vit
ImageNet-1k
822.
Sainv__hase_windooee 2tt__wiidow__2256mmm.mm.m 256.ms_in1e
vit
ImageNet-1k
823.
swiny2_land lr2__llrg___iddwww 1to_ 12to16__11_ 2 192to256.ms_nn
mino266.sm_n22k... i___k_222_2_ ft_ink
vit
ImageNet-21k
824.
swinv2 large window112t22 192to384.ms in22k inlk
ImageNet-21k
825.
swinv2 small window 256.ms inlk
vit
ImageNet-lk
826.
swinv2 small window8 256.ms inlk
vit
ImageNet-1k
827.
wwinv2_tiny_windoww 2__2666_m__mnnmm6m6e11
vit
ImageNet-1k
828.
wwinv2_tiny_windon
ImageNet-Ik
829.
twins ciinsspppvt_basc.ngg
timm
vit
ImageNet-1k
830.
twins_pepvt caans_pepvt__arree
sepyt_largeeinl1
timm vit ImageNet-1k
831.
twins_ twins_pepvt_______ plppppppppppppeeeeeeeeeeeeepeeee |
timm vit ImageNet-1k
832.
twins_ stt_sstt_aaaass vwt_base.in lk
timm vit ImageNet-Ik
833.
twins_svt_lare hwwpssst_aaargnnnll
ImageNet-Ik
834.
twins svt swallmall.nnk
timm vit ImageNet-Il
835.
vit b6
torch
vit ImageNet-
836.
vit_b_ 16___wa
torch
vit Instagram
837.
vit_b_32
torch
vit ImageNet-1k
838.
vit vit_base_mn Shass__atchh16 www.aagggg_nnll111 (_____nn2kkk__ll__
timm
ImageNet-21k
839.
vit base patch16 224.augreg inlk
timm
vit ImageNet-1k
840.
vit base, 1 224 224.augreg in21k ft inlk
timm
vit ImageNet-211
841.
vit base potch16 224.orig in21k ft inlk
timm vit
ImageNet-211
Table continues onto next page
43

---

Model summaries (continued from previous page)
No
Model ID
Source
Arch
Dataset
842.
vit base base_patch16. 224.sam 224.sam_inlk sam_inkk
timm
ImageNet-1k
843.
vit base base_patchll patch16 224. miil.in21k inlk
timm
vit
ImageNet-211
844. vit base baise_paccll6._.. 384.augreg inlk
tim
vii
ImageNet-1K
846. vit faise_patchl6..co 384 384.augreg in21k inlk
845. vit base
Gmm
vii
ImageNet-21I
ImageNet-21K
847. vit
ft_in12k_ink
timm
vit
LAION-2B
848. vit vit_base_pacchl6_cclp22222
timm
vit
LAION-2B
849. vit base_patch16 2iiiip222..peaaii [t_in|2k_inle
timm
vii
WIT
850. vit base_patch16......m. clip 224.openai ft inlk
timm
vit
WIT
851. vit base_patchh66. 1111666_______e11ee 384.laion2b. in12k_inll
timm
LAION-2B
852. vit base base_patchll6clij 384.laion2b inlk
timm
vit
LAION-28
853.
vit www.baasscclh__oeeee httpchll___clpp_88.peeeeaaaaaal 104..opeaiift_in12222. ft_jnn2k_innl
timm
vii
WIT
854.
vit base_patchhl....... 3ip.pppe.pnpni______ letoopeniift_inkk
timm
vit
WIT
855.
vit m__base__pacch6.__
2________n_22_________
222222_22illl inlk
timm
vit
ImageNet-1k
856.
vit ____bs___accii2_222aarrge___
timm
ImageNet-1k
857. vit
224.angreg_in221
timm
vit
ImageNet-21k
858. vit base paccheppac3322211111111 224.sam inlk
timm
vii
ImageNet-1K
859. vit base bacsee__ach32____11 384.augreg inlk
timm
vit
ImageNet-1k
860. vit base pacpllpaaahh22211 384.augreg in21k inlk
timm
ImageNet-21k
861.
vit ait_base_pateie www.bass__acchh__cippppp httpe_aaichhs_llpo2222aaiiobbbb 222.laion2b_ttiin22222
timm
vit
LAION-2B
862.
vit mt__ase_patchh__clle htt.:paalcciit_llpl22laaaaaabbb
timm
vit
LAION-2B
863. vit mat__aas_paahhh__ce [ttipp224.opcnaa_ft_iiee.
timm
vit
WIT
864. vit_ bas__pattcc_________ 3itiip334.laiiobbbb... ft in1k inlk
timm
vit
LAION-2B
865. vit base caciss__pahhh2____ cclip_344.opeaa
inlk inlk
timm
WIT
866. vit base waws__patccs2_cliiiieee __tpp_c___.___i1110 laino.....onbbbb.... ft__fffii______11 _nnk
timm
vit
LAION-2B
867. vit base _as__paacsh____________a 22g_a__r2222_22222..
j___nn21k1 ft_ikkk
timm
vit
ImageNet-21k
868. vit brase_patch$ 224.augreg
in21k inlk
timm
vit
ImageNet-211
869. vit Slaas__50__sse___ 1150_s16_384.oriigg
in21k inlk
timm
ImageNet-21K
870. vit vit_h_14_swag
torch
Instagram
871. vit Ht__huge_patch44_clig 2a1p1444__lpp222__aa_e 2aw.aiionn....iiin..b__. ftaoh__ff_nkk__iit1
timm
vit
LATON-2B
872. vit www.hhhatchhh__eee1 22..laion2b_ft.cmmm ff_____kkkkk__
timm
vit
LAION-2B
873. vit huge patch14 clip 336.laion2b in inlk
timm
vii
LAION-2B
874. vit
ImageNet-Ik
875. vit 16 swag __a_swag
torch
viu
Instagram
876. vit 1.32
torch
vii
ImageNet-Ik
877. vit Jarge_patchh4, Saa___llp__2____
2e..22i..22___....___ fttrb__tt_nllkkk____ innkk
timm
vit
LAION-2B
878. vit
l24.laioo2b... ft_ikkkkkkkkk
LAION-2B
879. vit www_aa___aacch__chll Baapcchl_clii_224oppeeee.
timm
vit
WIT
880.
vit lww_larrepacchh______
inlk
timm
vit
WIT
881. vit htnpge__pacc14__ccc_33...iiii bhiionnhff_inlkk_inn innkkinnk
timm
vit
LAION-2B
882. vit ii__large_patch44 aai.app...aiioohh.......aa.m.aa. fatlonnh_f__inl in_k
vit
LAION-2B
883. vit large facpee_pacchl_..1 _111___pp___p1p__1 33e.oienai in12k
WIT
884. vit large faclle_paachll.ee 224.augreg in21k ft inlk
timm
vit
ImageNet-21k
885.
vit large patch16 384.augreg in21k ft inlk
timm
vit
ImageNet-21k
886.
vit
in
timm
vit ImageNet-21k
887.
vit vit_large_r50_ss www.__22..ugregaaae_
in2k__n22k_____ ft_jntt_ j_inkk
ImageNet-21k
888. vit vit_largeer50_$$2 3s2__34.augrrg__1
ft
timm
vit
ImageNet-21k
889.
vit relpos bas__baa__________ baasepptchh16..1111 224.sw 22cdhl1._2.wwwwwoooo
inlk
timm
vit ImageNet-lk
890.
vit relpos_ Saappos_bae__peee bosseppacche6__.111pp 21166__ls________
224.sw 224.sp_224.wwwoo.oo
inlk
timm
vit ImageNet-1k
891. vit relpos base pptchatcccpppp1 plus rpn, 256.sw
inlk inlk
timm vit ImageNet-Ik
892. vit melrroos_meddumm pptpplaachhp.pp
sw..2224..wwj.. st__si_lll inlk
ImageNet-Ik
893. vit celpos_mediiee hacpiiiuuupptahhh-ph
224.sw inlk
timm vit
ImageNet-
894. vit_res calipoo_mmdium__ppe_ paatch16_rpp...
timm vit ImageNet-1k
895.
vit_ 6atllpos_ssalll_phhh111_111 2e___2222wwwm
timm vit ImageNet-1k
896.
vit mttsmall_ 1 2etpl622222aaaa_____1 inlk
timm
vit ImageNet-1k
897.
vit_small patpllpaaph111111 221662222mmmmmm 2aa_hh11_22____reeee_ Baaree__inn______e
timm
ImageNet-21k
898.
vit small patch16 384.augreg inlk
timm
vit ImageNet-Ik
899.
vit small patch16 384.augreg in21k ft inlk
timm
vit ImageNet-211
900.
vit small patch32 224. 224.augreg in21k inlk
timm vit
ImageNet-211
Table continues onto next page
44

---

Model summaries (continued from previous page)
No
Model ID
Source
Arch
Dataset
901. vit_small patch32 patcha3__34.aauuuupp 384.augreg_in22 ja____n2k_____ @o21k__tt [t_inkk
timm vit
ImageNet-211
902. vit small r26 s32 augree in21k ft inlk
timm
ImageNet-21K
903. vit small r26 s32 3ug.eg in21k inlik
tim
vii
ImageNet-21K
904. vit srelpos medium patch16 224.sw inlk
tim
vit
ImageNet-Ik
905. vit_ mit_steepos_smaa patch saatch16_224.ce 2___22.ss__imlm
ImageNet-Ik
906. vit jir_tiny___e watpathh6__22.aapppppp
in21k_ft it_inl
timm
vit
ImageNet-21k
907. vit vit_tiny_patchh1 www.hhh.e1__..8....ag___ inngreg_i22lk___iim
timm
vit
ImageNet-21K
908. vit tiny_tiy_____ r_wp____166___ p8 228_222uuggrrgg... in2 ft inlk
timm
vii
ImageNet-21K
909. vit iit_tiny___st 10116__8______1 384.augreg in2 1k inlk
timm
vit ImageNet-21k
910. volo dl _01__222.saii inlk
timm
ImageNet-Ik
911
volo dl _d____844.ssll inlk
timm
vit
ImageNet-1k
912.
volo_d2_221 sio__d_224.sall___ saik
timm
vit
ImageNet-1k
913.
volo 32_.sll.ail. inlk
6im
vit
ImageNet-Ik
914.
volo d3 2a__d3_22,44 224.,22ll_inl
vit ImageNet-Ik
915.
volo_d3_443,
sail_inkk
timm
ImageNet-Ik
916.
volo_d4_224. in1k
timm
vit
ImageNet-1k
917. volo d4 448.sail inlk
timm
vii
ImageNet-Ik
918.
volo d5 224.sail inlk
tim
vit
ImageNet-Ik
919.
volo 448.sail inlk
ImageNet-Ik
920. 1o__dd__11.sail_nnk
timm
ImageNet-lk
921.
xeit_large_24_21
timm
vit
ImageNet-1k
922.
timm
vit
ImageNet-Ik
923. xcit large 24_p16. fB4..8... (ibb__oh_ disk_inkk inlk
timm
vit
ImageNet-Ik
924. xcit large 24_p8 224.fb dist inlk
ImageNet-Ik
925. xcit large__2_ 24_p8___pp__________ 224.fb inlk
timm
vit ImageNet-lk
926. xcit Jarge 24_ 24___________________________2 _________fb_ 384.fb distditt____ inlk
timm
vit
ImageNet-IK
927. xcit mediume 24_pp 24_p16 224.fb inlk
6mm
vit ImageNet-Ik
928. xeit medium 24_p16 224.fb inlk
ImageNet-Ik
929. xcit xcit_meee medium 24_p16_384.fb _34..8.f.b... feffb_dist____ dist_inkk
timm
ImageNet-Ik
930. xcit_medium 24_p8__ 24_p8_224.10 224.fb_dde ff__dist_nnk
timm
vit
ImageNet-lk
931. xcit_moe icit_medium_ @ecdium_24_pss _4_p8__ 24_p8_224.fb
timm
vit
ImageNet-1k
932. xcit medium 24 p8 384.fb dist inlk
Gimm
vit
ImageNet-Ik
933. xcit nano 12 p16 224.fb dist inlk
ImageNet-Ik
934. xcit nano ______2__ppp _____1______________________1___ 224.fb inlk
timm
viu
ImageNet-1k
935. xcit nano 12_ _2_p16__________ [2_p16_384.fb _344.fb
dist inlk
timm
vii
ImageNet-Ik
936. cct__nano_1__p 224.fb bo_ddst__nnll
dmm
vit ImageNet-Ik
937.
xcit_nano_
_2_____________________pp_____ 224.fb inlk
ImageNet-Ik
938.
xcit_nano_112__ 384.fb dist_iill inlk
timm vit
ImageNet-1k
939.
2app___2...________1 f24.fb distddi_________________ dilk__nnkk
timm vit ImageNet-1k
940. cci_smmll_12_p16 f24.fb inlk
timm vit ImageNet-1k
941. xcit_small _2________6________ 1/2_p16_384.ft
33..ff fal_f_ddiis_ dist inlk
vit ImageNet-
942. xcit small p8 224.1b dist inlk
ImageNet-Ik
943. xcit small 12 p8 224.1b inlk
timm vit
ImageNet-1k
944. xcit small 12 p8 384.Ib dist inlk
vit
ImageNet-Ik
945.
soit_small_24_pp 2f66_224.bbjmm
222.fb_22kii sil_ffdddit_iillle inlk
vit
ImageNet-1k
946.
xcit_small 2ei_smmlll22__p66_ 24_p16_220
ImageNet-Ik
947.
xcit_small _4_p16_____ $4_pp6__34.f1 184.fb, dilt_inkk
vit
ImageNet-Ik
948. xcit small 24 24_p8 224.fb dist inlk
timm vit ImageNet-1k
949.
xcit small 24 24_p8 224.f2 inlk
timm vit ImageNet-1k
950.
xcit small 24 24_p8_ _pp__88.11____ dist inlk
vit ImageNet-Ik
951.
xcit_ tiny 12_p16_2244
ImageNet-Ik
952.
xcit tiny ____________________1_______11__ pl6__22.ffbbbbbbpppppp 224.fb inlk
tim vit
ImageNet-
953.
xcit_ting tint__nn_ylll_______ee_ 12__11__________ (16__384.fbooee inlk
timm vit ImageNet-
954.
xcit_ ttt__innn______ 12____1____________________1____
12__8_224_ff f24.fb fh_dist_inkk
timm vit ImageNet-1k
955.
xcit_ cit_ttny____ _2n___1__________ 1t__11____22.f___p___ 2224fb_inlk
vit ImageNet-
956.
xcit tttt_inny22 1____12___________ __2____34.ffb____ fet.ff___ssiiiiieeee dist_inlk
timm
ImageNet-1k
957.
xcit tiny 24 p16 224.fb dist inlk
timm
vit ImageNet-1k
958.
xcit tiny 24 p16 224.fb inlk
timm
vit ImageNet-
959.
xcit tiny 24 p16 384.fb dist inlk
timm vit
ImageNet-Ik
Table continues onto next page
45

---

Model summaries (continued from previous page)
No
Model ID
Source
Arch
Dataset
960.
xcit tiny_ ______________________ p8 222.fb_diss dist_inlk
timm
ImageNet-11 1k ImageNet- 1k
961.
xcit tiny 24 p8 224.fb inlk
timm
vit
962.
xcit tiny 24 p8. 384.fb dist inlk
timm
vii
ImageNet-Il
F,4 Fine-grained Superclasses
In this section, detail the superclasses use for detecting fine-grained and fine-grained OOV errors and their constituting IMAGENET classes. When defining these superclasses, we focused on visual and semantic similarity rather than positions in the WordNet hierarchy. We believe that the latter is fundamentally unsuitable to obtain high-quality groupings, as visually similar or closely related classes often end up having a large WordNet distance, while very different classes are close. Further, we ensured that superclass definitions are not overly generic, providing a much more fine- grained split than prior works. The superclasses defined programmatically in our code and be easily reused and adapted by others.
List of superclass definitions:
1. butterfly- - classes:
(n02276258, admiral) (n82279972, monarch)
(n62277742, ringlet)
(n02280649, cabbage butterfly)
(n02281486, sulphur butterfly)
(n82281787, lycaenid)
2. beetle_plus- classes:
(n02165105, tiger beetle)
(n02165456, ladybug)
(n82167151, ground beetle)
(n82168699, long- beetle)
(n82169497, leaf beetle)
(n@2172182, dung beetle)
(n82174001, rhinoceros beetle)
(n82177972, weevil)
(n82226429, grasshopper)
(n02229544, cricket)
(n82233338, cockroach)
(n02256656, cicada)
(n02259212, leafhopper)
insect_rest 13 classes:
(n82190166, fly)
(n02286856, bee)
(n82219486, ant)
(n02226429, grasshopper)
(n02229544, cricket)
(n02231487, walking stick)
(n82233338, cockroach)
(n02236044, mantis)
(n82256656, cicada)
(n82259212, leafhopper)
(n82264363, lacewing)
(n02268443, dragonfly)
(n82268853, damselfly)
fropod_trillbbite.chh
classes:
(n01768244, trilobite) (n01990800, isopod)
(n01955084, chiton)
5. crab_scorpion - - classes:
(n01770393, scorpion) (n01978455, rock crab)
(n01978287, Dungeness crab)
(n01980166, fiddler crab)
(n81981276, king crab)
(n01983481, American lobster)
(n01984695, spiny lobster)
(n01985128, crayfish)
(n81986214, hermit crab)
spider classes:
(n01770081, harvestman)
(n01773157, black and gold garden spider)
(n01773549, barn spider)
(n01773797, garden spider)
(n01774384, black widow)
(n01774750, tarantula)
(n81775062, wolf spider)
(n01776313, tick)
7.
- classes:
46

---

(n81914689, anemone) (n81950731, sea slug)
(n01917289, brain coral) (n02317335, starfish)
(n82319895, urchin) (n12985857, coral fungus)
(n09256479, coral reef)
conch.chambered_nnaae nautillus - classes: (n81943899, conch)
(n01968897, chambered nautilus)
9. invertebrate-rest- classes:
(n81784675, centipede) (n81930112, nematode) (n81945685, slug) (n82321529, cucumber)
(n01924916, flatworm)
(n01944390, snail)
(n01950731, sea slug)
10. salamander_lizard 16 classes:
(n01629819, European fire sa salamander) (n01631663, eft)
(n01630670, common newt)
(n01632458, spotted salamander)
(n01632777, axolotl)
(n01675722, banded gecko)
(n01677366, common iguana)
(n01682714, American chameleon)
(n01685808, whiptail)
(n01687978, agama)
(n81688243, frilled lizard)
(n01689811, alligator lizard)
(n01692333, Gila monster)
(n01693334,
green lizard)
(n01694178, African chameleon)
(n01695060,
Komodo dragon)
11. frog -3 classes:
(n01641577, bullfrog) (n81644960, tailed frog)
(n01644373, tree frog)
12. shark_ann__auatiiccmmeee - classes:
(n81484850, great white shark)
(n81491361, tiger shark) (n82066245, grey whale)
(n81494475, hammerhead) (n82071294, killer whale)
(n02074367, dugong)
(n82877923, sea lion)
13. ray 2 classes:
(n81496331, electric ray)
(n01498041, stingray)
14. fish-rest 11 classes:
(n01440764, tench)
(n01443537, goldfish)
(n02514041, barracouta) (n02536864, coho)
(n@2526121, eel)
(n82606052, rock beauty)
(n82607872, (n82641379, gar)
fish)
(n02648242, sturgeon)
(n82643566, lionfish)
(n82655020, puffer)
15. turtle classes:
(n01664065, loggerhead) (n81667114, mud turtle) (n01669191, box turtle)
(n01665541, leatherback turtle) (n01667778, terrapin)
16. crocodillan__eptille classes:
(n81697457, African crocodile)
(n01698640, American alligator)
17. snake 17 classes:
(n01728572, thunder snake) (n81729322, hognose snake)
(n01728920, ringneck snake)
(n01729977, green snake)
(n01734418, king snake)
(n01735189, garter snake)
(n81737621, water snake)
(n01739381, vine snake)
(n01740131, night snake)
(n@1742172, boa constrictor)
(n81744401, rock python)
(n01748264, Indian cobra)
(n81749939, green mamba)
(n01751748, sea snake)
(n81753488, horned viper)
(n01755581, diamondback)
(n81756291, sidewinder)
18. green_snaae_and_laa lizards -5 classes:
47

---

(n81682714, American chameleon)
(n01693334, green lizard)
(n81729977, green snake)
(n01739381, vine snake)
(n81749939, green mamba)
19. non-green_snakes 15 classes:
(n01689811, alligator lizard)
(n01728572, thunder snake)
(n81728920, ringneck snake)
(n01729322, hognose snake)
(n81734418, king snake)
(n01735189, garter snake)
(n81737021, water snake)
(n81740131, night snake)
(n81742172, boa constrictor)
(n01744401, rock python)
(n81748264, Indian cobra)
(n01751748, sea snake)
(n81753488, horned viper)
(n01755581, diamondback)
(n81756291, sidewinder)
cock_hen.gaallincceuus classes:
(n91514668, cock)
(n01514859, hen)
(n81795545, black grouse)
(n01796340, ptarmigan)
(n01797886, ruffed grouse)
(n01798484, prairie chicken)
(n81806143, peacock)
(n01896567, quail)
(n81807496, partridge)
(n82017213, European gallinule)
(n82018267, American coot)
(n02018795,
bustard)
small_birdd classes:
(n81530575, brambling)
(n81531178, goldfinch)
(n81532829, house finch)
(n81534433, junco)
(n81537544, indigo bunting)
(n01558993, robin)
(n81566419, bulbul)
(n01580077, jay)
(n81582220, magpie)
(n01592084, chickadee)
(n81601694, water ouzel)
(n01824575, coucal)
(n01828970, eater)
(n01833805, hummingbird)
(n01843065, jacamar)
22. bird_of-prey 3 classes:
(n81608432, kite) (n81616318, vulture)
(n01614925, bald eagle)
23. parrot 4 classes:
(n81817953, African grey) (n01819313, sulphur-crested cockatoo)
(n01818515, macaw) (n01820546, lorikeet)
24. big-beak_bird classes:
(n81829413, hornbill)
(n81843383, toucan)
25. aquatic_birrLaandLstridd 25 classes:
(n01518878, ostrich)
(n01847000, drake)
(n01855032, red- ad-breestee merganser)
(n01855672, goose)
(n01860187, black swan)
(n02002556, white stork)
(n82002724, black stork)
(n02096656, spoonbill)
(n82007558, flamingo)
(n82089229, little blue heron)
(n82009912, American egret)
(n02011460, bittern)
(n82012849, crane)
(n02013706, limpkin)
(n82017213, European gallinule)
(n02018207, American coot)
(n82018795, bustard)
(n02025239, ruddy turnstone)
(n02027492, red-backed sandpiper)
(n02028035, redshank)
(n02033041, dowitcher)
(n02037110, oystercatcher)
(n02051845, pelican)
(n02056570,
king penguin)
(n82058221, albatross)
26. tusker_elephant- classes:
(n81871265, tusker)
(n02584013, Indian elephant)
(n82504458, African elephant)
27. primate - 20 classes:
48

---

(n82480495,
orangutan)
(n02480855, gorilla)
(n82481823,
chimpanzee)
(n82483362, gibbon)
(n82483708,
siamang)
(n62484975, guenon)
(n82486261,
patas)
(n02486410, baboon)
(n82487347, macaque)
(n02488291, langur)
(n82488762,
colobus)
(n02489166, proboscis monkey)
(n02490219, marmoset)
(n02492035, capuchin)
(n02492660, howler monkey)
(n02493509, titi)
(n02493793, spider monkey)
(n02494079, squirrel monkey)
(n02497673, Madagascar cat)
(n02500267, indri)
28. bear-panda_marsupiaa. 10 classes:
(n01877812, wallaby)
(n01882714, koala)
(n01883870, wombat)
(n02132136, brown bear)
(n02133161, American black bear)
(n02134084, ice bear)
(n02134418, sloth bear)
(n02457408, three-toed sloth)
(n02509815, lesser panda)
(n02510455, giant panda)
29. pwterrannennstteeeee.eeeeeeeeeon
19 classes:
(n81872481, echidna)
(n01873310, platypus)
(n81877812, wallaby)
(n82137549, mongoose)
(n82138441, meerkat)
(n02342885, hamster)
(n02346627, porcupine)
(n02356798, fox squirrel)
(n02361337, marmot)
(n02363005, beaver)
(n02364673, guinea pig)
(n02441942, weasel)
(n02442845, mink)
(n02443114, polecat)
(n02443484, black-footed ferret)
(n82444819, otter)
(n82445715, skunk)
(n02447366,
badger)
(n02454379, armadillo)
30. rabbit classes:
(n02325366, wood rabbit) (n02328150, Angora)
(n02326432, hare)
31. ungulate- 17 classes:
(n82389026, sorrel)
(n02391049, zebra)
(n82395496, hog)
(n02396427, will boar)
(n82397896, warthog)
(n02398521, hippopotamus)
(n82403063, ox)
(n02488429, water buffalo)
(n02410509, bison)
(n02412080, ram)
(n02415577, bighorn)
(n02417914, ibex)
(n02422106, hartebeest)
(n02422699, impala)
(n82423622, gazelle)
(n82437312, Arabian camel)
(n82437616, Ilama)
32. domestic_cat classes:
(n82123645, tabby) (n82123394, Persian cat) (n82124075, Egyptian cat)
(no2123159, tiger cat)
(n82123597, Siamese cat)
33. wild_cat_biigcct classes:
(n82125311, cougar)
(n02127052, lynx)
(n82128385, leopard)
(n02128757, snow leopard) (n02129165, lion) (n02130308, cheetah)
(n02128925, jaguar) (n82129664, tiger)
34. wolf nolff 12 classes:
(n82114367, timber wolf)
(n62114548, white wolf)
(n82114712, red wolf)
(n02114855, coyate)
(n82115641, dingo)
(n82115913, dhole)
(n82116738, African hunting dog)
(n02117135, hyena) (n62119789, kit fox)
(n82119622, red fox)
(n82126879, Arctic fox)
(n02128505, grey fox)
49

---

33. pug_boxer_bulldog- classes: (n82093256, Staffordshire bullterrier)
(n62093428, American Staffordshire terrier)
(n82096585, Boston bull)
(n02106550, Rottweiler)
(n82108889, boxer)
(n82108422, bull mastiff)
(n82108915, French bulldog)
(n62118958, pug)
(n02112706, Brabancon griffon)
36. toy_dog 31 classes:
(n82085620, Chihuahua)
(n82085782, Japanese spaniel)
(n02085936, Maltese dog)
(n02086079, Pekinese)
(n82086240, Shih-Tzu)
(n02086646, Blenheim spaniel)
(n02086910, papillon)
(n02087046, toy terrier)
(n02093754, Border terrier)
(n82094114, Norfolk terrier)
(n82094258, Norwich terrier)
(n02094433, Yorkshire terrier)
(n02096177, cairn)
(n02096294, Australian terrier)
(n02096437, Dandie Dinmont)
(n02096585, Boston bull)
(n82097847, miniature schnauzer)
(n82097474, Tibetan terrier)
(n82097658, silky terrier)
(n02098105, soft-coated wheaten terrier)
(n82098286, West Highland white terrier)
(n62098413, Lhasa)
(n82102318, cocker spaniel)
(n02107312, miniature pinscher)
(n82108915, French bulldog)
(n82110627, affenpinscher)
(n82112018, Pomeranian)
(n92112706, Brabancon griffon)
(n82113624, toy poodle)
(n02113712,
miniature poodle)
(n82113799, standard poodle)
37. corgi - 2 classes:
(n82113823, Pembroke)
(n62113186, Cardigan)
38. terrier 26 classes:
(n82090721, Irish wolfhound) (n82093647, Bedlington terrier)
(n82091635, otterhound)
(n02093754, Border terrier)
(n82093859, Kerry blue terrier)
(n02093991, Irish terrier)
(n82094114, Norfolk terrier)
(n02094258, Norwich terrier)
(n82094433, Yorkshire terrier)
(n82095314, wire-haired fox terrier)
(n82095570, Lakeland terrier)
(n02095889, Sealyham terrier)
(n82096951, Airedale)
(n82096177, cairn)
(n02096294, Australian terrier)
(n02096437, Dandie Dinmont)
(n82097647, miniature schnauzer)
(n02097130, giant schnauzer)
(n82897269, standard schnauzer)
(n02097298, Scotch terrier)
(n02097474, Tibetan terrier)
(n82097658, silky terrier)
(n02098165, coated wheaten terrier)
(n02098286, West Highland white terrier)
(n02098413, Lhasa)
(n02110627, affenpinscher)
39. loodle_like - classes:
(n82093647, Bedlington terrier) (n82099429, curly-coated retriever) (n82113624, toy poodle)
(n02093859, Kerry blue terrier)
(n82102973, Irish water spaniel)
(n02113712, miniature poodle)
(n82113799, standard poodle)
40. standardd.shepherd..dg classes:
(n82091467, Norwegian elkhound)
(n82104365, schipperke)
(n02105056, groenendael)
(n02105162, malinois)
(n82105412, kelpie)
(n02105855, Shetland sheepdog)
(n82106830, collie)
(n02106166, Border collie)
(n02106550, Rottweiler)
(n02106662, German shepherd)
(n82112350, keeshond)
41. hairy-shepherrd_dg- classes:
(n82097474, Tibetan terrier)
(n02105251, briard) (n02185641, 0ld English sheepdog) (n02118627, affenpinscher)
(n82105585, komondor)
(n82106382, Bouvier des Flandres)
50

---

42.
27 classes:
(n02087394, Rhodesian ridgeback)
(n02088238, basset)
(n02088364, beagle)
(n02088466, bloodhound)
(n82088632, bluetick)
(n02089078, black-and- coonhound)
(n82089867, Walker hound)
(n02089973, English foxhound)
(n02090379, redbone)
(n62092339, Weimaraner)
(n02100236, German short-haired pointer)
(n02100583, vizsla)
(n82100735, English setter)
(n02180877, Irish setter)
(n82101086, Gordon setter)
(n02101388, Brittany spaniel)
(n82101556, clumber)
(n62102040, English springer)
(n82102177, Welsh springer spaniel)
(n02102318, cocker spaniel)
(n02102480,
Sussex spaniel)
(n02102973, Irish water spaniel)
(n82107142, Doberman)
(n02109047, Great Dane)
(n82110341, dalmatian)
(n02110806, basenji)
(n82113978, Mexican hairless)
43. spaniels classes:
(n02086646, Blenheim spaniel) (n02102040, English springer) (n02102318, cocker spaniel)
(n02101388, Brittany spaniel)
(n02102177, Welsh springer spaniel)
44. skimmy_hound_rest- 12 classes:
(n82088094, Afghan hound)
(n82098622, borzoi)
(n82090721, Irish wolfhound)
(n02091032, Italian greyhound)
(n82091134, whippet)
(n02091244, Ibizan hound)
(n82091635, otterhound)
(n82091831, Saluki)
(n82092002, Scottish deerhound)
(n02107142, Doberman)
(n82110806, basenji)
(n02113978,
Mexican hairless)
45. pinscher classes:
(n82106550, Rottweiler)
(n02107142, Doberman
(n02107312, miniature pinscher)
46. beanentundee_ndd.big..ddd - 15 classes:
(n82088364, beagle)
(n02106166, Border collie)
(n02106550, Rottweiler)
(n62187574, Greater Swiss Mountain dog)
(n82107683, Bernese mountain dog)
(n02107908, Appenzeller)
(n02108060, EntleBucher)
(n82108089, boxer)
(n82108422, bull mastiff)
(n02108551, Tibetan mastiff)
(n82109047, Great Dane)
(n02189525, Saint Bernard)
(n82111129, Leonberg)
(n02111277,
Newfoundland)
(n82112137, chow)
47. retriever_like 10 classes:
(n02090379, redbone)
(n02099267, flat-coate retriever)
(n02099429, curly-coated retriever)
(n02099601, golden retriever)
(n02099712, Labrador retriever)
(n02099849, Chesapeake Bay retriever) (n62184029, kuvasz)
(n02101006, Gordon setter) (n82111277, Newfoundland)
(n02111500, Great Pyrenees)
48. husky classes:
(n82091467, Norwegian elkhound) (n82116963, malamute) (n82111889, Samoyed)
(n82189961, Eskimo dog)
(n82110185, Siberian husky) (n02112350, keeshond)
49. spitz 4 classes:
(n02111889, Samoyed) (n02112137, chow)
(n62112018, Pomeranian! (n02112350, keeshond)
50. fungi - 7 classes:
(n87734744, mushroom)
(n12998815, agaric)
(n13037486, gyromitra)
(n13040303, stinkhorn)
(n13044778, earthstar)
(n13052670, hen- ten-of-the-woods)
(n13054560, bolete)
51

---

51. geological_formmttonn classes: (n89193705, alp) (n89288635, geyser) (n89399592, promontory) (n89428293, seashore) (n89472597, volcano)
(n09246464, cliff)
(n09332890, lakeside)
(n09421951, sandbar) (n09468604, valley)
52. bread classes: (n87684084, French loaf) (n07695742, pretzel)
(n07693725, bagel) (n07850988, dough)
53. cruciferous..eggeabbee---e classes:
(n07714571, head cabbage) (n87715103, cauliflower)
(n07714990, broccoli)
54. cur classes: (n87715103, cauliflower)
(n87760859, custard apple)
55. zucchini_cucummerr classes: (n87716358, zucchini)
(n07718472, cucumber)
56. squash 4 classes:
(n87716358, zucchini) (n87717410, acorn squash)
(n07716906, spaghetti squash) (n07717556, butternut squash)
57. reproductiive_struuuue 12 classes:
(n07720875, bell pepper)
(n07742313, Granny Smith)
(n87745940, strawberry)
(n07747607, orange)
(n87749582, lemon)
(n07753113, fig)
(n07754684, jackfruit)
(n07760859, custard apple)
(n87768694, pomegranate)
(n12267677, acorn)
(n12620546, hip)
(n12768682, buckeye)
58. corn classes:
(n12144580, corn)
(n13133613, ear)
59. cardoon classes: (n87718747, artichoke) (n87753275, pineapple)
(n87730033, cardoon)
60. dessert 3 classes:
(n87613480, trifle) (n07836838, chocolate sauce)
(n07614500, ice cream)
61. bady_bed classes:
(n02804414, bassinet) (n83131574, crib)
(n03125729, cradle)
62. chair classes:
(n82791124, barber chair) (n84899969, rocking chair)
(n83376595, folding chair) (n04429376, throne)
entertaiimeett.eetter....o.e. theatre classes:
(n83290653, entertainment center)
(n83529860, home theater)
64. cabinet 7 classes: (n02870880, bookcase)
(n03016953, chiffonier)
(n83018349, china cabinet) (n83337140, file) (n84550184, wardrobe)
(n03298653, entertainment center)
(n83742115, medicine chest)
65. table classes:
(n83179701, desk)
(n03201208, dining table)
66. robee.overgammett.ccook 10 classes:
52

---

(n82667893, abaya)
(n02669723, academic gown)
(n82730930, apron)
(n83045698, cloak)
(n83404251, fur coat)
(n03617480, kimono)
(n83630383, lab coat)
(n03980874, poncho)
(n84479846, trench coat)
(n04532106,
vestment)
67. sweater classes:
(n82963159, cardigan) (n84370456, sweatshirt)
(n03980874, poncho)
68. suits - classes:
(n83763968, military uniform) (n84358985, suit)
(n63877472, pajama)
69. woolen_pieces classes:
(n02963159, cardigan)
(n03297743, dishrag) (n03980874, poncho) (n04325704, stole)
(n03775071, mitten)
(n84229816, ski mask)
(n84599235, wool)
70. caps - classes:
(n82807133, bathing cap) (n84209133, shower cap)
(n02869837, bonnet)
swimsuit -4 classes:
(n02837789, bikini) (n83716637, maillot)
(n02892767, brassiere)
(n03710721, maillot)
72. woman_dress - classes:
(n83456230, gown) (n03866082, overskirt)
(n03534589, hoopskirt)
73. neckwear- - classes:
(n02865351, bolo tie) (n83814906, necklace)
(n02883205, bow tie)
(n84591157, Windsor tie)
74. sock_and_glove classes:
(n83026506, Christmas stocking) (n84254777, sock)
(n03775071, mitten)
75. breathing-device 3 classes:
(n03424325, gasmask) (n04251144, snorkel)
(n03868863, oxygen mask)
76. helmet classes:
(n83127747, crash helmet) (n83929855, pickelhaube)
(n83379051, football helmet)
77. pickethaaue_bbearkkin 2 classes:
(n82817516, bearskin)
(n03929855, pickelhaube)
78. hat - classes: (n02869837, bonnet) (n04259630, sombrero)
(n03124170, cowboy hat)
79. armor 4 classes: (n82895154, breastplate) (n83146219, cuirass)
(n83098247, chain mail) (n04192698, shield)
80. cuirasssbullettrooff.est classes: (n82916936, bulletproof vest)
(n03146219, cuirass)
81. footwear_rest classes:
(n83047690, clog) (n03680355, Loafer) (n84133789, sandal)
(n03124043, cowboy boot) (n84128489, running shoe)
82. top_cover 4 classes:
(n02877765, bottlecap)
(n03657121, lens cap) (n04019541, puck)
(n83717622, manhole cover)
53

---

83. lamp_light classes:
(n02948072, candle) (n83637318, lampshade) (n84380533, table lamp)
(n03590841, jack-o' -lantern) (n04286575, spotlight)
84. tandle_torch- classes: (n82948072, candle)
(n04456115, torch)
85. ship_boat - 15 classes:
(n82687172, aircraft carrier)
(n62951358, canoe)
(n02981792, catamaran)
(n03095699, container ship)
(n03344393, fireboat)
(n03447447, gondola)
(n03662601, lifeboat)
(n03673027, liner)
(n83947888, pirate)
(n04147183, schooner)
(n84273569, speedboat)
(n84347754, submarine)
(n84483307, trimaran)
(n04686251,
wreck)
(n84612504, yawl)
86. tank 3 classes:
(n02704792, amphibian) (n84389033, tank)
(n03478589, half track)
87. snow_sled classes:
(n02860847, bobsled) (n84252877, snowmobile)
(n03218198, dogsled)
88. motor_cycle-66clsee (n82835271, bicycle-built-for-e (n83791053, motor scoater) (n84482393, tricycle)
(n83785016, moped) (n03792782, mountain bike) (n84589417, unicycle)
89. aircraft- classes:
(n82690373, airliner) (n84552348, warplane)
(n62692877, airship)
90. balloon 3 classes: (n02692877, airship) (n03888257, parachute)
(n02782093, balloon)
91. 6 classes:
(n82917867, bullet train) (n03393912, freight car) (n04310018, steam locomotive)
(n03272562, electric locomotive) (n03895866, passenger car) (n04335435, streetcar)
92. car_bus_truck 23 classes:
(n02701002, ambulance)
(n02704792, amphibian)
(n02814533, beach wagon)
(n02930766, cab)
(n03100240, convertible)
(n03345487, fire engine)
(n83417842, garbage truck)
(n03594945, jeep)
(n03670268, limousine)
(n03769881, minibus)
(n03770679, minivan)
(n03777568, Model
(n03796401, moving van)
(n83930630, pickup)
(n83977966, police van)
(n84037443, racer)
(n64065272, recreational vehicle)
(n84146614, school bus)
(n84252225, snowplow)
(n04285008, sports car)
(n84461696, tow truck)
(n04467665, trailer truck)
(n04487081, trolleybus)
93. racer-go_kart - classes: (n83444034, ga-kart)
(n04037443, racer)
94. work_cart classes:
(n83384352, forklift) (n83445924, golfcart)
(n03444034, go- ga-kart)
(n03649909, lawn mower)
54

---

95. open_cart 4 classes:
(n02797295, barrow) (n83599486, jinrikisha)
(n03538406, horse cart) (n03868242, oxcart)
96. rocket_missile- classes:
(n83773584, missile) (n84256814, space shuttle)
(n04008634, projectile)
97. keyboard_innttrument classes:
(n83452741, grand piano) (n84515063, upright)
(n03854965, organ)
98. striingd classes:
(n02676566, acoustic guitar) (n82992211, cello) (n84536866, violin)
(n02787622, banjo) (n03272010, electric guitar)
99. Sannd_instrument.brass classes: (n03110569, cornet) (n84141076, sax)
(n03394916, French horn) (n04487394, trombone)
100. windd_nsttrument..wwwwddd classes: (n02804610, bassoon) (n03838899, oboe)
(n03372029, flute) (n04141076, sax)
101. wiinddinstrumenn.wwitteeeee classes: (n82804610, bassoon) (n83838899, oboe)
(n03372029, flute) (n84579432, whistle)
102. percussion_instruuentt classes:
(n83017168, chime) (n83447721, gong)
(n83249569, drum) (n83721384, marimba)
(n84311174, steel drum)
103. wwwppcpplacceeeeeppppppppppaeaap classes:
(n02776631, bakery)
(n02791270, barbershop)
(n82871525, bookshop)
(n02927161, butcher shop)
(n83032252, cinema)
(n83089624, confectionery)
(n83461385, grocery store)
(n03661043, library)
(n84081281, restaurant)
(n04200800, shoe shop)
(n84443257, tobacco shop)
(n04462240,
toyshop)
104. bridge_pier_dam- - classes:
(n83168389, dam)
(n83933933, pier)
(n84311004, steel arch bridge) (n84532670, viaduct)
(n04366367, suspension bridge)
105. gale_ball 9 classes:
(n02799071, baseball)
(n02802426, basketball)
(n83134739, croquet ball)
(n03445777, golf ball)
(n83942813, ping-pong ball)
(n04118538, rugby ball)
(n84254680, soccer ball)
(n84489515, tennis ball)
(n84540053, volleyball)
106. computer_monitor classes:
(n03180011, desktop computer) (n03642806, laptop) (n83832673, notebook)
(n03485407, hand-held computer)
(n03782006, monitor)
(n83857828, oscilloscope) (n04484412, television)
(n84152593, screen)
107. keyboard classes:
(n03085013, computer keyboard) (n03642806, laptop)
(n03485407, hand-hell computer) (n03832673, notebook) (n04505470, typewriter keyboard)
(n84264628, space bar)
55

---

108. player 4 classes:
(n02979186, cassette player) (n84041544, radio)
(n02988304, CD player)
(n84392985, tape player)
109. smaalleeeleetrroiicc.eiippeee classes:
(n02978881, cassette)
(n02992529, cellular telephone)
(n83271574, electric fan)
(n03485407, hand-held computer)
(n83492542, hard disc)
(n03584254, iPod)
(n83602883, joystick) (n84372370, switch)
(n04074963, remote control)
110. phone 33 classes:
(n82992529, cellular telephone) (n83902125, pay-phone)
(n03187595, dial telephone)
111. radio_modem 3 classes:
(n83492542, hard disc) (n84041544, radio)
(n63777754, modem)
112. biggeeleettroncc...quupmeeet- classes: (n83691459, loudspeaker) (n83857828, oscilloscope) (n04004767, printer)
(n63777754, modem)
(n03924679, photocopier) (n04009552, projector)
113. clock classes:
(n02708093, analog clock) (n83706229, magnetic compass)
(n82794156, barometer)
(n03841143, odometer)
(n83891332, parking meter) (n84328186, stopwatch) (n84548280, wall clock)
(n04141975, scale)
(n04355338, sundial)
114. digital_clock 5 classes:
(n03196217, digital clock) (n84141975, scale) (n84328186, stopwatch)
(n03197337, digital watch) (n84149813, scoreboard)
115. ruler - classes: (n84118776, rule) (n84376876, syringe)
(n84238763, slide rule)
116. firearm classes:
(n02749479, assault rifle) (n84090263, rifle)
(n04086273, revolver)
117. aympasticcapparrtuu - classes:
(n82777292, balance beam) (n03888605, parallel bars)
(n03535789, horizontal bar)
118. barbell_dumbbell - 2 classes: (n82790996, barbell)
(n03255030, dumbbell)
119. dish 13 classes:
(n64263257, soup bowl) (n07583066, guacamole)
(n87579787, plate)
(n07584110, consomme)
(n87590611, hot pot)
(n07697313, cheeseburger)
(n87697537, hotdog)
(n07711569, mashed potato)
(n87831146, carbonara)
(n07871810, meat loaf)
(n87873807, pizza)
(n87875152, potpie)
(n87880968, burrito)
120. beverage-pot-juge 18 classes:
56

---

(n82815834, beaker)
(n02823750, beer glass)
(n83062245, cocktail shaker)
(n83063599, coffee mug)
(n83063689, coffeepot)
(n03443371, goblet)
(n83733805, measuring cup)
(n03956228, pitcher)
(n84131690, saltshaker)
(n04254129, soap dispenser)
(n84398644, teapot)
(n04522168, vase)
(n04560804, water jug)
(n04579145, whiskey jug)
(n07892512, red wine)
(n07920052, espresso)
(n07930864, cup)
(n07932039,
eggnog)
121. spatulla_soon_taale - classes: (n03633091, ladle)
(n04270147, spatula)
(n84597913, wooden spoon)
122. utensils 11 classes:
(n02909870, bucket)
(n02939185, caldron)
(n03133878, Crock Pot)
(n03259280, Dutch oven)
(n83400231, frying pan)
(n03775546, mixing bowl)
(n83786961, mortar)
(n83992509, potter's wheel)
(n84263257, soup bowl)
(n04332243, strainer)
(n84596742, wok)
123. bucket_pot-22 classes: (n02909870, bucket)
(n03991062, pot)
124. bottle classes:
(n82823428, beer bottle) (n84557648, water bottle)
(n03983396, pop bottle) (n04591713, wine bottle)
125. busin 3 classes: (n02808440, bathtub) (n04553703, washbasin)
(n04493381, tub)
126. barrel_can classes:
(n02747177, ashcan) (n03764736, milk can)
(n82795169, barrel) (n04049303, rain barrel)
127. appaaneei classes: classes:
(n03207941, dishwasher)
(n83297495, espresso maker) (n04070727, refrigerator) (n84338267, stove)
(n03761084, microwave)
(n84111531, rotisserie)
(n84554684, washer)
128. kitchen_appliance classes:
(n83297495, espresso maker) (n84442312, toaster)
(n03761084, microwave) (n84542943, waffle iron)
129. smatLapptiance classes:
(n03483316, hand blower) (n84179913, sewing machine)
(n83584829, iron) (n84517823, vacuum)
130. - classes:
(n03761084, microwave) (n84330267, stove)
(n04111531, rotisserie) (n84442312, toaster)
(n84542943, waffle iron)
131. space_heater_rraiitoor- classes: (n84040759, radiator)
(n04265275, space heater)
132. paper_tissue classes:
(n03887697, paper towel)
(n15075141, toilet tissue)
133. cover_presentation 10 classes:
(n82786958, Band Aid)
(n02849245, binder)
(n83291819, envelope)
(n83598930, jigsaw puzzle) (n06359193, web site) (n06785654, crossword puzzle) (n87565083, menu)
(n03871628, packet)
(n86596364, comic book)
(n87248320, book jacket)
57

---

134. toiletry 5 classes:
(n03314780, face powder) (n83690938, lotion) (n84357314, sunscreen)
(n83476991, hair spray) (n03916031, perfume)
135. bed 2 classes:
(n83388549, four-posters
(n84344873, studio couch)
136. fabric 8 classes:
(n82808384, bath towel) (n83207743, dishrag) (n03485794, handkerchief) (n84209239, shower curtain)
(n02834397, bib)
(n63223299, doormat)
(n03998194, prayer rug)
(n04525038, velvet)
137. reltsiouss.cettte.plleee 12 classes:
(n82699494, altar)
(n02825657, bell cote)
(n82988441, castle)
(n03028079, church)
(n83220513, dome)
(n83781244, monastery)
(n03788195, mosque)
(n03877845, palace)
(n03956157, planetarium)
(n04346328, stupa)
(n04486054, triumphal arch)
(n04523525, vault)
138. pole -3 3 classes:
(n03355925, flagpole) (n83976657, pole)
(n03733131, maypole)
139. column 4 classes: (n83837869, obelisk) (n83976657, pole)
(n83983868, pedestal)
(n84458633, totem pole)
140. stick - 26 classes:
(n82666196, abacus)
(n02783161, ballpoint)
(n82906734, broom)
(n83141823, crutch)
(n03250847, drumstick)
(n03355925, flagpole)
(n03388183, fountain pen)
(n03658185, letter opener)
(n03729826, matchstick)
(n03759954, microphone)
(n83804744, nail)
(n03873416, paddle)
(n83876231, paintbrush)
(n83944341, pinwheel)
(n83970156, plunger)
(n04033901, quill)
(n84039381, racket)
(n84067472, reel)
(n84141327, scabbard)
(n04153751, screw)
(n84208210, shovel)
(n04228054, ski)
(n84277352, spindle)
(n04367489, swab)
(n04485082, tripod)
(n04501370, turnstile)
141. big_standing.machine classes: (n82977058, cash machine) (n84243546, slot)
(n03425413, gas pump)
(n04525305, vending machine)
142. farm_machine 6 classes: (n83496892, harvester) (n03967562, plow) (n84428191, thresher)
(n83874293, ledheee (n04252225, snowplow)
(n04465501, tractor)
143. ax_meat.knife 2 classes:
(n03041632, cleaver)
(n03498962, hatchet)
144. hand_tool= 8 classes:
(n02951585, can opener) (n83000684, chain saw) (n03481172, hammer) (n03995372, power drill)
(n02966687, carpenter's kit)
(n03189150, corkscrew)
(n03954731, plane)
(n04154565, screwdriver)
145. sunglass classes: (n84355933, sunglass)
(n84356056, sunglasses)
58

---

146. structureenear__wtee classes: (n82814960, beacon) (n82894685, breakwater)
(n92859443, boathouse) (n63216828, dock)
147. -ater--ower- classes: (n82814860, beacon)
(n04562935, water tower)
148. camera_binoculars classes:
(n02841315, binoculars) (n84069434, reflex camera)
(n03976467, Polaroid camera)
149. safe 2 classes:
(n83075370, combination lock)
(n04125021, safe)
150. lock 2 classes:
(n03075370, combination lock)
(n03874599, padlock)
151. box classes:
(n02727426, apiary)
(n02843684, birdhouse)
(n02971356, carton)
(n03014705, chest)
(n83127925, crate)
(n83482405, hamper)
(n83710193, mailbox)
(n84294238, shopping basket)
(n84204347, shopping cart)
152. ashcan_mailbox classes:
(n02747177, ashcan)
(n03710193, mailbox)
153. bag 4 classes:
(n82769748, backpack) (n84826417, purse)
(n83789823, mailbag)
(n84548362, wallet)
154. purse classes:
(n83908618, pencil box) (n84548362, wallet)
(n84026417, purse)
155. shoji_stiding classes: (n84201297, shoji)
(n04239074, sliding door)
156. construction.site 2 classes:
(n83126707, crane)
(n83249683, drilling platform)
157. patterned_structure- 16 classes:
(n02788148, bannister)
(n82999410, chain)
(n03000134, chainlink fence)
(n03065424, coil)
(n03347037, fire screen)
(n03459775, grille)
(n03530642, honeycomb)
(n03930313, picket fence)
(n84005630, prison)
(n84048759, radiator)
(n84265275, space heater)
(n04275548, spider web)
(n84326547, stone wall)
(n84589899, window screen)
(n84590129, window shade)
(n04604644, worm fence)
158. roof - classes:
(n84417672, thatch)
(n84435653, tile roof)
159. small_ojeect - 20 classes:
(n02865351, bolo tie)
(n82910353, buckle)
(n03314780, face powder)
(n03476684, hair slide)
(n03532672, hook)
(n83666591, lighter)
(n03676483, lipstick)
(n03692522, loupe)
(n83794856, rausetrap)
(n83814906, necklace)
(n83846681, ocarina)
(n03843555, oil filter)
(n83908714, pencil sharpener)
(n83929660, pick)
(n04116512, rubber eraser)
(n04127249, safety pin)
(n84131690, saltshaker)
(n04372370, switch)
(n84423845, thimble)
(n04579432, whistle)
59

---

160. houssing.famiinggstuuuuueee classes: (n02793495, barn)
(n02859443, boathouse)
(n03457962, greenhouse)
(n03697007, lumbermi (t) (n03899768, patio)
(n03776460, mobile home)
(n84613696, yurt)
161. soleeeddssh...elescpp classes: (n84044716, radio telescope)
(n04258138, solar dish)
162. Without superclass 74 classes:
(n81622779, great grey owl)
(n01784323, triceratops)
(n81910747, jellyfish)
(n02672831, accordion)
(n02879718, bow)
(n02892201, brass)
(n82956826, cannon)
(n02965783, car mirror)
(n02966193, carousel)
(n02974903, car wheel)
(n83042490, cliff dwelling)
(n03188531, diaper)
(n03208938, disk brake)
(n03325584, feather boa)
(n03388043, fountain)
(n03467068, guillotine)
(n03494278, harmonica)
(n03495258, harp)
(n83527444, holster)
(n03544143, hourglass)
(n83594734, jean)
(n03595614, jersey)
(n03623198, knee pad)
(n83627232, knot)
(n83720891, maraca)
(n03724870, mask)
(n83733281, maze)
(n83743016, megalith)
(n83778439, miniskirt)
(n03787032, mortarboard)
(n83788365, mosquito net)
(n03792972, mountain tent)
(n03793489, mouse)
(n03803284, muzzle)
(n03814639, neck brace)
(n03825788, nipple)
(n03884397, panpipe)
(n03891251, park bench)
(n83920288, Petri dish)
(n03935335, piggy bank)
(n83937543, pill bottle)
(n83938244, pillow)
(n83958227, plastic bag)
(n03961711, plate rack)
(n83982430, pool table)
(n84023962, punching bag)
(n84033995, quilt)
(n04136333, sarong)
(n84162706, seat belt)
(n04235869, sleeping bag)
(n84296562, stage)
(n04317175, stethoscope)
(n84336792, stretcher)
(n04371430, swimming trunks)
(n84371774, swing)
(n04399382, teddy)
(n84418357, theater curtain)
(n04447861, toilet seat)
(n84476259, tray)
(n04507155, umbrella)
(n84584267, wig)
(n84592741, wing)
(n86794110, street sign)
(n06874185, traffic light)
(n87615774, ice lolly)
(n07753592, banana)
(n87802826, hay)
(n89229709, bubble)
(n89835506, ballplayer)
(n10148035, groom)
(n10565667, scuba diver)
(n11879895, rapeseed)
(n11939491, daisy)
(n12057211, yellow lady's slipper)
60


Reviews:
[{"review_id": "frDqLdXuxL", "reviewer": "Reviewer_GgcH", "summary": "The paper proposes an automatic error classification pipeline for ImageNet models. In particular, they categorize mistakes into overlapping classes, multi-object images, fine-grained and fine-grained with out-of-vocabulary (OOV), non-prototypical, the examples influenced by spurious correlations and unexplained mistakes. To detect fine-grained OOV errors the authors used WordNet and an open world CLIP classifier. They evaluated 121 models and showed that better performing models (in terms top-1 accuracy or multi-label accuracy) have a lower _portion_ of unexplained errors, and higher portions of errors which are explained by class ambiguity and fine-grained cases. They analyzed error portion trends separately for objects and organisms ImageNet classes, and analyzed the effect of training dataset and model architecture.", "questions": "1. Line 210: “we manually reviewed all 1000 ImageNet classes and defined semantically similar superclasses guided by the question of whether an untrained human could reasonably confuse them”\nWhy did you choose to manually review and group classes instead of using WordNet tree?\nThis also doesn’t match well with the claims about the error classification pipeline being automatic.\n\n2. What are the examples of spurious correlations that you identified in Section 3.6? How are they related to the findings from prior works studying spurious correlations in ImageNet like e.g. Salient ImageNet [3] and Hard ImageNet [4]?\n\n3. Line 370: “Arguably, we observe vision transformers achieving lower model failure rates on artifacts at very high MLA when compared to ConvNeXts.”\nDoes this observation hold even when we condition on the size of pre-training dataset?\n\n**References**\n\n\n[3] Singla, Sahil, and Soheil Feizi. \"Salient ImageNet: How to discover spurious features in Deep Learning?.\" \n\n[4] Moayeri, Mazda, Sahil Singla, and Soheil Feizi. \"Hard imagenet: Segmentations for objects with strong spurious cues.\"", "limitations": "The authors adequately addressed the limitations.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "While prior works studied questions similar to this paper such as [1] analyzed error types of one or a few high performing ImageNet classifiers such as ViT-3B, and [2] looked at how multi-label accuracy improves with model scale, this paper studies a more detailed question of how the error type distribution changes as we scale up the models in terms of the architecture size and pre-training dataset.\nThey also propose an automatic way to identify fine-rained out-of-vocabulary mistakes which relies on an external open world classifier model CLIP (assuming that we filtered out overlapping and fine-grained classes).\n\n**References**\n\n[1] Vasudevan, Vijay, et al. \"When does dough become a bagel? analyzing the remaining mistakes on imagenet.\" \n\n[2] Beyer, Lucas, et al. \"Are we done with imagenet?.\"", "weaknesses": "1. **Limited novelty.**\n\nGiven the results from the prior works that study label mistakes and ambiguity in ImageNet dataset like [1] and [2], the novel insights from this paper seem limited.\nI would suggest that the authors explicitly emphasize that the error categorization and definitions as in Figure 1 and most subsections in Section 3 (except for novel approach for fine-grained OOV) were proposed in prior works, and that they separate the background and results from prior works and their novel approaches and insights. \n\nIn particular, both [1] and [2] categorize mistakes in a similar fashion, and the categorization in [2] is the one this paper relies on. [1] focus on analyzing mistakes of a few large-scale models like ViT-3B and report the percentage of different error types and their severity in their Section 4.1 Table 2. Similarly to this paper they report mistakes separately for objects and organisms classes from ImageNet. They also propose ImageNet-M — a subset of only 68 examples where high-performing models still consistently make major mistakes.\n[2] also discuss a similar error categorization but they focus on analyzing how scaling up models affects top-1 and multi-label accuracy. \nThis paper combines two types of analysis from [1] and [2] and studies how scaling up models affects the error types distribution, and concludes that for stronger performing models the percentage of major mistakes significantly decreases. While this experiment and analysis is new, the conclusions are somewhat straightforward given the results from [1] and [2]? \n\nCould the authors please list the new observations and conclusions that were made in their analysis and that were not known in prior work?\n\n\n\n2. **The extent to which the pipeline of error type classification is automatic seems to be overclaimed.**\nThe introduction and abstract of the work make an impression that the paper proposes a fully automatic way to classify model’s mistakes, without relying on manual annotation unlike prior works [1, 2].\nHowever, after reading Section 3 it turns out that class overlap (Sec 3.1), multi-object cases (Sec 3.2), and spurious correlations (Sec 3.6) rely on manually obtained labels from [1] and [2], the definition of non-prototypical examples (Sec 3.5) relies on the ViT model’s mistakes which were analyzed in [1], and fine-grained errors (Sec 3.3) rely on manual grouping of classes into super-classes (see another comment re this case in Questions).\nThe only case of automatic error categorization is the fine-grained out-of-vocabulary error in Section 3.4 which involves WordNet tree and open world CLIP classifier. \n I suggest that the authors adjust their claims accordingly since such a pipeline can’t be called automatic.\n\nIn particular, if the proposed error classification pipeline was indeed automatic and didn’t have limitations discussed in lines 45-47 in terms of relying on human labellers, how would one apply the proposed pipeline to another large-scale dataset where we don’t have these labels from [1] and [2]? If it is possible, it would be helpful to see the application of such error classification to another dataset.\n\n\n3. **The design choices for detection of fine-grained out-of-vocabulary mistakes is not fully clear.**\nThe only automatic and novel part of the error classification pipeline is detection of fine-grained OOV mistakes: (1) first we find 10 most visually similar images in ImageNet train set in terms of the CLIP embedding, (2) check that at least one label belongs to the same superclass as model’s prediction, and (3) measure CLIP similarity with the correct class and WordNet in-vocabulary and OOV sibling classes related to both correct class and model's prediction.\n\nFirst, for the last step the description of the label proposal set is a bit confusing, and it would be helpful if the authors could clarify the exact set of proposals they are considering.\nSecond, it would be helpful to have more examples like in Figure 10 to illustrate step-by-step fine-grained OOV mistakes identification and show for each example what were the 10 closest images in embeddings space and which labels they had, and then which label proposal set CLIP scored (in particular, where those proposals were taken from and what were the CLIP scores exactly). It would be helpful to have more intuition for each step of this process.  \n\n4. [Minor] While the paper is generally easy to follow, I highlight a few points below which were not as clear:\n\n-  For Figures like 3, 5 and other trends, please clearly define what exactly is shown on y axis. I am assuming it is (# of mistakes of type X ) / (# of all mistakes) for each model.\n- On the same Figures, does each marker correspond to one model? Is it correct that each model is shown twice: once for objects and once for organisms?\n\n- Please fix captions for Figures 7, 12 and 14.\n\n\nTypos:\n\n- Line 54: and -> an\n- Line 99: find\n- Line 113: need to\n- Line 189: the\n\n**References**\n\n[1] Vasudevan, Vijay, et al. \"When does dough become a bagel? analyzing the remaining mistakes on imagenet.\" \n\n[2] Beyer, Lucas, et al. \"Are we done with imagenet?.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer $\\RG$ for their detailed review, insightful feedback, helpful suggestions, and interesting questions. We will fix the mentioned typos and update the Figure captions to provide further clarity. Below we answer the reviewer’s remaining questions.\n\n**Q: Can you highlight the main novelties of this work given prior investigations?**  \nYes, please see our reply to Q1 in our General Response.\n\n**Q: Is the error classification pipeline truly automatic? How would it be adapted to other large-scale datasets?**  \nOur pipeline is automated in the sense that it can be continuously run on a large number of models, where previous approaches required the manual review of each new incorrect model prediction, which is infeasible for all but the most accurate models. This automatization even for relatively inaccurate models is what enables us to study how error distributions evolve, over meaningful ranges of (top-1) error rates.\n\nFor a discussion of how to adapt it to other datasets, please see Q3 in our General Response.\n\n**Q: Can you clarify the exact design choices for fine-grained OOV mistakes?**  \nIntuitively, we consider an error to be a fine-grained OOV mistake if it satisfies two requirements: first, the object that the model aims to classify is not included in the ImageNet label set, i.e., it is indeed out-of-vocabulary, and second, for this object, the predicted class is (only) a fine-grained error. To this end, we first retrieve images that show visually similar objects and check if these belong to the same superclass as the predicted label, thus indicating that the prediction is only a fine-grained error. To confirm that the true label is indeed OOV, we collect a set of label proposals (discussed in detail below) and check whether an open-world classifier prefers a label not in the ImageNet label set. We collect this set of label proposals as the union of the following: All labels in the same superclass as the model’s prediction, all direct WordNet siblings of the model’s prediction, and all WordNet ancestors of the model’s prediction up to but excluding the first common ancestor with the ImageNet label. Intuitively, this provides the open-world model the choice between the most closely related in-vocabulary classes (same superclass) and out-of-vocabulary classes (direct siblings and ancestors).\n\n**Q: In Figures 3 & 5 what is shown on y axis? Is it (# of mistakes of type X ) / (# of all mistakes) for each model?**  \nThe proportions are computed individually for each group, i.e., for a given model Y and group Z (organisms or artifacts) it shows the (# of mistakes of type X for model Y with an ImageNet label belonging to group Z)/(# of mistakes for model Y with an ImageNet multi-label belonging to group Z). We will clarify this.\n\n**Q: In Figures 3 & 5, does each marker correspond to one model? Is each model shown twice, once for objects and once for organisms?**  \nYes, each model is shown with one marker per group. We will clarify this.\n\n**Q: What are the examples of spurious correlations that you identified in Section 3.6? How do they relate to SalientImaget [2] and HardImageNet [3]?**  \nAs we outline in L312-317, we base our labels on Beyer et al. [1] and we obtain 965 pairs of classes that may be spurious correlations. Out of these 965 class pairs we observed 749 as actual spurious correlation errors and have included example images in the PDF attached to the main response.\nWe thank the reviewer for highlighting the works on SalientImaget [2] and HardImageNet [3] and are happy to discuss them in our related work section. For each class in ImageNet, Singla and Feizi [2] collect 5 salient or core (visual) features. We checked all 965 of our co-occurrence pairs, considered candidates for spurious correlation, and found 301 contain two classes that share at least one salient visual feature. This substantial overlap has an almost 0 chance of occurring at random (less than $10^{-200}$), showcasing the alignment between the two methods. Further, we will extend our discussion on the detection of spurious correlations by considering overlapping core features as a criterion. HardImageNet [3] extends the analysis of Singla and Feizi [2] by considering the 15 class pairs with the strongest spurious features and turning them into a segmentation task.\n\n**Q: Do vision transformers outperform ConvNeXts even when conditioned on the size of the pre-training datasets?**  \nConditioning on the same multi-label accuracy and same pre-training dataset, we observe that ViTs exhibit significantly fewer severe model failures on artifacts than ConvNeXts. Interestingly this is not the case for organisms. There, ConvNeXts perform either similarly or better than ViTs, sometimes even outperforming ViTs pretrained on a larger corpus (at slightly lower MLAs). We speculate that this might be due to the ConvNeXts convolutional backbone leveraging textures more heavily [4], which could be particularly helpful for organisms, while ViTs are better at dealing with the clutter often present in artifact scenes. We believe this to be an interesting example of the type of trends that can only be discovered using a large-scale, automated analysis such as ours.\n\n**References**  \n[1] Beyer et al. \"Are we done with imagenet?.\" arXiv  \n[2] Singla and Feizi. \"Salient ImageNet: How to discover spurious features in Deep Learning?.\"  \n[3] Moayeri, Mazda, Sahil Singla, and Soheil Feizi. \"Hard imagenet: Segmentations for objects with strong spurious cues\"  \n[4] ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. Geirhos et al., ICLR 2019\n"}, {"review_id": "eTRIjd8xRs", "reviewer": "Reviewer_rfV8", "summary": "This paper aims to create an automated pipeline for defining the types of errors made by ML models on ImageNet. Error categories are from previous work, and the paper compares the results between the previous human based pipeline and the newly automated version. The authors use the pipeline to evaluate over 100 models, and provide some analysis on the findings.", "questions": "See weaknesses", "limitations": "I believe that limitations are addressed adequately", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Paper structure is clear\n- Design decisions for automating each step of the pipeline make sense\n- Results are validated and put in context with prior work", "weaknesses": "- Analysis of results seems limited. I would like to see additional analysis that builds upon the benefits of such pipelines. For example, if we can now easily get to severe model errors, can we do human analysis on these to create categories or better understand them?\n- I would like to better understand the motivation behind this work, such as potential use cases for the pipeline to improve machine learning research", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer $\\Rr$ for their review and interesting questions, which we answer below.\n\n**Q: Can you provide an in-depth analysis of the results obtained using your automated pipeline, e.g., by manually categorizing model failures?**  \nWhile we include some more high-level observations (e.g. the differences in error distribution trends for artifact vs. organism classes) and some more detailed observations (e.g., ViTs suffering from significantly fewer model failures than ConvNeXts on artifacts conditioned on the same multi-label accuracy), we believe a more detailed and especially manual analysis to be an interesting item for future work and out of scope here given the focus on the automation of the analysis.\n\n**Q: Can you provide a more in-depth motivation for this work, in particular how it can be used to improve machine learning?**  \nThe first important motivation for our work was to put recent methodological advances in computer vision into context. Indeed, we observe that the *portion* of errors that we categorize as severe model failures decreases with both top-1 and multi-label accuracy, suggesting that these metrics not only remain valuable measures for progress, in contrast to recent suggestions that they mostly improve by overfitting on labeling biases but even under-report this progress. Second, we believe that trends in error distributions such as the dominance of fine-grained misclassifications for organisms or the prevalence of fine-grained out-of-vocabulary errors for artifacts, can help inform methodological research.\nFinally, our automated fine-grained error evaluation could allow developers of novel training techniques to quickly assess the kind and distribution of errors their models still make, allowing interventions targeted at specific types to be evaluated more effectively.\n"}, {"review_id": "dDOwxoJgn1", "reviewer": "Reviewer_HYyY", "summary": "The paper introduces an automated pipeline to classify ImageNet's errors from 4 categories: (i) fine-grained categories; (ii) fine-grained OOV; (iii) Non-prototypical instances; and (iv) Spurious correlations. The main contribution of the paper is this automated pipeline with which various errors can be categorized.  While there exists a plethora of recent works in classifying ImageNet errors, the paper does a good job of summarizing all these errors and also designing an automated pipeline to detect them for ImageNet. \n", "questions": "See Weaknesses", "limitations": "- While the paper provides a good overview of ImageNet errors and provides an automated pipeline to explain a subset of those errors, the analysis is still restricted to ImageNet and is not extended to other ImageNet test sets where the drop in accuracy is more severe. \n- Although the paper tests a large number of models, each component (e.g., pre-training strategy, augmentations) is not well dissected deeply. \n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Good overview of ImageNet errors and a simple, but effective automated pipeline for error detection.\n- A wide-range of models are tested which is a strong point for the paper. ", "weaknesses": "- The analysis is restricted to only analysing ImageNet errors. While it is a good direction to inform vision research -- it will be beneficial to extend the framework to understand error types in other ImageNet variants such as ImageNet-A. What are the error distributions in such sets? The current analysis though good, is restrictive.\n- The paper does not discuss recent works such as ImageNet-X (https://facebookresearch.github.io/imagenetx/site/home)  which also looks at understanding model failures. While the categories considered in this paper is a subset of ImageNet-X error types, can the authors compute an overlap metric between some of the error types detected in this paper and ImageNet-X? \n- While a range of models are tested, can the authors distinguish between the pre-training strategies more in the plots? E.g., highlight SSL methods, Vision-Language pre-training methods and Supervised Pre-training methods using different colors / notations. This would make the paper stronger.\n- Can the authors give some intuitions on the unexplained failures? What are the potential causes? \nHow can the error analysis learnings be useful in improving model training? Can the authors provide some intuition on this? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer $\\RH$ for their insightful feedback, helpful suggestions, and interesting questions which we address below.\n\n**Q: Can the proposed method be extended to ImageNet variants, such as ImageNet-A. What are the error distributions in such sets?**  \nIn principle, our pipeline can be extended to ImageNet-A. However, this would require multi-object annotations. To the best of our knowledge, no such annotations exist. While collecting such labels in general (see our response to $\\RG$ for details) could be automated via the Re-Label approach, we believe on ImageNet-A this would be not fruitful as the images were specifically selected to be challenging (wrong or low-confidence). \n\n**Q: Are the error types considered in this paper a subset of ImageNet-X error types? If so, how do they map to each other?**  \nThe types of errors considered in ImageNet-X are mostly entirely orthogonal to the types of errors considered by us. Their works focus on confounders that make a particular image challenging such as an atypical pose, background, and lighting, while we focus more on systematic errors caused (partially) by labeling (set) choices and aggregate effects such as atypical pose, background, etc. into the non-prototypical instance category. Similarly, their broad subclass confounder, could, depending on context, be considered class overlap, OOV, or fine-grained misclassification errors in our categorization.\n\n**Q: Can you distinguish different pre-training strategies in your plots and analysis?**  \nGreat suggestion. Since the original submission, we have collected over 800 additional models and more detailed annotations including the exact pre-training method used. We are happy to update our plots correspondingly and include a more detailed analysis in an updated version of our paper.\nWe include an updated plot corresponding to Figure 16 in the PDF attached to the main response. Generally, we observe that trends remain dominated by MLA and top-1 accuracy with the best-performing models all pretrained using self-supervised learning.\n\n**Q: Can you provide some intuitions on the unexplained failures?**  \nUnfortunately, we can not give general intuitions on the unexplained failures, as their name implies. These are model failures that we believe no human (aware of the ImageNet label set) would make. Further, while better (more top-1 or MLA-accurate) models make fewer such errors, they do not follow a strict set inclusion, making an inherent hardness to these images that is not apparent to humans unlikely. \n\n**Q: How can this analysis be used to improve model training?**  \nPlease see our answer in the main response."}, {"review_id": "UUgCTPkP70", "reviewer": "Reviewer_kbqQ", "summary": "The paper introduces an automated analysis on the model errors in ImageNet.\nThe analysis sequentially categorizes into six types of errors with increasing\ncriticality, unveiling the true model failures. The results are contextualized\nwith previous work.", "questions": "- While not critical, I am wondering why the paper did mention to use the\n  WordNet parent classes for the super classes. A single sentence on the\n  fitness of the parent classes as super classes would we interesting.", "limitations": "The paper addresses its major limitations well.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This work is of high interest to the computer vision and machine learning\n  community, given the extensive use of ImageNet as a benchmark.\n\n- The quality and presentation of the analysis is excellent.\n\n- The full code to conduct the pipeline presented in the paper is provided with\n  the paper, which is extremely value considering the reproducability under the\n  complexity of the analysis (wrt. the number of models, etc.). Although the\n  analysis may seem very large-scale, it is surprisingly affordable and, with\n  the code, is very easily reproducable.\n\n- The analysis builds upon and refines the previous NeurIPS's work by Vasudevan\n  et al..\n\n- The sequential categorization, starting with the least critical category of\n  error, is a very good idea.", "weaknesses": "- I could not find any major weakness of the paper that would\n  impede its acceptance in the current state.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the kind review. We are encouraged that you see our work as a valuable contribution to the community. Regarding your question about using the WordNet hierarchy to form superclasses, we refer you to the general response and are happy to answer any further questions."}, {"review_id": "TvYHAALppP", "reviewer": "Reviewer_hvnS", "summary": "This paper proposes a pipeline to automatically categorize ImageNet classification misclassifications. Previous work [1] defined and categorized those misclassifications into 4 types manually and this is the main motivation of automating the process. The results show that their pipeline has an accuracy of 60% based on human annotations of model misclassifications.\n\n\n[1] analyzing the remaining mistakes on imagenet\n\nI have read the author’s rebuttal and adjust the rating accordingly!", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "The paper is easy to follow as they mostly based on the previous work [1].", "weaknesses": "- The novelty of this paper is minimal and the motivation is not strongly supported. For example, the definitions and categorization of misclassifications have been done before in [1]. In writing, I do not see the reason why we need such categorization of misclassifications (i.e. the motivation). In Related work where I expect to see the motivation, the authors did not contrast their work with the literature then it is hard for me to see the contribution. I believe Related work can be greatly improved. Automating [1] is not considered novel at NeurIPS standard in my opinion.\n- The authors only tested on 1 model and 1 dataset. It does not make the findings generalize. Another problem is the number of samples being examined. Based on Table 1, only 378 were tested!!!\n- It looks like the authors are trying to do 4-way classification. In this case, it makes more sense to build a model to do that rather than having different methods for different misclassifications. The four types of misclassifications can be easily collected or synthesized.\n- The authors only compared with [1]. The reviewer encourage them to extend the comparisons to others (e.g. [2]) and with other datasets and classifiers.\n\n[2] ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer $\\Rh$ for their review and answer their questions below.\n\n**Q: Can you discuss the motivation for your method? Where is the novelty in automating the categorization proposed by [1]?**  \nPlease see our general response for a more detailed motivation. Further, we believe the large number of works investigating ImageNet errors to be a testament to the exceptional interest in this area. While prior manual analyses such as Vasudevan et al.’s [1] can achieve very high precision, they have several disadvantages:  a) it is time-consuming even for precise models and intractable for imprecise ones, b) it requires a panel of expert reviewers, typically not available, and c) introduces inconsistencies due to human expert error. We alleviate all of these with our novel and effective automated categorization pipeline (see line 42 ff).  \n\n**Q: Do you only evaluate a single model on 378 samples?**  \n*No*, in fact, we evaluate *121* different models (trained classifiers), and always consider all images from the ImageNet validation set with multi-label annotations [1,2] (over 18.000). Perhaps the reviewer is referring to ViT-3B’s 378 multi-label errors, manually classified by Vasudevan et al. [1] and thus used by us to validate our automated approach in Table 1.\n\n**Q: Why don’t you simply train a classification model to classify the different error types?**  \nFirstly, prior to our automated error categorization pipeline, only 548 labeled misclassifications from Vasudevan et al. [1] were available. Given that, the error classification depends not only on the image itself but also on the original label and model prediction (1.000.000 combinations for ImageNet), that would be an extremely small data set. Secondly, our pipeline, by its very nature, only considers images where the analyzed model failed, leaving a particularly hard subset of (training) images for any neural classifier. Finally, even if we could train such a classifier, it would suffer from the same lack of interpretability as the original model, making it ill-suited for an evaluation pipeline aiming to understand model errors.\n\n**Q: Could you compare with the ImageNet-Hard dataset and evaluate on other datasets and classifiers?**  \nSee our general response with regards to ImageNet-Hard and other datasets in general. As for other classifiers see the above response. \n\n\nThese questions lead us to believe that reviewer $\\Rh$ misunderstood the setting and high-level goal of our paper. While we will aim to improve the presentation to avoid such misunderstandings, we respectfully ask the reviewer to reconsider their review and acceptance score in light of these clarifications.\n\n**References**  \n[1] Vasudevan et al. \"When does dough become a bagel? Analyzing the remaining mistakes on ImageNet\", NeurIPS’22  \n[2] Shankar et al. \"Evaluating machine accuracy on imagenet.\" ICML’20.\n"}]

Author rebuttal:
$\newcommand{RG}{\textcolor{green}{GgcH}}$
$\newcommand{Rr}{\textcolor{blue}{rfV8}}$
$\newcommand{RH}{\textcolor{purple}{HYyY}}$
$\newcommand{Rk}{\textcolor{orange}{kbqQ}}$
$\newcommand{Rh}{\textcolor{teal}{hvnS}}$

We thank all reviewers for their work and are delighted that they appreciate the structure and quality of the presentation ($\Rr$, $\Rh$, $\Rk$), connection to prior work ($\Rr$,$\RH$,$\Rk$) and design choices in building the automatic pipeline ($\Rr$,$\RH$,$\Rk$). Here we address questions raised by multiple reviewers and reply to individual concerns below.

**Q1: Can you highlight the main novelties of this work? ($\RG$, $\Rh$)**  
The novelty and contribution of our work lies mostly in the automatization of the analysis done by Vasudevan et al. and the large-scale analysis enabled by this automatization. More concretely, our automated analysis of over 100 models (extended to over 900 by the time of this rebuttal) allows us to make much more general and well-supported observations than the manual or much coarser investigations performed before:
* While prior works have investigated the relation of multi-label (MLA) and top-1 accuracies, to the best of our knowledge, we are the first to go beyond this and perform a large-scale study of *the distributions of the different error types*.
* Organism and artifact classes exhibit significantly different trends in terms of error distributions. In particular, the main source of errors for organism classes are fine-grained classification mistakes, while artifacts suffer much more from spurious correlations and out-of-vocabulary errors. While this observation agrees well with our intuition, we believe we are the first to confirm it empirically. 
* The *portion* of errors that we categorize as severe model failures decreases significantly with increasing top-1 and multi-label accuracy, suggesting that even top-1 accuracy remains a meaningful measurement of model improvement and recent progress might in fact have been under- instead of over-reported.

**Q2: Why did you manually review and group classes into superclasses (L210) instead of using WordNet tree? ($\RG$, $\Rk$)**  
Our superclasses were designed to group classes that humans perceive to be very similar to each other such that, e.g., an untrained human could easily confuse them. The ImageNet labels are nodes from WordNet which is itself organized in a hierarchy of synonym sets (synsets), each expressing a distinct concept. The WordNet hierarchy implies a natural subset relationship between the synsets which indeed forms the basis for our superclass organization. However, this hierarchy is suboptimal for our purposes.
For example, the direct WordNet parent of ``cock`` and ``hen`` is ``bird``, which has 59 ImageNet labels as successors, including small birds, birds of prey, parrots, aquatic birds, etc., which are substantially different from ``cock`` and ``hen``. In contrast, we manually grouped ``cock`` and ``hen`` together with ``gallinaceous birds``, resulting in a more refined superset with just 9 classes. Further, some visually similar ImageNet classes have large very broad super classes, such as ``physical entity``.

**Q3: How would the error analysis pipeline be adapted to other large-scale datasets (where multi-label or re-annoations are not present)? ($\RG$, $\Rh$)**  
We believe extending this work to new datasets as outlines below, to be an interesting item for future work. While less precise than using a human annotated gold-standard, multi-labels could be generated using a Re-Label [1] approach, allowing (given sufficient scale) spurious correlation pairs to be extracted from co-occurrence frequencies. After defining superclasses manually as required by their very definition, this would allow all error types except for the rare non-prototypical instances to be categorized.

**References**  
[1] Re-Labeling ImageNet: From Single to Multi-Labels, From Global to Localized Labels. Yun et al., CVPR 2021


Meta-review: The paper proposes an automatic error classification pipeline for ImageNet models, they break down the various error types into overlapping classes, multi-object images, fine-grained and fine-grained with out-of-vocabulary (OOV), non-prototypical, the examples influenced by spurious correlations and unexplained mistakes. While the reviewers do cite that there are other works that do things in this direction and the impact could be limited since this is one dataset, the reviewers generally agree the contribution is valuable and the experimental work is thorough. Thus the decision will be to accept.
Decision: Accept (poster)