When does dough become a bagel? Analyzing the remaining mistakes on ImageNet
Authors: Vijay Vasudevan
Benjamin Caine
Raphael Gontijo-Lopes
Sara Fridovich-Keil
Rebecca Roelofs
Venue: NeurIPS 2022 Accept (2022)
Keywords: ImageNet
image classification
multi-label
benchmarking

Abstract: Image classification accuracy on the ImageNet dataset has been a barometer for progress in computer vision over the last decade. Several recent papers have questioned the degree to which the benchmark remains useful to the community, yet innovations continue to contribute gains to performance, with today's largest models achieving 90%+ top-1 accuracy. To help contextualize progress on ImageNet and provide a more meaningful evaluation for today's state-of-the-art models, we manually review and categorize every remaining mistake that a few top models make in order to provide insight into the long-tail of errors on one of the most benchmarked datasets in computer vision. We focus on the multi-label subset evaluation of ImageNet, where today's best models achieve upwards of 97% top-1 accuracy. Our analysis reveals that nearly half of the supposed mistakes are not mistakes at all, and we uncover new valid multi-labels, demonstrating that, without careful review, we are significantly underestimating the performance of these models. On the other hand, we also find that today's best models still make a significant number of mistakes (40%) that are obviously wrong to human reviewers. To calibrate future progress on ImageNet, we provide an updated multi-label evaluation set, and we curate ImageNet-Major: a 68-example "major error" slice of the obvious mistakes made by today's top models -- a slice where models should achieve near perfection, but today are far from doing so.


When does dough become a bagel? Analyzing the remaining mistakes on ImageNet
Vijay Vasudevan, Benjamin Caine, Raphael Gontijo-Lopes, Sara Fridovich-Keil's Rebecca Roelofs
{vrV, rofls) com Google Research, Brain Team, UUiivessity of California, Berkeley
Abstract
Image classification accuracy on the ImageNet dataset has been barometer for progress in computer vision over the last decade. Several recent papers have questioned the degree to which the benchmark remains useful the community [3] 36], yet innovations continue to contribute gains to performance, with today's s largest models achieving 90%+ top-1 accuracy. To help contextualize progress on ImageNet and provide more meaningful evaluation for today's ss state- of-the-art models, we manually review and categorize every remaining mistake that few top models make and provide insights into the long-tail of errors on one of the most henchmarked datasets in computer vision. We focus on the multi-label subset evaluation of ImageNet, where today's 's best models achieve upwards of 97% top-1 accuracy. Our analysis reveals that nearly half of the supposed mistakes are not mistakes at all, and we uncover new valid multi-labels, demonstrating that, without careful review, we are significantly underestimating the performance of these models. On the other hand, also find that today's 's best models still make significant number mistakes (40%) that are obviously wrong to human reviewers. To calibrate future progress ImageNet. a 68-example "major error"
3
B1
42
provide updated multi-
label evaluation set, and we curate ImageNet- slice of the obvious mistakes made by today's top models slice where models should achieve near perfection, but today are far from doing so.
Introduction
Computer vision models often evaluate their performance on the ImageNet classification dataset [5] 30
and many variants (2 12 I iI 38, as
signal of capability for visual understanding. As performance on the standard sets have reached diminishing returns to top-1 and top-5 accuracy, much
[7] has focused on understanding what is left for the computer vision community to solve, and where the community should be driving toward. Prior studies of ImageNet errors have identified issues stemming from lack multi-labels, label noise, under-specified classes, and more 33
recent work 33
3 29
36
Label: dough; Model: bagel. When does dough become a bagel?
36 I8].
Label errors and label noise affect the evaluation of any model ImageNet is no exception. Many studies above have spent effort to correct and improve these labels, showing that while ImageNet performance improvements are approaching diminishing returns, the dataset continue to shepherd As the best models improve, however, it becoming increasingly challenging to assess the often novel predictions these models make. For example, should we penalize models for
2.5 and
remain useful to the community, but only if we collectively
Dataset and analysis available at http://///ttaaaaooooooo////e//ee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

being the first to predict that a pre- pre-baked bagel may be bagel, as one of the models we review in this work does?
Machine learning models tend to make mistakes with varying severity and importance (a function of both the prediction well as the label definitions), and previous studies R3 3 on ImageNet have shown that non-experts find it challenging to determine the correctness of a model's 's prediction. Our own experience, highlighted by the doughy-bagel, is that many of the remaining mistakes these top models make are quite reasonable and probably should not be considered mistakes unnduksksnnundrrttdddnnnnnnn the severity and type of these remaining mistakes can help calibrate our barometer of progress.
Indeed, to our knowledge, there has not been an expert-review, categorization, and severity assessment of the remaining long-tailed mistakes, which becomes particularly important at these margins. Our experience working with production teams on deployed applications has suggested that manual triage and assessing individual failures provides useful indicator of model performance that aggregate measures can fail to capture. Thus, in this work we attempt to analyze (as expert reviewers) every remaining mistake that few state-of-the-ar models make better understand (a) which of the remaining mistakes remain egregious errors, (b) what error category they might fall in, and (c) what evaluations might capture the most important long-tail failures that remain.
In this paper we analyze the ImageNet multi-label validation subsets [31,, in which expert labelers were used to assess the correctness of model predictions through the year 2020, and on which 1000-image human-evaluutee subset provides a direct comparison to expert human performance. By analyzing the mistakes of two large 2022-era ImageNet models, found that:
Nearly half of each model's mistakes were deemed correct under a careful, expert multi- label re-evaluation, halving the error rate. Had we not analyzed the models' mistakes, we would be severely underestimating the models' actual performance.
Approximately 40% of the remaining mistakes can be classified as 'major' errors: errors that most humans would likely not make, suggesting that many of the taigetilldd mistakes aren't 't simply label noise, but legitimate mistakes that leave room for improvement.
What do these lessons portend for the future of ImageNet evaluation? Top-1 will become increasingly noisy as our best models get better (though we have not yet completely saturated top-1). Our work shows that multi-label accuracy, while better at capturing "true" errors compared to top-1, suffers from a lack of a comprehensive, accurately labeled large evaluation set, which expensive to procure and challenging to maintain.
We therefore propose ImageNet-M, 68-example evaluation split of the multi-label evaluation set composed of "major" mistakes that several top-performing models make; we believe this subset is one on which future image classification models can achieve near perfect accuracy, and provides three clear benefits: (1) we attempt to comprchensively-labee all examples for multi-label annotations to prevent the need review novel correct predictions, (2) we endeavor to maintain and provide way for the public to add new correct predictions; (3) the evaluation set is small enough to encourage completeness and allow the community to inspect their own errors.
2 Related Work
Multi-label annotations on ImageNet. As models continue to improve ImageNet top-1 accuracy, there has been an increased interest in evaluating ImageNet multi-label accuracy 33
3
31
42
36
Stock et al. (33] use non-expert human studies and explanations (e.g., model criticisms) on predictions
of SOTA model on ImageNet, finding that machine accuracy underestimated and advocating for multi-label evaluation. Beyer et. al. [3] introduce a set of Reassessed Labels (ReaL) for the ImageNet validation set containing multi-labed annotations. The researchers first collected proposal labels from model predictions using a testbed of 19 models, and then, in order to reduce the number of predictions to review, they narrowed the set of models down to the models that had the highest precision and recall above 97% % on a set of 256 images reviewed by 5 experts. The top predictions from these 6 models were then reviewed by human annotators from a crowdsourcing platform. In a similar vein, Shankar et. al. 31 provide multi-label annotation dataset for 20,000 of the 50,000 ImageNet validation images and find that roughly 20% of images have more than one valid label. To generate the annotations, the researchers first collected predictions from testbed of 70 models for

---

each image and then reviewed the unique model predictions using panel of three human experts. Additionally, human accuracy was evaluated on a subset of 1,000 images and a panel of 5 human experts reviewed human and model predictions on this subset. Tsipras et al. (36) also find that 20% of images in the validation set contain objects from multiple classes, and identify sources of ambiguous label classes. Hooker et. al [14] use human evaluations to label a subset of examples from ImageNet, finding that they contained multiple labels 40-60% of the time. More recently, Yun et al.
42 obtained pixel-level multi-label ground truths for the ImageNet training set using machine annotator, and found that training a model with the dense annotations leads to small improvements in both t 1 and multi-label ImageNet accuracy.
Our paper focuses on (and adds to) the multi-label evaluation of ImageNet using expert labelers, updating the dataset to collapse classes that are overlapping or subset relationships to better capture the remaining real errors. We adopt the labeling methodology of Shankar et al. [31] and use a panel of human experts to determine the validity of novel predictions. In contrast to prior work, we re-visit all remaining model mistakes using human expert review, and categorize them by type and severity.
ImageNet Label Error. Multi-label annotation datasets for ImageNet (including our updated anno- tations) identify set of images that have no correct ground truth label (i.e. label errors). Van Horn et. al (37] used experts from the Cornell Lab of Ornihhlogy to estimate that at least 4% of the bird images are misclassified, and more recently Northcutt et. al [25] used MTURK workers to review algorithmically identified potential errors and found a label error rate of 5.83% in ImageNet. Lee et. al [8] sample 400 ImageNet mistakes and perform a categorization, similarly finding significant label error and label ambiguity.
Mistake analysis. Recent work on analyzing model mistakes on ImageNet has mainly focused on similarities or differences between mistake sets of independently trained classifiers. Mania et al. [20] found that predictions of different models on ImageNet are more similar than one would expect if the models were making mistakes independently. Geirhos et al. [9] studied a 16-class ImageNet classification task and similarly found that CNNs make remarkably consistent mistakes with one another but CNNs and humans have an error consistency that is only slightly above what can be expected from chance. In follow up work, Mania al. [2i] studied the dominance probabilities for pairs of ImageNet models, which capture the probability that a higher accuracy model will make a mistake on a particular image that a lower accuracy model correctly classifies. Their empirical analysis of dominance probabilities on ImageNet implies that the mistakes of higher accuracy models are typically subsets of the mistakes of lower accuracy models. However, Gontijo- Lopes et al. [TO] and Andreassen et al. [ found that training on larger and more diverse datasets as well zero-shot evaluation can lead to models with more mistake diversity, which in turn can be used to build more accurate ensembles. Similarly, Nguyen et al. (23 found systematic differences in the errors between wide and deep ResNets on ImageNet. Ciregan et al. 4 demonstrated that powerful models on MNIST allowed for remaining error analysis in the single-label context, highlighting the ambiguity in many of those remaining errors. More recently, ImageNet-9 [40) studied the influence of background on object recognition ImageNet, Salient ImageNet [32) annotated core and spurious features on ImageNet, and Domino 7 automatically detected systematic errors based on clustering.
In contrast to this prior work, we take step towards fully calibrating SOTA model evaluations by exhaustively and visually reviewing every remaining model mistake using manual expert review in the multi-label context; we believe the remaining errors identified to be legitimately incorrect and tempered by severity and category ratings that we hope prove useful to future evaluations.
3 Mistake analysis method and taxonomy
To obtain an initial set of mistakes remaining on ImageNet, we used standard ViT 66 model scaled to 3B parameters (ViT-3B) that was pre-trained on JFT-3B B4 and fine-tuned on ImageNet- 1K (3). achieving a t accuracy of 89.5% (details in Appendix ??). We also later review mistakes made by the Greedy Soups model (39] Using the imagenet2012 1t1220222mullaa mmultilln tilabel dataset [31], we measured the initial multi-label accurucy (MLA) of the ViT-3B model to 96.3%. The model made a total of 676 apparent mistakes, which we sought to manually review in detail. Exhaustively examining every mistake has been made more convenient and practical due to the quality of today's 's top models as well as the smaller subset of 20k validation images present in the multi-labe set.
3

---

Deemed Correct: 294
VIT3B MLA Mistakes: 676
ViT3B Major Mistakes: 155
After Novel Prediction Review: 536
ImageNet-Mr 68
After Categorization Review: 382
VTT3B Minor Mistakes: 227
Figure 1: Review flow: Starting with 676 mistakes, we employed two phases of mistake review, identifying 382 re-evaluated mistakes at the end. The categorization phase identified 155 major errors made by the model. ImageNet-M was then selected from these major errors via model-based selection (Section 5.1 294 of the original 676 mistakes were determined to be correct after review.
3.1 Panel Review
To exhaustively and accurately assess every remaining mistake, formed panel of five reviewers and followed process similar to [3I] to evaluate the predictions made by this model on the 676 mistakes avoided using non-expert crowd-source platforms specifically because the remaining mistakes are often difficult to assess by non-expert annotators [36 For every mistake, the panel determined: (1) Did the model make mistake? (2) Was the original ground truth annotation correct? (3) the model made a mistake, what is the category, type, and severity of the mistake? The dataset contains field for every image indicating which classes
imagenet22 220gennet201.... 1120012mmullaai awuuulllallit large previous suite of models predicted that were determined to be incorrect (wrong_ma multi labels). Of these 676 initial mistakes, 221 were novel: they not reviewed in the original multi-label annotation process since none of the models evaluated made the same prediction. Each member of the panel reviewed all 221 novel mistakes.
Similar to [1], we built a review tool that allowed each panelist to see a) the predicted class, b) the predicted top softmax score, c) the set of ground-truth labels, d) the set of previously incorrect labels, and e) the image. We also employed the labeling guide produced by the authors of [31] when investigating the definition of a class, and a tool to iterate through the images of every ImageNet validation example for that class, using the validation images to help define the class boundaries (rather than the gloss or lay definition of the class). See Appendix ?? for screenshots of the review tools. In addition, we collapsed small number classes for which previous work has identified exhibited extreme overlap (24), such 'missile' and 'projectile missile'. In Appendix ?? we provide these new collapsed class mappings.
We also used Google Image Search to help provide context to some assessments; in one interesting but not isolated case, a prediction of a taxi cab (with no obvious taxi cab indicators beyond yellow color) present in the image; determined the prediction to be correctly taxi cab and not just standard vehicle by identifying a landmark bridge in the background order to localize the city, and subsequent image search for taxis in that city yielded the images of the same taxi model and license plate design, validating the model's 's actually correct prediction.
Each panelist rated whether these novel mistakes had mislabeled ground truth, whether the prediction should be added to the set of correct, unclear, or wrong multi-labels. As a group, we reviewed any image where there was no unanimous agreement, allowing those in the minority to make their case and change minds, or highlight potential oversights. The use a panel and discussion was important: in non-trivial number of cases, a single panelist found unique evidence for against a prediction that no other panelist saw that led to a different outcome. After locking in final votes, we took the majority assessment, used "unclear" for tic. After this re-assessment, 140 of the novel predictions were deemed correct (or the ground truth deemed incorrect), leaving 536 remaining mistakes to assess.
3.2 Mistake severity and category
The remaining mistakes then comprised images that had either previously been deemed wrong by the panel in Shankar et al. (31], or deemed wrong by the current panel. We then began review of the

---

(a) Major mistake
(b) Minor mistake
(c) New multilabel Label: tape player
(d) Problematic
Label: dough
Label: kuvasz
Label: bee
Model: jigsaw puzzle
Model: Great Pyrenees
Model: cassette
Model: fly
Figure 2: Mistake Severity. Examples of the two mistake severities (a-b), a correct model prediction where the model identifies previously missing multi-lahel (c); and problematic example (d) where the label (bee) is incorrect (object a bee-fly, which is type of fly).
(c) Spurious correlation
(a) Fine-grained Label: wall clock
(b) Fine-grained w/ O0V
Label: mouse, desk, moni- (d) Non-prototypical
Label: syringe
tor, screen
Label: stove
Model: sundial
Model: hamster
Model: desktop computer Model: hamper
Figure Mistake Category. Examples of the four mistake categories. In the One-grained with OOV example, the animal is chinchilla, which is not an ImageNet class but is visually similar to hamster, which an ImageNet class. the spurious correlation example, the scene contains relevant context for desktop computer, but there is such object in the image.
mistakes for the category and severity of mistakes. During this second phase review, we re-reviewed all images in detail, potentially overturning decisions made by the previous panel, where they missed the presence of object that was in example the decision was inconsistent with the labeling guide or validation examples we relied on
Severity:
We assessed each mistake's s severity with the assumption that not all mistakes are equal: some mistakes are extremely borderline, particularly because the ImageNet class definitions are imprecise or because the image itself provides ambiguous or incomplete information. For any remaining mistake, we broke down the severity levels into major and minor mistakes. A major mistake is model prediction that human who understands the class definitions would find obviously incorrect. For example Figure 2(a) shows an example image where the prediction jigsaw puzzle but the label is dough; although the pieces somewhat wwwpww-uzzee-sstee
untrained humans more likely to classify this as dough than jigsaw puzzle. minor mistake is a model prediction that human who understands the class definitions would probably find to be incorrect, but in more subtle way than a major mistake. Some minor mistakes are so subtle that even expert-traiie trained humans might debate their correctness.
We recognize that these severities are subject to the influences of the worldview of the panelists (and provide the panel
the web) and should be judged accordingly (see Section 5.3). 3. For transparency, assessments the severity for all mistakes, and in Figure we provide some examples of the severities of various mistakes made by this model for the reader's calibration. In Appendix ?? we provide many additional examples of every severity and category.
Category: After reviewing many mistakes. we formulated four mistake types (Figure 3
"The labeling guide in Shankar et al. [31] was constructed after the initial panel review but prior to the human accuracy assessment, resulting in some validation labels that would be inconsistent with the labeling guide we used.
5

---

(1) Fine-grained errors are where the model makes mistake between two similar types of organisms or objects, one of which a groundtruth label. These mistakes often occur when the two confused classes are already similar (e.g. two dog breeds), or when either of them is very broad or ill-scoped (e.g. the "bake shop" class includes any baked good or bakery).
(2) F'ne-grriiee with out-of-vocabularyy there is an object in the image that is not in the ImageNet class hierarchy but is similar predicted class that is in ImageNet. We separate out this category because it highlights the possible benefits of training models to expect new classes to appear at test time, and the importance of model uncertainty and calibration in the face of this 'open world'.
(3) Spurious correlations:
Either (a) the predicted object is not plausibly in the image but surround- ing cues may have been used, (b) the predicted object does not match the groundtruth. In extreme cases, there is no clear indication of the predicted object; in more subtle cases, it is more clear the model is trying predict the class of object in the image but the predicted object would not be considered either semantically or visually similar to the groundtruth class.
(4) Non-prototy
labels: The predicted label is not present but the groundtruth object is non- prototypical example of the class that bears resemblance to the predicted label. Non-prototypical mistakes are relatively rare, and capture the 'long tail' of examples for each class. These mistakes highlight the internal diversity of each class, and the difficulty of modeling the long within-class tail.
Analyzing the remaining mistakes
After review of all original 676 mistakes (comprising both novel predictions and previously reviewed mistakes), we found that a total of 298 were either correct or unclear, or determined the original groundtruth incorrect or problematic. Our evaluation of the ViT-3B model on this re- re-labeled dataset shown in Table II with the model making total of 378 mistakes on the dataset. In other words, approximately 44% of the initial mistakes made by this model were determined to be correct!
All
Organisms
Objects
MLA
MLA Re-labeled MLA
MLA Re-labeled
MLA 96.4%
MLA Re-labeled
VIT-3B 96.3%
97.9%
96.3%
97.8%
98 98.0%
Table I: Multi-label accuracy (MLA) of ViT-3B model before and after our re-labeling,
4.1 Mistake category and severity
Each of the 378 remaining mistakes was assigned both mistake category and severity (Table 2 by the expert panel.
Model
Dataset
Categories
Severities
Fine Fine- pred-grained (FC)
HG w/ OOV
Spurious
Noo-prototypical
Major
Minor
VIT.3B VIT-3B
ImageNet Ima
64.0%
14.3%
13.8%
7.9%
40.7%
59.3%
66.0%
9.4%
15.1%
9.4%
41.5%
58.5%
Greedy Soups
ImageNet
69.1%
10.4%
12.9%
7.6%.
35.75
64.3%
Table 2: Mistake Category and Severity: We classified the majority of the ViT-3B mistakes as fine-grained or a variant of fine- grainedi many of the mistakes were considered "major" mistakes; these distributions held on ImageNetV2 as well for the Greedy Soups model.
Category: 78.3% of errors were assessed to be fine-grained in nature (either in- in-voaabulary or OOV), and 13.8% categorized as a spurious correlation. To measure whether these categories are meaningful, we measured the "hierarchy distance" " between the groundtruth label and the model's prediction using the WordNet hierarchy (22). For example, hierarchy distance of means that the groundtruth and model prediction share the same parent; a distance found that 80.9% of errors with hierarchy distance of 1 were assessed as fine- fine-grained. In contrast, 54.3% of errors with a hierarchy distance of were fine-grained, matching intuition that predictions very likely to be fine-grained errors, and predictions far in the hierarchy
means they share the same grandparent. We
close in the hierarchy are more likely to be spurious correlations, fine-grained with out-ol-vocabulary, or non-prototypical examples. We note that WordNet does not provide a perfect (automated) category function: "goblet" and "vase" are a hierarchy distance of 4 apart, and we encountered one model mistake for this pair that we nonetheless assessed as fine- mneigddd

---

Analyzing class confusions: Given that most failures were fine- grained, we tried identify any patterns present in the class confu- sions made by the model, but found no consistent pattern. The inline figure shows the frequency of occurrence for confused class pairs. The distribution is long-tailed in nature-th theeeeeteeeee majority of class pairs occur exactly once or twice and only a handful of class pairs occur three or more times. The long-tailed nature of the class confusions suggests that we will not be able to resolve a large fraction of model mistakes by focusing on cleaning up adding additional data to only small number of classes.
400
200
100
5
Frequency of occurrence
Severity:
We determined that around 40% of errors were assessed to be "major" errors, indicating that this model still appears to make mistakes that a human familiar with the class definitions would not make, despite the fact that the model on average performs better than an expert human. We return to 'major' errors later in Section 5.1 as we believe that subset of these errors can be a useful evaluation slice for future ImageNet benchmarking.
4.2 Out-of-distributtoo generalization
We evaluated the VTT-3B model on the ImageNetV2 multi-label subset which produced over 900 unreviewed errors. To assess what aspects of our analysis generalize to other datasets, we sampled 100 of these errors using the same panel review system employed for ImageNetV1. 1. We discovered that 47 of the 100 2 predictions were either correct or had problematic labels, leaving 53 mistakes that we reviewed for category and severity. For ImageNetV1, we had previously found that 44% (296/676) of mistakes were either problematic
correct and we found no statistically significant difference between the two datasets this regard (x2 N = 776) = 0.36, p = .55). These proportions suggest that large models are frequently uncovering new correct multi-labels, suggesting that mistake analysis and label correction needs to be part of the lifecycle and maintenance of benchmark development of long-tailed errors properly assess performance as benchmark saturates. Table compares the category and severity breakdowns between the two datasets-ovent the model is making similar types and severities of mistakes both datasets. A chi-square test of independence shows that there is no significant difference between either the mistake categorization x² N 434) 0.97, p .00 mistake severity x (1, N 434) = 0.01, p .92.
4.3
Generalization to new models
As models produce higher top-1 accuracy, how do the types of mistakes they make and improve upon change? We use the Greedy Soups model that obtains 90.9% top-1 accuracy on ImageNet validation (39], measuring its MLA after our initial re-labeling at 98. 1,, and yielding 341 total remaining (and partially unreviewed) errors.
The Soups model corrected 209 mistakes that the ViT-3B model made, while the model made 170 mistakes where the ViT-3B model was correct, yielding an overall accuracy improvement; 28 mistake examples were common with ViT-3B but with different prediction. In total there were 198 novel predictions made by this model that needed to be reviewed; upon review using the same panel method, we found 46.5% (92/198) were problematic (10) or actually correct (82), showing with second model that model predictions on mistakes need to be reviewed, and that the single label expected by top-1 is often insufficient. The Soups model in the end made only 249 errors, for an MLA of 98.6%, a 0.5% absolute increase compared to unreviewed mistakes. The categorization and severity of these errors is shown as the last line in Table 2 Again, chi-square test of independence shows that there is no significant difference between either the mistake categorization (x2 (3, N = 629) = 2.41, p 49) or mistake severity (1, N 629) 1.61, p -20).
4.4 Comparison to humans
We compare the performance of the VIT-3B and Greedy Soups models to the best human labeler from Shankar et al. [I by evaluating on the subset 1,000 ImageNet val images used to compute human accuracy in the prior work. To fairly compare to the models, re-compute the human accuracy
Best human is chosen based on MLA on all classes (as opposed to object organism subset) and corresponds Human E in the original work.
7

---

scores using the original human predictions and our updated label set. Overall, the re-labeling did not significantly change human accuracy; the best human labeler achieved 97.3% MLA on the original multi-labels and 97.4% MIA on the updated multi-labels.
Table comppaes the re-labeled MIA of the VTT-3B and Greedy Soup model to the best human for all ImageNet classes as well as the subset of ImageNet classes corresponding to objects and organisms. Similar to Shankar et al. [3I] we also find that the performance of the models more uniform across the object and organism classes, but humans do substantially better on the object classes than the organism classes. However, unlike prior work, current models outperform the best human when evaluated on all ImageNet classes (though humans still achieve slightly better performance on the object classes).
All Classes
Organisms
Objects
VTT-3B
97.8%
97.4%
98.1% %
Greedy Soup 98.6%
98.4%
98.8%
Best Human [1 97.4%
95.4%
99.0%
Table 3: Multi-label accuracy compared to humans. Both the ViT-3B and Greedy Soup model
achieve better MLA on all ImageNet classes than the best human labeler from Shankar al. (31. However, on the object classes, where the class boundaries are substantially less ambiguous for humans, the best human labeler still outperforms the models.
4.5 Analyzing the Training Data
Finally, we investigate how much we can understand model validation set errors through the lens of the training data. To do so, we inspect the K = 10 nearest neighbor training images using JFT- pretrained (before ImageNet fine tuning) embeddings for the ViT-3B model. Doing this, we rediscover (originally documented in Sun et al. 3 and Kolesnikov et al. [16] that 797 (1.59%) ImageNet validation images exist the training set as exact duplicates. Interestingly, we are the first to notice that every single leaked image has different label in the training set than the validation set. In Appendix ?? we show removing these images has relatively little impact on the model's performance, and detail more pernicious leakage pattern of "near duplicates" in Appendix ?? that is hard to fully quantify. Finally, we show how the training data sometimes explain spurious correlations in Appendix ??.
5 Recommendations and Discussion
In this section we provide recommendations for future evaluation, starting with ImageNet-M: our curated multi-label evaluation set of "major mistakes" that we suggest should be reported on in addition to metrics such as top-1 and multi-label accuracy.
5.1 ImageNet-M: A "major mistakes" evaluation split
Studies over the last several years seeking to understand whether ImageNet remains an informative benchmark have typically concluded that aspects of ImageNet remain useful but top-1 accuracy less These works encourage alternative related metries for ImageNet such multi-label accuracy 31 or object vs. organism breakdowns (31) In many of these cases, stronger, more generalizable models often continue to incrementally but inevitably improve on these metries.
With an emphasis on understanding which long-tail errors are unambiguously errors, we suggest that benchmark focusing on the most egregious long-tail tail errors could provide useful additional signal about whether the improvements are meaningful. In particular, we desire a long- benchmark where we believe that 100% accuracy is achievable. Because the minor mistakes are more subject to interpretation and discussion than the major mistakes, we believe a benchmark focused on the latter will help the community judge what is meaningful improvement ImageNet.
To that end, we leverage our expert-reviewed analysis to produce small slice of the ImageNet multi-label set where (1) today's best top-1 models are still more wrong than right, and (2) the mistakes are largely unambiguous to human given a reasonable understanding of the ImageNet label set. We call this evaluation slice ImageNet-Major.
8

---

ImageNet-M example selection method. The ViT-3B model made 155 "major" mistakes, for which we analyzed whether each example was labeled correctly for three additional models: (1) the Greedy Soups model, (2) model pre-trained on Instagram data but fine-tuned on ImageNet that achieves 85.4% top-1 [19], and (3) A zero-shot evaluation [28 15 27 using a CoCa [41] model pretrained on JFT and noisy image-text data. In order to maximize prediction diversity, we purposefully selected models with varying pre-training data and training methodologies, including a zero-shot model that does not see ImageNet image-label associations directly [TO].
From this suite of four models, we assembled a subset where three or more of the models make a major mistake, yielding 68 such major mistakes. This process is similar in spirit to ImageNet-A [13] except we use four high-performing but diverse models, and we restrict the set to ImageNet images and the corresponding model prediction that were rated as "major" errors. We analyzed the predictions of all models on these examples (including any novel predictions made by these additional models) and verified that none of them were correct new multi-labels, and that any model's 's mistakes were major mistakes. In addition, we attempted to comprehensively manually label additional labels that no model has yet predicted but would be correct, in an attempt to reduce the likelihood that future models are penalized for making novel but unreviewed correct predictions. The failure categories of these 68 examples are broken down into 36 fine- fine-grained failures, 15 spurious correlation failures, 12 out-of- vutoo---ocaalary failures, and 5 non- non-prototypical failures.
We design the ImageNet-M M 68- 68-example subset as an additional split of the validation set that we believe has the following properties: (a) many top performing models trained in different ways make mistakes on this set; (b) the mistakes are all major mistakes as determined by expert-reviewers; (c) the example set is small enough to permit manual inspection by model evaluators; (d) strives to be comprehensively labeled with respect to the ImageNet label set; (e) in theory provides a subset that future models could achieve perfect accuracy on without worrying about underspecified class definitions. We anticipate stronger models may yet make correct unreviewed predictions this slice, so we will endeavor to update the set of multi-labels as they are reported to us.
ViT-3B model achieves 0% accuracy on ImageNet-M: the Instagram- pretrained model gets 9 of the 68 correct, while the Greedy Soups model gets 19 correct. The zero-shot model gets the best performance with 24 correct, even though the zero-shot model overall achieves lower multi-label accu- racy (94.2%) than any of the other models. Because we use these four models to help choose the mistakes, these specific models comparatively will perform poorly on this benchmark. Like with ImageNet-A [13]. a dataset cho- sen using models may bias selection in such a way that future models may easily get very high accuracy on this subset, though we try to mitigate this effect by using mul- tiple differently-traiiee -rainedd models in our selection criteria; we discuss this selection criteria in more detail in Appendix ??.
Evaluation. By construction,
0.65
0.0
0.25
680
4.5
0.90
0.55
1.0
Ttp www.keaeggee..o.... Accuracy
Trend of models ImageNet-M, using Clopper-Pearson inter- vals. Red dashed line indicates total number of images in ImageNet-M as an upper-bound.
top-1 ImageNet vs.
How do models that were not used to select this dataset perform? We evaluate the suite of 70 models from Shankar et al. 3T on this dataset, in addition four recent top models not directly used to help filter the ImageNet-M set: a ViT-G/14 model 43) (90.5% top-1), a BASIC model [27] fine-tuned on ImageNet (90.7% top-1), an ALIGN model (15) fine-tuned on ImageNet (88.1% 1% top-1), and a
CoCa model [41) fine-tuned on ImageNet (91.0% top-1). The plot shown here shows that most models as far back as AlexNet through ResNets get between 10-25 examples correct, but recent high accuracy models such as ViT-G/14, BASIC-FT, and CoCa-FT are starting to solve more of these "major' mistakes: CoCa-FT gets 42 of the 68 examples correct. We reviewed the mistakes made by these four new models, which yielded a total of 5 novel predictions; 4 of them were verified to be wrong (and major), and additional new valid prediction, for which we updated the label set accordingly. Future models could predict classes on these examples that are "minor" mistakes, since the definition of severity linked to the (prediction, example) pair; the dataset slice could be augmented with attribute to provide more fine- fine-grained signals.
9

---

5.2 Evaluation recommendations.
One might view ImageNet-M as another quantitative hill-climbing target, but while benchmarks like this invite quantitative comparison, we caution against using ImageNet-M purely quantitative benchmark for a few reasons.
First, the evaluation set is relatively small; enror bars will be wide enough that it will be hard to claim one model is quantitatively better than another. Second, ImageNet-M is predominantly qualitative in nature; the relatively small evaluation set by design permits qualitative analyses of the results that a much larger set would effectively prohibit. Hypothetically, assume two models A and B where A gets 40 of the 68 correct and B gets 55 of the 68 correct, but where all of the mistakes model A makes are minor/borderline, and all of the mistakes B makes are still major mistakes. Qualitatively, model A might be the better model for some use cases because it makes no major mistakes. To even understand whether a mistake is major or minor today requires human analysis, which is why ImageNet-M is reasonably small.
Our overall recommendations would be to look at model mistakes on ImageNet-M qualitatively: (I) Check that the mistakes are actually mistakes, and if not, update ImageNet-M labels. (2) Check that the mistakes are major; for those that are, identify what error category the mistake belongs in, and apply some appropriate remediation. A model that does not improve upon accuracy, but makes fewer (or no) "major" mistakes, could still be very noteworthy. (3) As one iterates on a model, evaluating on ImageNet-M understand how their changes changed the distributions of mistakes model is making could provide additional insight into whether progress is being made on major mistakes.
Lastly, although we did not find statistical evidence that stronger models solve major errors first, we hope progress on image classification can be evaluated by whether improvements help reduce egregious mistakes before reducing nebulous ones. Overall, encourage reporting on several slices, including ImageNet-M: to measure the strengths and weaknesses of various models.
5.3 Limitations of Analysis
Much of our analysis on mistake categorization, severity, and data cleaning depends greatly on qualitative factors determined by the authors and experiment design, which we briefly discuss here.
Limitations due to mistake subset. We only reviewed multi-labe annotation examples where the ViT-3B model and Greedy Soups model we chose were incorrect; while the multi-label dataset itself has undergone review of mistakes from a suite of many models, we never review any validation image whose groundtruth might be wrong, if all models evaluated also make the incorrect groundtruth prediction. We do partially review some of the mistakes made by a few other models, and build upon dataset where mistakes made by many other models have already been reviewed, but most of the work here assesses these two specific model's predictions.
Definition of a mistake. We re-iterate that we used qualitative judgments to decide whether prediction was a mistake, and so, its severity and categorization. Our qualitative judgments are biased worldview [35] 8 26] comprising the five panelists; moreover, we
therefore based not world experts on dog or animal species, though we believe our assessments are at least as good or better than the original labeling process used for the validation set, given the research effort we made on each mistake. As a mitigation against imbuing too biased a worldview on class definitions, we relied heavily on the validation data to define the boundaries of the class, even when those examples did not match up with our personal definitions of the class. Moreover, updating (and evaluating) multi-labels potentially allows for different world-views to be expressed.
Acknowledgements
We thank the reviewers for their helpful comments and discussion that improved this paper. We thank Lucas Beyer, Zhifeng Chen, Aleksandra Faust, Wei Han, Alexander Kolesnikov, Simon Kornblith, Jiquan Ngiam, Abhijit Ogale, and Jiahui Yu for their comments, suggestions, and infrastructure support. Additionally, thank the larger Google Research organization for their support.
10

---

References
[1] A. Andreassen, Y. Bahri, B. Neyshabur, and R. Roelofs, The evolution of out-of-distribution robustness throughout fine-tuning, 2021. http::/arriv.oo//ab/////// 15831
[2] D. Baker. Datasets have worldviews. attp:://pair..iihhgoogt/ cotpe/pooooeelelees/
kotaset-wooldvveesss Accessed: 2022-044 04-28..
[3] L. Beyer, O. Hénaff, A. Kolesmikov, X. Zhai, and A. v. d. Oord. Are we done with imagenet? CVPR, 2020.
07159
[4] D. Ciregan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. In 2012 IEEE conference on computer vision and pattern recognition, pages 3642-3649. IEEE, 2012.
[5] J. Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, 2009. https
//ieeexplore. ieee. .rree.rg//eoe httg:/dcuuentt52000 5206848
[6] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbomn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[7] S. Eyuboglu. M. Varma, K. Saab, J.-B. Delbrouck, Lee-Messer, J. Dunnmon, J. Zou, and C. Ré. Domino: Discovering systematic errors with cross-modal embeddings. In International Conference on Learning Representations, 2022.
[8] S. A. Friedler, C. Scheidegger, and S. Venkatasubramanian. On the (im) possibility of fairness. arXiv preprint arXiv:1609.07236, 2016.
[9] R. Geirhos, K. Meding, and F. A. Wichmann. Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. NeurlPS, 2020. https 16736
7/arxiv. org//
[10] R. Loneijo-Lopes, Y. Dauphin, and E. D. Cubuk. No one representation to rule them all: Overlapping features of training methods, 2021. https://arxiv..rrg wttp///g/ab///oo/oo////////////a 12899
[11] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang. E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of dut-o--distribution generalization. arXiv preprint arXiv:2006. /6241, 2020.
[12] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
[13] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. CoRR, abs/1997.07174, 2019.
[14] S. Hooker, A. Courville, G. Clark, Y. Dauphin, and A. Frome. What do compressed deep neural networks forget? arXiv preprini arXiv: 19/105244 2019.
[15] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le. Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning. pages 4904-4916. PMLR, 2021.
[16] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision, pages 491-507. Springer, 2020.
[17] Kornblith, J. Shlens, and Q. Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
11

---

[18] H. S. Lee, A. A. Agarwal, and J. Kim. Why do deep neural networks still not recognize these images?: A qualitative analysis on failure cases of imagenet classification. arXiv preprint arXiv: 1it..090990399 2017.
[19] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181-196, 2018
[20] H. Mania, Miller, L. Schmidt, M. Hardt, and B. Recht. Model similarity mitigates test set overuse. NeurIPS, 2019. https://arxiv org/ abs/ 1905, 12580
[21] H. Mania and S. Sra. Why do classifier accuracies show linear trends under distribution shift? arXiv preprini arXiv:2012./5448 2020. https:///axiv: abgg/bs//// 2012. 15483
[22] G. A. Miller. Wordnet: A lexical database for english. Communications of the ACM, 38:39-41, 1995.
[23] T. Nguyen, M. Raghu, and S. Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations (ICLR), 2021.
[24] C. Northcutt, L. Jiang, and Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2021.
[25] C. G. Northcutt, A. Athalye, and J. Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103.14745 2021. https: org/ abs/2100 14749
[26] A. Paullada, I. D. Raji, E. M. Bender, E. Denton, and A. Hanna. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2(11))100336 2021.
[27] H. Pham, Z. Dai, G. Ghiasi, H. Liu, A. W. Yu, M.-T. Luong, M. Tan, and Q. V. Le. Combined scaling for zero-shot transfer learning. arXiv preprint ar 10050, 2021.
(28) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[29] B. Recht, R. Roclofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400, 2019.
[30] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, 7. Huang. A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and F.-. Li. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 2015. https://arxiv.. org/ /a///b///////// 0575
[31] V. Shankar, R. Roelofs, H. Mania, A. Fang. B. Recht, and L. Schmidt. Evaluating machine accuracy on imagenet. International Conference on Machine Learning (ICML), 2020. http://proceedings. mlr.
http:///ww.....eeeesssss//eeeeae presss/v11//hhaaar2200 htt.aaaannarrreeeaaaaaaaaaa0 swwnkar pdf
[32] S. Singla and S. Feizi. Salient imagenet: How to discover spurious features in deep learning?
In International Conference on Learning Representations, 2022.
[33] Stock and M. Cissé. Convnets and imagenet beyond accuracy: Explanations, bias detection, 2017.
adversarial examples and model criticism. CoRR,
[34] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In International Conference on Computer Vision (ICCV), 2017. https://arxiv: org/ abaaab/s//// abb/ 02968
[35] A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.
12

---

[36] D. Tsipras, S. Santurkar, L. Engstrom, A. Ilyas, and A. Madry, From imagenet to image classification: Contextualizing progress on benchmarks. CoRR, abs/2005. 1295, 2020.
[37] G. Van Horn, Branson, R. Farrell, S. Haber, Barry, P. Ipeirotis, Perona, and Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine grained dataset collection. In CVPR, 2015. https: /vission. cornell. www.ell.eduueeeeee......... /e3/ wp cwantttntupllass///eeeoo 20010/////onn..... Haro_oonnnn_ee Buw.dlnulllddaaBneea Biri____iidd_11___1111 2015__VPR__appe.. pdf
[38] H. Wang, S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506- 10518, 2019.
[39] M. Wortsman, G. Ilharco, S. Y. Gadre, Roclofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv preprint arXiv:2203.05482, 2022.
[40] K. Xiao, L. Engstrom, A. Ilyas, and A. Madry. Noise signal: The role of image backgrounds in object recognition. In International Conference on Learning Representations, 2021.
[41] Yu, Z. Wang, Vasudevan, L. Yeung. M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[42] S. Yun, S. Oh, B. Heo, D. Han, Choe, and S. Chun. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2340-2350, 2021. https://arxiv. org/ 0bs/2101.05022
[43] X. Zhai, A. Kolesnikov, N. Houlsby, and L.. Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.000 04560, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Yes.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss the fundamental subjectivity of labels and potential biases in the Limitations section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have ensured that our paper conforms with the ethical review guidelines.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We detail the method by which we assessed mistakes. We have prepared GitHub containing our mistake assessments for the two models for others to verify. We also will release the updated multi-label set for others to build upon our work.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? (Ys]] We largely use pre-trained model outputs for most of our analysis. For the VTT-3B model, we provide training details in the Appendix, though we note that reproducing the exact model is not the contribution our paper. We will, however, release the logits on the full ImageNet validation set for any model we analyzed, if those logits are not available already.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] We use Clopper-pearse intervals for our evaluation of ImageNet-M and use significance tests for our distributional comparisons.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Our work is largely based analyzing pre-trained models; we did no training specific this work except for the fine-tuning required for measuring the impact of de-duplication of validation leakage examples, which we provide details the appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] Each asset has information pertaining
to the license on their respective websites.
(c) Did you include any new assets either the supplemental material or as a URL? [Yes] We release our analysis metadata as well as updated multi-labels.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] The authors of the paper generated all released data metadata and consent to its release.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We only release metadata statistics about mistakes made the existing dataset, or add new ImageNet labels to the existing validation set; we do not add any new potentially personally identifiable information beyond what is already present in the ImageNet validation set.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and sereenshots, if applicable? [Yes] The authors are the participants, and we provide screenshots of the review tools used.
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14

---

Author Contributions
All authors contributed reviewing and categorizing ImageNet mistakes, participated in panel discussions and conducted analysis, and helped write the paper.
VV initiated the project, built review and analysis tools, and helped design and implement experiments and various analyses.
BC performed the K-nearest neighbor error and duplicate analysis (and built tools to do so), helped write the paper, and contributed to the discussions on analysis and project direction.
RGL performed comparisons to human-evaluated labels, helped write the paper, and contributed to discussions on analysis and project direction.
SFK helped with writing and figure generation, and contributed to discussions on analysis and project direction.
RR helped assemble team, provided expertise on previous work related to labeling imagenet:202__mm 1a100022mmllllbbbeeeeeeeeaa dataset, contributed to writing, advised on high level research di- rection and experiments to conduct.
15


Reviews:
[{"review_id": "8SDNxkRPLJL", "reviewer": "Reviewer_VG4e", "summary": "The paper conducts extensive manual reviews on the mistakes made by a few top models. The ​mistakes are further annotated with severity and category. Two major findings are (1) about half of the mistakes are actually correct; (2) about 40% of the mistakes are major mistakes—predictions that humans find obviously wrong. The authors propose ImageNet-Major (ImageNet-M), an evaluation split based on 68 “major mistakes” images, to better benchmark models' progress on remaining mistakes.", "questions": "Questions\n* I request the authors to answer if the “high error bar” problem (see weakness) would exist on ImageNet-M. If so, whether or not it can prevent ImageNet-M from serving as an evaluation dataset.\n* Besides, could the authors clarify the distribution of these 68 images in terms of the mistake category (i.e., how many images are in the “Fine-grained” category and other categories)? Are the ImageNet-M the 68 images listed in Appendix E?\n\nMinor suggestion\n* Although it is good to present many details about the reviewing process, I feel it becomes very easy to get lost while reading the paper. The paper will be better presented if the authors add a figure (e.g., a flowchart) as an overview to summarize how the initial 676 mistakes are narrowed down to 378 mistakes and the final 68 major mistakes.\n", "limitations": "Yes, the authors adequately addressed them in Sec. 5.2.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* The paper studies an important problem to understand the remaining mistakes on ImageNet better.\n* The human evaluations and the experiments are extensive.\n* The manual review provides interesting findings (e.g., the model’s performance is underestimated due to label errors) and good insights (e.g., the categories of the mistakes).\n\n\nWeaknesses\n\n* My major concern is if the ImageNet-M can serve as an evaluation set for future models to benchmark. Due to its small size (i.e., only 68 images), the error bars (due to model selection or random seeds) could be very high.\n* Some related works are not discussed. In terms of mistake analysis on ImageNet (L109-125), Salient ImageNet (Singla and Feizi 2022) annotated core and spurious features on ImageNet; Domino (Eyuboglu et al. 2022) designs a method to automatic detect systematical errors based on clustering. Regarding spurious correlation (L207-211) for the context, ImageNet-9 (Xiao et al. 2021) studies the influence of background on object recognition on ImageNet.\n\nSingla, Sahil, and Soheil Feizi. 2022. “Salient ImageNet: How to Discover Spurious Features in Deep Learning?” In International Conference on Learning Representations. https://openreview.net/forum?id=XVPqLyNxSyh\n\nEyuboglu, Sabri, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Re. 2022. “Domino: Discovering Systematic Errors with Cross-Modal Embeddings.” In International Conference on Learning Representations. https://openreview.net/forum?id=FPCMqjI0jXN.\n\nXiao, Kai Yuanqing, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. 2021. “Noise or Signal: The Role of Image Backgrounds in Object Recognition.” In International Conference on Learning Representations. https://openreview.net/forum?id=gl3D-xY7wLq.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2_TKKVrGq76", "reviewer": "Reviewer_NAFQ", "summary": "The paper analyses the remaining mistakes by the state of the art image classifiers on ImageNet. Several models with >95% on the multi-label accuracy (MLA) metric are considered and their mistakes are analysed. The paper discusses the remaining challenges in ImageNet and proposes a benchmark ImageNet-M that will quantify how well the future models would address those challenges.", "questions": "I think it's more natural to include the questions in the \"strengths and weaknesses\" section above.", "limitations": "I think it's more natural to include the limitations in the \"strengths and weaknesses\" section above.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nWhen evaluating a model, we often report only a number (e.g. classification accuracy) or a set of numbers (ImageNet variants). Then they miss out on the rich information about *what kind of mistakes* they make and *how severe* they are. Given that models are getting closer to 100% accuracy (in MLA), it is important to delve into the remaining errors of ImageNet to distinguish whether\n- remaining errors are mostly due to labelling errors or beyond-the-common-sense difficulty; or \n- they are reasonable errors that could be conquered.\n\nThe paper exactly does that. The conclusion is that there are both impossible and doable cases in the remaining errors. I believe this is interesting news for many who follow and contribute models on ImageNet.\n\nThe paper is quite interesting to read. The authors are often quite frank about the procedure and possible limitations there. This paper lets you think. I think it's a well-written paper.\n\n## Weaknesses\n\nImageNet-M is a subset of the ImageNet validation set that combines the obvious (i.e. humans wouldn't really get confused - line 325) \nerrors made by four state-of-the-art models. The authors suggest this subset as another benchmark dataset for evaluating future close-to-perfect models. \n\nI'm worried about the representativeness of ImageNet-M for the remaining challenges. Indeed, (roughly speaking) they are the intersection of the mistakes made by four state-of-the-art image classifiers. However, as the authors have observed as well, even the best models today make a different set of mistakes. I wonder if there is anything special about those 68 ImageNet-M samples. I'm worried that they are rather unlucky ones for the four particular classifiers.\n\nAnother way to formulate my worry is as follows: what can you conclude when model A gives 60/68 correct and model B gives 50/68 correct? \n- Can you say model A is addressing the \"remaining challenges\" of ImageNet better than model B?\n- Are you sure model A is not making more mistakes outside of ImageNet-M than model B?\n- Isn't it sufficient to compare the multi-label accuracy (MLA) rather than ImageNet-M?\n  - What additional benefit does ImageNet-M bring on top of MLA?\n  - Will there be convincing use cases where ImageNet-M ranking would differ from MLA ranking?\n\nOne possible way to verify the representativeness of the 68 ImageNet-M images would be to do the following experiment: You prepare a sequence of K models M1, M2, M3, M4, M5, M6, ....., MK. Then, plot the size of ImageNet-M against the number of models used for generating ImageNet-M increases from 1 to K. I wish to see if \n- the size of ImageNet-M converges to a non-zero number (ideally close to 68) as you include more models; or\n- the size of ImageNet-M effective converges to zero as each model gets included.\n\nIf the former holds, then I would agree that ImageNet-M is indeed representative. If the latter holds, then ImageNet-M is rather a transient existence that results from considering the particular 4 models. I believe the trend from M1 up to M1,..,M4 (the four models for generating 68 images) would already tell us a lot. Could you share them?\n\nThe paper also seems to hint at the possibility of manually looking through the 68 samples to get more information about the model performance and mechanism than what you would get with numeric metrics (line 346). It would be great if the paper discusses how one could run such a qualitative analysis on the 68 samples - e.g. what kind of insights they could get and how they can improve the engineering based on that.\n\n## Conclusion\n\nThe analysis of remaining errors for current models is interesting and useful (see \"strengths\"). However, I'm not too confident about ImageNet-M as a valid evaluation set for the remaining challenges for future models (see \"weaknesses\"). Unless the issues with ImageNet-M are successfully resolved in the rebuttal, I'd consider the paper's contribution to be insufficient for publication.\n\n## After the rebuttal\n\nI believe the authors have proved the minimal value for the ImageNet-M benchmark in their response. I would therefore change my score to 5 - borderline acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1bK2OXEaQBO", "reviewer": "Reviewer_c29F", "summary": "The paper dives deeper on mistakes in ImageNet dataset by the latest models.  To help contextualize progress on ImageNet and provide a more meaningful evaluation for today’s state-of-the-art models, authors manually review and categorize the mistakes that a few top models make and provide insights into the long-tail of errors. Since ImageNet images contains multiple labels the paper focuses on the multi-label\nsubset evaluation of ImageNet, where today’s best models achieve upwards of 97% top-1 accuracy. The main contribution of the paper is what the analysis reveals:  nearly half of the supposed mistakes are not mistakes at all (significantly underestimating the performance\nof these models). At the same time the models still make a significant number of mistakes (40%) that are obviously wrong to human\nreviewers. To calibrate future progress on ImageNet, authors also provide an updated multi- label evaluation set.\n", "questions": "Have authors done similar analysis on another dataset to see how much the observations generalize? Even if it is a closed (proprietary) dataset? \n\nCan we use this analysis to build better datasets or give guidelines to human reviewers or use it to quickly correct some other major datasets used by the community?\n\nHow do the authors think of this work translating to video classification where the accuracies are still on the lower side and the models are not as accurate?", "limitations": "1. Doing this evaluation on datasets that are larger like ImageNet22K (even if subsampled) would throw more light on the limitation of models. Right now, a lot of conclusions and observations is on the dataset rather than a particular set of models etc. \n\n2. Also, going beyond classification into detection and segmentation would also have been a strong direction to think about.\n\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Thorough and useful analysis of mistakes made by current models on ImageNet\n2. Significant clarity on the mistakes both qualitative (thanks to the supplementary material) and quantitative\n3. A refined multi-label dataset for folks to compare in future\n4. categorizing the mistakes into four: Fine-grained errors, Fine-grained with out-of-vocabulary, Spurious correlations, Non-prototypical labels.\n5. Useful conclusions like 40% of errors are not errors, remaining errors have the four above categories, deduplication and nearest neighbor filtering from validation set.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept