Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper dives deeper on mistakes in ImageNet dataset by the latest models.  To help contextualize progress on ImageNet and provide a more meaningful evaluation for today’s state-of-the-art models, authors manually review and categorize the mistakes that a few top models make and provide insights into the long-tail of errors. Since ImageNet images contains multiple labels the paper focuses on the multi-label
subset evaluation of ImageNet, where today’s best models achieve upwards of 97% top-1 accuracy. The main contribution of the paper is what the analysis reveals:  nearly half of the supposed mistakes are not mistakes at all (significantly underestimating the performance
of these models). At the same time the models still make a significant number of mistakes (40%) that are obviously wrong to human
reviewers. To calibrate future progress on ImageNet, authors also provide an updated multi- label evaluation set.

strengths_and_weaknesses: 1. Thorough and useful analysis of mistakes made by current models on ImageNet
2. Significant clarity on the mistakes both qualitative (thanks to the supplementary material) and quantitative
3. A refined multi-label dataset for folks to compare in future
4. categorizing the mistakes into four: Fine-grained errors, Fine-grained with out-of-vocabulary, Spurious correlations, Non-prototypical labels.
5. Useful conclusions like 40% of errors are not errors, remaining errors have the four above categories, deduplication and nearest neighbor filtering from validation set.
questions: Have authors done similar analysis on another dataset to see how much the observations generalize? Even if it is a closed (proprietary) dataset? 

Can we use this analysis to build better datasets or give guidelines to human reviewers or use it to quickly correct some other major datasets used by the community?

How do the authors think of this work translating to video classification where the accuracies are still on the lower side and the models are not as accurate?
limitations: 1. Doing this evaluation on datasets that are larger like ImageNet22K (even if subsampled) would throw more light on the limitation of models. Right now, a lot of conclusions and observations is on the dataset rather than a particular set of models etc. 

2. Also, going beyond classification into detection and segmentation would also have been a strong direction to think about.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper conducts extensive manual reviews on the mistakes made by a few top models. The ​mistakes are further annotated with severity and category. Two major findings are (1) about half of the mistakes are actually correct; (2) about 40% of the mistakes are major mistakes—predictions that humans find obviously wrong. The authors propose ImageNet-Major (ImageNet-M), an evaluation split based on 68 “major mistakes” images, to better benchmark models' progress on remaining mistakes.
strengths_and_weaknesses: Strengths

* The paper studies an important problem to understand the remaining mistakes on ImageNet better.
* The human evaluations and the experiments are extensive.
* The manual review provides interesting findings (e.g., the model’s performance is underestimated due to label errors) and good insights (e.g., the categories of the mistakes).


Weaknesses

* My major concern is if the ImageNet-M can serve as an evaluation set for future models to benchmark. Due to its small size (i.e., only 68 images), the error bars (due to model selection or random seeds) could be very high.
* Some related works are not discussed. In terms of mistake analysis on ImageNet (L109-125), Salient ImageNet (Singla and Feizi 2022) annotated core and spurious features on ImageNet; Domino (Eyuboglu et al. 2022) designs a method to automatic detect systematical errors based on clustering. Regarding spurious correlation (L207-211) for the context, ImageNet-9 (Xiao et al. 2021) studies the influence of background on object recognition on ImageNet.

Singla, Sahil, and Soheil Feizi. 2022. “Salient ImageNet: How to Discover Spurious Features in Deep Learning?” In International Conference on Learning Representations. https://openreview.net/forum?id=XVPqLyNxSyh

Eyuboglu, Sabri, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Re. 2022. “Domino: Discovering Systematic Errors with Cross-Modal Embeddings.” In International Conference on Learning Representations. https://openreview.net/forum?id=FPCMqjI0jXN.

Xiao, Kai Yuanqing, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. 2021. “Noise or Signal: The Role of Image Backgrounds in Object Recognition.” In International Conference on Learning Representations. https://openreview.net/forum?id=gl3D-xY7wLq.

questions: Questions
* I request the authors to answer if the “high error bar” problem (see weakness) would exist on ImageNet-M. If so, whether or not it can prevent ImageNet-M from serving as an evaluation dataset.
* Besides, could the authors clarify the distribution of these 68 images in terms of the mistake category (i.e., how many images are in the “Fine-grained” category and other categories)? Are the ImageNet-M the 68 images listed in Appendix E?

Minor suggestion
* Although it is good to present many details about the reviewing process, I feel it becomes very easy to get lost while reading the paper. The paper will be better presented if the authors add a figure (e.g., a flowchart) as an overview to summarize how the initial 676 mistakes are narrowed down to 378 mistakes and the final 68 major mistakes.

limitations: Yes, the authors adequately addressed them in Sec. 5.2.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper analyses the remaining mistakes by the state of the art image classifiers on ImageNet. Several models with >95% on the multi-label accuracy (MLA) metric are considered and their mistakes are analysed. The paper discusses the remaining challenges in ImageNet and proposes a benchmark ImageNet-M that will quantify how well the future models would address those challenges.
strengths_and_weaknesses: ## Strengths

When evaluating a model, we often report only a number (e.g. classification accuracy) or a set of numbers (ImageNet variants). Then they miss out on the rich information about *what kind of mistakes* they make and *how severe* they are. Given that models are getting closer to 100% accuracy (in MLA), it is important to delve into the remaining errors of ImageNet to distinguish whether
- remaining errors are mostly due to labelling errors or beyond-the-common-sense difficulty; or 
- they are reasonable errors that could be conquered.

The paper exactly does that. The conclusion is that there are both impossible and doable cases in the remaining errors. I believe this is interesting news for many who follow and contribute models on ImageNet.

The paper is quite interesting to read. The authors are often quite frank about the procedure and possible limitations there. This paper lets you think. I think it's a well-written paper.

## Weaknesses

ImageNet-M is a subset of the ImageNet validation set that combines the obvious (i.e. humans wouldn't really get confused - line 325) 
errors made by four state-of-the-art models. The authors suggest this subset as another benchmark dataset for evaluating future close-to-perfect models. 

I'm worried about the representativeness of ImageNet-M for the remaining challenges. Indeed, (roughly speaking) they are the intersection of the mistakes made by four state-of-the-art image classifiers. However, as the authors have observed as well, even the best models today make a different set of mistakes. I wonder if there is anything special about those 68 ImageNet-M samples. I'm worried that they are rather unlucky ones for the four particular classifiers.

Another way to formulate my worry is as follows: what can you conclude when model A gives 60/68 correct and model B gives 50/68 correct? 
- Can you say model A is addressing the "remaining challenges" of ImageNet better than model B?
- Are you sure model A is not making more mistakes outside of ImageNet-M than model B?
- Isn't it sufficient to compare the multi-label accuracy (MLA) rather than ImageNet-M?
  - What additional benefit does ImageNet-M bring on top of MLA?
  - Will there be convincing use cases where ImageNet-M ranking would differ from MLA ranking?

One possible way to verify the representativeness of the 68 ImageNet-M images would be to do the following experiment: You prepare a sequence of K models M1, M2, M3, M4, M5, M6, ....., MK. Then, plot the size of ImageNet-M against the number of models used for generating ImageNet-M increases from 1 to K. I wish to see if 
- the size of ImageNet-M converges to a non-zero number (ideally close to 68) as you include more models; or
- the size of ImageNet-M effective converges to zero as each model gets included.

If the former holds, then I would agree that ImageNet-M is indeed representative. If the latter holds, then ImageNet-M is rather a transient existence that results from considering the particular 4 models. I believe the trend from M1 up to M1,..,M4 (the four models for generating 68 images) would already tell us a lot. Could you share them?

The paper also seems to hint at the possibility of manually looking through the 68 samples to get more information about the model performance and mechanism than what you would get with numeric metrics (line 346). It would be great if the paper discusses how one could run such a qualitative analysis on the 68 samples - e.g. what kind of insights they could get and how they can improve the engineering based on that.

## Conclusion

The analysis of remaining errors for current models is interesting and useful (see "strengths"). However, I'm not too confident about ImageNet-M as a valid evaluation set for the remaining challenges for future models (see "weaknesses"). Unless the issues with ImageNet-M are successfully resolved in the rebuttal, I'd consider the paper's contribution to be insufficient for publication.

## After the rebuttal

I believe the authors have proved the minimal value for the ImageNet-M benchmark in their response. I would therefore change my score to 5 - borderline acceptance.
questions: I think it's more natural to include the questions in the "strengths and weaknesses" section above.
limitations: I think it's more natural to include the limitations in the "strengths and weaknesses" section above.
review_title: null

Meta-review: metareview: All reviewers are positive about this paper, leaning toward accept. The AC does not find sufficient grounds to overrule the consensus. 
confidence: Certain
Decision: Accept