Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces Imagenet-X a set of meta-annotations on the validation set of imagenet,  plus 12.000 randomly selected images from the imagenet training dataset. These meta-annotations provide information across 16 different attributes such as pose, brightness, occlusions, etc. Then they performed a systematic study comparing 2200 different models. 
strength_and_weaknesses: Strengths: 

* They introduced a new set of meta-annotations on the validation set of imagenet. I think there is a lot of value and effort in providing new datasets that can inspire new angles in evaluating new models. 
* They can show that some data augmentation techniques can improve across a dimension but affect performance among others. 

Weakness: 

* They claim in the introduction: " A hurdle to research progress is understanding not just that, but also why model failures occur." ... However, from the annotations presented and their results is hard to see how to motivate changes in the architecture itself, most of the evaluations seem to be in the data domain. Which seems to be a missing opportunity for the usage of all the data collected and the models evaluated. For instance, why not inform what errors are introduced in a transformer like architectures vs convolutional ones?  is there any impact on the deepness and certain errors presented by the networks? Is there any influence on the number of parameters and some errors? Activation functions, etc, etc. 
* Another missing opportunity, in my opinion, is that they could provide ideas of how to select models given the applications the models can have. 
* It was not entirely clear from the paper how to use the dataset to improve the modeling.  Perhaps a little more intuition in the expected usage of the dataset can provide much more elements to measure the potential impact. So far the paper only seems to address the issues from the data perspective, but not clear how these extra annotations can be used to improve model/architecture choice. 

clarity,_quality,_novelty_and_reproducibility: The paper is clear and provides a good intuition on why imagenet by itself can fail to test model robustness. The work done for them is really impressive in the amount of data collected. I think the main concern is that the paper, as it is, seems to fail to provide more justification on why they think this kind of dataset would provide an advantage for a researcher who attempts to use it and how should be used to improve in the dimensions they suggest. I think there is a lot of potential given the detail of the data collected, but so far the novelty appears very limited. For instance, one of the main conclusions is that "mistakes are surprisingly consistent across architecture, learning paradigms,.." However, they do not seem to unpack this much more besides the graph showing that in general models that have been trained with more data perform on average better. I think they could zoom in a bit more and provide more low-level intuition about this. The second main claim is that these issues can be addressed with data augmentation, which I think is not a novel claim and it is not clear how this specific dataset can help in that direction. 

summary_of_the_review: Overall, I think this paper introduces new angles to look at imagenet, and I think the effort of collecting more data is always appreciated. I feel though that there is a missing opportunity in how the frame the paper and it seems to provide only limited reasons to use the dataset for improving model selections or training routines.   This seems to restrict the novelty of the paper and provide conclusions that are not that novel. That said I think there is potential if they take their data and all the models that they have diligently gathered and find other directions of usage, focusing more on the model selection than on the data diet as the current paper strongly leans to.

------- Post Rebutal -----

I would like to thank the authors for addressing my concerns and providing a new set of experiments in the questions I formulated. I think with these edits this paper shows different applications of the dataset and has a stronger message. I believe the community would benefit from the availability of this dataset. I am raising the score for the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work introduces ***ImageNet-X***, which is a set of human annotations of the ImageNet validation set. It aims to ***explain the model mistakes/predictions*** from detailed visual attributes such as background, object pose, and lighting condition. On ImageNet-X, this work studies 2,200 models with different architectures, learning paradigms, and training procedures. Then, this work gives some observations. For example, all models fail consistently across ImageNet-X categories; dataset augmentation can improve robustness; diverse and more training data would be helpful to learn robust models.
strength_and_weaknesses: **[Strength]**

- The motivation is good, the idea is interesting, and the construction of ImageNet-X is reasonable and clearly presented. 
- The 16 attributes/categories are good. Figure 1 is very clear and helpful. I like the idea of dividing visual changes into several categories.
- Prototypical image defined with ResNet-15 is clever (Section 2).
- Selecting the top factor per image is also a smart idea
-The observations probing model robustness are clear. I would point out that I like the analysis of "is accuracy enough?".  

**[Weakness]**
- The major question would be ImageNet-X labels the factor on ImageNet validation set, which is an in-distribution dataset. If the annotations on some out-of-distribution datasets (e.g., ImageNet-R and ImageNet-A) could be provided, then this work would be super solid and strong. But it is also fine to provide annotations on ImageNet-Val.

- Please clarify some details. ***1)*** what is the "1-word difference" in Figure 3? 
***2)*** why does "person blocking" affect the models (shown in Fig.4)? One guess is that models learn person-related features to classify the input image, which is undesirable. Please comment on this. ***3)*** "Model weaknesses coincide with labeling errors" is not very clear to me. Why this observation offers a potential explanation for the model biases? Please clarify this. My guess is "the labels contain errors, so the models fit these errors and thus show some consistent bias". Please comment on this. ***4)*** some related works [a-c]might be useful to illustrate the effect of data augmentation. (Just a small suggestion for reference)

    [a] Deng, W., Gould, S. and Zheng, L., 2022. On the Strong Correlation Between Model Invariance and Generalization. NeurIPS, 2022

    [b] Mintun, E., Kirillov, A. and Xie, S., 2021. On interaction between augmentations and corruptions in natural corruption robustness.NeurIPS, 2021

    [c] Hendrycks, Dan, et al. "The many faces of robustness: A critical analysis of out-of-distribution generalization." ICCV, 2021

--- ***Post Rebuttal*** ---

Thanks for providing the rebuttal, which nicely addressed most of my concerns. Therefore, I vote for acceptance and would like to have a discussion with other reviewers and ACs.


clarity,_quality,_novelty_and_reproducibility: - The work is very clear, original, novel and of high quality. 

- Moreover, I believe the response will address my questions above.

- This work promises to release ImageNet-X, so I think reproducibility should be high as well.



summary_of_the_review: ***The strengths of this work outweigh the weaknesses***
- First, using 16 visual factors to explain model failures is interesting and reasonable. 
- Second, ImageNet-X is well-constructed and clearly illustrated. 
- Third, observations and analyses are helpful. 

--- ***Post Rebuttal*** ---

This work provides annotations on ImageNet to study failure cases of models. The paper ***provides several interesting and useful insights***, such as the visual attributes that lead to misclassification, the importance of each visual attribute for out-of-distribution datasets. Therefore, ***I vote to accept*** and look forward to a discussion with other reviewers and ACs.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an enhanced ImageNet validation dataset (ImageNet-X) with a more detailed labeling about variation factors of an object instance against its class. Each validation image has been labeled by human editor with 16 different variation factors including pose, background, color and etc.. Hundreds of image classification models have been evaluated on this new enhanced dataset to provide a detailed view towards model failure modes in terms of these variation factors.  
strength_and_weaknesses: Strengths:
    
- This work is well motivated and presented. It proposes a systematic way to label dominant variation type of an instance image against the corresponding canonical class pattern. This is in turn used to provide a concrete picture of model error modes in different classes and across different methods. 
    
- A large number of training models have been evaluated in the new benchmark and some technical insights have been revealed quantitatively, though previous work is already aware of many empirical observations (e.g. Figure 6).

Weaknesses:
     
- It is not clear whether the additional variation factor label can benefit training and further improve the generalization of existing models. The authors have also labeled some training images but they did not show how these newly labeled training images could boost performance. 

- In general, some data is hard to be labeled with only a single dominant factor or it can be perceived differently by different editors. This work did not discuss in detail about how to resolve label inconsistency and how to utilize multi-factor setting generally. 
clarity,_quality,_novelty_and_reproducibility: The overall presentation quality is good but I would suggest to move related work section after introduction to highlight the difference between this work and previous ones. Also, the paper states 2200 trained models are included in the experiment study but later only 209 models are used to create plots? Please clarify this.

The release of ImageNet-X and the corresponding experiments will be helpful for other model development and data usages.  
summary_of_the_review: I think this paper presents a scientific point of view to analyze existing recognition methods and contributes a useful enhanced ImageNet variant for a potentially more diverse purpose. Some critical questions (in weakness section) related to using this new approach/dataset may be also discussed/addressed in detail to solidify the value of this work. 

-- Post Rebuttal --

The authors' responses address most of my concerns. The new experiment results are good examples to show useful applications downstream. It would be nice that the authors can move this new additional results to the main paper. Given the current form of the submission, I am leaning towards acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a set of meta-annotations for ImageNet (entire validation set + 12.000 randomly selected images from the training set). These meta-annotations provide information across 16 different attributes including pose, brightness, occlusions, etc. This extended dataset is then used to diagnostic failure modes of different in a large-scale study of SOTA models. The work appears to be solid with a zoo of models tested and multiple analyses conducted to evaluate the relative benefits of different architectures and training procedures. 

justification_for_why_not_higher_score: The work appears solid and received unanimous support but a number of reported results appear to already be known. While the current paper differs from previous work in terms of the level of details and systematicity of the analysis no single result really standout.
justification_for_why_not_lower_score: Because this is an extension of imagenet this is likely to generate broad interest at the conference.
Decision: Accept: notable-top-25%