ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations
Authors: Badr Youbi Idrissi
Diane Bouchacourt
Randall Balestriero
Ivan Evtimov
Caner Hazirbas
Nicolas Ballas
Pascal Vincent
Michal Drozdzal
David Lopez-Paz
Mark Ibrahim
Venue: ICLR 2023 notable top 25% (2023)

Abstract: Deep learning vision systems are widely deployed across applications where reliability is critical. However, even today's best models can fail to recognize an object when its pose, lighting, or background varies. While existing benchmarks surface examples challenging for models, they do not explain why such mistakes arise. To address this need, we introduce ImageNet-X—a set of sixteen human annotations of factors such as pose, background, or lighting the entire ImageNet-1k validation set as well as a random subset of 12k training images. Equipped with ImageNet-X, we investigate 2,200 current recognition models and study the types of mistakes as a function of model’s (1) architecture, e.g. transformer vs. convolutional, (2) learning paradigm, e.g. supervised vs. self-supervised, and (3) training procedures, e.g., data augmentation. Regardless of these choices, we find models have consistent failure modes across ImageNet-X categories. We also find that while data augmentation can improve robustness to certain factors, they induce spill-over effects to other factors. For example, color-jitter augmentation improves robustness to color and brightness, but surprisingly hurts robustness to pose. Together, these insights suggest to advance the robustness of modern vision models, future research should focus on collecting additional data and understanding data augmentation schemes. Along with these insights, we release a toolkit based on ImageNet-X to spur further study into the mistakes image recognition systems make.

Published as a conference paper at ICLR 2023
IMAGENET-X: UNDERSTANDING MODEL MISTAKES WITH FACTOR OF VARIATION ANNOTATIONS
Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazirbas Nicolas Ballas, Pascal Vincent, Michal Drozdzal, David Lopez-Paz, Mark Ibrahim" Fundamental AI Research (FAIR), Meta Al (byoubi, , ma rark brabbriiii memmmmmmmmm. @m a com
ABSTRACT
Deep learning vision systems widely deployed across applications where liability is critical. However, even today's ' best models can fail to recognize an object when its pose, lighting, or background varies. While existing benchmarks surface examples that are challenging for models, they do not explain why such mistakes arise. To address this need, we introduce ImageNet --aa set of six- teen human annotations of factors such as pose, background, or lighting for the entire ImageNet-Il 1 validation set as well as a random subset of 12k training im- ages. Equipped with ImageNet-X, we investigate 2,200 current recognition models and study the types of mistakes as a function of model's 's (1) architecture e.g. transformer vs. convolutional vs. self-supervised -, and (3) training procedures e.g. data augmentation. Re- find models have consistent failure modes across
(2) learning paradigm e.g. supervised
gardless of these choices, ImageNet-XX categories. We also find that while data augmentation can improve robustness to certain factors, they induce spill-over effects other factors. For example, color-jitter augmentation improves robustness to color and brightness, but surprisingly hurts robustness to pose. Together, these insights suggests that advance the robustness of modern vision models, future research should focus on collecting additional diverse data and understanding data augmentation schemes. Along with these insights, we release toolkit based on ImageNet-X to spur further study into the mistakes the image recognition systems make: https: / //ffacebookresearrreee gi github. io/ www.ammtt///////e/eo/oe/.e.teaet site/home.
INTRODUCTION
Despite deep learning surpassing human performance on ImageNet (Russakovsky et al., 2015; He et al., 2015), even today's best vision systems can fail in spectacular ways. Models are brittle to variation in object pose (Alcorn et al., 2019), background (Beery et al., 2018), texture (Geirhos et al., 2018), and lighting (Michaelis al., 2019).
Model failures are of increasing importance as deep learning is deployed in critical systems spanning fields across medical imaging (Lundervold and Lundervold, 2019), autonomous driving (Grigorescu et al., 2020), and satellite imagery (Zhu et al., 2017). One example from the medical domain raises reasonable worry, "recent deep learning systems detect COVID-19 rely confounding factors rather than medical pathology, creating an alarming situation in which the systems appear accurate, but fail when tested in new hospitals" (DeGrave et al., 2021). Just worrisome evidence that model failures are pronounced for socially disadvantaged groups (Chasalow and Levy, 2021; Buolamwini and Gebru, 2018; De Vries al., 2019; Idrissi et al., 2021).
Existing benchmarks such as ImageNet-A,-O, and -V2 surface more challenging classification examples, but not reveal why models make such mistakes. Benchmarks don't indicate whether a model's failure is due to an unusual pose an unseen color dark lighting conditions. Researchers, instead, often measure robustness with respect these examples' average accuracy. Average accuracy captures a model's mistakes, but does not reveal directions to reduce those mistakes. A hurdle to research progress is understanding not just that, but also why model failures occur.
hands-on and advising contributions

---

Published as a conference paper at ICLR 2023
To meet this need, introduce ImageNet-22 of human annotations pinpointing failure types for the popular ImageNet dataset. ImageNet-X labels distinguishing object factors such as pose, size, color, lighting, occlusions, co-occurences, and a random subset of 12,000 training samples. Along with explaining how images in ImageNet vary, these annotations surface factors associated with models' mistakes (depicted in Figure 1).
for each image in the validation set and
a) ImageNet-X annotation form
Prototypes
Sample
Given the group of Prototypical images to the left and Sample image to the right, please answer the following questions in blue:
Select all factors that make the Prototypical images different from the Samp le image: Ø pose / positioning; K object partially present: Ø object partially blocked by another object; [ object partially blocked by a person; O another object is present; object is small relative to the image frame; D object is large relative to the image frame; lightning brighter; O lightning darker; &d/ background; M color; D. shape:
texture; L pattern; D media style; D subcategory.
Describe more about your selections in question 2: cow in water at the beach
Describe one word what the primary difference between the left images and right image: beach, far-away
b) Robustness analysis enabled by ImageNet-X
Supervised on more data Supervised on ImageNNtt 1I
Self-mperviod
2.5
Robastness intervention
Error ralio
preee
pattora
brighter
partial view
background
Ianger
dacker
amalher
style
multiple objjecta
shupe
persen blocking
object blocking
subeategory
Lestive
Figure 1: Models, regardless of architecture, training dataset size, and even robustness inter- ventions all share similar failure types. ImageNet-X annotations allow to group images into Factors of Variation such as pose, pattern or texture (subfigure and full definitions in Appendix A.2). A model can be evaluated on each of these factors, revealing where it makes the most mistakes. We compare еrror ratios (-acorfactor) i-ace(overall) on each factor for wide groups of models. Subfigure shows that differences in texture, subcategories (e.g., breeds), and occlusion are most associated with models' mistakes. Transparent bars show the factors where there is no significant difference between the 4 groups (p value 0.05 with Alexander Govern test).
By analyzing the ImageNet X labels, in section 3, we find that in ImageNet pose and background commonly vary, that classes can have distinct factors (such dogs more often varying in pose compared to other classes), and that ImageNet's 's training and validation sets share similar distributions of factors. We then analyze, in section the failure types of more than 2,200 models. We find that models, regardless of architecture, training dataset size, and even robustness interventions all share similar failure types in section 4.1. Additionally, differences in texture, subcategories (e.g., breeds),
2

---

Published as a conference paper at ICLR 2023
Dataset
Description
ImugeNet Imisges
Ref.
images
val. set
FoY
ImageNet-C
algorithmic corruptions animated perturbations
X
Hendrycks and Dietterich (2019)
ImageNet-P
X
x
v
X
Hendrycks and Dietterich (2019)
ImageNet-R
artistic renditions
Hendryeks (2021)
ImageNet-A
natural adversurial examples
X
/
K
X
Hendrycks et al. (2021)
ImageNet-O)
out-of oi-siiiiittttin examples
K
/
X
Hendrycks al. (2021)
ImageNet-V2
validation set drawn sketches
/
Recht (2019)
ImageNet Sketch
X
X
X
X
Wang et (2019)
ImageNet-ML.
human multi-labels
Shankar (2020)
ImageNet ReaL.
buman multi labels
Beyer al. (2020a)
ImageNet-RR Meeeeeeeealeee Rel ahel
machine pixelwise multilabels
Yun al. (2021)
ImageNet- Stylized
randomly textured images
Geirhos (2018)
ImageNet-X
buman FoV annotations
V
Table I: Extensions of the ImageNet benchmark extensions designed to inspect failure modes of the ImageNet trained models. We characterize each dataset by looking whether (1) the dataset images only coming from the ImageNet validation set - ImageNet images (2) the dataset images natural images or are created with algorithmic and artistic perturbations of natural images Natural images -; (3) the dataset annotates the entire ImageNet validation set Entire val. set and (4) whether the dataset contains human annotations of image factors of variations - Human FoV Our proposed ImageNet- is the first dataset based ImageNet include human annotations of multiple factors of variation for the entire ImageNet validation set.
and occlusion are most associated with models' mistakes (See Figure 1 and section 4.3. 1). Among modeling choices such as architecture, supervision, data augmentations, regularization methods, find data augmentations can boost models' robustness. Common augmentations such as cropping and volor-jittering however, can have unintended consequences by affecting unrelated factors (see section 4.3.2). For example, cropping improves robustness to pose and partial views, expected, all the while affecting unrelated factors such as pattern, background, and texture. Together these findings suggests that to advance the robustness of modern vision models, future research should focus on improving training data - by collecting additional data and improving data augmentation schemes and deemphasize the importance of other aspects such as choice of architecture and learning paradigm.
We release all the ImageNet -X annotations along with an open-source toolkit probe existing or new models' failure types. The data and code are available at
https:/
github. ic/ /ttp///eeee/tteee//eeeeeeeeeeeee
With ImageNet we equip the research community with tool to pin-point models' failure types. We hope this spurs new research directions to improve the reliability of deep learning vision systems.
2 RELATED WORK
The research community has developed approaches for testing models' robustness on extended versions of the ImageNet dataset see Figure for visual on different levels of evaluation granularity and Table for an overview of datasets created to test the failure modes of the models trained on the ImageNet data. One common approach to introduce artificial augmentations. e.g. image corruptions and perturbations (Hendrycks and Dietterich, 2019), renditions (Hendrycks et al., 2021), 1), sketches (Wang et al., 2019: Bordes et al., 2021), etc. These artificial variations capture changes arising from corruptions. but are unlikely to capture changing arising from natural distribution shifts or variation such as changes in pose, lighting, scale, background etc. Consequently, researchers also collected additional natural images study the performance of the classification models under moderate to drastic distribution shifts Hendrycks and Dietterich (2019); Recht et al. (2019), However, most of these datasets are built assess the model performance when going away from training data distribution and, thus, provide almost no understanding about the nature of the in-distribution errors. Currently, the only ImageNet extensions that help analyzing the in in-distribution model errors the multiclass relabelling or saliency of the validation set (Shankar et al., 2020; Beyer et al., 2020a; Yun al., 2021; Singla and Feizi, 2021). However, this relabelling only explains type of model error

---

Published as a conference paper at ICLR 2023
that is caused by the co-oceurrences of other objects in the scene. Our contribution, ImageNet-2 builds on this line work to provide granular labels for naturally occurring factors such as changes in pose, background, lighting, scale, etc. pinpoint the underlying modes of failure.
3 IMAGENET-XX ANNOTATING IMAGENET WITH VARIATION LABELS
ImageNet- X contains human annotations for each of the 50,000 images the validation set of the ImageNet dataset and 12,000 random sample from the training set. Since it's difficult to annotate factors of variations by looking at single image in isolation, we obtain the annotation by comparing a validation set image to the three class-proton paas-prototyppcall images and ask the annotators describe the image by contrasting it with the prototypical images. We define the prototypical images as the most likely images under ResNet-50 model' (He et al., 2015). Trained annotators select among sixteen distinguishing factors, possibly multiple, and write text description as well one-word summaries of key differences. The form illustrated in Figure 1. The factors span pose, various forms of occlusion, styles, and include a subcategory factor capturing whether the image is of a distinct type or breed from the same class (full definitions in Appendix A.2). The text descriptions account for factors outside the sixteen multi-review on a subset, each image is annotated by one trained human annotator. For example, the annotator marks whether the object depicted in the Sample image is larger or more occluded than P Prototypical images. We provide datasheet following Gebru et al. (2021) 1) in Appendix A.I.
provide. After training the annotators and verifying quality with
One word summaries confirm the list of factors considered. Since our pre-selected list of 16 factors may not encompass every type of variation needed to account for model bias, we also asked annotators to provide one-word summaries to best distinguish given image from its prototypes. We assess whether these free-form responses are encompassed within the pre-defined categories. We find the top-20 one-word annotation summaries are: pattern, close-up, top-view, front-view, grass, black, angle, white, color, background, brown, blue, red, position, facing-left, trees, person, side-view, low-angle, all falling within the 16 categories defined. For example top-view, front-view, facing-left, low-angle, side-view are captured by the pose factor.
3.1 WHAT IMAGENET-X REVEALS ABOUT IMAGENET
To better understand the proposed annotations, we explore the distribution of the different factors among ImageNet images. We identify the most common varying factors in ImageNet, confirm factor training and validation set distributions match, and find factors can vary in ways that are class-specific.
traln of factos
val fodiors
Liiin top Socioe
val big facko
2535
4.077
6258
0,305
5,00%
200
2536-
0.06
000g
nuthora
sdage
partial vabeatyeory
Lestttee
dutter
mudtappi olpend
happrr
bnjgttor
Figure 2: Some factors are selected for most images; choosing the top factor allows to focus on the main change the image. Figure shows the distribution of each factor in the training and validation set both with all factors selected and only top factor selected.
Pose and background are commonly selected factors. As we can see in Figure when aggre- gating all the annotations, pose, background, pattern, and color factors are commonly selected. For instance, pose and background are active for around 80% of the validation images. Since images are not likely to share background and the objects within are unlikely to share the same pose, annotators selected these factors for many images. Pattern and color, which are the next most common, are also unlikely to be identical across images.
'The prototypical images for a class are those correctly classifed with the highest predicted probability by ResNet-50. For 115 models with accuracies better than ResNet-50's, the average accuracy on the prototypical examples is 99.4%. Therefore all good models seem to agree on this set of prototypes.
4

---

Published as a conference paper at ICLR 2023
Selecting the top factor per image. While the annotators could select multiple factors, select a unique, top factor per image. To do so, we use the free-from text justification that the annotator provided. We embed the text justification using pretrained language model provided by Spacy (Honnibal and Montani, 2017), and we compare it to that of selected factors' embeddings. This selection allows us extract the main change in each image, thus avoiding over-representing factors that are triggered by small changes, such as pose or background. We see a clear reduction of those factors in Figure (all top factors).
Training and validation sets have similar distributions of factors To see if there is distribution shift between training and validation, we annotate a subset of 12k training samples in a similar fashion the validation". Figure 2 reports the counts of each factor in this subset (denoted train all and train top). Comparing with the validation dataset, we see most represented factors are very similar to the validation set. We confirm this statistically by performing x²=test on the factor distribution counts, confirming we cannot reject the null hypothesis that the two distributions differ (with < 0.01). Similarly, Appendix A.3 we see that the distribution of factors ticked by the human annotator (refemee to as active) per image are close.
To ease our analysis, we consider for each image its meta-label. A meta-label regroups many ImageNet classes into bigger classes. We consider 17 meta-labels that we extract using the hierarchical nature of ImageNet classes: we select 16 common ancestors the wordnet tree of the original ImageNet 1000 classes. These chosen meta-labels are device, dog, commodity, bird, structure, covering, wheeled vehicle, food, equipment, insect, vehicle, furniture, primate, vessel, snake, natural object, and other for classes that don't 't belong to any of the first 16.
Classes can have distinct variation factors. In Appendix A.4 (right) we also observed statistically significant correlations between factors and meta-labels. For instance, the dog meta-label is negatively correlated with pattern, color, smaller, shape, and subcategory while being positively correlated with pose and background. This suggests that the images dogs in the validation set have more variation for pose and background while having less variation for pattern, color, smaller, shape, and subcategory. The commodity meta-label, which contains clothing and home appliances, is positively correlated with pattern and color and negatively correlated with pose.
In addition to revealing rich information about ImageNet, ImageNet-X can be used to probe at varying levels of granularity model's robustness (see Figure 3).
Previous works Distribattin shifts
ImageNet-X (Ours)
TmagpeNot volidation
ImageNNe A, O V2, Slatch, etc.
Justifications 1-word differences
Classes
Factors
Closses Factors
Performance evaluation granularity
Figure 3: Performance can be evaluated at different levels of granularity. ImageNet - pro- vides factors of variation aspects that makes an image different from typical images of its class. ImageNet -X allows to more precisely pinpoint models' weaknesses and compare them.
PROBING MODEL ROBUSTNESS
By measuring when factors appear among a model's mistakes relative to the overall dataset, we can characterize any ImageNet model's robustness across the 16 ImageNet factors. Robustness
In the training annotation scheme, we separated "pose" into two factors: "pose" and "subject location". where the latter covers cases where the subject in the same pose but a different part of the image. We reconcile the two here into unique "pose" to compare with the validation annotations.
5

---

Published as a conference paper at ICLR 2023
via ImageNet-X goes beyond revealing model mistakes pinpointing the underlying factors associated with each mistake.
Here systematically study the robustness of 2,200 trained models including many models from the ImageNet test bed Taori et al. (2020a) and additional self-supervised architectures such as DINO and SimCLR- reveal the limitations of average accuracy, understand model biases, and characterize which choices in architecture, learning paradigm, data augmentation, and regularization affect robustness. For our analysis we focus on the most salient factor for each image by ranking the selected factors by their similarity to the text-justification.
4.1
MANY DEEP LEARNING MODELS HAVE THE SAME WEAKNESSES AND STRENGTIIS
Supariied
Supercies] IntqoeNii-1K
Self-opprrvial
intersontion
Top Aeautor (Panto)
mocill
wund
sulboalagsey
boodlune
obiject blocking
uuilipil ubjjecl
6026
6036
2079
035
shapo
smelller
blolling
darlor
larger
Tep Aestary (betoe)
80%
4036
2036
Top Arenncy (Pestee)
color
boclggrermd
partial view
brighter
pattern
pose
80%
603
207
03%
40%
600%
80%
103%
647%
aios
40%% 610%
8005
400%
60%
80S5
40%
60%
80PS
40%
60%
800
Ascunacy
Ascunacy
Figure 4: Most deep learning models, when trained, finetuned evaluated ImageNet, have the same biases. We plot top-1 accuracy for the subset of images labeled with the given factor (y-axis) relative overall top-1 accuracy (x-axis). The dashed line is an ideal robust models' performance, i.e. performance on each factor is the same as the overall performance. We show the performance 209 models. We also show the accuracy for the worst factor, and for the images of the worst 100 classes.
To what extent does the commonly reported average accuracy capture a model's robustness? To answer this, inspect 209 models from the ImageNet testbed Taori et al. (2020a) which includes many architectures (most of which are convolutional, with a few vision transformers), training procedures (losses, optimizers, hyperparameters), and pretraining data. We include additional self supervised models for completeness.
Models with similar overall accuracies have very similar per factor accuracies. With all this variety, the scatter plots in Figure 4 exhibit surprising consistency; overall accuracy is a good predictor of per factor accuracies. Most models, even with improving overall accuracy, seem to struggle with the same set of factors: subcategory, texture, object blocking, multiple objects and shape. Conversely, they seem do well on the same set of factors: pose, pattern, brighter, partial view, background, color, larger, and darker. There are some factors where mate-of-theearr models seem to have closed the robustness gap such as person blocking or style.
More training data helps, but robustness interventions do not. Models trained with larger datasets (blue circles in Figure 4) exhibit higher accuracy across the factors suggesting larger training datasets do help others have shown Taori et al. (2020b). Surprisingly, models trained with

---

Published as a conference paper at ICLR 2023
robustness interventions (such as CutMix, AdvProp, AutoAugment, etc..., which are directly aimed at improving robustness don't show a significant improvement in per factor accuracy as prior work also shows Taori et al. (2020b).
Model weaknesses coincide with labeling errors. The ImageNet labels known contain labeling errors Beyer et al. (2020b). With new labels from previous work, we assess the accuracy of the original ImageNet labels and study the distribution of eπrois across factors. Interestingly, we find that the original label accuracies on all factors coincide with the state of the art models. This offers a potential explanation for the model biases. We leave future work to investigate if training on more accurate labels leads to more robust models. We provide details in appendix A.8.
Is accuracy enough? Since our annotations are on the ImageNet validation set, 100% overall accuracy necessarily means a 100% per factor accuracy. So as models get better overall, they necessarily get better per factor accuracies. To disentangle performance from robustness need to investigate the distribution of errors across factors for a given model.
Robust
Susceptilbl
vessel
suale
commacdity
dog
overall
object blocking
preson blooking
style
multigle objects pemum bllcking
texture subradiggrry
tecture
objinn sheking
durier
larger
darbos
lorgrt
subeataguty
object boockang
datker-
pottien
poso pestorn
pattera-
potticn
brighter
pose
pose
boighter
style
mnitippl objects
beighter
pertial stew
pone
2.5
25
2.5 Errot ratão
Error ratão
Error zatio
Error ratio
Ervor ratio
Figure 5: An illustration of how ImageNet-X can identify robustness weaknesses and strengths for Vision Transformers (VIT). Here we visualize the 3 most susceptible, and most robust factors for the three worst-performing metalabels along with dog. We also include the overall validation set for comparison. We see ViT is susceptible texture, occlusion, and subcategory, but is robust to pattern, brightness, and pose. While those overall types also show up across metalabels, we see robustness can distinct by metalabel.
4.2 MOVING BEYOND AVERAGE ACCURACY: ASSESSING FAILURE TYPES WITH IMAGENET-X X
With ImageNet: X we can go beyond average accuracy, to identify the types of mistakes a model makes. To do we measure the error ratio across each of the 16 ImageNet -X factors: Specifically,
- accuracy(factor model)
0)
Error ratio(factor, model) = 1 - (scuracy(model)
(ffattor)
This quantifies how many more mistakes a model makes on given factor relative to its overall performance. It also measures the increase or decrease in likelihood of a factor when selecting model's errors vs the overall validation set. perfectly robust model would have the same error rate across all factors, yielding error ratios of across all factors.
4.2.1 1
AN EXAMPLE OF VISION TRANSFORMER 'S MISTAKES
We illustrate how the error ratio can be used to identify the types of failures and strengths for the popular Vision Transformer (ViT) model. Despite impressive 84% average top-1 accuracy, we find VIT's mistakes are associated with texture, occlusion, and subcategory (appearing 2.02-2. 1x more times among misclassified samples than overall) as shown in Figure 5. On the other hand we find ViT are robust to pose, brightness, pattern, and partial views. We also see that these strenghts or weaknesses can vary by metalabel. For example, VIT is susceptible to occlusion for vessel and snake, but not the commodity metalables where mistakes are associated with style, darkness, and texture. For the dog metaclass, ViT is quite robust different poses. Instead, ViT's mistakes for dogs associated with the presence of multiple objects and differences among dog breeds (subcategory). The full list of failure types across meta-labels is in Appendix A.12.
7

---

Published as a conference paper at ICLR 2023
4.3 WHICH LEVERS CAN AFFECT MODEL ROBUSTNESS?
In practice model developers have many choices from architecture, learning paradigm, and training regularization to data augmentations. What impact do each of these choices have on model robustness? We systematically examine how each choice affects robustness.
4.3.1 ROLE OF SUPERVISION
We first group models into supervised (Ik and with more data), self-supervised, and trained with ro- bustness interventions in Figure 1. We measure the error ratio for each factor across in ImageNet-X We find all model types have comparable error ratios, meaning models make similar types of mistakes. There are few minor differences. For instance, self supervised models seem to be slightly more robust to the factors: color, larger, darker, style, object blocking, subcategory and texture. Supervised models trained on more data are more robust to the person blocking factor. We isolate whether some of the effects may be due to difference in data augmentation next.
4.3.2 DATA AUGMENTATION
6.735
pose
pottern
1.4
partial View
backgroond
Lo
smaller
Foue aato
20-
6.700
1,066
1.5
18
1:000
0.730
L.0
.,000
Cnpp smogth
Cnp enonghh
Chip eirrnghh
Chop ennegth
Crop rinwogth
Chop etrrughh
color
16
darlor
Imre
boolggrond
1.56
smaller
82
prse
3.10
L4
1.10
12-
T
1.40
Chice strraghh
Color strength
Crior strength
Ohior stroaghh
Oolor srrenghh
Oslor strenghh
1.75
atheeregory
color
testune
12
Intgiter
bockgponnd
10
poce
Ennor calo
2.50
1.5
10
08
03-
1.0
50
100
30
39
109
100
Scaoeiling struughh
Suastling dmugth
Suuorllan strough
Suantliaz smupth
Staecling sinapth
Suootlin straulli
Figure 6: Evaluation of the impact of DA towards robustness of the trained model. We experiment with random crop (top row), color jittering (middle row) and Gaussian blur (bottom row); in all cases, vary the strength of the respective DA (x-axis) and train multiple independent models for each setting. We observe that decreasing the random crop lower-bound (i.e. increasing the DA strength going from left to right) leads more robust performance when pose varies, or partial views of the objects are present. Interestingly this also reduce the model robustness to objects that appear smaller and to background variations i.e. the random crop augmented models become robust towards background and texture. Color jittering improves robustness to darker objects and to objects with varying color, Surprisingly, color jittering also increases robustness to larger objects. Gaussian blur also presents a dual effect i.e. benefits towards subcategory and color changes and increases sensitivity to change in pose. In short, it seems that regardless of the DA, the benefits are always accompanied by detrimental impacts on unexpected factors hinting at possible limitation of DA to produce improved robustness throughout all factors.
Data- autaa-agmmetttion (DA) is an ubiquitous technique providing significant average test performance gains (Shorten and Khoshgoftaar, 2019). In short, DA leverages priori knowledge to artificially augment a training set size. The choice of DA policies e.g. image rotations, color jittering, translations is rarely questioned as it is given by our a priori understanding of which transformations will produce nontrivial new training samples whilst preserving their original semantic content. Although intuitive at first glance, DA has recently seen much attention as it can unfairly impact different classes (Balestriero et al., 2022), affect invariance (Deng et al., 2022), and potential to improve various forms of robustness (Hendrycks et al., 2021; Mintun et al., 2021). Equipped with ImageNet X, we now propose more quantitative and principled understanding of the impact of DA.
8

---

Published as a conference paper at ICLR 2023
To that extend, we propose the following controlled experiment. We employ ResNet50 wwhich is one of the most popular architecture employed by practitioners- and perform multiple independent training runs with varying DA policy. Each run across all policies share the exact same optimizer (SGD), weight-decay (le-5), mini-batch size (512), number epochs (80), and data ordering through training. For each DA setting, multiple runs performed to allow for statistical testing. Only the strength of the DAs and the random seed vary within those runs, In all scenarios, left-right flip is used both during training and testing.
Data augmentations can improve robustness, but with spill-over effects to unrelated factors. We report in Fig. 6 the statistically significant effects on error ratio due to three data augmentations: random crop, color jittering and Gaussian blur. For each setting, measure prevalence shifts i.c. how much more or less likely a factor is to appear among a models' misclassifications.
For random crop, we vary the random crop area lower-bound i.e. how small of region can the augmented sample be resized from the original image (varying from 0.08 which is the default value, to 1 which is synonym of no DA). We find the prevalence shift decreases for pose and partial view as expected. We also find that pattern, which unrelated to random cropping, improves as well. However, also observe decrease in robustness for subcategory, unrelated factor.
For color jittering, we vary both the probability to apply some standard color transformations and their strength. In particular, for any given value E (0, 1) we employ the composition of ColorJitter with probability 0.8/ with brightness strength 0.4t, contrast strength 0.4/, saturation strength 0. 0.14, and hue strength 0.14, followed by random grayscale with probability 0.54, and finally followed by random solarization with probability 0.3t. Those parameters are found so that when t nears 1, the DA is similar most aggressive DA pipelines used in recent methods. We naturally observe that the predictions become more robust to change in color and brightness. Interestingly, the model becomes more sensitive to pose variations.
For Gaussian blur, we vary the standard deviation of the Gaussian filter between 0.1 and 0.1 + 3.5t for a filter size the standard setting used, for example, in SSL. In supervised training, GaussianBlur DA is rarely employed on ImageNet. We observe that the model becomes more robust to texture which might come from blurring removing most of the texture information and forcing the model to rely other features. Surprisingly, subcategory is also much less of an impact on performances while change in pose becomes more problematic for the Gaussian blur trained model.
x 13. Note that when / approaches the strength of this DA goes beyond
CONCLUSION
We introduced ImageNet-XX an annotation of the validation set and 12,000 training samples of the ImageNet dataset across 16 factors including color, shape, pattern, texture, size, lightning, and occlusion. We showed how ImageNet X labels can reveal how images in popular ImageNet dataset differ. We found that images commonly vary in pose and background, that classes can have distinct factors (such dogs more often varying in pose compared to other classes), and that ImageNet's training and validation sets share similar distributions of factors. Next, we showed how models mistakes are surprisingly consistent across architectures, learning paradigms, training data size, and common robustness interventions. We identified data augmentation as a promising lever to improve models' robustness to related factors, however, it can also affect unrelated factors. These findings suggest need for a deeper understanding of data augmentations ImageNet X serves as an useful resource to build a deeper understanding of the failure modes of computer vision models, and as tool to measure their robustness across different environments.
model robustness. We hope that

---

Published as a conference paper at ICLR 2023
Reproducibility Statement The results and figures in the paper be reproduced using the open-source code and the ImageNet -X annotations, which we also release. The annotations were collected by training annotators contrast three prototypical images from the same class. This setup that can be replicated using the questionnaire we provide here as well as the freely available ImageNet dataset. For detailed description of the ImageNet-> dataset, please A.
REFERENCES
Michael Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In CVPR, 2019. URL https::///// /ttp://aaavvvii.
www.xiv.org//bba 111/1.11111 11553.
Randall Balestriero, Leon Bottou, and Yann LeCun. The effects of regularization and data augmenta- tion are class dependent. arXiv preprint 220.:220040066.2,, 2022.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In ECCV, 2018. URL https: /ttpps:///rxxv wwwp/as/////.oo.ooo.o.........st abs 807.0000000 04975.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are done with imagenet? arXiv, 2020a. URL
/2gg 071559.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? (arXiv:2006.07155)) Jun 2020b. doi: (0-488550arXxi.2000.711 URL
http::/ /tetp://arxiv. htttp://arxxw..or//bbb...s./ www./rrg///......ooo..o....o.../ 2006 07159. arXiv:2006.077 [cs].
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self-supervised representation knows about. arXiv preprint arXiv:21 2021.
Diane Bouchacourt, Mark Ibrahim, and Ari Morcos. Grounding inductive biases in natural images: invariance stems from variations in data. Advances in Neural Information Processing Systems, 34: 19566-19579, 2021.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, 2018. URL https::/prrceeee
mllr www.sppesss//eme vvl//vv///// http:///ooowwwwnnnn.m.eemmmaaaaa wwwwwniiiiii.. html.
Kyla Chasalow and Karen Levy. Representativeness in statistics, politics, and machine learning. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021.
URL https:ttp//.cc....tt /arxiviivvv wwtg/abss..ee /2011 03827.
Alex DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, 2021. URL https: wwww naturee com/
021-00338 0030000330000
Weijian Deng, Stephen Gould, and Liang Zheng. On the strong correlation between model invariance and generalization. arXiv preprint #rXiv:2207.07066 2022.
Terrance Dries,, Ishan Misra, Changhan Wang, and Laurens van der Maaten. Does object recogni- tion work for everyone? download pdf. arXiv, 2019. URL httpt:tppseeoo htttt:ps///axxx..or///eeeee /
/9906.
02659.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12)) 86-92, 2021.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 1.2231, 2018.
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 2020. URL https: /arxiv. org//bs//911 9191 07738.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In /CCV, 2015. URL https://///xxi org/abs 1502. 01852.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations, arXiv, 2019. URL
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of tittof-dditributiin generalization. In ICCV, 2021. URL
abs/2006. 6241.
Matthew Honnibal and Ines Montani. spaCy 2:
Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group peist-group-accurage CLeaR, 2021. URL https://arxiv org/abs/2111 2110 14503.
Alexander Selvikvåg Lundervold and Arvid Lundervold. An overview of deep learning in medical imaging focusing on mri. Zeitschrift für Medizinische Physik, 2019. URL https: /link. springer.. cwm/ http//rricce/0000
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der S Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter coming, arXiv, 2019. URL https: /tattps///axiv....t org/ /bs/ 1.90700488
Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and corruptions in natural corruption robustness. Advances Neural Information Processing Systems, 34:3571-3583, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019. URL /ttpapppp///....o/////////////// 19111111111111..1111111111111111
10811.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015. URL https: wttpiaixiiiii/ii///aaa//aaaaavva org/ abs/
Vaishaal Shankar, Rebecca Roclofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt. Evaluating machine accuracy on imagenet. In ICML, 2020. URL httpps:///aaxi..////at/sss mlr. press/ / vlvvl11999 /
0575.
Connor Shorten and Taghi M Khoshgoftaar. A survey image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning?
In International Conference on Learning Representations, 2021.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Ad- vances in Neural Information Processsing Systems, volume 33, page 18583-18599, Curran Asso- ciates, Inc., 2020a. URL https: httpa::/proceeeding neurips ct/ cctpaper//02///////ecpp html.
7c53d21 4ee 4833005555415552220004444e666653
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt, Mcasuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems,
2020b.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. NeurIPS, 2019. URL https: /aaxixxivv.////////aaa////aa org/ abs/ 1905. 13549.
11

---

Published as a conference paper at ICLR 2023
Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In CVPR, 2021. URL https: / wwwpiaiiivvoooo. org/ abs 2211. 05022.
Xiao Xiang Zhu, Devis Tuia, Lichao Mou, Gui-Song Xia, Liangpei Zhang, Feng Xu, and Friedrich Fraundorfer. Deep learning in remote sensing: A comprehensive review and list of resources. IEEE Geoscience and Remote Sensing Magazine, 2017. URL https: / //rxivv..vv////////////a///aaaav org/ 03959.
12

---

Published as a conference paper at ICLR 2023
A
APPENDIX
We include additional experiments and dataset details in this appendix. In the first several sections, provide additional information regarding the ImageNetX dataset: annotation details in A. factor definitions A.2, number factors selected in A.3, and factor co-occurence in A.4. Finally, we show sample annotations in Appendix A.5.
Next, we show additional experiments. First in A.6 we show details of ImageNet-X robustmess analysis for additional architectures and learning paradigms. We show how such comparisons inform modeling choices in 10 In Appendix A.7 we detail our experiments in applying tailored augmentations per class and show in A.8 how label bias measured in ImageNet-Real correspond to ImageNet-X factors. Next we show how ImageNet-X factors can probe downstream performance on OOD datasets in Appendix A.9. Then, we replicate our analysis using multi-factor labels A.II. Finally, in A.12 we illustrate how ImageNet-X factors can reveal granular model robustness strengths and weaknesses for specific class categories.
A.1 DATA SHEET
We follow the recommenattoons from by constructing datashect (Gebru et al., 2021) for ImageNet below.
For what purpose was the dataset created? We created ImageNet-X enable research on understanding the biases and mistakes of computer vision classifiers on the popular ImageNet benchmark.
What do the instances that comprise the dataset represent? Each instance represent set of binary labels and free-form text descriptions of variation for each ImageNet validation image.
How many instances are there in total? There are 50,000 validation and randomly sampled 12,000 training image labels.
What data does each instance consist of Each data point is a json string indicated whether each of the 16 factors was selected and two free-form text fields.
Are there any errors, sources of noise, redundancies in the dataset? Each sample was labeled by a single human annotator. There are no other redundancies or sources of errors.
Is the dataset self-contained? The full annotations each corresponding ImageNet filename will be provided in self-contained repo. The original ImageNet images however, must be downloaded separately.
Does the dataset identify any subpopulations (e.g., by age, gender)? No.
Is the software that was used to preprocess/ceenn/abell the data available? Yes, the data was preprocessed using Pandas and Numpy, both freely available Python packages.
13

---

Published as a conference paper at ICLR 2023
A.2
FACTOR OF VARIATION DEFINITIONS
Factor of variation
Description
The object has a different pose or is placed in different location within the image.
Pose
The object visible only partially due to the camera field of view that did not contain the full object e.g. cropped out.
Partial view
Object blocking
The object occluded by another object present in the image. The object occluded by person human body part this might include objects manipulated by human hands. There is, at least, one another prominent object present the image.
Person blocking
Multiple objects
Smaller
Object occupies only small portion of the entire scene.
Larger
Object dominates the image.
The lighting in the image is brighter when compared to the prototypical images.
Brighter
The lightning in the image is darker when compared to the prototypical images.
Darker
The background of the image is different from backgrounds of the prototypical images.
Background
Color
The object has different color.
Shape
The object has different shape.
Texture
The object has different texture - e.g., sheep that's sheared.
Pattern
The object has different pattern e.g., striped object.
Style
The overall image style is different- e.g., a sketch.
The object distinct type breed from the same class e.g., a mini-van within the car class.
Subcategory
A.3 NUMBER OF IMAGES WITH ACTIVE FACTORS
Figure/ shows the count of samples depending on how many factors are selected by the annotator.
train
Count
10
102
10a
109
1
3
5
8
10
11
Figure 7: Many images have multiple factors selected. Very few have no factors selected since some factors are sensitive Distribution of the per image number of factors annotated active, in validation vs train.
A.4 FACTOR CO-OCCURRENCES AND CORRELATIONS.
Table reports the exact count of each factor. Furthermore, Figure 8 (left) shows the spearman correlation between each factor and meta-class and (right) between each pair of factors for the validation set. Figure shows the same for the training set for comparison. The major correlations the same between the two heatmaps (for instance pattern and color pattern and dog). We can see that some factors are slightly correlated, for example color and pattern, larger and partial view, subcategory and shape positively correlated.
14

---

Published as a conference paper at ICLR 2023
bwdgrend
23
13
65
1018
129
489
24
62
60
-12
63
-30
-85
Aaghe
41
-22
40
-12
-15
-16
ddjec Mailing
toglere
41
17
Figure 8: Factor/factor and factor/meta label spearman correlatton heatmaps for validations set. The heatmap shows only significant correlations (p value < 0.05).
-55
81
57
-42
48
311
15
-35
Indgreendl
83
4.4
-38
-33
-29
-18
-5.1
B3
pellect-
482
31
-30
stndke-
52
38
-25
25
pattil sloe-
-2.
29
20
atbeatager-
-13
abjer Jaaiiing
28
58
33
3.5
brighor-
Maef
modile
pire
rodor
Maye
mericd otee
Figure 9: Factor/factor and factor/meta-le meta-labllllll spearman correlation heatmaps for training set. The heatmap shows only significant correlations (p value < 0.05).
Split
Factor selection
Factor
count
factor
count
Factor
count
factor
count
39862
background
35467
pottern
15401
color
12995
All factios
partial view
subcategory
3606
1504
texture
smaller
2866
878
shape
style
1889
574
darker
multiple objects
1837
515
Validation
larger
390
object blocking
person blocking
brighter
pose
1503]
background
14743
color
6138
puttern
$938
Top factor
smaller
1449
shape
675
partial view
640
subcategory
586
texture
283
larger
darker
123
object blocking
person blocking
60
brighter
45
style
44
multiple objects
41
pose
10349
background
9618
pattern
color
smaller
1192
subcuitegory
717
shape
$14
multiple objects
462
All factors
purtial view
362
style
357
texture
293
darker
99
Training
larger
object blocking
69
person blocking
brighter
7
pose
3250
hackground
2936
pattern
2525
coloe
1438
Top factor
smaller
494
subcategury
shape
purtial
testure
93
larger
35
darker
26
style
17
object blocking
person blocking
16
multiple objects
brighter
4
Table 2: Counts of each factor of variation in ImageNet-X.
A.5 EXAMPLE ANNOTATIONS
In Figure 11 we present some sample annotations. We show the prototypical images for each class along with the sample annotators labeled.
15

---

Published as a conference paper at ICLR 2023
(a) Factor/factor co-occurrence training subset.
IT CHOKES
Artichoke
A.
pose background
"top-view"
left
right
Sea
pose
object small pwntow fame background
"Farvieww red bail"
right
Pitcher
color pattern background
Antique, Victorian"
left
right
Figure 11: ImageNet-X Annotations Examples illustrates three annotation examples from the ImageNet validation set. In panel, A. In panel B., we see the effect of an object being small and in panel C. we see color and pattern as distinctive varying factors. We provide the full set of annotations with their image paths in our repo.
example of background and pose for Artichoke.
A.6 RESULTS ON POPULAR MODELS AND METHODS
A.6.1 ASSESSING THE ROBUSTNESS OF VISION TRANSFORMERS
To move beyond average accuracy, can probe model robustness with varying levels of granularity by measuring bias towards or against ImageNet-X factors. Here we walk through the biases of ViT (vision transformer model) trained ImageNet-21k. While the average accuracy is >> 80%, probe further to uncover VIT biases towards texture, occlusion, and subcategory.
Assessing VIT's bias modes We first examine the prevalence shift of each factor among a models' misclassified samples relative to their overall prevalence. Despite impressive average top-1 accuracy, we find VIT susceptible to several factors with texture, occlusion, and subcategory appearing +1.02-1.11 1x more times among misclassified samples than overall shown in Figure 12. On the other hand we find ViT are robust to naturally occurring pose, brightness, pattern, and partial views.
Granular biases by meta-label We further probe robustness by examining how model bias differs among meta-labels of classes groups. We find, as shown Figure 13, the three meta-labels with lowest accuracies are vessel, snake, and commodity. We find ViT is most bias towards occlusion (+2.0x-2.6x) for vessel and snake and style for commodity. While ViT robust to darkness and brightness (-1.0x) for vessel, it's susceptible to darkness (+1.2-1.66 (+1.2x1.6x) for snakes and the commodity labels Among these three meta-labels VIT is susceptible to shape (+ ++4)) for the snake meta-label. Model bias across all 16 meta-labels is in Appendix Figure 12.
16

---

Published as a conference paper at ICLR 2023
model
less robust
shift
0.5
texture
object blocking
Bubcategory
person. blocking
darker
smaller
multiple, objects
style
shape
larger
color
background
partial. view
pattem
brighter
pose
factor
Figure 12: VIT model bias
VIT
vessel
snake
commodity
dog
top-1)
(72.40% top-1)
(72.98% top-1)
(75.69% tep-1)
object_blocking
person_ blocking
style
multiple_ode multiple_objects objjects
robust
texture
object_ blocking
darker
subcategory
larger
darker
texture
darker
susceptible
su subcategory
shape
larger
person_blocte blocking
smaller
subcategory
smaller
object_
blocking
background
partial view
object_blocking
object
larger
shape
smaller
subcategory
brighter
partial_vier
texture
view
smaller
multiple_obpe _bjjects
color
background
color
jorson_blocking
background
shape
bexture
pose
brighter
multiple_objects
background
style
color
style
darker
larger
person person_blocking blocking
shape
pattern
pattern
pose
pose
brighter
pose
pattern
pattern
style
multiple_objects
brighter
partial_ view
-10 1
DMN
013
Figure 13: Robustness by meta-label shows robustness of the three worst meta-labels for ViT (vision transformer) well the dog class which encompases large number of ImageNet classes.
A.6.2 ARCHITECTURE AND LEARNING PARADIGM
models exhibit similar overall biases : As shown in Figure 14. ResNet-50 and ViT exhibit similar prevalence in factors among their misclassified samples suggesting both models exhibit robustness similar factors. Both ViT and ResNet-50 show bias among misclassification towards texture, occlusion by person or object, subcategory, darkness and the prescence of multiple objects. For example, texture more likely to appear among misclassified compared the overall validation set by +1 +1.48x for ResNet-50 and +1.1x for VIT. This confirms finding in Geirhos et al. (2018) illustrating CNNs are bias towards texture. We extend this finding by illustrating transformer based models also biased towards texture. The overall similarity between CNN and transformer based models matches the findings in Bouchacourt et al. (2021) 1) for invariance augmentations by illustrating the similarity holds even for natural factors of variation. While the
Convolution and transformer-bee
17

---

Published as a conference paper at ICLR 2023
model
1.5
1
ResNet50
shift
less rabust
0.5
texture
person beategory
object blocking
multiple, objects
darker
style
shape
smaller
larger
color
background
partial. view
pattern brighter
pose
factor
Figure 14: Comparison of factor error ratios for VIT (transformer-based) and ResNet-50 (CNN- based) models. The y-axis measures the shift in how likely a factor is to appear in a model's s misclassified samples relative to their overall prevalence. likelihood a factor is model bias towards factor suggesting a model is susceptible to the factor. A shift below zero implies a factor less likely to appear among misclassified suggesting the model robust the factor.
shift greater than zero measures the
overall trends are similar we observe the largest difference between ViT and ResNet-50 in their biases towards texture, the presence of multiple objects, and style with VIT exhibiting smaller bias.
Role of learning supervision is important for SimCLR versus ResNet-50, but not transformer based VIT versus DINO In Figure 15b, we compare the factor bias in self-supervised versus supervised learning for both ResNet-based and transformer-ban based models. We observe remarkably similar biases when comparing ViT versus DINO in 15a. For ResNet based models, we observe self-supervised learning in SimCLR leads to less bias towards the presence of multiple objects, style, and darkness. Specifically, for supervised ResNet-50 the prevalence with multiple objects is +1 +1.03x (versus +0.22x for SimCLR), among misclassified samples. This suggests while learning paradigm leads to simlar biases for transformers, standard supervised training does lead to differences in bias towards texture, multiple objects, and darkness. We isolate the effect data augmentation in next Section.
A more gramular comparison of learning paradigm for transformers by meta-label: We compare the worst meta-labels for ViT versus DINO, finding both more share the worst three meta-labels with remarkably similar biases: occlusion and texture for vessel, style and darkness for commodity, and occlusion for snake. There is difference in the magnitude of some factors. For example, DINO is more susceptible to partial views for snakes compared to ViT (+1.76x versus +0,81x). Aside from minor differences, the granular meta-labels analysis confirms DINO and ViT exhibit similar robustness for their worst meta-labels.
A.7 IMPROVING TRAINING
A.7.1 TAILOR AUGMENTATIONS BY CLASS
To test whether tailoring augmentations by class improves robustness, we identify meta-labels benefiting from the robustness induced by two types of augmentations: color jitter and random resized crop. For color jittering, we select meta-labels with prevalence shift ; 1.0 for either darker, color, or larger factors (whose robustenss improves with color jittering). The resulting 402 classes then augmented with additional color jittering augmentations
belonging those meta-labels top of the standard training recipe with random resized cropping. We show results against basclines of only cropping, only color jittering, and both augmentations applied all classes in Figure 17.
18

---

Published as a conference paper at ICLR 2023
model
less robust
DINO
shift
0.5
texture
object blocking
person blocking
darker subcategory
style
smaller
shape
multiple. _bojects
larger
color
background
partial. view
brighter
pattern
pose
factor (a) Supervised versus self self-uupeviied vision transformer
model
1.5
ResNet50
less robust -
SimCLR
shift
0.5
texture
person person_blat blocking
object blocking
Subcategory
shape
multiple. objects
smaller
style
darker
larger
color
background
partial. view
brighter
pattern
pose
factor
(b) Supervised versus self-supervised ResNet-50.
Figure 15: Comparison of error ratios across self-supervised and supervised learning. The y-axis measures the shift in how likely factor is to appear in model's misclassified samples relative to their overall prevalence. A shift greater than zero measures the likelihood factor is model bias towards factor suggesting model is susceptible to the factor. A shift below zero implics factor is less likely to appear among misclassified suggesting the model is robust to the factor.
A.8 EVALUATING LABEL BIAS
In Figure 18 we measure the ReAL accuracy relative to the original validation accuracy.
A.9 CORRELATION WITH IMAGENET DISTRIBUTION SHIFTS
Imagenet-X provides fine grained labels to evaluate model robustness. To link the performance of this work with related work in Imagenet robustness, we provide a pearson correlation score between factor error ratios and distribution shift error ratios in figure 19.
ImageNet-R
ObjectNet- 1.0-beta
ImageNet-A
Greyscale
ImageNet- Vid- hod-RRoust_pmoO
texture (0.93)
texture (0.98)
texture (0.96)
color (0.79)
texture (0.77)
shape (0.90)
subcategory (0.97)
shape (0.96)
smaller (0.74)
smaller (0.77)
subcutegory (0.90)
shape (0.96)
smaller (0.94)
person blocking (0.72)
style (0.77)
Table 3: show the Pearson correlation (shown parenthesis) between factor error ratios and performance distribution shifts of ImageNet for the 3 most correlated factors.
19

---

Published as a conference paper at ICLR 2023
model
1.5
1
ResNet50
shift
less rabust
SimCLR
0.5
DINO
texture
person b_ocbllooiingggg
object_ blocking
subcategory
shape
smaller
style
multiple _bbjects
darker
larger
color
background
partial. view
brighter
pattern
pose
factor
Figure 16: Models tend to have similar biases for the measured factors. The largest difference being between ResNet and Sim CLR likely to color jitter.
Crop
Full Colp
Color
Per class color jitter
Error ratio
1.5
000
Josc
pattorn
brigltter
backroand
calor
partial view
larger
multiple objects
darker
mualler
style
dbject blocking
shape
atheadegory
person blocking
Losture
Figure 17: Prevalence shift comparison of tailored versus universal color jittering augmentation
In Table 3, show the three most correlated factors with each OOD dataset. We compute the Pearson correlation between each factor's error ratio (shown in parenthesis) and the performance on each OOD dataset. We find intuitive correspondence between the factors in ImageNet-X and downstream OOD performance. For example, ImageNet-R, which contains renditions such as art, cartoons, embroidery, graphics, origami, paintings, patterns of ImageNet classes, most correlated with shape and texture as well as subcategory. On the other hand, Greyscale which discards color information is most influence by each model's reliance color, smaller objects, and occlusion. ImageNet-A, which contains natural images explicitly mined to be challenging for standard vision models, often contains images where the object is small or blends in with the background. We see the top factors for ImageNet-A are texture, shape, and whether an object can be identified even if it's small. We hope this analysis reveals the potential for ImageNet-X factors to better probe downstream performance. This unlocks for example the potential for future work to assess/optimize downstream OOD performance only by tuning model performance on the ImageNet-X factors.
A. INFLUENCE OF ARCHITECTURE CHOICES
Although the general conclusion of our per factor performance analysis indicates consistent perfor- mances across architectures, training data and losses, there still some differences that arise. We study some model choices to shed some light on where those differences might arise in Figures 20, 21 and 22.
A.11 MULTI FACTOR RESULTS
In the paper we chose to select the top factor per image. In this section we present the main figures of the paper with multiple factors per image selected. The main conclusions remain valid, and the
20

---

Published as a conference paper at ICLR 2023
Suprevised data Supeevised on Imagrevet IK
Self Self-nueesstood
Robruttres insevvationn
Original vubdation
160 classes
wonst factor
toxtaze
subeategocy
shope
bloeking
Real. Acenney (Focto))
60%
mit Leck
4036
-lyle
bargee
partial camr
mullipphe aljiela
sunalar
daa kauker
Real Acentey (Butot)
8036
60%
40%
20%
abject blucking
Iackgroand
Ieal Accuroyy (Pacttt
6006
4035
0/%
61F Roal. Accuzary
50%
0/% 60% Roal Accurary
03%
60% 89% Boal Aocuracy
405i 60% 80%
4006 60/5 Real. Accuracy
805E
005E
60% 50%
Roal Accuracy
Real, Accuracy
Figure 18: We measure the ReAL accuracy Beyer et al. (2020b) of the original validation labels and show that these labels have consistent errors on the same factors as models shown previously.
objectnet-1.0-11 D-bbet imagenet-a
Imagenet-r imagenettviddrrauutt pmo
greyscale
Pearson correlatio
pose
pattern
brighter
partial view
larger
subcategory
multiple objects
background
object blocking
style
texture
shape
person, blocking
darker
color
smaller
Factor
Figure 19: Pearson correlation between factor error ratios and distribution shifts of ImageNet. The correlation strengths change depending on the distribution shift. For instance for greyscale images, the color factor is maximally correlated with relative performance.
results don't it drastically change. We include the multifactor setting for completeness. The provided code allows generating figures and loading annotations both in top and multi factor settings. See figures 23 and 24
A.12 GRANULAR MODEL BIASES BY META-LABELS
In Table 4 we show the robustness susceptibility of factors for DINO, ResNer50, SimCLR and ViT for each meta-label. We see while many factors are shared, some meta-labels are especially susceptible to distinct factors. For example, ViT is susceptible style for vehicles and insects.
21

---

Published as a conference paper at ICLR 2023
Transformer
Error ratio
2.0
1.5
1.0
pose
pattern
brighter
partial view
background
color
larger
darker
smaller
multiple objects
shape
person blocking
style
object blocking
subcategory
texture
Figure 20: We measure the error ratios for multiple resnets (ResNet50, ResNer101, ResNet1 and for multiple Transformers (ViT small/16, base/16, large/16). The main differences in the factors partial view, darker and multiple objects, with transformers having an advantage. The better handling of multiple objects Transformers might be due the lack of average pooling.
resnet50
resmet152
resnet101
Error ratio
2.0
less robust
1.5
1.0
pattern
brighter
partial view
background
color
larger
darker
smaller
shape
multiple objects
style
person blocking
object blecking
subcategory
texture
Figure 21: We measure the enror ratios for multiple resnets (ResNer50, ResNet101, ResNet152) to study the effect of depth. For most factors, the deeper the resnet the better, except for the "object blocking" factor that gets considerably worse the model gets deeper.
efficientret-b0 efficientnet-11
efficientnet-b3
efficientnet-b4 officientnet-bo b5
Error ratio
2.5
afficientnet-bz b2
2.0
S5S robust
1.5
1.0
pose
pattern
brighter
partial view
background
color
larger
darker
smaller
multiple objects
person blocking
shape
object blocking
style
subcategory
texture
Figure 22: We measure the error ratios for scales EfficienNet models from bo to b5. As the model grows in size, the error ratios grow, especially for the b5 scale.
Supervised on more data Supervised on ImngeNet-11K
Self-aupervised
Robustuss Intervention
Error ratio
los
poso
pattera
badkgronnd
ntyle
partial vimw
larger
culor
mmaller
durker
brighter
object blooking
peman blodking
shape
muitiple objecta
texture
mubentenuuty
Figure 23: Model comparison with multiple factors per image.
22

---

Published as a conference paper at ICLR 2023
Supervised
Supercisel TroapeNet-1K
Self-opemisel
woest 100 clanses
wonst factor
suboategocy
textirr
multiple objects
shape
Top Aeennry (Peeto)
203
035
blodiing
object blocking
brighter
darker
smalee
Top Acennee (Focte)
80%
603
4036
2036
'2p Accurse (Puter)
Iurge:
portial view
Inckgrooaa
pallarn
p000
skyle
8006
2076
009
4075
6005
50%
407
60%
80%
609%
80%
40%
609S
30%
40%
600%
30%
40F
60% Aocurary
80%
Accmary
Accuaayy
Top
Accunay
Accunary
Acconacy
Top
Figure 24: High level scatter plot with multiple factors per image.
23

---

Published as a conference paper at ICLR 2023
Model
Metacloss
Rebrett
Smerpiibbe
Nid
pesson blockin (0.0), ssye (0.0) toonue (1.0), largrr (1.0)
daller (2.5) smalls (2.7) agjor bocking (12.5) shape (12.9)
communiy
yhier (0U0) teature (..)) puttern (8.Ss pone (0:5)
psesill (1.7%
obrest blocking St, wyyl
pose
(0.5),
desice
60.8) styls 1030), beigttr (048)
smaller 50 bocking
(2.5) objoct Nocking (2.7)
6og
pontrr (0) T3 pous 00.8) partal siew (1.9) unaiter (0.99
object Nocking (1 sebeoteprcy (3 perss bllocking (8.5) mitiipt otgerts ohoiept--hgeeeee (4.5)
dulker (0.0), siskc D0, lareer (0.3s, pattons (03)
setaller 5S. oboect blocking 11 subcalzzeey (2.2), passen blocking (5.8)
food
egyla (h, targer (0.00,
hrightor (0.00, posr (o-1)
purrol siew ailcanogory (766 tectum (7.50
largir syle muttile otgens
shape ahoorppryy
darkor
DINO
aniiipi
begine:
able
(4.2, dasker
satand object
beigltes (0.0) (0.5), puntial view (0.9)
largr (2.7) darker (2.7) subcattaocy (3.2), object bocking (5.3)
other
posr (0.7)
eabeniegsey (hill, mulipp objects (2 Dt ayls (2 penson Nocking (1i)
petratz
pontol view (0.0s langer (000) style (0.00, pos (0.1)
objoc blockng
schostepery c5 stope (5 1t loctim
saate
mullippe stjects
pose 10.8) putten (1.0)
sabeategoc partal
peoco Modking
object bloking (3.3)
blockking
wchicle
blacking (0.0s souare (9.0), muliple obrects (0.0) putien (0.5)
smaller 00 15 darker (3 0) shape (3.0
soec
beghher (0U0), patem (1.)), pos (009) c1.)
larger SS. agee socure (130 objon Nodkny (2.5)
wheele whille
object blekking (0 pesson blocking (M secur 10.0) style 00.00
stape mnlter A duiker 6 largn (4.9)
hird
10.0, auligle objects (0.0), beighter (0.0)
Mockay
Modkang (31.5) (11.5)
commoniit
poten (0.G) person Mocking (.5), pose (0.5)
testure 2 bager (2.6) biggher (5.T) ayte (3.3)
coczring
peaten beger (0.5% pose 00.50 person Nanding (009
smaler St iccure (1.5) abject blacking (2.0s ayle:1333
desice
potern 6o pential view (0.7% pose (USS) ootor 11.00
numpple ohgeets somure 2.5 objest Modkig (2.3) bitghter (2.5)
dog
pottrn sosture 10.8) pose
portial (0.8)
abject Mocking
subeoregey
peros blocking
extiipt oljoess44.0)
fool
larger (0.0) atyle 10.0) brighier (0.00, pasr (0.4)
shape (LS)) adhoaepory 12.9) darker (2 9) iostare (2.9)
fimbbure
object hlokknng 01.01, larger (1.0o, mastigle otgeces (0.0) styte (0.8)
enalle (1 6a sextan (1 geose Stockig (3.0) (3.1
ResNer50
he ghaer (b.l powe (1.6) patern (0.3. birger (1)))
stape
ducker whorggrr subestoeery
multiple ohjers 50, d5.eer.5.. (5.8)
sstani object
begittes (Cil), pattem (0.2) poss (0.5)
Mocking (5.7)
miiliple
(2.3)
blocking (3.5)
perratz
(0.5)
v0.0), pou: (0.80
41. eaheatogory et shaps (444
stsee
derler (h, malipp -bpers (l.t). pour 05.85 patzm (1.9)
testons 12 D dhpe (L.) object Nooking (1.7) panum blocking (52)
squccuu
beghts (0.0) pattem (0.7). pose (0.5) coler (0.5)
testuee anyl (2.5) abhonggryy (2.5) aulliaee oljors (2.5)
vehide
pesson bocking (0.0),
10.0) multipe objects poters (0.5)
sabeutzpoyy stape
wheckel vchicle
iesture 10.5) bngghr (0.0) (0.0)
(044)
(2.5) object blacking (2.55 leyer (3A), blocking (5
bird
pesson blocking (0.0) 2ocas 10.0). laper DJ nultille objecs (8.)
partal view 11.8s andler (1.)) shape (5.7) docd blocking (5.7)
communiit
blocking (6.71 putter (1S). shage (0:8)
5t
12.5)
pasa
desicz
larger (017) pontal view (0-5), putzom
(049)
haacking 11
loxtem (2.1) abject blacking (2.5)
dog
muliiple objects (0.0), pantial view (0.5), pose (0.5)
dbject bocking (1.), daleer 4) subootogec T), penos blocking (2.9)
daaleer (000), style (1.00, pottem (1.6) mutiiple olject (0.8)
object Rocking (.5) subeoreecc (1 5) tommce (2.7) pesson bodkig (3.0)
food
(b,0), larger (0.0), eyle 10.0) pone (0u)
smaller no, blocking 2200 sabcalcesyy (2.1) Insture 12.2)
ubjoch (600) (0.7)
StoCLR
Irsect
(0.6) b0ger pose
eubeategeyy
muligle
(38), d
ratinl otject
he ghaer 05.05 poso (11 fo beckgroend (1.0)
larger 80. wheaggey (3.) darker (377, object Mocking (3.7)
ether
pase 10U7) brizkter (0.8)
subesiczoey (1.7, shape (1.7) soour (L.). poisa blocking (2.7)
péinatz
larnce abde 10.0% pantall (0.5), darie: (6:8)
smaller 20, subeatgoory
testure
pows
Nicking
onamer
glaer (0.0), derker (3.) orker (044
objon bloking paron blacking
ochaatgoryy (2.)) testom (1.7)
velidl
pesson blocking (0.0) Jocure 10.0) mullille obeets (L.D) penteaa (0.7)
object Hooking (1.5) dalerr (2.4) stble (2.4) shape 12.4)
vescel
be glgee (0.0) dakker (0.5) style (0.6) pateem (0.1)
sttaller (1 21, pottial viek (144 tomae (1 5) dijec Mooking (1.9)
whookal vchiile
shyos blecking (0.0), bryther (00% patum S), shape (0.5)
laryer (2.5) pencon Hockone (13) asyle (14), reatre (3.8)
Nit
booking (0100 object booking (10) leger (000 mamer (1
esheatggey deser moleer shape (15-4)
contrndit
begiee (C.0) pattem (0.3), 100% peress blocking (0.9)
testure (1.s lager (1.) dakerr (2.1) ubde (3.5)
cokcring
petson blokkin (0.5), paimaa 10.7) patial view (0) SX, youd (0.)
subesicaccy (1.), stalle (2.0) style (2 7) obgect bloking (2.9)
dester
he ghaer (bul) pattm (1.2) pantal vice (07% pose by
maliipl abyicsc object blocking (2:6)
dog
powe
penon blocking
(0.7
90 subeatogery
object blocking (2.0)
food
larger (0.00, wighher (1.00, style (1.0), pose (0.5)
dakker (1.7) smalle (1 sibcitegery (2.3). testue (2.4)
fumiuue
obyos blecking (ll lange (1,0) mattpple objocs (000) stds (Di)
testune (1.3 auhcaiggry 2t endler 11 darker (6.1)
texture 10.0% brightr (0.0), mulliple abposs (600 posc (0.7)
larger aznaatoeryy
ayle darker (2.1)
ratind ofject
portial
panera
pose (8.0s
beighter sehooroocy sibeatogeyy (2.5.
otject
blocking (3.0)
peinate
largee (0.0s pantal view (04)) pase (0.7)
dunker (405 teature (4.0 atye (4.) aubcaigoory (4.6)
stake
muliiple ebjecis (D.O), pouc (0.3)
shape 3.3% pensen blecking 4), object bookkng (34): dakker (3.4)
porson blodking (I io. patuens (0-7% pertol Rt color (0.8)
trstom 13 aahcanggryy St eyle 23.5 mulippl onjarss (7.5)
machiple
(1.0),
sougs (0.0)
(0.01,
subeutegoy
jestare
koollag (2.9)
whczled velicle
shape (000) Iniglaer (1.O) style (0.0) patieca (0.6)
dbject Hocking (2.6) larger (3.5) person Booking (5.2) testaee (5.2)
Table 4: Best and worst 4 factors for DINO, ResNet50, SimCLR and ViT by Metaclass. The values between parenthesis are the error ratios for the associated factor.
24


Reviews:
[{"review_id": "TwoI3xbFRzr", "reviewer": "Reviewer_cwD7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces Imagenet-X a set of meta-annotations on the validation set of imagenet,  plus 12.000 randomly selected images from the imagenet training dataset. These meta-annotations provide information across 16 different attributes such as pose, brightness, occlusions, etc. Then they performed a systematic study comparing 2200 different models. ", "summary_of_the_review": "Overall, I think this paper introduces new angles to look at imagenet, and I think the effort of collecting more data is always appreciated. I feel though that there is a missing opportunity in how the frame the paper and it seems to provide only limited reasons to use the dataset for improving model selections or training routines.   This seems to restrict the novelty of the paper and provide conclusions that are not that novel. That said I think there is potential if they take their data and all the models that they have diligently gathered and find other directions of usage, focusing more on the model selection than on the data diet as the current paper strongly leans to.\n\n------- Post Rebutal -----\n\nI would like to thank the authors for addressing my concerns and providing a new set of experiments in the questions I formulated. I think with these edits this paper shows different applications of the dataset and has a stronger message. I believe the community would benefit from the availability of this dataset. I am raising the score for the paper. ", "strength_and_weaknesses": "Strengths: \n\n* They introduced a new set of meta-annotations on the validation set of imagenet. I think there is a lot of value and effort in providing new datasets that can inspire new angles in evaluating new models. \n* They can show that some data augmentation techniques can improve across a dimension but affect performance among others. \n\nWeakness: \n\n* They claim in the introduction: \" A hurdle to research progress is understanding not just that, but also why model failures occur.\" ... However, from the annotations presented and their results is hard to see how to motivate changes in the architecture itself, most of the evaluations seem to be in the data domain. Which seems to be a missing opportunity for the usage of all the data collected and the models evaluated. For instance, why not inform what errors are introduced in a transformer like architectures vs convolutional ones?  is there any impact on the deepness and certain errors presented by the networks? Is there any influence on the number of parameters and some errors? Activation functions, etc, etc. \n* Another missing opportunity, in my opinion, is that they could provide ideas of how to select models given the applications the models can have. \n* It was not entirely clear from the paper how to use the dataset to improve the modeling.  Perhaps a little more intuition in the expected usage of the dataset can provide much more elements to measure the potential impact. So far the paper only seems to address the issues from the data perspective, but not clear how these extra annotations can be used to improve model/architecture choice. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IKn027DmSAO", "reviewer": "Reviewer_HVUi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces ***ImageNet-X***, which is a set of human annotations of the ImageNet validation set. It aims to ***explain the model mistakes/predictions*** from detailed visual attributes such as background, object pose, and lighting condition. On ImageNet-X, this work studies 2,200 models with different architectures, learning paradigms, and training procedures. Then, this work gives some observations. For example, all models fail consistently across ImageNet-X categories; dataset augmentation can improve robustness; diverse and more training data would be helpful to learn robust models.", "summary_of_the_review": "***The strengths of this work outweigh the weaknesses***\n- First, using 16 visual factors to explain model failures is interesting and reasonable. \n- Second, ImageNet-X is well-constructed and clearly illustrated. \n- Third, observations and analyses are helpful. \n\n--- ***Post Rebuttal*** ---\n\nThis work provides annotations on ImageNet to study failure cases of models. The paper ***provides several interesting and useful insights***, such as the visual attributes that lead to misclassification, the importance of each visual attribute for out-of-distribution datasets. Therefore, ***I vote to accept*** and look forward to a discussion with other reviewers and ACs.", "strength_and_weaknesses": "**[Strength]**\n\n- The motivation is good, the idea is interesting, and the construction of ImageNet-X is reasonable and clearly presented. \n- The 16 attributes/categories are good. Figure 1 is very clear and helpful. I like the idea of dividing visual changes into several categories.\n- Prototypical image defined with ResNet-15 is clever (Section 2).\n- Selecting the top factor per image is also a smart idea\n-The observations probing model robustness are clear. I would point out that I like the analysis of \"is accuracy enough?\".  \n\n**[Weakness]**\n- The major question would be ImageNet-X labels the factor on ImageNet validation set, which is an in-distribution dataset. If the annotations on some out-of-distribution datasets (e.g., ImageNet-R and ImageNet-A) could be provided, then this work would be super solid and strong. But it is also fine to provide annotations on ImageNet-Val.\n\n- Please clarify some details. ***1)*** what is the \"1-word difference\" in Figure 3? \n***2)*** why does \"person blocking\" affect the models (shown in Fig.4)? One guess is that models learn person-related features to classify the input image, which is undesirable. Please comment on this. ***3)*** \"Model weaknesses coincide with labeling errors\" is not very clear to me. Why this observation offers a potential explanation for the model biases? Please clarify this. My guess is \"the labels contain errors, so the models fit these errors and thus show some consistent bias\". Please comment on this. ***4)*** some related works [a-c]might be useful to illustrate the effect of data augmentation. (Just a small suggestion for reference)\n\n    [a] Deng, W., Gould, S. and Zheng, L., 2022. On the Strong Correlation Between Model Invariance and Generalization. NeurIPS, 2022\n\n    [b] Mintun, E., Kirillov, A. and Xie, S., 2021. On interaction between augmentations and corruptions in natural corruption robustness.NeurIPS, 2021\n\n    [c] Hendrycks, Dan, et al. \"The many faces of robustness: A critical analysis of out-of-distribution generalization.\" ICCV, 2021\n\n--- ***Post Rebuttal*** ---\n\nThanks for providing the rebuttal, which nicely addressed most of my concerns. Therefore, I vote for acceptance and would like to have a discussion with other reviewers and ACs.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "GZ6-M-8Mc-", "reviewer": "Reviewer_8YJy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an enhanced ImageNet validation dataset (ImageNet-X) with a more detailed labeling about variation factors of an object instance against its class. Each validation image has been labeled by human editor with 16 different variation factors including pose, background, color and etc.. Hundreds of image classification models have been evaluated on this new enhanced dataset to provide a detailed view towards model failure modes in terms of these variation factors.  ", "summary_of_the_review": "I think this paper presents a scientific point of view to analyze existing recognition methods and contributes a useful enhanced ImageNet variant for a potentially more diverse purpose. Some critical questions (in weakness section) related to using this new approach/dataset may be also discussed/addressed in detail to solidify the value of this work. \n\n-- Post Rebuttal --\n\nThe authors' responses address most of my concerns. The new experiment results are good examples to show useful applications downstream. It would be nice that the authors can move this new additional results to the main paper. Given the current form of the submission, I am leaning towards acceptance.  ", "strength_and_weaknesses": "Strengths:\n    \n- This work is well motivated and presented. It proposes a systematic way to label dominant variation type of an instance image against the corresponding canonical class pattern. This is in turn used to provide a concrete picture of model error modes in different classes and across different methods. \n    \n- A large number of training models have been evaluated in the new benchmark and some technical insights have been revealed quantitatively, though previous work is already aware of many empirical observations (e.g. Figure 6).\n\nWeaknesses:\n     \n- It is not clear whether the additional variation factor label can benefit training and further improve the generalization of existing models. The authors have also labeled some training images but they did not show how these newly labeled training images could boost performance. \n\n- In general, some data is hard to be labeled with only a single dominant factor or it can be perceived differently by different editors. This work did not discuss in detail about how to resolve label inconsistency and how to utilize multi-factor setting generally. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%