Trivial or Impossible --- dichotomous data difficulty masks model differences (on ImageNet and beyond)
Authors: Kristof Meding
Luca M. Schulze Buschoff
Robert Geirhos
Felix A. Wichmann
Venue: ICLR 2022 Poster (2022)
Keywords: CNNs
Cognitive Science
Vision Science
Psychophysics
Neuroscience
Visual perception
Inductive bias
ImageNet
CIFAR
RSA
Representation similarity analysis
Error consistency
Datasets

Abstract: "The power of a generalization system follows directly from its biases" (Mitchell 1980). Today, CNNs are incredibly powerful generalisation systems---but to what degree have we understood how their inductive bias influences model decisions? We here attempt to disentangle the various aspects that determine how a model decides. In particular, we ask: what makes one model decide differently from another? In a meticulously controlled setting, we find that (1.) irrespective of the network architecture or objective (e.g. self-supervised, semi-supervised, vision transformers, recurrent models) all models end up with a similar decision boundary. (2.) To understand these findings, we analysed model decisions on the ImageNet validation set from epoch to epoch and image by image. We find that the ImageNet validation set, among others, suffers from dichotomous data difficulty (DDD): For the range of investigated models and their accuracies, it is dominated by 46.0% "trivial" and 11.5% "impossible" images (beyond label errors). Only 42.5%  of the images could possibly be responsible for the differences between two models' decision boundaries. (3.) Only removing the "impossible" and "trivial" images allows us to see pronounced differences between models. (4.) Humans are highly accurate at predicting which images are "trivial" and "impossible" for CNNs (81.4%). This implies that in future comparisons of brains, machines and behaviour, much may be gained from investigating the decisive role of images and the distribution of their difficulties.

Published as a conference paper at ICLR 2022
TRIVIAL OR IMPOSIBLEE-DICHTOONOO DATA DIFFICULTY MASKS MODEL DIFFERENCES (ON IMAGENET AND BEYOND)
Kristof Meding"
Luca M. Schulze Buschoff+
University of Tübingen
University of Tübingen
kristof meding@uni -tweunnggrteeeeerr...... de
luca.schules bacc..schleee-bucccofffffeeeee
uni mnii-tuebingen. de
Robert Geirhos
Felix A. Wichmann
University of Tübingen & IMPRS-IS
University of Tübingen
ABSTRACT
"The power of generalization system follows directly from its biases" " (Mitchell 1980). Today, CNNs are incredibly powerful generalisation systems-but what degree have we understood how their inductive bias influences model decisions? We here attempt to disentangle the various aspects that determine how a model decides. In particular, ask: what makes one model decide differently from another? meticulously controlled setting, we find that (1.) irrespective of the network architecture objective (e.g. self-supervised, semi-supervised, vision transformers, recurrent models) all models end up making similar decisions. (2.) To understand these findings, analysed model decisions the ImageNet validation set from epoch to epoch and image by image. We find that the ImageNet validation set, among others, suffers from dichotomous data difficulty (DDD): For the range of investigated models and their accuracies, it is dominated by 46.0% "trivial" and 1.5% "impossible" images (beyond label errors). Only 42.5% of the images could possibly be responsible for the differences between two models' decision boundaries. (3.) Only removing the "impossible" and "trivial" images allows us see pronounced differences between models. (4.) Humans are highly accurate at predicting which images are "trivial" and "impossible" for CNNs (81.4%). This implies that in future comparisons of brains, machines and behaviour, much may be gained from investigating the decisive role of images and the distribution their difficulties.
1
2
3
4
5
6
foooo
UK
Figure 1: Can you predict which of these images are "tricky" for CNNs? Out of every of the six pairs, one image is always correctly classified and one always incorrectly (answers on the next pagq"). On ImageNet, image difficulty appears largely dichotomous: CNNs make highly systematic errors irrespective of inductive bias (architecture, optimiser. ...). Humans can reliably differentiate between images that are "trivially easy" and "impossibly hard" for CNNs (81.4% accuracy).
joint first authors in alphahetical order; "corresponding author

---

Published as a conference paper at ICLR 2022
0.7
Images from validation set
0.6
Label errors
0.6
Binomial distribution
Fraction of images
0.5
Fraction of images
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
None
10
None 3 4 5
Number models that classify image correctly
Number of models that classify image correctly
(a) ResNet- 18 variants
(b) State-of of-e---rtttttt models
Figure 2: Dichotomous Data Difficulty (DDD) in nutshell: Irrespective of model differences (e.g- architecture, hyperparameters, optimizer), most ImageNet validation images are either "trivial" (in the sense that all models classify them correctly) or "impossible" (all models make an error). This dichotomous difficulty masks underlying differences between models (as we will show later), and affects the majority of the ImageNet dataset i.e. not only images with label errors (red) as identified by the cleanlab package (Northcutt et al.) (021a). For comparison, a binomial distribution of errors is shown in green: this is the distribution errors expected for completely independent models all images were equally difficult.
INTRODUCTION
Let's play game call Find those tricky images! In Figure show pairs of images. One image impossible for a CNN regardless of its architecture, optimiser, random seed etc. never gets the label correct. The other image always yields a correct classification you find the tricky images?
Done? We will wait. You have probably never seen these images before, and neither have CNNs seen them during training. How exactly decision maker-be neural network, or biological brain-generaliset to previously unseen images is influenced by the decision maker's ' inductive bias (Goyal and Bengio, 2020)- in fact, as already recognised in 1980, "the power of generalisation system follows directly from its biases" (Mitchell) 1980). Commonly, the inductive bias is defined as set of assumptions and choices that determine which hypothesis space available to the model, before the model is exposed to data. For instance, starting from the set of all possible hypotheses, the hypothesis space of linear models a tiny subset (lincarity is one example of a strong inductive bias). After the "choice" of the inductive bias, the dataset then influences which particular decision boundary (or concrete hypothesis) is selected from the model's 's hypothesis space. Finding the right inductive bias for a given problem is at the core of machine learning. Therefore it is only consequent that a tremendous amount of work is being invested in improved architectures (Alzubaidi et al. (021]. optimisers (Ruder 2016), learning would expect these choices to make a difference on the resulting model's decisions even if trained on the exact same dataset. However, in the present work, we have tested various factors related the inductive bias-amon, other aspects, architecture. optimiser, learning rate, and initialisation and yet, on ImageNet, all models agree in the sense that they all make largely similar errors. This is shown in Figure even radically different state-of-the-art (SOTA) models make surprisingly similar errors on the ImageNet validation set. To certain degree, image difficulty appears dichotomous: nearly 60% of all images are either "trivial" (all models correct) "impossible" (all models wrong). As we will demonstrate later, this dataset issue masks and overshadows hidden differences between models.
and Hutter (2016), etc. etc.-surely
'The tricky (=misclassified) images are: 1. bottom, 2. bottom, 3. top, 4. top, 5. bottom, 6. top. This game is homage to "Name that dataset" by Torralaa and Eiros 2011).
2

---

Published as a conference paper at ICLR 2022
1.1 RELATED WORK
Metries for CNN comparisons Given the scientific, practical and engineering implications of model inductive biases, it is perhaps not surprising that many studies investigated differences between neural networks. For this purpose, the standard metric is accuracy, but some studies also focus on learned features and decision boundaries (c.g. Heemman and Lampinen 2020, Nguyen al. [2200] Wang et al. 2018 Hermann et al. 2019 Shah et al. 2020). or internal representations (Kriegeskorte et al., 2008; Kornblith et al. 2019) Using representational similarity analysis (RSA) and most similar to our work, |Meheer et al. (2000) and Akbarriiaa and Gegenfurtner (2019) investigated whether different CNNs yield correlated representations and found that many neural networks show differences representational level. How intermediate representations largely remains unclear. In order to compare networks on a behavioural level directly, metrics such as error consistency can be used. Error consistency (measured by λ) assesses the degree of agreement between two decision-makers on an image-by-image basis, not just average performance (Geirhos) al. 2020a b).
related to classification behaviour
Consistent model errors Tramer et al. (2017) observe that the decision boundaries of two models are highly similar, an issue that is related to the transfcrability of adversarial examples between models. Additionally, it has been shown that standard vanilla models systematically agree on their errors both on IID (independent and identically distributed) data (Mania et al.] 2019) and OOD distiff-iistibbution)) data (Geirhos et al.) 2020a). It is unclear whether, if at all, there is connection between model inductive bias, dataset difficulty and consistent model errors. Another line of work investigated fairness metric consistency (Qian al. 2021).
Relationship between DDD and OOD Evaluating models on nut-oo-disttrbutiin (OOD) datasets is an important way to differentiate between models. However, different models show highly consistent errors even when evaluated on OOD data, according to Geirhos et al. (2020a) Figure 3). Here, CNN-to-CNN consistency is at .62, .48 and .67 depending on the OOD dataset, which is closer to perfect consistency than it is chance level. Therefore, while OOD data can distinguish models in terms of overall accuracy, OOD testing is insufficient in terms of revealing deeper differences overshadowed by dichotomous data difficulty (DDD). Furthermore, OOD datasets might also exhibit DDD. Looking forward, OOD testing as well as curating datasets without DDD are not mutually exclusive and should be employed in combined fashion for comprehensive understanding of model similarities and differences.
Problems of datasets The ImageNet dataset (Russakovsky et al.] 2015) has numerous issues, including some that affect most datasets, like dataset bias (Torralba and Efros 2011). Northcutt et al.) (202Tb) showed that around 6% of ImageNet validation images suffer from label errors. Additionally, many images require more than a single label since multiple objects are present, and the distinctions between classes seem rather arbitrary at times (Tsipras et al., 2020 Beyer et al. 2020). Even when trying to replicate the original ImageNet labeling procedure in order to create a new test set, models trained on ImageNet have an accuracy drop of 11-14% on this new test set (Recht et al., (019). Finally, ImageNet labels are based on the WordNet hierarchy, which contains many problematic categories. For instance, many categories in the "person" subtree have labels ranging from outdated to outrageous and racist (Crawford and Paglen; Yang et al. 2020). Furthermore and similar to our work, authors already investigated image sampling strategies during training (Jiang et al.) 2019 Katharopoulos and Fleuret, 2018). However, these studies focused on accelerating the training and not how the Net issues may obscure differences between models
explore here.
Example difficulty A number of papers have investigated what makes images easy or difficult-e.2 e.g. Agarwal al. (2020); [Baldokk et al. (202T); Mangalam and Prabhu (2019); Paul et al. (202T) for MNIST/CIFAR, and e.g. Hacohen et al. (2020) for ImageNet. Additionally, it is well-known that models often make similar errors and often learn examples in the same order, see e.g. Toneva et al.
(2018): Kalimeris al. (2019).
To summarise: it was clear that there are easier and harder images and that models often make similar errors. However, the relationship between these two findings has not commonly been recognised. We here show for the first time the implication thereof: That underlying model differences are masked by dichotomous data difficulty.
3

---

Published as a conference paper at ICLR 2022
2
METHODS
All details regarding our software, hardware and dataset can be found the appendix (section A.2 Similarity measure For the investigation of network similarities, mainly use the behavioural measure error consistency (x) (Geirhos et al. (020aa) based on Cohen's work (Cohen 1960). A > indicates that two socision-makenn systematically make emrors the same images: K 0 indicates no more error overlap than what could be expected by chance alone. R < implies that two decision- makers systematically disagree.
Network variations In our experiments we investigated the systematic agreement between CNNs, varying not only architecture but carefully controlling for number of epochs, optimiser, batch size, random initialisation, learning rate, hardware randomness, data order, architecture, and disjoint data sampling. Unless stated otherwise, we only changed one of the above parameters at time. Our main results are based on the ImageNet ILSVRC dataset (Russakovsky et al., 2015). We first used systematic variations on ResNet- 18 (called ResNet- variants). Details can be found in section A..] in the Appendix. In total, 30 networks were trained on each of the three data sets (See below: ImageNet, CIFAR-I00, Gaussian) presented the main text, as well as 60 more networks for control experiments reported in the Appendix. We stored all network states and all responses for each epoch. This allows us to analyse the agreement on different training stages epoch by epoch (and image by image). Later, we investigated different state-of- the-art network architectures. When investigated these SOTA models, implementations provided by mode 1 Ivshuman (Geirhos et al.] (021) were used (which focuses various out-of-distributiin datasets but not ImageNet do). Psychophysical experiment We conducted two psychophysical experiments. For both experiments detailed methods can be found in section A. in the appendix.
3
RESULTS
0.8
Error Consistency
0.6
0.4
Ease condition
plus
plus
Different sptimizer
Different
Different
Different
CUDA
Different dataorder
Different architecture
Half
Combined
Different data
lep
10ep
batchaize
initielisation
nondetermniste
LA
data
Figure 3: Error consistencies between the different conditions and the base network on the ImageNet validation set after 90 epochs. For conditions for which multiple models were trained the mean over all models of a condition is plotted in black.
3.1 MODEL ERRORS ARE ALIGNED DUE TO DICHOTOMOUS DATA DIFFICULTY (DDD)
Figure 3 shows the result of our controlled study of model differences ImageNet. A positive error consistency score means that the networks agree beyond what is expected by independent models. Regardless of the parameter changed (architecture, optimiser, etc.), find very high error consistencies (around 0.7)-thus thus all models agree which images are easy or difficult to classify irrespective of the model differences investigated Strikingly, changes that we hypothesized would
In addition to the mean across several runs. we also plot the consistencies of single runs in gray. However, these are non-visible since the variance within the conditions is very small (except, perhaps, for the "cuda nondeterministic condition").
4

---

Published as a conference paper at ICLR 2022
make larger difference, e.g. different architecture, have basically the same error consistency as "minor" changes like enabling hardware randomness on the GPUs. All networks achieved similar top- 1 accuracies (mean: 69.05% after 90 epochs; range: 65.87% to 71.47%; standard deviation: 1.60%, cf. Figure 13 in the Appendix). Another popular method for agreement analysis is RSA. All our results also hold here, see Figure i the Appendix. Additionally, switching the base architecture VGG-11 or DenseNet-121 does not make a difference either (see Figures 15 and [
in the Appendix). deeper analysis of single network's 's decisions can be found in section A.5 in the appendix.
The findings from Figure 3 becomes even more prominent in @ we overlay the previous figure for all of the 13 networks with different hyperparameters, architectures etc. (explained in section 2 A very light red entry indicates that all networks correctly classify the image, a very dark red entry that all networks misclassify the corresponding image; shades of red indicate the cases in-between (where, e.g., some but not all networks make errors). The figure illustrates that the previous findings hold across very different inductive biases for ResNet- 18 variants: We observe that 48.2% % images are learned by all models regardless of their inductive bias; 14.3 % images are consistently misclassified by all models only roughly a third (37.5%) of images are responsible for the differences between two model's decisions. We call this phenomenon dichotomous data difficulty (DDD): While the inductive bias restricts the hyperparameter space for a given model, the nature of the Mataset-and especially its highly non-uniform image difficulties-seemt to be important cause for the high similarity in the decisions of different networks. Model differences may play a bigger role for images of intermediate difficulty-where there is substantial consistency variation across models-but only minor role for easy and hard images, As the dataset primarily consists of images that all models either classify correctly or incorrectly, all models end up with similar classification behaviour.
Let us consider two extreme cases in order to put these findings into context. On one end of the spectrum, if all images were equally difficult and if all networks were independent (i.e. their different inductive biases would result in independent decision boundaries), then we could expect binomial distribution of model errors: out of 13 investigated ResNet-18 18 models, very few images should be misclassified by all models and very few correctly classified by all models-instcad, most images should be correctly classified by a handful of models. Figure 2h shows, in green, exactly this distribution expected for independent models and equally difficult data [ On the extreme end of the spectrum, if the inductive bias had no influence at all and the dataset only contained "trivial" and "impossible" images, we would expect a histogram with only two "spikes": given ImageNet accuracies of 69.05% on average, one spike at "None" (30.95% for ImageNet) and one at "All" (69.05 % for ImageNet). Clearly, the empirically obtained histogram (blue) much more resembles the latter, i.c. the scenario where the (nearly) dichotomous data difficulty dominates over inductive bias. We observe that DDD on ImageNet is amplified. but not caused, by label errors (Northcutt et al.) 202 Beyer et al. 2020 Tsipras et al., 2020) which only have minor influence on the "None-Bar from our histogram in Figure Hence: removing erroneous labels is beneficial and laudable, but it will not solve DDD.
Is dichotomous data difficulty (DDD) only a problem for ImageNet? This is not the case: DDD is also present in CIFAR-100 and the synthetic Gaussian dataset we (purposefully) generated. As a first indication, for both of these datasets we find similarly high error consistencies between different models, just like we found for ImageNet (see section A.6|in the appendix).
3.2 DICHOTOMOUS DATA DIFFICULTY EVEN AFFECTS RADICALLY DIFFERENT STATE-OF-THE-ARU MODELS
In the previous section, we found that changing different aspects within one model class does not change the decisions significantly. However, is unclear whether these results generalize across model classes. Therefore, we apply the analysis from Figure Ph with a number of models specifically chosen be radically different from each: self-supervised model (SimCLR, Chen et al. (2020)) semi-sun cumi-super model (SWSL, Yalniz et al. (0019). a vision transformer (VTT, Dosovitskiyy et al. (2020). ) a recurrent model (CORnet-Ril, Kubilius et al. (2019)) very deep model (ResNet-152.
Of course, these numbers change if one uses an architecture with higher top-1 accuracy, see next section A As one of our reviewers suggested, we also tested model in which the image difficulty is decaying exponentially-men images are simple, then less and less are more difficult. However, this also does not lead to DDD like distribution (see A.9.
5

---

Published as a conference paper at ICLR 2022
"Trivial"
Image from validation set
"in-between"
"Impossible"
10
35
40
45
65 70
75
Epoch
Figure 4: Decisions on all 50K ImageNet validation images of all 13 networks with different inductive biases (architectures, ...). Dark red indicates that the respective item falsely classified by all networks. Light red indicates that the image was correctly classified by all networks. Images are ordered according the mean accuracy across networks in the last epoch.
He et al. () highly compressed model (SqueezeNet, landola al. (2016)) adversarially trained model (ResNet-50 with epsilon L22-rbuutnesss on ImageNet, Salman et al. (2020),
aag-of-local-featuee model (Bagnet-33, Brendel and Bethge (2099) ) network trained on stylized ImageNet (ResNet-50) trained on SIN, Geirhos et al. (2019)), ), deep high resolution neural network (HRNET, Wang et al. (2200), and OpenAl's CLIP model (Radford al. [02T]) with a transformer architecture and joint image-language training objective (11 models in total). Individual accuracies of these networks can be found in the appendix (see Figure 14). Again, we find the same pattern in Figure Pb. In total, 46.0 % "trivial" images are learned by all except one model; 11.5 % "impossible" images are consistently misclassified by all except one model. (42.5%) of images are responsible for the differences between two model's decisions.
3.3 DATASET SUBSAMPLING ACCORDING DICHOTOMOUS DATA DIFFIULLTY REVEALS DIFFERENCES BETWEEN MODELS
So far we have that models agree despite markedly different choices of architecture, training objectives, and many other aspects. While we hypothesized DDD a dataset problem- to be the cause, alternative explanation would be that models simply agree irrespective of the choice of data difficulty. order to differentiate between these two competing hypotheses we performed an experiment where removed both the "trivial" and the "impossible" images from the ImageNet

---

Published as a conference paper at ICLR 2022
Original ImageNet validation-set
Subsampling: Only in-between images
a
Base condition
Les
exn
1:70 ont ono ans ans san dan
DIN
CO0
0:0
020
038
033
504
811
Plus lep
ax7
cin can oen oan dan san Eoe
LCO
C:2
OX1
030
030
011
810
018
Plus 10ep
5:0
can oan aee den s4n
5.0
010
033
023
030
013
513
015
Different optimizer
em
6M
1:0
(:99
0n0 675 075 544 6.0
035
1:0
0.5
0.1
0.5
934
536
510
810
Different batchsize
8:6
6:0
1.00
0.00 975 079
148 6.0
0.5
025
922
037
831
815
0.6
Different initialisation
K39
656
6.7
0.0
0.0 0.0 079 140 660
924
0.0
020
927
039
820
Different LR
0.0
0.0
930 979
8:0 600 600
029
020
027
029
335
820
10.4
CUDA nondeterministic
A12
0.0
070 079
sda
020
027
Different dataorder
0.2
Different architecture
Half
0.00
Combined condition
condition
Plus 10ep
Different optimizer
Different betchsize
Different initiallsatiion
Different LR
CUDA nondeformniistic
Different dataorder
Different architecture
Half data Combined condition
Plus 100p
Different optimizer
Different batchsize
Different initiallsation
Different LR
CUDA nondetorministic
Different detaorder
Different architecture
data Combined condition
Base condittion Plus
Plus
b
CLIP
1:0
338
6:02
037
3:13
0:30
1:30
FBLL
031
9:0
1.0
bagnet33
840
180
35L
045
030
8:08
611
5.0
0.8
hmet w44
8:6
8:0
040
O5T
0.5
36L
0.55
2:0
030
032
8:33
6:08
615
860
018
resnet152
reseet50 12 eps1
339
03T
035
0.5
022
resnet50 swsl
8:4
4:00
6
304
939
LM
055
0.64
4:42
8.09
031
030
034
K.14
410
340
-0.4
resnet50 trained SIN
simclr resnet50x1
-0.2
squeezenet1 1
8:7
6:57
1000
6.9
3m
FM
03T
200
-0.0
vit base patch16 224
035
0.66
039
CLIP
CORnet RT
bagnet33
hmet w44
resnet152
resnet50 eps1
resner50 swsl
resnet50 trained on SIN
simcir resm
squeezenet1
vit base patch16 224
CLIP
CORnet RT
bagget33
hrmst md4
resnet152
resnet50 12 eps]
resnet50 trained on SIN
simmir resnet50r3
squeezenet1]
vi base patch16 224
resnet50
Figure 5: Error consistency on the original ImageNet test-set (left panel) and on the test-set with in-between images only (right panel) for the ResNet-vvriants (a) and the SOTA networks (b). Both networks were trained on the whole ImageNet training set. Error consistency around 0 indicates independent responses. A diagonal element of 1 represents that only one network for comparison was available, otherwise the within condition consistency is calculated, see section 2
validation dataset. The training dataset was not altered. If model agreement is indeed caused by DDD, then we should find much stronger differences between different models (as indicated through lower error consistency scores). The results are presented in Figure [ Indeed, model differences are now much more pronounced, in many cases the consistency between different models even approaches zero, indicating that some networks make truly independent decisions, i.c. have learned independent decision boundaries whilst being similarly accurate their different inductive biases now show. This shows that the high agreement between different models (as observed e.g. by Geirhos et al. (2020a) 202I) and Mania et al. (2019) is a result of dataset DDD problems, not that inductive bias does not matter much. Please note that the reduced consistency is not trivially caused by the removal of impossible and trivial images: Even when removing extreme images (all models correct/incorree). two models could agree disagree the remaining images of intermediate difficulty (error consistency is calculated pair-wise), Finally, we show that there are some particularly easy and hard classes (Section A.8 in the Appendix).
7

---

Published as a conference paper at ICLR 2022
3.4 DIFFERENCES BETWEEN "TRIVIALS", "IMPOSSIBLES" AND "IN-BETI IN-BETWEENNS"
Since we found DDD to affect very different models, we were interested to understand the nature of their differences. To this end, we asked in our first experiment whether humans could identify which images were "trivial" and "impossible" for CNNs. If they can, this would mean that there is-at least some degrec-a shared notion of image difficulty between humans and CNNs. We therefore conducted a psychophysical experiment where subjects were asked to identify which of two images was easier for a neural network classify. We found that human observers were able to do so well beyond chance (50%): on average, with accuracy of 81%. The accuracies of the different subjects ranged from 72% 9 to 89%, with a standard deviation of 6.29%. The mean error consistency 0.59. For all combinations of different subjects, the enror consistency ranged from 0.41 to 0.75, with a standard deviation of 0.09. In conclusion, even naive human observers without machine learning experience can reliably and consistently predict which images are easy and difficult for CNNs. In the Appendix, we also show that the ImageNet "superclasses" are not equally distributed for the three image subsets (see section A. 12 and observe that label ambiguity does not seem to be the major cause for the emergence of "in betweens" (see section A.13).
between the subjects
Additionally, we conducted second follow-up human experiment which aims find decisive factors for the trivial, in-between and impossible images. We randomly chose 100 "trivial", "impossible" and "in-between" images, each of our eight human observers had answer three questions; Q1: "How many objects belonging to different categories are in the image?', Q2: "Is there a main category in the image?" Q3: "Is the presentation of the objects unusual in any manner?". The main results are summarised in Figure 6
Question
Question
Question
Relative image class propartion
1.0
0.8
0.6
Trivial Inbetween Impossible
0.4
0.7
0.0
2
Yes
Maybe
No
Slightly
Very
Number of categanies image
Is there main category?
Is the prosentation of objects add?
Figure 6: Barplot displaying the proportions of answers over all observers and for the three questions mentioned below the subfigures. Colour coded is the image subclass.
For this figure, we removed images which were found to have label errors by Northcutt et al. (202 and balanced the image classes have the same number of images (n 57 per class, otherwise our plots would be misleading if we do not have the same number of images per class). The bars are normalized so that the proportions of the different classes add up to for each answer category. We see a clear trend, that with increasing number of categories, increasing uncertainty about the presence of main category and increasing oddity of presentation, the proportion of "trivials" decreases while the proportion of "impossibles" increases. The Pearson correlation coefficient between the mean number of categories over all observers for each image and the respective mean model accuracy is -0.37. Furthermore, we found that items with clear main category had mean accuracy of 0.64. Items where observers indicated that they "maybe" had main category had a mean accuracy of 0.44 while items with no main category had mean accuracy of 0.37. Finally, items with normal presentation had a mean model accuracy of 0.61. Items with a slightly odd presentation had a mean model accuracy of 0.55 and items with very odd presentation had mean model accuracy of 0.34. We also find high consistencies between the observers, see section A. the appendix. For completeness, we show raw pooled data in Figure 26 25 (with/without label errors) as well as individual data with and without label errors in Figure 27 28 All our results also hold when we include images with label errors.
*Our experimental paradigm here is high- nigh-powered small N- N-Design (mith and Little 2018 Even our worst performing observer with 72% accuracy yields p-value of 5 10 for the null hypothesis of chance nerformance.
8

---

Published as a conference paper at ICLR 2022
In summary, experiment one shows that human can reliably and with high accuracy distinguish trivial and impossible image. Second, we found in our second experiment that the number of categories in the image, whether there is main category to contribute to whether an image belongs to the "trivial", "in-between" or "impossible" subsets.. While the factors investigated show a clear effect, it is evident that there is no single factor that completely explains the differences between the three subsets of images. We are clearly still in need of an explanation what makes images "trivial", "in-between" or "impossible" for neural networks. To a certain degree, this is perhaps not surprising. There is a long history of investigating the factors underlying image difficulty in vision research which we could not capture with our only three questions. Future studies might draw on this line of research and ideally explore more possible factors which could be related to the dichotomous difficulty inherent popular image datasets.
not, and the oddness of presentation all seem
DISCUSSION
We investigated the influence of dataset difficulty on model decisions. We found that model decisions not only determined by the inductive bias (such as their architecture) influenced by the dichotomous difficulty of images in common datasets (DDD): many ImageNet images are either "trivial" or "impossible", but only third in-between. This has implications for model design. Viewed positively, results for one network may generalise towards different networks, which can be advantageous in some circumstances. This is in line with previous findings that some results transfer between different model classes, e.g. adversarial examples (Szegedy et al. 2013 Papernot et al., 2016). However, if models are trained on datasets with DDD, design decisions like architectural improvements may not be able to show their full potential since the resulting models, due to DDD, have a high likelihood of ending up in a very similar decision- reeiime as other (already existing) models- andells-nd might even inherit their vulnerabilities. In comparison to underspecification described by D' Amour et al. (2020) we observe that models behave very similarly because of DDD. When removing trivial and impossible images, the differences between models are unmasked which potentially can be used to accelerate training (Jiang et al. 2019 Katharopoulos and Fleuret 2018). Furthermore, our method also offers an easy to implement method to curate DDD-free datasets. One only needs to train different models on the same data, followed by removing impossible and trivial images.
are even more
Previous investigations found label errors to be a problem in a number of datasets. Here we show a dataset issues that affects a much larger number of ImageNet images than those affected by label errors. In order to be able to improve our ability to differentiate between models and give their inductive bias chance to truly make a difference, we will need datasets that are more balanced with respect to image difficulty or use only in-between images. This is far from trivial since we do not know precisely what causes DDD. Inspiration could come from psychology and vision science, where investigations into what makes an image or object difficult have long history. At least since Eleanor Rosch's 's (Rosch] 1973) pioneering work we know that for some object categories there are "natural prototypes". i.c. particularly representative exemplars of category. Thus not all members of a category are equally easy to recognize and classify. Second, for human vision it is well known that the recognition of an object depends its viewpoint: objects are easier to see from a "canonical" viewpoint (Biederman) 1987 Bülthoff and Edelman| 1992: Freeman 1994] Tarr and Kriegman| 2001] Tarr al. (996). Third, object recognition also depends on its context and surroundings. Humans can recognize objects remarkably quickly (Thorpe et al. 1996). but this is only true if they are effectively segmented from their background by the photographer' selection of focus point, focal length and aperture (Wichmann et al., 2010). As result one can make a real-world dataset arbitrarily trivial (or impossible) for human observers. Perhaps it was somewhat naive to believe that large automatically generated datasets would "get the mix right" and result in images where the difficulty within and between categories is approximately the same.
Our human experiment shows that humans can reliably identify the impossible images from ImageNet (see Figure 18 in the Appendix for more examples). Inspection of those images left us with the impression that impossible images often contain multiple objects and sometimes "unusual" objects and viewpoints which is verified in our follow-up experiment. From a cognitive science or neuroscience perspective DDD might thus also provide new opportunities for insight: Perhaps the impossible images are the ones which can reveal differences between humans and CNNs and are thus those which neuroscience and cognitive science should be interested in?

---

Published as a conference paper at ICLR 2022
5 ETHICS STATEMENT
Potential social harm. We do not expect that our work causes harm to people or groups. Environmental aspects. We roughly used 250 GPU days for this paper. Each GPU unit cluster (together with CPU and RAM) consumes on average 300W. total, this paper consumed 1800kWh. The CO2 emission in the country of the authors is roughly 400g/kW resulting in CO2 equivalent of 720kg-this corresponds to roughly 45% of the emission of a flight from London to New York. We will compensate the amount of CO2 with a certified CO2-compensation company. Furthermore, we will make sure that other researchers have access to the trained models, see below. We can not distribute all models yet (several GBs) because of the size limit of the supplementary materials.
Psychophysical experiment. Prior to the experiment written consensus was collected from all participants. Recently, some issues around ImageNet were discussed e.g. by https::/ /www excavating aij Thus, we removed some images in our psychophysical experiment and do not show any images containing humans in this paper. Otherwise, we do not see potential participant risks in our experiment.
REPRODUCIBILITY STATEMENT
Code to reproduce our findings can be found on github: wichmann -tteriiiill----ioosss-ese--sssss
https://github.com com/
ACKNOWLEDGMENTS & FUNDING DISCLOSURE
Funding was provided, in part, by the Deutsche Forschungsgemeinschaaff (DFG, German Research Foundation)- project number 276693517 SFB 1233, TP Causal inference strategies in human vision (L.S.B, K.M. and F.A.W.). The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting R.G.; and the German Research Foundation through the Cluster of Excellence "Machine Learning -New Perspectives for Science", EXC 2064/1, project number 390727645, for supporting F.A. W. The authors declare no competing interests.
We would like to thank Silke Gramer, Leila Masri and Sara Sorce for administrative and the "Cloud- masters" of the MI--Cluster at University Tübingen for technical support. We thank the group of Ludwig Schmidt at UC Berkeley for discussions during early stages of this project and David- Elias Künstle for helpful comments on the manuscript. This work benefited considerably from conscientious peer-review: We would like to express our sincere gratitude to all our reviewers.
AUTHOR CONTRIBUTIONS
Motivated by previous work (Geirhos et al. 2020a), the project was initialized by K.M. and jointly developed forward with R.G. and F.A. W. Later, but still an early stage, L.S.B. joined the project L.S.B. wrote the code for computing and analysis and was supervised by K.M.; R.G and F.A. W. gave input during the entire project. R.G. and F.A. W. had the idea of the psychophysical experiment, All authors planned, structured and wrote the manuscript; all figures were made by L.S.B. based on joint discussion.
REFERENCES
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. arXiv preprint M.Xiv:2011110040 2020.
Tom M. Mitchell. The need for biases learning generalizations. 1980.
Laith Alzubaidi, Jinglan Zhang, Amjad J. Humaidi, Ayad Al-Dujaili, Ye Duan, Omran Al-Shamma, J Santamaria, Mohammed A. Fadhel, Muthana Al-Amidic, and Laith Farhan. Review of deep learning: concepts, cnn architectures, challenges, applications, future directions. Journal of big Data, 8(1):1-74, 2021.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.0000 1609.04747, 2016.
10

---

Published as a conference paper at ICLR 2022
Iyy Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 2016.
Curtis Northeutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2021a.
Antonio Torralba and Alexei A. Efros. Unbiased look dataset bias. In CVPR 201 pages 1521-1528. IECE, 2011.
Katherine L. Hermann and Andrew K. Lampinen. What shapes feature representations? exploring datasets, architectures, and training. arXiv preprint arXiv:2006. 12433. 2020.
Thao Nguyen, Maithra Raghu, and Simon Korbllihh. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010.15327, 2020.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hoperoft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. arXiv preprint arXiv:1810.1 11750 2018.
Katherine L. Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. arXiv preprint arXiv:1911.00000 2019.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli, The pitfalls of simplicity bias in neural networks. arXiv preprint arXiv:2006.07711 2020.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A. Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4, 2008
Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Leariing pages 3519-3529. PMLR, 2019.
Johannes Mehrer, Courtney J. Spoerer, Nikolaus Kriegeskorte, and Tim C. Kietzmann. Individual differences among deep neural network models. Nature communications, 11(1):1-1,, 2020.
Arash Akharinia and Karl R. Gegenfurtner. Paradox in deep neural networks: Similar yet different while diffcrent yet similar, arXiv preprint arXiv: arXiv: 0903,04772, 2019.
Robert Geirhos, Kristof Meding, and Felix A. Wichmann. Beyond accuracy: quantifying trial-hy-trial behaviour of CNNs and humans by measuring error consistency. Advances in Neural Information Processing Systems, 33, 2020a.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. On the surprising similarities between supervised and self-supervised models. arXiv preprint arXiv:2010.08377, 2020b.
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint 174.00377 2017.
Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. arXiv preprint arXiv: 1905. 105800000,,, 12580, 2019.
Shangshu Qian, Hung Pham, Thibaud Lutellier, Zeou Hu, Jungwon Kim, Lin Tan, Yaoliang Yu, Jiahao Chen, and Sameena Shah. Are my deep learning systems fair? an empirical study of fixed-seed training. Advances in Neural Information Processing Systems, 34, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International journal of computer vision, 11 (15())211-252) 2015.
Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2 14749, 2021b.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew llyas, and Aleksander Madry. From ImageNet to image classification: Contextualizing progress on benchmarks, In International Conference on Machine Learning, pages 9625-9635. PMLR, 2020.
Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with ImageNet? arXiv preprint arXiv:2006.07159 2020.

---

Published as a conference paper at ICLR 2022
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize PMLR, 2019.
to ImageNet? In International Conference on Machine Learning, pages
Kate Crawford and Trevor Paglen. Excavating AI: The politics of training sets for machine learning. https //ecavaa ing.ai7
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasels: Filtering and balancing the distribution of the people subtree in the ImageNet hicrarchy. In Proceedings of the 2020 Conference on Fairness, Accountabilityy and Transparency, pages 547 -47-558, 2020.
Angela H. Jiang, Daniel L.K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton. et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.0077
2019.
Angelos Katharopoulos and François Fleuret. Not all samples created equal: Deep learning with importance sampling. In International conference on machine learning, pages 2525-2534, PMLR, 2018.
Chirag Agarwal, Daniel D'souza, and Sara Hooker. Estimating example difficulty using variance of gradients.
arXiv preprint arXiv:2008. 1600, 2020.
Robert JN Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. arXiv preprint arXiv:2106..m 106.09647, 2021.
Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow learnable examples first? In Proceedings of the Workshop on Identifying and Understanding Deep Learning Phenomena at 36th International Conference on Machine Learning, 2019.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training, arXiv preprint arXiv:2 2021.
Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agree to agree: Neural networks share classifica- tion order real datasets. In International Conference on Machine Learning, pages 3950-3960. PMLR, 2020.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Comhes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:/812.05159, 2018.
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. SGD on neural networks learns functions of increasing complexity. Advances in Neural Information Processing Systems, 32:3496 350666 2019.
Jacob. Cohen. A coefficient of agreement for nominal scales. Educutional and psychological measurement. 20 (1):37-46, 1960.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. In Advonces Neural Information Processing Systems 34, 2021.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
I 7eki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXiv preprint arviv: / 09050005466 2019.
Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 2020.
Jonas Kubilius, Martin Schrimpf, Ha Hong. Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel K. Yamins, and James J. DiCarlo. Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs. In H. Wallach, H. Larochelle, A. Beygelzimer. F. D' Alché-Buc, Buc, E. Fox, and R. Garnett, editors, Neural Information Processing Systems 32, pages 12785 12796. Curran Associates, Inc., 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 770-778, 2016.
12

---

Published as a conference paper at ICLR 2022
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07366 2016.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? arXiv preprint arXiv:2007.084499 2020.
Wieland Brendel and Matthias Bethge. Approximating cnns with bag-of-local-featle models works surprisingly well on imagenet. arXiv preprint
2019.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wicland Brendel. Imagenct-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang. Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, al. Deep righ-resolution representation learning for visual recognition. IFEE transactions on pattern analysis and machine intelligence, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 00t:2103.00020 2021.
Philip I. Smith and Daniel R Little. Small is beautiful: defense of small-n design. Psychonomic bulletin review, 25(6):2083-21)) 2018.
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 13tXiv::312.611// 2013.
Nicolas Papernot, Patrick McDaniel, and lan Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint (tf::1605.0722 2016.
Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility modern machine learning, arXiv preprint 2rX1v:2011.03395, 2020.
Eleanor H. Rosch. Natural categories. Cognitive Psychology, 4(3):328-350, 1973.
Irving Biederman. 94(2):115-147, 1987.
cenntttii--yyyoommmmmtttt A theory of human image understanding. Psychological Review,
Heinrich H. Bülthoff and Shimon Edelman. Psychophysical support for two-dimensional view interpolation theory of object recognition. Proceedings the the National Academy Sciences, 89(1):60-66 64, 1992.
William T. Freeman. The generic viewpoint assumption in framework for visual perception. Nature, 368 (6471):542-545, 1994.
Michael Tarr and David J. Kriegman. What defines view? Vision Research, (1155):9911-0044 2001.
Michael J. Tarr, Heinrich H. Bülthoff, Marion Zabinski, and Volker Blanz. To What Extent Do Unique Parts Influence Recognition Across Changes in Viewpoint? Psychological Science, 8(4):282-289, 1996.
Simon Thorpe, Denis Fize, and Catherine Marlot. Speed of processing in the human visual system. Nature, 381 (6582):520-522, 1996.
Felix A. Wichmann, Jan Drewes, Pedro Rosas, and Karl R. Gegenfurtner. Animal detection in natural scenes: Critical features revisited. Journal of Vision, 10(4))6-6, 2010.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learnin.. In International conference on machine learning, pages 1139- 1147. PMLR. 2013.
Duncan Riach. Determinism deep learning. GPU Technology Conference, 2019.
Lukas Muttenthaler and Martin N. Hebart. Thingsvision: python toolbox for streamlining the extraction of activations from deep neural networks. bioRxiv preprint bioRxiv:2021.00 1.449999 2021.
Alex Krizhevsky and Geoffrey E, Hinton. Learning multiple layers of features from tiny images. 2009.
Felix Wichmann and Frank Jäkel. Methods in Psychophysics, pages 1-42. John Wiley & Sons, Inc, 2018.
13

---

Published as a conference paper at ICLR 2022
Klaus Krippendorff. Content analysis introduction its methodology, 2004.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Féi. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition. pages 248-255. IEEE, 2009.
Jacob Whitehill, Tang-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. Whose vote should count more: Optimal integration of labels from labelers unknown expertise. Advances in neural information processing systems, 22:2035 2043, 2009.
Joshua Peterson, Ruairidh Battleday. Thomas L Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617-9626, 2019.
Mitchell Gordon, Kaidyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael Bernstein. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2 CHI Conference on Human Factors in Computing Systems, pages 1-14, 2021.
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 I
RESNET- 18 VARIANTS
Our systematic variations for the ResNet-18 variants are:
Base condition: a standard ResNet-18 trained on ImageNet in PyTorch" was used as the baseline network for all comparisons; one instance trained.
Plus lep: network trained for one additional epoch compared to the base network; one instance trained.
Plus 10ep: a network was trained for ten additional epochs compared to the base network; one instance trained.
Different optimizer: a network was trained using SGD with Nesterov Coonentum(SStskeeer el 2013) instead of vanilla SGD; one instance trained.
Different batch size: for this condition we split the batch size in half (128 instead of the 256). This was done by drawing the same batches from data loader and splitting them half. then input halves sequentially into the model, effectively doubling the number of gradient updates; one instance trained.
Differrntt inttialisation: networks were varied in the initialisation of their layer weights by choosing different random seed for each network; five instances trained.
Different learning rafe: the networks were trained using initial learning rates varying from 0.148 to 0.152 instead of the default learning rate of 0.1. We narrowed range such that they still reach the same accuracy level; five instances trained.
CUDA non determmmmmiii:::: Training networks without CUDA determinism is the standard procedure. However, graphic card operations are not necessarily deterministic, e.g. functions like reduce_ _um (Riach 2019). This non determinism might not influence accuracy but may influence agreement between instances; five instances trained.
Different dataorder: networks were trained with the exact same training data, however the order of the samples was varied for cach model by choosing a different random seed before initialisation the data loader; five instances trained.
Different architectures: we trained a DenseNet- 121 a different architecture. Due to hardware constraints, we had to use a batch size of 64 for this condition; one instance trained.
Half data: the network was trained on only half of the data but compared to the base condition with all data; one instance trained.
Combined condition: for this condition. we combined multiple conditions. Here, we trained networks of the different architecture condition with training data different order, using SGD with Nesterov momentum, varying learning rates from 0.148 to 0.152, and different initialisations for each network; instances trained.
two networks were trained on the first and second half of the ImageNet training set respectively. Thus two different, disjoint training datasets For this condition we compared the networks to each other instead comparing against the base condition.
Different data:
used-of course from the distribution (ImageNet).
A.2 SOFTWARE, HARDWARE AND DATA
The networks were trained on GeForce RTX 2080 Ti GPUs with CUDA Version 11.1, CPU cores and 32 GB RAM shared between the cores. All code was written in PyTorch using Python 3 and the code to reproduce our findings is available in the supplementary material. For the RSA analysis. we used the thingsvision toolkit Muttenthaler and Hebart 2021). We used three data sels: ImageNet (Russakovsky et al. 2015). 110AR-100 Krizhevsky and Hinton 2009) and the third dataset ("Gaussian noise") was generated by ourselves investigate the effect of training on a dataset that does not contain any "natural image structure". It was generated by drawing pixel- wise uncorrelated Gaussian noise for each of the three cca-nnass.s.. The dataset consisted of 100 classes with 20000 train and 50 test images per class. The i-th class has mean of 128 and standard deviation of which is how classes can be identified by model.
A.3 CONTROL EXPERIMENTS WITH VGG AND DENSENET
To ensure that our findings generalize across different architectures and different datascts, we reran our main experiment with number of variations:
'See https: /ttpt/tttttttt/// www hoom/pytorch httaa//ppls//rreeasstrre/raaaae 256, 90 epochs, the SGD optimizer and initial learning rate of 0.1 that was divided by 10 every 30 epochs.
hatch size of
15

---

Published as a conference paper at ICLR 2022
First, tested different architectures; ImageNet with Densenet- 121 base network: Using Denenet- 121 base network with slightly altered training paradigm using only instead of epochs -to reduce the environmental impact of our study and a batch size of 64 due to GPU RAM limitations. ResNet-50 was used as comparison architectures.
Imagenet with VGG-I1 buse network: For VGG-11 as the base network, we used starting learning rate of 0.1 according the standard PyTorch implementation. Again, only trained networks in this paradigm for 30 epochs and with learning rate steps every 10 epochs. Additionally, we used AlexNet as different architecture.
Second, addition ImageNet our Gaussian dataset we used another dataset, namely CIFAR- 100: Again, we followed the standard ResNet-18 8 PyTorch implementation with the modification that we only used a total of 30 epochs to reduce the environmental impact of our study.
A.4 CONTROL EXPERIMENT REPRESENTATIONAL SIMILARLY ANALYSIS
Additionally, to check whether results reproducible outside of behavioural measure, we applied tool representational similarly analysis (RSA). RSA a method originated the brain sciences. quantifies whether the inner representation here the activation of kernels by single images similar across networks (Kriegeskorte et al. 2008 Mchrer et al. 22200). An RSA between two networks yields correlation index between 1 and 1, indicating anti- corr-corrltton, important note that the correlation values from RSA and from error consistency are comparable, although they have the same limits.
correlation (0) and perfect correlation respectively.
Representational
consistency
0.6
0.4
0.2
0.0
Base condition
Plus
lep
Pus
10ep
Different optimizer
Different
batchelze
Different init initialisation
Different
nandetermnnistic
LR
CUDA
Different dateorder
Different architecture
Half
data
Containe
Different data
Figure Correlations between the last fully connected layers the different conditions and the base network on the ImageNet validation set after 90 epochs. For conditions in which multiple models were trained, only the first model was used.
16

---

Published as a conference paper at ICLR 2022
A.5 ANALYSING SINGLE NETWORK
Figure 8 shows for our network, whether ImageNet validation images are classified correctly (white) or incorrectly (blue) across epochs. There are three take-aways from visualization. (1.), one immediately notices the influence of the standard learning rate steps after 30 and 60 epochs. However, after this step, some images (bottom) also "forgotten" (classified correctly before step but incorrectly afterwards), which contrasts with the usual expectation that model gradually improves during very first epoch and never forgotten later (top right region). while some are never learned all. We will later see that this is not an effect of label errors, see Figure 2a (3.), while accuracy usually only improves minimally from one epoch to the next (e.g. 0.04% from epoch 89 to epoch 90, or 14 additionally correctly classified images out of 50,000), epoch, corresponding 68184 images! (See Figure 17 in the Appendix for plot which shows the number of swapped labels from epoch to epoch). The key takeaways from Figure [ are already known from previous works investigating model errors over training time (Toneva el al.] [0188] Mangalam and Prabhu 2019) Kalimeris et al.)
time. (2.), some images learned immediately
average 12.37% of the models' image classification decisions swap every
-e we do not intend to claim any conceptual novelty in this regard, Figure
simply intends to visualise
these intriguing patterns clearly.
Image from validation set
100
15
20
25
30
35
40
45
60
65
70
75
80
85
90
Epoch
Figure 8: Decisions on all 50K ImageNet validation images of the single base network over the epochs. Blue indicates that the respective item was falsely classified during the specific epoch, while white indicates that was correctly classified. The items from the ImageNet validation set are ordered according to the accuracy the base network achieved them over the course of the 90 epochs. Therefore, items which were classified correctly from epoch are on top and items which were classified incorrectly from epoch are on the bottom.
17

---

Published as a conference paper at ICLR 2022
A.6 DDD IN CIFAR AND THE GAUSSIAN DATASET
this not the case on two different datasets. CIFAR-100 (Krizhevsky and Hinton 2009) and the third dataset ("Gaussian noise") was generated by ourselves to investigate the effect of training on dataset that does not contain any "natural image structure". It was generated by drawing pixel-wise uncorrelated Gaussian noise for each of the three RGBB-hannell. The dataset consisted of 100 classes with 20000 train and 50 test images per class. The which classes can be identified by mode
dichotomous difficulty (DDD) only problem for ImageNet? We here
class has mean and standard deviation
As a first indication, for both of these datasets we find similarly high error consistencies hetween different models, just like we found for ImageNet (sce Figure 21 Furthermore, training models CIFAR-100 1-00 and the Gaussian data leads
very similar result pattern for natural data sets like ImageNet and CIFAR- (shown in panel (b) and (c) of Figure (2). This is strong indication -togthher with imbalanced class accuracies in Figure 20 that highly consistent model errors are result of DDD and not an artefact of natural images.
A.7 KL DIVERGENCE
We constructed a third dataset ("Gaussian noise"). It was generated by drawing pixel-wise uncorrelated Gaussian noise for each of the three RGB-channels. The dataset consisted of 100 classes with 20000 train and 500 test images per class. The i-th class has mean 128 and standard deviation of i, which is how classes be identified by a ML model. With this procedure, the KL. AL-Diiergeece
LCCaass Class.41) log( (1+1) + 2 2.0741 2
(I)
between class and is decreasing, see Figure
KL(CCasss Class, 1
63.
Acc. Acc. (Class))
ACc.(Classi)
Class; (Std
Class; (Std
KL(Class; Class;+
J000 0000
Figure 9: Kl-Divergence vs. accuracy for the Gaussian dataset. (Left) KL-divergence between Class; and Class (Centre) Ace, Class, (Right) Scatterplot between KL-divergence and accuracy. For the last plot we skip the first 20 classes (with accuracy close to 1) for better visibility.
A.8 CLASS ACCURACIES
datasets, some classes are very easy to classify (e.g. up 100% top-1 accuracy on ImageNet), while other classes are very difficult (c.g. down to 10% top-1 accuracy on ImageNet, for a list of top- 10 casiest and hardest classes see Table []. This means that there are both casy and hard images as well easy and hard classes.
Figure 20 clearly shows
We constructed the dataset with decreasing KI. -ivveergee hetween classes. Thus some classes easier than others. In fact, show (Figure that the KI. divergence is very good predictor for class accuracies.
18

---

Published as a conference paper at ICLR 2022
Highest accuracy
Lowest accuracy
"arrtsstar'
'screen, CRT screen' 'velvet'
yelloo lady's slipper, yellow lady-slipper,
Cypripedium calceolus, Cypripedium parviflorum'
"proboscis monkey, Nasalis larvatus'
'sunglass' 'ladle'
"Leonberg'
'reiight car'
"tiger cat' 'notebook, notebook computer' hhook, claw'
"echidna, spiny anteater, anteater'
Affican hunting dog, hyena dog, Cape hunting dog, Lycaon pictus'
'limpkin, Aramus pictus'
'cleaver, meat cleaver, chopper' 'letter opener, paper knife, paperknife' 'spatula'
'hamster'
'three-toed sloth, ai, Bradypus tridactylus'
Table 1: Table displaying the ten classes, for which the base network achieved with highest and lowest accuracies respectively. Items are in a descending order, so that 'earthstar' has the highest accuracy and 'screen, CRT screen' has the lowest accuracy.
19

---

Published as a conference paper at ICLR 2022
A.9 MODELING IMAGE DIFFICULTY
We modelled the image difficulty Figure as a delta peak, all images have the same difficulty. One of our reviewers suggested model the image difficulty with an exponential decay instead-this means that we assumed that there are many simple images, then less and less more difficult images.
We therefore extend our previous approach by simulating exponential and a sigmoid function. For each of the two functions, binned the "difficulty" in bins of size 0.01 in the range of 0.00 1.00. We then calculated how many images are to expected in each the difficulty bins given underlying function. For each of difficulty bins, we then sampled from a binomial distribution. The mean difficulty of images corresponds to the observed mean difficulty (p=0.689) for ResNet-18 variants. Both functions still yield very different histograms compared to the observed histogram of image difficulties. In order to better reproduce the observed histogram, U-shaped function (bimodal distribution) required.
5000
0.7
Sigmold decay
Sigmald decay
Exponential
0.6
Exponential decay
4000
Images from validation set
Number of images
Fraction of images
0.5
0.4
0.3
2000
1000
0.1
None
None 1
3
8
10
Probability of correct classification
Number of models that classify image correctly
Figure 10: DDD is neither explained by exponentia//sigmodaa decay in image difficulty, nor by uniform example difficulty. (a) Binned functions modelling (exponentially) decaying (orange) and sigmoidal (purple) image difficulty. (b) Observed histogram (blue) with histograms obtained by sampling with a binomial observer given the exponential (orange) and sigmoid (purple) functions.
20

---

Published as a conference paper at ICLR 2022
A.10 METHODS OF THE HUMAN EXPERIMENTS
Experiment I: Can humans distinguish trivials and impossibles? order to test whether humans can infer which images are easy hard CNNs, we conducted a psy- chophysical two-alteraative forced choice experiment (Wichmann and Jäkel 2018). In the experiment, observers were instructed to indicate by button press which image of image pair they helieve to he more difficult for network classify correctly-Is the right the left image easier to classify for CNN? An example trial can found the appendix, see Figure 23 Images were chosen from ImageNet validation set such that the image pains consisted of one image which all networks with different inductive bias classified correctly and another image which all networks misclassified (see also Figure (8]. Stimuli were normalzad images of size 224 x 224 px. Observers performed 149 self-paced trials. Overall, nine observers (mean age 34.6 yrs, female, male) participated. Two observers were entirely naive CNN rescarch, a further four were naive the purpose of the experiments, but knew about CNNs. Subjects received monetary compensation of 10 É per hour. The total duration of the experiment was 30 minutes per observer.
Experiment 2: What makes trivial, inbetween and impossible images different?
Thanks to the suggestion of our reviewers, designed follow-up experiment better understand the differ- ences between "trivials", "impossibles" the correct response). 100 "impossible" (no network gives a correct response) and 100 "in-between" images. Each observer had to answer the following questions for each image:
"in betweens". We randomly chose 100 "trivial" (all networks give
Q1: How many objects helonging different categories are in the image (e.g. three dogs still category: dog. But two dogs and one cat are two categories: dog and cat)?
Q2: Is there main category in the image? (No, maybe, Yes)
Q3: the presentation of the objects unusual in any manner (e.g. orientation, location, size, view- point)? (No, slightly, very)
Stimuli noremmmmmlledd images of size 224 224 px white background with the trial number the
left, see example image Figure 24 All observer rated the same 300 images.
Overall, 8 observers (mean age 36.0 yrs, female, 7 male) participated. Two observers were entirely naïve to
CNN research, a further three were naive to the purpose of the experiments but have experience in working with CNNs and three observers were non-naïve. The total duration of the experiment was 90 minutes per observer. Subjects received compensation worth 15E.
21

---

Published as a conference paper at ICLR 2022
A.11 CONSISTENCY IN THE SECOND HUMAN PSYCHOPHYSICAL EXPERIMENT
Due a suggestion of our reviewers, checked the consistencies between our observers this follow-up experiment, For question one (number of categories), the average pairwise correlation was 0.62 [min 0.42. max = 0.76). For question number two (is there a main category) and three (presentation oddness) we calculated Krippendorff's s alpha (0 indicates no agreement between raters, implies perfect agreement which is suited for ordinal data. For question two, also removed subject from the analysis, since they reported that they did not use object categories
instead semantic categories like playing music or partying in the debriefing. Here we obtained on average of a2 = 0.50[min = 0.38, max 0.63] for question and dx 0.33 = 0.10, max 0.48] for question 3. Furthermore, we made sure that these numbers are not affected by the pre-knowledge of the observers. Thus there reasonably high agreement between the observers despite the rather open (or vague) nature of the task and instructions.
*Krippendorff offers in his book (Krippendorfi 2004 p. 241) lower bound only for very high agreement.
22

---

Published as a conference paper at ICLR 2022
A.12 SUPERCLASS ANALYSIS
We analysed the proportions of images from the three image categories ("trivials", "in-betweens" and "impossi- bles") belonging to each the ImageNet superclasses. The superclasses result from the WordNet hierarchy and are sometimes also referred to as subtrees Deng et al. (2009). We mapped the unique image identifiers from the ImageNet validation images to their respective superclasses using file from the git repository of Tiipras et al. (020).
This analysis is visualised in Figure Here we the distribution image subsets ("triviall", "in- netwbeeeeen", "impossible") within each superclass. Since we have less impossibles than trivial, they are overall more rare. The two most extremely unhalanced superelasses are "Implements, containers, misc. objects" and "Birds". For "Implements [...]", only 20% of images are in the "trivial" subset and 10% are in the "impossible" subset. In contrast, the "Birds" superclass consists of 60% "trivials" and <5% "impossibles". We therefore find that the image subsets are not equally distributed not equally distributed over the image subsets, see Figure 12 Both Figures point towards the same conclusion: there are easier and harder superclasses.
superclasses. Furthermore, we show the superclasses are
Trivials
Inbetween
0.8
Impossibles
Relative image propartion
0.6
0.5
0.3
0.2
implements, containers, misc. objects
0.0
Clothes, covering
Structures, furnishing
Devices
Reptiles, fish, amphibians
Food, plants, fungi
Dogs
Other mammals
inverterbrates
vehicles
Birds
ImageNet superclass
Figure 11: Barplot showing the proportions of items from each of the three image subsets ("trivial", "in-betwer betteeteee"" "impossible") belonging to each superclass. Here, the values are normalized so that the proportions of items each superclass sum to The superclasses are ordered according to proportion of impossibles to trivials.
23

---

Published as a conference paper at ICLR 2022
0.25
All
Trivials
Proportion of items
0.20
Inbetween
Impossibles
0.15
0.10
0.05
0.00
Clothes, covering
Birds
Devices
Implements, containers, misc. objects
Food, plants, fungi
Dogs
Other mammals
Reptiles, fish, amphibians
Structures, furnsshngg
Inverterbrates
vehicles
ImageNet superclass
Figure 12: Proportion of items from each of the three image categories ("Trivials", "Inbetweens" and "Impossibles") belonging to each superclass. The values are normalized so that the proportions of each subset sum 1.
24

---

Published as a conference paper at ICLR 2022
A.13 LABEL AMBIGUITY AS CAUSE FOR THE "IN-BETWEEN"
One of our reviewers suggested that label ambiguity might be a major cause emergence "in-between" images.
We agree that there is a possibility that disagreement between human annotators could reduce the accuracy for the "in-between" images. Label ambiguity is known to affect image datasets (Whitchill et al. [009 Peterson al. [2019] Gordon et al. 2021) although interestingly the ImageNet creators tried to mitigate this, Russakovsky et al. (2015) p.7 onward). We agree that label ambiguity can affect model accuracies. Thus
we decided to investigate this hypothesis (label ambiguity as cause for disagreement) using two different,
independent datasets to analyse label ambiguity in ImageNet.
First we revisit the dataset Northeutt et al. (2021b). The authors automatically detected label errors in the ImageNet validation and used Amazon Mechanical Turk manually check every possibly falsely labelled image with human raters. If label ambiguity do not a agree on the "in-betweens". Thus we combine combine Northcutt's 's data and with our previous analysis. Overall, for the MTTurk analysis Northcutt proposes label errors on 5440 images in the ImageNct validation From these 5440 potential label errors 2643 are in the "in-between" class. Out of the potential 2643 images, 1945 at least rater not agreement with the others. However please note that this high rate expected, since the 2643 images are those already identified as possibly having ambiguous labels by automatic approach. We have to compare this number to the total number of images in the "in-between" subset, which is 21248 images. Hence, only 9% (=1945/21444 (445) of "in- between" images suffer from label ambiguity, compared to an overall rate of 8,48(4400000000) label ambiguity for the entire dataset.
cause the "in-between" we expect that the human raters
This provides evidence that label (dis-Jagreement may be main confounder in our experiment, but of course an automated approach might miss certain images. We therefore make use another dataset: that of Geirhos ct al. (02T) with humans (which does not rely on any automated assessment).
Geirhos et al. 2021) used four observers, which performed classification task in a highly controlled psy- subset of the ImageNet validation set. In total the observers classified 607 colored images belonging the "in-hetween" subset. Here the four observer agreed on 85% (513 of 607) of the images. Thus, the disagreement between raters fairly small.
chophysical setting
Both papers used completely independent raters and different strategies (MTurk vs. highly controlled psy- chophysics). Still, analysing data from both approaches point towards the same result, providing evidence that only a minor fraction of "in-beeween"" are affected by label ambiguity.
25

---

Published as a conference paper at ICLR 2022
A.14
FIGURES
1.0
0.8
Accuracy
0.6
0.4
0.2
0.0
condition
Base
Plus lep
Pius
10ep
aptimizer
Different batchalze
Different initialisation
Different
nondeterministe mstti
Li
CUDA
Different dataorder
Differnt archifecture
Half
data
Different data
Figure 13: Accuracies of the different conditions and the base network on the ImageNet validation set after 90 epochs. The mean over all models of condition is displayed here.
1.0
Accuracy
0.6
0.2
0.0
resnet5o Swil
nrnet W44
resner152 52
wit base patch16 224
resnet50 I2 epsl
simelr resnet50r1
bagnet233
resnet5o trained on SIN
CLIP
squeezenet1 1
CORnet RT
Figure 14: Accuracies for the SOTA models the ImageNet validation set. Mean accuracy of all
models is 0.689
26

---

Published as a conference paper at ICLR 2022
1.0
0.8-
Error Consistency
0.2
0.0
Bace condition
Plus
lep
Plus
10ep
optimizer
Different
batchsize
initiallaation
Different
Different
nondetermnnssti
Li
CUDA
Differant dataorder
Differant architecture
Half
data
Combined
condition
Different
data
Figure 15: Error consistencies on the ImageNet validation set with VGG-11 the base network. All variations performed are the same as outlined in the Methods section. In this case, the different architecture is an AlexNet. The conditions validation set for ResNet-18 18 as the base network (see Figure 3}. For conditions in which multiple models were trained, the model-wise error consistencies are plotted with lower opacity compared to the mean over all models for the conditions.
ordered by the mean error consistency on the ImageNet
1.0
0.8
Error Consistency
0.2
0.0
Base condition
Plus
lep
Plus
10mp
Different
batchsize
inttialisation
Different
Different
nondetermnistic
LA
CUDA
Differant datatrder
Differant architecture
Half data
Combined
Different
data
Figure 16: Error consistencies the ImageNet validation set with DenseNet-122 121 as the base network. All variations performed are the same as outlined in the Methods section. In this case, the different architecture is ResNet-50. The conditions ImageNet validation set for ResNet-18 18 as the base network (see Figure For conditions in which multiple models were trained, the model-wise error consistencies are plotted with a lower opacity compared to the mean over all models for the conditions.
ordered by the mean error consistency
the
27

---

Published as a conference paper at ICLR 2022
Base condition
50000
Plus lep
Plus 10ep
45000
Different optimizer
Different batchsize
Number of different decisions
40000
Different initialisation
Different LR
CUDA nondeterministic
35000
Different dataorder
Different architecture
30000
Different data
Half data
25000
Combined condition
20000
15000
10000
5000
0
10
20
30
40
50
60
70
80
90
Epoch
Lineplot showing the number of decisions that change from the current to the following epoch. For epoch 0, this means that the number of decisions that different between epoch 0 and epoch 1 are shown. For conditions in which multiple model instances were trained, only the last instance is shown for the sake of simplicity.
Figure 17:
28

---

Published as a conference paper at ICLR 2022
High School Mio
XHhi COUT
UX
Figure 18: Pairs of impossible (top) and trivial images (bottom) from ImageNet.
29

---

Published as a conference paper at ICLR 2022
Figure 19: Pairs of impossible (top) and trivial images (bottom) from CIFAR-100.
1.0
0.8
Accuracy
Accuracy
Accuracy
0.6
0.4
0.2
0.2
0.0
0.0
500
1000
100
75
Class number
Class number
Class number
(a) ImageNet
(b) CIFAR-100
(c) Gaussian
Figure 20: Class-wise accuracy per dataset. Shown is the decreasing accuracy for all classes in the validation sets and for the fully trained base network.
30

---

Published as a conference paper at ICLR 2022
1.0
Error Consistency
0.4
Gaussian
0.2
CIFAR-100
0.0
Base condition
Plus
Plus
Different optimizer
Different batchsize
Different
Different
CUDA
Differen dataorder
Differnt architecture
Half date
Combined
Different
lsp
10ap
initielisation
nondeterminsste
LR
condition
data
Condition
Figure 21: Error consistencies between the different conditions and the base network for the validation sets of CIFAR-10 100 and our Gaussian dataset. The conditions are ordered by the mean error consistency on the ImageNet validation set (see Figure 33. For conditions in which multiple models were trained, the model-wing models for the conditions.
consistencies plotted with lower opacity compared to the mean over all
0.7
0.7
0.7
Images from validation set
0.5
Label emrrs
0.6
0.8
0.5
Sinomial distribution
Frocten of Images.
0.5
9.4
0.4
0.4
0.3
Irection
0.3
0.2
Number
Number modals
Number medels
(a) ImageNet
(b) CIFAR-100
(c) Gaussian
Figure 22: Histogram showing how many models correctly classify validation sets images in the last epoch. In blue, the densities of how many items were answered correctly are shown. "None" indicates that no models classified the items correctly (impossibles), while for "All" items were classified correctly by all models ("trivial images"). For the sake of simplicity, only the last model was used for conditions where multiple models were trained. In green, samples are drawn from binomial distribution with equal to the number of models and p equal to the mean accuracy over the models. Additionally for ImageNet, the distribution 5000 label errors as identified by the cleanlab package are shown in red (Northcutt et al. 202
31

---

Published as a conference paper at ICLR 2022
R
10
THE
STR
Figure 23:
Example trial from the first psychophysical experiment. Observers were asked: "Is the right or the left image easier classify for neural network?"' The number on the left indicates the trial number and the letters "R" and "L" above the images were entered into the answer sheet by the observers.
32

---

Published as a conference paper at ICLR 2022
10
Figure 24: Example trial from the second psychophysical experiment. Observers were asked: "How many objects belonging to different categories are in the image (e.g. three dogs are still one category: dog. But two dogs and one cat are two categories: dog and cat)?". "Is there a main category in the image? (No, maybe, Yes)" and "Is the presentation of the objects unusual in any manner (e.g. orientation, location, size, viewpoint)? (No, slightly, very)". The number on the left indicates the trial number.
0.8
Proportion of images
Inbetween Impossible
0.6
0.2
0.0
10
Maybe
No
Slightly
Very
Number of categorics image
there main category?
the presentation objects odd?
Figure 25: Barplot displaying the proportions of answers over all observers. We did not remove label errors for this plot. The bars normalized that the proportions of the different answers add up to 1 for each question.
33

---

Published as a conference paper at ICLR 2022
0.8
Trivial
Proportion of images
Inbetween
0.6-
Impossible
0.0
10
Maybe
Slightly
Very
Number of categories image
main category?
presentation objects odd?
Figure 26: Barplot displaying the proportions of answers over all observers. For this plot, we removed images which classes to have the same number of images. The bars are normalized so that the proportions of the different answers add up to I for each question.
found have label errors by Northcutt al. (2021b) and balanced the image
34

---

Published as a conference paper at ICLR 2022
160
Question
Question
Question
140
120
Observer 1 Counts
100
00
Inbetween Impossible
60
40
20
Maybe
No
Very
Number Categories
there cetegory?
presentation objects odd?
160
140-
120-
Observer 2
100
80
60-
40
10
Maybe
No
Number
category?
otjjects odd!
160
140-
120
3
100
80-
60
00
No
Number
catepory?
presentation odi?
160
1.10
120
100
80
60
Very
Number categories image
category?
presentation odi??
35

---

Published as a conference paper at ICLR 2022
100
Question
Question
Question)
140
120
Observer 5
100
80
60
40
10
Maybe
No presentation odd?
Very
Number categories
catepory?
160
140-
120
6
100
80
60
10
No
Number categories image
category?
presentation odd?
160
140
120
100
Obserear
80
Slightly
Number categories imagge
category?
presentation objects odd?
180
140
120
Obsanver 3
100
Maybee category?
Slightly presentation objects
categories imagee
Figure 27: Barplots displaying the proportions of answers for each individual observer. We did not remove label enrors for this plot. The bars are normalized so that the proportions of the different answers add up to for each question.
36

---

Published as a conference paper at ICLR 2022
Question
Question
Question
250-
Observer 1
150
Inbetween Impossible
50
10
Maybe
No
Very
Number Categories
there cetegory?
presentation objects odd?
300
Observer 2
150
50
10
Maybe
No
Number
category?
otjjects odd!
300
3
150
50
No
Very
Number
catepory?
presentation odi?
300
200
150
50
No
Very
Number categories image
category?
presentation odi??
37

---

Published as a conference paper at ICLR 2022
300
Question
Question
Question)
250
Observer 5
150
50
10
Maybe
No presentation odd?
Very
Number categories
catepory?
300
6
150
50
10
No
Very
Number categories image
category?
presentation odd?
300
200
Obsereer ?
150
50
Slightly
Number categories imagge
category?
presentation odd?
300
200
Obsenver 3
50
Maybee cetegory?
Slightly presentation
categories imagee
Figure 28: Barplots displaying the proportions of answers for each individual observer. removed images which were found to have label errors by Northcutt et al. Northcutt et al. (2021b). The bars are normalized so that the proportions the different answers add up to for each question.
38


Reviews:
[{"review_id": "dfyJvRTksTi", "reviewer": "Reviewer_oQTc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strengths\n- 13 variants to ResNet-18 as well as 11 SOTA models where analyzed on\n  ImageNet, CIFAR-100 and a gaussian toy dataset\n- the writing is very engaging and overall very good\n- the paper is very well structured, and offers many control experiments in the\n  supplement\n- the experiments are well documented for reproducibility\n\n### Weaknesses\n\n- minor: in A.4, the last paragraph is a copy from the first paragraph and footnote 6/7 are the same\n\nI could not find any obvious weaknesses in this paper.\nI think the paper is very well written, and the empirical experiments are\nconducted thoroughly and without any obvious flaws.\nThe insights are very significant and novel, and I feel this paper is a very important contribution.\n", "summary_of_the_paper": "The authors investigate why networks with highly different architectures and\nobjectives seem to produce similar decision boundaries.\nBy analyzing the model behavior of each sample of the validation set during\ntraining, they find that over half of the samples of the ImageNet validation\nset are either trivial or impossible for almost all analyzed models, which they\nname \"dichotomous data difficulty\". They then show that the model agreement is\nalmost only caused by these trivial and impossible samples, by measuring the\nagreement over all models with and without the trivial examples.\nFinally, they show through human trial that humans can with a relatively high\naccuracy (~81.36%) predict which samples are easier for neural networks to\npredict, showing that there is a level of difficulty to the image samples.", "summary_of_the_review": "Unless I have not missed anything very obvious, this submission seems very\npolished, significant and novel.\nTherefore, I recommend 8: accept, good paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "a7-UDVbh0_7", "reviewer": "Reviewer_3xZg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The authors tackle a timely and important problem.\n2. I found the paper to be relatively clearly written and concise.\n3. The experiments that the authors conduct are fascinating and in my view point to many potential challenges in today's evaluation of image classification models which must be further explored.\n\nWeaknesses:\nOverall, I wish the authors had focused more on what seems to me as the most supported claim the authors can make given their experiments: around half of the images in existing test sets are not providing meaningful signal about the performance of today's models, and the consequence of this might be that existing evaluation approaches are hiding or at least downplaying the differences between models. All the authors show is that within this \"difficult\" set, models make different errors. Are these errors just randomly distributed between all the images? Are they clustered around certain classes? If we just focus on this \"difficult\" set, how do standard classification metrics differ between models?\n\nTo me, the claims the authors make about \"difficulty\" are far less supported and require significant additional work. As it stands, the wording used by the authors may be potentially misleading. How can we be confident that these images are \"difficult\"? Isn't it possible that these images may simply contain multiple possible plausible labels, and annotators disagreed? Existing work has shown relatively high disagreement rates [1], [2] in image classification datasets among annotators.\n\n[1] Joshua C. Peterson*, Ruairidh M. Battleday*, Thomas L. Griffiths, & Olga Russakovsky (2019). Human uncertainty makes classification more robust. In Proceedings of the IEEE International Conference on Computer Vision. \n[2] Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S. Bernstein. 2021. The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems.\n\nThe authors also need to detail the exact procedure used in their human study. Exactly what questions were asked, what did the interface look like, how many participants were there, how were they compensated, etc. Also, why did the authors not ask annotators whether an image would fall into the 3rd \"difficult\" category?", "summary_of_the_paper": "The authors demonstrate that, irrespective of model architecture or hyperparameters, many modern image classification models are always correct on 46.0% “trivial” and 11.5% “impossible” images. The authors describe the remaining 42.5% of images as \"difficult\", and claim that by focusing on these difficult images, it is possible to see pronounced differences between models. They also find that humans can\npredict which images are “trivial” and “impossible” for CNNs at 81.4% accuracy.", "summary_of_the_review": "The authors tackle an important topic and perform fascinating experiments, but there is insufficient evidence for the claims they make and, in my view, the claims they could better make were not deeply explored.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3ZlR2FteZRB", "reviewer": "Reviewer_MrqK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- The experiments on error consistencies across (i) models trained with different hyperparameters (figure 3) and (ii) state-of-the-art models (figure 2 and figure 5) are thorough and clearly showcase the effect of dichotomous data difficulty on model predictions.\n\n- The psychophysical experiment results in section 3.4 are quite surprising, and suggest that humans and CNNs share a similar notion of image difficulty (albeit at a coarse level).\n\n- The experiment on dataset subsampling is insightful, as it suggests that that the effect of model architecture and task complexity can be amplified in the absence of DDD / dataset issues. The drop in error consistency is quite drastic, so it would be good to have the individual model accuracies reported in the paper as well.\n\n**Weaknesses**\n\n- Novelty vis-a-vis prior work.\n    - Papers on example difficulty [R1-R4] (and references therein) already show that a non-trivial fraction of data points are systematically \"easy\" and \"hard\" across different models etc. In this context, DDD is simply a different way to frame previously known results.\n\n    - The purpose of Figure 4 is unclear. The takeways reported from figure 4 are already known observations from previous works. [R5, R6] discusses learning order and [R4] discusses training time as a proxy for example difficulty.\n\n    - Hard examples can also originate from \"noisy\" datasets that contain data points with label errors (such as imagenet) .The analysis in this paper does not take into account the effect of label errors on example difficulty (the impossible set of points). Label errors can erroneously inflate the metrics used to describe DDD in the paper.\n\n    - \"...all models end up with a similar decision boundary\". This statement is incorrect. While models with similar decision boundaries will make similar errors, it is not necessary that models that make similar errors have similar decision boundaries. There is a separate line of work [R7, R8] that study feature learning and decision boundaries despite differing inductive biases of different architectures.\n\n- Scope of empirical analysis limited. While all experiments in the paper serve to identify DDD, it is not clear if this is a dataset \"issue\" per se. Is it reasonable or even possible to curate large-scale datasets that do not have DDD? Answering these questions requires additional analysis on what it means for examples to be \"trivial\" or \"impossible\". For example, are all \"trivial\" examples within a class similar in some aspect? are \"in-between\" images contributing the most to the model's performance? Experiments or preliminary analysis in this direction would have made the paper (and its conclusions) stronger.\n\n---\n\n**References**\n\n[R1] Hacohen, G., Choshen, L. and Weinshall, D., 2020, November. Let’s agree to agree: Neural networks share classification order on real datasets. In International Conference on Machine Learning (pp. 3950-3960). PMLR.\n\n[R2] Agarwal, C., D'souza, D. and Hooker, S., 2020. Estimating example difficulty using variance of gradients. arXiv preprint arXiv:2008.11600.\n\n[R3] Baldock, R.J., Maennel, H. and Neyshabur, B., 2021. Deep Learning Through the Lens of Example Difficulty. arXiv preprint arXiv:2106.09647.\n\n[R4] Mangalam, K. and Prabhu, V.U., 2019. Do deep neural networks learn shallow learnable examples first?.\n\n[R5] Toneva, M., Sordoni, A., Combes, R.T.D., Trischler, A., Bengio, Y. and Gordon, G.J., 2018. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159.\n\n[R6] Kalimeris, D., Kaplun, G., Nakkiran, P., Edelman, B., Yang, T., Barak, B. and Zhang, H., 2019. Sgd on neural networks learns functions of increasing complexity. Advances in Neural Information Processing Systems, 32, pp.3496-3506.\n\n[R7] Wang, L., Hu, L., Gu, J., Wu, Y., Hu, Z., He, K. and Hopcroft, J., 2018. Towards understanding learning representations: To what extent do different neural networks learn the same representation. arXiv preprint arXiv:1810.11750.\n\n[R8] Shah, H., Tamuly, K., Raghunathan, A., Jain, P. and Netrapalli, P., 2020. The pitfalls of simplicity bias in neural networks. arXiv preprint arXiv:2006.07710.", "summary_of_the_paper": "This paper analyzes the effect of dichotomous dataset difficulty (DDD) on model predictions; it has three key findings. First, it shows that a large fraction of imagenet images are either trivial (most models classify such images correctly) or impossible (most models classify such images correctly) for several state-of-the-art models. Second, it shows that differences between model predictions get pronounced when models are trained on \"in-between\" images only (i.e., in the absence of DDD). Third, it suggests that humans and CNNs have similar notions of image difficulty by showing that humans are highly accurate at predicting which images are difficult for CNNs", "summary_of_the_review": "Overall, I think the weaknesses of this paper outweigh its strengths. While the experiments are thorough and the human study results are quite interesting, I am primarily concerned about the novelty of the empirical findings (+ missing related work) and the limited scope/breadth of experiments (e.e. analysis of easy and hard examples). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1MwcSF6ORx", "reviewer": "Reviewer_rhYo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tThorough and solid empirical studies on the decision consistency between CNNs trained on the same dataset.\n2.\tClear analysis showing the bimodal distribution of consistency across images. One large part is almost trivial for all CNNs and the other small part is almost impossible. The middle part is what differentiates the CNNs.\n3.\tThe authors further show deeper analysis on the reason of this consistency through training models without the trivial and the impossible parts and showing that these models are much more different.\n4.\tThe paper also shows that this consistency pattern happens for different datasets, including ImageNet, Cifar, and self-constructed Gaussian vectors.\n\nWeaknesses:\n1.\tIt seems to me that a very straightforward hypothesis about these two parts would be that the trivial part is what’s very simple, either highly consistent to what’s in the training set, or the images with very typical object pose in the center of the images; and for the impossible part, it might be the images with ambiguous labels, atypical object pose or position. I think the human test results would support this hypothesis, but I wonder whether the authors could provide more evidence to either prove or disprove this hypothesis.\n2.\tThe figure 6 is very confusing to me. The caption says that the right part is original ImageNet test set, but the texts on the image actually say it’s the left part. If the texts on the image are right, then the right panel is the consistency on the validation images between the two parts. If I understand the experiments correctly, these results are for models trained on ImageNet training set without the trivial or the impossible part and then tested on ImageNet validation set without the two parts. Although it’s good to see the lower consistency, it should be compared to the consistency between models trained on the whole ImageNet training set and tested on ImageNet validation set without the two parts, which I cannot find. Is the consistency lower because of the changed training process or the changed validation set?\n3.\tIt is also unclear how surprising we should be towards the consistency distribution, is this a result of an exponential distribution of the general “identification” difficulty (most images are simple, then less and less are more difficult)?\n", "summary_of_the_paper": "This paper shows the high consistency between the decisions of CNNs trained on the same dataset, regardless of the algorithms, architectures, hyperparameters in training, and optimizers. The authors further show that the validation set of ImageNet contains a large part that’s “trivial” for all CNNs and a small part that’s almost “impossible”. Removing these parts in the training set indeed makes models more different. Finally, the authors state that humans can easily tell what images are “trivial” and what images are “impossible”.", "summary_of_the_review": "The authors show the high consistency between the decisions of CNNs trained on the same dataset. It is also shown that the validation images can be split into three parts: the trivial part, the impossible part, and the part that’s between them. However, the current work lacks the test of a naïve hypothesis about these two parts: the trivial part is the easy images, and the impossible part is the difficult images. Some results are also not described clearly. I can only recommend for acceptance with confidence marginally above the threshold, but fixing these points would increase my confidence.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The key contribution of the paper is identifying that datasets have the so-called DDD property. In short, datasets are predominantly composed of examples that are either consistently trivial or challenging (often misclassified) for neural networks.

Reviewer MrqK pointed out that it is well known (and provided four references) that many examples are consistently very hard or very easy for neural networks. This is true. It is somewhat novel how the authors attribute this phenomenon to datasets. Here, I would like to note that I slightly disagree with the attribution of the phenomenon to dataset alone. While it is a property of datasets, it is not self-evident that deep nets trained with SGD have to learn these trivial examples. Attributing it either only to dataset or to model/optimization seems to be oversimplistic.

The second key issue of the paper is that it is somewhat inconclusive. Many datasets have the DDD property, but the Authors provide a somewhat unclear motivation for why it matters. In particular, the fact the two models make correlated errors on a dataset does not mean we cannot distinguish them. In fact, we have been able to distinguish models using IID and OOD datasets. They make correlated errors, but one makes, with a significant margin, less errors than the other. Having said that, I agree that we without the DDD property we would be able to more easily distinguish models. This is a useful perspective.

Reviewers appreciated added experiments that help better characterize what are these trivial and impossible examples. 

Despite the issues with novelty and framing, I think it is a useful perspective and hopefully will encourage more research into understanding the interaction between data and training. It is my pleasure to recommend acceptance and thank you for submitting the paper.

In the camera ready, please: (a) describe much more clearly and openly relation to prior work; (b) bring to the main text more data from the psychophysical experiments; and (c) address any other remarks made by reviewers.
Decision: Accept (Poster)