Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes the effect of dichotomous dataset difficulty (DDD) on model predictions; it has three key findings. First, it shows that a large fraction of imagenet images are either trivial (most models classify such images correctly) or impossible (most models classify such images correctly) for several state-of-the-art models. Second, it shows that differences between model predictions get pronounced when models are trained on "in-between" images only (i.e., in the absence of DDD). Third, it suggests that humans and CNNs have similar notions of image difficulty by showing that humans are highly accurate at predicting which images are difficult for CNNs
main_review: **Strengths**

- The experiments on error consistencies across (i) models trained with different hyperparameters (figure 3) and (ii) state-of-the-art models (figure 2 and figure 5) are thorough and clearly showcase the effect of dichotomous data difficulty on model predictions.

- The psychophysical experiment results in section 3.4 are quite surprising, and suggest that humans and CNNs share a similar notion of image difficulty (albeit at a coarse level).

- The experiment on dataset subsampling is insightful, as it suggests that that the effect of model architecture and task complexity can be amplified in the absence of DDD / dataset issues. The drop in error consistency is quite drastic, so it would be good to have the individual model accuracies reported in the paper as well.

**Weaknesses**

- Novelty vis-a-vis prior work.
    - Papers on example difficulty [R1-R4] (and references therein) already show that a non-trivial fraction of data points are systematically "easy" and "hard" across different models etc. In this context, DDD is simply a different way to frame previously known results.

    - The purpose of Figure 4 is unclear. The takeways reported from figure 4 are already known observations from previous works. [R5, R6] discusses learning order and [R4] discusses training time as a proxy for example difficulty.

    - Hard examples can also originate from "noisy" datasets that contain data points with label errors (such as imagenet) .The analysis in this paper does not take into account the effect of label errors on example difficulty (the impossible set of points). Label errors can erroneously inflate the metrics used to describe DDD in the paper.

    - "...all models end up with a similar decision boundary". This statement is incorrect. While models with similar decision boundaries will make similar errors, it is not necessary that models that make similar errors have similar decision boundaries. There is a separate line of work [R7, R8] that study feature learning and decision boundaries despite differing inductive biases of different architectures.

- Scope of empirical analysis limited. While all experiments in the paper serve to identify DDD, it is not clear if this is a dataset "issue" per se. Is it reasonable or even possible to curate large-scale datasets that do not have DDD? Answering these questions requires additional analysis on what it means for examples to be "trivial" or "impossible". For example, are all "trivial" examples within a class similar in some aspect? are "in-between" images contributing the most to the model's performance? Experiments or preliminary analysis in this direction would have made the paper (and its conclusions) stronger.

---

**References**

[R1] Hacohen, G., Choshen, L. and Weinshall, D., 2020, November. Let’s agree to agree: Neural networks share classification order on real datasets. In International Conference on Machine Learning (pp. 3950-3960). PMLR.

[R2] Agarwal, C., D'souza, D. and Hooker, S., 2020. Estimating example difficulty using variance of gradients. arXiv preprint arXiv:2008.11600.

[R3] Baldock, R.J., Maennel, H. and Neyshabur, B., 2021. Deep Learning Through the Lens of Example Difficulty. arXiv preprint arXiv:2106.09647.

[R4] Mangalam, K. and Prabhu, V.U., 2019. Do deep neural networks learn shallow learnable examples first?.

[R5] Toneva, M., Sordoni, A., Combes, R.T.D., Trischler, A., Bengio, Y. and Gordon, G.J., 2018. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159.

[R6] Kalimeris, D., Kaplun, G., Nakkiran, P., Edelman, B., Yang, T., Barak, B. and Zhang, H., 2019. Sgd on neural networks learns functions of increasing complexity. Advances in Neural Information Processing Systems, 32, pp.3496-3506.

[R7] Wang, L., Hu, L., Gu, J., Wu, Y., Hu, Z., He, K. and Hopcroft, J., 2018. Towards understanding learning representations: To what extent do different neural networks learn the same representation. arXiv preprint arXiv:1810.11750.

[R8] Shah, H., Tamuly, K., Raghunathan, A., Jain, P. and Netrapalli, P., 2020. The pitfalls of simplicity bias in neural networks. arXiv preprint arXiv:2006.07710.
summary_of_the_review: Overall, I think the weaknesses of this paper outweigh its strengths. While the experiments are thorough and the human study results are quite interesting, I am primarily concerned about the novelty of the empirical findings (+ missing related work) and the limited scope/breadth of experiments (e.e. analysis of easy and hard examples).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors investigate why networks with highly different architectures and
objectives seem to produce similar decision boundaries.
By analyzing the model behavior of each sample of the validation set during
training, they find that over half of the samples of the ImageNet validation
set are either trivial or impossible for almost all analyzed models, which they
name "dichotomous data difficulty". They then show that the model agreement is
almost only caused by these trivial and impossible samples, by measuring the
agreement over all models with and without the trivial examples.
Finally, they show through human trial that humans can with a relatively high
accuracy (~81.36%) predict which samples are easier for neural networks to
predict, showing that there is a level of difficulty to the image samples.
main_review: 
### Strengths
- 13 variants to ResNet-18 as well as 11 SOTA models where analyzed on
  ImageNet, CIFAR-100 and a gaussian toy dataset
- the writing is very engaging and overall very good
- the paper is very well structured, and offers many control experiments in the
  supplement
- the experiments are well documented for reproducibility

### Weaknesses

- minor: in A.4, the last paragraph is a copy from the first paragraph and footnote 6/7 are the same

I could not find any obvious weaknesses in this paper.
I think the paper is very well written, and the empirical experiments are
conducted thoroughly and without any obvious flaws.
The insights are very significant and novel, and I feel this paper is a very important contribution.

summary_of_the_review: Unless I have not missed anything very obvious, this submission seems very
polished, significant and novel.
Therefore, I recommend 8: accept, good paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper shows the high consistency between the decisions of CNNs trained on the same dataset, regardless of the algorithms, architectures, hyperparameters in training, and optimizers. The authors further show that the validation set of ImageNet contains a large part that’s “trivial” for all CNNs and a small part that’s almost “impossible”. Removing these parts in the training set indeed makes models more different. Finally, the authors state that humans can easily tell what images are “trivial” and what images are “impossible”.
main_review: Strengths:
1.	Thorough and solid empirical studies on the decision consistency between CNNs trained on the same dataset.
2.	Clear analysis showing the bimodal distribution of consistency across images. One large part is almost trivial for all CNNs and the other small part is almost impossible. The middle part is what differentiates the CNNs.
3.	The authors further show deeper analysis on the reason of this consistency through training models without the trivial and the impossible parts and showing that these models are much more different.
4.	The paper also shows that this consistency pattern happens for different datasets, including ImageNet, Cifar, and self-constructed Gaussian vectors.

Weaknesses:
1.	It seems to me that a very straightforward hypothesis about these two parts would be that the trivial part is what’s very simple, either highly consistent to what’s in the training set, or the images with very typical object pose in the center of the images; and for the impossible part, it might be the images with ambiguous labels, atypical object pose or position. I think the human test results would support this hypothesis, but I wonder whether the authors could provide more evidence to either prove or disprove this hypothesis.
2.	The figure 6 is very confusing to me. The caption says that the right part is original ImageNet test set, but the texts on the image actually say it’s the left part. If the texts on the image are right, then the right panel is the consistency on the validation images between the two parts. If I understand the experiments correctly, these results are for models trained on ImageNet training set without the trivial or the impossible part and then tested on ImageNet validation set without the two parts. Although it’s good to see the lower consistency, it should be compared to the consistency between models trained on the whole ImageNet training set and tested on ImageNet validation set without the two parts, which I cannot find. Is the consistency lower because of the changed training process or the changed validation set?
3.	It is also unclear how surprising we should be towards the consistency distribution, is this a result of an exponential distribution of the general “identification” difficulty (most images are simple, then less and less are more difficult)?

summary_of_the_review: The authors show the high consistency between the decisions of CNNs trained on the same dataset. It is also shown that the validation images can be split into three parts: the trivial part, the impossible part, and the part that’s between them. However, the current work lacks the test of a naïve hypothesis about these two parts: the trivial part is the easy images, and the impossible part is the difficult images. Some results are also not described clearly. I can only recommend for acceptance with confidence marginally above the threshold, but fixing these points would increase my confidence.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors demonstrate that, irrespective of model architecture or hyperparameters, many modern image classification models are always correct on 46.0% “trivial” and 11.5% “impossible” images. The authors describe the remaining 42.5% of images as "difficult", and claim that by focusing on these difficult images, it is possible to see pronounced differences between models. They also find that humans can
predict which images are “trivial” and “impossible” for CNNs at 81.4% accuracy.
main_review: Strengths:
1. The authors tackle a timely and important problem.
2. I found the paper to be relatively clearly written and concise.
3. The experiments that the authors conduct are fascinating and in my view point to many potential challenges in today's evaluation of image classification models which must be further explored.

Weaknesses:
Overall, I wish the authors had focused more on what seems to me as the most supported claim the authors can make given their experiments: around half of the images in existing test sets are not providing meaningful signal about the performance of today's models, and the consequence of this might be that existing evaluation approaches are hiding or at least downplaying the differences between models. All the authors show is that within this "difficult" set, models make different errors. Are these errors just randomly distributed between all the images? Are they clustered around certain classes? If we just focus on this "difficult" set, how do standard classification metrics differ between models?

To me, the claims the authors make about "difficulty" are far less supported and require significant additional work. As it stands, the wording used by the authors may be potentially misleading. How can we be confident that these images are "difficult"? Isn't it possible that these images may simply contain multiple possible plausible labels, and annotators disagreed? Existing work has shown relatively high disagreement rates [1], [2] in image classification datasets among annotators.

[1] Joshua C. Peterson*, Ruairidh M. Battleday*, Thomas L. Griffiths, & Olga Russakovsky (2019). Human uncertainty makes classification more robust. In Proceedings of the IEEE International Conference on Computer Vision. 
[2] Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S. Bernstein. 2021. The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems.

The authors also need to detail the exact procedure used in their human study. Exactly what questions were asked, what did the interface look like, how many participants were there, how were they compensated, etc. Also, why did the authors not ask annotators whether an image would fall into the 3rd "difficult" category?
summary_of_the_review: The authors tackle an important topic and perform fascinating experiments, but there is insufficient evidence for the claims they make and, in my view, the claims they could better make were not deeply explored.
review_title: null

Meta-review: The key contribution of the paper is identifying that datasets have the so-called DDD property. In short, datasets are predominantly composed of examples that are either consistently trivial or challenging (often misclassified) for neural networks.

Reviewer MrqK pointed out that it is well known (and provided four references) that many examples are consistently very hard or very easy for neural networks. This is true. It is somewhat novel how the authors attribute this phenomenon to datasets. Here, I would like to note that I slightly disagree with the attribution of the phenomenon to dataset alone. While it is a property of datasets, it is not self-evident that deep nets trained with SGD have to learn these trivial examples. Attributing it either only to dataset or to model/optimization seems to be oversimplistic.

The second key issue of the paper is that it is somewhat inconclusive. Many datasets have the DDD property, but the Authors provide a somewhat unclear motivation for why it matters. In particular, the fact the two models make correlated errors on a dataset does not mean we cannot distinguish them. In fact, we have been able to distinguish models using IID and OOD datasets. They make correlated errors, but one makes, with a significant margin, less errors than the other. Having said that, I agree that we without the DDD property we would be able to more easily distinguish models. This is a useful perspective.

Reviewers appreciated added experiments that help better characterize what are these trivial and impossible examples. 

Despite the issues with novelty and framing, I think it is a useful perspective and hopefully will encourage more research into understanding the interaction between data and training. It is my pleasure to recommend acceptance and thank you for submitting the paper.

In the camera ready, please: (a) describe much more clearly and openly relation to prior work; (b) bring to the main text more data from the psychophysical experiments; and (c) address any other remarks made by reviewers.
Decision: Accept (Poster)