Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper addresses a very interesting and important question: to what extent the recent improvements in object recognition models make them closer to human perception. The authors do this by collecting human data on a wide range of ood benchmarks modeled after ImageNet, evaluating the behavior of a large number of recently developed ImageNet models on these benchmarks, and comparing them with the human responses. The authors also open-source all their data and analysis tools so people can evaluate their own models on the same benchmarks and compare them with human behavior in a fine-grained manner (on an image-by-image basis).
main_review: I think this is overall a solid paper packed with important and, to my knowledge, novel contributions and insights:

1) The human data itself and the open-source tools made available for evaluating and analyzing new models are very useful contributions.

2) The authors show that several recent models make significant progress in closing the gap with human behavior. This seems to be true even with respect to the most stringent comparison metric: i.e. image-by-image error consistency with humans. The most important contributor to this progress seems to be pretraining on large datasets (e.g. CLIP, SWSL, BiT).

3) Self-supervised models do not perform any differently from their supervised counterparts when differences in data augmentation pipelines are accounted for.

4) Transformer models in general appear to be better than comparable convnets both in terms of ood performance and in terms of consistency with human observers.

Perhaps, one small suggestion for improvement I can make is some kind of factorial quantitative analysis of the effects of different factors considered here both on ood performance and on consistency with human observers. Currently the authors are making these judgements qualitatively by looking at some plots and concluding, for example, “models pre-trained on large datasets perform better on ood benchmarks and are more consistent with human observers” or “transformers perform better than convnets” etc., which is fine, but I think a more quantitative, factorial analysis would be even better, because, among other things, it would, for example, also potentially give us more information about the interactions and comparative effect sizes.

limitations_and_societal_impact: I think the authors do discuss the limitations of this work adequately in their discussion section. For me, the most important limitation is that static object recognition is an important, but limited facet of human perception. Extending this kind of detailed analysis to a broad range of other human perceptual skills in the future would be important.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors performed a large set of behavioral experiments to measure human psychophysical performance on an image classification task using a broad range of out-of-distribution datasets. They then evaluated a pool of convolutional neural networks (CNNs) on how well they matched human judgements and observed that while there is still a big gap between human and machine vision, models trained on very large datasets are starting to close that gap. The main contribution of the study is the release of the behavioral data for benchmarking models of computer vision. EDIT: Score updated 5 -> 6

main_review: The paper is very well written, clear, and addresses one of the most important problems in computer vision: the difference in robustness between human and machine vision. It does so by directly comparing the performance of models in several out-of-distribution datasets with human psychophysics. Despite this, I have some concerns

This paper follows a trend of studies comparing human behavioral performance on an object recognition task with machine vision models. Studies such as  Rajalingham et al 2018 The Journal of Neurosciences, Geirhos et al 2018 NeurIPS, Geirhos et al. 2019 ICLR and Geirhos et al 2020 NeurIPS have all compared CNN performance with humans in different datasets. In that sense this study can be seen not entirely original but more as an extension of this line of research. It is difficult to see what is exactly original in this study since most of the comparisons with human behavior described here had already been done elsewhere: there is a large overlap with datasets from Geirhos et al 2018 and Geirhos et al 2020.

Furthermore, the methodology that the authors have used is also not without issues. To compare CNN models with 1,000 classes outputs with human behavioral performance in a task with 16 object categories, the authors have used a simple fixed behavioral decoder to map these different output spaces using the WordNet hierarchy. Other researchers (e.g. Rajalingham et al 2018) have used different strategies to decode behavioral choices from the model features and it is not clear which option is better. Even if the authors do not implement other strategies for decoding behavioral outputs from the models, they should at least discuss this in the paper and justify their choice. 

Finally, it is not clear in which format the datasets will be made available. While I salute the authors for providing an evaluation platform to facilitate access to the benchmarks described here, it would also be important that the authors provide the raw behavioral data as well as the stimuli used so that other researchers can implement alternative metrics for comparing models to humans. 

Minor issues:
I was surprised to see the omission of a citation to Rajalingham et al 2018, as it was one of the first studies to perform a detailed comparison between models to human behavioral performance in an object recognition task.

EDIT: Score updated 5 -> 6

limitations_and_societal_impact: The authors properly addressed some of the study's limitations. The authors do not address potential negative societal impact since these are non-existent.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper compares human and model performance on a variety of OOD benchmarks (e.g., texture-swapped images, blurred images, silhouettes, sketches) and for a variety of the most recent models (eg., CLIP, ViT, BiT, SimCLR) and shows that the best models are now on par with humans on these tasks. The paper also shows interesting remaining differences between humans and models in terms of the errors made (humans make different errors than models) and the texture bias (much stronger in models than humans).
main_review: *Summary of the review:*
I recommend acceptance of this paper for its important and non-trivial conclusions, and for its impressive thoroughness (52 recent models tested on 17 OOD datasets, 90 human observers, 85,120 psychophysical trials). However, I believe that the conclusion that 'the longstanding robustness gap between humans and CNNs is closing, with the best models now matching or exceeding human performance on most OOD datasets' is too broad in scope and not currently supported by the results. I would feel comfortable increasing my rating if this claim was re-worded to more adequately reflect the scope of the results (see Main Concern).

*Strengths*
- The study is very thorough (52 recent models, 17 OOD datasets, 90 human observers, 85,120 psychophysical trials) and its main conclusions are non-trivial and have far-reaching implications for the field of computer vision.
- I found the article very well and clearly written, and pleasant to read.

*Main Concern*

I believe that the main conclusion of the study, as currently stated in the abstract  ('The longstanding robustness gap between humans and CNNs is closing, with the best models now matching or exceeding human performance on most OOD datasets'), lacks support for two reasons:
- There isn't a clear consensus on what constitutes relevant OOD benchmarks for images. The OOD tasks chosen in this paper follow a somewhat narrow definition of OOD, with the distortions only affecting pixel-level details of the images and object textures, as opposed to the overall structure of the image. For example, it would be important to see if models are now robust to unusual poses of objects in images (Alcorn et al.). See references below for more OOD datasets which affect the global structure of images and other properties.
- Humans were presented images for a short 200ms followed by a backward mask and in a 3x3 visual angle frame. I understand the motivation to remove recurrent processing, but given this potentially strong impairment to human vision, I believe that the claim that 'best models now matching or exceeding human performance' currently lacks support.

*Suggestions to improve clarity*
- Is OOD a well-defined notion for networks trained on proprietary datasets such as CLIP? For example, are we sure that the distortions presented in the OOD datasets were not present in the training set? What is the exact definition of OOD used here?
- How to reconcile the strong difference in shape/texture bias between human and models in fig3, with the overall performance match in fig1? Is the stylized dataset a particularly difficult dataset for the models tested?
- 'Observed consistency' seems to be a less informative quantity than 'Error consistency' because the overall accuracy factors in 'Observed consistency'. Why not just report 'Error consistency'?

*Other suggestions*
- Error consistency: it would be useful to have a qualitative interpretation and illustrations of the images that confuse networks and not humans and vice-versa.
- Figure 3 apparently shows a drastic failure of models to capture shape information, but would a simple linear readout trained on top of the representations of the last layer change these results? Maybe the shape information is readily available at the level of these representations but the network chose to ignore it to focus on texture. Conversely, humans naturally pay attention to shape but could probably switch to a texture-bias if asked to focus on texture.

*References* 

Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects
Michael A. Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, Anh Nguyen
https://arxiv.org/abs/1811.11553

Exposing Previously Undetectable Faults in Deep Neural Networks
Isaac Dunn, Hadrien Pouget, Daniel Kroening, Tom Melham
https://arxiv.org/abs/2106.00576

Small in-distribution changes in 3D perspective and lighting fool both CNNs and Transformers
Spandan Madan, Tomotake Sasaki, Tzu-Mao Li, Xavier Boix, Hanspeter Pfister
https://arxiv.org/abs/2106.16198

Natural Adversarial Examples
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, Dawn Song
https://arxiv.org/abs/1907.07174

Towards Non-I.I.D. Image Classification: A Dataset and Baselines
Yue He, Zheyan Shen, Peng Cui
https://arxiv.org/abs/1906.02899
http://nico.thumedialab.com/


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors employ a large-scale comparison between human classification behavior and a diverse range of machine vision models on 17 out-of-distribution (OOD) datasets as a way of attempting to comprehensively measure whether the field is making progress in closing the human-machine gap in this domain. They find that particular architectures and training objectives, as well as larger training datasets appear to be increasingly closing the gap when measuring OOD accuracy overall, but less so at the level of error patterns for individual images. They further identify subsets of their OOD task set where the human-machine gap is narrowing versus those where it is not.
main_review: While many smaller experiments have previously compared humans to CNNs, the current work is one of the most ambitious and comprehensive comparisons to date, including many architectures that have received far less study in the cognitive sciences. The conclusions are interesting, and the task suite is said to be planned for public release, which would be a very useful tool for the community.

Given the breadth of the work, it's hard to complain about many of the details. My only real concern was that the conclusions may be severely limited by the behavioral dataset used. That is, the authors used a single experimental paradigm wherein a very small set of laboratory subjects provide a larger number of judgements for a much smaller range of classes than the models which are points of comparison. There are existing behavioral datasets that are larger and pool information across many more annotators to assess image-level error patterns without having to alter the categorization task (e.g., CIFAR-10H). Other datasets focus on much richer OOD problems (ObjectNet, Omniglot) that may elicit very different human-machine deviations. For these reasons, I would have gladly traded some of the depth in model exploration in the paper for additional depth in behavioral comparison.

In the end, I have little doubt that the work represents an important analysis of the field, but I don't agree that it provides the so-called "missing human baseline", nor that datasets employed allow one to draw conclusions about the current human-machine vision gap beyond the sort of image augmentations explored by the authors.
limitations_and_societal_impact: The authors acknowledge many limitations and there doesn't appear to be much potential for negative social impact.
review_title: null

Meta-review: The reviewers were enthusiastic about the breadth and quality of this work. I agree; this is an exceptional paper that will be an important resource for the community in comparing human and machine vision, especially with the open-source evaluation software.

The reviewers offered important suggestions about the wording of some of the conclusions, which the authors were receptive to in the rebuttal. The regression analysis suggested by R-bjmw is also very useful. These changes will further improve the final manuscript.
Decision: Accept (Oral)