Partial success in closing the gap between human and machine vision

1 Introduction . Looking back at the last decade , deep learning has made tremendous leaps of progress by any standard . What started in 2012 with AlexNet [ 1 ] as the surprise winner of the ImageNet Large-Scale Visual Recognition Challenge quickly became the birth of a new AI “ summer ” , a summer lasting much longer than just a season . With it , just like with any summer , came great expectations : the hope that the deep learning revolution will see widespread applications in industry , that it will propel breakthroughs in the sciences , and that it will ultimately close the gap between human and machine 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . perception . We have now reached the point where deep learning has indeed become a significant driver of progress in industry [ e.g . 2 , 3 ] , and where many disciplines are employing deep learning for scientific discoveries [ 4–9 ] —but are we making progress in closing the gap between human and machine vision ? IID vs. OOD benchmarking . For a long time , the gap between human and machine vision was mainly approximated by comparing benchmark accuracies on IID ( independent and identically distributed ) test data : as long as models are far from reaching human-level performance on challenging datasets like ImageNet , this approach is adequate [ 10 ] . Currently , models are routinely matching and in many cases even outperforming humans on IID data . At the same time , it is becoming increasingly clear that models systematically exploit shortcuts shared between training and test data [ 11–14 ] . Therefore we are witnessing a major shift towards measuring model performance on out-of-distribution ( OOD ) data rather than IID data alone , which aims at testing models on more challenging test cases where there is still a ground truth category , but certain image statistics differ from the training distribution . Many OOD generalisation tests have been proposed : ImageNet-C [ 15 ] for corrupted imaes , ImageNet-Sketch [ 16 ] for sketches , Stylized-ImageNet [ 17 ] for image style changes , [ 18 ] for unfamiliar object poses , and many more [ 19–29 ] . While it is great to have many viable and valuable options to measure generalisation , most of these datasets unfortunately lack human comparison data . This is less than ideal , since we can no longer assume that humans reach near-ceiling accuracies on these challenging test cases as they do on standard noise-free IID object recognition datasets . In order to address this issue , we carefully tested human observers in the Wichmannlab ’ s vision laboratory on a broad range of OOD datasets , providing some 85K psychophysical trials across 90 participants . Crucially , we showed exactly the same images to multiple observers , which means that we are able to compare human and machine vision on the fine-grained level of individual images [ 30–32 ] ) . The focus of our datasets is measuring distortion robustness : we tested 17 variations that include changes to image style , texture , and various forms of synthetic additive noise . Contributions & outlook . The resulting 17 OOD datasets with large-scale human comparion data enable us to investigate recent exciting machine learning developments that crucially deviate from “ vanilla ” CNNs along three axes : objective function ( supervised vs. self-supervised , adversarially trained , and CLIP ’ s joint language-image training ) , architecture ( convolutional vs. vision transformer ) and training dataset size ( ranging from 1M to 1B images ) . Taken together , these are some of the most promising directions our field has developed to date—but this field would not be machine learning if new breakthroughs weren ’ t within reach in the next few weeks , months and years . Therefore , we open-sourced modelvshuman , a Python toolbox that enables testing both PyTorch and TensorFlow models on our comprehensive benchmark suite of OOD generalisation data in order to measure future progress . Even today , our results give cause for ( cautious ) optimism . After a method overview ( Section 2 ) , we are able to report that the human-machine distortion robustness gap is closing : the best models now match or in many cases even exceed human feedforward performance on most of the investigated OOD datasets ( Section 3 ) . While there is still a substantial image-level consistency gap between humans and machines , this gap is narrowing on some—but not all—datasets when the size of the training dataset is increased ( Section 4 ) . 2 Methods : datasets , psychophysical experiments , models , metrics , toolbox . OOD datasets with consistency-grade human data . We collected human data for 17 generalisation datasets ( visualized in Figures 7 and 8 in the Appendix , which also state the number of subjects and trials per experiment ) on a carefully calibrated screen in a dedicated psychophysical laboratory ( a total of 85,120 trials across 90 observers ) . Five datasets each correspond to a single manipulation ( sketches , edge-filtered images , silhouettes , images with a texture-shape cue conflict , and stylized images where the original image texture is replaced by the style of a painting ) ; the remaining twelve datasets correspond to parametric image degradations ( e.g . different levels of noise or blur ) . Those OOD datasets have in common that they are designed to test ImageNet-trained models . OOD images were obtained from different sources : sketches from ImageNet-Sketch [ 16 ] , stylized images from Stylized-ImageNet [ 17 ] , edge-filtered images , silhouettes and cue conflict images from [ 17 ] 1 , and the remaining twelve parametric datasets were adapted from [ 33 ] . For these parametric datasets , [ 33 ] collected human accuracies but unfortunately , they showed different images to different observers implying that we can not use their human data to assess image-level consistency between humans and machines . Thus we collected psychophysical data for those images ourselves by showing exactly the same images to multiple observers for each of those twelve datasets . Additionally , we cropped the images from [ 33 ] to 224 × 224 pixels to allow for a fair comparison to ImageNet models ( all models included in our comparison receive 224× 224 input images ; [ 33 ] showed 256× 256 images to human observers in many cases ) . Psychophysical experiments . 90 observers were tested in a darkened chamber . Stimuli were presented at the center of a 22 ” monitor with 1920× 1200 pixels resolution ( refresh rate : 120 Hz ) . Viewing distance was 107 cm and target images subtended 3× 3 degrees of visual angle . Human observers were presented with an image and asked to select the correct category out of 16 basic categories ( such as chair , dog , airplane , etc. ) . Stimuli were balanced w.r.t . classes and presented in random order . For ImageNet-trained models , in order to obtain a choice from the same 16 categories , the 1,000 class decision vector was mapped to those 16 classes using the WordNet hierarchy [ 34 ] . In Appendix I , we explain why this mapping is optimal . We closely followed the experimental protocol defined by [ 33 ] , who presented images for 200 ms followed by a 1/f backward mask to limit the influence of recurrent processing ( otherwise comparing to feedforward models would be difficult ) . Further experimental details are provided in Appendix C. Why not use crowdsourcing instead ? Our approach of investigating few observers in a highquality laboratory setting performing many trials is known as the so-called “ small-N design ” , the bread-and-butter approach in high-quality psychophysics—see , e.g. , the review “ Small is beautiful : In defense of the small-N design ” [ 35 ] . This is in contrast to the “ crowdsourcing approach ” ( many observers in a noisy setting performing fewer trials each ) . The highly controlled conditions of the Wichmannlab ’ s psychophysical laboratory come with many advantages over crowdsourced data collection : precise timing control ( down to the millisecond ) , carefully calibrated monitors ( especially important for e.g . low-contrast stimuli ) , controlled viewing distance ( important for foveal presentation ) , full visual acuity ( we performed an acuity test with every observer prior to the experiment ) , observer attention ( e.g . no multitasking or children running around during an experiment , which may happen in a crowdsourcing study ) , just to name a few [ 36 ] . Jointly , these factors contribute to high data quality . Models . In order to disentangle the influence of objective function , architecture and training dataset size , we tested a total of 52 models : 24 standard ImageNet-trained CNNs [ 37 ] , 8 selfsupervised models [ 38–43 ] ,2 6 Big Transfer models [ 45 ] , 5 adversarially trained models [ 46 ] , 5 vision transformers [ 47 , 48 ] , two semi-weakly supervised models [ 49 ] as well as Noisy Student [ 50 ] and CLIP [ 51 ] . Technical details for all models are provided in the Appendix . Metrics . In addition to OOD accuracy ( averaged across conditions and datasets ) , the following three metrics quantify how closely machines are aligned with the decision behaviour of humans . Accuracy difference A ( m ) is a simple aggregate measure that compares the accuracy of a machine m to the accuracy of human observers in different out-of-distribution tests , A ( m ) : R→ [ 0 , 1 ] , m 7→ 1 |D| ∑ d∈D 1 |Hd| ∑ h∈Hd 1 |Cd| ∑ c∈Cd ( accd , c ( h ) − accd , c ( m ) ) 2 ( 1 ) where accd , c ( · ) is the accuracy of the model or the human on dataset d ∈ D and condition c ∈ Cd ( e.g . a particular noise level ) , and h ∈ HD denotes a human observer tested on dataset d. Analogously , one can compute the average accuracy difference between a human observer h1 and all other human observers by substituting h1 for m and h ∈ HD \ { h1 } for h ∈ HD ( which can also be applied for the two metrics defined below ) . 1For those three datasets consisting of 160 , 160 and 1280 images respectively , consistency-grade psychophysical data was already collected by the authors and included in our benchmark with permission from the authors . 2We presented a preliminary and much less comprehensive version of this work at the NeurIPS 2020 workshop SVRHM [ 44 ] . Aggregated metrics like A ( m ) ignore individual image-level decisions . Two models with vastly different image-level decision behaviour might still end up with the same accuracies on each dataset and condition . Hence , we include two additional metrics in our benchmark that are sensitive to decisions on individual images . Observed consistency O ( m ) [ 32 ] measures the fraction of samples for which humans and a model m get the same sample either both right or both wrong . More precisely , let bh , m ( s ) be one if both a human observer h and m decide either correctly or incorrectly on a given sample s , and zero otherwise . We calculate the average observed consistency as O ( m ) : R→ [ 0 , 1 ] , m 7→ 1 |D| ∑ d∈D 1 |Hd| ∑ h∈Hd 1 |Cd| ∑ c∈Cd 1 |Sd , c| ∑ s∈Sd , c bh , m ( s ) ( 2 ) where s ∈ Sd , c denotes a sample s ( in our case , an image ) of condition c from dataset d. Note that this measure can only be zero if the accuracy of h and m are exactly the same in each dataset and condition . Error consistency E ( m ) [ 32 ] tracks whether there is above-chance consistency . This is an important distinction , since e.g . two decision makers with 95 % accuracy each will have at least 90 % observed consistency , even if their 5 % errors occur on non-overlapping subsets of the test data ( intuitively , they both get most images correct and thus observed overlap is high ) . To this end , error consistency ( a.k.a . Cohen ’ s kappa , cf . [ 52 ] ) indicates whether the observed consistency is larger than what could have been expected given two independent binomial decision makers with matched accuracy , which we denote as ôh , m . This can easily be computed analytically [ e.g . 32 , equation 1 ] . Then , the average error consistency is given by E ( m ) : R→ [ −1 , 1 ] , m 7→ 1 |D| ∑ d∈D 1 |Hd| ∑ h∈Hd 1 |Cd| ∑ c∈Cd ( 1|Sd , c| ∑ s∈Sd , c bh , m ( s ) ) − ôh , m ( Sd , c ) 1− ôh , m ( Sd , c ) ( 3 ) Benchmark & toolbox . A ( m ) , O ( m ) and E ( m ) each quantify a certain aspect of the humanmachine gap . We use the mean rank order across these metrics to determine an overall model ranking ( Table 2 in the Appendix ) . However , we would like to emphasise that the primary purpose of this benchmark is to generate insights , not winners . Since insights are best gained from detailed plots and analyses , we open-source modelvshuman , a Python project to benchmark models against human data.3 The current model zoo already includes 50+ models , and an option to add new ones ( both PyTorch and TensorFlow ) . Evaluating a model produces a 15+ page report on model behaviour . All plots in this paper can be generated for future models—to track whether they narrow the gap towards human vision , or to determine whether an algorithmic modification to a baseline model ( e.g. , an architectural improvement ) changes model behaviour .

Summary: This paper addresses a very interesting and important question: to what extent the recent improvements in object recognition models make them closer to human perception. The authors do this by collecting human data on a wide range of ood benchmarks modeled after ImageNet, evaluating the behavior of a large number of recently developed ImageNet models on these benchmarks, and comparing them with the human responses. The authors also open-source all their data and analysis tools so people can evaluate their own models on the same benchmarks and compare them with human behavior in a fine-grained manner (on an image-by-image basis).