Partial success in closing the gap between human and machine vision

1 Introduction . Looking back at the last decade , deep learning has made tremendous leaps of progress by any standard . What started in 2012 with AlexNet [ 1 ] as the surprise winner of the ImageNet Large-Scale Visual Recognition Challenge quickly became the birth of a new AI “ summer ” , a summer lasting much longer than just a season . With it , just like with any summer , came great expectations : the hope that the deep learning revolution will see widespread applications in industry , that it will propel breakthroughs in the sciences , and that it will ultimately close the gap between human and machine 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . perception . We have now reached the point where deep learning has indeed become a significant driver of progress in industry [ e.g . 2 , 3 ] , and where many disciplines are employing deep learning for scientific discoveries [ 4–9 ] —but are we making progress in closing the gap between human and machine vision ? IID vs. OOD benchmarking . For a long time , the gap between human and machine vision was mainly approximated by comparing benchmark accuracies on IID ( independent and identically distributed ) test data : as long as models are far from reaching human-level performance on challenging datasets like ImageNet , this approach is adequate [ 10 ] . Currently , models are routinely matching and in many cases even outperforming humans on IID data . At the same time , it is becoming increasingly clear that models systematically exploit shortcuts shared between training and test data [ 11–14 ] . Therefore we are witnessing a major shift towards measuring model performance on out-of-distribution ( OOD ) data rather than IID data alone , which aims at testing models on more challenging test cases where there is still a ground truth category , but certain image statistics differ from the training distribution . Many OOD generalisation tests have been proposed : ImageNet-C [ 15 ] for corrupted imaes , ImageNet-Sketch [ 16 ] for sketches , Stylized-ImageNet [ 17 ] for image style changes , [ 18 ] for unfamiliar object poses , and many more [ 19–29 ] . While it is great to have many viable and valuable options to measure generalisation , most of these datasets unfortunately lack human comparison data . This is less than ideal , since we can no longer assume that humans reach near-ceiling accuracies on these challenging test cases as they do on standard noise-free IID object recognition datasets . In order to address this issue , we carefully tested human observers in the Wichmannlab ’ s vision laboratory on a broad range of OOD datasets , providing some 85K psychophysical trials across 90 participants . Crucially , we showed exactly the same images to multiple observers , which means that we are able to compare human and machine vision on the fine-grained level of individual images [ 30–32 ] ) . The focus of our datasets is measuring distortion robustness : we tested 17 variations that include changes to image style , texture , and various forms of synthetic additive noise . Contributions & outlook . The resulting 17 OOD datasets with large-scale human comparion data enable us to investigate recent exciting machine learning developments that crucially deviate from “ vanilla ” CNNs along three axes : objective function ( supervised vs. self-supervised , adversarially trained , and CLIP ’ s joint language-image training ) , architecture ( convolutional vs. vision transformer ) and training dataset size ( ranging from 1M to 1B images ) . Taken together , these are some of the most promising directions our field has developed to date—but this field would not be machine learning if new breakthroughs weren ’ t within reach in the next few weeks , months and years . Therefore , we open-sourced modelvshuman , a Python toolbox that enables testing both PyTorch and TensorFlow models on our comprehensive benchmark suite of OOD generalisation data in order to measure future progress . Even today , our results give cause for ( cautious ) optimism . After a method overview ( Section 2 ) , we are able to report that the human-machine distortion robustness gap is closing : the best models now match or in many cases even exceed human feedforward performance on most of the investigated OOD datasets ( Section 3 ) . While there is still a substantial image-level consistency gap between humans and machines , this gap is narrowing on some—but not all—datasets when the size of the training dataset is increased ( Section 4 ) . 2 Methods : datasets , psychophysical experiments , models , metrics , toolbox . OOD datasets with consistency-grade human data . We collected human data for 17 generalisation datasets ( visualized in Figures 7 and 8 in the Appendix , which also state the number of subjects and trials per experiment ) on a carefully calibrated screen in a dedicated psychophysical laboratory ( a total of 85,120 trials across 90 observers ) . Five datasets each correspond to a single manipulation ( sketches , edge-filtered images , silhouettes , images with a texture-shape cue conflict , and stylized images where the original image texture is replaced by the style of a painting ) ; the remaining twelve datasets correspond to parametric image degradations ( e.g . different levels of noise or blur ) . Those OOD datasets have in common that they are designed to test ImageNet-trained models . OOD images were obtained from different sources : sketches from ImageNet-Sketch [ 16 ] , stylized images from Stylized-ImageNet [ 17 ] , edge-filtered images , silhouettes and cue conflict images from [ 17 ] 1 , and the remaining twelve parametric datasets were adapted from [ 33 ] . For these parametric datasets , [ 33 ] collected human accuracies but unfortunately , they showed different images to different observers implying that we can not use their human data to assess image-level consistency between humans and machines . Thus we collected psychophysical data for those images ourselves by showing exactly the same images to multiple observers for each of those twelve datasets . Additionally , we cropped the images from [ 33 ] to 224 × 224 pixels to allow for a fair comparison to ImageNet models ( all models included in our comparison receive 224× 224 input images ; [ 33 ] showed 256× 256 images to human observers in many cases ) . Psychophysical experiments . 90 observers were tested in a darkened chamber . Stimuli were presented at the center of a 22 ” monitor with 1920× 1200 pixels resolution ( refresh rate : 120 Hz ) . Viewing distance was 107 cm and target images subtended 3× 3 degrees of visual angle . Human observers were presented with an image and asked to select the correct category out of 16 basic categories ( such as chair , dog , airplane , etc. ) . Stimuli were balanced w.r.t . classes and presented in random order . For ImageNet-trained models , in order to obtain a choice from the same 16 categories , the 1,000 class decision vector was mapped to those 16 classes using the WordNet hierarchy [ 34 ] . In Appendix I , we explain why this mapping is optimal . We closely followed the experimental protocol defined by [ 33 ] , who presented images for 200 ms followed by a 1/f backward mask to limit the influence of recurrent processing ( otherwise comparing to feedforward models would be difficult ) . Further experimental details are provided in Appendix C. Why not use crowdsourcing instead ? Our approach of investigating few observers in a highquality laboratory setting performing many trials is known as the so-called “ small-N design ” , the bread-and-butter approach in high-quality psychophysics—see , e.g. , the review “ Small is beautiful : In defense of the small-N design ” [ 35 ] . This is in contrast to the “ crowdsourcing approach ” ( many observers in a noisy setting performing fewer trials each ) . The highly controlled conditions of the Wichmannlab ’ s psychophysical laboratory come with many advantages over crowdsourced data collection : precise timing control ( down to the millisecond ) , carefully calibrated monitors ( especially important for e.g . low-contrast stimuli ) , controlled viewing distance ( important for foveal presentation ) , full visual acuity ( we performed an acuity test with every observer prior to the experiment ) , observer attention ( e.g . no multitasking or children running around during an experiment , which may happen in a crowdsourcing study ) , just to name a few [ 36 ] . Jointly , these factors contribute to high data quality . Models . In order to disentangle the influence of objective function , architecture and training dataset size , we tested a total of 52 models : 24 standard ImageNet-trained CNNs [ 37 ] , 8 selfsupervised models [ 38–43 ] ,2 6 Big Transfer models [ 45 ] , 5 adversarially trained models [ 46 ] , 5 vision transformers [ 47 , 48 ] , two semi-weakly supervised models [ 49 ] as well as Noisy Student [ 50 ] and CLIP [ 51 ] . Technical details for all models are provided in the Appendix . Metrics . In addition to OOD accuracy ( averaged across conditions and datasets ) , the following three metrics quantify how closely machines are aligned with the decision behaviour of humans . Accuracy difference A ( m ) is a simple aggregate measure that compares the accuracy of a machine m to the accuracy of human observers in different out-of-distribution tests , A ( m ) : R→ [ 0 , 1 ] , m 7→ 1 |D| ∑ d∈D 1 |Hd| ∑ h∈Hd 1 |Cd| ∑ c∈Cd ( accd , c ( h ) − accd , c ( m ) ) 2 ( 1 ) where accd , c ( · ) is the accuracy of the model or the human on dataset d ∈ D and condition c ∈ Cd ( e.g . a particular noise level ) , and h ∈ HD denotes a human observer tested on dataset d. Analogously , one can compute the average accuracy difference between a human observer h1 and all other human observers by substituting h1 for m and h ∈ HD \ { h1 } for h ∈ HD ( which can also be applied for the two metrics defined below ) . 1For those three datasets consisting of 160 , 160 and 1280 images respectively , consistency-grade psychophysical data was already collected by the authors and included in our benchmark with permission from the authors . 2We presented a preliminary and much less comprehensive version of this work at the NeurIPS 2020 workshop SVRHM [ 44 ] . Aggregated metrics like A ( m ) ignore individual image-level decisions . Two models with vastly different image-level decision behaviour might still end up with the same accuracies on each dataset and condition . Hence , we include two additional metrics in our benchmark that are sensitive to decisions on individual images . Observed consistency O ( m ) [ 32 ] measures the fraction of samples for which humans and a model m get the same sample either both right or both wrong . More precisely , let bh , m ( s ) be one if both a human observer h and m decide either correctly or incorrectly on a given sample s , and zero otherwise . We calculate the average observed consistency as O ( m ) : R→ [ 0 , 1 ] , m 7→ 1 |D| ∑ d∈D 1 |Hd| ∑ h∈Hd 1 |Cd| ∑ c∈Cd 1 |Sd , c| ∑ s∈Sd , c bh , m ( s ) ( 2 ) where s ∈ Sd , c denotes a sample s ( in our case , an image ) of condition c from dataset d. Note that this measure can only be zero if the accuracy of h and m are exactly the same in each dataset and condition . Error consistency E ( m ) [ 32 ] tracks whether there is above-chance consistency . This is an important distinction , since e.g . two decision makers with 95 % accuracy each will have at least 90 % observed consistency , even if their 5 % errors occur on non-overlapping subsets of the test data ( intuitively , they both get most images correct and thus observed overlap is high ) . To this end , error consistency ( a.k.a . Cohen ’ s kappa , cf . [ 52 ] ) indicates whether the observed consistency is larger than what could have been expected given two independent binomial decision makers with matched accuracy , which we denote as ôh , m . This can easily be computed analytically [ e.g . 32 , equation 1 ] . Then , the average error consistency is given by E ( m ) : R→ [ −1 , 1 ] , m 7→ 1 |D| ∑ d∈D 1 |Hd| ∑ h∈Hd 1 |Cd| ∑ c∈Cd ( 1|Sd , c| ∑ s∈Sd , c bh , m ( s ) ) − ôh , m ( Sd , c ) 1− ôh , m ( Sd , c ) ( 3 ) Benchmark & toolbox . A ( m ) , O ( m ) and E ( m ) each quantify a certain aspect of the humanmachine gap . We use the mean rank order across these metrics to determine an overall model ranking ( Table 2 in the Appendix ) . However , we would like to emphasise that the primary purpose of this benchmark is to generate insights , not winners . Since insights are best gained from detailed plots and analyses , we open-source modelvshuman , a Python project to benchmark models against human data.3 The current model zoo already includes 50+ models , and an option to add new ones ( both PyTorch and TensorFlow ) . Evaluating a model produces a 15+ page report on model behaviour . All plots in this paper can be generated for future models—to track whether they narrow the gap towards human vision , or to determine whether an algorithmic modification to a baseline model ( e.g. , an architectural improvement ) changes model behaviour .

Summary: The authors employ a large-scale comparison between human classification behavior and a diverse range of machine vision models on 17 out-of-distribution (OOD) datasets as a way of attempting to comprehensively measure whether the field is making progress in closing the human-machine gap in this domain. They find that particular architectures and training objectives, as well as larger training datasets appear to be increasingly closing the gap when measuring OOD accuracy overall, but less so at the level of error patterns for individual images. They further identify subsets of their OOD task set where the human-machine gap is narrowing versus those where it is not.