Passive attention in artificial neural networks predicts human visual selectivity
Authors: Thomas A Langlois
Haicheng Charles Zhao
Erin Grant
Ishita Dasgupta
Thomas L. Griffiths
Nori Jacoby
Venue: NeurIPS 2021 Oral (2021)
Keywords: Cognition
Attention
Interpretable AI
Computer Vision
Human Visual Perception

Abstract: Developments in machine learning interpretability techniques over the past decade have provided new tools to observe the image regions that are most informative for classification and localization in artificial neural networks (ANNs). Are the same regions similarly informative to human observers? Using data from 79 new experiments and 7,810 participants, we show that passive attention techniques reveal a significant overlap with human visual selectivity estimates derived from 6 distinct behavioral tasks including visual discrimination, spatial localization, recognizability, free-viewing, cued-object search, and saliency search fixations. We find that input visualizations derived from relatively simple ANN architectures probed using guided backpropagation methods are the best predictors of a shared component in the joint variability of the human measures. We validate these correlational results with causal manipulations using recognition experiments. We show that images masked with ANN attention maps were easier for humans to classify than control masks in a speeded recognition experiment. Similarly, we find that recognition performance in the same ANN models was likewise influenced by masking input images using human visual selectivity maps. This work contributes a new approach to evaluating the biological and psychological validity of leading ANNs as models of human vision: by examining their similarities and differences in terms of their visual selectivity to the information contained in images.

Passive Attention in Artificial Neural Networks Predicts Human Visual Selectivity
Thomas A. Langlois lab H. Charles Zhao Erin Grant Ishita Dasgupta',
Thomas L. Griffiths and Nori Jacoby 2.b
T.A.L. and H.C.Z. contributed equally this work. T.L..G. and N.J. contributed equally to this work. DDepartment of Computer Science, Princeton University "Computational Auditory Perception Research Group, Max Planck Institute for Empirical Aesthetics 'Deparmment of Electrical Engineering and Computer Sciences, UC Berkeley MDeepMind, New York *Department of Psychology, Princeton University
Abstract
Developments in machine learning interpretability techniques over the past decade have provided new tools to observe the image regions that are most informative for classification and localization in artificial neural networks (ANNs). Are the same regions similarly informative to human observers? Using data from 79 new experiments and 7,810 participants, reveal significant overlap with human visual selectivity estimates derived from 6 distinct behavioral tasks including visual discrimination, spatial localization, recognizability, free-viewing, cued-object search, and saliency search fixations. We find that input visualizations derived from relatively simple ANN architectures probed using guided backpropagation methods are the best predictors of shared component in the joint variability of the human measures. We validate these correlational results with causal manipulations using recognition experiments. We show that images masked with ANN attention maps were easier for humans to classify than control masks in a speeded recognition experiment. Similarly, we find that recognition performance in the same ANN models was likewise influenced by masking input images using human visual selectivity maps. This work contributes a new approach to evaluating the biological and psychological validity of leading ANNs models of human vision: by examining their similarities and differences in terms of their visual selectivity to the information contained images.
show that passive attention techniques
1 Introduction
The last decade has witnessed the rise of artificial neural networks (ANNs) that can match and even exceed human performance on variety of perceptual and cognitive tasks, ranging from image recognition [ to natural language processing and reinforcement learning [2]. Alongside the rapid development of these technologies, significant body of work aimed at improving the interpretability of these systems and comparing them to biological ones has also grown [3] 4 6. 77. In computer vision, techniques for probing which visual regions ANNs "attend to" when classifying images have been developed to visualize the receptive fields of convolutional layers as well as regions a visual input that most influence the class activations of the models [8] 9 10] II [2] In neuroscience.
researchers began to quantify the functional fidelity of leading ANNs as models the human visual system using both neural and behavioral benchmarks [33 [4] 15). Finally, cognitive scientists have developed techniques to compare the structure of ANN learned representations to human
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

psychological representations [7 [8]. All these efforts have contributed to our understanding of the biological and psychological validity of leading ANNs as models of biological vision, beyond just assessing their performance on standard object categorization benchmarks.
Many previous analyses of the correspondence between ANNs and human vision have focused on the representations used by the systems. However, a natural question is whether ANNs select information in the same way, and in particular whether they attend the same visual regions as humans when extracting information for visual object recognition and localization. Prior work has developed ANNs trained explicitly to predict human visual gaze [9), and even incorporated simulated foveated systems into the model design [20]. In addition, work comparing human attention to computational attention [2T 22 23 6 7 revealed that computational attention tends not to resemble human attention. However, this work also provides evidence that using human attention to supervise the training of network architectures devised to emulate the parallel pathways in the human visual system can improve performance, and yields more human-like visual features [6]. Still, relatively little work has attempted a more comprehensive examination of how a wide range of ANNs compare to multiple human behavioral measures, using the large variety of interpretability techniques that are now available to probe what visual information ANNs use.
Methods for gaining insight into what is learned by ANNs started with "passive" attention gradient- based approaches designed to reveal which visual inputs influence the class activation score the most [24]. More advanced techniques using deconvolution and guided backpropagation methods followed
9 252 as well as techniques that introduced novel design alterations, such as global average pooling layers and class activation mapping (CAM) to localize class-specific visual regions in the input images [0] Finally, more general approaches that could be applied to architectures without global average pooling [26] appeared, with some of the most recent contributions proposing class activation mapping techniques that do not rely on gradients at all [27]. Aside from this range of "passive" techniques, computer scientists have also developed CNN models that incorporate end-to-end trainable attention modules (28 [2] as a means for both improving interpretability and boosting performance. The full range of techniques now available for visualizing the information that is most relevant to ANNs offer an unprecedented opportumity to compare their results to biological analogues such as estimates of human attention, discrimination accuracy, and visual recognition over image regions.
Since the early 19th century 30 vision scientists devoted to the study of biological vision have also developed variety of experimental techniques for estimating the visual information used by the primate visual system when engaged in similar perceptual and cognitive tasks such as visual search, localization, and recognition. Among these are measures of visual change sensitivity (discrimination accuracy), visuospatial memory (spatial localization estimation), as well as explicit reports of visual recognizability. In this work, we obtained estimates from six different perceptual tasks, including explicit visual recognizability estimates using recent behavioral task [31 32 (Fig. [A and Fig. A
two-alternative forced choice (2AFC) change sensitivity task (Fig. B and Fig. 24, eye-tracking
fixations (Fig. C and Fig. 2 and a spatial localization task (Fig. D and Fig. A).
We then compared them to estimates of ANN visual selectivity using variety of pretrained models and visualization techniques including guided backpropagation and techniques based on class activa- tion mapping. We used range of model types, including early convolutional networks like AlexNet I.
as well as recent of the-art models, such as Visual Transformer (ViT) (33 and EfficientNet models.
34
We find that only a select class of ANN models and passive attention techniques capture the shared variance across all human visual selectivity measures. This work contributes to current efforts aimed at evaluating the biological and psychological validity of contemporary ANNs by investigating the similarity between artificial and biological vision systems at the level of the visual inputs rather than the learned representations or correspondence to patterns of neural activation in the visual cortex [3] We see our primary contributions as two-fold: (1) In a departure from prior work, we use larger range of human behavioral measures, ANN models, and attention techniques in our comparisons. These reveal a significant range in the degree to which different models combined with different attention techniques produce human-like results. In addition, they show that more human-like ANN attention captures shared component of the joint variability between the different human measures, rather than any particular human measure, (2) We provide causal evidence that more human- ANN attention directly influences both human classification performance as well as predictions made by ANNs.
2

---

2 Human visual selectivity measures
We computed 6 behavioral measures with 25 images, performing experiments in which total of 4,050 participants took part (see Table 1 in the Appendix). We ran experiments for 3 distinct human behavioral tasks for each image (see Fig. 1 and Fig. for representative examples). A total of 1,575 participants took part in the discrimination accuracy experiments (an average of 63 participants for each of the 25 images), and a total of 225 participants took part in each of the image patch ratings tasks (9 participants per image). Finally, the spatial memory serial reproduction chain experiments were completed by a total of 2,250 participants (an average 90 participants for each the images). Participants were recruited anonymously over Amazon Mechanical Turk (AMT), and provided informed consent. Participants were paid approximately S7 per hour. Details of the experimental procedures, design, and map estimation for each task can be found in Appendix Fig. S1.
image patch recognizability ratings
spatial memory reproduction
PATCH
CONTEXT
Flam
TASK: How informative & the conortt of the sotch?
TASK Nomenbee the sract position of the rol dos in the image
B. Human charge sensitivity (discrimination) task
E. Artificial Neural Network (ANN) Attention Map Example
Presentation
ged
Parsive Attention Exampler Guided Backpropagation
204 c4
12
EXITTE-
DIPRIEE
TESITION
SOFTMAK LAVER
TASK: Paln same shiftes!
C. Human traciing: free view, abject saliency search
INPUT IMAGE
1016400 TETCION
FCB
FC7 FG
DONV
CONV5
CONV3
(VGG MODEL EXAMPLE)
DONV
CONV1
Human behavioral tasks, and ANN attention. A. Informativeness patch ratings task. B. Discrimination accuracy 2AFC task. C. Eye-tracking for free search, object search, and saliency search. D. Spatial memory serial reproduction task. E. ANN attention (passive attention example). Details of the experimental procedures and map estimation are included in Appendix Fig. 1.
Figure 1:
designed to measure the 1. Visual recognizability. We adopted a recent behavioral task [3 32
informativeness and recognizability local image regions by using explicit self-reports (see Fig. [ In the task, participants view small circular image patches sampled from full images and rate how "recognizable" or "informative" the content of the patch on a six-point Likert scale ranging from rating of "Very low recognizability" to "Very high recognizability" The patches were sampled from regular 12 x 12 grid over the image. Fig. 2 shows representative examples of the results following averaging over all the ratings in different spatial areas of the images, smoothing, and interpolation to produce continuous maps for each image. We ran 25 experiments with an average of participants per image (see Appendix for details of the map generation procedure). A total of 225 participants took part in the patch ratings experiments. Participants were paid $2 to complete 144 experimental trials.
2. Change sensitivity (Discrimination). We measured change sensitivity using a two-alternative forced choice (2AFC) discrimination task. In this task, participants viewed an image with small red dot superimposed on it for 1000 milliseconds. Following a 1000 millisecond delay, the same image was presented again with the dot in either the same exact location or in a slightly displaced location (see Fig BB) Participants were then asked to indicate if the dot was shifted or unchanged in the second presentation. Crucially, the locations of the initial dot locations were sampled densely from all possible locations a regular grid that spanned the dimensions of the image, in order to measure changes visual change sensitivity over the entire image (see Appendix for details). This task measures changes in visual acuity conditioned on different visual areas in an image, and has
3

---

been used as proxy for measuring variable encoding precision of different image regions [35.. We ran 25 experiments (one for each image) with an average of 63 participants per experiment. The overall number of participants for the discrimination tasks equalled 1,575. Participants were paid $1.50 to complete 120 trials the discrimination task.
3. Visuospatial localization. We used a recent behavioral paradigm based on serial reproduction that can reveal intricate spatial memory priors that guide visual localization estimation in humans [35]. In this paradigm the first participant views point superimposed on an image and then reproduces its location from memory. The next participant views the same image but with the point located in the position reconstructed by the previous participant. As in the "telephone game," the process is repeated, forming a chain of participants. For each image, there were a total of 20 iterations in the chains, for 250 initial random seed dot positions. This experimental procedure is known to reveal the spatial landmarks in visual scenes that bias human allocentric visuospatial representations (see (35 and Fig. [D). We ran 25 experiments (one for each image) with an average of 90 participants each. The overall number of participants was 2,250. Participants were paid a base rate of $1.00 for completing 105 trials in the spatial memory experiment but could earn up to $1.50 depending on accuracy in the task. Additional details are provided in the Appendix.
4. Fixations. Finally, we used existing dataset of human fixations obtained via eye-tracking when human participants were engaged in free-viewing task, a cued object search task, and a saliency search task 336]| (Fig. T.). We used the published data from the 75 experiments reported in (36).
3 ANN models, passive and active attention
We evaluated three standard deep convolutional neural network architectures (AlexNet (I) VGGNet 24], ResNet (37), as well as two state-of-the-art architectures (Vision Transformer (ViT) 33 and
EfficientNet (4) ). For each of these models, which do not have active attention modules, we obtained attention maps using range of passive methods described below. We also evaluated two built-in end-to-end trainable attention modules, described below. All models were pretrained on ImageNet 44). Unlike the ImageNet and CIFAR-100 is comprised of images of complex natural scenes. See Table 2 in the Appendix for a list of all models.
2012 (38], CIFAR-100 (3), or Places365-Standee
datasets, which are comprised of images of objects,
3.1 Passive attention
We used gradient-based techniques including guided backpropagation methods, as well as more recent techniques based on class activation mapping (see Fig. L for a schematic example of passive attention). The methods based on guided backpropagation effectively try to compute the sensitivity of the model's ss output with respect to each pixel in the input image, using various techniques for increasing the signal in these maps and decreasing noise. These methods include guided backpropa- gation (GBP) (25), guided gradients times the image (GBPxIM) 41], and SmoothGrad with guided backpropagation (SGBP) [42]. The methods based on class activation mapping compute linear combinations of the activation maps in the final convolutional layer of the model in order to determine the discriminative regions of the image used by the model. These methods include Grad-CAM (26..
Score- CAMe (27), and CAMERAS 43). See the Appendix for details on these passive attention methods.
3.2 Active attention
We evaluated two different active attention modules. These active attention modules are trainable, and they learn to generate masks which are applied to the input image (or to intermediate convolutional layers). These masks are effectively explicit attention maps, we do not have to use passive attention methods to try to discern the models' attention. These attention modules can be incorporated into essentially any standard CNN architecture, so chose few for which we were able to obtain pretrained weights (see the Appendix for details).
One attention module is described in the paper Learn to Pay Attention 28 to as I.PA.. LTPA inserts attention at three intermediate convolutional layers within a VGGNet architecture. The other active attention module is used in Attention Branch Networks (ABNs) [29].
which we will refer

---

ABN has separate "attention branch" that runs in parallel with a "perception branch," and is based on class activation mapping [T].
Passive attention predicts shared variance across human measures
Human experiments. Fig. 2 shows representative results from each of the human behavioral experiments. We found that the human maps correlated with one another though the correlations varied from r 0.14 to 0.86 (see Fig. 2B) due to variations in the visual regions that were the most implicated from one task to another. We obtained a single factor that captures the maximal amount of shared variance across all behavioral maps ("Human PC": see Fig. 2 by computing a linear combination of the results of the six experiments via Principal Component Analysis (see SI Appendix for details and formal description). The Human PC was correlated with each of the six behavioral measures (r 75, .74, .42, 81,, .81 and .73 for the informativeness maps, change sensitivity maps, spatial localization maps, and each of the fixation maps, respectively).
behavioral map examples, map examples
B. Human map cross-correlations
PATCH
D. P-RRIME
Saliency
Object
SPATIAL
Free
PC
.74
0.8
.59
.39
.32
.31
D-PRIME
.4
.4
.32
.28
0.6
EYE-TRACKING
Fixation
.86
.75
0.4
Saliency Search
0.2
Object
.14
C. Human PC map examples obtained via PCA
HUMAN PC
Figure 2: Human behavioral task maps and ANN maps. A. Representative examples of the human maps obtained for the (1) patch "informativeness" ratings task (2) The discrimination accuracy task (3) the serial reproduction spatial memory task (4) Free fixations (5) saliency search fixations and (6) cued object search fixations. Examples of ANN maps for the same images are also shown. B. Cross correlations of all the maps, including the linear combination of all the maps that predicts the maximal shared variance (Human PC), are shown. While most methods are relatively highly intercorrelated, there clear differences. Fixations were the most highly intercorrelated (r -72--66)) while spatial memory Kernel Density Estimates (KDEs) are only weakly correlated (r = .14-177) to the fixations, in line with previous findings [35]. Factor loadings of each of the six measures to the Human PC were uniformly high (r= .1). C. Human PC map examples.
ANN maps. We then compared the maps computed by the ANN attention methods (Fig. 3A Raw ANN maps are included in Appendix Fig. S3). We found that ANN maps varied significantly in terms of their level "smmoothness" depending on the attention method. Because of this, and in order to compare the human and ANN maps in way that is agnostic to the raw smoothness of the ANN maps, we introduced a smoothing parameter when comparing the two. We optimized the smoothing parameter for each of the ANN maps based on the correlation of the result to each of the human behavioral maps including the Human PC. We did this by applying the same smoothing to each of the individual 25 image ANN maps, and then computing the average Pearson correlation of those maps to the corresponding human maps for each of the 25 images. We repeated this process for each ANN attention method, and for each human task.
The relation between ANN and human maps. Next, we explored relation between each human measures (including the Human PC) and the ANN attention maps. Fig. 3 shows the optimal correlations between human and ANN maps for each human map type including the Human PCC The peak correlations between the Human PC maps and the most highly correlated ANN maps exceed the
5

---

mapo
Conclations ANN shared PC, and Patch
PEAK CORRELATIONS HIGHER FOR PATCH RATINGS
0.3
Example ANN PC human behawicral maos
0.4
2 Corve
FRST IC MANAA DANHGAATIONL 1APPS
CORRELATIONS FOR HUNAN PC
Human labals original ol imagee
maps
Figure 3: Human behavioral maps and ANN attention. A. cross-correlations between each of the human maps (including the shared Human PC) and the ANN maps. Results are sorted according to the average peak correlations across the six human measures and PC maps. Blocks highlighted in red correspond to ANN maps obtained using SGBP, GBP, or GBPxIM methods. Blocks high- lighted in green correspond to maps obtained from LTPA and ABN active attention models. Blocks with blue highlights correspond to maps obtained using 2AM-based methods. Results show that maps obtained using GIBP-based attention methods better aligned with human behavioral maps than both CAM--asedd results and active attention results (p < 0.001, and p < 0.001). The nam- ing convention for passive attention maps is <architecturee-ccassiiee attention methods-<l/C/P for ImageNet/CIFAR J00/Places3655. The naming convention for active attention maps is <attention module>-<ardhe <archhiecture--acivv--Iii for ImageNet/CIFAR- 100>. B. Correlations between all ANN maps and Human PC (x-axis) and the correlation between all ANN maps and the Human Informa- tiveness ratings. Error bars were estimated from 100 bootstrapped samples of the human data. The red boxes indicate the ANN maps used for the human speeded recognition task. The blue shaded area represents the scenario where the Human PC correlations to ANN maps (optimized to Human PC maps) are higher than the corresponding correlations between human informativeness maps and ANN maps (optimized to human informativeness maps). C. Representative examples of the ANN maps that were highly predictive of the Human PC maps (r 0.71, p < 0.001), and not highly predictive of the Human PC maps (r = 0) are shown. Also shown are the Human PC maps for the same images.
peak correlations of the same ANN maps to each of the six behavioral maps (Fig. 3: PC average r r= 0.71, compared with r = 0.36-0.65 for each of the human maps). This suggests that the peak ANN maps predict the shared component of the variance between all the human measures rather than the variance of any particular human map type (such as human fixations or discrimination accuracy for instance). This is significant because it indicates that some intrinsic aspects of visual information contained in the images captured by the peak ANN models are predictive of human visual selectivity regardless of the behavioral task, and in spite of the systematic differences between them [35]. Fig. BB illustrates this fact for the human patch ratings task results. For most ANN maps, the correlation values under the diagonal in the plot indicate that the peak correlations of the ANN models to the Human PC were significantly higher than the peak correlations of the same ANN models to the Human patch ratings results. Fig. S4 illustrates the same finding for all individual human behavioral measures. In addition, Fig. B3 shows the top correlations for all ANN maps reaching a peak of r= 0.71 for the SGBP method applied to the AlexNet network pretrained In a separate analysis, we repeated the smoothing parameter fitting using split-half cross- vross-validation, and found that performance of smoothed hold-out test set maps using smoothing parameters fit to random training set maps produced nearly identical ranges in peak correlations to the human PC (between 0.73 and -0.01 for the training set, and between 0.72 and -0.04 for the testing set). as well as a nearly identical rank order in peak correlations to the human PCC Details of this analysis are included in the SI Appendix (see Fig. S8).
ImageNet (alexnet-SGBP-I).
natural question concerns whether the peak correlations achieved by the leading ANN maps are due to the model architecture, attention method, or training set. Fig. 3A shows correlations of the

---

ANN maps to the human measures by attention type. We used the average correlations across the 6 measures and images as a dependent variable and found that the attention method category explains 35.9% of the variance, while architecture and training set categories explain 25.3% and 1.5% of the variance, respectively (see Fig. AA for the average performance across all map types, and for each of the human map types). This result suggests that attention type plays the most significant role in our findings. Surprisingly. SGBP applied to one of the simplest architectures (the AlexNet network pretrained on ImageNet) showed the highest peak correlation to the Human PC (r = 0.71), and was significantly more predictive of the Human PC and human patch ratings than most of the other ANN maps (p 0.001 with Bonferroni corrections applied).
Passive attention models were consistently predictive of the human maps across behavioral tasks (See Appendix Fig. S2, S3 and S4). However, there was significant variation in the performance of these methods. For the human PC results, Guided saleeeddbbccpoopgattioon based attention results (GBP, SGBP and GBPxIM; averaged r = 0.51) consistently outperformed active attention results (averaged r= 0.38. = =..001, and these in turn outperformed Class Activation Mapping (CAM) based attention results (Grad-CAM, Score-CAM and CAMERAS, averaged r= 0.27, p 0.001). In addition, we found that the resnet models (averaged r= 0.54) produced more human-like maps than the remaining models (averaged ranged between 0.25 0.42. p 0.001 in all cases; with the Bonferroni correction applied). Finally, comparing results from the model types for which we could extract both active and passive attention maps reveals that biinded-bbckrropgggae
based attention produces more human- like results than active attention architectures (See SI Appendix for details).
5 Validation experiment: ANN visual selectivity influences human recognition
We showed that the shared component of the variability across human behavioral measures is best predicted by attention maps computed using a particular class of passive attention methods, suggesting that recognition performance in both humans and machines is derived from the same visual information in images. However, this finding is based on indirect correlational evidence. In order provide direct evidence for this claim, we tested whether human recognition performance is improved in real-time using the leading ANN maps via a speeded recognition experiment. We reasoned that recognition performance in humans will be better for images masked using their own attention maps ("correct masking") obtained from the best ANN maps (as defined by their peak correlation to the Human PC) compared with images masked using maps obtained for different images ("incorrect masking." see Fig. A for the masking procedure, and Fig. 4 for the speeded recognition task design). Furthermore, we predicted that the difference in recognition performance on this task between these two conditions (correct masking vs. incorrect masking) would be greater the masks are generated using the leading ANN maps than if they are generated using ANN maps that are very dissimilar to the Human PC maps.
To test these hypotheses, we ran three additional behavioral experiments with a total of 3,600 participants recruited from AMT. Each participant viewed briefly flashed masked images (for 200 milliseconds), and had to select the best descriptors from a set of word pairs obtained from a separate labelling experiment in which a total of 160 participants took part (see Appendix). Masked images were generated by an element-wis wise product of grayscale image with the map produced by the ANN model, see Fig. AAA. Correct masking consisted in masking an image with the map produced by the ANN for that image. Incorrect masking consisted in masking an image with the attention map produced by the ANN for different images. Representative examples of the masks along with the original color images are shown in Fig. C
Based on the rankings of the ANN maps in terms of their peak correlations to the Human PC, we predicted that human recognition accuracy should be significantly more sensitive to the visual regions revealed by SGBP applied to the AlexNet model than those revealed by the peak maps produced by the same model using the scorecam or Grad-CAM attention methods. Paired t-tests revealed a significantly larger difference (p < 0.001) between the d' scores across map types in the correct masking condition (blue bars Fig. 4D) relative to the differences in the d' scores across map types in the incorrect masking condition (red bars in Fig. 4D; see Appendix for definition of d' scores). This interaction confirms the prediction that human recognition accuracy is in fact significantly more sensitive to the visual regions revealed by one of the models and attention methods with the highest peak correlations to the Human PC, but less sensitive to those revealed by passive attention techniques applied to model that yielded significantly lower peak correlations to the Human PC, even though
7

---

A. Human recognition: correct and incorrect masking
B. Experiment Design
ELENENTTWSS PRODUCT
COnTECT
Chai, iudo
image
Ligttouss mon
nAFC
WASKED IMAGE
WMM MASK
GRAYSCALE IMAGE
MAGKED IMAGE
IMACE
Gus street
C. Examples of correctly masked and incorrectly masked images
D. Recognition d' results
MASKING
COPANCT
Discrimination Accuracy (d)
ALEXNET- TEXKET-SGEI-IMAGEEEE MASKED EXAMPLES
8
ALEXNET-GRADCAE IMAGENET EXAMPLES
EDAT
SOHTHOLSE
Masking ANN similarity to Human PC
Human labels for original images
Figure 4: Human recognition results. A. Masking procedure. We masked images by an elementwise product of the grayscale image with the ANN map. B. Experimental design. Participants viewed a masked image for 200 milliseconds. They then selected the word descriptors that best described the image. C. Example original images, and corresponding masked versions obtained from the SaageNet-pretrainee alcxnet-SGBP maps (high peak correlation to Human PC), and from the same model also pretrained ImageNet using Grad-CAM attention (low peak correlation to Human PC). D. Overall d' results for the experiments using SGBP, Score- CAM, and Grad-CAM maps. Results reveal significantly higher d' for correctly masked images (blue bar) relative incorrectly masked images (red bar) for all 3 ANN map types (p < 0.001, < 0.001, p 0.05 for SGBP, Score-CAM, and Grad-CAM maps, respectively). In the correct masking condition (blue bars), there
significant difference between results for the SGBP maps relative to the results for the Score-CAM and Grad-CAM maps (p < 0.001). Error bars were computed from 1000 bootstrapped samples of the human responses.
correct masking did produce boost in recognition accuracy incorrect masking for all three map types (p < 0.001, 4). d' scores broken down by individual images can be found the SI Appendix, including the same results shown in terms of simple accuracy (% correct) rather than d' (See SI Fig. S6).
Validation experiment: human visual selectivity influences ANN recognition
The results of the human speeded recognition experiments suggested that we evaluate the same prediction in the opposite direction by asking whether ANN classification performance is similarly sensitive to the visual regions revealed by the human behavioral maps. To do this, we masked images using the six different human behavioral maps including the Human PC maps (Fig. 3). As with the human recognition experiments, we evaluated if ANNs show better performance in classifying correctly masked images rather than incorrectly masked images (i.e., images paired with the mask from a different image). Furthermore, we tested whether ANNs show improvements when the correct masks were obtained from the behavioral measures that had the highest peak correlations to the ANN maps (Fig. 3). See Fig. 5A for an illustration the whole procedure.
Measuring recognition. In this experiment, we evaluated how masking affects ANN recognition directly. We therefore used the ANN's classification of the original, unmasked image as a baseline, and then compared it to classification the corresponding (correctly or incorrectly) masked image.

---

Similar classifications in both cases would indicate good performance. We defined similarity across classifications as follows. For a given ANN, we took the top-1 category (i.e., the category predicted with highest confidence) for the unmasked image, and computed its rank in the predictions on the masked image (r). We then divided this rank by the total number of categories (N) to normalize for differences in the number of categories across ImageNet, CIFAR-100, and Places365 trained models. This new quantity, (r/N), inversely tracks recognition quality- it is lower when recognition is good, and higher when recognition is poor. To convert it into measure that gives higher values for better recognition and lower values for poorer recognition (as with the human d' accuracy metric), we use N/(r+ + N) our final measure. We refer to this measure the imverse-rant raerse-rank and computed it for all (correctly and incorrectly) masked images. Note that due to domain shift we expected the inverse rank to be less than one even in the correctly masked case.
Results. We computed the inverse-rank across all models, for all types of human maps (Section 2) as well as for both kinds of masking (correct, incorrect). We found that the correctly masked images are universally more recognizable (have higher mean inverse-rank) than incorrectly masked images, across each of the different human maps (Fig. [ p < 0.001). This finding validates our core prediction that ANNs should be sensitive to visual regions revealed by the human measures.
Masked images were also more or less easy to recognize based on the human map type, and we found a significant main effect of behavioral map type on the overall mean inverse-rank (F(6, (76) = 49.6999999 49.65, p < 0.001) in the predicted direction: human maps that had higher peak correlations to the ANN maps also produced higher overall inverse-rank scores. We also found significant interaction between masking condition (correct vs. incorrect) and map type, indicating that different human behavioral maps have direct impact on recognizability (the difference in inverse-rank between the correct and incorrect masking conditions (F(6,176) = 6.54. p < 0.001). This finding is key, because it confirms a change in recognition performance that is predicted from the differences in overall peak correlations of the ANN maps to the different human maps. Like the interaction we observed with the human recognition experiment, it shows that behavioral measures that had higher peak correlations to the ANN maps (such as the patch rating maps) also gave higher inverse-rank score differences across masking conditions, while others, like the fixation maps, gave smaller inverse-rank score differences across masking conditions. More details of the analysis and results can be found in Appendix Fig. S6.
ANN recognition: masking and rank change
B. ANN recognition. Meon Inverse Rank
ORIGINAL RGB
Get prod
1.
CONNECT
INCORRECT
Mean Inverse Rank Nirr+)
FX
Pass DORRECTLYY musked RC3B image
Messure rank di d'ange
AAA
0.7
INCORORECT mashed insagu
Messure rank drange
Sallenay
Figure 5: ANN recognition results. A. Measuring recognition. As outlined in the text, we directly evaluate how masks derived from human maps affect ANN recognition by examining how the rank of the top image category prediction when classifying an unmasked image changes when that same image masked with either the correct or incorrect mask. B. Mean Inverse-rank across different human maps grouped by correct vs incorrect masking, averaged across all ANN models and images. All human maps give higher inverse-ran -eeers--ankkaa for correct vs incorrect masking, validating the main hypothesis that ANNs are sensitive to visual regions highlighted by human maps. We also find an interaction in the predicted direction: the effect of correct masking tended to be greater when maps were from the behavioral results that were the most highly correlated to the ANN maps overall.
9

---

7 Discussion
Using a range of human behavioral measures, ANN models and attention techniques (Fig. 2 we attempted a comprehensive examination of the similarities and differences between humans and machines with respect their visual selectivity to image information. In departure from prior work, we found that ANN maps are optimally predictive of a latent human visual selectivity feature that captures the joint variability between our human behavioral measures (Human PC; see Fig. 3
Fig.
Surprisingly, simple architectures and passive guided backpropagation based attention techniques showed the highest peak correlations to the human data, and performed significantly better than the maps produced by active attention or class activation mapping based techniques. These results suggest that the same visual regions are informative to humans and machines. In addition, comparing models for which we could extract both active and passive attention maps revealed that active attention architectures consistently produce less human-like results (SI appendix section D.2). This noteworthy because active attention models have often been conceived with the goal of emulating human attention mechanisms.
In another departure from prior work comparing human attention to ANN attention, we validated our correlational findings by running two causal experiments. In the first, we took the ANN maps and used them to mask images presented to human participants. We found that humans were better classifying images that were masked with the correct ANN map compared with incorrect maps. and that the difference in performance between these two conditions was greater when the ANN maps used were more highly correlated to the Human PC maps (Fig. 4.. In the second experiment, we used human maps to mask images and measured the effect on recognition performance for the ANN models. Again, we found that incorrect masking was more destructive to ANN recognition performance than correct masking (Fig. [T]. We also found that the change in performance between the two conditions tended to be greater when the masking was done using human maps that were the most highly correlated to the ANN maps.
Our results suggest that the regions that discovered by attention techniques in both humans and ANNs are indeed mutually important for recognition. Morcover, these results show that we can use off-the-shelf ML methods (i.e., ANN and interpretability methods) to produce saliency maps that can predict human visual selectivity just as well as custom models (i.e. non-deep-learning visual-attention models) designed specifically for that purpose. We include predictions made by two such models: braph-based Visual Saliency (GBVS) and Boolean Map based Saliency (BMS) 45 in the SI Appendix (see Fig. S2 and S4).
One of the main findings of this paper is the fact that more human-like ANN maps tend to be more predictive of the shared component of the joint variability between human measures rather than any single human measure. Although this finding may be due in part to a reduction in measurement noise from summing the individual behavioral maps, the fact that we observed systematic variations in the visual regions that are implicated depending on the behavioral task (Fig. B), as well as relatively high internal reliability for each of the estimates (split-half correlations between p = 0.63 0.92, see SI Appendix), suggests otherwise.
The main limitation of this work is the inclusion of relatively small number of images. This is due to the large number of participants needed in order to create detailed estimates for all the behavioral maps for each of the images (requiring over a hundred participants for every image). In addition, further work will be required to fully explain the factors that contribute to greater similarity between artificial attention and human visual selectivity, and ways in which we can devise systems that better models of biological vision. In addition, while making artificial networks more human- like has practical advantages for improving their interpretability, pitfalls include introducing undesirable biases [46 47 48)
Overall, our results pave the way for developing new psychologically relevant benchmarks for evalu- ating leading ANN models, beyond comparing them to the neural basis of biological vision, or the distributions of their learned representations to the structure of human psychological representations. These results showcase new ways of combining the perspectives of machine learning and cognitive science towards developing more human-like intelligent systems.
10

---

Acknowledgements
This work was funded by Princeton University (Grant number 1718550 from the National Science Foundation) as well as the Max Planck Society.
References
[1] Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25:1097- 1105, 2012.
[2] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7559::4364444 2015.
[3] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James DiCarlo. performance-optimii hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111 (1((2))8619
2014.
[4] Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman Norman-Haignent Haignere. and Josh H McDermott. A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3):630-644, 2018.
[5] Daniel Yamins. An optimization approach understanding sensory systems. The Cognitive Neurosciences, 4(81)) 2020.
[6] Drew Linsley, Dan Shiebler, Sven Eberhardt, and Thomas Serre. Learning what and where to attend. arXiv preprint arXiv: 1805.08819 2018.
[7] Drew Linsley, Sven Eberhardt, Tarun Sharma, Pankaj Gupta, and Thomas Serre. What are the visual features underlying human versus machine vision? In Proceedings of the IEEE International Conference Computer Vision Workshops, pages 2706-2714, 2017.
[8] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048-2057 PMLR, 2015.
[9] Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, pages 818-833. Springer, 2014.
[10] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2921-2929, 2016.
[11] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. In Proceedings of the 27th International Conference on Neural Information Processing Systems, volume 2, pages 2204-2212. 2014.
[12] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017.
[13] Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Ha Hong, Najib J Majaj. Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Jonathan Prescott-Roy. Kailyn Schmidt, et al. Brain-like object recognition with high-performing shallow recurrent ANNs. arXiv preprint arXiv: 999.06161, 2019.
[14] Bria Long and Talia Konkle. The role of textural statisties outer contours in deep and neural responses to objects. In Conference on Computational Cognitive Neuroscience, page 4, 2018.
[15] Talia Konkle and George Alvarez. Deepnets not need category supervision to predict visual system responses to objects. Journal of Vision, 20(11):498 498, 2020.

---

[16] Ruairidh M Battleday, Joshua C Peterson, and Thomas L Griffiths. Capturing human cat- egorization of natural images by combining deep networks and cognitive models. Nature Communications, 11 11(1):1-14, 2020.
[17] Joshua C Peterson, Paul Soulos, Aida Nematzadeh, and Thomas I. Griffiths. Learning to generalize like humans using basic-level object labels. Journal of Vision, 19(10):60a-60a 2019.
[18) Brenden M Lake, Wojciech Zaremba, Rob Fergus, and Todd M Gureckis. Deep neural networks predict category typicality ratings for images. CogSei, 2015.
[19] Andronicus A Akinyelu and Pieter Blignaut. Convolutional neural network- methods for eye gaze estimation: A survey. IEEE Access, 1:14581-14220 2020.
[20] Arturo Deza and Talia Konkle. Emergent properties of foveated perceptual systems. arXiv preprint arXiv:2006. 07991, 2020.
[21] Qiuxia Lai, Salman Khan, Yongwei Nie, Sun Hanqiu, Jianbing Shen, and Ling Shao. Under- standing more about human and machine attention in deep neural networks. IEEE Transactions on Multimedia, 2020.
[22] Abhishek Das, Harsh Agrawal, Larry Zitnick, Devi Parikh, and Dhruv Batra. Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding, 163:90-100, 2017.
[23] Mohammad K Ebrahimpour, Ben Falandays, Samuel Spevack, and David C Noelle. Do humans look where deep convolutional neural networks "attend"? In International Symposium on Visual Computing, pages 53-65. Springer, 2019.
[24] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.600 2013.
[25] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 14126664, 2014.
[26] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 118-626,22 2017.
[27] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-CAM: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 24-25, 2020.
[28] Saumya Jetley, Nicholas Lord, Namhoon Lee, and Philip HS Torr. Learn to pay attention. arXiv preprint 1rXiv:1804.0339) 2018
[29] Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Attention branch network: Learning of attention mechanism for visual explanation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10705-10714, 14, 2019.
[30] Stanley Finger. Origins of neuroscience: A history of explorations into brain function. Oxford University Press, USA, 2001.
[31] John M Henderson, Taylor R Hayes, Gwendolyn Rehrig, and Fernanda Ferreira. Meaning guides attention during real-world scene description. Scientific Reports, 8(1):199, 2018
[32] John M Henderson and Taylor R Hayes. Meaning-based guidance of attention in scenes as revealed by meaning maps. Nature Human Behaviour. 1(10):433744,, 2017.
12

---

[33] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[34] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference Machine Learning, pages 6105-61 6114. PMLR, 2019.
[35] Thomas A Langlois, Nori Jacoby, Jordan W Suchow, and Thomas L Griffiths. Serial reproduc- tion reveals the geometry of visuospatial representations. Proceedings of the National Academy of Sciences, 118(13), 2021.
[36] Kathryn Koehler, Fei Guo, Sheng Zhang, and Miguel P Eckstein. What do saliency models predict? Journal of Vision, 14(3):14 14, 2014.
[37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[38] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hicrarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 258-255. IEEE, 2009.
[39] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[40] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[41] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. just a black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713. 2016.
[42] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: Removing noise by adding noise. arXiv preprint
2017.
[43] Mohammad AAK Jalwana, Naveed Akhtar, Mohammed Bennamoun, and Ajmal Mian. CAM- ERAS: Enhanced resolution and sanity preserving class activation mapping for image saliency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16327-16336, 2021.
[44] Jonathan Harel, Christof Koch, and Pietro Perona. Graph-based visual saliency. 2007.
[45] Jianming Zhang and Stan Sclaroff. Exploiting surroundedness for saliency detection: a boolean map approach. IEEE Transactions Pattern Analysis and Machine Intelligence, 38(5):889-99))) -02, 2015.
[46] Morgan Klaus Scheuerman, Kandrea Wade, Caitlin Lustig, and Jed R Brubaker. How we' vee taught algorithms see identity: Constructing race and gender in image databases for facial analysis. Proceedings of the ACM on Human-Compunn Interaction, (CCSCW1):1-2 2020.
[47] Angelina Wang, Arvind Narayanan, and Olga Russakovsky. Revise: A tool for measuring and mitigating bias in visual datasets. In European Conference on Computer Vision, pages 733-751. Springer, 2020.
[48] Zeyu Wang. Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8919-8928, 2020.
13


Reviews:
[{"review_id": "JSJanZtvWJf", "reviewer": "Reviewer_Mcif", "summary": "This paper investigates visual selectivity (i.e. which areas in images are most informative) in humans and artificial neural networks. Human attention maps are generated using different behavioral tasks, while artificial attention maps are generated from different ANN architectures. A simple ANN architecture with a passive attention mechanism is revealed as the best predictor of human visual selectivity. Using recognition experiments, the authors show that ANN-based attention maps enhance human recognition and viceversa. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper describes a comprehensive investigation of the correspondence between human and machine attention. The authors compare behavioral and eye tracking data to an array of ANN architectures with different attention mechanisms. The question of whether the areas in images that ANNs pay attention to are similar to those humans pay attention to is an interesting one, and it's great to see it addressed using a wealth of experimental data and ANN architectures.\n\nThe authors mention some previous work on this topic, but it's not very clear to me where this work sits in relation to this context (e.g. more comprehensive approach/different methods?) Furthermore, the work they cite mostly concludes that visual selectivity is different between ANNs and humans; it would be interesting to see some discussion of why the authors think their work reaches a different conclusion.\n\nAlthough correlations between the different behavioral measures are reported, it would be great to also get an idea of the cross-participant reliability in the behavioral experiments. This could help understand the level of noise in the data and could even be used to estimate a noise ceiling for how well a good model would be expected to fit the data.\n\nFurthermore, I found the combination of results from different behavioral tasks and the human PC presented in a slightly confusing way. Given that most behavioral measures do correlate fairly well, I am not sure the authors' interpretation that the human PC reflects a shared latent component is the only possible one. Would it not be possible that reducing the data in this way removes some of the variability due to noise and thus improves the results? I would also like to better understand how the human PC is calculated – is this the first component resulting from PCA and if so, how much variance does it explain? The explanation in appendix C1 seems a bit unclear on this. \n\nFurthermore, since the human PC is used as the 'best measure' of human behavior, I am wondering why it was not used to mask the images presented to the ANN in Section 6 (as far as I can tell this section discusses separate masks based on the different behavioral tasks).\n\nThe optimization of the smoothing parameter is another aspect that could be explained a bit more clearly (especially in lines 171-175). Since the parameter was optimized based on the target correlation, I wonder whether this introduced some circularity in the analysis. Do the ANN architectures rank the same if the correlation is calculated between the raw?\n\nIt is great to see validation experiments addressing the overlap between human and machines in terms of visual selectivity. Although these experiments show that human/ANN maps improve recognition compared to randomized maps, they don't address the usefulness of these maps relative to comparable methods (e.g. masks derived from different participants or different ANN architectures). It is hard to tell from these results whether these maps really provide an 'advantage' in recognition, and it would be great to see these aspects mentioned in the discussion. Furthermore, it is difficult to judge from Figure 4D how meaningful the difference in d' is - what would be the expected d' of the 'true' model? Such difficulties in model comparison are common in this type of work, but could be discussed a bit more.\n\nFinally, the main finding of this paper seems to be that a relatively simple model, without an active attention module, captures human visual selectivity patterns better than more complex models. However, the discussion seems relatively limited in interpreting the results, e.g. are there any conclusions that can be drawn from this about humans/ANNs? How do the authors reconcile the fact that the type of attention explains much of the variance in performance, but on average active attention models do as well as passive attention ones? \n\nMinor comment: In places, some analyses could be described more clearly (eg. lines 195-198, the analysis done to quantify variance is described rather vaguely)\n\n-----------------------------------------\nAfter author response:\n\nI thank the authors for addressing my comments so thoroughly. I think many of these issues just required more clarification, and my concerns have been addressed well in this rebuttal. In regards to Q8 - I did not mean to suggest that a different measure of accuracy should be used. By 'true' model, what I meant was that there is some individual variability in human behavior too, and so a 'great' result might actually be expected to sit below 100%. I think expanding the discussion, plus the additional checks performed by the authors, will improve the paper, and so I'm increasing my rating to 7. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors acknowledge the main limitations of the work, as well as potential impacts. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Cpmhhtw4n5v", "reviewer": "Reviewer_EJ2y", "summary": "This paper creates artificial attention maps for 25 images, using a number of passive methods for creating attention maps crossed with four different convnet architectures, as well as two “active attention” models. These attention maps are correlated with human maps from three different experiments designed to test sensitivity or information across an image and three existing datasets of human eye tracking experiments in three different tasks using the same 25 images. The three experiments run here were 1) an experiment to measure how informative humans consider patches of an image compared to the original image, resulting in an information map; 2) a change sensitivity experiment where a dot is moved slightly, at a fixed grid over the image, resulting in a d’ map over the image; and 3) a kind of mmc with people experiment where subjects indicate where they think a dot was in an image and the next subject has to make the same judgment using the dot placed by the previous subject. \n\nFrom these six experiments, they compute the first principal component of the six maps, which they term the “human PC”. They then compute the correlation of the various combinations of attention method with architecture, and find that Smooth Grad with guided backpropagation (sgbp) applied to AlexNet trained on ImageNet provides the highest correlation with the human maps. Since this is correlational, they then test how well humans can make judgments of images using the maps, and again, AlexNet with sgbp works best compared with a low-performing model, and in general, images gated by the correct maps for the image vs. the map for another image are statistically significantly better. Similarly, they came up with a measure of how well the human maps worked for the different models. \n", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, this review is a bit brief because the paper is obviously excellent.\n\nOriginality: This is highly original, although there have been several previous attempts to do this, that are all cited. I would have preferred that the differences between what had been done in the past and the current experiments had been described, instead of just citing the three prior works. \n\nQuality: High. One slightly odd measurement was the way they decided how to rate the models’ performances using the human maps. They took the highest output for each network (e.g., which of the 1k outputs was highest for a network trained on imagenet) and measured how far that output fell in the ranking of outputs when given the correct human map vs. the incorrect one for the image. This was done because the masked images were out of distribution for the models. I would have preferred that they used transfer learning on the images to train a final linear layer to distinguish the 25 images, and then see which mask  changed the output the most. But I’m ok with what they did.\n\nI would have liked it slightly better if they had completed the 2x2 table and tested humans with the human maps and the networks with the network maps, providing an upper bound to compare with. \n\nClarity: The paper is clearly written.\n\n Significance: The results are quite interesting, as they give a new methodology for rating how human-like the information used by the neural nets is, and which method for computing an attention map gives the most human-like results\n. \nLimitations and societal impact. They discuss how human biases might creep into their models and suggest that ways to ameliorate this should be investigated. They also mention the limitation that all this is based on only 25 images, but they justify this because of the large number of subjects needed for each experiment, which ran into the thousands.\n\nHaving read the authors' response, and the other reviews, I am now going to \"make a case for this paper [I] scored so well.\" Since the only really negative review is now from reviewer tunC (marginally below acceptance threshold), I will respond to that specifically.\n1) Cognitive Science has always had a place at NeurIPS. As a cognitive scientist myself, and a reviewer for over 30 years, I increasingly find that newer reviewers make this statement more and more often. It's a bit unfortunate that at a \"neural information processing systems\" conference, that reviewers ding papers that compare these systems to the best example we have of one, humans. Sigh.\n2) This paper makes substantial contributions to the science! They tested over 6k subjects! They compared four architectures and k different saliency mechanisms, where k is some small integer! They backed up the correlational results with causal studies! What more do you want? Yes, they didn't test hand-designed non-deep model saliency mechanisms, and now they did. One of these mechanisms worked as well as AlexNet with sgbp. Ok, but the point here is that saliency is an emergent property of the model and mechanism used - AlexNet was not designed as a model of saliency. \n3) They effectively responded to all of your complaints, as far as I can tell.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "8iaOzvgsVff", "reviewer": "Reviewer_tunC", "summary": "In this paper, the authors systematically compare visual saliency maps obtained from deep learning models against human visual selectivity maps obtained through a series of large-scale online experiments involving a variety of perceptual tasks. The correlation between maps greatly varies depending on neural network architecture, visualization technique and behavioral task. In some cases (e.g., SGBP technique applied to AlexNet architecture) the model’s map achieves a very good match with human’s maps, suggesting a remarkable similarity between humans and machines in terms of how they select information contained in visual stimuli. This finding is further corroborated by two recognition experiments, where performance of both humans and deep nets is shown to decrease when images are masked using “incorrect” maps.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This research work is interesting and novel: a systematic comparison between human visual selectivity and state-of-the-art machine vision models is still lacking. The methodology used to build human maps is taken from very recent experimental approaches and the methodology used to build ANNs maps is based on state-of-the-art techniques.\n\nQuality: The methodology is overall sound, and the perceptual tasks used to probe human observers are appropriate. It is also remarkable that the authors implemented a wide range of passive and active attention techniques to derive saliency maps and tested a good range of deep learning architectures. However, the paper as it stands has several weaknesses, which I list here:\n\n-\tI am not fully convinced that NeurIPS would be the best venue to present these findings (maybe a more cognitively oriented conference / journal could be a better option). However, validation of neural network models against humans is one of the subject areas of NeurIPS, hence I should acknowledge that this is a matter of opinion.\n-\tAll models considered have been trained on datasets (ImageNet and CIFAR) containing mostly single-object images, in contrast with the testing stimuli that represent complex visual scenes. I doubt that the visual saliency of a complex scene is simply the sum of visual saliency of single objects.\n-\tIt would be useful to include a “non-deep learning” baseline to better establish whether neural networks really represent the state-of-the-art in modeling human visual selectively. I was surprised that the authors did not even mention classical saliency-based methods (for a review, see Borji and Itti, “State-of-the-art in visual attention modeling” IEEE TPAMI 2012).\n-\tI think the results presented in Fig. 4D are particularly informative, but only if the “Low” column refers to the AlexNet-Scorecam (as mentioned in Line 235). However, from the caption of Fig. 4 it seems that the “Low” column refers instead to the AlexNet-Gradcam maps, which I think is not particularly interesting to show given that such masking condition returns almost unperceivable images. It would be more informative to show that SGBP is still better than Scorecam, even if that method achieves a decent correlation with human PC.\n-\tFig. 5 is never referred to in the text (it is mistakenly called Fig. 3 or 4).\n\nClarity: The paper is written in a very clear way and properly organized. I enjoyed reading it. The discussion has a good balance between technical details and more general topics related to human perception and cognitive psychology.\n\nSignificance: This work could promote further research and stimulate the interest of the deep learning community toward more systematic comparisons between neural network models and human observers.\n\n---------------------------\nI think the authors properly addressed my concerns. I am curious to read the final version of the paper with the new results. My scored changed to 6.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors explicitly discuss the limitations of their research approach.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see ethical issues related to the content of this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a real tour-de-force, containing "data from 78 new experiments and 6,610 participants"  and "6 distinct behavioral tasks including visual discrimination, spatial localization, recognizability, free-viewing, cued-object search and saliency search fixations."  Though this paper is different than the majority of NeurIPS submissions, I think the breadth of the experimentation, and the relation to current trends in DL make it a great contribution and a breath of fresh air.  I think it will be of interest to NeurIPS attendees, could spark some interesting discussion and inspire future work.
Decision: Accept (Oral)