SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guy Gaziv*
Michael J. Lee*
James J. DiCarlo
McGovern Institute for Brain Research, Dept. of Brain and Cognitive Sciences Massachusetts Institute of Technology
The visual object category reports of artificial neural networks (ANNs) are notoriously sensitive to tiny, adversarial image perturbations. Because human category reports (aka human percepts) are thought to be insensitive to those same small-norm perturbations – and locally stable in general – this argues that ANNs are incomplete scientific models of human visual perception. Consistent with this, we show that when small-norm image perturbations are generated by standard ANN models, human object category percepts are indeed highly stable. However, in this very same “human-presumed-stable” regime, we find that robustified ANNs reliably discover low-norm image perturbations that strongly disrupt human percepts. These previously undetectable human perceptual disruptions are massive in amplitude, approaching the same level of sensitivity seen in robustified ANNs. Further, we show that robustified ANNs support precise perceptual state interventions: they guide the construction of low-norm image perturbations that strongly alter human category percepts toward specific prescribed percepts. In sum, these contemporary models of biological visual processing are now accurate enough to guide strong and precise interventions on human perception.
Code Webpage
Based on empirical alignment, some fully-trained artificial neural networks (ANNs) are the current leading scientific models of the integrated mechanisms of the adult primate visual ventral stream – in particular, its multi-level neural computations and some of the visually-guided behaviors it supports [1-10]. However, individual ANNs are also notoriously susceptible to adversarial attack: Adding of a tiny (e.g., ultra-low norm  $\ell_2$ -norm) pixel perturbation to the model's input, which is optimized to disrupt the model's categorization of the original image [11-13]. Because human object category perception has been shown to be only weakly sensitive to those same small-norm pixel perturbations [14-16], if at all, this argues against those ANNs as scientific models of human visual perception [17, 18].
Moreover, human object category perception is empirically robust to random image perturbations of sufficiently low-norm, here estimated to be  $||\delta ||_2 \leq 30$  (see Methods). The corresponding assumption is that humans are robust to any perturbation with low norm (Fig 1a, but see [19, 20] for special cases). Our primary goal here was to re-examine this prevailing assumption, and to systematically compare human and ANN model categorization behavior in this low-norm pixel perturbation regime.
We were motivated by three recent observations: First, much recent work has been aimed at addressing the adversarial sensitivity of ANNs [21], including adversarial training [22] – a training scheme in which insensitivity to low-norm pixel perturbations is explicitly imposed by generating adversarial examples during model optimization (a procedure we refer to here as "robustification"). Intriguingly, the resultant robustified ANNs are not only less susceptible to adversarial attacks on new clean
Figure 1: Robustified models discover low-norm image perturbations that strongly modulate human category percepts. The prevailing assumption: Human object category percepts have complicated topology in pixel space, but are robust (i.e., stable) inside a low pixel budget envelope around most natural images (a). Guided by robustified ANNs, we discovered image perturbations within that envelope that strongly disrupt (b1) or precisely change (b2) human category percepts. (c1) Image category disruption rates of humans (black) and models on the same modulated images, replicating a large gap in behavioral alignment for a non-robust "vanilla" model, which is largely closed by adversarial robustification. Model curves show reports by "surrogate" model of the Guide Model type (i.e., "gray-box"). (c2) Robust models allow for precise Targeted Modulation (TM) of human behavior in the "human-presumed-insensitive" pixel budget regime. Shown are TM examples by  $\ell_2$  pixel budget of 30.0 in the low pixel budget regime, demonstrating that arbitrary source images can be locally modulated toward arbitrary target human category percepts. The presented robust model in all panels is adversarily-trained at  $\ell_2$  pixel budget of 3.0. Error bars,  $95\%$  CI over source images & human raters.
images [22, 23], but, unlike vanilla ANN attacks, those attacks introspectively seem to engage human perception [24-28]. Second, when tested on clean natural images, robustified ANNs have internal "neural" representations that are more closely aligned with the corresponding neural population representations along the primate ventral stream, relative to vanilla ANNs [29, 30]. Third, the adversarial sensitivity of the responses of individual "neurons" in deep layers of robustified ANNs has been reported to be comparable with the sensitivity of the responses of individual biological neural sites at the corresponding layer of the primate visual ventral stream [31]. In addition, the robustified ANNs were found to reliably predict [ultra-]low-norm perturbation directions in pixel space for which biological neural sites turned out to indeed exhibit unexpectedly high sensitivity.
Taken together, these observations suggested the possibility that robustified ANNs may have closed the human-to-model behavioral gap in low-norm perturbation sensitivity. If so, then because robustified ANNs still exhibit ubiquitous, dramatic category report changes with precisely targeted low-norm pixel perturbations, then humans must also exhibit such not-yet-detected, ubiquitous, massive "bugs" in their visual processing. And those bugs could be utilized as image-dependent "features" that may give rise to very strong, yet very efficient (i.e., low pixel budget) human perceptual modulation.
We therefore asked: Q1) have robustified ANN models closed the model-human gap in adversarial sensitivity? And, Q2) can arbitrary human percepts be specifically induced in the low pixel budget regime, where human percepts are believed to be highly robust? We asked these two questions in the context of nine-way visual categorization tasks that are performed both by ANN models (all parameters frozen) and by human observers: a test image is presented and one of nine category reports is made (by models and by humans). To create each such test image, a frozen ANN model is utilized by a standard image generation algorithm as a "guide" in pixel space (dimensionality  $\mathrm{D} = 150,528$ ; see Methods) to construct a low-norm pixel perturbation (length D) relative to an arbitrary start image for a prescribed "perceptual" goal. For Q1) the prescribed goal is: "Perturb this start image to disrupt the model's currently category judgement." And, for Q2) the prescribed
goal is: "Perturb this start image to induce a prescribed category judgement [X]." We use different guide models to generate these perturbations, and we then measure the effect (if any) of such image perturbations on the categorization behavior of: (i) the original ANN guide model (aka "white-box"), (ii) a surrogate ANN model from the same family as the guide model (aka "gray-box"/transfer test), and (iii) a pool of human observers at natural viewing times.
- We provide evidence for the existence of low-norm image perturbations of arbitrary natural images that strongly disrupt human categorization behavior.
- We provide empirical evidence for the existence of local (i.e., low-norm) perturbations in image space that support changes from the category percept induced in a human subject by an arbitrary start image to semantically very different perceptual categories.
- We show that some robustified ANN models have largely, but not completely, closed the model-to-human behavioral correspondence gap in low-norm perturbation sensitivity.
- We show that robustified ANN models coupled with a generator algorithm can support surprisingly precise, low-norm perceptual state modulations for humans.
2 Overview of approach and experiments
To study the effects of small image perturbations on human visual object categorization, we used a two-stage methodology: (i) generate small image perturbations predicted to modulate human behavior by highly-ranked models of the ventral visual stream and control models, then (ii) collect human object categorization reports in a nine-way choice task, identical to that performed by the model using Amazon Mechanical Turk surveys. This methodology allowed us to directly assess the alignment between model category judgements and (pooled) human category judgements in response to the same start images and the same perturbations of those start images.
2.1 Generating image perturbations predicted to modulate human behavior
We focused on two image perturbation modes conceptually illustrated in Fig 1b: (i) Disruption Modulation (DM), involving perturbations intended to induce model errors by driving the model's category judgement away from the ground-truth category label, irrespective of the alternatively-induced model category judgement (Q1 above); and (ii) Targeted Modulation (TM), involving perturbations designed to induce specific, "target" model category judgements (Q2 above). These two modes are also referred to as untargeted and targeted attacks, respectively [22, 32, 18]. Image perturbations were created by one of the four model families we considered here, without any guidance from human experimenters and without guidance from human subjects data.
To have a tractable perceptual category report set that avoids the problem of humans being unfamiliar with the 1000 fine-grained ImageNet classes, we focused on a subset of ImageNet that has been mapped to a basic set of nine classes, called "Restricted ImageNet" [32]. Using this nine-category image set, we randomly sampled starting images (aka "clean" images) from each class and, for each such starting image, produced a DM-perturbation image and eight TM-perturbation images (one for each of the eight alternative possible categories). Each such perturbation image was required to be within the pixel "vicinity" of the starting image by imposing a pre-chosen upper limit on the  $\ell_2$ -norm of the perturbation (aka "pixel budget"  $\epsilon$ , Fig 1a). To systematically test perturbation efficiency, we repeated this procedure for a wide range of pre-chosen pixel budget limits, focusing on the "human-presumed-stable" low-norm pixel budget regime ( $\epsilon \leq 30$ ; see Sec 1).
For Guide Models (GM) used to direct the construction of image perturbations, we focused on the ResNet50 ImageNet-pretrained model family [33]. These ANNs are among the currently leading models of primate ventral visual processing stream and its supported behaviors [34, 2, 31], and particularly so in their more recent adversarially-trained, "robustified" version [22, 32]. We thus considered four families of ResNet50 models, which differ only in their robustification level, starting from the non-robust, "vanilla" variant, through those commonly employed ( $\ell_2$  pixel budget at training of 3.0) [32], to new models which we trained for other robustification levels. Notably, we denote this adversarial training budget by  $\varepsilon$ , to be distinguished from the perturbation budget,  $\epsilon$ , which applies to any image-perturbation method.
For a fair quantitative comparison of model category judgements and human category judgements we addressed the "gray-box" setting [12, 11, 13], where we ask "surrogate" models - a model from
Perturbation pixel budget (€) [l2-norm]
Figure 2: Low-norm image perturbations discovered by robustified models strongly disrupt human category judgements. (a) The Guide Models used for Disruption Modulation (DM) image generation. (b) Disruption rates of humans and models. All panels share the same set of start images, and the four sets of perturbed images (one for each of the four Guide Models, color-coded). Y-axis shows the categorization "error" rate with respect to the category label of the (clean) start image. Human curves show average across subjects and trial repeats per image, with lapse-rate correction applied subject-wise. "White-box" refers to reports made by the Guide Model itself. "Gray-box" refers to reports by a second seed "surrogate" model of Guide Model type. Error bars,  $95\%$  CI by bootstrap over images & subjects. (b) Example of modulated images generated by robust model [3.0] (in red) and, in contrast, by vanilla model (in blue) in the low pixel budget regime.
the Guide Model's family, but which is initialized and trained using a different seed – to report its category judgements of the perturbed images (i.e., within family transfer test).
2.2 Measuring human object category percepts
We used Mechanical Turk for large-scale behavioral experiments of nine-way image categorization. Randomly-selected human raters provided their trial-by-trial category reports via a sequence of trials. In each trial a single test image – randomly-chosen from the full set of TM or DM perturbations images – was presented at the center of gaze for a fixed duration of 200ms, after which the rater was shown a category choice screen with nine category options ('dog', 'cat', 'frog', 'turtle', 'bird', 'primate', 'fish', 'crab', 'insect'). No pre-mask or post-mask was used. Raters were asked to report the category that best describes the presented image.
3.1 Disruption Modulation results
Fig 2 shows the results of the Disruption Modulation (DM) experiment. The DM image generation Guide Models include a non-robust (vanilla) model and three versions of robustified models, trained to different robustification levels, indicated by their perturbation pixel budget at adversarial training, namely, 1.0, 3.0, and 10.0. Each panel corresponds to measured category report "errors" on a different system: the guide model itself, a guide-surrogate model, and on humans. Note that "errors" here are defined with respect to the ground truth label of the start (unperturbed) image.
Examination of the effect of image perturbations on the Guide Model that designed them (Fig 2b, left) confirms previous reports that robustification strongly reduces white-box perturbation sensitivity near novel start images, as previously reported [11-13]. And the human DM results confirms previous reports [14-16] that human category percepts are only very weakly sensitive to low-norm image perturbations designed by vanilla ANNs (Fig 2b right, blue line).
In contrast, we report this novel finding: human percepts are massively disrupted by low-norm image perturbations discovered by robustified ANNs trained at an  $\ell_2$  pixel budget of 3.0 (red line). At an image perturbation budget of 30,  $\sim 90\%$  of human category reports are no longer the category of the start image. Visual inspection of some example test images (Fig 2c) allows introspection about the perceptual differences between image perturbations generated by vanilla guide models vs. those generated by robustified guide models. Notably, we report these strong disruptions in human category percepts when perturbing at budgets well above the budget used for model robustification, yet still well-below the typical pairwise natural image distance regime (see Supplementary Material, Fig 1). This result was not a priori guaranteed.
We sought to determine whether human perceptual disruptions are quantitatively aligned with the predictions of robustified ANNs. Because we cannot perform white box attacks on the human visual system, the fairest assessment of this is via comparison of effects on humans with effects on surrogate models. To facilitate this comparison, we re-plot the results of the robustified ANN model (level 3.0, red) on the same panel as the human data (Fig 1c1, black line). Those plots demonstrate: (i) the originally motivating qualitative "gap" between the very high sensitivity of vanilla (surrogate) ANNs and the almost complete lack of human perceptual sensitivity to the same perturbations, and (ii) that some robustified ANNs have dramatically closed that gap. Nevertheless, some human-model misalignment still exists, with humans typically requiring about twice the perturbation strength to produce a quantitatively matched level of perceptual disruption as that particular ANN model family (red vs. black in Fig 1c1 right).
We also noted that robustified models show stronger "white-box"-to-"gray-box" alignment. Namely, image perturbations driven by robustified models not only transfer effectively to humans but also exhibit stronger alignment to surrogates of their own model class, compared to vanilla models (compare Fig 2b left vs. middle graphs).
We further assessed the impact of robustification training level on our findings. We found that models, which were adversarially trained at reduced allowable budget of 1.0, were less human-aligned (cf. Fig 2b right, yellow vs. red lines). Increasing the allowable budget at training to 10.0 did not result in a significant qualitative difference in model-human alignment. However, it did lead to less effective human category disruptions per budget level (cf. Fig 2b right, brown vs. red lines).
In follow up experiments, we found these human perceptual disruption effects to be largely unaffected by test image viewing times in the natural viewing time range of 100-800ms (see Supplementary Material). In summary, our results demonstrate that robustified models reveal at least a subset of behaviorally meaningful image perturbations in the low pixel budget regime.
3.2 Targeted Modulation results
Fig 3 shows the results of the Targeted Modulation experiments – tests of each Guide Model's ability to shift human percepts toward prescribed target categories from arbitrary start images (see Sec 2.1). We first randomly selected start images from all nine object categories and perturbed each source image toward all different-class targets. We used several Guide Models: the non-robust (vanilla) model, three variations of robustified models with different levels of robustification, and an image interpolation baseline approach using randomly selected images from the intended target categories. We experimented with a range of  $\ell_2$  pixel budgets in the low pixel budget regime, using the same source images across all conditions. Fig 3a provides examples of images generated by the  $\varepsilon_{tr}[3.0]$  robustified model at a 30.0 pixel budget in the low budget regime.
We found that robustified models trained with an  $\ell_2$  pixel budget of 3.0 and 10.0 give rise to perturbations that are reliably reported by humans as the target category (Fig 3b). These models discovered specific image modifications that resulted in a  $60\%$  probability of humans choosing the prescribed target category at an  $\ell_2$  pixel budget of 30 (chance:  $\sim 11\%$ ). In comparison, the interpolation approach remained at baseline level throughout the low pixel budget regime. Consistent with the findings in the Disruption Modulation experiment, we found that perturbations made by vanilla model had no targeted effect on human category percepts.
The specificity of our Targeted Modulation is further demonstrated in Fig 4a. This panel shows examples of successful TM toward 'dog' and 'cat' for a robustified model  $(\varepsilon_{tr}[3.0])$ , and other baseline methods. The precise control of a model is shown in confusion matrices in Fig 4b. They further highlight the strong specificity of TM for the robust model at  $\ell_2$  pixel budget of 30 relative to a image interpolation baseline approach.
Figure 3: Robustified models enable budget-efficient precise targeted behavioral modulation.
Dependence on start images & target category percepts. To test the generality of these behavior modulation findings across different distributions of the start images, we conducted similar experiments (all at pixel budget 30) using source images from (i) Out Of Distribution (OOD) images of the same categories ('frog', etc.), (ii) Arbitrary Natural Images (ANI), collected from the internet and social media, and (iii) Uniform Noise Images (UNI). We found that the TM effect magnitude remained comparable even when start images were drawn from these different distributions. For In-Distribution (ID), OOD, ANI, and UNI we achieved normalized target category induction rates of of  $60\%$ ,  $51\%$ ,  $55\%$ , and  $70\%$ , respectively (Fig 4c). Notably, TM often fails to produce convincing images for some target classes when using UNI source images (see Supplementary Material).
To further test the generality of our approach, we conducted experiments using new, arbitrary target categories. We selected nine target classes from an external animal image dataset, using 60 randomly selected images to represent each class. We modified our TM method to optimize towards the centroid feature representation of the target class, obviating the need to train a new classifier for the new class space (see Sec 4). We found high TM specificity also under this setting (Fig 4d), with lower TM scores on average compared to previous settings. This is despite the challenges posed here, which include, in addition to using new target classes, the presence of source-image distribution shifts (i.e., OOD, ANI; For UNI see Supplementary Material).
Induction of composite target category percepts We next sought to test whether model-guided perturbations could move human percepts toward pairs of object categories. Because subjects were only allowed to report one of nine categories, a perfect success in this test would be inducing humans to report each of the two prescribed target categories  $\sim 50\%$  of the time. Fig 4e shows results for these "two-composite" experiments. These results indicate that, while not perfect, even under this unusual setting, robustified models often discover highly creative ways to combine multi-class features in a single image that are human-recognizable. Visual inspection of the modulated images further highlights that the multi-class modulation cannot be trivially attributed to tiling the relevant class features across different spatial locations – The budget constraint guides the model to find clever ways to combine rather than "just add". Notably, scaling to 3-composite directions under the restrictive budget of 30 makes TM highly challenging, and more sensitive to the specific start images (see Supplementary Material).
Optimal robustification level. We defined the Model TM Efficiency (MTME) to be the budget at which a modulatory effect size exceeding  $10\%$  probability of choosing the target is observed. To this end, we interpolated the curves from Fig 3 uniformly to 100 points in  $\ell_2$  pixel budget range of [0, 30], and identified the first budget at which the threshold is exceeded. We rank the models by
Figure 4: Robustified models achieve precise Targeted Modulation (TM) from any image start point to any target direction.  $(a, b)$  Behavioral modulation results obtained when using a robust model to drive toward all possible cardinal class directions (i.e., 1-hot).  $(c, d)$  Analysis of distribution dependence of source images and target directions, generalizing our precise targeted modulation finding beyond Restricted ImageNet.  $(e)$  Targeted Modulation toward hybrid directions of two classes. All plots and images are due to robust model  $(\varepsilon_{tr}[3.0])$  that perturbs natural source images at an  $\ell_2$  pixel budget of 30.0. All panels show directly comparable numbers. Error bars,  $95\%$  CI over source images & human subjects.