Strong and Precise Modulation of Human Percepts via Robustified ANNs
Authors: Guy Gaziv
Michael J. Lee
James J. DiCarlo
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision
Object Recognition
Human
Primate
Ventral Stream
Adversarial Examples
Behavior Modulation
Behavioral Alignment
TL;DR: Robustified ANNs reliably discover image perturbations in the "human-presumed-stable" pixel budget regime that can strongly disrupt human behavior, or can precisely drive human perceptual state transitions to prescribed target states.

Abstract: The visual object category reports of artificial neural networks (ANNs) are notoriously sensitive to tiny, adversarial image perturbations. Because human category reports (aka human percepts) are thought to be insensitive to those same small-norm perturbations -- and locally stable in general -- this argues that ANNs are incomplete scientific models of human visual perception. Consistent with this, we show that when small-norm image perturbations are generated by standard ANN models, human object category percepts are indeed highly stable.  However, in this very same "human-presumed-stable" regime, we find that robustified ANNs reliably discover low-norm image perturbations that strongly disrupt human percepts. These previously undetectable human perceptual disruptions are massive in amplitude, approaching the same level of sensitivity seen in robustified ANNs.  Further, we show that robustified ANNs support precise perceptual state interventions: they guide the construction of low-norm image perturbations that strongly alter human category percepts toward specific prescribed percepts.  In sum, these contemporary models of biological visual processing are now accurate enough to guide strong and precise interventions on human perception.

Strong and Precise Modulation of Human Percepts via Robustified ANNs
Guy Gaziv"
Michael Lee"
James J. DiCarlo
McGovern Institute for Brain Research, Dept. of Brain and Cognitive Sciences Massachusetts Institute of Technology
Abstract
The visual object category reports of artificial neural networks (ANNs) are notori- ously sensitive tiny, adversarial image perturbations. Because human category reports (aka human percepts) are thought to be insensitive to those same small-norm perturbations and locally stable general this argues that ANNs are incomplete scientific models of human visual perception. Consistent with this, we show that when small-norm image perturbations are generated by standard ANN models, human object category percepts are indeed highly stable. However, in this very same "human-press weeeen--resudddddeeeee cover low-norm image perturbations that strongly disrupt human percepts. These previously undetectable human perceptual disruptions are massive in amplitude, approaching the same level of sensitivity seen in robustified ANNs. Further, we show that robustified ANNs support precise perceptual state interventions: they guide the construction of low-norm image perturbations that strongly alter human category percepts toward specific prescribed percepts. In sum, these contemporary models of biological visual processing now accurate enough to guide strong and precise interventions on human perception.
regime, we find that robustified ANNs reliably dis-
Code
Wehpage
Introduction
Based on empirical alignment, some fully-trained artificial neural networks (ANNs) are the current leading scientific models of the integrated mechanisms the adult primate visual ventral stream in particular, its multi-level neural computations and some of the visually-guided behaviors it supports [1-10]. However, individual ANNs are also notoriously susceptible to adversarial attack: Adding of tiny (e.g., ultra-low norm formm)) pixel perturbation to the model's input, which is optimized to disrupt the model's categorization of the original image [11-13]. Because human object category perception has been shown to be only weakly sensitive to those same small-norm pixel perturbations [1 [14-16], if at all, this argues against those ANNs as scientific models of human visual perception [17, 18).
Moreover, human object category perception empirically robust to random image perturbations of sufficiently low-norm, here estimated to be ||8||| S 30 (see Methods). The corresponding assumption that humans are robust to any perturbation with low norm (Fig 1a, but see [19, 20] for special cases). Our primary goal here was to re-examine this prevailing assumption, and to systematically compare human and ANN model categorization behavior in this low-norm pixel perturbation regime.
We were motivated by three recent observations: First, much recent work has been aimed at addressing the adversarial sensitivity of ANNs [21], including adversarial training [22] a training scheme in which insensitivity to low-norm pixel perturbations is explicitly imposed by generating adversarial examples during model optimization (a procedure we refer to here as "robustification") Intriguingly. the resultant robustified ANNs are not only less susceptible to adversarial attacks on new clean
*Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Human category percepts over pixel space
Prevailing
(b1) Disruption Modulation (DM)
(b2)
Targeted Modulation (TM)
assumption
Perturbation disrupting original category percept
Small budget
crab
"dog'
Pixel /
Arbitrary start image
Perturbation inducing specilic category percept
Human calegory percepts presumed stable inside small C
'frog'
start Images
Example target categories 'insect'
(c1)
(c2)
'primate'
'fish'
Vanilla guide model
Robustified guide model
'crab'
Normalized behavioral
disruption (% errors]
100
Robustifed model
Vanilla model
ground truth "turtle" 'frog'
Gap
Gap closure
Humans
Humains
10
100
10
10
100
10
Perturbation pixel budget (c) (4-m
Figure 1: Robustified models discover low-norm image perturbations that strongly modulate
human category percepts. The prevailing assumption: Human object category percepts have complicated
topology in pixel space, but are robust (i.e., stable) inside a low pixel budget envelope around most natural images (a). Guided by robustified ANNs, we discovered image perturbations within that envelope that strongly disrupt (bl) or precisely change (b2) humun cutegory percepts. (cl) Image category disruption rates of humans (black) and models on the same modulated images, replicating large gap in behuvioral alignment for non-robust "vanilla" model, which is largely closed by adversarial robustification. Model curves show reports by "surrogate" model of the Guide Model type (i.e.. "gray-box") (c2) Robust models allow for precise Targeted pixel budget regime. Shown are TM
Modulation (TM) of human behavior in the "human-pres wwwpppeeddddiieeseeteeeeee examples by pixel budget of 30.0 in the low pixel budget regime, demonstrating that arbitrary source images can be locally modulated toward arbitrary target human category percepts. The presented robust model in all panels adversarially- trained at f2 pixel budget of 3.0. Error bars, 95% CI over source images human raters.
images [22, 23], but, unlike vanilla ANN attacks, those attacks introspectively seem to engage human perception [24-28]. Second, when tested on clean natural images, robustified ANNs have internal "neural" representations that are more closely aligned with the corresponding neural population representations along the primate ventral stream, relative to vanilla ANNs [29, 30]. Third, the adversarial sensitivity of the responses of individual "neurons" in deep layers of robustified ANNs has been reported to be comparable with the sensitivity of the responses of individual biological neural sites at the corresponding layer of the primate visual ventral stream [31]. In addition, the robustified ANNs were found to reliably predict [ultra- [ultra-]low-nne Jlow-norm perturbation directions in pixel space for which biological neural sites turned out to indeed exhibit unexpectedly high sensitivity.
Taken together, these observations suggested the possibility that robustified ANNs may have closed the human-to-mote behavioral gap in low-norm perturbation sensitivity. If so, then because robustified ANNs still exhibit ubiquitous. dramatic category report changes with precisely targeted oowmmorm pixel perturbations, then humans must also exhibit such not-yet-detected, ubiquitous, massive "bugs" their visual processing. And those bugs could utilized as image-dependent "features" that may give rise to very strong, yet very efficient (i.e., low pixel budget) human perceptual modulation.
We therefore asked: Q1) have robustified ANN models closed the model-human gap in adver- sarial sensitivity? And, Q2) can arbitrary human percepts be specifically induced in the low pixel budget regime, where human percepts are believed to by highly robust? We asked these two questions in the context of nine- way visual categorization tasks that are performed both by ANN models (all parameters frozen) and by human observers: a test image is presented and one of nine category reports is made (by models and by humans). To create each such test image, a frozen ANN model is utilized by a standard image generation algorithm as a "guide" in pixel space (dimensionality D = 150,528; see Methods) to construct a low-norm pixel perturbation (length D) relative to an arbitrary start image for a preseribed "perceptual" goal. For Q1) the preseribed goal "Perturb this start image disrupt the model's 's currently category judgement." And, for Q2) the prescribed

---

goal is: "Perturb this start image to induce a prescribed category judgement [X]," We use different guide models to generate these perturbations, and we then measure the effect (if any) of such image perturbations on the categorization behavior of: (i) the original ANN guide model (aka "white-box") (ii) a surrogate ANN model from the same family as the guide model (aka "gray-box" bor-a/xaxx///eerrr test), and (iii) pool of human observers at natural viewing times.
Our main contributions are as follows:
We provide evidence for the existence of low-norm image perturbations of arbitrary natural images that strongly disrupt human categorization behavior.
We provide empirical evidence for the existence of local (i.e., low-norm) perturbations in image space that support changes from the category percept induced in a human subject by an arbitrary start image to semantically very different perceptual categories.
We show that some robustified ANN models have largely, but not completely, closed the model-to- human behavioral correspondence gap in low-norm perturbation sensitivity.
We show that robustified ANN models coupled with generator algorithm can support surprisingly precise, low-norm perceptual state modulations for humans.
2 Overview of approach and experiments
To study the effects of small image perturbations on human visual object categorization, we used a two-stage methodology: (i) generate small image perturbations predicted to modulate human behavior by highly-ranked models of the ventral visual stream and control models, then (ii) collect human object categorization reports in a nine-way choice task, identical to that performed by the model using Amazon Mechanical Turk surveys. This methodology allowed us to directly assess the alignment between model category judgements and (pooled) human category judgements in response to the same start images and the same perturbations of those start images.
2.1 Generating image perturbations predicted to modulate human behavior
We focused on two image perturbation modes conceptually illustrated in Fig 1b: (i) Disruption Modulation (DM), involving perturbations intended to induce model errors by driving the model's category judgement away from the ground-truth category label, irrespective of the alternatively- induced model category judgement (QI above); and (ii) Targeted Modulation (TM), involving perturbations designed to induce specific, "target" model category judgements (Q2 above). These two modes are also referred to as untargeted and targeted attacks, respectively [22, 32, 18). Image perturbations were created by one of the four model families considered here, without any guidance from human experimenters and without guidance from human subjects data.
To have tractable perceptual category report set that avoids the problem of humans being unfamiliar with the 1000 fine-grained ImageNet classes, we focused on subset of ImageNet that has been mapped to basic set of nine classes, called "Restricted ImageNet" [32]. Using this nine-category image set, we randomly sampled starting images (aka "clean" images) from each class and, for cach such starting image, produced a DM-perturbation image and eight TM-perturbation images (one for each of the eight alternative possible categories). Each such perturbation image was required to be within the pixel "vicinity" of the starting image by imposing a pre-chosen upper limit on the Eg-no fg-norm of the perturbation (aka "pixel budget" c, Fig la). To systematically test perturbation efficiency, we repeated this procedure for a wide range of pre-chosen pixel budget limits, focusing on the "human--resuumed-ttahee"
low-norm pixel budget regime (e S 30; see Sec 1).
For Guide Models (GM) used to direct the construction of image perturbations, we focused on the ResNet50 ImageNet-pretrained model family [33]. These ANNs are among the currently leading models of primate ventral visual processing stream and its supported behaviors [34, 2, 31], and particularly so in their more recent adversarially-trainn considered four families of ResNet50 models, which differ only in their robustification level, starting from the non-robust, "vanilla" variant, through those commonly employed (l2 pixel budget at training of 3.0) [32], to new models which we trained for other robustification levels. Notably, we denote this adversarial training budget by E, to be distinguished from the perturbation budget, e, which applies any amage-perturbatin method.
"robustified" version [22, 32]. We thus
For fair quantitative comparison of model category judgements and human category judgements we addressed the "gray-box" setting [12, 11, 13], where we ask "surrogate" models a model from
3

---

(a)
(b)
Effect on Guide Model
Effect on surrogate model (aka gray-box)
Effect on humans
(aka white-box)
Proposed
100
Guide Models:
Normllzzed behavioral
budget range
disruption 5% erross
used to train
Vanilla
Robust E, [3.0]
Robust E, [3.0]
Robust E, [1.0]
Robust e, [10.0]
30.0
1011
100
10
10
10
10
100
101
Perturbation pixel budget (c)
(c)
ground truth
primatee
primesee
'dog'
'Irog'
'insect'
'turtle'
'bird'
Tish"
insect'
'insect'
start Images
Perturbation
pixel budgat
C-10
6=00
Vanilla guide model
Robustified guide model (3.0)
Figure 2: Low-norm image perturbations discovered by robustified models strongly disrupt
human category judgements. (a) The Guide Models used for Disruption Modulation (DM) image generation. (b) Disruption rates humans and models. All panels share the set of start images, and the four sets perturbed images (one for each of the four Guide Models, color-coded). Y-axis shows the categorization "error" rate with respect the category label of the (clean) start image. Human curves show average and trial repeats per image, with lapse-rate correction applied subject-wise. "White-box" refers to reports made by the Guide Model itself. "Gray-box" refers reports by second seed "surrogate" model of Guide Model type. Error bars, 95% CI by bootstrap over images & subjects. (b) Example of modulated images generated by robust model [3.0] (in red) and, in contrast, by vanilla model blue) the low pixel budget regime.
subjects
the Guide Model's family, but which is initialized and trained using a different seed - to report its category judgements of the perturbed images (i.e., within family transfer test).
2.2 Measuring human object category percepts
We used Mechanical Turk for large-scale behavioral experiments of wine-way image categorization. Randomly-selected human raters provided their trial-by-trial category reports via sequence of trials. In each trial single test image randomly-chosen from the full set of TM DM perturbations images was presented at the center gaze for a fixed duration of 200ms, after which the rater was shown category choice screen with nine category options (" ('dog', 'cat', 'frog', 'turtle', 'bird', 'primate', 'fish', 'crab', 'insect'). No pre-mask or post-mask was used. Raters were asked to report the category that best describes the presented image.
3 Results
3.1 Disruption Modulation results
Fig 2 shows the results of the Disruption Modulation (DM) experiment. The DM image generation Guide Models include non-robust (vanilla) model and three versions of robustified models, trained to different robustification levels, indicated by their perturbation pixel budget at adversarial training, namely, 1.0, 3.0, and 10.0. Each panel corresponds to measured category report "errors" on a different system: the guide model itself, guide-surrogate model, and on humans. Note that "errors" here are defined with respect to the ground truth label the start (unperturbed) image.
Examination of the effect of image perturbations on the Guide Model that designed them (Fig 2b, left) confirms previous reports that robustification strongly reduces white-box perturbation sensitivity near novel start images, previously reported [11-13]. And the human DM results confirms previous reports [14-16] that human category percepts are only very weakly sensitive to low-norm image perturbations designed by vanilla ANNs (Fig 2b right, blue line).

---

In contrast, we report this novel finding: human percepts are massively disrupted by low-norm image perturbations discovered by robustified ANNs trained at an Eg pixel budget of 3.0 (red line). At an image perturbation budget of 30, ~90% of human category reports are no longer the category of the start image. Visual inspection of some example test images (Fig 2c) allows introspection about the perceptual differences between image perturbations generated by vanilla guide models vs. those generated by robustified guide models. Notably, we report these strong disruptions in human category percepts when perturbing at budgets well above the budget used for model robustification, yet still well-below the typical pairwise natural image distance regime (see Supplementary Material, Fig 1). This result was not priori guaranteed.
We sought to determine whether human perceptual disruptions are quantitatively aligned with the predictions of robustified ANNs. Because we cannot perform white box attacks on the human visual system, the fairest assessment of this via comparison of effects on humans with effects on surrogate models. To facilitate this comparison, we re-plot the results of the robustified ANN model (level 3.0, red) on the same panel as the human data (Fig 1c1, black line). Those plots demonstrate: (i) the originally motivating qualitative "gap" between the very high sensitivity of vanilla (surrogate) ANNs and the almost complete lack of human perceptual sensitivity to the same perturbations, and (ii) that some robustified ANNs have dramatically closed that gap. Nevertheless, some human-model misalignment still exists, with humans typically requiring about twice the perturbation strength to produce quantitatively matched level of perceptual disruption as that particular ANN model family (red vs. black in Fig lel right).
We also noted that robustified models show stronger "white-box"" boo--ooo---ooe "t--a------- alignment. Namely, image perturbations driven by robustified models not only transfer effectively to humans but also exhibit stronger alignment to surrogates of their own model class, compared to vanilla models (compare Fig 2b left vs. middle graphs).
We further assessed the impact of robustification training level on our findings. We found that models, which were adversarially trained at reduced allowable budget of 1.0, were less human-aligned (cf. Fig 2b right, yellow vs. red lines). Increasing the allowable budget at training to 10.0 did not result in a significant qualitative difference in model-human alignment. However, did lead less effective human category disruptions per budget level (cf. Fig 2b right, brown vs. red lines).
In follow up experiments, found these human perceptual disruption effects to be largely unaffected by test image viewing times in the natural viewing time range of 100-800ms (see Supplementary Material). In summary, our results demonstrate that robustified models reveal at least a subset of behaviorally meaningful image perturbations in the low pixel budget regime.
3.2 Targeted Modulation results
Fig 3 shows the results of the Targeted Modulation experiments tests of each Guide Model's ability to shift human percepts toward prescribed target calegories from arbitrary start images (see Sec 2.1). We first randomly selected start images from all nine object categories and perturbed each source image toward all different-class targets. We used several Guide Models: the non-robust (vanilla) model, three variations of robustified models with different levels of robustification, and an image interpolation baseline approach using randomly selected images from the intended target categories. We experimented with range of fg pixel budgets in the low pixel budget regime, using the same source images across all conditions. Fig 3a provides examples of images generated by the [3.00 robustified model at 30.0 pixel budget in the low budget regime.
We found that robustified models trained with an &2 pixel budget of 3.0 and 10.0 give rise to perturbations that are reliably reported by humans as the target category (Fig 3b). These models discovered specific image modifications that resulted in a 60% probability of humans choosing the prescribed target category at an E2 pixel budget of 30 (chance: ~11%) In comparison, the interpolation approach remained at baseline level throughout the low pixel budget regime. Consistent with the findings in the Disruption Modulation experiment, we found that perturbations made by vanilla model had no targeted effect on human category percepts.
The specificity of our Targeted Modulation is further demonstrated in Fig 4a. This panel shows examples of successful TM toward 'dog' and 'cat' for robustified model (e., (3.0). and other baseline methods. The precise control of a model is shown in confusion matrices in Fig 4b. They further highlight the strong specificity of TM for the robust model at la pixel budget of 30 relative to image interpolation bascline approach.
5

---

(a)
'insect'
'cat'
"crab"
(b)
100
Catch Goligg
Example:
Vanilla
'dog' 'caab'
humans choose target (%6]
Robust E, [3.0]
Robust ε [1.0]
Normalized rate
Robust ε, [10.0]
"turtle"
'dog'
www.
Image interp.
-60%
'bird!
budget range used to train
Robust E, [3.0]]
Pixel
'frog'
primate'
'fish"
6=30 Perturbation pixel budget
Dasoling
10
30
Perturbation pixel budget (E) [6--norm]
Figure 3: Robustified models enable budget-efficient precise targeted behavioral modulation.
(a) Targeted Modulation (TM) example: A 'dog' start image is here independently modulated toward each of eight alternative target categories by Robust Ecr [3.0] at an & pixel budget of 30.0. (b) Efficacy of Guide Models in modulating human reports toward specific, prescribed target categories (TM). Images were drawn from all nine categories and each image was independently perturbed towards each of the eight alternative possible larget categories (as illastrated in (a)). Curves show the lapse-normalized probability of human choosing the prescribed larget calegory. Baseline marks refer to unperturbed imagex, indicating human rates reporting the larget calegory the source image (' ('Catch") j. Error bars, 95% CI by bootstrap over images, target categories, and subjects. Robust models trained with la pixel budget 3.0 and 10.0 gave rise to the most hudget-efftcient behavior modulation.
Dependence on start images & target category percepts. To test the generality of these behavior modulation findings across different distributions of the start images, we conducted similar experi- ments (all pixel budget 30) using source images from (i) Out Of Distribution (OOD) images of same categories (" ('frog" etc.), (ii) Arbitrary Natural Images (ANT), collected from internet and social media, and (iii) Uniform Noise Images (UNI). We found that the TM effect magnitude remained comparable even when start images were drawn from these different distributions. For In Di-Distribution (ID), OOD, ANI, and UNI we achieved normalized target category induction rates of of 60%, 51 %. 55%, and 70%, respectively (Fig 4c). Notably, TM often fails to produce convincing images for some target classes when using UNI source images (see Supplementary Material).
To further test the generality of our approach, we conducted experiments using new, arbitrary targer categories. We selected nine target classes from an external animal image dataset, using 60 randomly selected images to represent each class. We modified our TM method to optimize towards feature representation of the target class, obviating the need to train new classifier for the new class space (see Sec 4). found high TM specificity also under this setting (Fig 4d), with lower TM scores on average compared previous settings. This is despite the challenges posed here, which include, in addition to using new target classes, the presence of source- source-image image distribution shifts (i.c., OOD, ANI; For UNI see Supplementary Material).
centroid
Induction of composite target category percepts We next sought to test whether model-guided perturbations could move human percepts toward pairs of object categories. Because subjects were only allowed to report one of nine categories, a perfect success in this test would inducing humans to report each of the two prescribed target categories ~50 of the time. Fig 4e shows results for these "two-composite" experiments. These results indicate that, while not perfect, even under this unusual setting, robustified models often discover highly creative ways to combine multi-class features in a single image that are human- recognizacc.................. Visual inspection of the modulated images further highlights that the multi-class modulation cannot be trivially attributed to tiling the relevant class features across different spatial locations The budget constraint guides the model to find clever ways to combine rather than "just add". Notably, scaling to 3-composite directions under the restrictive budget of 30 makes TM highly challenging, and more sensitive to the specific start images (see Supplementary Material).
Optimal robustification level. We defined the Model TM Efficiency (MTME) to be the budget at which a modulatory effect size exceeding 10% probability of choosing the target is observed. To this end, we interpolated the curves from Fig uniformly to 100 points in & pixel budget range of [0, 30). and identified the first budget at which the threshold is exceeded. We rank the models by

---

Image interp.
Robust E [3.0]
100
Robust e, [3.0]
Nornalized rate
80
Vanilla Interp.
dog
G81-
Nommalzzed rate [%6
75
targel class
humans choose
60
chance
turrle
50
40
primato
fish
25
20
crab
insedt
dog
frog
lumo
primato
fin
insect
dog
cat
frou
tumto
beo
primate
fiah
csab
insect
dogg
con
Rog
Rumo
Dirg
primata
Roh
crab
drgg
cat
Rop
tumb
primato
Rot
crab
(c)
(d)
Robust E [3.0]
humanss choose class [%6]
(OOD source)
(ANi source)
Robust e, [3.0]
OOD target class
goot
Nommllzzed rate [%)
-55
Normalized rate
80
ID
OOD
ANI
koala
llon
50
60
octopus
40
ovl
-55
20
Resticted
Disrib.
Distr
Isame
Arbitrary Natural Images
Uniform Noise Images
OOD
reindeer
squlimal
zebra
Source image distribution
glar
Kobday
Kon
lobette
oréspus
Obai rerrdooe
equirme Rackin
grea
Kouls
Intenio
Claf actpoo
mondeo
equiro
200rg
OOD-crab bird
ANI + turtle
UNI + primate
OOD-turtle +
koala
ANI + zebra
Hybrid-class Targeted Modulation
Nomalized rate [%)]
+ frog/bird
ID- frog/insect
ID primate/crab
primate/fish
40
20
cat
fog
tomte
bad pomats Ros
crab
inseer
dogg cat
trap
forto Did primate fos creb insectt
nog cet
frog torio
bed primno rih
crop inssect
dog cat fag tonte cor poimate fesh crab inseor
Figure 4: Robustified models achieve precise Targeted Modulation (TM) from any image start point to any target direction. (a, b) Behavioral modulation results obtained when using a robust model to drive toward all possible cardinal class directions (i.e., I-hot). (c, d) Analysis of distribution dependence source images and target directions, generalizing our precise targeted modulation finding beyond Restricted ImageNet. (e) Targeted Modulation toward hybrid directions robust model (3xr(3.0) that perturbs natural source images at an & pixel budget of 30.0. All panels show directly comparable numbers. Error bars, 95% over source images human subjects.
classes. All plots and images due
ascending efficiency order (marking MTME score, lower better): Rank 5) Vanilla, which never reaches the 10% threshold within the considered range, Rank 4) Robusified [1.0] (196), Rank 3) Image interpolation (47), Rank 2) Robustified ,,, [10.0] (11), and the best, Rank 1) Robustified Exr [3.0] (9). This suggests that there exist optimal robustification level for strongest modulation effects. While this precise optimization is beyond the scope of this work, we highlight that the Robust [3.0) model is currently the most TM-efficient amongst all tested.
Other pixel budget constraints. In addition to the & perturbation constraint, we also explored the of lo constraint for TM. Visual inspection of the resulting images revealed that both constraints induce comparable modulation effects, with the & constraint giving rise to images that appeared slightly more visually "clean" and aesthetically pleasing. We also found that TM by loc foc-treieed models was significantly weaker compared to the fz-trained models.
4 Methods
4.1 Distances in image space
We exclusively focused on x 224 RGB pixel space (D 150, 528), where each value ranges between 0 and 1. Thus, the maximal distance between any two images is [| z 388, For
7

---

reference, we found the typical distance between ImageNet images to be = 130. We also estimated the regime in which human object perception is empirically robust random perturbations to be || I 30 (see Supplementary Materials; also see [35]).
4.2 Stimuli generation by model-guided perturbations
Disruption Modulation (DM). We used the Projected Gradient Descent (PGD) attack algo- rithm [31, 36, 37], using an adapted version of the robustness library [32]. Given an initial input image and its ground-truth label in 1-hot representation x, y, and classification model to logits for C' classes, f Rxx x3 + RRC we optimize for the perturbation, 6, such to be disruptive of model classification to the ground-truth class label under pixel budget restriction:
argmax LCe (ʃ (x + +6) .))
where is the pixel budget, and LCz is the cross-entropy loss. Unless otherwise mentioned, we focus on f2-adversariilllytraines models and attacks, i.e., p = 2. Optimization is performed in steps of Stochastic Gradient Descent (SGD), each followed by projection step to pixel budget e:
5 1 Proj, (6x + VVV (S (x oz)
(2)
where & denotes the step, and is the step-size. All original input images are resized to 256 x 256 (bilinear, anti-aliased) and center-cropped to 224 x 224 [22]. We focused on the following range of lg-pixel budgets in the low-budget regime: [0.1, 0.5, 1.0, 2.0, 3.0, 5.0, 7.5, 10, 15, 20, 25, 30, 40, 50]. We set the number of PGD steps and the step-size, (ksteps. 17), such to match the pixel budget [32], ranging from (200, 0.02) for 0.1 to (2000, 2) for 50.
Targeted Modulation (TM). The algorithm identical to DM, aside from y changing to be a given target class label, and the optimization solving for minimization in lieu of maximization (flipping of gradient sign). Notably, unlike DM, this modulation can be performed toward multiple target class labels from single source image.
Hybrid-Clan Targeted Modulation. To modulate toward non-cardinal directions we defined random composite directions involving multiple classes. These directions were constructed as probability vectors with all all-zero except for randomly selected classes, which were assigned equal probabilities. We generated four 2-composite directions: 'frog-bing bird'. Our objective function was based on cross-entropy similar to Eq 1, with additional logit maximization on the non-zero target classes, which drives higher usage of the allowable budget.
crimate-crab"" 'primate-fish
Features Targeted Modulation. To modulate images toward target feature representation compiled a subset of 60 images per class and computed their mean feature representation using the GM. This defines the target classes via class-centroids. The optimization criterion is then the MSE between the predicted feature representation the target class-centroid.
Image interpolation modulation. To generate baseline image perturbations using an image interpola- tion approach, we randomly drew images from a target class, used them for linear interpolation with the source image, and projected back the pixel budget envelope.
Dataset. We sought to focus on a dataset with the following key properties: (i) Has a tractable class-space for exploration; (ii) Will mitigate confounds originating from typical-human unfamiliarity with the class labels (as is commonly the case in ImageNet); (iii) Is widely used in the adversarial robustness context. Based on these considerations, we found partial and mapped version of ImageNet to a basic set of nine classes, termed "Restricted ImageNet", to be the most suitable choice (see Supplementary Material for class mapping) [32].
Training robustified models. To adversarially -tain -train publicly released code [32]. Specifically, we trained ResNet50 models at &g pixel budgets of 1.0. 3.0, and 10.0, using PGD (steps, step-size) of (7, 0.3), (7, 0.5), (10, 1.5), respectively.
models on ImageNet [38]. we followed
Runtime. Our stimuli generation completes within Smin for a single batch of size 100 on a single A100 GPU. Adversarial--raining of models completes within ~8 days on 4 A100 GPUs
8

---

4.3 Human behavioral measurements
We measured human behavior in a nine-way image categorization task. Our primary experimental objective was, for any given test image, estimate the probability that humans would on average select each of the nine possible choices.
Task paradigm. Human subjects performed "sessions" comprising multiple trials. Each trial had three stages: (i) First, the subject pressed a button at the center of their screen, encouraging them to center-fixate. (ii) A test image (covering approximately 6 degrees of visual angle) appeared at the center of their screen for 200ms. This was followed by 200ms of a blank gray screen. (iii) An array of nine labeled buttons (one for each of the nine reportable categories) appeared. These buttons were arranged radially, outside of the location of where the test image appeared. The locations of the buttons were randomly permuted on each trial, and no time limit was imposed. The subject was asked to select the button which most accurately described the image.
In total, there were n 130 such trials in each session. Among these, there were four types of trials: (i) Warmup trials (n 10), which were always presented at the beginning of the session. Test images were randomly sampled "clean" (unperturbed) images drawn from the Restricted ImageNet validation set [32]. (ii) Calibration trials (n 10), included to measure the lapse rate of each human subject (see Lapse-rate correction). Stimulus images consisted of either a blue triangle or blue circle, and two randomly selected buttons on the choice screen were replaced with buttons for 'circle' and 'trianglc'. (iii) Reference trials (n 10), consisting of unperturbed stimulus images randomly selected from Restricted ImageNet validation set. (iv) Main trials (n = 100), consisting of potentially perturbed versions of source images. Trials types (ii)-ivv were run randomly interleaved, following the warmup trials. In general, subject feedback was only delivered for unperturbed images. Upon an incorrect selection by the subject, black "x" was displayed for 1000ms; No feedback delivered upon correct selection. We implemented the task paradigm using JSPsych, and presented the stimuli using the
plugin.
Human data collection. We used Amazon Mechanical Turk to recruit and collect behavioral mea- surements in human subjects, following protocol approved by the [anonymized IRB information] The protocol has no greater than minimal risk involved for the subjects. We did not attempt to collect any personally identifying information; subjects were kept anonymous. We screened subjects by their performance on a similar "demo" object categorization task (see Supplementary Material). In total, we recruited population of 119 subjects (n = 43 female; = 76 male). Subjects were compensated for all time spent in this study, including in the recruitment phase. Subjects were free to opt out of the study at any time. We ensured that each image we included in our analysis had measurements from at least n = 2 different subjects.
Lapse-rate correction. We assumed that measurements of human reports was possibly corrupted by a base "lapse rate", where a subject makes a random report independent of the presented test image's content. To account for that, we measured each subject's 's lapse rate, then adjusted their measurements using a formula that estimates their behavior under lapse rate of zero (see Supplementary Material).
5 Conclusion
In this work, we systematically examine the prevailing assumption that human categorization is highly robust to low-norm image perturbations. Our findings challenge that assumption by establishing that the low-norm image perturbations suggested by robustified ANNs (but not vanilla ANNs) can strongly and precisely change human object category percepts. These results suggest two novel conclusions: (i) for arbitrary starting points in image space, there exists nearby "wormholes", local (i.e., low-norm) perturbations in image space that support "travel" from the current category perceptual state induced in the subject into a semantically very "distant" perceptual state, and (ii) contemporary scientific models of ventral visual processing are accurate enough to point us to those wormholes.
5.1 Limitations
Our findings raise many new questions: given start image, does there always exist a nearby human perceptual wormhole to at least one another category? Are there nearby wormholes to any possible category? Which wormholes are common to all humans and which are more idiosyncratic? And do robustified ANNs always find the wormholes that do exist? And how often they predict wormholes that do not actually exist? We do not yet know the answers to these questions, but the results presented

---

here offer some insight. For example, Fig 2b right panel, showing a ~90% disruption effect, is consistent with the hypothesis that wormholes are abundant in image space. Figs 3 and 4 (~-60% effect) suggests that multiple category portals are nearby, but that portals to all categories are not always nearby. Further work could produce tighter lower-bounds on such probabilities.
We emphasize that these are lower bound estimates because we are probing human vision with models that are imperfect approximations of the underlying neurobiology. Indeed, do not claim that robustified ANN models discover all low-norm wormholes or that they never point to wormholes that do not actually exist. Notably, because we still see a residual gap in behavioral aligmment (Figs 1c and 2b), other models, which are even more behaviorally aligned, must still exist. Specifically, our analyses focused on the ResNet50 model architecture, and mainly on lz normo budget robustified models and image perturbation type (see Sec 3.2 for other perturbation constraints). While generalizing to other architectures image generation algorithms is not foundational the our claims presented in this paper, those are interesting directions to explore in future work.
Furthermore, while this paper supports that adversarial training (AT) gives rise an improved estimate of the adults state of the ventral stream and its supported behavior, we do not claim that AT is the mechanism by which robustness in humans emerges. Future work may consider alternative, more biologically plausible mechanisms (e.g., Topographic ANNs [39]) that may give rise to a comparably aligned models in terms of predictivity.
5.2 Societal impact
Image generation methods such as those used here have potential societal benefits, but also potential risks [40]. The scientific knowledge brain function contributed here may present additional future risks, in that it may widen the ability of malicious actors to subtly and con-consensually disrupt the perception of humans. Building defenses against those manipulations, and orienting the use of this knowledge toward societal benefits (e.g., consensually enriching perception, improving mental health, accelerating human visual learning) are crucial considerations in future work.
Acknowledgments
This work was partially funded by the Office of Naval Research (N00014-20-1-2588) JJD); (MURI, N00014-21-1-225 JJD), the National Science Foundation (2124136, JJD) and the Simons Founda- tion (542965, JJD).
10

---

References
[1] P. Bashivan, K. Kar, and J. J. DiCarlo, "Neural population control via deep image synthesis," Science, vol. 364, 2019.
[2] M. Schrimpf. J. Kubilius, M. J. Lee. N. A. Ratan Murty, R. Ajemian, and J. J. DiCarlo, "Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence." 11 2020.
[3] S. Grossman, G. Gaziv, E. M. Yeagle, M. Harel, P. Mégevand, D. M. Groppe, S. Khuvis, J. L.. Herrero, M. Irani, A. D. Mehta, and R. Malach, "Convergent evolution of face spaces across human face-selective neuronal groups and deep convolutional networks," Nature Communications, vol. 10, pp. 1-13, 12 2019.
[4] C. F. Cadieu, H. Hong, D. L. K. Yamins, N. Pinto, D. Ardila, E. A. Solomon, N. J. Majaj, and J. J. DiCarlo, "Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition," PLoS Computational Biology, vol. 10, p. e e1003963, 12 2014.
[5] D. L. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert, and J. J. DiCarlo, "Performance-optine hicrarchical models predict neural responses in higher visual cortex," Proceedings of the National Academy of Sciences of the United States of America, vol. 111, pp. 8619-8624, 6 2014.
[6] J. Kubilius, M. Schrimpf, K. Kar, R. Rajalingham. H. Hong. N. J. Majaj, E. B. Issa, P. Bashivan. J. Prescott- Roy. K. Schmidt,, A. Nayebi. D. Bear. D. L. K. Yamins, and J. Dicarlo, "Brain-Like Object Recognition with PerfomPerfoning Shallow Recurrent ANNs," tech. rep., r0p.. 2019.
[7] J. Dapello, T. Marques, M. Schrimpf, F. Geiger, D. Cox, and J. DiCarlo, "Simulating Primary Visual Cortex at the Front of CNNs Improves Robustness to Image Perturbations," Advances in Neural Information Processing Systems, 2020.
[8] K. Storrs, K. C. A. Storrs @ mre- Cbu, J. Mehrer, J. C. A. Mehrer@mre- Chure-Cu,,AA A. Walther, N. Kriegeskorte, and N. C. A. Kriegeskorte mre-Cbu, "Architecture matters: How well neural networks explain IT representation does not depend on depth and performance alone," "
[9] T. Golan, P. C. Raju, and N. Kriegeskorte, "Controversial stimuli: Pitting neural networks against each other as models of human cognition." " Proceedings of the National Academy of Sciences of the United States of America, vol. 117, pp. 29330-293333 29337, 11 2020.
[10] K. M. Jozwik, N. Kriegeskorte, R. M. Cichy, and M. Mur, "Deep convolutional neural networks, features, and categories perform similarly at explaining primate high-level visual representations" tech. rep.
[11] J. Goodfellow, J. Shlens, and C. Szegedy, "EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES," International Conference on Learning Representations (ICIR), 2015.
[12] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, "Intriguing properties of neural networks," in 2nd International Conference Learning Representations, ICLR 2014- Conference Track Proceedings, 2014.
[13] F. Tramer, A. Kurakin, G. Brain, N. Papernot, I. Goodfellow, D. Boneh, and P. Medaniel, "ENSEMBLE ADVERSARIAL TRAINING: ATTACKS AND DEFENSES," International Conference on Learning Representations (ICLR), 2018
[14] G. F. Elsayed, G. Brain, S. Shankar, B. Cheung, N. Papernot, and A. Kurakin Google Brain Ian Goodfellow Google Brain Jascha Sohl-Dickstein Google Brain, "Adversarial Examples that Fool both Computer Vision and Time-Limited Humans." "
[15] G. F. Elsayed, S. Shankar, B. Cheung, N. Papernot, A. Kurakin, I. Goodfellow, and J. Sohl-Dickstein, " "Adversarial examples influence human visual perception," Journal of Vision, vol. 19, pp. 190c-190c, 9 2019.
[16] Z. Zhou and C. Firestone, "Humans decipher adversarial images," Nature Communications 2019 10:1, vol. 10, pp. 1-9, 3 2019.
[17] M. Dujmović, G. Malhotra, and J. S. Bowers, "What do adversarial images tell us about human vision?." eLife, vol. 9, pp. 1-29, 9 2020.
[18] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Mądry, A. Kurakin, G. Brain, O. Evaluating, and A. Robustness, "On Evaluating Adversarial Robustness," 2019.
[19] S. Ullman, L. Assif, E. Fetaya, and D. Harari, "Atoms of recognition in human and computer vision." Proceedings of the National Academy of Sciences of the United States of America, vol. 113, pp. 2744 2749, 2016.
11

---

[20] F. Tramër. J. Behrmann, N. Carlini, N. Papernot, and J.-H. Jacobsen, "Fundamental Tradeoff's between Invariance and Sensitivity Adversarial Perturbations," " 2020.
[21] H. Xu, Y. Ma, H. C. Liu, D. Deb, H. Liu, Tang, and A. K. Jain, "Adversarial Attacks and Defenses in Images. Graphs and Text: A Review," International Journal of Automation and Computing, vol. 17, pp. 151-178, 2020.
[22] A. Mądry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant adversarial attacks," 2018.
[23] F. Croce and M. Hein, "Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free free Attacks." 2020.
[24] L.. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, B. Tran, and A. Madry, "Adversarial Robustness as Prior for Learned Representations." 6 2019.
[25] S. Santurkar, D. Tsipras, B. Tran, A. Dyas, L.. Engstrom, and A. Madry, "Image Synthesis with Single (Robust) Classifier," Advances in Neural Information Processing Systems, vol. 32, 2019.
[26] C. Ledig, I.. Theis, F. Huszár, Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi Twitter, "Photo-Realiste Single Image Super-Resolution Using a Generative Adver- sarial Network."
[27] Feather, G. Leclere, A. Mądry, and H. McDermott, "Model metamers reveal divergent invariances between biological and artificial neural networks," Nature Neuroscience 2023 26:/1, vol. 26, pp. 2017- 2034, 10 2023.
[28] A. Harrington and A. Deza, "Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks," ICLR, 2022.
[29] J. Dapello, K. Kar, M. Schrimpf, R. Geary, M. Ferguson, D. D. Cox, and J. DiCarlo, "Aligning Model and Macaque Inferior Temporal Cortex Representations Improves Model-to-Humal Behavioral Alignment and Adversarial Robustness," " bioRxiv. p. 2022.07.01 4984999 7 2022.
[30] M. Schrimpf, J. Kubilius, M. J. Lee, N. A. Ratan Murty, R. Ajemian, and J. J. DiCarlo, "Integrative Benchmarking Advance Neurally Mechanistic Models of Human Intelligence," Neuron, vol. 108, pp. 413 423, 11 2020.
[31] C. Guo, M. J. Lee. G. Leclere, J. Dapello, Y. Rao, A. Madry, and J. Dicarlo, "Adversarially trained neural representations are already as robust as biological neural representations," 6 2022.
[32] H. S. S.S. D. T. Logan Engstrom, Andrew Ilyas, "Robustness (Python Library)." 2019.
[33] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," tech. rep.
[34] M. Schrimpf, J. Kubilius, H. Hong, J. Majaj, R. Rajalingham, F. B. Issa, K. Kar, P. Bashivan, J. Prescott- Roy, F. Geiger, K. Schmidt, D. I.. Yamins, and J. J. DiCarlo, "Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?." 2018.
[35] H. Jang, D. McCormack, and F. Tong, "Noise-trained deep neural networks effectively predict human vision and its neural responses to challenging images." PLOS Biology, vol. 19, p. e3001418, 12 2021.
[36] N. Carlini and D. Wagner, "Towards Evaluating the Robustness of Neural Networks," in Proceedings IEEE Symposium on Security and Privacy, pp. 39-57, 2017.
[37] C. Qin, J. Martens, S. Gowal, D. Krishnan, G. Krishnamurthy, D. Deepmind, A. Fawzi, S. De, R. Stanforth, D. Pushmeet, and K. Deepmind, "Adversarial Robustness through Local I Linearization,"
[38] J. Deng, W. Dong, R. Socher, L.-J. Li, Kai Li, and Li Fei-Fei, "ImageNet: A large-scale hierarchical image database," in 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, IEEE, 2009.
[39] E. Margalit, H. Lee, D. Finzi, J. DiCarlo, K. Grill-Spector, and D. L. Yamins, "A Unifying Principle for the Functional Organization Visual Cortex," bioRxiv, p. 2023.05.18.55 18.541361, 2023.
[40] M. Masood, M. Nawaz, K. M. Malik, A. Javed, A. Irtaza, and H. Malik, "Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward," Applied Intelligence, vol. 53, pp. 3974-44440 2023.
12


Reviews:
[{"review_id": "0GISkJlukj", "reviewer": "Reviewer_Xb7f", "summary": "This paper studies the problem of adversarial images in artificial and biological visual systems. On standard (\"vanilla\") image models (e.g. ResNet-50), adversarial perturbations that successfully disrupt a vanilla model are quite small and do not significantly alter human perception. This paper generates adversarial perturbations on adversarially-trained (\"robustified\") image models, and tests whether those perturbations more effectively disrupt human perception, and find that they do (especially as the perturbation norm grows larger). The paper finds that it is both possible to perturb in an arbitrary direction, or in a targeted direction towards another class. It appears that you need to adversarially train the network used to generate the perturbation with a strong perturbation strength in order to see these effects (networks trained with no or weak adversarial perturbations did not yield perturbations that generalized to humans).", "questions": "- I would appreciate more details on the four adversarially trained networks. What is their performance on restricted imagenet? What do the training curves look like?\n- I would also like to get some better intuition for what a pixel norm of 30 means in this context. ", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Experiments are clear and straight-forward\n- Effect size is significant, although the perturbation size required is large.\n- Experiments contain nice controls and baselines (comparing perturbations from four different adversarially trained networks trained with different perturbation strengths; comparing targeted perturbations to interpolation; hybrid-targeted modulation, etc.)\n", "weaknesses": "- My main concern with the paper has to do with the interpretation. A perturbation size of 30 (for normalized images) seems quite large, and the example images do not (in my estimation) seem \"close\" to the original image. The original definition of adversarial images were images that were quite close in pixel space (imperceptibly so); by contrast the perturbations needed to modulate human perception are quite large. I don't think that in and of itself is an issue, it's still interesting that the large perturbations from an adversarially-trained network are sufficient to modulate human perception (especially in a targeted way). What I disagree with is that the claim that these perturbations are \"tiny\" (e.g. first sentence of the abstract, Fig 1 caption, etc). Throughout the text, the paper suggests that these are really small perturbations, which doesn't feel correct to me at a norm of 30.\n- Instead, my interpretation of the paper would be something like: \"Perturbations from adversarially trained networks are semantically meaningful\". If you look at some of the example perturbations, they are intuitive (unlike adversarial perturbations on vanilla networks, which look like noise). For example, in Fig 1c2, to perturb the image of a frog, the perturbation simply replaces the frog's head with that of a primate, and replaces the frog's skin texture with fur. By keeping the same background color (black) and pose of the animal, the perturbation minimizes l2 norm in pixel space while still being semantically meaningful. In some sense, the perturbation has become more _efficient_. Rather than a random noise pattern fooling the network, the perturbation needs to be more careful in how it modifies the target image. It is no surprise then that these more effective perturbations also modify human perception. I don't think this interpretation is any less interesting, but to me it feels like a more accurate description of the results in the paper.\n- I am curious if the authors looked at similarities or differences in the representations across the four trained networks. If one expects the robustified networks to be more similar to biological representations, this could make predictions for experimental visual neuroscientists to test.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Interpretation of the low budget regime**\n\nWe thank the reviewer for this feedback. Please refer to our global response on this point.\n\n**I am curious if the authors looked at similarities or differences in the representations across the four trained networks. If one expects the robustified networks to be more similar to biological representations, this could make predictions for experimental visual neuroscientists to test.**\n\nWe thank the reviewer for this suggestion. Some connections between adversarial robustness and increased representation alignment between models and primate neural activity have been established in previous works ([Dapello et al. NeurIPS’20](https://proceedings.neurips.cc/paper/2020/hash/98b17f068d5d9b7668e19fb8ae470841-Abstract.html), [Guo et al. ICLR’22](https://proceedings.mlr.press/v162/guo22d/guo22d.pdf), [Dapello et al. ICLR’23](https://openreview.net/pdf?id=SMYdcXjJh1q)), as we discuss in lines 36-46. Further evaluating the correspondences between the model(s) we study in this work and biological representations is an interesting line of future work. \n\n**I would appreciate more details on the four adversarially trained networks. What is their performance on restricted imagenet? What do the training curves look like?**\n\nPlease refer to our global response on the topic.\n\n**I would also like to get some better intuition for what a pixel norm of 30 means in this context.**\n\nPlease refer to our global response on the pixel norm value.  Notably most of the studied low pixel budget range (i.e., < 30) is beyond the robustification budget range (1.0, 3.0, or 10.0). In this context, please also refer to our global response on distinguishing these two budgets (adversarial training pixel budget vs. the budget used in our behavior modulation experiments on the pretrained models).\n\nTo better illustrate the meaning of a 30 pixel budget, we show examples of pixel-norm 30 perturbations (alongside others) Rebuttal Panel B for a variety of guide approaches.\n"}, {"review_id": "6IxopBzsmZ", "reviewer": "Reviewer_keVx", "summary": "The paper under review brings to light a crucial and fascinating issue in deep learning: adversarial attacks. The authors contend that a neural network trained adversarially, which is logically more robust against attacks, produces perturbed images that humans perceive as differing from the original when it is finally attacked. They present evidence that humans perceive images generated for both non-targeted and targeted attacks similar to how the neural network does. The concept of comparing human perception with the performance of artificial neural networks holds merit. However, it's necessary to critique the exaggeration of novelty in the authors' findings. They find that an adversarially trained network, which necessitates higher pixel perturbations for successful attacks due to its robustness, aligns well with human perception. Further, the authors' claim of their 'novel finding' that \"human percepts are massively disrupted by low-norm image perturbations discovered by robustified ANNs”. The purported surprise and novelty is  questionable as it simply reiterates the foundational goal of adversarial training and warrants scrutiny. Firstly, the paper does not offer a clear foundation as to who assumes that humans are necessarily resilient all perturbation with low norm, casting doubt on the originality of the claim. Secondly, as the authors themselves acknowledge (figure 1)  that object classes are intricately entangled in pixel space, hinting that in high dimensions small pixel-space deviations can be found to cause perceptual shifts to other classes. Therefore, it's counter-intuitive to assert that such a phenomenon is a surprising novelty. Moreover, the paper employs a pixel budget of 30 to define 'small perturbations', but it falls short of explaining why this particular value is low (average and maximum distances in the ImageNet dataset's high-dimensional space is 130 and 338 pixels if these distances are to be compared to the pixel budget for adversarial generation). To conclude, despite reservations regarding the claimed novelty, the paper does contributes valuable insights. Its methodology is solid, and its attention to human perceptual measurements is commendable. The authors carry out a diligent study quantifying human perception with respect to adversarial attacks, the results of which could spark important discussions in the field of adversarial robustness. However, the paper lacks claimed novelty. ", "questions": "Explain better the value of a pixel budget perturbation (i.e. 30). For example you can compare to contrast of the image to provide a more intuitive meaning of the level of perturbation needed to change classes for different networks. ", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "Quantification of human perception relating it to adversarially trained neural networks. ", "weaknesses": "-Lacks novelty and main results are expected based on the fact that the whole purposed that neural networks are trained with adversarial attacks is to make robust and more aligned with human perception \n- Not clear why in Figure 1c1 there is not shift of the red curve to the left compared to the blue one which is what one would expect from robustified neural net. The authors show this expected shift in Figure 2b where the blue is clearly shift to the left. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reviewing our work, for their positive comments about our behavioral experimental methods, and for their critical analysis of our empirical findings. We agree that the novelty and significance of our findings are dependent on 1) the prior beliefs in the field on perceptual sensitivity in humans, 2) the extent to which our results are straightforward consequences or restatements of adversarial training, and 3) whether any natural image may, with high probability, be a distance of < 30 to other images from other categories in pixel space (which would make it wholly unsurprising that perceptual shifts via perturbations of norm < 30 are possible ).\n\nWe provide our clarifications and justification on each of these points below.\n\n**Who assumes that humans are necessarily resilient to all perturbations of low norm?**\n\nWe contend that human vision is widely believed to be robust to “ultra low norm” perturbations (l2 norm < 3); e.g. [Wichman and Geirhos, 2023](https://arxiv.org/pdf/2305.17023.pdf) and [Dujmovic et al. 2020](https://elifesciences.org/articles/55978#s3). \n\nHowever, we should have been careful to not make the logical leap that this implies that the field also widely believes humans are invariant to merely “low norm” perturbations (which we define as norms < 30; see our global response for justification), where we found evidence of human perceptual sensitivity. We implicitly make this leap throughout our paper (e.g. language in line 30; line 323), and intend to be more precise in a future revision. Indeed, it would be more accurate for us to describe our findings not as challenging a “prevailing assumption”, but rather as resolving wide uncertainty and an absence of evidence around the question of whether and to what extent humans are sensitive in this < 30 “low norm” regime. We are unaware of any other prior work systematically establishing the perturbative sensitivity of human perception in this norm regime (and would welcome any key references in this regard). \n\nWe thank the reviewer for challenging us on this point, which will help us more accurately contextualize our findings relative to existing beliefs.\n\n**Is alignment with human perceptual sensitivity to “small” pixel perturbations a restatement of adversarial training?**\n\nThe adversarial training (AT) objective is to make a system’s behavior robust to pixel perturbations less than or equal to a certain norm (which is determined by a hyperparameter). This objective could be aligned with the objective of driving similarity to human perceptual sensitivity, to the extent that humans are **also** robust to perturbations below that hyperparameter-specified norm. \n\nHowever, our results focus on the fact that we found correspondences in sensitivity between AT-trained guide models and human perception at perturbation budgets **outside** of the budget range used for AT. This result could not be a straightforward consequence of the AT objective, which provides no explicit constraints on perturbations greater than 3 (for the primary, epsilon=3 robust model we considered).  Indeed, the degree of alignment between humans and models has not been previously shown or systematically tested and analyzed prior to this study.\n\n**To what extent are images from different object classes “close” to each other in pixel space?**\n\nAs shown in Rebuttal Panel A, and as the reviewer mentioned in their answer, natural images are estimated to be typically farther than what we termed “low norm” (median pairwise l2 distance ~130; 99.9% quantile range [69, 268], min ~45; max ~306; from Imagenet Restricted images). \n\nAs the reviewer correctly points out, if we had found that humans were perceptually sensitive from perturbations with norms of “typical” size, this would have been a trivial and unsurprising result (in that humans are obviously perceptually sensitive to different natural images). \n\nHowever, we found humans were perceptually sensitive to perturbations with norms as low as ~10. This is roughly one order of magnitude lower than the trivial result. \n\n**Our response to: “Not clear why in Figure 1c1 there is not shift of the red curve to the left compared to the blue one which is what one would expect from robustified neural net. The authors show this expected shift in Figure 2b where the blue is clearly shift to the left.”**\n\nThank you for noticing this. Some context: you are absolutely correct that a robustified version of a model should be more invariant to perturbations, namely the red curve should be to the right of the blue curve (i.e. requires a higher pixel budget to produce the same level of disruption). We indeed replicate that previously reported result in Fig 2a (left panel). However,  please note that this rightward shift is only guaranteed in the situation in which the model is used to design attacks against itself (aka “white box”).  The colored curves in Fig 1c1 are the results of “gray box” attacks. Specifically, these are tests of attacks designed by one member of the model family and tested on another member of that same model family. We plot these against the human data in Figure 1 as they are the proper way to make a quantitative comparison to the behavioral of human visual neural networks (as we do not have white box access to those networks), and we explain this in Overview. For gray box comparisons, the rightward shift of the red curve relative to the blue curve is, unlike the white-box situation, not guaranteed.  Indeed, while not the primary focus of our work, it is interesting that we found that robust models and vanilla models had similar gray box sensitivities (e.g. as shown in Figure 1 and Fig 2b; left panel).  We will add a clarification to the caption of Fig 1 to indicate that these are gray-box results"}, {"review_id": "NsnTPZNQ44", "reviewer": "Reviewer_LMUo", "summary": "The paper systematically challenges the common assumption that human categorization of images remains highly robust to small-scale image perturbations (low pixel budget). The authors find that small-scale image perturbations, guided by adversarially trained artificial neural networks (robustified ANNs), can significantly and precisely alter human perception of object categories. Not only does this finding challenge the above assumption, but it also reveals the existence of \"wormholes\" in the image space that can lead to a significant change in human object category perception. Moreover, the study demonstrates that contemporary models of visual processing are precise enough to locate these \"wormholes\".", "questions": "1) Have the authors tried more models besides ResNet?\n2) Could the connection between deep models and human perception be explored from more angles, such as the similarity of intermediate features in the model?\n3) Wormholes, a core part of the paper, are not prominently discussed. The authors should seek and describe the regularities of wormholes in robustified ANNs, instead of simply saying that they can generate images that interfere with human judgment from any source image distribution.", "limitations": "No", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The authors discuss from an intriguing perspective how deep neural network models (robustified ANNs) can induce anticipated human behavior (i.e., misjudgments) with generated images (slight changes on the original image).\n2) The authors reveal that only models generated through adversarial training have this ability to induce.\n3) The authors seek to explore human perceptual wormholes from modern visual models (robustified ANNs).", "weaknesses": "1) The paper revolves around the premise that images generated by robustified ANNs (under a low pixel budget) can interfere with human classification judgments. This conclusion doesn't seem surprising. Humans often misjudge many images; it's just that robustified ANNs can generate highly deceptive images.\n2) The authors believe that robustified ANNs and human perception have similar responses to image perturbations. The only evaluation criterion seems to be the final classification error rate. Having similar error rates doesn't equate to having similar responses.\n3) The idea of using robustified ANNs to search for human perceptual wormholes seems impractical, since even from the perspective of robustified ANNs, the authors do not provide any patterns of wormholes in robustified ANNs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The paper revolves around the premise that images generated by robustified ANNs (under a low pixel budget) can interfere with human classification judgments. This conclusion doesn't seem surprising. Humans often misjudge many images; it's just that robustified ANNs can generate highly deceptive images**\n\nWe thank the reviewer for this comment.\nWe agree that attacks on robustified models were previously shown to engage human perception (as addressed in Introduction, ln38). However, to the best of our knowledge, there is yet no evidence for systematically quantifying the model-human gap in adversarial sensitivity. Further, it was unknown whether in a low budget-regime – in which, as we show, human perception is stable under baseline attacks (image interpolation or random attacks) – it is possible to disrupt or induce specific category percepts, and to what degree. Furthermore, and even more surprising, is that the strong modulations by robustified guide models are prominently evident when perturbing at budgets well beyond the budget range used for model robustification. Namely beyond the “robustification perimeter”. \n\nFor perspective, consider some of the following live hypotheses prior to this study: Image perturbations discovered by robustified ANNs (1) have no effect on human category reports, regardless of the perturbation budget, (2) have some effect of on human category reports, but only in the typical-norm budget regime (i.e. typical of pairwise distances, ~130), (3) have perceptually interpretable effects in the low-norm regime, but have no effect on human categorization behavior.  \n\nTo better clarify our results and their novelty we now revise in Results:\n\n\n*“Notably, we report these strong disruptions in human category percepts when perturbing at budgets well above the budget used for model robustification, yet still well-below the typical pairwise natural image distance regime. This result was not a priori guaranteed.”*\n\n**Have the authors tried more models besides ResNet?**\n\nWhile generalizing to other architectures is not foundational to the our claims presented in this paper, we agree that exploring other architectures is an interesting question. \n\nBecause model adversarial training takes ~8 days to complete (on 4 A100 GPUs), in conjunction with the requirement to train two such models per architecture and robustification level for white-box/gray-box analysis, for this study, we focused our analysis on ResNet50 only.\n\nPrevious studies on model adversarial training suggest that the results obtainable by other architectures (e.g., ResNet101) may be comparable (Singh, Croce & Hain 2023). We have further mentioned in Conclusion ln 339-343 that our findings are in a lower bound sense, and that there may be other models that are more behaviorally aligned:\n\n*“We emphasize that these are lower bound estimates because we are probing human vision with models that are imperfect approximations of the underlying neurobiology. Indeed, we do not claim that robustified ANN models discover all low-norm wormholes or that they never point to wormholes that do not actually exist. Notably, because we still see a residual gap in behavioral alignment (Figs 1c and 2b), other models, which are even more behaviorally aligned, must still exist.”*\n\n\n**Model-human alignment evaluation criterion focusing on final classification**\n\nWe agree with the reviewer that finding a good model of biological vision has multiple facets to it, including internal representation. We note that while our disruption modulation (DM) experiments were indeed simply measures of error rate (accuracy), our Targeted Modulation (TM) tests were tests of specific types of errors, which is a more stringent test in the spirit of the reviewer’s question. Internal (feature) similarities of these models and primate internal neural representations have been reported prior to this study, and were key motivation points for us to focus the present paper on testing the alignment in the downstream behavioral output, which is conceived as supported by those internal responses in both the brain and the models. We refer to those prior findings about the similarities in representation between the biological visual system and the artificial counterpart in the Introduction in lines 39-46.\n\n**Using robustified ANNs to search for human perceptual wormholes**\n\nWe thank the reviewer for raising this concern, which understandably can be elusive. If correctly parsed, this comment has to do with better explaining what we mean by “wormholes”. The “wormholes” is a metaphor for our empirical finding that, from every tested starting “location” (i.e., starting image), there exists a “nearby” (i.e., low-norm) perturbation step that will induce drastic changes in human behavioral categorization report. The metaphor is intended to conceptually express the idea that when a human subject is in one perceptual “universe”, s/he can “walk” a very short distance in pixel space to then be in another perceptual “universe.” We acknowledge that this metaphor may not be perfect, and may be missed, but we are inclined to believe this helps convey the conceptual essence of our findings in a faithful way.  We will add a sentence in Introduction to clarify this.\n\nWe agree with the reviewer’s related comment that we have not yet fully characterized the statistical distribution of such human perceptual wormholes at any given pixel distance, but we believe the work we presented here – demonstrating that some computational models can reliably point to and reveal those human perceptual phenomena – is a necessary precursor for that follow on work.  We have added a sentence in the revised manuscript (under “Limitations”) to express this.\n\n"}, {"review_id": "6Y1lr9FiQE", "reviewer": "Reviewer_StMT", "summary": "I have read the rebuttal and will keep my (relatively high!) score as is.\n\nIt is a folk-theorem that human categorization behavior is robust to adversarial perturbations that are under an L2 norm of 30 or less. This paper shows that networks that have received adversarial training can be used to generate relatively low pixel-budget “adversarial” examples that are reliably mis-classified by humans under an L_2 norm of 30 or less. This closes one gap between human behavior and network behavior, in that the humans are disrupted to nearly the same extent that networks are.", "questions": "lines 23-25: “However, individual ANNs are also notoriously susceptible to adversarial attack: the addition of tiny (e.g., small l2-norm) pixel perturbations to the model’s input, optimized to disrupt the model’s categorization of the original image [11–13].”\nThis sentence is awkward: one expects a verb in the clause after the colon, but it is just a noun phrase describing what an attack is. So, it’s ok, it’s just hard to parse - almost a garden path. \n\nlines 36-39: for an example of a robust model that doesn’t require adversarial training, and causes computed adversarial examples to appear as the category they are trying to spoof, see https://arxiv.org/abs/2002.07405.\n\nline 126: Fig 2a -> Fig. 2\n\nline 160: Fig 2a -> Fig, 2b\n\nFigure 4: I’m not clear whether these are confusion matrices or error matrices? I.e., are the entries the percent of times the network returns that answer, or are they the times that it doesn’t. The caption seems to mean the latter (1-hot). A confusion matrix usually shows the number or percent of times that the answer goes in that box, so if the network is doing well, the diagonal has high values. \n\nline 184: do you mean precision in the technical sense (i.e., as in precision and recall)? If not, better to choose a different word.", "limitations": "The authors adequately discuss the possibly misuse of this technique.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ the paper shows a novel result: That when adversarially-trained networks, which are a better match to primate neural representations than “vanilla” networks, the adversarial examples that they generate can disrupt human categorization with a much lower budget than is seen with vanilla networks.\n\n+ the paper shows that these “robustified” networks can modulate relatively arbitrary images to relatively arbitrary targets with low budgets. \n\n+ The paper is well-written and the experiments are thorough.", "weaknesses": "- Just a bit of terminology complaint: Generally, an adversarial example is by definition one that fools networks but not humans. So once humans perceive these as the target category, I would no longer call them adversarial.\n\n- A bit of a philosophical complaint: The paper focuses on the fact that these low-budget examples fool humans and networks to conclude that now these networks are better models of the human visual system. I find that a bit distracting from the goal of discovering network properties that cause the network to be a better model of the human visual system. That is, humans don’t *need* adversarial training to not be fooled by adversarial examples. A more biologically plausible model would be one that shares this property with humans. I find adversarial training to be a kind of dumb response to the susceptibility of networks to adversarial examples - again, we should be looking for properties that make them not susceptible to these images in the first place - e.g., perhaps the right kind of recurrence, feedback, a capsule architecture (see below), or some other property will achieve this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for supporting our work.\n\n**Terminology complaint**\n\nWe agree that “adversarial image” is not well defined in the field and that the work we presented exposes the need for a clear definition of this phrase.  Our working definition – which is inline with the original methodological approach – has been that an adversarial perturbation for a model is an image perturbation that intends to cause the model to make a mistake (relative to a ground truth label). Strictly speaking, this definition is *not dependent* on the effect of that perturbation on human perception.  However, because the difference between model perception and human perception is what made adversarial examples most interesting to many, the reviewer’s assumed definition (different than ours) is understandable. To avoid this confusion, we propose to revise the manuscript accordingly:\n\nln48: “adversarial sensitivity” -> “low-norm perturbation sensitivity”\n\nln75: “low-norm adversarial sensitivity” -> “low-norm perturbation sensitivity”\n\nln133: “adversarial sensitivity” -> “perturbation sensitivity”\n\nln240: “adversarial perturbations” -> “model-guided perturbations”\n\n**Philosophical complaint**\n\nWe agree that finding a model that is both predictive and biologically plausible in its developmental/learning trajectory is a major goal in the field. Indeed we are not claiming that adversarial training (AT) is the mechanism by which robustness in humans emerges. We rather consider AT as a method by which we arrive at an improved estimate of the adult state ventral stream and supported behavior, and a contribution of our paper is to demonstrate that.  Future work may consider alternative, more biologically plausible mechanisms that may give rise to a comparably aligned model in terms of predictivity (e.g., Topographic ANNs, Margalit et al. 2023).  We will add a line in the new “Limitations” section to make this clear.  \n\n**Phrasing in lines 23-25**\n\nWe see the point. Now changed to:\n\n“However, individual ANNs are also notoriously susceptible to \\emph{adversarial attack}: Adding of a tiny amplitude (e.g., ultra-low $\\ell_{2}$-norm) pixel perturbation to the model's input, which is optimized to disrupt the model's categorization of the original image.”\n\n**An example of a robust model that doesn’t require adversarial training and causes computed adversarial examples to appear as the category they are trying to spoof**\n\nWe thank the reviewer for this reference. As mentioned in our comment about biological plausibility, we do not claim nor highlight adversarial training or the specifics of our algorithm for image generation given a robustified model, as key to behavioral modulation. For this we dedicated a conclusion paragraph in lines 339-343 clarifying:\n\n“We emphasize that these are lower bound estimates because we are probing human vision with models that are imperfect approximations of the underlying neurobiology. Indeed, we do not claim that robustified ANN models discover all low-norm wormholes or that they never point to wormholes that do not actually exist. Notably, because we still see a residual gap in behavioral alignment (Figs 1c and 2b), other models, which are even more behaviorally aligned, must still exist.”\n\nline 126: Fig 2a -> Fig. 2\n\nline 160: Fig 2a -> Fig, 2b\n\n**Fig 2 referencing typos**\n\nThanks! Corrected.\n\n**Figure 4 matrices**\n\nIndeed confusion matrices, with high values on the diagonal for robustified models, but not for the control methods (Image interpolation, see Supplementary Material for the vanilla). The values shown are normalized rates for choosing the target class, same as in panels (a) and (c), and as denoted on the color bar.\n\n**line 184: do you mean precision in the technical sense (i.e., as in precision and recall)? If not, better to choose a different word.**\n\nWe thank the reviewer for this suggestion. We now change: “precision” -> “specificity” in ln184, ln202. \n"}, {"review_id": "RDNCVWUSfc", "reviewer": "Reviewer_ibQ6", "summary": "The paper presents a novel approach to find categorical perceptual changes in humans using artificial neural networks (ANNs). Notably, the paper presents compelling evidence that an adversarially trained resnet50  is better at generating these adversarial attacks on humans on a low budget pixel regime. To test this the authors tested on humans subjects how the categorical percept change in relation with the pixel  budget assigned to a PGD attack on the network. On low pixel budget, adversarially robust  resnet50 are able to generate stimuli that can generate more successful targeted attacks that affect humans.", "questions": "Why are humans as sensitive to image perturbation at an l2 budget of 3.0 and 10.0? What happens with a pixel budget of 5 for example? And more than 10? How can you explain that by changing more content in the human, the images get less semantically different for humans?\n\nHow do you explain that DM images have more effects on human errors than TM perturbations?\n\nIt was not completely clear what is the performance of both models in the 9 categories described.", "limitations": "No explicit limitation section is written.\nIt would be nice to mention that this work is restricted to only one norm of adversarial attacks and one type of architecture. Those results show an increased alignment between humans and models’ robustness to adversarial attacks but a lot remains to be analyzed to completely claim of closing the gap.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall the paper is well-written and  well-executed. \n\nThe question asked is interesting and very relevant to the community. Even though it’s not completely solved, the authors propose a first set of answers that could lead to interesting future work.", "weaknesses": "Minor points: \n\n I think having methods presented after the results was a bit unexpected and perhaps confusing. A reorganization may help understand a bit better the results, by putting the foundations first. \nBe careful with Figure 2, the caption doesn’t exactly correspond to the figure : 3 panels in the figure but only two in the caption. Also in panel c, second row and “frog column\", some writing appears which are probably not supposed to be part of the image.\n\nthe authors sometimes mentioned a pixel of 30 (or 30.0) and sometimes of 3.0. A revision to make sure the units are coherent through the paper would help.\n\nThe term “robustified models” can be a bit miss-leading and not commonly used. It would be better to refer to “adversely trained models”.\n\n\nMajor points: \n\nSome of the contributions seem a bit  over-claimed considering that only one type of attack is tested and also  only one type of model is analyzed.  \n\nAlthough the pixel budget is well defined, there seems strong semantic distance even in low-pixel budget regime. For instance, in figure 1, the primate example, it seems like in 10^1 budget regime there is crucial semantic  information lost and added that would explain the impact on performance for humans. There seems to be potential in this direction to understand image classification, but the authors may need to clarify exactly why taking this route.\n \nIt seems what the paper is proposing is a categorical morpher, why taking this route and what does it tell us about the human judgements? Also, why using an adversarial approach and not a stable diffusion approach?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Errors/typos & paper organization**\n\nWe thank the reviewer for spotting those important errors/typos, now all corrected. As far as organization is concerned, and in accord with the reviewer’s suggestion, we designed the “Overview of approach and experiments” section to provide a succinct and high-level description of our methodology, while leaving mostly the more technical details for Methods. If there are additional elements, which if moved from Methods to Overview may help ease the reading further, we are happy to do so.\n\n**Budgets 30.0 and 3.0**\n\nIndeed, the 30.0 and 3.0 budgets are two distinct budgets, corresponding to the perturbation budget in our experiments when attacking *pre-trained models*, and that allowed *during model (adversarial) training*, respectively.  We also add a visual to explain these regimes to the global response figure and to Figs 2,3. \n\nWe now further clarify this point by adding:\n\nIn Overview:\n\n*“Notably, we denote this \\textit{adversarial training budget} by $\\varepsilon$, to be distinguished from the perturbation budget, $\\epsilon$, which applies to any image-perturbation method.”*\n\nKindly refer also to our global comment about this distinction.\n\n**The term “robustified models”**\n\nAs correctly pointed out, our “robustified models” are colloquially known as “adversarially trained models” or “robust models”. However, adversarial training does not guarantee a robust model – not to adversarial attacks nor in broader senses of robustness. We thus believe that coining the term “robustified models” has a merit in referring to just the process performed on the model towards becoming more robust. We further use it interchangeably with “adversarially trained models”.\n\n**Limited attack/model types**\n\nWe agree that analyzing other attacks and/or model types can better support our contributions. We thus include in our Supplementary Material additional analyses titled “Different attack strategy: linf attacks and linf-robust models”. We describe and refer to this section from the main text in line 228 under “Other pixel budget constraints”.\n\nWe refactored our conclusion section to more explicitly call out the limitations (specifically, we focused on one architecture and mainly l2). \n\n**Inducing interpretable semantic shifts**\n\nAs human observers ourselves, we agree with the reviewer (another human observer) that we also perceive a *strong semantic shift* even in the low-pixel budget regime.   Systematically testing and quantifying such measures of human perception relative to model “perception” is a key contribution of our work. The ability to induce those shifts in human reports in such a pixel budget regime – which is otherwise a \"perceptually stable regime” as we demonstrated via random attacks experiments or a contrast blend approach – was not known to be possible before this study.\nNotably we have in fact conducted experiments also by a Stable Diffusion modulation approach. While those are outside the scope of this paper, they reflected that unlike robustified-models’ guided perturbations, compelling modulations by LDMs require *larger* budgets than our defined low-norm budget (i.e., < 30).\n\n**Why are humans as sensitive to image perturbation at an l2 budget of 3.0 and 10.0**\n\nFigs 2b or 3b show human reports for Disruption or Targeted Modulations respectively, for perturbation/attack budgets in range < 30, which are performed on an array of models, including vanilla and a few robustified models. Those robustified models were trained to various degrees of robustification levels, 1, 3, and 10. The curves on both Figs 2b or 3b for all robustified models 3, 10, and mostly 1 are monotonically increasing, indicating the humans are more strongly modulated by higher allowable budget perturbations. Kindly refer to our previous clarification (and the global) regarding distinguishing between the training and the experiment perturbation budget.\n\n**How do you explain that DM images have more effects on human errors than TM perturbations?**\n\nThere are 8/9 (n categories=9) ways to disrupt a category but only 1/9 to induce a specific one, making the former an easier objective compared with the latter under a given pixel budget constraint.\n\nIt was not completely clear what is the performance of both models in the 9 categories described.\n\nPlease refer to our global response on the topic.\n\n**It would be nice to mention that this work is restricted to only one norm of adversarial attacks and one type of architecture. Those results show an increased alignment between humans and models’ robustness to adversarial attacks but a lot remains to be analyzed to completely claim of closing the gap.**\n\nAs noted previously, we have shown that these results extend also to linf attacks and have analyzed linf models. Previous studies on model adversarial training suggest that the results obtainable by other architectures (e.g., ResNet101) may be comparable. We have further mentioned in Conclusion ln 339-343 that our findings are in a lower bound sense, and that there may be other models that are more behaviorally aligned:\n\n*“We emphasize that these are lower bound estimates because we are probing human vision with models that are imperfect approximations of the underlying neurobiology. Indeed, we do not claim that robustified ANN models discover all low-norm wormholes or that they never point to wormholes that do not actually exist. Notably, because we still see a residual gap in behavioral alignment (Figs 1c and 2b), other models, which are even more behaviorally aligned, must still exist.”*\n"}]

Author rebuttal:
We thank the reviewers for finding our work interesting and for your support. We appreciate your constructive feedback and we agree that those suggestions would clarify the contributions of this work.

**Interpreting the low-pixel budget regime (< 30)**

All reviewers asked for clarification about the “low-norm” pixel budget regime that we focused on in the paper.  We agree that the original version of the paper did not provide full clarity on this issue.  
Our definition of the “low-norm” perturbation pixel budget regime (l2 norm of < 30) was/is primarily based on empirical evidence that we provided in Figure 1 in our Supplementary Material. We show this empirical evidence in Rebuttal Panel A, and it illustrates that: 1) that human reports are virtually unaffected by random noise image perturbations or to image perturbations guided by a vanilla ANN, when those perturbations are restricted to pixel budgets less than 30, and 2)  that the typical (median) pairwise distance in natural images is ~130 with the *closest observed* pair of natural images (out of n=101,025 pairs) is ~45 apart. These empirical observations support the notion that perturbations of norm less than 30 can be reasonably referred to as “low-norm.”   

As shown in Rebuttal Panel A, we define our perturbation budget regimes as: 

- __“Ultra low-norm”__: < 3, a typical range studied in adversarial literature (e.g., for performing adversarial training) 
- __“Low-norm”__: < 30, a range in which human categorization reports are insensitive to attacks by Gaussian noise perturbations or vanilla ANN attacks.
- __“Typical-norm”__ :  69-268, the range that we estimated contains 99.9% of pairwise distances for two independently sampled natural images. 


In our revision, we will strictly adhere to these terms, and as suggested, we will refrain from calling the perturbations in our experiments “tiny” and simply call them “low-norm”.

To better illustrate the state of affairs prior to this study, we now also include in Rebuttal Panel B examples of pixel-norm 30 perturbations (alongside others) for three baseline image perturbation generation approaches.

As correctly pointed out by several reviewers, the perturbations discovered by robustified models – despite being low-norm – are indeed semantically meaningful, which is another way of saying that they are likely to produce specific changes in human perceptual report, and quantifying that was the focus of our study.  The primary novelty is the observation that, from apparently any starting image, robust guide models can be used to induce strong shifts in human perceptual reports in such a low-norm budget regime – otherwise a perceptually stable regime under naive baseline approaches.  This finding is novel in that it was not a priori guaranteed or otherwise known (see below).  

**Distinguishing between pixel budget for adversarial training and pixel budget for designing perturbations experiments to test on humans and (fixed) models**

To further clarify the distinction between these two pixel budget regimes and their relationship to our results we now add in the revised manuscript:

In Overview:

*“Notably, we denote this \textit{adversarial training budget} by $\varepsilon$, to be distinguished from the perturbation budget, $\epsilon$, which applies to any image-perturbation method.”*

Please note that the adversarial training procedure used to train the robust guide models does not a priori guarantee that the resultant representations of robust models will be aligned with human perceptual semantics – rather, that procedure attempts to establish *complete insensitivity* to all pixel perturbations below a certain (ultra low) norm (determined by the epsilon-training hyperparameter), without any explicit constraint attempting to explicitly match measurements of human perception.  Thus, the fact these guide models automatically discover human-aligned perturbations in the (untrained) low-norm regime was not guaranteed, and is thus, in our view, surprising and novel.  

To clarify this, we now add in Results:

*“Notably, we report these strong disruptions in human category percepts when perturbing at budgets well above the budget used for model robustification, yet still well-below the typical pairwise natural image distance regime (see Supplementary Material, Fig~1).  This result was not a priori guaranteed.”*

We also add a visual to explain these regimes to the global response figure and to Figs 2,3.


**Details on model performance**

We thank the reviewers for this feedback.
Given that these are ImageNet trained models they were evaluated on the 1000-way task on natural images and adversarial images (and for various evaluation budgets for the adversarial attacks). Their performance on natural images ranged from 45-76% Top-1 accuracy with the least robustified (vanilla) highest; on adversarial images ranged from 10-53% Top-1 accuracy (when evaluating at the same budget as the training one) with the least robustified (excluding the vanilla) highest. As shown in prior work, robustification tends to reduce natural image accuracy (Tsipras et al. 2019, Zhang et al. 2019). Further note that robustifying at a higher training budget level entails evaluation at the same (higher) budget level, resulting in a lower validation accuracy on adversarial images overall for the models trained for higher robustification levels.

For reference we include in our response figures summarizing their performance and learning curves.

**Fig2 typos**

We thank the reviewers (StMT, ibQ6) for spotting Fig 2 typos, which are now all fixed.


**Public availability of behavioral data**:
If this work is accepted, we will release an anonymized copy of the behavioral data used in this study, following the regulations for privacy, anonymity, and storage in our protocol, which was approved by our institutional board. We will update the Methods section paper with a reference to this protocol, and a link to these data. 


Meta-review: In this paper, the authors explore the problem of adversarial images in both artificial and human visual systems. Typically, adversarial perturbations applied to convolutional neural networks (e.g. ResNet-50) may be small and imperceptible to humans, but may profoundly influence the predictions of the CNN. In this work, the authors demonstrate that adversarial perturbations targeted to an adversarially trained network (i.e. 'robustified') may generate perturbations that are semantically meaningful to a human. This effect is quite noticeable especially as the norm of the perturbation is relaxed. Furthermore, the authors find that they perturbations may be targeted to selectively change the classification of the object to arbitrary objects of choice.

The reviewers commented positively on the quality of the results, the semantic saliency of the resulting images (i.e. strong effect), and the clear and straight-forward experiments. The reviewers also surfaced concerns about the interpretation of the results. In particular, the pixel budget of 30 permitted in the adversarial attack seemed quite large and might confuse the original definition of adversarial images (e.g. Szegedy et al, 2013; Goodfellow et al, 2014). 

That said, all reviewers were impressed by the visual saliency of the results but clearly there are some unresolved concerns about how to interpret these results. I found the suggestion by one reviewer (and concurred by a second reviewer) that one way to clarify the interpretation of the paper is to potentially retitle the contribution to be "Perturbations from adversarially trained networks are semantically meaningful". This would add far more clarity than introducing esoteric and unnecessary nomenclature such as "wormholes" and would bring clarity to the presentation. Regardless, these results are compelling and the larger community would benefit from their publication. For these reasons, this work will be conditionally accepted so long as the title and main description of the paper better align with the proposed phrasing above.
Decision: Accept (poster)