Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks

Abstract: Recent work suggests that feature constraints in the training datasets of deep neural networks (DNNs) drive robustness to adversarial noise (Ilyas et al., 2019). The representations learned by such adversarially robust networks have also been shown to be more human perceptually-aligned than non-robust networks via image manipulations (Santurkar et al., 2019, Engstrom et al., 2019). Despite appearing closer to human visual perception, it is unclear if the constraints in robust DNN representations match biological constraints found in human vision. Human vision seems to rely on texture-based/summary statistic representations in the periphery, which have been shown to explain phenomena such as crowding (Balas et al., 2009) and performance on visual search tasks (Rosenholtz et al., 2012). To understand how adversarially robust optimizations/representations compare to human vision, we performed a psychophysics experiment using a metamer task similar to Freeman \& Simoncelli, 2011, Wallis et al., 2016 and Deza et al., 2019 where we evaluated how well human observers could distinguish between images synthesized to match adversarially robust representations compared to non-robust representations and a texture synthesis model of peripheral vision (Texforms a la Long et al., 2018).  We found that the discriminability of robust representation and texture model images decreased to near chance performance as stimuli were presented farther in the periphery.  Moreover, performance on robust and texture-model images showed similar trends within participants, while performance on non-robust representations changed minimally across the visual field.  These results together suggest that (1) adversarially robust representations capture peripheral computation better than non-robust representations and (2) robust representations capture peripheral computation similar to current state-of-the-art texture peripheral vision models. More broadly, our findings support the idea that localized texture summary statistic representations may drive human invariance to adversarial perturbations and that the incorporation of such representations in DNNs could give rise to useful properties like adversarial robustness.

Reviews and author responses:

This paper examines whether an adversarially trained (“robust”) ResNet-50 classification model might better resemble the representation of human peripheral vision than a non-adversarially trained (“non-robust”) model. It does so by means of a human perceptual experiment: test images from Restricted ImageNet (a collapsed subset of ImageNet comprising 9 super-classes) are resynthesized via gradient descent from random noise initial conditions by minimizing the L2 difference in the penultimate feature activation of both a robust and non-robustmodel. 
In addition, the same stimuli are resynthesized according to previously validated models of summary statistic based peripheral computation (Texforms). A range of perceptual discrimination experiments (odd-one-out and two alternative forced-choice) are carried out with human subjects among different pairs of stimuli (original, robust, non-robust and texform) at varying degrees of peripheral eccentricity. The experiments show that the discriminability of robust and texform stimuli vs original stimuli degrade similarly with increasing eccentricity, while non-robust stimuli are relatively easy to pick out vs original stimuli at all levels of eccentricity. 
 # Strengths

1. I thought the topic of the paper was interesting and relevant for ICLR since it probes at a link between the representations learned by adversarially trained image classifiers and the robustness of human peripheral vision, and suggests some possible paths forward (e.g. spatially-adaptive computation of continuous representations)

2. The experiments appear to me to be carefully thought through and executed, and the results are well explained and documented.

3. The paper describes related literature well and I feel like I learned some useful things from reading it, so thank you!

# Weaknesses

1. I find the assertion that "robust representations capture peripheral computation similar to current state-of-the-art texture peripheral vision models" a little strong. It's certainly the case that the synthesized metamers have similar (but not matching) behavior curves in the precision vs eccentricity discrimination tasks of Fig 5, but is this sufficient to claim that they capture computation similarly? Is it possible that a model could yield synthesized images which also have similar curves but have nothing to do with how peripheral computation takes place? What if you were to learn a decoder from the representations of robust and non-robust networks, and train it to reconstruct the images from those representations, then use those reconstructions in place of the current synthetic images. Do you think these relationships would hold up? Would that be an invalid way of producing synthetic images?

2. Along that line, I’d like to see some discussion and analysis about the role of optimization in generating the stimuli. In particular, do the Standard and Robust images share the same final L2 minimization error? Does gradient descent find sub-optimal local minima more often for either Standard or Robust? How often? Does it matter? Could the optimization landscape of the non-robust and robust models lead to incorrect experimental conclusions or is it not a concern? Quantitatively, since Standard and Robust models use the same network, what are the distributions of converged L2 distances in both sets of tested stimuli? Similarly, what are the L2 distances vs the Original representation if, for example, you take a Robust image and feed it into a Standard model, and vice versa? Is one model more invariant to the other’s stimuli? 

3. There are various details missing from the experiments section which I think are important. Was vanilla gradient descent used, as currently implied in the paper, or a more adaptive optimizer, e.g. Adam, as in Feather et al., or gradient descent with momentum, as in Mahendran et al.? How was the experiment carried out - remote? in-lab? What controls were in place? Was an eye tracker used to ensure subject fixations? Did all subjects view all stimuli pairs?

4. I would love to understand the role of visual angle in the experiments. It seems the choice was made to map 256x256 pixel stimuli to 7x7 d.v.a. (p4) - how was this decided? Presumably the results are sensitive to this choice -- if say a 14x14 d.v.a. were used instead, would it become easier to discriminate the Robust and Original stimuli? Would it make sense to check that similar trends continue to exist between Texform and Robust stimuli even despite such changes (maybe just causing shifts in the curves in Fig 5)?

5. I think it should be made clearer at an earlier point in the abstract / introduction, why human peripheral vision (as opposed to foveal vision) is important to this problem. I like the clarity of the statement at the bottom of page 3: “we’d like to investigate if the following statement is true: a transformation resembling peripheral computation in the human visual system can closely be approximated by an adversarially trained network”. I think something along these lines should be stated in the abstract to improve readability. Also, I found the term “Biological plausibility” to be a little vague, and unqualified until the Discussion section - and as mentioned above, I am not yet convinced that these experiments definitely "find biological plausibility". 

6. I would like to understand why the choice was made to take the representation at the second-to-last network layer, *after* average pooling, which destroys some potentially useful local information. Did you try to optimize for the representation before average pooling? Or is there a particular reason why doing so would not make sense? I would be very curious to see whether that closes the gap between the robust and non-robust discriminability curves.

# Minor points

1. Figure clarity. The clarity of Fig 4 would be helped by adding what kind of stimulus each one is in the frame (not easy to tell without zooming in a lot). Fig 6 red and pink are a little hard to distinguish on my screen - maybe choose a more distinct color? Fig 7 is similarly difficult to make out without zooming in - a clearer alternative might be e.g. filled and unfilled circles?

2. Sec2.1: “We used gradient descent to minimize the difference between the representation of the second-to-last network layer of a target image and an initial noise seed as shown in Figure 9.” -- rather than “an initial noise seed”, would it be clearer to say “a synthetic image initialized with random noise”?

3. I understand the code used in the paper is already largely public from other sources, but the work may have better impact if the various synthetic and original stimuli could be released alongside the paper, perhaps along with the generation scripts.
 I think this paper is marginally below the acceptance threshold, but this may be due in part to the exposition of the work. I like the effort to try to link these two areas of human peripheral vision and robust representations, and I think the paper does quite a good job to describe the experiments undertaken. If the authors can respond convincingly to the various questions I and other reviewers may have, I would be willing to upgrade my rating.

EDIT: following the rebuttal from the authors and taking into consideration their response to other reviewer comments, I am upgrading my score.

 Thank you to the authors for their thorough rebuttal to all the reviewer comments. I appreciate the responses to my uncertainties and am happy to upgrade my score, although for the benefit of future readers it would be great to see certain clarifications made to the text in line with the authors' responses here. In particular:

- since the others seem to agree that the claim "These results together suggest that (1) robust representations capture peripheral computation similar to current state-of-the-art texture peripheral vision models" is a little strong, it seems fair that this language should be weakened - e.g. robust representations *may* capture, or capture certain important aspects of?

- I agree that the experiments probe a "necessary but not sufficient condition" as the authors mention in several responses, and I think this language should be mentioned in the main text for clarity.

- I would also appreciate seeing the explanation for why the representation was taken in the penultimate layer added to the text, because it is provides useful context to the reader for an otherwise arbitrary choice

- I think that the details regarding optimization are sufficiently important to be stated explicitly in the main text - after all the results may not necessarily hold if you used a different optimizer, step size, number of iterations etc. 

Thank you!

This paper presents a psychophysical comparison between a peripheral summary statistics model ("texforms"), adversarially robust and non-robust neural networks. Images are generated from each model / network using gradient ascent, then presented to humans at different retinal eccentricities. The key finding is that images generated by non-robust networks don't look anything like natural images, and so are easily discriminable, whereas robust and texforms become more difficult to tell apart from natural images as they are moved further into the periphery. The main claim is that because of this similar performance falloff, training on adversarial noise may cause similar representations to be learned as in the human periphery. By comparing the generated images in terms of physical and perceptual distances, the claim is further strengthened as "suggesting their models compute the same transformations". The paper ends with the interesting conjecture that "peripheral computation may implicitly act as a natural visual regularizer". 
 ## Strengths

- The paper raises some interesting ideas linking network robustness with human perception; specifically about the roles of eye movements and peripheral computations in inducing useful invariances
- The psychophysical experiments appear well conducted (though see below)

## Weaknesses

- Purely in terms of writing, I found the motivation and results of the paper difficult to understand until the discussion. The key motivating question posed on page 2 ("Could it be that adversarially trained networks are robust because they encode object representations similar to human peripheral computation?") comes out of the blue. Why would they? We are of course also robust (perhaps more so) in the fovea, so this didn't seem to make sense on the face of it.
- The paper claims that the texform and robust models are doing similar things (or even "the same computations"; Figure 6), based on (1) the finding of similar psychometric functions (Figure 5) and similar IQA scores (Figure 6). This is quite weak. As the paper itself notes, "there are many distortions (derived from the synthesized model stimuli) that can potentially yield the same perceptual sensitivity in a discrimination task". I do not find the IQA analysis convincing either. If I understand correctly, these measure the distance between the original and the synthesized, and synthesized vs synthesized images *within the same model*. But isn't the key issue whether the generated images look similar to each other? Presumably if they are performing "the same computations" they should? I think a stronger way to assess this point would be to perform some variant of "maximum differentiation competition" (MAD; Wang & Simoncelli, 2008). Loosely: synthesize an image that maximally changes the robust synthesis while keeping the texform representation unchanged, and vice versa. If the two models are performing similar computations, then moving in some direction in one model's space while keeping the other fixed should create little perceptible change.
- Missing experimental details. Was the human data collected in a lab or online? Was eye tracking performed to monitor fixation? Was the study conducted according to relevant ethical guidelines and approved by a review board or similar? I could also find no details on this in the supplement.

## Minor

- p. 6 Typo "Mainly Human observers" 
- the differences in the terms in equation 7 were hard to read ($\tilde{x}$ vs $\hat x$ look very similar, so I was confused for a while)
- A relevant reference for the conjectures on page 9 around learning invariances from fovea to periphery is: Nandy, A. S., & Tjan, B. S. (2012). Saccade-confounded image statistics explain visual crowding. *Nature Neuroscience, 15*(3), 463–469.
- p. 9: "Counter-intuitively, the fact that our visual system is spatially-adaptive could give rise to a more robust encoding mechanism of the visual stimulus as observers can encode a distribution rather than a point as they move their center of gaze." -- does this imply the converse, that non-foveated biological vision systems are *less* robust? I don't know of evidence that speaks specifically to this, but there are certainly organisms with non-foveated visual systems that get along just fine in their niche. The paper makes a few interesting arguments linking robustness to foveated visual systems. I find the evidence that the texform and robust models are doing similar / the same computation to be only weakly supported. The story of the paper could also be more clearly constructed.

EDIT after discussion: Given report of new experimental evidence that the texform and robust models generate perceptually-similar images, I am raising my score from 5 to 8.

 The main reason for my previous score of 5 was that I found the evidence supporting the claim that texforms and robust networks compute the same / similar things to be weak. I suggested one way to strengthen evidence for this claim, which the authors report:

> We have added a variation of this experiment with results in the main body of the paper, where we find that there is little low spatial frequency variability across the synthesized samples, and in some cases, the difference is unnoticeable unless doing foveal inspection.

Given this report and the discussion above, I am happy to raise my score.

> We will keep participants' demographics and other apparatus details to the supplement, but are open to suggestions if you think it is better to add this in the main body (we did not want to interrupt flow hence adding it in the current Supplement A.6 via a small reference in page 6 that we will expand on).

I agree this is not important to appear in the manuscript.

 > I downloaded the current PDF linked from OpenReview. I note that there are changes from the original (e.g. the Nandy and Tjan reference is added -- but I think would fit better in Discussion on page 9), but I do not find reference to the new experiment added in the main body of the paper. Have I just missed it (which page?) or has that not appeared in the update yet? In addition, I request that the experimental details given to the reply to s6dV above (e.g. in lab, with no eye-tracker) and also noting human subjects research approval, be added to the manuscript.

> If those changes are made then I would be happy to raise my score.


Thank you again Reviewer teJp! Apologies for the confusion on our part, as we did add such new results + information (the new experimental variation with high resolution samples and a statement of approved IRB details, recruitment process, demographics, and apparatus details in the current Supplementary Material available in the OpenReview Link above (See Appendix A.5 and A.6). We also have additional details of the synthesis procedure across all models and single-observer psychophysics attached in the Supplement. We will definitely be more clear in saying that the experiment was in-lab with no eye-tracker and an approved IRB in the main body. We will keep participants' demographics and other apparatus details to the supplement, but are open to suggestions if you think it is better to add this in the main body (we did not want to interrupt flow hence adding it in the current Supplement A.6 via a small reference in page 6 that we will expand on).

We are a bit bummed that we can no longer make changes to the manuscript at this stage of reviewing to showcase these edits, however we will definitely update our manuscript with the details above and move Nandy and Tjan to the Discussion section on eye-movements and peripheral processing ( [...] and can eventually learn where to make a saccade given candidate object peripheral template ) in the camera ready version if accepted! 

Much appreciated once again for your feedback and insights Reviewer teJp!


 Thank you for your reply. I'm impressed that you have added that experimental variation.

I downloaded the current PDF linked from OpenReview. I note that there are changes from the original (e.g. the Nandy and Tjan reference is added -- but I think would fit better in Discussion on page 9), but I do not find reference to the new experiment added in the main body of the paper. Have I just missed it (which page?) or has that not appeared in the update yet?

In addition, I request that the experimental details given to the reply to s6dV above (e.g. in lab, with no eye-tracker) and also noting human subjects research approval, be added to the manuscript.

If those changes are made then I would be happy to raise my score.

 Thank you for your interest in our paper and your feedback Reviewer teJp! We will address your very important concerns below:

> Purely in terms of writing, I found the motivation and results of the paper difficult to understand until the discussion. The key motivating question posed on page 2 ("Could it be that adversarially trained networks are robust because they encode object representations similar to human peripheral computation?") comes out of the blue. Why would they? We are of course also robust (perhaps more so) in the fovea, so this didn't seem to make sense on the face of it.

This is a bit tricky to articulate as we aimed to write the paper in an accessible way, hoping to engage scientists and engineers with a variety of backgrounds in machine learning, computer vision, and vision science (perceptual psychology) -- and you are right, perhaps this critical question does come-out of the blue. We have re-written some parts of the introduction to smoothen the introduction of this idea. As a minor side-note: your point that humans are more robust (to adversarial noise-like PGD based perturbations) in the fovea than in the periphery has not been tested, and may not necessarily be true given the differences in preference for high-spatial frequency processing (fovea) vs low-spatial frequency processing (periphery). It may in fact be the opposite (see even Ziemba \& Simoncelli 2021), and we are currently testing this in follow-up experiments. 

> “[...] As the paper itself notes, "there are many distortions (derived from the synthesized model stimuli) that can potentially yield the same perceptual sensitivity in a discrimination task".” 

Reviewer U8eW has had the same concern, and so do we (as we write on our paper). You are correct, that perhaps the claim for perceptual isomorphism is strong. We have toned down the claim in our paper, however we do think that the equivalence of the psychometric functions and the IQA metric tests, in addition to the other DeepNet experiments (Figure 7) are “necessary, but not sufficient conditions for a perceptual relationship across transformations”. Our paper presents a first test-bed of a wide-variety of conditions where we will continue to go on a quest to try to falsify this claim. One current follow-up project is performing a similar flavor to these experiments on macaque and getting their V1 and V2 population neural recordings in the fovea and periphery to find any perceptual signatures from these sites and visual areas evoked from robust stimuli.

> “[...] Loosely: synthesize an image that maximally changes the robust synthesis while keeping the texform representation unchanged, and vice versa. If the two models are performing similar computations, then moving in some direction in one model's space while keeping the other fixed should create little perceptible change.”

This is a great suggestion! We have added a variation of this experiment with results in the main body of the paper, where we find that there is little low spatial frequency variability across the synthesized samples, and in some cases, the difference is unnoticeable unless doing foveal inspection.

Thank you once again Reviewer teJP for your feedback and suggestions! We have corrected the typos and added the missing citations. If you think these new updates and comments raise your views on our paper -- in addition to our update manuscript and Supplementary Material --  please feel free to update/raise your score, as we think this paper could generate a lot of new ideas in the computer vision and machine learning community if published in ICLR.


The authors hypothesize that description of visual textures/objects using summary statistics (as may be doing the human vision in the periphery) may induce robustness to adversarial examples. 

In the discussion they argue that this may be the case because imposing separability of classes with high-variance (responses of peripheral vision) leads to enlarged class separation of small-variance sets in foveal vision which have the same mean. 

The authors check their initial hypothesis in an indirect way: they check that robust representations to adversarial attacks resemble human representations. They do so by comparing \tilde{x} (synthesized images from networks which are robust to adversarial examples) and \hat{x} (synthesized images from models of human peripheral vision). They argue that given the fact that these classes of images (\tilde{x} and \hat{x}) are equally discriminable by humans from the original images, x, the image representation of robust networks is similar to the (periphery) image representation in humans. The authors also compare the discriminability of images, x_s, generated from standard (non-robust) networks.

The specific way in which \tilde{x}, \hat{x} and x_s are compared is measuring the psychometric function (over eccentricity) so that these images are discriminable by humans from the original image. Interestingly, these images are also compared using subjective image distortion metrics. While the idea of connecting summary statistics with the robustness to adversarial attacks is interesting, I feel the paper has a number of points that should be clarified by the authors before it can be accepted for publication at ICLR.

1. Two representations with the same null-space are not necessarily the same.

The authors say that the representation of robust nets is similar to the representations of the models of peripheral vision only because the images coming from the inverse of such models are metameric (similarly non-discriminable) for humans.
Note that this only means that these representations share the null space and this null space is also shared by humans. However, this is far from being the same as saying that the representations are equivalent or similar.

An example of this difference can be obtained from color science (where the metamer concept comes from). Imagine the space of spectral radiances with vectors living in a 100-dimensional space (say one value every 3 nm in the 400-700 nm interval).
Consider a perceptual system defined by a linear operator, P, that projects this 100-dimensional space into a 3-dimensional space (P is a 3*100 matrix that performs the integration by the cone sensitivities). Note that this perceptual system, P, has a null-space (the inverse is not unique) and many different spectra lead to the same "tristimulus" representation. We say that these physically different stimuli are metameric (they are the same for the perceptual system P).
They are perceptually same because these different spectra differ in vectors that belong to the null-space of the projection operator P.
With this in mind, if we consider any arbitrary rotation of the 3-d vectors, all these would be *different* color representations, but they would share the same null-space with the same metamers.
Does this mean that all these different color spaces are equivalent?
Not at all: actually, spaces that separate the achromatic information from the chromatic information (as opposed to LMS cones) are closer to the internal physiological and psychophysical representations of color...  And their statistical properties of these opponent spaces are quite different too. Therefore, sharing some null space does not mean having equivalent representations.

Now, imagine the complication of going from the 100 dimensions of spectral radiance, to the millions of dimensions in color images... This complexity implies that the rationale followed in the paper is not obvious.
The authors sould give additional evidences of why their rationale is correct or acknowledge the problems in this assumption.

2. Authors should give additional evidence for the suggested connection between summary statistics and robustness to adversarial attacks.

Specifically, in the discussion [in fig. 8] the authors argue that this conection may come from the fact that imposing separability of classes with high-variance (responses of peripheral vision) leads to enlarged class separation of small-variance sets in foveal vision, with samples that have the same mean. I think this interesting suggestion in the discussion is not justified in any way (or I totally missed the justification -sorry if that is the case-).

3. Generation of \hat{x} implies the minimization of a perceptual distance with regard to \tilde{x} (as explained in Section 2.2, eq. 7). In this situation, who is responsible for the human-like behavior of \tilde{x}?. More specifically, who is responsible for the similar behavior of humans in \hat{x} and \tilde{x}? (a) the fact that the robust representation is human-like (as argued by the authors), or (b) the fact that the images \hat{x} were selected so that they were close to \tilde{x} for humans?.

It is interesting to note that the texture distortion metric DISTS is based in computing summary statistics which are similar in spirit to the synthesis method taken as model of peripheral vision. Consider that the procedure in Freeman and Simoncelli 11 reduces to enforcing the replication of certain statistical descriptions as in the old Portilla and Simoncelli 01 paper. In a similar vein, in DISTS texture similarity is measured according to mean and variance of the responses in different layers of a VGG net, and in fact, in Ding et al. PAMI 21, they also propose enforcing the replication of these descriptors as a texture synthesis method. All this seems quite overlapped.

4. Related to the previous question, Fig. 6, that shows that the perceptual distance of \tilde{x} and \hat{x} wrt x are similar seems like a consequence of the selection of \hat{x} described in section 2.2, no?

5. Moreover, given the fact that x_s are so subjectively different from natural images, the whole psychophysical procedure seems unnecessary. Simple inspection of the synthesized images x_s and \tilde{x} is enough to see that \tilde{x} is closer to whatever \hat{x} and x, and that x_s simply is just not right. As a result, it is not surprising that the psychometric functions for \tilde{x} are more similar to those of \hat{x} than those of x_s. 
In this way, I feel the psychophysical experiments and the perceptual image quality metrics do not add much with regard to the qualitative visualizations shown in Tsipras et al. ICLR 2019.

6. Finally, given the above concerns, I suggest another way of assessing the similarity between the robust networks and human vision. Why not trying to simulate other human tasks? such as in Geirhos et al. ICLR 19 (cited in the work) or classical psychophysical responses as in Martinez-Garcia et al. Frontiers in Neuroscience 2019 (not cited in the work). I am not suggesting to do this now, but I feel a comment on alternative comparisons with humans is due. 

 The attempt to assess the human-nature of the image representation of networks which are robust to adversarial attacks is very interesting both for machine learning as well as for computational neuroscience. However, a number of points need to be clarified before the work can be published at ICLR:

1. Representations with shared null-space (the central idea that is checked in the work) are not necessarily the same: simple linear examples  tell us that they can differ in arbitrary rotations or extra linear invertible transforms, and these may represent quite different qualitative nature!

2. Authors should give additional evidence for the suggested connection between the use of summary statistics and robustness to adversarial attacks (fig. 8).

3. Authors should mention other ways to assess the similarity between the robust networks and humans which may be centered in the response space rather than in the null space.

See extra minor comments at the detailed review.

 Despite the above minor technical point, I think this paper makes interesting points that should be further discussed at the conference!
Thereffore, I raise the score from 6 to 8 and I vote for the acceptance of the paper.

 I was not meaning that a "real" human selected the images \hat{x}. My point does not depend on using a real or a simulated human:
if the selected distortion metric (simulated human) is good (as DISTS is supposed to be) then it is equivalent to using real humans. As a result, the similarity in behavior in the \tilde{x} and the \hat{x} cases maybe also a merit of the algortihm to select \hat{x}, and not only of the algorithm to select \tilde{x}. And in this case, both synthesis algorithms are related (both are based on matching the statistics of a target image).


 Thank you for your very positive review Reviewer 5yZa! We have addressed some of your comments below!

> [...] However, the data also suggests a discrepancy between robust CNNs and peripheral vision, as discriminability remains above chance even for 30 visual degrees. The authors might consider addressing this discrepancy in the text and in subsequent analyses, analyzing what humans see that the model can't.

Yes you are right. We are currently investigating a way to close this gap in another follow-up project. Some work by Feather et al. (Cosyne 2021), suggests that earlier layers of an adversarially trained resnet can already aid in decoding the image class, and it is likely that one could close the gap by rendering model metamers at earlier layers (Feather et al. NeurIPS 2019). The scope of our paper however, is to even see if texform stimuli and robust stimuli are related at all.

> Doesn't fitting the two free parameters of the Freeman & Simoncelli model to the robust CNN weaken the conclusion that the two models are equivalent in their predictive power? What if the Texforms are formed independently of the CNN?

It would not, as the model fitting is only done once before any psychophysics, and estimated for a “simulated observer” with simulated fixation and scaling hyper-parameters. No human was used in the perceptual optimization of Figure 3, but rather a IQA metric (DISTS). The estimated and simulated scaling/fixation values are held *constant* at multiple eccentricities to actually inspect the rate of decay of perceptual discriminability for the texform stimuli wrt robust stimuli for both tasks (2AFC + Oddity) and conditions (Synth vs Synth & Original vs Synth).

> It is unclear to me whether it's fair to generate metamers to be tested in the periphery without simulating visual acuity limitations during stimulus synthesis. The standard CNN might be driven by high-frequency information that isn't even there when the stimulus is presented at 20 or 30 degrees. At least, the authors might want to evaluate posthoc whether the metamers remain metamers under high-pass filtering emulating retinal constraints at the different eccentricities.

We have yet to run a condition where we high-pass filter the stimuli and show them at different eccentricities. We will test this in the future. It is worth noting however, that the standard stimuli does not interact with eccentricity and that observers are at chance at discriminating between standard stimuli samples close to the foveal region (5 deg) away. This would suggest that standard stimuli are missing critical low pass spatial frequency information that is critical in object recognition.

> The synthetic vs. synthetic condition is included without sufficient motivation or discussion. How should this condition be interpreted differently than the natural vs. synthetic condition? This point is particularly relevant for the standard CNN where these two conditions diverge.

This is a great question. The synthetic vs synthetic condition was used in the original works of Freeman & Simoncelli from 2011. Since then Wallis et al. 2016 and Deza et al. 2019 have used a similar paradigm to test for metamerism in addition to a stronger test: original vs synthesized. In this paper the synthesis vs synthesis condition gives us an additional curve to test for the similarities of robust stimuli and texform when they are compared to each other and to original stimuli. In addition, this condition also serves to make the experiment more challenging. We added this in our original manuscript (and revised version) at the end of page 5 and start of page 6. We added a small subsection in the Appendix re-articulating this idea in case it may have flown under the radar.

> Do subject demographics, ethical approval, and subject compensation appear in the text?

We have added a more detailed discussion of the subject demographics, compensation and yes, we did have an IRB approved.

> Open-sourcing the code, stimuli, and behavioral results would increase the impact of this work.

Absolutely, we will open-source the stimuli rendering code, and the per-trial psychophysical data per observer while keeping their ID anonymized (as we initially planned in our IRB). This is the plan to motivate other groups to continue to explore this line of work.

> Learning invariances shared between foveal and peripheral processing is an interesting idea! Perhaps this approach can replace adversarial training. This direction should be pursued further in another study. 

Thank you! This was indeed our initial motivation and a current project in the works that would compliment the results shown in this paper.

> [References]

We have added the suggested references in our updated paper.


Thanks once again Reviewer 5yZa for your valuable insight!

 > [Standard vs Robust Stimuli: Were they optimized correctly/fairly? & How were they optimized?] I’d like to see some discussion and analysis about the role of optimization in generating the stimuli.

We had added this plot in the Supplement. The short answer is yes, they converge to the same loss value and are synthesized with the same loss and constraints. We used the pre-trained models from the work of Santurkar et al. (NeurIPS 2019), and Engstrom et al. (NeurIPS 2019). We used the Engstrom et al. parameters to set the number of iterations on the synthesis procedure, they are publicly available here:

* https://github.com/MadryLab/robustness_applications
* https://github.com/MadryLab/robust_representations

More generally, it is well known that synthesizing stimuli from a Standard Trained Network will yield noise-like images. This has been shown across several papers including the ones above, and also Feather et al. (NeurIPS, 2019), and Ilyas et al. (NeurIPS 2019). The only cases in which the synthetic stimuli does not yield noise-like images when rendered from a Standard Trained Network is when the loss is regularized with expressions such as Total Variation (See for example: Neural Population Synthesis from Bashivan et al., Science 2019)

> [Role of Visual Angle] “ It seems the choice was made to map 256x256 pixel stimuli to 7x7 d.v.a. (p4) - how was this decided? “

We wanted to map 256x256 to a value close to 7x7 degrees of visual angle so as to heuristically find a sweet spot at which the image resolution remains high/intact, but when placed in the periphery it may potentially all lie within as few receptive fields as possible, thus emphasizing the effects of visual crowding. Having an image that is bigger would reduce crowding effects. We do not know how the curves would look like in terms of overlap if the stimuli was bigger -- but our guess is that they would be similar, but that they would not reach close to chance by 30 degrees of retinal eccentricity, but rather a greater value.

The precise value of the d.v.a of the stimuli is 6.67 x 6.67 d.v.a, and this was determined from standard psychophysical protocol where we compute (with trigonometry) the degrees per visual angle given our monitor measurements and the distance of the observer from the monitor. Monitor resolution was 3440x1440 and the monitor size was 80 x 34 cm. We have updated this on our paper and added how we arrived at the stimuli d.v.a value in detail in Appendix A.6 (Apparatus)


> I would like to understand why the choice was made to take the representation at the second-to-last network layer, after average pooling, which destroys some potentially useful local information. Did you try to optimize for the representation before average pooling? Or is there a particular reason why doing so would not make sense? I would be very curious to see whether that closes the gap between the robust and non-robust discriminability curves.

This is a great question. We decided to perform our experiments on the stimuli synthesized after average pooling as we wanted to build off as much as possible from the critical works of Santurkar et al., Ilyas et al., and Engstrom et al. who all show impressive results in terms of human perceptual alignment (without psychophysics) by choosing the post average pooling layer of ResNet-50.

Overall, we certainly agree that there is still more work to do, such as continuing to re-evaluate or psychophysical experiments in the lab by synthesizing more stimuli at different stages of the ResNet-50. There is both critical previous and ongoing work by Feather et al. (NeurIPS 2019, Cosyne 2021) that suggests that using earlier layers would close this gap. However, for us it is impressive already that using the penultimate layer yields such similar psychometric functions for the robust vs the texform stimuli as we manipulate retinal eccentricity and the type of metameric discrimination tasks. 

Also see Supplementary Material section A.2 (Texform Stimuli), where our goal is not to create a new metamer model from robust stimuli. While that would be interesting, it is not the goal of this paper. The goal of this paper is to suggest that the transformations performed in the human visual periphery (loosely) resemble those done in an adversarially robust network. This potentially suggests that learning via peripheral computation (as adversarial training is non-biologically plausible) could lead to adversarial robustness in humans. 

> Additional Points:

Yes, we will be releasing our code, image generation process, and the raw data from each psychophysical trial from the 12 human observers to further increase the research on this topic. We have extended our Supplementary Material, with more details and we encourage you to give it a quick look as it may suggest you to increase your score. 

Thank you once again for your feedback Reviewer U8eW!


 Thank you for your time in reviewing our paper, finding it interesting, and for also posing a very relevant set of follow-up questions with regards to our work. We will provide clarifications with regards to these points, and please feel free to let us know if we may have missed something!

> [...] Is it possible that a model could yield synthesized images which also have similar curves but have nothing to do with how peripheral computation takes place?

We agree and we think this is definitely possible. Indeed, we think that the results present in our experiments show that the perceptual curve similarities should be a necessary but not sufficient condition for perceptual equivalence (or some type of relationship) between the adversarially robust transformations and those performed in human peripheral computation. However, what we find quite remarkable, is that the robust stimuli, which have no prior or inductive bias on localized receptive fields, texture, V2-like computation, visual crowding, etc ... somehow manage to synthesize images that resemble texform stimuli to a high degree by pure virtue of adversarial training(!) 

It is also worth emphasizing that  peripheral computation is not just reminiscent of adaptive gaussian blur in case general readers find it odd that we are using stimuli based on texture-like distortions (See Deza & Konkle, 2021). This fact has not been well known in computer vision, but has been well known in human vision since the last decade (Freeman & Simoncelli, 2011 & Rosenholtz 2011).

> [...] (Strong claim) is this sufficient to claim that they capture computation similarly? 

We agree, and we are indeed currently extending our experiments to neural population recordings of macaque to probe visual responses of area V1 and V2 in the fovea and periphery to see if the texform stimuli vs the robust stimuli yield similar responses -- analogous to the experiments of Ziemba et al. from PNAS 2016 that probe selectivity and tolerance of texture in V2. We think our paper proposes a very first step in the direction of finding a relationship of computations between adversarially robust transformations and peripheral processing, as it sets the stage to now test carefully designed experiments on macaques with robust stimuli with the goal of seeing if we can find perceptual signatures in the periphery and/or V2 via neural recordings of such stimuli.

> "What if you were to learn a decoder from the representations of robust and non-robust networks, and train it to reconstruct the images from those representations, then use those reconstructions in place of the current synthetic images. Do you think these relationships would hold up? Would that be an invalid way of producing synthetic images?"

We think a decoder would not be ideal given that the decoder in itself requires its own training procedure and requires defining a loss function as well. In addition, training a decoder to invert the representation breaks the “Synth vs Synth” discriminability procedure making it intractable as a decoder by definition will make the “synthesis” procedure deterministic as it would only produce one stimuli given a single latent feature vector. Altogether, while the idea is appealing, it introduces more variables to our framework, and finding a null result in the psychophysical procedure (if we are to use a Decoder) may be due to not training the decoder properly rather than not defining the correct feature transformation via the encoder. Altogether, the more variables we would add (e.g. via a Decoder), the more difficult it is to distill why the two psychometric functions would (tentatively) overlap. We hope this response is generally helpful not just for the reviewer, but to the general readers as well who may be thinking about conducting similar experiments in the near future.


 >“Moreover, given the fact that x_s are so subjectively different from natural images, the whole psychophysical procedure seems unnecessary. Simple inspection of the synthesized images x_s and \tilde{x} is enough to see that \tilde{x} is closer to whatever \hat{x} and x, and that x_s simply is just not right. As a result, it is not surprising that the psychometric functions for \tilde{x} are more similar to those of \hat{x} than those of x_s. In this way, I feel the psychophysical experiments and the perceptual image quality metrics do not add much with regard to the qualitative visualizations shown in Tsipras et al. ICLR 2019.”

The goal in our paper is not to say that Adversarially Robust transformations are perceptually aligned to humans (suggested by other in previous work), but rather our goal is to suggest **why** it is perceptually aligned to human visual computation as adversarial training is biologically implausible (hence our title).

The papers of Tsipiras et al., and also those of Engstrom et al., Santurkar et al. and Ilyas et al. have without a doubt been influential to our work. However, we believe the term 'human perceptual alignment' is imprecise (Engstrom et al.) [hence the need for rigorous experiments].  Ilyas et al. also lay out the idea that more human priors might be needed for adversarial robustness and high standard accuracy based on their experiments.  We propose that information regarding peripheral computations could be one of those priors.

As a step toward understanding if our proposal is true, we designed our psychophysics experiment to test the human discriminability of robust stimuli in the visual periphery, which we believe has never been done before.  Our results demonstrate that retinal eccentricity interacts in a similar way with discriminability performance on human observers with the texform and robust stimuli, which may suggest that there is are important similarities between adversarially robust representations and human peripheral vision representations.

In addition, our experiments:
* Manipulate presentation of the stimuli as a function of retinal eccentricity
* Are in-lab (and not over Mechanical Turk) thus providing a more/highly controlled setting for data collection with fixed displays, viewing distance and short presentation times to waive the effect of eye-movements over 12 observers. 
* No eye-tracking device was used due to equipment constraints, but we intentionally chose short presentation times (100ms) to minimize the potential for eye movements.  Participants also used a chin rest to better control viewing conditions
* Are forced-fixation trials (where observers are instructed to fixate at the center of the screen) while engaging in peripheral-to-peripheral template discrimination (oddity task) or foveal-to-peripheral matching (2AFC task).
* Force humans to make discrimination tasks rather than classification tasks. In most studies regarding adversarial images on humans, humans are instructed to *classify* the image class (ex: cat or dog?), while we present an even more fine-grained task where observers must *discriminate* between synthesized or original samples (ex: are these stimuli same or different?)

>“Finally, given the above concerns, I suggest another way of assessing the similarity between the robust networks and human vision. Why not trying to simulate other human tasks? such as in Geirhos et al. ICLR 19 (cited in the work) or classical psychophysical responses as in Martinez-Garcia et al. Frontiers in Neuroscience 2019 (not cited in the work). I am not suggesting to do this now, but I feel a comment on alternative comparisons with humans is due.”

In our updated pre-print, we have cited the paper of Martinez-Garcia, and even the most recent paper by Geirhos et al.  (NeurIPS 2021), that provides additional experiments benchmarking adversarial trained networks with humans on tasks such as shape bias (adversarially trained networks have high shape bias!). However, we want to make clear that our claim in this paper is not to say that adversarially trained networks are more perceptually aligned to humans. This has already been explored in many papers from the group of Aleksander Madry, but rather to provide an even more fine grained connection on how and why this is the case, as adversarial training is non-biologically plausible. Indeed, by establishing a shared null space between adversarial trained models and models of human peripheral computation, we rather propose that localized texture summary statistic representations  -- akin to those found in human peripheral processing -- may drive human invariance to adversarial perturbations.

Thank you once again for your valuable feedback Reviewer s6dV! Please consider checking our updated paper, Supplementary Material, raising your score and/or following-up with more questions in the ongoing rebuttal phase if something is not clear.

 Thank you for your encouraging comments Reviewer s6dV!

>“Two representations with the same null-space are not necessarily the same.”

This is a great point and we agree! We think that the results present in our experiments show that the perceptual curve similarities should be a necessary but not sufficient condition for isomorphic/equivalent computation (or another type of relationship such as sub/super-class relationships) between the adversarially robust transformations, the texform model, and the operations performed in human peripheral computation. We have weakened our tone and clarified the claim in our paper (to not suggest isomorphism, but rather a relationship).

However, by knowing that these models all share the same null-space, this can lead us into investigating more complex questions such as in on-going work where we are testing the responses of macaque on populations of V1 and V2 neurons of the robust stimuli when placed in the fovea and the periphery similar to the works of Ziemba et al. from PNAS 2016.

> “Authors should give additional evidence for the suggested connection between summary statistics and robustness to adversarial attacks.”

Figure 8 is indeed a conjecture figure that we debated whether to include in the paper or not. However, we do provide partial evidence for this conjecture through Figure 7 when we do a 2D projection from the final logit outputs in a 2-class discriminability scenario, and where we can observe wider variance for the adversarially trained network vs the standard trained network when performing recognition on the original stimuli.

There is also a non-trivial idea revealed in Ilyas et al., NeurIPS 2019 where standard training (no pgd, SGD + cross entropy loss) on robust stimuli leads to adversarial robustness.  The degree of adversarial robustness is similar to performing adversarial training on original images. With this in mind, and with the possibility that there is a perceptual alignment with human peripheral computation and robust stimuli as suggested through our psychophysical experiments, it may be the case that these wide-class covariances arise from such perceptual transformations inducing adversarial robustness (Figure 7 & Discussion).

> “Generation of $\hat{x}$ implies the minimization of a perceptual distance with regard to $\tilde{x}$ (as explained in Section 2.2, eq. 7). In this situation, who is responsible for the human-like behavior of $\tilde{x}$?. More specifically, who is responsible for the similar behavior of humans in $\hat{x}$ and $\tilde{x}$? (a) the fact that the robust representation is human-like (as argued by the authors), or (b) the fact that the images $\hat{x}$ were selected so that they were close to $\tilde{x}$ for humans?.”

The human was not involved in the perceptual optimization to define $\hat{x}$. The DISTS metric is responsible for equalizing the distortions of $\tilde{x}$ (robust) wrt original ($x$) and equalizing that distance to $\hat{x}$ (texform) wrt original ($x$). Perhaps the confusion here is the scaling factor and fixation distance parameter in Figure 3. These variable do not correspond to a human but are rather the texform generator parameters borrowed from the Freeman \& Simoncelli Model. In that regard, scaling factor and horizontal fixation are simulated hyperparameters for tuning the texform, and do not involve a human.

> “Related to the previous question, Fig. 6, that shows that the perceptual distance of $\tilde{x}$ and $\hat{x}$ wrt $x$ are similar seems like a consequence of the selection of $\hat{x}$ described in section 2.2, no?”

Yes you are correct, although the Optimization in Figure 3 was done with only a small subset of images (45), while the plot in Figure 6 shows all the stimuli used in our experiments (nearly 900).


This study draws a surprising connection between adversarially trained CNNs and human peripheral perception. Using the synthesized metamers, the authors show that the human ability to discriminate between natural images and their synthetic metamers drops in a similar fashion for metamers generated by inverting adversarially trained CNNs and for metamers generated by an inversion of a well-studied model of peripheral vision (i.e., "Texforms") as display eccentricity increases. This may indicate that adversarially trained models bear some resemblance to human visual processing at the retinal periphery. I enjoyed reading the paper and found the connections it makes inspiring. 

Thoughts:

* Overall, the metameric manipulation seems to make a simple prediction: if a model correctly captures human vision for a given eccentricity, then the metamers should not be discriminable by humans when presented at that eccentricity. The authors interpret the decreasing discriminability as indicative that robust CNNs enjoy a similar relation to peripheral vision as the Freeman & Simoncelli model does. I agree that this is a reasonable interpretation. However, the data also suggests a discrepancy between robust CNNs and peripheral vision, as discriminability remains above chance even for 30 visual degrees. The authors might consider addressing this discrepancy in the text and in subsequent analyses, analyzing what humans see that the model can't.

* Doesn't fitting the two free parameters of the Freeman & Simoncelli model to the robust CNN weaken the conclusion that the two models are equivalent in their predictive power? What if the Texforms are formed independently of the CNN?

* It is unclear to me whether it's fair to generate metamers to be tested in the periphery without simulating visual acuity limitations during stimulus synthesis. The standard CNN might be driven by high-frequency information that isn't even there when the stimulus is presented at 20 or 30 degrees. At least, the authors might want to evaluate posthoc whether the metamers remain metamers under high-pass filtering emulating retinal constraints at the different eccentricities.

* The synthetic vs. synthetic condition is included without sufficient motivation or discussion. How should this condition be interpreted differently than the natural vs. synthetic condition? This point is particularly relevant for the standard CNN where these two conditions diverge.

* Do subject demographics, ethical approval, and subject compensation appear in the text?

* Open-sourcing the code, stimuli, and behavioral results would increase the impact of this work.

* Learning invariances shared between foveal and peripheral processing is an interesting idea! Perhaps this approach can replace adversarial training. This direction should be pursued further in another study. 

Minor points:

* The discussion treats standard CNNs as spatially uniform. However, in practice, this is not the case due to effects related to pooling (Azulay &  Weiss, 2018) and padding (Alsallakh et al., 2020). The latter study even indicates implicit foveation.

* Some of the cited ArXiv citations are outdated and should be replaced with citations of the corresponding conference proceedings.

References:
Alsallakh, B., Kokhlikyan, N., Miglani, V., Yuan, J., & Reblitz-Richardson, O. (2020). Mind the Pad--CNNs can Develop Blind Spots. arXiv preprint arXiv:2010.02178.

Azulay, Aharon, and Yair Weiss. "Why do deep convolutional networks generalize so poorly to small image transformations?." arXiv preprint arXiv:1805.12177 (2018). Although this work has some weaker aspects, this is a mature research project whose acceptance to ICLR would promote more important research into the connections between CNNs, robustness, and peripheral vision. Therefore, I recommend acceptance.

Meta-review: This paper shows that images synthesized to match adversarially robust
representations are similar to original images to humans when viewed 
peripherally.  This was not true for adversarially non-robust
representations.  Additionally the adversarially robust
representations were similar to the texform model image from a model
of human peripheral vision.


Reviewers increased their score a lot during the rebuttal period as  
the authors provided more details on the experiments and agreed to 
tone down some of the claims (especially the strong claim that the
robust representations capture peripheral computation similar to
current SOA texture peripheral vision models).  As well stated by 
reviewer s6dV, two representations with the same null-space are not
necessarily the same.  

With reviewer scores of 8 across the board, reviewers agree that this
is interesting work that should be presented at the conference.  I agree.

Decision: Accept (Spotlight)