Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors hypothesize that description of visual textures/objects using summary statistics (as may be doing the human vision in the periphery) may induce robustness to adversarial examples. 

In the discussion they argue that this may be the case because imposing separability of classes with high-variance (responses of peripheral vision) leads to enlarged class separation of small-variance sets in foveal vision which have the same mean. 

The authors check their initial hypothesis in an indirect way: they check that robust representations to adversarial attacks resemble human representations. They do so by comparing \tilde{x} (synthesized images from networks which are robust to adversarial examples) and \hat{x} (synthesized images from models of human peripheral vision). They argue that given the fact that these classes of images (\tilde{x} and \hat{x}) are equally discriminable by humans from the original images, x, the image representation of robust networks is similar to the (periphery) image representation in humans. The authors also compare the discriminability of images, x_s, generated from standard (non-robust) networks.

The specific way in which \tilde{x}, \hat{x} and x_s are compared is measuring the psychometric function (over eccentricity) so that these images are discriminable by humans from the original image. Interestingly, these images are also compared using subjective image distortion metrics.
main_review: While the idea of connecting summary statistics with the robustness to adversarial attacks is interesting, I feel the paper has a number of points that should be clarified by the authors before it can be accepted for publication at ICLR.

1. Two representations with the same null-space are not necessarily the same.

The authors say that the representation of robust nets is similar to the representations of the models of peripheral vision only because the images coming from the inverse of such models are metameric (similarly non-discriminable) for humans.
Note that this only means that these representations share the null space and this null space is also shared by humans. However, this is far from being the same as saying that the representations are equivalent or similar.

An example of this difference can be obtained from color science (where the metamer concept comes from). Imagine the space of spectral radiances with vectors living in a 100-dimensional space (say one value every 3 nm in the 400-700 nm interval).
Consider a perceptual system defined by a linear operator, P, that projects this 100-dimensional space into a 3-dimensional space (P is a 3*100 matrix that performs the integration by the cone sensitivities). Note that this perceptual system, P, has a null-space (the inverse is not unique) and many different spectra lead to the same "tristimulus" representation. We say that these physically different stimuli are metameric (they are the same for the perceptual system P).
They are perceptually same because these different spectra differ in vectors that belong to the null-space of the projection operator P.
With this in mind, if we consider any arbitrary rotation of the 3-d vectors, all these would be *different* color representations, but they would share the same null-space with the same metamers.
Does this mean that all these different color spaces are equivalent?
Not at all: actually, spaces that separate the achromatic information from the chromatic information (as opposed to LMS cones) are closer to the internal physiological and psychophysical representations of color...  And their statistical properties of these opponent spaces are quite different too. Therefore, sharing some null space does not mean having equivalent representations.

Now, imagine the complication of going from the 100 dimensions of spectral radiance, to the millions of dimensions in color images... This complexity implies that the rationale followed in the paper is not obvious.
The authors sould give additional evidences of why their rationale is correct or acknowledge the problems in this assumption.

2. Authors should give additional evidence for the suggested connection between summary statistics and robustness to adversarial attacks.

Specifically, in the discussion [in fig. 8] the authors argue that this conection may come from the fact that imposing separability of classes with high-variance (responses of peripheral vision) leads to enlarged class separation of small-variance sets in foveal vision, with samples that have the same mean. I think this interesting suggestion in the discussion is not justified in any way (or I totally missed the justification -sorry if that is the case-).

3. Generation of \hat{x} implies the minimization of a perceptual distance with regard to \tilde{x} (as explained in Section 2.2, eq. 7). In this situation, who is responsible for the human-like behavior of \tilde{x}?. More specifically, who is responsible for the similar behavior of humans in \hat{x} and \tilde{x}? (a) the fact that the robust representation is human-like (as argued by the authors), or (b) the fact that the images \hat{x} were selected so that they were close to \tilde{x} for humans?.

It is interesting to note that the texture distortion metric DISTS is based in computing summary statistics which are similar in spirit to the synthesis method taken as model of peripheral vision. Consider that the procedure in Freeman and Simoncelli 11 reduces to enforcing the replication of certain statistical descriptions as in the old Portilla and Simoncelli 01 paper. In a similar vein, in DISTS texture similarity is measured according to mean and variance of the responses in different layers of a VGG net, and in fact, in Ding et al. PAMI 21, they also propose enforcing the replication of these descriptors as a texture synthesis method. All this seems quite overlapped.

4. Related to the previous question, Fig. 6, that shows that the perceptual distance of \tilde{x} and \hat{x} wrt x are similar seems like a consequence of the selection of \hat{x} described in section 2.2, no?

5. Moreover, given the fact that x_s are so subjectively different from natural images, the whole psychophysical procedure seems unnecessary. Simple inspection of the synthesized images x_s and \tilde{x} is enough to see that \tilde{x} is closer to whatever \hat{x} and x, and that x_s simply is just not right. As a result, it is not surprising that the psychometric functions for \tilde{x} are more similar to those of \hat{x} than those of x_s. 
In this way, I feel the psychophysical experiments and the perceptual image quality metrics do not add much with regard to the qualitative visualizations shown in Tsipras et al. ICLR 2019.

6. Finally, given the above concerns, I suggest another way of assessing the similarity between the robust networks and human vision. Why not trying to simulate other human tasks? such as in Geirhos et al. ICLR 19 (cited in the work) or classical psychophysical responses as in Martinez-Garcia et al. Frontiers in Neuroscience 2019 (not cited in the work). I am not suggesting to do this now, but I feel a comment on alternative comparisons with humans is due. 


summary_of_the_review: The attempt to assess the human-nature of the image representation of networks which are robust to adversarial attacks is very interesting both for machine learning as well as for computational neuroscience. However, a number of points need to be clarified before the work can be published at ICLR:

1. Representations with shared null-space (the central idea that is checked in the work) are not necessarily the same: simple linear examples  tell us that they can differ in arbitrary rotations or extra linear invertible transforms, and these may represent quite different qualitative nature!

2. Authors should give additional evidence for the suggested connection between the use of summary statistics and robustness to adversarial attacks (fig. 8).

3. Authors should mention other ways to assess the similarity between the robust networks and humans which may be centered in the response space rather than in the null space.

See extra minor comments at the detailed review.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper examines whether an adversarially trained (“robust”) ResNet-50 classification model might better resemble the representation of human peripheral vision than a non-adversarially trained (“non-robust”) model. It does so by means of a human perceptual experiment: test images from Restricted ImageNet (a collapsed subset of ImageNet comprising 9 super-classes) are resynthesized via gradient descent from random noise initial conditions by minimizing the L2 difference in the penultimate feature activation of both a robust and non-robustmodel. 
In addition, the same stimuli are resynthesized according to previously validated models of summary statistic based peripheral computation (Texforms). A range of perceptual discrimination experiments (odd-one-out and two alternative forced-choice) are carried out with human subjects among different pairs of stimuli (original, robust, non-robust and texform) at varying degrees of peripheral eccentricity. The experiments show that the discriminability of robust and texform stimuli vs original stimuli degrade similarly with increasing eccentricity, while non-robust stimuli are relatively easy to pick out vs original stimuli at all levels of eccentricity. 

main_review: # Strengths

1. I thought the topic of the paper was interesting and relevant for ICLR since it probes at a link between the representations learned by adversarially trained image classifiers and the robustness of human peripheral vision, and suggests some possible paths forward (e.g. spatially-adaptive computation of continuous representations)

2. The experiments appear to me to be carefully thought through and executed, and the results are well explained and documented.

3. The paper describes related literature well and I feel like I learned some useful things from reading it, so thank you!

# Weaknesses

1. I find the assertion that "robust representations capture peripheral computation similar to current state-of-the-art texture peripheral vision models" a little strong. It's certainly the case that the synthesized metamers have similar (but not matching) behavior curves in the precision vs eccentricity discrimination tasks of Fig 5, but is this sufficient to claim that they capture computation similarly? Is it possible that a model could yield synthesized images which also have similar curves but have nothing to do with how peripheral computation takes place? What if you were to learn a decoder from the representations of robust and non-robust networks, and train it to reconstruct the images from those representations, then use those reconstructions in place of the current synthetic images. Do you think these relationships would hold up? Would that be an invalid way of producing synthetic images?

2. Along that line, I’d like to see some discussion and analysis about the role of optimization in generating the stimuli. In particular, do the Standard and Robust images share the same final L2 minimization error? Does gradient descent find sub-optimal local minima more often for either Standard or Robust? How often? Does it matter? Could the optimization landscape of the non-robust and robust models lead to incorrect experimental conclusions or is it not a concern? Quantitatively, since Standard and Robust models use the same network, what are the distributions of converged L2 distances in both sets of tested stimuli? Similarly, what are the L2 distances vs the Original representation if, for example, you take a Robust image and feed it into a Standard model, and vice versa? Is one model more invariant to the other’s stimuli? 

3. There are various details missing from the experiments section which I think are important. Was vanilla gradient descent used, as currently implied in the paper, or a more adaptive optimizer, e.g. Adam, as in Feather et al., or gradient descent with momentum, as in Mahendran et al.? How was the experiment carried out - remote? in-lab? What controls were in place? Was an eye tracker used to ensure subject fixations? Did all subjects view all stimuli pairs?

4. I would love to understand the role of visual angle in the experiments. It seems the choice was made to map 256x256 pixel stimuli to 7x7 d.v.a. (p4) - how was this decided? Presumably the results are sensitive to this choice -- if say a 14x14 d.v.a. were used instead, would it become easier to discriminate the Robust and Original stimuli? Would it make sense to check that similar trends continue to exist between Texform and Robust stimuli even despite such changes (maybe just causing shifts in the curves in Fig 5)?

5. I think it should be made clearer at an earlier point in the abstract / introduction, why human peripheral vision (as opposed to foveal vision) is important to this problem. I like the clarity of the statement at the bottom of page 3: “we’d like to investigate if the following statement is true: a transformation resembling peripheral computation in the human visual system can closely be approximated by an adversarially trained network”. I think something along these lines should be stated in the abstract to improve readability. Also, I found the term “Biological plausibility” to be a little vague, and unqualified until the Discussion section - and as mentioned above, I am not yet convinced that these experiments definitely "find biological plausibility". 

6. I would like to understand why the choice was made to take the representation at the second-to-last network layer, *after* average pooling, which destroys some potentially useful local information. Did you try to optimize for the representation before average pooling? Or is there a particular reason why doing so would not make sense? I would be very curious to see whether that closes the gap between the robust and non-robust discriminability curves.

# Minor points

1. Figure clarity. The clarity of Fig 4 would be helped by adding what kind of stimulus each one is in the frame (not easy to tell without zooming in a lot). Fig 6 red and pink are a little hard to distinguish on my screen - maybe choose a more distinct color? Fig 7 is similarly difficult to make out without zooming in - a clearer alternative might be e.g. filled and unfilled circles?

2. Sec2.1: “We used gradient descent to minimize the difference between the representation of the second-to-last network layer of a target image and an initial noise seed as shown in Figure 9.” -- rather than “an initial noise seed”, would it be clearer to say “a synthetic image initialized with random noise”?

3. I understand the code used in the paper is already largely public from other sources, but the work may have better impact if the various synthetic and original stimuli could be released alongside the paper, perhaps along with the generation scripts.

summary_of_the_review: I think this paper is marginally below the acceptance threshold, but this may be due in part to the exposition of the work. I like the effort to try to link these two areas of human peripheral vision and robust representations, and I think the paper does quite a good job to describe the experiments undertaken. If the authors can respond convincingly to the various questions I and other reviewers may have, I would be willing to upgrade my rating.

EDIT: following the rebuttal from the authors and taking into consideration their response to other reviewer comments, I am upgrading my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a psychophysical comparison between a peripheral summary statistics model ("texforms"), adversarially robust and non-robust neural networks. Images are generated from each model / network using gradient ascent, then presented to humans at different retinal eccentricities. The key finding is that images generated by non-robust networks don't look anything like natural images, and so are easily discriminable, whereas robust and texforms become more difficult to tell apart from natural images as they are moved further into the periphery. The main claim is that because of this similar performance falloff, training on adversarial noise may cause similar representations to be learned as in the human periphery. By comparing the generated images in terms of physical and perceptual distances, the claim is further strengthened as "suggesting their models compute the same transformations". The paper ends with the interesting conjecture that "peripheral computation may implicitly act as a natural visual regularizer". 

main_review: ## Strengths

- The paper raises some interesting ideas linking network robustness with human perception; specifically about the roles of eye movements and peripheral computations in inducing useful invariances
- The psychophysical experiments appear well conducted (though see below)

## Weaknesses

- Purely in terms of writing, I found the motivation and results of the paper difficult to understand until the discussion. The key motivating question posed on page 2 ("Could it be that adversarially trained networks are robust because they encode object representations similar to human peripheral computation?") comes out of the blue. Why would they? We are of course also robust (perhaps more so) in the fovea, so this didn't seem to make sense on the face of it.
- The paper claims that the texform and robust models are doing similar things (or even "the same computations"; Figure 6), based on (1) the finding of similar psychometric functions (Figure 5) and similar IQA scores (Figure 6). This is quite weak. As the paper itself notes, "there are many distortions (derived from the synthesized model stimuli) that can potentially yield the same perceptual sensitivity in a discrimination task". I do not find the IQA analysis convincing either. If I understand correctly, these measure the distance between the original and the synthesized, and synthesized vs synthesized images *within the same model*. But isn't the key issue whether the generated images look similar to each other? Presumably if they are performing "the same computations" they should? I think a stronger way to assess this point would be to perform some variant of "maximum differentiation competition" (MAD; Wang & Simoncelli, 2008). Loosely: synthesize an image that maximally changes the robust synthesis while keeping the texform representation unchanged, and vice versa. If the two models are performing similar computations, then moving in some direction in one model's space while keeping the other fixed should create little perceptible change.
- Missing experimental details. Was the human data collected in a lab or online? Was eye tracking performed to monitor fixation? Was the study conducted according to relevant ethical guidelines and approved by a review board or similar? I could also find no details on this in the supplement.

## Minor

- p. 6 Typo "Mainly Human observers" 
- the differences in the terms in equation 7 were hard to read ($\tilde{x}$ vs $\hat x$ look very similar, so I was confused for a while)
- A relevant reference for the conjectures on page 9 around learning invariances from fovea to periphery is: Nandy, A. S., & Tjan, B. S. (2012). Saccade-confounded image statistics explain visual crowding. *Nature Neuroscience, 15*(3), 463–469.
- p. 9: "Counter-intuitively, the fact that our visual system is spatially-adaptive could give rise to a more robust encoding mechanism of the visual stimulus as observers can encode a distribution rather than a point as they move their center of gaze." -- does this imply the converse, that non-foveated biological vision systems are *less* robust? I don't know of evidence that speaks specifically to this, but there are certainly organisms with non-foveated visual systems that get along just fine in their niche.
summary_of_the_review: The paper makes a few interesting arguments linking robustness to foveated visual systems. I find the evidence that the texform and robust models are doing similar / the same computation to be only weakly supported. The story of the paper could also be more clearly constructed.

EDIT after discussion: Given report of new experimental evidence that the texform and robust models generate perceptually-similar images, I am raising my score from 5 to 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This study draws a surprising connection between adversarially trained CNNs and human peripheral perception. Using the synthesized metamers, the authors show that the human ability to discriminate between natural images and their synthetic metamers drops in a similar fashion for metamers generated by inverting adversarially trained CNNs and for metamers generated by an inversion of a well-studied model of peripheral vision (i.e., "Texforms") as display eccentricity increases. This may indicate that adversarially trained models bear some resemblance to human visual processing at the retinal periphery.
main_review: I enjoyed reading the paper and found the connections it makes inspiring. 

Thoughts:

* Overall, the metameric manipulation seems to make a simple prediction: if a model correctly captures human vision for a given eccentricity, then the metamers should not be discriminable by humans when presented at that eccentricity. The authors interpret the decreasing discriminability as indicative that robust CNNs enjoy a similar relation to peripheral vision as the Freeman & Simoncelli model does. I agree that this is a reasonable interpretation. However, the data also suggests a discrepancy between robust CNNs and peripheral vision, as discriminability remains above chance even for 30 visual degrees. The authors might consider addressing this discrepancy in the text and in subsequent analyses, analyzing what humans see that the model can't.

* Doesn't fitting the two free parameters of the Freeman & Simoncelli model to the robust CNN weaken the conclusion that the two models are equivalent in their predictive power? What if the Texforms are formed independently of the CNN?

* It is unclear to me whether it's fair to generate metamers to be tested in the periphery without simulating visual acuity limitations during stimulus synthesis. The standard CNN might be driven by high-frequency information that isn't even there when the stimulus is presented at 20 or 30 degrees. At least, the authors might want to evaluate posthoc whether the metamers remain metamers under high-pass filtering emulating retinal constraints at the different eccentricities.

* The synthetic vs. synthetic condition is included without sufficient motivation or discussion. How should this condition be interpreted differently than the natural vs. synthetic condition? This point is particularly relevant for the standard CNN where these two conditions diverge.

* Do subject demographics, ethical approval, and subject compensation appear in the text?

* Open-sourcing the code, stimuli, and behavioral results would increase the impact of this work.

* Learning invariances shared between foveal and peripheral processing is an interesting idea! Perhaps this approach can replace adversarial training. This direction should be pursued further in another study. 

Minor points:

* The discussion treats standard CNNs as spatially uniform. However, in practice, this is not the case due to effects related to pooling (Azulay &  Weiss, 2018) and padding (Alsallakh et al., 2020). The latter study even indicates implicit foveation.

* Some of the cited ArXiv citations are outdated and should be replaced with citations of the corresponding conference proceedings.

References:
Alsallakh, B., Kokhlikyan, N., Miglani, V., Yuan, J., & Reblitz-Richardson, O. (2020). Mind the Pad--CNNs can Develop Blind Spots. arXiv preprint arXiv:2010.02178.

Azulay, Aharon, and Yair Weiss. "Why do deep convolutional networks generalize so poorly to small image transformations?." arXiv preprint arXiv:1805.12177 (2018).
summary_of_the_review: Although this work has some weaker aspects, this is a mature research project whose acceptance to ICLR would promote more important research into the connections between CNNs, robustness, and peripheral vision. Therefore, I recommend acceptance.
review_title: null

Meta-review: This paper shows that images synthesized to match adversarially robust
representations are similar to original images to humans when viewed 
peripherally.  This was not true for adversarially non-robust
representations.  Additionally the adversarially robust
representations were similar to the texform model image from a model
of human peripheral vision.


Reviewers increased their score a lot during the rebuttal period as  
the authors provided more details on the experiments and agreed to 
tone down some of the claims (especially the strong claim that the
robust representations capture peripheral computation similar to
current SOA texture peripheral vision models).  As well stated by 
reviewer s6dV, two representations with the same null-space are not
necessarily the same.  

With reviewer scores of 8 across the board, reviewers agree that this
is interesting work that should be presented at the conference.  I agree.
Decision: Accept (Spotlight)