Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes using feature-level adversarial perturbation to explore interpretable adversarial attacks. The proposed method can generate targeted, universal, disguised, physically realizable, and black-box attacks at the ImageNet scale. The method can also be used as an interpretability tool for detecting bugs in networks, and to evaluate this, the paper designs "copy/paste" attacks to cause targeted misclassification. 
strengths_and_weaknesses: Strengths:
+ The paper is well written and easy to follow.
+ The method is evaluated on a large-scale ImageNet dataset.

Weaknesses:
- The evaluation of the "disguise" or "realistic" objective is not very convincing.
- The paper lacks a quantitative comparison with previous methods.
See the detailed questions in the next section.
questions: - Figure 3 is confusing to me. Are all of the methods generating patches or localized perturbations? Why are patches (top) and generalized patches (bottom) shown as a complete picture instead of "a patch" (localized region) in a full image, while the region (middle) shown as a localized patch? I am asking because a patch will accompany by a "location" where the patch is located inside an image. This location will determine whether the patch requires to occlude the salient features of the attacked images, which will tell more about the attack (as shown in Figure 3 middle). In addition, because this paper is about "interpretability tools" as mentioned in the title, where the patch locates inside an image will also help to understand more about the interpretability of the network. 

- Can the method control the location of the region attack, or whether the patch is sensitive to location? As shown in Figure 3 (middle), we can see that the patch seems to occlude the salient features to create the adversarial effect.  

- How effective is the method compared with previous works? The paper seems to lack a quantitative comparison with previous methods. For example, to compare robustness, the paper should use a traditional metric of Attack Success Rate to measure how successful the patch can generalize and evade a large number of images to quantify the "universal" claim as well as to compare with other patch attacks. Similarly, for the "interpretability", how effective is the method compared to other interpretability tools mentioned in the literature in terms of interpreting networks? I recommend the paper conduct further experiments to clarify these points to quantify the superiority of the proposed method.  

- What is the purpose of the "disguise" or "realistic" objective, and how to evaluate that better? I also wonder why the paper considers the "disguise" or "realistic" objective. I assume it is to fool human beings or at least to be realistic to a human. If that is the case, I suggest the paper use a better metric such as similarity metrics rather than simply relying on Inception-V3 or conduct a user study to evaluate this because it is hard to be convinced that the patches shown in Figures 3 and 4  are more realistic than previous patch methods.  

- Minor questions: 
	- In Fig 9. "Attacks further up and right are better?" ==> should it be further up and left?
limitations: I think the paper adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a method for generating feature-level adversarial examples. In the most successful variant of the attacks, adversarial perturbations in representation space of a generative model allow generation of “patches” which will then be added to natural images.The patches are optimized to be classified as “real” by a discriminator, increase entropy of a surrogate classifier, and misclassify the targeted classifier. The authors empirically verified the robustness of the adversarial examples, including its correctness in physical world. Then the authors study how the proposed method can be used for interpreting mistakes made by DNNs.
strengths_and_weaknesses: Strength
The general method of the paper is presented relatively clear, and the idea of using feature-level adversarial examples for model interpretability.

Weakness
The focus of the paper is not clear: the proposed method has its advantage that it can be used as a interpretability tool, whereas as an attack the authors said it may be detectable by a human. However, the paper focus more on the attack than the interpretability. Besides, some important contents are placed in the appendix.

questions: 1. The authors seem to be using a different objective for adversarial examples: they want the examples to be (1) easily-describable, (2) not classified as the target class, but have a rather loose restriction on how much perturbatitions can be made. The authors also agree their attacks may be detectable to humans. Thus to the best of my understanding, the contributions of this work is more on the interpretability side and not on the adversarial ML side, but the authors only talked about interpretability in one subsection--it would be great if the authors can elaborate more on how one could use the proposed method to diagnose the flaws. 
2. Also related to the authors’ definition on disguised adversarial examples--it seems to me that natural images from any classes other than the targeted class also satisfy this definition, so a baseline to add would be use natural images from different classes to replace the adversarial patches. It would be nice to show why the proposed method is better than this baseline since it requires more computationally cost than the baseline.
3. Given the authors suggest this method can be used for interpretability, it would be good to test it on some language dataset since it would give some more intuitive outputs for understanding. This should be doable since there also exist generative language models.
It is worth to clarify what is the advantage of using feature-level adversarial examples for interpretability over other existing interpretability methods.
4. Some important parts of the paper is in the appendix, e.g. the black-box setting. It may be worth to reorganize the paper (e.g. in my mind most subfigures of figure 6 can be moved to the appendix).
The authors said the proposed method could be used for feature visualization. How would it be different from the existing feature visualization techniques?
5. It is not very intuitive to me why the surrogate classifier C’ is needed. Is it meant to make the adversarial examples transferable? 
A minor typo: in figure 4, “Mean Target Confidence” is at the y-axis but in the caption it is called “fooling confidence”.

limitations: The authors stated the limitations of their work in the discussion and claimed that they are future works. Compared to other works on adversarial examples, the negative societal impact of this work is not significant since the generated adversarial examples here are easily detectable by human beings.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors use a pre-trained generative model to automatically identify feature-level adversarial examples, which are adversarial perturbations that are both interpretable to a human and robust (i.e. still serve as adversarial attacks in related contexts). They target image classifier networks. To do this, they use several loss terms: A term that penalizes the predicted image class for being similar to the true image class; a term that penalizes high-frequency patterns; a term that encourages the realistic-ness of the image; a term that encourages the image to look like a particular class; and a term that encourages that class not to be the true class. They demonstrate the utility of all these loss terms with an ablation study. They evaluate three kinds of adversarial attack - (square) patch, region, and generalized (any shape) patch.  Impressively, they show that the attacks generated by their method are physically realizable and also transfer to a held out classifier. They are also able to use the attacks generated by the model to hand-craft copy/paste attacks, which is very useful for interpreting mistakes made by the target network. 
strengths_and_weaknesses: The paper is very high quality, clearly presented, and probably high significance. Although the method itself makes from only minor alterations to previous work, what the paper lacks in originality, it more than makes up for in other factors including extensiveness of analyses, evaluation of societal impact, and provides a valuable tool to neural network interpretability.

A few criticisms of an otherwise great paper:

- It's often hard to identify the perturbation from the original image in e.g. fig 3. That hinders the reader's ability to understand the difference. I appreciate that an additional side-by-side comparison might make the figure unmanageably large, but I encourage the authors to figure out ways to make this key figure more useful to the reader.  
- The description and labels in Fig 3 caption are unclear. "Each patch and generalized patch is labeled with its mean fooling confidence under random insertion in source images (labeled ‘Adv’) and the confidence for the disguise class (labeled ‘Img’)". Is 'disguise class' missing a -d, as in 'disguised class'? And the Img and Adv labels are confusing to me - it isn't perfectly obvious which is the class of the original and which is the predicted class after adversarial attack. 
- The authors should make it even clearer what the difference between their work and Brown et al. (2017) is.  Is it simply that the present work uses a generator and Brown et al. do not? 
- Figure 6 could be clearer. It isn't obvious at a glance which row ought to be paired with which other(s). Moving associated rows closer to each other would solve this. 
- I feel like the 'bad' example in Fig 11 is perhaps a bit contrived: It's not obviously caucasian skin. Certainly not in the patch adversary. And caucasian would not have been my first guess for the skin colour of the region adversary or generalized patch adversary. I'd encourage the authors to find a more convincing example. 

questions: - How easy in general is it to construct copy/paste examples from adversarial examples generated by the method? The copy/paste attacks that were exhibited, were they cherry picked examples that worked? Or were they some of the few that the authors constructed and yet they worked?
- One suggestion for future research:  It is clearly be beneficial for interpretability to know where in the network an error is occurring. Perhaps the authors could regularize the features so that the adversarial features attack a particular layer of the target network.  

limitations: The authors do an exceptionally good job of discussing the limitations and potential social impact of their work. They clearly make as much effort as can be expected of them to mitigate risks. Moreover they also engage constructively with a lay audience. 

Although their work focuses on images, clearly there is much interest in the interpretability and robustness of non-image-based networks, such as language models. It isn't clear that gradient-based optimization (central to their method) will work in that domain due to the difficulty of generating realistic language examples from a generative model that uses an optimizable latent space, as is possible using image GANs. Given the considerable interest in the language domain, the authors might consider providing a slightly more detailed discussion than they currently do.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a new method for generating adversarial perturbations of images. These perturbations operate at a higher "feature-level" rather than at a pixel-level. As such, the perturbations are more understandable to humans and can potentially provide insights into model behavior.
strengths_and_weaknesses: Strengths:

1. Through extensive experiments, the authors demonstrate that their method can provide insights on the behavior of deep-learning-based image classifiers. The perturbations produced by the authors' proposed method are indeed interpretable, and I found it quite interesting to see how the suggested perturbations changed model outputs in the provided examples. As such, I believe this method could be a valuable addition to the model-debugging toolbox to help machine learning practitioners better diagnose spurious correlations learned by their models.
2. To my knowledge (I am not an expert in the adversarial example literature), the method proposed by the authors is novel and the literature review seems very thorough.
3. I generally (with a couple exceptions, see Weaknesses section) found the writing to be high-quality and easy to follow.

Weaknesses:
1. My biggest issue with this work is not with the method itself, but rather with the vocabulary used to describe/frame it. Specifically:
    - To my knowledge (and indeed, this is in the second sentence of the paper), an "adversarial" attack involves modifying an image with a small-norm perturbation such that a deep learning model produces a wildly different output while a human does not realize the image has been modified. In this work the authors describe their method as a form of adversarial attack, even though the method is designed to produce perturbations that are obvious to human observers. For example, I wouldn't expect the "attacked" image in Figure 1d to be correctly classified after such a large perturbation. I would encourage the authors to find another way to frame their method (perhaps focusing more on the interpretability benefits?), as the current framing confused me when I first read the paper.
    - Similarly, I would highly recommend that the authors choose another word instead of "disguised" to describe something that is "perceptible and resembling something other than the target class". On my first reading, I assumed that "disguised" referred to something that is _imperceptible_.
2. The authors may wish to cite/discuss cycle-consistency-based methods for discovering human-interpretable perturbations that change the output of deep learning image classifiers (see e.g. [1] and an application in [2]).

[1]: "Explanation by Progressive Exaggeration" https://arxiv.org/abs/1911.00483
[2]: "AI for radiographic COVID-19 detection selects shortcuts over signal" https://www.nature.com/articles/s42256-021-00338-7
questions: As mentioned in the "Weaknesses" section, I would strongly encourage the authors to use different wording when describing/framing their method.
limitations: The authors discussed the limitations of their work in a satisfactory manner.
review_title: null

Meta-review: metareview: This paper proposed the use of generative models to create feature level adversarial perturbations.  The resulting attack images have unusual transferability properties.  The reviewers agree that the threat model is interesting, and the clarity and thoroughness of the paper is above the bar.
confidence: Certain
Decision: Accept