Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Prior works generally thought non-robust features, which are vulnerable to small perturbations, are not semantically meaningful but are useful for generalization. This work challenges these traditional beliefs by pointing out that non-robust features can also be human-perception aligned and be less useful for generalization, if these non-robust features are discovered via universal adversarial perturbations (rather than via image-dependent perturbations). Extensive experiments are provided to justified these arguments.




*Pros:

(1) This paper is well written and easy to follow.

(2) The idea of utilizing universal adversarial perturbations to analyze non-robust features is novel, and lead to many interesting findings of non-robust features, e.g., non-robust features can also be semantically meaningful.

(3) Most arguments in this paper are rigorously justified by careful experiment designs. For example, by analyzing the effects of perturbation scaling on model performance, this paper successfully persuades the reviewer to believe universal adversarial perturbations indeed leverage non-robust features.



*Cons:

(1) The reviewer has a major concern about the experiments that show non-robust features can also fail to generalize. For example, Table 1 suggests that with a large base set, non-robust features' generalization decrease. Nonetheless, the reviewer worries that this effect may be caused by the reduced diversity of training samples. That's said, when training with a base set of K (i.e., K training samples will share the sample universal perturbation), the reviewer tends to think the training dataset's "effective" sample size will then be greatly reduced to (total sample size)/K, which will decrease training quality and lead to inferior generalization. To rule out this possibility, the authors should also investigate the setting where the number of universal perturbations is equal to the number of training samples, regardless of the value of the size of base set K. The dataset preparation could be like this: first, we randomly sample K images, and generate a universal perturbation for them; second, we randomly sample one image from K' samples (which successfully fool networks), add the generated universal perturbation, and put it to the new training set; third, repeat the first & second step until we collect enough images in this new training set. Only with the performance analysis using these datasets, we can then confidently justify if non-robust features can generalize.

(2) The reviewer is a little bit curious (and worry) why different batch sizes lead to quite different performance as shown in Figure 8? As already shown in the "Training ImageNet in 1hr" paper (https://arxiv.org/pdf/1706.02677.pdf) and many others, if you set the learning rate to be proportional to the batch size, then these training settings should get nearly the same results. Why this paper shows batch size is a critical hyperparameter for impacting model performance (e.g., in Figure 8, the settings of K=64, lr=0.5 batch size=32 and K=64, lr=1.0, batch size=64 seem to get very different results)?

(3) why this paper only considers the targeted attack setting? will the conclusions here generalizable to the setting of non-targeted attack? Also what is the motivation for showing universal perturbations has spatial invariance (since image-dependent perturbation can also be spatial invariance if such transformation is considered by attackers)? 

In general, the reviewer thinks it is an interesting paper, and is happy to raise the score if the authors successfully address the concerns above. 
review_title: an interesting study on universal adversarial perturbations

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper studies the link between non-robust features and universal adversarial perturbations. This paper shows that universal perturbation leverage non-robust features in data in a different way than standard adversarial attacks. Experiments are based on a universal version of projected gradient descent (PGD). The findings are that universal perturbations are more aligned with visual semantics and human perception that general adversarial attacks. Moreover, it is shown to be difficult to obtain generalisation or transferability between models based on universal signals, as opposed to standard adversarial samples. Generalization seems to decrease with the size of the set used for generating a given universal perturbation, while semantics of the features improve.

Strong points:
- The paper uncovers novel, previously unknown properties of non-robust features.
- The subject of the paper is of interest in general and relevant for the ICLR community.
- The experiments chosen for the exploration are appropriate.
- The paper is clear and well-written.

Concerns:
- While the present work advanced the state of understanding of non-robust features, it seems we are still far from a full picture of the topic. Many of the novel ideas proposed in the paper are still at a hypothesis or reasonable explanation state (e.g., universal perturbations reflect the intersection of non-robust features for the input set they were computed on).

Questions / suggestions:
- It is unclear why universal perturbations being bounded in norm implies that they must be leveraging non-robust features.
- Fig. 4: how is the perturbation shifted? Is is rolled over the edge of the image or is the excess discarded?
- The current work can probably be extended to a similar analysis on adversarial patches.
review_title: Novel insights into the properties of non-robust features 

final_score: rating: 5: Marginally below acceptance threshold
review_content: -> Summary:

The authors investigate universal perturbations for adversarial robustness. They find that while universal perturbations are based on non-robust features, they are more human-aligned and spatially invariant. They also show that they contain less predictive
signal than other non-robust features.

-> Reasons for score:

I recommend weak reject for this paper, as I think the scope and impact of this work is too limited to universal perturbations. While I am not that familiar with interpretability literature, what experiments on generalization seem to show, most of the features learned by models are in fact not universal perturbations. So I am not sure whether findings of this paper would be interesting for broad ICLR community.

-> Pros:

I like investigation of the properties of non-robust features in the context of interpretability by humans. Paper is generally well written and easy to follow. Also, findings in this paper are novel as far as I know.

-> Cons:

While I am not that familiar with interpretability literature, based on my knowledge, I am not so sure of the impact of the findings presented in this work. There are essentially three main findings in this paper: universal perturbations are human aligned, non-robust features can be semantic and universal perturbations contain less signal than other non-robust features. I feel these are all interesting, but I am not sure how surprising they are. Universal perturbations are just a small type of possible adversarial perturbations and it is interesting to investigate their properties. As the experiments here show, most of the non-robust features, the ones that are actually useful for classification, are in fact not universal perturbations. So I am not sure what would be the key takeaway for the broader community that does not work on universal perturbations. 

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper analyzes the existence of non-robust features through the lens of universal perturbations. The concept of non-robust features and universal perturbations is not new, but it is interesting to study the two concepts together. 

Strength: the paper gives a detailed study of the property of non-robust features via universal perturbations. Visualization of qualitative results is provided.

Weakness: 

1. Conclusion is not well supported by the experiment. It is hard to measure what is human-aligned properties, even the concept is not well-defined. By visualization and spatially invariant experiments, it is fine to hint towards this, but seems unconvincing to me.

2. The paper does not offer related improvements for existing models. How would the findings of this paper be beneficial to the models? Or guide the research community to build more robust models, or launch successful attacks?

3. The findings are not that interesting and kind of known to the community. Wouldn't universal perturbations be optimized to be invariant such that they contain less non-robust signal (but more invariant/general signal), otherwise how would it be universal? 

4. The paper defines the adversarial perturbations to be non-robust features, however, the existence of non-robustness features is still an open question. It is one interpretation, but other interpretation also works. For example, it can be the common vulnerability of CNN models, thus the authors need to rethink whether a paper based on this assumption is solid.
review_title: This paper provides detailed analysis of the property of universarial adversarial attacks. It is interesting to study non-robust features and universal perturbations together. 

Meta-review: The reviews were a bit mixed, and there was some concern on the usefulness and actual novelty of this work. On one hand, the authors did a nice job in visualizing their findings and conducting a wealth of interesting experiments. On the other hand, the submission suffers severely from hand-waving definitions and arguments. Many terms were not precisely defined, various hyperparameters were not thoroughly investigated, and yet conclusions were made based on indirect experimental results. The AC agrees with the reviewers that it is not very clear how this work would impact the field. For exploratory work like this one, there is also a great danger that one may simply overfit the observations and squeeze conclusions from thin air. It would be more convincing if the authors could largely quantify their definitions and results. For example: what do we mean by human-aligned? robust / nonrobust feature? (this definition depends on the perturbation size hence needs more elaboration.) Is there any way to quantify the results in Fig 2, including the impact of epsilon? Should these adversarial examples be called universal if their ASR falls below what threshold? Are (some of) the conclusions (e.g. translation invariance, semantic) a direct consequence of the perturbation being universal? At this stage this work would be an excellent workshop paper but a bit more rigor would be needed for publishing at the ICLR conference. 
Decision: Reject